Tag: VKS

TKG, Supervisor and VKS Nodes Cannot Resolve .local Hostnames: Cause and Workarounds

Kubernetes nodes on vSphere fail to resolve names ending in .local while the DNS server answers fine. Why systemd-resolved sends .local to mDNS, and the lab-only workarounds for TKG, the Supervisor and VKS clusters (KB 317072, KB 323444).

Short answer: Kubernetes nodes on vSphere (Tanzu Kubernetes Grid, Supervisor control plane VMs and VKS cluster nodes) run Linux with systemd-resolved, which by default sends names ending in .local to Multicast DNS, not to your DNS server. Lookups such as vcenter.corp.local therefore fail even though the DNS server knows the name. The workaround is to add the .local domain as a search domain on the nodes, but Broadcom supports it only for labs and proofs of concept: .local is reserved for mDNS by RFC 6762, and production environments should use a different domain (KB 317072, KB 323444).

Symptoms

  • On TKG or VKS nodes, any name that ends in .local fails to resolve: Temporary failure in name resolution. Other names resolve fine.
  • The Supervisor shows an Error state in Workload Management with a message that a host ending in .local cannot be resolved on a control plane VM and that local must be included in the management DNS search domains.
  • The DNS server is correct, reachable and answers when you ask it directly.

A quick test on a node separates the two layers:

resolvectl status                              # which DNS servers and domains each link uses
resolvectl query vcenter.corp.local            # goes through systemd-resolved: fails
dig @<dns-server-ip> vcenter.corp.local        # asks the DNS server directly: works

If the direct query works and the resolver query fails, the DNS server is fine and the node’s resolver is the problem.

Why it happens

This is not a TKG or VKS bug; it is how systemd-resolved works on any modern Linux. RFC 6762 reserves the .local top-level domain for Multicast DNS on the local link. systemd-resolved follows that: names ending in .local go to mDNS on the interfaces where it is enabled, and they are not sent to unicast DNS servers unless .local (or the specific domain, such as corp.local) is configured as a search or routing domain for that DNS server. Older systems and Windows clients simply asked the DNS server, which is why the same names work from your laptop or from older appliances.

Labs are hit most often, because many lab and nested environments still use .local domains.

Workaround for the Supervisor and VKS clusters (lab only)

KB 323444 has three parts. First, add the .local domain to the Supervisor’s management DNS search domains. On the vCenter appliance as root:

# find the Supervisor cluster ID (for example domain-c8)
dcli com vmware vcenter namespacemanagement clusters list

# current search domains
dcli com vmware vcenter namespacemanagement clusters get --cluster domain-c8 | grep -A 1 master_DNS_search_domains

# set the .local search domain
dcli com vmware vcenter namespacemanagement clusters update --cluster domain-c8 --master-dns-search-domains corp.local

Note that the update sets the list, so include any search domains you already had. Second, on each Supervisor control plane VM, make sure /etc/hosts contains the vCenter FQDN with its IP address. Third, on each VKS cluster node, set the search domain for the node’s interface:

resolvectl domain eth0 corp.local
resolvectl status

The node setting survives reboots but not node replacement: every node created by an upgrade, a scale-out or a machine health check needs it again. That alone makes it a lab-only fix.

Workaround for standalone TKG (lab only)

For standalone TKG management and workload clusters, KB 317072 uses the custom nameserver overlays from the TKG documentation (Nameservers on vSphere). Create the overlay files vsphere-overlay-dns-control-plane.yaml and vsphere-overlay-dns-workers.yaml as described there and add a search domain line at the end of both, so they finish like this:

nameservers: ["10.0.0.53"]
searchDomains: ["corp.local"]

Then deploy the clusters with that plan, so every node gets the search domain at creation, including nodes created later by upgrades.

The real fix: do not use .local

The latest versions of both KBs are clear: Broadcom does not recommend .local for any component, including vCenter, ESXi, NSX Manager, NSX Edges, Avi, Harbor and any endpoint that Kubernetes clusters use. The workarounds are strictly for proof of concept and lab use; they are not validated for production. For a new environment, use a domain you own (for example a subdomain of your company domain) or one reserved for private use such as home.arpa. For an existing production environment on .local, plan the rename before you deploy the Supervisor or VKS.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.

VKS Cluster Stuck in Provisioning: “failed to create volume” with POST /vsanHealth 503 Service Unavailable

A VKS cluster stays in provisioning, control plane nodes run but workers never power on, and volume creation fails with POST /vsanHealth 503. Why the vCenter vsan-health service matters for CNS and how to fix it (KB 409491).

Short answer: When a vSphere Kubernetes Service (VKS) cluster stays in provisioning, its control plane nodes are up but the worker nodes never power on, and the events show failed to create volume. Error: POST “/vsanHealth”: 503 Service Unavailable, the volume service on vCenter is not answering. Cloud Native Storage (CNS), which creates every Kubernetes volume, runs inside the vmware-vsan-health service on the vCenter appliance. Check it with service-control --status vmware-vsan-health and start it if it is stopped. If it is running, check the connection from the Supervisor to vCenter (Broadcom KB 409491).

Symptoms

  • The VKS cluster stays in provisioning for a long time.
  • Control plane nodes are created and powered on; worker nodes are created but do not power on.
  • The cluster description or the namespace events show ProvisioningFailed from csi.vsphere.vmware.com with POST “/vsanHealth”: 503 Service Unavailable, followed by ExternalProvisioning events saying the volume is still being waited for.

Check it from the Supervisor context, in the vSphere Namespace of the cluster:

kubectl describe cluster <cluster-name> -n <namespace>
kubectl get machines -n <namespace>
kubectl get pvc -n <namespace>                     # volumes stuck in Pending
kubectl get events -n <namespace> --sort-by=.lastTimestamp | grep -iE "provision|vsanHealth"

Why a vSAN service blocks a Kubernetes cluster

The name is misleading: you do not need vSAN storage to hit this. CNS, the vCenter component that creates, attaches and deletes container volumes on any datastore, is hosted by the vmware-vsan-health service, and the CSI driver calls it through the /vsanHealth endpoint. A 503 means vCenter accepted the request but the service behind it is stopped, still starting or overloaded.

The cluster needs volumes before its workers can run, for example node volumes defined in the node pool (such as a separate disk for /var/lib/containerd) or persistent volumes that system components claim during bootstrap. Without CNS, those volumes stay Pending and the rollout stops.

Fix: check and start vmware-vsan-health

Open an SSH session to the vCenter appliance and run:

service-control --status vmware-vsan-health

# if it is stopped
service-control --start vmware-vsan-health

# while you are there: any other stopped services?
service-control --status --all

If the service does not start, look at why before you retry: a full appliance partition is a common reason for vCenter services to stop or fail to start (df -h; see VCSA root partition full), and the service log under /var/log/vmware/vsan-health/ shows the error. If it still fails, open a case with Broadcom support.

Once the service is running, the CSI provisioner retries on its own. Watch the volumes bind and the workers power on:

kubectl get pvc -n <namespace> -w
kubectl get machines -n <namespace> -w

The service is running, but the 503 stays

The latest version of the KB adds what to check when vmware-vsan-health is already started:

  • Connectivity between the Supervisor and vCenter, including the workload network VIPs and load balancer: Broadcom KB 388260, Troubleshooting vSphere Supervisor Workload Cluster VIP Connection Issues.
  • vCenter service dependencies: a service can report started while one it depends on is not healthy; Broadcom KB 328009, Check vCenter Server essential service status and dependencies.
  • The general provisioning checks in the TechDocs page Troubleshoot VKS Cluster Provisioning Errors.

It also helps to check the storage policy of the StorageClass named in the error: the policy must be compatible with a datastore that every host in the Supervisor cluster can reach, otherwise volume creation fails even with a healthy CNS.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.

vSphere CSI Node Pods in CrashLoopBackOff with “Unauthorized” on a VKS Cluster: Cause and Fix

New vsphere-csi-node pods on a VKS cluster crash with “failed to get VirtualMachines for the node … Unauthorized”? The CSI controller still uses an old token. Restart the vsphere-csi-controller deployment (KB 411871).

Short answer: When new vsphere-csi-node pods on a vSphere Kubernetes Service (VKS) cluster stay in CrashLoopBackOff and the logs end with failed to get VirtualMachines for the node … Error: Unauthorized, the vSphere CSI controller in that cluster is still using an old token to talk to the Supervisor. Restart the controller so it reads the current credentials: kubectl rollout restart deployment vsphere-csi-controller -n vmware-system-csi. The node pods recover on their own afterwards (Broadcom KB 411871).

How vSphere CSI works in a VKS cluster

A VKS cluster does not talk to vCenter directly for storage. Its vSphere CSI driver (paravirtual CSI) runs in the vmware-system-csi namespace and forwards volume requests to the Supervisor, which talks to vCenter. Two parts matter here:

  • vsphere-csi-controller (Deployment): authenticates to the Supervisor with a token stored in the secret pvcsi-provider-creds in the VKS cluster.
  • vsphere-csi-node (DaemonSet, one pod per node): registers the CSI plugin on its node. During registration it asks for the node’s topology, which needs the node’s VirtualMachine object from the Supervisor.

If the controller keeps an outdated token in memory after the secret has been updated, every call to the Supervisor returns Unauthorized. Existing node pods that registered earlier keep running, which is why the problem usually shows up only on new nodes: after a scale-out, an upgrade rollout or a node replaced by a machine health check.

Recognize the problem

In the VKS cluster context:

kubectl get pods -n vmware-system-csi -o wide

# registration error on a crashing node pod
kubectl logs <vsphere-csi-node-pod> -n vmware-system-csi -c node-driver-registrar

# Unauthorized in the controller
kubectl logs deployment/vsphere-csi-controller -n vmware-system-csi -c vsphere-csi-controller | grep -i unauthorized

# when was the secret last changed, and when did the controller start?
kubectl get secret pvcsi-provider-creds -n vmware-system-csi -o jsonpath='{.metadata.creationTimestamp}{"\n"}'
kubectl get pods -n vmware-system-csi -l app=vsphere-csi-controller

The node pods on the new nodes are in CrashLoopBackOff, and the registrar log says plugin registration failed because the topology of the node could not be retrieved: failed to get VirtualMachines for the node … Unauthorized. Pods with PersistentVolumeClaims cannot start on those nodes, because the CSI plugin is not registered there.

Fix: restart the CSI controller

# VKS cluster context
kubectl rollout restart deployment vsphere-csi-controller -n vmware-system-csi
kubectl rollout status deployment vsphere-csi-controller -n vmware-system-csi

# watch the node pods recover
kubectl get pods -n vmware-system-csi -w

The new controller pods read the current pvcsi-provider-creds secret at start. The crashing node pods retry registration with back-off, so they can need a few minutes to reach Running. To speed it up, restart the DaemonSet as well: kubectl rollout restart daemonset vsphere-csi-node -n vmware-system-csi.

If it does not help

  • Check that the pvcsi-provider-creds secret exists and is not empty. If the Supervisor cannot update it, the problem is on the Supervisor side; look at the cluster status in the Supervisor context.
  • If the error is not Unauthorized but a timeout or connection refused, the CSI controller cannot reach the Supervisor at all; check the network path from the VKS cluster to the Supervisor control plane.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.

VKS Workload Cluster Upgrade Stuck Draining a Node: Pinniped kube-cert-agent and MachineDrainRule

A VKS 3.3+ workload cluster upgrade hangs because an old node never finishes draining: a pinniped kube-cert-agent pod keeps coming back. How to recognize it and fix it with a MachineDrainRule (KB 410900).

Short answer: On vSphere Kubernetes Service (VKS) 3.3 and later, a workload cluster upgrade can stop because one old node never finishes draining. In clusters with an identity provider (Pinniped) enabled, a kube-cert-agent pod is recreated on the draining node every few seconds, so the drain never completes and the upgrade waits forever. The fix from Broadcom KB 410900 is a MachineDrainRule in the Supervisor that tells Cluster API to skip that pod during the drain.

Why the upgrade waits

A VKS cluster upgrade is a rolling replacement driven by Cluster API in the Supervisor: a new node with the target VKr (Kubernetes release) is created, the old node is cordoned and drained, and only then deleted. Control plane nodes go first; worker nodes do not start until every control plane node runs the new version and the old ones are gone. From VKS 3.3 the drain handling changed: if a node does not drain within the cluster’s node drain timeout, the upgrade does not move on.

Pinniped’s kube-cert-agent is the problem pod. It is recreated on the node it lives on, so each time the drain evicts it, a new one appears on the same cordoned node. The drain never reaches zero pods. Deleting the node or the Machine by hand does not help; the controller is still waiting for the drain.

Recognize the problem

In the Supervisor context, one Machine of the cluster stays in Deleting on the previous VKr version, and its conditions say the drain is not finished because of a pinniped pod:

kubectl get machines -n <cluster-namespace>
kubectl describe machine <deleting-machine> -n <cluster-namespace>
#   Type: DrainingSucceeded  Status: False  Reason: Draining
#   Message: Drain not completed yet ... Pod <ns>/...pinniped-kube-cert-agent-...
#            deletionTimestamp set, but still not removed from the Node

In the workload cluster context, the old node is Ready,SchedulingDisabled on the old version, and a pinniped kube-cert-agent pod keeps appearing on it (often in Pending, ContainerCreating or ImagePullBackOff):

kubectl get nodes
kubectl get pods -A -o wide | grep -i pinniped

Note the namespace of the recreating pod. It differs by environment: pinniped-concierge for the pod named pinniped-concierge-kube-cert-agent-…, or vmware-system-tmc for cluster-auth-pinniped-kube-cert-agent-… when the cluster is attached to Tanzu Mission Control.

Fix: a MachineDrainRule in the Supervisor

A MachineDrainRule is a Cluster API object that changes how drains treat selected pods. With behavior: Skip, the drain ignores the matching pods, so the node can finish draining and the upgrade continues. Create it in the Supervisor context, in the vSphere Namespace of the workload cluster.

Save this as skip-pinniped.yaml and replace the namespace with the one you noted above:

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDrainRule
metadata:
  name: vks-pod-drain-skip-pinniped
spec:
  drain:
    behavior: Skip
  pods:
  - namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: <pinniped-pod-namespace>
    selector:
      matchLabels:
        kube-cert-agent.pinniped.dev: v3

Apply it and check that it exists:

# Supervisor context
kubectl apply -f skip-pinniped.yaml -n <cluster-namespace>
kubectl get machinedrainrule -n <cluster-namespace>

# watch the stuck Machine finish and the rollout continue
kubectl get machines -n <cluster-namespace> -w

Within a few minutes the old node finishes draining, its Machine is deleted, and the upgrade moves on to the next node. If pinniped pods run in two namespaces, add a second entry under pods (or a second rule) for the other namespace. A MachineDrainRule without a machine selector applies to every Machine in its vSphere Namespace, so it also covers the other clusters in that namespace and later rollouts, such as the next upgrade or a node replaced by a machine health check.

Good to know

  • The issue affects clusters with an identity provider (Pinniped) enabled on VKS 3.3 and later. Clusters without it drain normally.
  • Do not delete nodes or Machines by hand to push the upgrade on; it does not help and can leave the cluster short of capacity.
  • Check the KB before you apply the rule: Broadcom updates the namespace guidance and may fix the behavior in a later VKS release.

Further reading