Category: K8S

The “K8S” blog category specializes in Kubernetes in the VMware environment, a key component of modern cloud and container technologies. This blog provides rich content on integrating Kubernetes with VMware platforms, including tutorials, best practices, technical overviews, and case studies. Readers will find valuable information on how to effectively implement and manage Kubernetes clusters within VMware infrastructure, optimize the performance and security of these systems, and integrate Kubernetes with other cloud and virtualization technologies.

Supervisor Stuck in Configuring: “ERROR – Configured Core Supervisor Services” with ReconcileFailed

Supervisor stays in Configuring with “ERROR – Configured Core Supervisor Services” and velero or tkg packages in ReconcileFailed because docker-registry.kube-system.svc returns Service Unavailable. The proxy is the cause (KB 411761).

Short answer: When the Supervisor stays in Configuring in Workload Management with ERROR – Configured Core Supervisor Services, and services such as velero.vsphere.vmware.com or tkg.vsphere.vmware.com show ReconcileFailed because https://docker-registry.kube-system.svc:5000/v2/ returns Service Unavailable, the Supervisor’s HTTPS proxy is blocking the package installation. Broadcom’s workaround (KB 411761): disable the HTTPS proxy on the Supervisor, restart the wcp service on vCenter, let the packages reconcile, then enable the proxy again and restart wcp once more.

Symptoms

In the vSphere Client, Workload Management shows the Supervisor in Configuring with an error that lists the core Supervisor services that failed. Each entry has Reason: ReconcileFailed and a message from vendir saying it could not fetch the package image, because the connection to the internal registry docker-registry.kube-system.svc:5000 returned Service Unavailable.

Core Supervisor services, such as the VKS (TKG) service and Velero, are installed as Carvel packages (PackageInstall objects) in the Supervisor. The latest version of the KB shows how to see the failure directly. In the Supervisor context:

# all core service packages and their state
kubectl get pkgi -n vmware-system-supervisor-services

# details of a failing one
kubectl describe pkgi -n vmware-system-supervisor-services svc-velero.vsphere.vmware.com
kubectl describe pkgi -n vmware-system-supervisor-services svc-tkg.vsphere.vmware.com

The status shows the condition ReconcileFailed, and usefulErrorMessage contains the same Unable to create round tripper … docker-registry.kube-system.svc:5000/v2/ … Service Unavailable text.

Why it happens

The package images for the core services are served from a registry inside the Supervisor, reached by its cluster-internal name docker-registry.kube-system.svc. When an HTTPS proxy is configured for the Supervisor, that internal request can be sent to the proxy instead of directly to the registry. The proxy cannot reach a Kubernetes-internal service name, answers Service Unavailable, and every package that needs an image from the internal registry stays in ReconcileFailed. The KB summarizes it simply: enabling the proxy can block package reconciliation.

Fix: proxy off, reconcile, proxy on

  1. Note the current proxy settings, then disable the HTTPS proxy: select the Supervisor, Configure > Network > Proxy Configuration.
  2. On the vCenter appliance, restart the Workload Management service so it reconciles again: service-control --restart wcp
  3. Wait until the packages reconcile; check with kubectl get pkgi -n vmware-system-supervisor-services that Velero, TKG and the others show Reconcile succeeded.
  4. Enable the HTTPS proxy again with the same settings, and restart wcp once more: service-control --restart wcp
  5. Check that all packages still reconcile and that Workload Management shows the Supervisor as Running.

Restarting wcp does not affect running workloads; it restarts the Workload Management service on vCenter, which drives the Supervisor configuration.

Prevent it: check the no-proxy list

The KB only describes the disable/re-enable cycle. To keep it from coming back after the next upgrade or package update, review the No Proxy list in the Supervisor proxy configuration: internal destinations should not go to the proxy. Typical entries are the Kubernetes service domain (.svc and .svc.cluster.local), the Supervisor service and pod CIDRs, the management and workload networks, vCenter and the internal registry. Check the exact list against the Broadcom documentation for your release before you change it.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.

Supervisor Certificates Expired? Check and Rotate vSphere Supervisor Certificates with certmgr

Supervisor control plane and spherelet certificates expire after one year. How to check them, rotate them from vCenter with Broadcom’s certmgr tool, persist the new keys, and which certificates certmgr does not cover (KB 322994).

Short answer: The Kubernetes certificates of a vSphere Supervisor (control plane VMs) and the spherelet certificates on its ESXi hosts are valid for one year. Before they expire, rotate them with Broadcom’s certmgr tool, which is attached to KB 322994 and runs on the vCenter appliance: ./certmgr certificates check-expiration, then ./certmgr certificates rotate, and finally re-encrypt the keys on each control plane VM with /usr/lib/vmware-wcp/hypercrypt.py --reencrypt. Always download the latest certmgr from the KB, because older versions miss certificates.

What expires and what breaks

Kubernetes issues its certificates with a one-year lifetime by default, and the Supervisor follows that. If nobody rotates them, a year after deployment you see the Supervisor go unhealthy: kubectl on the control plane returns Unauthorized, ESXi hosts (the Supervisor worker nodes) turn NotReady, Host Config Status stays Configuring, and vSphere Pods and VKS clusters can no longer be managed. A Supervisor upgrade renews many of the certificates, so environments that are upgraded regularly hit this less often, but do not count on it.

Check the expiration dates

With certmgr on the vCenter appliance (see the next section for how to get it), one command checks every control plane VM:

cd /root
./certmgr supervisors                                   # Supervisor cluster IDs (domain-c...)
./certmgr certificates list -c <cluster-id>            # every cert with NOTAFTER and ISEXPIRED
./certmgr certificates check-expiration --days 60       # only what expires in the next 60 days

Without the tool, SSH to each Supervisor control plane VM and read the dates directly:

# wcp/tls and kubernetes/pki certificates
find /etc -type f \( -name "*.cert" -o -name "*.crt" \) | xargs -L 1 -t -i bash -c 'openssl x509 -noout -enddate -in {}'

# kubelet
openssl x509 -noout -enddate -in /var/lib/kubelet/pki/kubelet.crt

# certificates embedded in admin.conf
grep client-certificate-data /etc/kubernetes/admin.conf | awk '{print $2}' | base64 -d | openssl x509 -noout -enddate

Before you rotate

  • Time is in sync across vCenter, the ESXi hosts and the Supervisor control plane VMs.
  • In Workload Management, both Config Status and Host Config Status of the Supervisor are Running.
  • All Supervisor nodes are Ready.
  • You have the latest wcp_cert_manager.zip from KB 322994. Older builds do not rotate spherelet certificates, which leaves the ESXi nodes NotReady after the rotation.

Rotate with certmgr

Copy the zip to /root on the vCenter appliance, unzip it, and run the tool from /root. certmgr copies itself to the control plane VMs and runs the work there.

scp wcp_cert_manager.zip root@<vcenter>:/root
# on vCenter
cd /root && unzip wcp_cert_manager.zip

# optional: back up the current certificates (directory must exist)
mkdir -p /tmp/cert-backup
./certmgr certificates backup --backup-dir /tmp/cert-backup

# rotate control plane and spherelet certificates
./certmgr certificates rotate -c <cluster-id>           # -c is needed with more than one Supervisor

# only spherelet certificates on the ESXi hosts
./certmgr certificates rotate --spherelet-only -c <cluster-id>

Then make sure the new keys survive a reboot. SSH to each Supervisor control plane VM (the KB 323407 describes how) and run:

/usr/lib/vmware-wcp/hypercrypt.py --reencrypt

On legacy vSphere 7.0 Supervisors, hypercrypt.py has no --reencrypt option; KB 322994 contains a small Python script that does the same job. Finally run ./certmgr certificates list again and check that nothing shows as expired. The tool logs to /var/log/vmware/certmgr.log on vCenter; if it returns nothing or exits with status 127, you are probably not running it from /root.

What certmgr does not cover, and known issues

The latest version of KB 322994 lists several gaps that are worth knowing before you start:

  • authproxy-client.crt and pinniped.crt in /etc/vmware/wcp/tls/ are not rotated by certmgr. They regenerate themselves at two thirds of their lifetime and cannot be replaced with custom certificates. If they have already expired, open a case with Broadcom.
  • Supervisor system pod certificates are managed by the cert-manager pod in the Supervisor and should renew on their own. If they expired, cert-manager itself has a problem; Broadcom has a separate KB for that.
  • admin.conf: some certmgr versions did not always rotate the certificates embedded in admin.conf. If they expire, kubectl on the control plane fails with You must be logged in to the server (Unauthorized), and they must be rotated by hand following the linked KB.
  • kubelet.conf: the embedded certificate was not always rotated either. A control plane node can then go NotReady with unable to load bootstrap kubeconfig; open a case with Broadcom.
  • ESXi hosts NotReady after rotation point to the spherelet certificates; Broadcom has a dedicated KB for that case.
  • --vmca (rotating the VMCA-signed certificates) is marked experimental.

VKS (guest) clusters

The same tool also rotates the control plane certificates of VKS clusters, from vCenter or from a jumpbox with kubectl (KB 323453):

./certmgr tkc list --all-namespaces
./certmgr tkc certificates list <cluster> -n <namespace>
./certmgr tkc certificates rotate <cluster> -n <namespace>

Add a reminder about eleven months after each Supervisor deployment or upgrade, and run check-expiration as part of your regular health checks.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.

Supervisor Deployment or Upgrade Stuck at “Installed and Started Kubernetes Node Agent on the ESXi Host”

Supervisor deployment or upgrade hangs in Configuring Host Nodes with repeated Apply Solution tasks? The spherelet VIB cannot install because the hosts are not compliant with the vLCM image, sometimes because of legacy BIOS boot (KB 393663).

Short answer: When a Supervisor deployment or upgrade hangs in Configuring Host Nodes with the pending task Installed and Started Kubernetes Node Agent on the ESXi Host and the vSphere Client keeps running Apply Solution tasks, the Kubernetes node agent (the spherelet VIB) cannot be installed because the ESXi hosts are not compliant with the cluster’s vSphere Lifecycle Manager (vLCM) image. Fix the image and the compliance errors, including hosts that still boot in legacy BIOS mode, and remediate the hosts; the Supervisor then installs spherelet and continues on its own (Broadcom KB 393663).

Symptoms

  • In Workload Management, the Supervisor shows Host Config Status: Configuring for a long time, during a new deployment or during a Supervisor upgrade.
  • The details list a pending task Installed and Started Kubernetes Node Agent on the ESXi Host, sometimes also Kubernetes Node Agent stopped on the ESXi Host for MaintenanceMode.
  • The vSphere Client shows the same Apply Solution task again and again on the cluster.
  • The Workload Management service log on vCenter says the solution specification in the image is incompatible with hosts and lists the ESXi hosts.

Confirm it on the vCenter appliance and on a host:

# vCenter appliance: Workload Management service log
grep -i "incompatible with hosts" /var/log/vmware/wcp/wcpsvc.log | tail -5

# ESXi host: is the spherelet VIB installed?
esxcli software vib list | grep -i spherelet

Why the Supervisor waits

Every ESXi host of a Supervisor runs spherelet, the agent that turns the host into a Kubernetes node for vSphere Pods. On clusters managed by a vLCM image, the Supervisor does not install the VIB directly: it adds spherelet to the cluster’s desired state as a solution and asks vLCM to apply it. vLCM applies a solution only to hosts that are compliant with the rest of the image. If a host is out of compliance or incompatible, the apply fails, the Supervisor retries, and you see the endless Apply Solution tasks.

Common reasons for non-compliance:

  • The image has a different ESXi version, vendor add-on, firmware add-on or component than the hosts actually run.
  • Hosts have extra VIBs that are not in the image, such as a driver installed by hand.
  • Legacy BIOS boot. The latest version of the KB adds this cause: hosts that still boot in legacy BIOS mode instead of UEFI can be reported as non-compliant, because legacy boot is deprecated in vSphere 8. See Broadcom KB 313152.

Fix: make the hosts compliant

  1. Select the Supervisor cluster, open Updates > Hosts > Image.
  2. Edit the image so it matches what the hosts should run: ESXi version and, for an OEM custom image, the right vendor add-on. Save.
  3. Run Check Compliance. Read the reason for every non-compliant host; searching the exact compliance message in the Broadcom knowledge base usually finds the cause.
  4. Fix the causes: remove or add components in the image, change the boot mode of legacy BIOS hosts to UEFI (plan this carefully; it needs a reinstall or a supported conversion), then Remediate the hosts.
  5. When every host is compliant, the next Apply Solution succeeds, spherelet is installed, and the Supervisor deployment or upgrade continues without any further action.

Tip: run a compliance check on the cluster before you enable the Supervisor or start a Supervisor upgrade. Ten minutes in the Updates tab saves hours of a stuck Workload Management wizard.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.

TKG, Supervisor and VKS Nodes Cannot Resolve .local Hostnames: Cause and Workarounds

Kubernetes nodes on vSphere fail to resolve names ending in .local while the DNS server answers fine. Why systemd-resolved sends .local to mDNS, and the lab-only workarounds for TKG, the Supervisor and VKS clusters (KB 317072, KB 323444).

Short answer: Kubernetes nodes on vSphere (Tanzu Kubernetes Grid, Supervisor control plane VMs and VKS cluster nodes) run Linux with systemd-resolved, which by default sends names ending in .local to Multicast DNS, not to your DNS server. Lookups such as vcenter.corp.local therefore fail even though the DNS server knows the name. The workaround is to add the .local domain as a search domain on the nodes, but Broadcom supports it only for labs and proofs of concept: .local is reserved for mDNS by RFC 6762, and production environments should use a different domain (KB 317072, KB 323444).

Symptoms

  • On TKG or VKS nodes, any name that ends in .local fails to resolve: Temporary failure in name resolution. Other names resolve fine.
  • The Supervisor shows an Error state in Workload Management with a message that a host ending in .local cannot be resolved on a control plane VM and that local must be included in the management DNS search domains.
  • The DNS server is correct, reachable and answers when you ask it directly.

A quick test on a node separates the two layers:

resolvectl status                              # which DNS servers and domains each link uses
resolvectl query vcenter.corp.local            # goes through systemd-resolved: fails
dig @<dns-server-ip> vcenter.corp.local        # asks the DNS server directly: works

If the direct query works and the resolver query fails, the DNS server is fine and the node’s resolver is the problem.

Why it happens

This is not a TKG or VKS bug; it is how systemd-resolved works on any modern Linux. RFC 6762 reserves the .local top-level domain for Multicast DNS on the local link. systemd-resolved follows that: names ending in .local go to mDNS on the interfaces where it is enabled, and they are not sent to unicast DNS servers unless .local (or the specific domain, such as corp.local) is configured as a search or routing domain for that DNS server. Older systems and Windows clients simply asked the DNS server, which is why the same names work from your laptop or from older appliances.

Labs are hit most often, because many lab and nested environments still use .local domains.

Workaround for the Supervisor and VKS clusters (lab only)

KB 323444 has three parts. First, add the .local domain to the Supervisor’s management DNS search domains. On the vCenter appliance as root:

# find the Supervisor cluster ID (for example domain-c8)
dcli com vmware vcenter namespacemanagement clusters list

# current search domains
dcli com vmware vcenter namespacemanagement clusters get --cluster domain-c8 | grep -A 1 master_DNS_search_domains

# set the .local search domain
dcli com vmware vcenter namespacemanagement clusters update --cluster domain-c8 --master-dns-search-domains corp.local

Note that the update sets the list, so include any search domains you already had. Second, on each Supervisor control plane VM, make sure /etc/hosts contains the vCenter FQDN with its IP address. Third, on each VKS cluster node, set the search domain for the node’s interface:

resolvectl domain eth0 corp.local
resolvectl status

The node setting survives reboots but not node replacement: every node created by an upgrade, a scale-out or a machine health check needs it again. That alone makes it a lab-only fix.

Workaround for standalone TKG (lab only)

For standalone TKG management and workload clusters, KB 317072 uses the custom nameserver overlays from the TKG documentation (Nameservers on vSphere). Create the overlay files vsphere-overlay-dns-control-plane.yaml and vsphere-overlay-dns-workers.yaml as described there and add a search domain line at the end of both, so they finish like this:

nameservers: ["10.0.0.53"]
searchDomains: ["corp.local"]

Then deploy the clusters with that plan, so every node gets the search domain at creation, including nodes created later by upgrades.

The real fix: do not use .local

The latest versions of both KBs are clear: Broadcom does not recommend .local for any component, including vCenter, ESXi, NSX Manager, NSX Edges, Avi, Harbor and any endpoint that Kubernetes clusters use. The workarounds are strictly for proof of concept and lab use; they are not validated for production. For a new environment, use a domain you own (for example a subdomain of your company domain) or one reserved for private use such as home.arpa. For an existing production environment on .local, plan the rename before you deploy the Supervisor or VKS.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.

VKS Cluster Stuck in Provisioning: “failed to create volume” with POST /vsanHealth 503 Service Unavailable

A VKS cluster stays in provisioning, control plane nodes run but workers never power on, and volume creation fails with POST /vsanHealth 503. Why the vCenter vsan-health service matters for CNS and how to fix it (KB 409491).

Short answer: When a vSphere Kubernetes Service (VKS) cluster stays in provisioning, its control plane nodes are up but the worker nodes never power on, and the events show failed to create volume. Error: POST “/vsanHealth”: 503 Service Unavailable, the volume service on vCenter is not answering. Cloud Native Storage (CNS), which creates every Kubernetes volume, runs inside the vmware-vsan-health service on the vCenter appliance. Check it with service-control --status vmware-vsan-health and start it if it is stopped. If it is running, check the connection from the Supervisor to vCenter (Broadcom KB 409491).

Symptoms

  • The VKS cluster stays in provisioning for a long time.
  • Control plane nodes are created and powered on; worker nodes are created but do not power on.
  • The cluster description or the namespace events show ProvisioningFailed from csi.vsphere.vmware.com with POST “/vsanHealth”: 503 Service Unavailable, followed by ExternalProvisioning events saying the volume is still being waited for.

Check it from the Supervisor context, in the vSphere Namespace of the cluster:

kubectl describe cluster <cluster-name> -n <namespace>
kubectl get machines -n <namespace>
kubectl get pvc -n <namespace>                     # volumes stuck in Pending
kubectl get events -n <namespace> --sort-by=.lastTimestamp | grep -iE "provision|vsanHealth"

Why a vSAN service blocks a Kubernetes cluster

The name is misleading: you do not need vSAN storage to hit this. CNS, the vCenter component that creates, attaches and deletes container volumes on any datastore, is hosted by the vmware-vsan-health service, and the CSI driver calls it through the /vsanHealth endpoint. A 503 means vCenter accepted the request but the service behind it is stopped, still starting or overloaded.

The cluster needs volumes before its workers can run, for example node volumes defined in the node pool (such as a separate disk for /var/lib/containerd) or persistent volumes that system components claim during bootstrap. Without CNS, those volumes stay Pending and the rollout stops.

Fix: check and start vmware-vsan-health

Open an SSH session to the vCenter appliance and run:

service-control --status vmware-vsan-health

# if it is stopped
service-control --start vmware-vsan-health

# while you are there: any other stopped services?
service-control --status --all

If the service does not start, look at why before you retry: a full appliance partition is a common reason for vCenter services to stop or fail to start (df -h; see VCSA root partition full), and the service log under /var/log/vmware/vsan-health/ shows the error. If it still fails, open a case with Broadcom support.

Once the service is running, the CSI provisioner retries on its own. Watch the volumes bind and the workers power on:

kubectl get pvc -n <namespace> -w
kubectl get machines -n <namespace> -w

The service is running, but the 503 stays

The latest version of the KB adds what to check when vmware-vsan-health is already started:

  • Connectivity between the Supervisor and vCenter, including the workload network VIPs and load balancer: Broadcom KB 388260, Troubleshooting vSphere Supervisor Workload Cluster VIP Connection Issues.
  • vCenter service dependencies: a service can report started while one it depends on is not healthy; Broadcom KB 328009, Check vCenter Server essential service status and dependencies.
  • The general provisioning checks in the TechDocs page Troubleshoot VKS Cluster Provisioning Errors.

It also helps to check the storage policy of the StorageClass named in the error: the policy must be compatible with a datastore that every host in the Supervisor cluster can reach, otherwise volume creation fails even with a healthy CNS.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.

vSphere CSI Node Pods in CrashLoopBackOff with “Unauthorized” on a VKS Cluster: Cause and Fix

New vsphere-csi-node pods on a VKS cluster crash with “failed to get VirtualMachines for the node … Unauthorized”? The CSI controller still uses an old token. Restart the vsphere-csi-controller deployment (KB 411871).

Short answer: When new vsphere-csi-node pods on a vSphere Kubernetes Service (VKS) cluster stay in CrashLoopBackOff and the logs end with failed to get VirtualMachines for the node … Error: Unauthorized, the vSphere CSI controller in that cluster is still using an old token to talk to the Supervisor. Restart the controller so it reads the current credentials: kubectl rollout restart deployment vsphere-csi-controller -n vmware-system-csi. The node pods recover on their own afterwards (Broadcom KB 411871).

How vSphere CSI works in a VKS cluster

A VKS cluster does not talk to vCenter directly for storage. Its vSphere CSI driver (paravirtual CSI) runs in the vmware-system-csi namespace and forwards volume requests to the Supervisor, which talks to vCenter. Two parts matter here:

  • vsphere-csi-controller (Deployment): authenticates to the Supervisor with a token stored in the secret pvcsi-provider-creds in the VKS cluster.
  • vsphere-csi-node (DaemonSet, one pod per node): registers the CSI plugin on its node. During registration it asks for the node’s topology, which needs the node’s VirtualMachine object from the Supervisor.

If the controller keeps an outdated token in memory after the secret has been updated, every call to the Supervisor returns Unauthorized. Existing node pods that registered earlier keep running, which is why the problem usually shows up only on new nodes: after a scale-out, an upgrade rollout or a node replaced by a machine health check.

Recognize the problem

In the VKS cluster context:

kubectl get pods -n vmware-system-csi -o wide

# registration error on a crashing node pod
kubectl logs <vsphere-csi-node-pod> -n vmware-system-csi -c node-driver-registrar

# Unauthorized in the controller
kubectl logs deployment/vsphere-csi-controller -n vmware-system-csi -c vsphere-csi-controller | grep -i unauthorized

# when was the secret last changed, and when did the controller start?
kubectl get secret pvcsi-provider-creds -n vmware-system-csi -o jsonpath='{.metadata.creationTimestamp}{"\n"}'
kubectl get pods -n vmware-system-csi -l app=vsphere-csi-controller

The node pods on the new nodes are in CrashLoopBackOff, and the registrar log says plugin registration failed because the topology of the node could not be retrieved: failed to get VirtualMachines for the node … Unauthorized. Pods with PersistentVolumeClaims cannot start on those nodes, because the CSI plugin is not registered there.

Fix: restart the CSI controller

# VKS cluster context
kubectl rollout restart deployment vsphere-csi-controller -n vmware-system-csi
kubectl rollout status deployment vsphere-csi-controller -n vmware-system-csi

# watch the node pods recover
kubectl get pods -n vmware-system-csi -w

The new controller pods read the current pvcsi-provider-creds secret at start. The crashing node pods retry registration with back-off, so they can need a few minutes to reach Running. To speed it up, restart the DaemonSet as well: kubectl rollout restart daemonset vsphere-csi-node -n vmware-system-csi.

If it does not help

  • Check that the pvcsi-provider-creds secret exists and is not empty. If the Supervisor cannot update it, the problem is on the Supervisor side; look at the cluster status in the Supervisor context.
  • If the error is not Unauthorized but a timeout or connection refused, the CSI controller cannot reach the Supervisor at all; check the network path from the VKS cluster to the Supervisor control plane.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.

vSphere Pod Traffic to ClusterIP Times Out on a VDS Supervisor: Cause and Workarounds

On a vSphere 8 Supervisor with VDS networking, vSphere Pods time out to ClusterIP services: ErrImagePull, CrashLoopBackOff or a stuck Supervisor service install. Why the gateway drops it and how to work around it (KB 312199).

Short answer: On a vSphere 8 Supervisor that uses VDS networking (not NSX), traffic from a vSphere Pod to a Kubernetes ClusterIP is sent to the gateway of the workload network, even when the backend sits in the same subnet. If that gateway or a firewall next to it drops traffic that comes back in on the same interface, or sees only half of a TCP session, the pod times out: DNS lookups fail with ErrImagePull, connections to the API fail with CrashLoopBackOff, and Supervisor services such as Harbor or Contour do not install. Allow the hairpin traffic on the gateway, use a simpler internal gateway, and allow kube-dns to reach the upstream DNS servers. Broadcom fixed the behavior in VCF 9.0 (KB 312199).

Symptoms

vSphere Pods run Supervisor services like Harbor, Contour or the image registry, so the problem usually shows up when you install a Supervisor service or upgrade the Supervisor. Typical signs:

  • ErrImagePull right after deployment. kubectl describe pod shows that the image could not be fetched because the DNS lookup of the registry name timed out (UDP), or the HTTPS connection to the registry timed out (TCP 443).
  • CrashLoopBackOff shortly after start. kubectl logs shows a timeout or a TLS handshake timeout when the pod connects to a ClusterIP, for example the Kubernetes API service.
  • ClusterIP fails, backend works. A connection from a vSphere Pod to a ClusterIP times out, while a direct connection to the backend pod IP works.
  • Supervisor upgrade stuck with an ImageRegistryUpgrade timeout, and the image disks for the service are missing in the Supervisor: kubectl get imagedisks -A -o yaml | grep <service-name> returns nothing.

Why it happens

A ClusterIP is a virtual address; it is not in the pod’s subnet. With VDS networking on vSphere 8, a vSphere Pod therefore sends every packet for a ClusterIP to its default gateway, the gateway of the Supervisor workload network. The backend behind the ClusterIP is often in the very same subnet, so the gateway has to send the packet back out of the interface it came in on. Two things can break this:

  • Hairpin routing is blocked. Routing a packet back out of the same interface can be denied by router security features such as reverse-path (RPF/uRPF) checks or a policy against same-interface forwarding. This hits UDP first, so DNS lookups during image fetching fail.
  • Asymmetric TCP. The request goes through the gateway, but the reply from the backend goes straight to the pod, because both are in the same subnet. A stateful firewall on or next to the gateway sees only the first and third packets of the TCP handshake and drops the session as invalid. The connection opens and then nothing passes.

A related variant: kube-dns forwards name lookups to the upstream DNS servers set in the workload network, and a third-party router or firewall between them does not allow that traffic. vSphere Pods then cannot resolve any external name.

Workarounds for vSphere 8

  1. Allow hairpin traffic on the gateway. Change the router and firewall configuration so that any traffic arriving from the workload network interface may be routed back out of the same interface: relax the RPF check and do not drop asymmetric TCP for that subnet.
  2. Use a simpler gateway for the workload network. Instead of the company edge firewall, point the Supervisor workload network to an internal router one hop before it, without stateful filtering on that segment.
  3. Allow kube-dns to reach the upstream DNS. Find the kube-dns address in the Supervisor context and allow DNS (UDP and TCP 53) between it and the DNS servers configured in the workload network:
# Supervisor context
kubectl get services -n kube-system | grep -i dns

After the change, delete the failing pod or retry the Supervisor service installation, and check that the image disks appear with kubectl get imagedisks -A.

Fixed in VCF 9.0

According to the KB, the underlying behavior of vSphere Pods on VDS networking is fixed in VCF 9.0. After you upgrade to 9.0 you can remove the router and firewall workarounds. The issue does not apply to Supervisors that use NSX or VPC networking.

Further reading

VKS Workload Cluster Upgrade Stuck Draining a Node: Pinniped kube-cert-agent and MachineDrainRule

A VKS 3.3+ workload cluster upgrade hangs because an old node never finishes draining: a pinniped kube-cert-agent pod keeps coming back. How to recognize it and fix it with a MachineDrainRule (KB 410900).

Short answer: On vSphere Kubernetes Service (VKS) 3.3 and later, a workload cluster upgrade can stop because one old node never finishes draining. In clusters with an identity provider (Pinniped) enabled, a kube-cert-agent pod is recreated on the draining node every few seconds, so the drain never completes and the upgrade waits forever. The fix from Broadcom KB 410900 is a MachineDrainRule in the Supervisor that tells Cluster API to skip that pod during the drain.

Why the upgrade waits

A VKS cluster upgrade is a rolling replacement driven by Cluster API in the Supervisor: a new node with the target VKr (Kubernetes release) is created, the old node is cordoned and drained, and only then deleted. Control plane nodes go first; worker nodes do not start until every control plane node runs the new version and the old ones are gone. From VKS 3.3 the drain handling changed: if a node does not drain within the cluster’s node drain timeout, the upgrade does not move on.

Pinniped’s kube-cert-agent is the problem pod. It is recreated on the node it lives on, so each time the drain evicts it, a new one appears on the same cordoned node. The drain never reaches zero pods. Deleting the node or the Machine by hand does not help; the controller is still waiting for the drain.

Recognize the problem

In the Supervisor context, one Machine of the cluster stays in Deleting on the previous VKr version, and its conditions say the drain is not finished because of a pinniped pod:

kubectl get machines -n <cluster-namespace>
kubectl describe machine <deleting-machine> -n <cluster-namespace>
#   Type: DrainingSucceeded  Status: False  Reason: Draining
#   Message: Drain not completed yet ... Pod <ns>/...pinniped-kube-cert-agent-...
#            deletionTimestamp set, but still not removed from the Node

In the workload cluster context, the old node is Ready,SchedulingDisabled on the old version, and a pinniped kube-cert-agent pod keeps appearing on it (often in Pending, ContainerCreating or ImagePullBackOff):

kubectl get nodes
kubectl get pods -A -o wide | grep -i pinniped

Note the namespace of the recreating pod. It differs by environment: pinniped-concierge for the pod named pinniped-concierge-kube-cert-agent-…, or vmware-system-tmc for cluster-auth-pinniped-kube-cert-agent-… when the cluster is attached to Tanzu Mission Control.

Fix: a MachineDrainRule in the Supervisor

A MachineDrainRule is a Cluster API object that changes how drains treat selected pods. With behavior: Skip, the drain ignores the matching pods, so the node can finish draining and the upgrade continues. Create it in the Supervisor context, in the vSphere Namespace of the workload cluster.

Save this as skip-pinniped.yaml and replace the namespace with the one you noted above:

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDrainRule
metadata:
  name: vks-pod-drain-skip-pinniped
spec:
  drain:
    behavior: Skip
  pods:
  - namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: <pinniped-pod-namespace>
    selector:
      matchLabels:
        kube-cert-agent.pinniped.dev: v3

Apply it and check that it exists:

# Supervisor context
kubectl apply -f skip-pinniped.yaml -n <cluster-namespace>
kubectl get machinedrainrule -n <cluster-namespace>

# watch the stuck Machine finish and the rollout continue
kubectl get machines -n <cluster-namespace> -w

Within a few minutes the old node finishes draining, its Machine is deleted, and the upgrade moves on to the next node. If pinniped pods run in two namespaces, add a second entry under pods (or a second rule) for the other namespace. A MachineDrainRule without a machine selector applies to every Machine in its vSphere Namespace, so it also covers the other clusters in that namespace and later rollouts, such as the next upgrade or a node replaced by a machine health check.

Good to know

  • The issue affects clusters with an identity provider (Pinniped) enabled on VKS 3.3 and later. Clusters without it drain normally.
  • Do not delete nodes or Machines by hand to push the upgrade on; it does not help and can leave the cluster short of capacity.
  • Check the KB before you apply the rule: Broadcom updates the namespace guidance and may fix the behavior in a later VKS release.

Further reading

Tanzu & Kubernetes for vSphere Admins: Complete…

Tanzu & Kubernetes for vSphere Admins: Complete Guided Workshop (HOL-2413-01-SDC)

Tanzu & Kubernetes for vSphere Admins: Complete…

Link to Lab: https://userui.learningplatform.vmware.com/HOL/catalog/lab/13926 🌟 Welcome to our VMware Expert-Led Guided Workshop Series! 🌟 In this series, you’ll explore various VMware Hands-On Labs, each led by our team of VMware experts. Dive deep into different VMware solutions, gaining hands-on experience and valuable insights! 🔹 Featured Lab: Tanzu and Kubernetes for vSphere Administrators (HOL-2413-01-SDC) 🔹 This workshop covers: 1️⃣ Module 1: Introduction to vSphere with Tanzu (30 min, Level: Intermediate) Get started with the basics of vSphere and Tanzu integration. 2️⃣ Module 2: What’s New for vSphere 8 Update 1 (15 min, Level: Beginner) Discover the latest updates and features in vSphere 8. 3️⃣ Module 3: Managing vSphere with Tanzu…Read More

VMware Social Media Advocacy