Tag: vSphere Supervisor

Supervisor Stuck in Configuring: “ERROR – Configured Core Supervisor Services” with ReconcileFailed

Supervisor stays in Configuring with “ERROR – Configured Core Supervisor Services” and velero or tkg packages in ReconcileFailed because docker-registry.kube-system.svc returns Service Unavailable. The proxy is the cause (KB 411761).

Short answer: When the Supervisor stays in Configuring in Workload Management with ERROR – Configured Core Supervisor Services, and services such as velero.vsphere.vmware.com or tkg.vsphere.vmware.com show ReconcileFailed because https://docker-registry.kube-system.svc:5000/v2/ returns Service Unavailable, the Supervisor’s HTTPS proxy is blocking the package installation. Broadcom’s workaround (KB 411761): disable the HTTPS proxy on the Supervisor, restart the wcp service on vCenter, let the packages reconcile, then enable the proxy again and restart wcp once more.

Symptoms

In the vSphere Client, Workload Management shows the Supervisor in Configuring with an error that lists the core Supervisor services that failed. Each entry has Reason: ReconcileFailed and a message from vendir saying it could not fetch the package image, because the connection to the internal registry docker-registry.kube-system.svc:5000 returned Service Unavailable.

Core Supervisor services, such as the VKS (TKG) service and Velero, are installed as Carvel packages (PackageInstall objects) in the Supervisor. The latest version of the KB shows how to see the failure directly. In the Supervisor context:

# all core service packages and their state
kubectl get pkgi -n vmware-system-supervisor-services

# details of a failing one
kubectl describe pkgi -n vmware-system-supervisor-services svc-velero.vsphere.vmware.com
kubectl describe pkgi -n vmware-system-supervisor-services svc-tkg.vsphere.vmware.com

The status shows the condition ReconcileFailed, and usefulErrorMessage contains the same Unable to create round tripper … docker-registry.kube-system.svc:5000/v2/ … Service Unavailable text.

Why it happens

The package images for the core services are served from a registry inside the Supervisor, reached by its cluster-internal name docker-registry.kube-system.svc. When an HTTPS proxy is configured for the Supervisor, that internal request can be sent to the proxy instead of directly to the registry. The proxy cannot reach a Kubernetes-internal service name, answers Service Unavailable, and every package that needs an image from the internal registry stays in ReconcileFailed. The KB summarizes it simply: enabling the proxy can block package reconciliation.

Fix: proxy off, reconcile, proxy on

  1. Note the current proxy settings, then disable the HTTPS proxy: select the Supervisor, Configure > Network > Proxy Configuration.
  2. On the vCenter appliance, restart the Workload Management service so it reconciles again: service-control --restart wcp
  3. Wait until the packages reconcile; check with kubectl get pkgi -n vmware-system-supervisor-services that Velero, TKG and the others show Reconcile succeeded.
  4. Enable the HTTPS proxy again with the same settings, and restart wcp once more: service-control --restart wcp
  5. Check that all packages still reconcile and that Workload Management shows the Supervisor as Running.

Restarting wcp does not affect running workloads; it restarts the Workload Management service on vCenter, which drives the Supervisor configuration.

Prevent it: check the no-proxy list

The KB only describes the disable/re-enable cycle. To keep it from coming back after the next upgrade or package update, review the No Proxy list in the Supervisor proxy configuration: internal destinations should not go to the proxy. Typical entries are the Kubernetes service domain (.svc and .svc.cluster.local), the Supervisor service and pod CIDRs, the management and workload networks, vCenter and the internal registry. Check the exact list against the Broadcom documentation for your release before you change it.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.

Supervisor Certificates Expired? Check and Rotate vSphere Supervisor Certificates with certmgr

Supervisor control plane and spherelet certificates expire after one year. How to check them, rotate them from vCenter with Broadcom’s certmgr tool, persist the new keys, and which certificates certmgr does not cover (KB 322994).

Short answer: The Kubernetes certificates of a vSphere Supervisor (control plane VMs) and the spherelet certificates on its ESXi hosts are valid for one year. Before they expire, rotate them with Broadcom’s certmgr tool, which is attached to KB 322994 and runs on the vCenter appliance: ./certmgr certificates check-expiration, then ./certmgr certificates rotate, and finally re-encrypt the keys on each control plane VM with /usr/lib/vmware-wcp/hypercrypt.py --reencrypt. Always download the latest certmgr from the KB, because older versions miss certificates.

What expires and what breaks

Kubernetes issues its certificates with a one-year lifetime by default, and the Supervisor follows that. If nobody rotates them, a year after deployment you see the Supervisor go unhealthy: kubectl on the control plane returns Unauthorized, ESXi hosts (the Supervisor worker nodes) turn NotReady, Host Config Status stays Configuring, and vSphere Pods and VKS clusters can no longer be managed. A Supervisor upgrade renews many of the certificates, so environments that are upgraded regularly hit this less often, but do not count on it.

Check the expiration dates

With certmgr on the vCenter appliance (see the next section for how to get it), one command checks every control plane VM:

cd /root
./certmgr supervisors                                   # Supervisor cluster IDs (domain-c...)
./certmgr certificates list -c <cluster-id>            # every cert with NOTAFTER and ISEXPIRED
./certmgr certificates check-expiration --days 60       # only what expires in the next 60 days

Without the tool, SSH to each Supervisor control plane VM and read the dates directly:

# wcp/tls and kubernetes/pki certificates
find /etc -type f \( -name "*.cert" -o -name "*.crt" \) | xargs -L 1 -t -i bash -c 'openssl x509 -noout -enddate -in {}'

# kubelet
openssl x509 -noout -enddate -in /var/lib/kubelet/pki/kubelet.crt

# certificates embedded in admin.conf
grep client-certificate-data /etc/kubernetes/admin.conf | awk '{print $2}' | base64 -d | openssl x509 -noout -enddate

Before you rotate

  • Time is in sync across vCenter, the ESXi hosts and the Supervisor control plane VMs.
  • In Workload Management, both Config Status and Host Config Status of the Supervisor are Running.
  • All Supervisor nodes are Ready.
  • You have the latest wcp_cert_manager.zip from KB 322994. Older builds do not rotate spherelet certificates, which leaves the ESXi nodes NotReady after the rotation.

Rotate with certmgr

Copy the zip to /root on the vCenter appliance, unzip it, and run the tool from /root. certmgr copies itself to the control plane VMs and runs the work there.

scp wcp_cert_manager.zip root@<vcenter>:/root
# on vCenter
cd /root && unzip wcp_cert_manager.zip

# optional: back up the current certificates (directory must exist)
mkdir -p /tmp/cert-backup
./certmgr certificates backup --backup-dir /tmp/cert-backup

# rotate control plane and spherelet certificates
./certmgr certificates rotate -c <cluster-id>           # -c is needed with more than one Supervisor

# only spherelet certificates on the ESXi hosts
./certmgr certificates rotate --spherelet-only -c <cluster-id>

Then make sure the new keys survive a reboot. SSH to each Supervisor control plane VM (the KB 323407 describes how) and run:

/usr/lib/vmware-wcp/hypercrypt.py --reencrypt

On legacy vSphere 7.0 Supervisors, hypercrypt.py has no --reencrypt option; KB 322994 contains a small Python script that does the same job. Finally run ./certmgr certificates list again and check that nothing shows as expired. The tool logs to /var/log/vmware/certmgr.log on vCenter; if it returns nothing or exits with status 127, you are probably not running it from /root.

What certmgr does not cover, and known issues

The latest version of KB 322994 lists several gaps that are worth knowing before you start:

  • authproxy-client.crt and pinniped.crt in /etc/vmware/wcp/tls/ are not rotated by certmgr. They regenerate themselves at two thirds of their lifetime and cannot be replaced with custom certificates. If they have already expired, open a case with Broadcom.
  • Supervisor system pod certificates are managed by the cert-manager pod in the Supervisor and should renew on their own. If they expired, cert-manager itself has a problem; Broadcom has a separate KB for that.
  • admin.conf: some certmgr versions did not always rotate the certificates embedded in admin.conf. If they expire, kubectl on the control plane fails with You must be logged in to the server (Unauthorized), and they must be rotated by hand following the linked KB.
  • kubelet.conf: the embedded certificate was not always rotated either. A control plane node can then go NotReady with unable to load bootstrap kubeconfig; open a case with Broadcom.
  • ESXi hosts NotReady after rotation point to the spherelet certificates; Broadcom has a dedicated KB for that case.
  • --vmca (rotating the VMCA-signed certificates) is marked experimental.

VKS (guest) clusters

The same tool also rotates the control plane certificates of VKS clusters, from vCenter or from a jumpbox with kubectl (KB 323453):

./certmgr tkc list --all-namespaces
./certmgr tkc certificates list <cluster> -n <namespace>
./certmgr tkc certificates rotate <cluster> -n <namespace>

Add a reminder about eleven months after each Supervisor deployment or upgrade, and run check-expiration as part of your regular health checks.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.

Supervisor Deployment or Upgrade Stuck at “Installed and Started Kubernetes Node Agent on the ESXi Host”

Supervisor deployment or upgrade hangs in Configuring Host Nodes with repeated Apply Solution tasks? The spherelet VIB cannot install because the hosts are not compliant with the vLCM image, sometimes because of legacy BIOS boot (KB 393663).

Short answer: When a Supervisor deployment or upgrade hangs in Configuring Host Nodes with the pending task Installed and Started Kubernetes Node Agent on the ESXi Host and the vSphere Client keeps running Apply Solution tasks, the Kubernetes node agent (the spherelet VIB) cannot be installed because the ESXi hosts are not compliant with the cluster’s vSphere Lifecycle Manager (vLCM) image. Fix the image and the compliance errors, including hosts that still boot in legacy BIOS mode, and remediate the hosts; the Supervisor then installs spherelet and continues on its own (Broadcom KB 393663).

Symptoms

  • In Workload Management, the Supervisor shows Host Config Status: Configuring for a long time, during a new deployment or during a Supervisor upgrade.
  • The details list a pending task Installed and Started Kubernetes Node Agent on the ESXi Host, sometimes also Kubernetes Node Agent stopped on the ESXi Host for MaintenanceMode.
  • The vSphere Client shows the same Apply Solution task again and again on the cluster.
  • The Workload Management service log on vCenter says the solution specification in the image is incompatible with hosts and lists the ESXi hosts.

Confirm it on the vCenter appliance and on a host:

# vCenter appliance: Workload Management service log
grep -i "incompatible with hosts" /var/log/vmware/wcp/wcpsvc.log | tail -5

# ESXi host: is the spherelet VIB installed?
esxcli software vib list | grep -i spherelet

Why the Supervisor waits

Every ESXi host of a Supervisor runs spherelet, the agent that turns the host into a Kubernetes node for vSphere Pods. On clusters managed by a vLCM image, the Supervisor does not install the VIB directly: it adds spherelet to the cluster’s desired state as a solution and asks vLCM to apply it. vLCM applies a solution only to hosts that are compliant with the rest of the image. If a host is out of compliance or incompatible, the apply fails, the Supervisor retries, and you see the endless Apply Solution tasks.

Common reasons for non-compliance:

  • The image has a different ESXi version, vendor add-on, firmware add-on or component than the hosts actually run.
  • Hosts have extra VIBs that are not in the image, such as a driver installed by hand.
  • Legacy BIOS boot. The latest version of the KB adds this cause: hosts that still boot in legacy BIOS mode instead of UEFI can be reported as non-compliant, because legacy boot is deprecated in vSphere 8. See Broadcom KB 313152.

Fix: make the hosts compliant

  1. Select the Supervisor cluster, open Updates > Hosts > Image.
  2. Edit the image so it matches what the hosts should run: ESXi version and, for an OEM custom image, the right vendor add-on. Save.
  3. Run Check Compliance. Read the reason for every non-compliant host; searching the exact compliance message in the Broadcom knowledge base usually finds the cause.
  4. Fix the causes: remove or add components in the image, change the boot mode of legacy BIOS hosts to UEFI (plan this carefully; it needs a reinstall or a supported conversion), then Remediate the hosts.
  5. When every host is compliant, the next Apply Solution succeeds, spherelet is installed, and the Supervisor deployment or upgrade continues without any further action.

Tip: run a compliance check on the cluster before you enable the Supervisor or start a Supervisor upgrade. Ten minutes in the Updates tab saves hours of a stuck Workload Management wizard.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.

vSphere Pod Traffic to ClusterIP Times Out on a VDS Supervisor: Cause and Workarounds

On a vSphere 8 Supervisor with VDS networking, vSphere Pods time out to ClusterIP services: ErrImagePull, CrashLoopBackOff or a stuck Supervisor service install. Why the gateway drops it and how to work around it (KB 312199).

Short answer: On a vSphere 8 Supervisor that uses VDS networking (not NSX), traffic from a vSphere Pod to a Kubernetes ClusterIP is sent to the gateway of the workload network, even when the backend sits in the same subnet. If that gateway or a firewall next to it drops traffic that comes back in on the same interface, or sees only half of a TCP session, the pod times out: DNS lookups fail with ErrImagePull, connections to the API fail with CrashLoopBackOff, and Supervisor services such as Harbor or Contour do not install. Allow the hairpin traffic on the gateway, use a simpler internal gateway, and allow kube-dns to reach the upstream DNS servers. Broadcom fixed the behavior in VCF 9.0 (KB 312199).

Symptoms

vSphere Pods run Supervisor services like Harbor, Contour or the image registry, so the problem usually shows up when you install a Supervisor service or upgrade the Supervisor. Typical signs:

  • ErrImagePull right after deployment. kubectl describe pod shows that the image could not be fetched because the DNS lookup of the registry name timed out (UDP), or the HTTPS connection to the registry timed out (TCP 443).
  • CrashLoopBackOff shortly after start. kubectl logs shows a timeout or a TLS handshake timeout when the pod connects to a ClusterIP, for example the Kubernetes API service.
  • ClusterIP fails, backend works. A connection from a vSphere Pod to a ClusterIP times out, while a direct connection to the backend pod IP works.
  • Supervisor upgrade stuck with an ImageRegistryUpgrade timeout, and the image disks for the service are missing in the Supervisor: kubectl get imagedisks -A -o yaml | grep <service-name> returns nothing.

Why it happens

A ClusterIP is a virtual address; it is not in the pod’s subnet. With VDS networking on vSphere 8, a vSphere Pod therefore sends every packet for a ClusterIP to its default gateway, the gateway of the Supervisor workload network. The backend behind the ClusterIP is often in the very same subnet, so the gateway has to send the packet back out of the interface it came in on. Two things can break this:

  • Hairpin routing is blocked. Routing a packet back out of the same interface can be denied by router security features such as reverse-path (RPF/uRPF) checks or a policy against same-interface forwarding. This hits UDP first, so DNS lookups during image fetching fail.
  • Asymmetric TCP. The request goes through the gateway, but the reply from the backend goes straight to the pod, because both are in the same subnet. A stateful firewall on or next to the gateway sees only the first and third packets of the TCP handshake and drops the session as invalid. The connection opens and then nothing passes.

A related variant: kube-dns forwards name lookups to the upstream DNS servers set in the workload network, and a third-party router or firewall between them does not allow that traffic. vSphere Pods then cannot resolve any external name.

Workarounds for vSphere 8

  1. Allow hairpin traffic on the gateway. Change the router and firewall configuration so that any traffic arriving from the workload network interface may be routed back out of the same interface: relax the RPF check and do not drop asymmetric TCP for that subnet.
  2. Use a simpler gateway for the workload network. Instead of the company edge firewall, point the Supervisor workload network to an internal router one hop before it, without stateful filtering on that segment.
  3. Allow kube-dns to reach the upstream DNS. Find the kube-dns address in the Supervisor context and allow DNS (UDP and TCP 53) between it and the DNS servers configured in the workload network:
# Supervisor context
kubectl get services -n kube-system | grep -i dns

After the change, delete the failing pod or retry the Supervisor service installation, and check that the image disks appear with kubectl get imagedisks -A.

Fixed in VCF 9.0

According to the KB, the underlying behavior of vSphere Pods on VDS networking is fixed in VCF 9.0. After you upgrade to 9.0 you can remove the router and firewall workarounds. The issue does not apply to Supervisors that use NSX or VPC networking.

Further reading