Short answer: The Kubernetes certificates of a vSphere Supervisor (control plane VMs) and the spherelet certificates on its ESXi hosts are valid for one year. Before they expire, rotate them with Broadcom’s certmgr tool, which is attached to KB 322994 and runs on the vCenter appliance: ./certmgr certificates check-expiration, then ./certmgr certificates rotate, and finally re-encrypt the keys on each control plane VM with /usr/lib/vmware-wcp/hypercrypt.py --reencrypt. Always download the latest certmgr from the KB, because older versions miss certificates.
What expires and what breaks
Kubernetes issues its certificates with a one-year lifetime by default, and the Supervisor follows that. If nobody rotates them, a year after deployment you see the Supervisor go unhealthy: kubectl on the control plane returns Unauthorized, ESXi hosts (the Supervisor worker nodes) turn NotReady, Host Config Status stays Configuring, and vSphere Pods and VKS clusters can no longer be managed. A Supervisor upgrade renews many of the certificates, so environments that are upgraded regularly hit this less often, but do not count on it.
Check the expiration dates
With certmgr on the vCenter appliance (see the next section for how to get it), one command checks every control plane VM:
cd /root
./certmgr supervisors # Supervisor cluster IDs (domain-c...)
./certmgr certificates list -c <cluster-id> # every cert with NOTAFTER and ISEXPIRED
./certmgr certificates check-expiration --days 60 # only what expires in the next 60 daysWithout the tool, SSH to each Supervisor control plane VM and read the dates directly:
# wcp/tls and kubernetes/pki certificates
find /etc -type f \( -name "*.cert" -o -name "*.crt" \) | xargs -L 1 -t -i bash -c 'openssl x509 -noout -enddate -in {}'
# kubelet
openssl x509 -noout -enddate -in /var/lib/kubelet/pki/kubelet.crt
# certificates embedded in admin.conf
grep client-certificate-data /etc/kubernetes/admin.conf | awk '{print $2}' | base64 -d | openssl x509 -noout -enddateBefore you rotate
- Time is in sync across vCenter, the ESXi hosts and the Supervisor control plane VMs.
- In Workload Management, both Config Status and Host Config Status of the Supervisor are Running.
- All Supervisor nodes are Ready.
- You have the latest
wcp_cert_manager.zipfrom KB 322994. Older builds do not rotate spherelet certificates, which leaves the ESXi nodes NotReady after the rotation.
Rotate with certmgr
Copy the zip to /root on the vCenter appliance, unzip it, and run the tool from /root. certmgr copies itself to the control plane VMs and runs the work there.
scp wcp_cert_manager.zip root@<vcenter>:/root
# on vCenter
cd /root && unzip wcp_cert_manager.zip
# optional: back up the current certificates (directory must exist)
mkdir -p /tmp/cert-backup
./certmgr certificates backup --backup-dir /tmp/cert-backup
# rotate control plane and spherelet certificates
./certmgr certificates rotate -c <cluster-id> # -c is needed with more than one Supervisor
# only spherelet certificates on the ESXi hosts
./certmgr certificates rotate --spherelet-only -c <cluster-id>Then make sure the new keys survive a reboot. SSH to each Supervisor control plane VM (the KB 323407 describes how) and run:
/usr/lib/vmware-wcp/hypercrypt.py --reencryptOn legacy vSphere 7.0 Supervisors, hypercrypt.py has no --reencrypt option; KB 322994 contains a small Python script that does the same job. Finally run ./certmgr certificates list again and check that nothing shows as expired. The tool logs to /var/log/vmware/certmgr.log on vCenter; if it returns nothing or exits with status 127, you are probably not running it from /root.
What certmgr does not cover, and known issues
The latest version of KB 322994 lists several gaps that are worth knowing before you start:
- authproxy-client.crt and pinniped.crt in
/etc/vmware/wcp/tls/are not rotated by certmgr. They regenerate themselves at two thirds of their lifetime and cannot be replaced with custom certificates. If they have already expired, open a case with Broadcom. - Supervisor system pod certificates are managed by the cert-manager pod in the Supervisor and should renew on their own. If they expired, cert-manager itself has a problem; Broadcom has a separate KB for that.
- admin.conf: some certmgr versions did not always rotate the certificates embedded in
admin.conf. If they expire,kubectlon the control plane fails with You must be logged in to the server (Unauthorized), and they must be rotated by hand following the linked KB. - kubelet.conf: the embedded certificate was not always rotated either. A control plane node can then go NotReady with unable to load bootstrap kubeconfig; open a case with Broadcom.
- ESXi hosts NotReady after rotation point to the spherelet certificates; Broadcom has a dedicated KB for that case.
--vmca(rotating the VMCA-signed certificates) is marked experimental.
VKS (guest) clusters
The same tool also rotates the control plane certificates of VKS clusters, from vCenter or from a jumpbox with kubectl (KB 323453):
./certmgr tkc list --all-namespaces
./certmgr tkc certificates list <cluster> -n <namespace>
./certmgr tkc certificates rotate <cluster> -n <namespace>Add a reminder about eleven months after each Supervisor deployment or upgrade, and run check-expiration as part of your regular health checks.
Further reading
More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.
- Broadcom KB 322994: Replace vSphere Supervisor certificates (download certmgr here)
- Broadcom KB 323453: Replace guest cluster certificates
- vCenter and VCF certificate troubleshooting: which tool or fix you need
- Supervisor stuck at “Installed and Started Kubernetes Node Agent on the ESXi Host”