Rotate vSphere Supervisor Certificates with certmgr

Short answer: The Kubernetes certificates of a vSphere Supervisor (control plane VMs) and the spherelet certificates on its ESXi hosts are valid for one year. Before they expire, rotate them with Broadcom’s certmgr tool, which is attached to KB 322994 and runs on the vCenter appliance: ./certmgr certificates check-expiration, then ./certmgr certificates rotate, and finally re-encrypt the keys on each control plane VM with /usr/lib/vmware-wcp/hypercrypt.py --reencrypt. Always download the latest certmgr from the KB, because older versions miss certificates.

Table of Contents

What expires and what breaks

Kubernetes issues its certificates with a one-year lifetime by default, and the Supervisor follows that. If nobody rotates them, a year after deployment you see the Supervisor go unhealthy: kubectl on the control plane returns Unauthorized, ESXi hosts (the Supervisor worker nodes) turn NotReady, Host Config Status stays Configuring, and vSphere Pods and VKS clusters can no longer be managed. A Supervisor upgrade renews many of the certificates, so environments that are upgraded regularly hit this less often, but do not count on it.

Check the expiration dates

With certmgr on the vCenter appliance (see the next section for how to get it), one command checks every control plane VM:

cd /root
./certmgr supervisors                                   # Supervisor cluster IDs (domain-c...)
./certmgr certificates list -c <cluster-id>            # every cert with NOTAFTER and ISEXPIRED
./certmgr certificates check-expiration --days 60       # only what expires in the next 60 days

Without the tool, SSH to each Supervisor control plane VM and read the dates directly:

# wcp/tls and kubernetes/pki certificates
find /etc -type f \( -name "*.cert" -o -name "*.crt" \) | xargs -L 1 -t -i bash -c 'openssl x509 -noout -enddate -in {}'

# kubelet
openssl x509 -noout -enddate -in /var/lib/kubelet/pki/kubelet.crt

# certificates embedded in admin.conf
grep client-certificate-data /etc/kubernetes/admin.conf | awk '{print $2}' | base64 -d | openssl x509 -noout -enddate

Before you rotate

Time is in sync across vCenter, the ESXi hosts and the Supervisor control plane VMs.
In Workload Management, both Config Status and Host Config Status of the Supervisor are Running.
All Supervisor nodes are Ready.
You have the latest wcp_cert_manager.zip from KB 322994. Older builds do not rotate spherelet certificates, which leaves the ESXi nodes NotReady after the rotation.

Rotate with certmgr

Copy the zip to /root on the vCenter appliance, unzip it, and run the tool from /root. certmgr copies itself to the control plane VMs and runs the work there.

scp wcp_cert_manager.zip root@<vcenter>:/root
# on vCenter
cd /root && unzip wcp_cert_manager.zip

# optional: back up the current certificates (directory must exist)
mkdir -p /tmp/cert-backup
./certmgr certificates backup --backup-dir /tmp/cert-backup

# rotate control plane and spherelet certificates
./certmgr certificates rotate -c <cluster-id>           # -c is needed with more than one Supervisor

# only spherelet certificates on the ESXi hosts
./certmgr certificates rotate --spherelet-only -c <cluster-id>

Then make sure the new keys survive a reboot. SSH to each Supervisor control plane VM (the KB 323407 describes how) and run:

/usr/lib/vmware-wcp/hypercrypt.py --reencrypt

On legacy vSphere 7.0 Supervisors, hypercrypt.py has no --reencrypt option; KB 322994 contains a small Python script that does the same job. Finally run ./certmgr certificates list again and check that nothing shows as expired. The tool logs to /var/log/vmware/certmgr.log on vCenter; if it returns nothing or exits with status 127, you are probably not running it from /root.

What certmgr does not cover, and known issues

The latest version of KB 322994 lists several gaps that are worth knowing before you start:

authproxy-client.crt and pinniped.crt in /etc/vmware/wcp/tls/ are not rotated by certmgr. They regenerate themselves at two thirds of their lifetime and cannot be replaced with custom certificates. If they have already expired, open a case with Broadcom.
Supervisor system pod certificates are managed by the cert-manager pod in the Supervisor and should renew on their own. If they expired, cert-manager itself has a problem; Broadcom has a separate KB for that.
admin.conf: some certmgr versions did not always rotate the certificates embedded in admin.conf. If they expire, kubectl on the control plane fails with You must be logged in to the server (Unauthorized), and they must be rotated by hand following the linked KB.
kubelet.conf: the embedded certificate was not always rotated either. A control plane node can then go NotReady with unable to load bootstrap kubeconfig; open a case with Broadcom.
ESXi hosts NotReady after rotation point to the spherelet certificates; Broadcom has a dedicated KB for that case.
--vmca (rotating the VMCA-signed certificates) is marked experimental.

VKS (guest) clusters

The same tool also rotates the control plane certificates of VKS clusters, from vCenter or from a jumpbox with kubectl (KB 323453):

./certmgr tkc list --all-namespaces
./certmgr tkc certificates list <cluster> -n <namespace>
./certmgr tkc certificates rotate <cluster> -n <namespace>

Add a reminder about eleven months after each Supervisor deployment or upgrade, and run check-expiration as part of your regular health checks.

Author: Daniel Micanek

Senior Service Architect, SAP/TDL Platform Services Team at Vivicta | vExpert ⭐⭐⭐⭐⭐⭐⭐ | Broadcom Knight: vSphere Kubernetes Service, VCF Networking (NSX), VCF Automation | Broadcom Software Knight: vSphere | Cisco Champion. Notes from running about 1,000 ESXi hosts on Cisco UCS, VCF 9 and SAP HANA on vSphere. View all posts by Daniel Micanek

Supervisor Certificates Expired? Check and Rotate vSphere Supervisor Certificates with certmgr