vCenter and VCF Certificate Troubleshooting: Which Tool or Fix You Need

Which vCenter or VCF certificate is failing, and which tool fixes it: an alarm-to-fix map for vCenter 7, 8 and 9, vCert vs lsdoctor, and auto-renewal.

Short answer: start by identifying the certificate, not the tool. Run the VECS expiry check below or a vCert health check (option 1), then match the alarm or error to the table on this page. vCert fixes most vCenter certificate problems: Machine SSL, solution users, STS signing, trust anchors and ESXi. lsdoctor fixes the Lookup Service registrations that point at them. From vCenter 8.0 U3h and VCF 9.0.2, VMCA-issued Machine SSL and ESXi certificates renew themselves, but only in vmca mode and only a few days before expiry.

This page is the index for every certificate article on vDan. Each link says when to read it.

First check: which certificate is expiring?

Most vCenter certificates live in VECS, the VMware Endpoint Certificate Store. This loop from Broadcom KB 396907 prints every store with the alias and expiry date of each entry. Run it as root in the vCenter appliance shell:

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done

The store name tells you which certificate you are dealing with:

Store or certificateWhat uses itRead
MACHINE_SSL_CERTThe vCenter endpoint certificate that browsers and API clients seeAuto-renewal: when you still need vCert
Solution users (machine, vsphere-webclient, vpxd, vpxd-extension, wcp, hvc)Internal service-to-service authenticationvCert guide · certificate-manager scripting
STS signing certificate (not a VECS store; check it with vCert)Signs every vCenter SSO tokenSTS signing certificate guide
STS_INTERNAL_SSL_CERTLegacy STS store on vCenters upgraded from 6.xSTS_INTERNAL_SSL_CERT and vCert’s LEGACY result
BACKUP_STORE, BACKUP_STORE_H5CRollback copies of replaced certificatesCertificate alarm after renewal: check the backup stores
TRUSTED_ROOTSCA certificates that vCenter trustsvCert guide (trusted root options)
ESXi host certificate (stored on the host, tracked by vCenter)Host management endpointESXi Host Certificate Status alarm

Alarm or error → fix

Find the message you see and go straight to the fix.

You seeLikely causeRead
“STS Signing Certificates are about to expire”, or “no healthy upstream” at loginSTS signing certificate within 90 days of expiry, or already expiredSTS signing certificate guide
“ESXi Host Certificate Status” alarmA host certificate expires within about 30 daysESXi Host Certificate Status: Renew, VMCA mode, vCert
A Certificate Status alarm that stays raised after you renewedAn old copy in a backup store, or a certificate that renewal doesn’t coverCheck the backup stores · What auto-renewal covers
vCert reports STS as LEGACY; an old certificate still shows after a Machine SSL renewalSTS still uses the pre-7.0 STS_INTERNAL_SSL_CERT storeSTS_INTERNAL_SSL_CERT explained
“HTTP Status 500 – Internal Server Error” at the vSphere Client loginExpired certificates, a corrupted STS cache after patching, or broken SSO registrationsHTTP 500: causes and fixes
Lookup Service errors, SSL trust mismatch in service registrationsRegistrations still point at an old certificatelsdoctor guide
lsdoctor fails with an IndexError on vCenter 8.0.3.00000A customised login bannerlsdoctor on 8.0.3.00000
VCF 9.0 “Certificate authorities update failed” (401 in vmware_vrlcm.log)# or & in the Microsoft CA service account password (KB 406901)KB 406901 fix
VCF 9 upgrade stops at the vRNI / Aria Operations for Networks SSL thumbprint checkSDDC Manager stores the thumbprint of a replaced certificatevRNI SSL thumbprint fix
vROps / Aria Operations: “Data Retriever is not initialized yet”Expired internal certificate (KB 71018)Replace the vROps internal certificate
VAMI: “Exception in invoking authentication handler User password expired”Not a certificate: the appliance root password expiredvCenter password expired

Tools: vCert, lsdoctor and certificate-manager

ToolUse it forWhere to get itGuide
vCert (vCert.py)Checking and replacing vCenter certificates: Machine SSL, solution users, STS signing, trust anchors, VMCA and ESXi. Supports vCenter 7.0, 8.0 and 9.x.Broadcom KB 385107vCert guide
lsdoctor (lsdoctor.py)Checking and repairing Lookup Service registrations, SSL trust and solution users. Start with the read-only --lscheck.Broadcom KB 320837lsdoctor guide
certificate-managerThe older built-in, interactive-only menu. Broadcom now recommends vCert for most certificate workflows.On the appliance: /usr/lib/vmware-vmca/bin/certificate-managerScripting certificate-manager, and when to use vCert instead

A rule that saves time: vCert changes certificates, lsdoctor fixes what points at them. If you replace a certificate and services then fail to register or trust each other, that is lsdoctor territory. Take an offline snapshot of every vCenter in the SSO domain before running either tool.

Auto-renewal in vCenter 8.0 U3h and VCF 9.0.2

Per Broadcom KB 427937, vCenter 8.0 U3h and VCF 9.0.2 renew two certificate types natively:

  • VMCA-issued Machine SSL, 5 days before expiry;
  • VMCA-issued ESXi host certificates, 10 days before expiry by default (configurable).

Both require vpxd.certmgmt.mode = vmca. The certificate alarm fires at 30 days, so a certificate that will renew itself can still show a red alarm for about three weeks. Solution user, STS signing, externally issued and legacy-store certificates are not covered. vCenter Certificate Auto-Renewal: When You Still Need vCert has the full table, including the 9.1 threshold change.

VCF 9: fleet certificates and custom CAs

In VCF 9, VCF Operations Fleet Management can renew certificates for VCF components, including ones from a Microsoft CA or OpenSSL. It isn’t a layer on top of the native vCenter renewal; it replaces it. Giving an ESXi host a CA-signed certificate through Fleet Management switches vCenter to custom mode, and new hosts then need a CA-signed certificate before you commission them.

Upgrade-time certificate traps

  • vCenters upgraded from 6.x can keep STS on the legacy store, so a Machine SSL renewal never reaches it. Check with vCert before the next upgrade.
  • HTTP 500 right after patching is often a corrupted STS Tomcat cache, not an expired certificate. Clear the cache.
  • VCF upgrade pre-checks compare stored thumbprints. After you replace a component certificate, SDDC Manager can still hold the old one. vRNI example.
  • Run a vCert health check after every upgrade. Stale trusted roots and mismatched trust anchors don’t expire into an outage on a schedule, so no renewal job finds them.

Other products and related certificates

Sources

Author: Daniel Micanek

Senior Service Architect, SAP/TDL Platform Services Team at Vivicta | vExpert ⭐⭐⭐⭐⭐⭐⭐ | Broadcom Knight: vSphere Kubernetes Service, VCF Networking (NSX), VCF Automation | Broadcom Software Knight: vSphere | Cisco Champion. Notes from running about 1,000 ESXi hosts on Cisco UCS, VCF 9 and SAP HANA on vSphere.