Month: September 2026

Short answer: In VCF 9, the certificate authority you choose decides who renews each certificate. Certificates signed by a CA configured in VCF Operations (VMCA, Microsoft CA or OpenSSL CA) can be renewed manually or automatically from Fleet Management, 60 days before expiry. Certificates signed by an external CA that you import are yours to renew. And once ESX hosts get CA-signed certificates through Fleet Management, vCenter switches to custom certificate mode, so every new host needs a CA-signed certificate before you commission it.

Part of the series “vCenter and VCF Certificate Management: Diagnosis, Renewal, and Operations”: vCenter Certificate Auto-Renewal: When You Still Need vCert (start here) · STS_INTERNAL_SSL_CERT Explained: What vCert’s LEGACY Result Means · vCenter Certificate Alarm After Renewal? Check the Backup Stores

Where certificate management lives in VCF 9

Fleet-wide certificate work is done in the VCF Operations console. The menu path depends on the version of the documentation you follow:

• VCF 9.0: Fleet Management > Certificates, then VCF Management (fleet components) or VCF Instances and pick an instance or domain.
• VCF 9.1: Manage in the top navigation bar, then Fleet management > Certificates in the left pane. The same VCF Management / VCF Instances split applies.

The scope split matters. The CA and the auto-renewal setting are configured separately for VCF management components (VCF Operations, VCF Automation, identity broker, logs, networks, fleet management services) and for each VCF Instance (vCenter, ESX, NSX Manager, SDDC Manager). Three instances means three instance-level CA configurations to keep consistent. The same screens show certificates and Certificate Alerts, and let you generate CSRs, import and replace; 9.1 adds bulk operations.

Configured CA versus imported certificate

This is the distinction that drives the whole operating model. The VCF 9.1 documentation lists three configured CA types:

• VMCA: the default signing CA, embedded in vCenter.
• Microsoft CA (AD CS): integrated through the /certsrv web enrollment URL, a least-privileged service account and a certificate template you create on the CA.
• OpenSSL CA: the CA embedded in SDDC Manager.

Scope is version-specific. The VCF 9.0 “Configure a Certificate Authority” page says that VCF management components support only Microsoft CA, while VCF Instance components support Microsoft CA and OpenSSL. The VCF 9.0 blog adds that the Fleet Management appliance acts as the CA for the management components when you leave them at their defaults. In 9.1, check the supported combinations for your build before you standardise.

Anything else counts as an external CA: a public CA, or an enterprise PKI that is not wired into VCF Operations. The 9.1 documentation is explicit about what that means in practice:

Summarised from the VCF 9.1 “Managing Certificates in VMware Cloud Foundation” page.

So an imported certificate is monitored but not renewed. For an external CA the cycle is manual: generate a CSR, have it signed outside VCF, import it (base64 .crt, .cer, .pem, .p7b or .p7c), then use Replace with imported certificate. Expired certificates cannot be replaced, so this must happen before expiry. The 9.1.1 auto-replace with VMCA is a fallback that changes the issuer, which your security team may not accept.

What auto-renewal actually covers

When you use Activate Auto-renewal at the management or instance level, VCF renews TLS certificates that meet two conditions: they were signed by a configured CA, and the component supports non-disruptive certificate (NDC) updates. Renewal starts 60 days before expiry. The 9.0 documentation lists the NDC-managed certificates: ESX SSL, vCenter machine SSL, NSX Manager (local manager and VIP), SDDC Manager, VCF Identity Broker, VCF Operations, VCF Automation, VCF Operations for logs, VCF Operations for networks, and Fleet Management.

Limits to plan around:

• Microsoft CA and OpenSSL auto-renewal needs the CA configured first. A Microsoft CA must be reachable from VCF Operations and time-synchronised with SDDC Manager.
• Root and intermediate certificates are never renewed.
• In 9.1, if a certificate is already inside the 60-day window, renew it manually before you turn auto-renewal on.
• System-managed certificates, such as internal certificates without a customer-facing API, are renewed by their own component. VCF Operations only monitors them and raises an alert if that renewal fails.
• vCenter also has its own native auto-renewal (vCenter 8.0 U3h / VCF 9.0.2 and later). According to KB 427937, it renews ESXi certificates 10 days before expiry (vpxd.certmgmt.certs.autoRenewThreshold) and the machine SSL certificate 5 days before, and it does not work when the certificate mode is custom. For VCF, Broadcom recommends configuring the CA in Fleet Management instead. The auto-renewal article in this series covers the vCenter side.

Custom certificate mode and new ESX hosts

This is the part that catches teams during expansion. KB 427937 says that the first time you configure an ESX host with a CA-signed certificate through Fleet Management, vCenter’s vpxd.certmgmt.mode advanced setting changes to custom. As a result, you must apply a CA-signed certificate to new hosts before you commission them. KB 417983 adds that the switch happens even when the replacement certificate comes from VMCA, because vCenter does not allow ESX certificate changes outside custom mode. After the switch, vCenter raises the alarm “When ESXi Certificate Mode is set to custom it is mandatory to add the custom CA certificates to vCenter trusted root store before updating the host certificates”. The KB treats this as expected and says you can reset it to green.

In custom mode, vCenter no longer provisions VMCA certificates to hosts, and certificate management becomes your responsibility (see vSphere 9.0 “Change the ESX Certificate Mode”). If you try to commission a host that still has its default certificate, it can fail validation with Host SSL certificate is not trusted by SDDC Manager / SDDC_TRUST_HOST_SSL_CERT_FAILED (KB 415767, VCF 5.2.x and 9.x). The commissioning documentation states the same rule for deployments done with external ESX certificates: all hosts added later must also use external certificates.

A repeatable onboarding procedure for a fleet in custom mode:

1. Check the mode on the target vCenter: Configure > Advanced Settings, filter on vpxd.certmgmt.
2. Prepare the host as usual: DNS forward and reverse records, host name equal to the FQDN, self-signed certificate regenerated for the FQDN.
3. Issue a certificate for the host from the same CA chain the rest of the domain uses. Include the FQDN in the SAN; in 9.1, VCF Operations requires both the FQDN and the IP in the SAN for certificates it generates or imports.
4. Install it on the host (/etc/vmware/ssl/rui.crt and rui.key, then restart hostd and vpxa), following “Configure ESX Hosts with Signed Certificates”.
5. Make sure the issuing root and any intermediates are trusted by vCenter and SDDC Manager, then commission the host.
6. After the host is added, confirm it appears under the instance in Fleet Management with Show ESX Hosts turned on, and that it falls under auto-renewal if its issuer is a configured CA.

KB 415767 also describes a workaround for environments that moved to external certificates after bring-up: temporarily set the mode back to vmca, commission the host, then revert. The new host then carries a VMCA certificate that you still have to replace, so treat this as an exception.

Certificate ownership matrix

A template to complete for your own fleet. Rows reflect the documented behaviour described above. Where the documentation does not settle a cell for every deployment, the cell says “check your deployment”.

In your own copy, add a named owner and the CA template used for each row. Use the same CA settings (URL, template, service account) in every VCF Instance, and keep imported external-CA certificates to a minimum, because each one is a manual renewal. If the CA wizard fails in VCF 9.0, see the KB 406901 write-up.

Sources

• TechDocs VCF 9.1: Managing Certificates in VMware Cloud Foundation (9.0 version)
• TechDocs VCF 9.0: Configure a Certificate Authority (9.1 version)
• TechDocs VCF 9.1: Set Up Automatic Renewal of Certificates
• TechDocs VCF 9.1: Import Certificates and Replace a Certificate with an External CA-Signed Certificate
• TechDocs VCF 9.0: Commission ESX Hosts
• TechDocs VCF 9.0: Configure ESX Hosts with Signed Certificates
• TechDocs vSphere 9.0: Change the ESX Certificate Mode
• KB 427937: SSL certificate auto-renew for Machine SSL and ESXi (VCF 9.0.2 and later / vCenter 8.0U3h and later)
• KB 417983: Replacing an ESX host certificate through VCF Operations triggers the alert in vCenter
• KB 415767: Host commission fails with “Host SSL certificate is not trusted by SDDC Manager”
• VCF Blog: Automatic Certificate Renewal in VMware Cloud Foundation 9.0

Related on vdan.cz

• vCenter Certificate Auto-Renewal: When You Still Need vCert (series entry point)
• STS_INTERNAL_SSL_CERT Explained: What vCert’s LEGACY Result Means
• vCenter Certificate Alarm After Renewal? Check the Backup Stores
• KB 406901: Fix “Certificate authorities update failed” in the VCF Management CA wizard
• ESXi Host Certificate Status alarm
• Installing and using the vCert tool

Short answer: Partly. From vCenter 8.0 Update 3h and VCF 9.0.2, vCenter renews VMCA-issued Machine SSL and ESXi host certificates automatically, but only when vpxd.certmgmt.mode is vmca, and only just before expiry: 5 days for Machine SSL, 10 days (configurable) for ESXi. Solution user, STS signing, externally issued and legacy-store certificates are not covered. You still need to know which mechanism owns each certificate, and keep vCert ready for recovery.

Part of the series “vCenter and VCF Certificate Management: Diagnosis, Renewal, and Operations”: this article is the entry point. Troubleshooting: STS_INTERNAL_SSL_CERT Explained: What vCert’s LEGACY Result Means and vCenter Certificate Alarm After Renewal? Check the Backup Stores. Fleet operations: VCF 9 Certificate Management: Custom CAs, Auto-Renewal, and Host Onboarding.

What changed in 8.0 U3h and VCF 9.0.2

Before vCenter 8.0 Update 3h, an expiring VMCA-issued Machine SSL certificate was an administrator task. If nobody acted, the certificate expired and services such as vpxd-svcs and sps stopped starting. Broadcom KB 396907 describes that failure and the manual fix with vCert.

The vCenter 8.0 U3h and VCF 9.0.2 release notes add native auto-renewal for two certificate types:

• vCenter Machine SSL: extended by 2 years once fewer than 5 days remain.
• ESXi host SSL: extended by 5 years once fewer than 10 days remain. This applies to hosts that support non-disruptive certificate renewal. The 8.0 U3h notes limit it to ESXi 8.0 Update 3 and later.

Both renewals apply only to certificates that VMCA issues. Both stop working if the vCenter advanced setting vpxd.certmgmt.mode is custom. KB 427937 adds that vCenter services are not restarted during the renewal.

Three renewal mechanisms, not one

“Auto-renewal” can mean three separate things, and they have different thresholds and eligibility rules. Before you trust any of them, check which one owns the certificate in front of you.

One side effect is easy to miss. KB 427937 says that the first time you give an ESXi host a CA-signed certificate through Fleet Management, vCenter switches vpxd.certmgmt.mode to custom. After that, new hosts need a CA-signed certificate before you commission them, and the native vCenter renewal in the first two rows stops applying. Fleet Management and native renewal are alternatives, not layers. Article 4 of this series covers the Fleet Management side.

The ESXi threshold depends on version and source

Published values for the native ESXi renewal threshold differ:

The most likely explanation is that the default changed in 9.1, but I have not found release notes that state this directly. Don’t rely on either number. Check the value your vCenter actually uses:

1. In the vSphere Client, select the vCenter Server and open Configure > Settings > Advanced Settings.
2. Filter on vpxd.certmgmt.
3. Check vpxd.certmgmt.mode. It must be vmca for native renewal.
4. Check vpxd.certmgmt.certs.autoRenewThreshold (days). Broadcom does not say whether this key is listed when it has never been set. If it is missing, assume the documented default for your version.
5. While you are there, note vpxd.certmgmt.certs.hardThreshold. TechDocs gives it a default of 30 days and describes it as the point where vCenter raises a red alarm.

Why the alarm fires before renewal is due

KB 427937 says the certificate pre-expiration alarm fires 30 days before expiry by default. KB 397104 ties the Certificate Status alarm to vpxd.cert.threshold and vpxd.certmgmt.certs.hardThreshold, both 30 by default. Native renewal does not start until 5 days (Machine SSL) or 10 days (ESXi, 8.0 U3h / 9.0.2) before expiry. That means a healthy, eligible certificate can keep a red alarm raised for about three weeks before renewal happens.

KB 427937 also says renewal follows the certificate’s expiration date, not the time the alarm fired. You have two options: renew early yourself, or confirm eligibility and let the alarm run its course. What you must not do is assume the alarm will clear on its own. It will not if the certificate is not eligible, for example a solution user certificate (KB 397104). It also will not clear if the certificate that triggers it lives in a store that renewal never touches. Article 3 covers the BACKUP_STORE case.

Where auto-renewal does not help, and vCert does

• An expired Machine SSL certificate. If the vCenter is below 8.0 U3h, or the certificate already expired while in custom mode, KB 396907 still points to manual replacement with vCert (Manage certificates > Machine SSL certificate) followed by a restart of all services.
• Solution user certificates. KB 397104 resolves expiring solution user certificates by replacing them, for example with vCert. Native renewal covers only Machine SSL and ESXi.
• Legacy STS configuration. On vCenters upgraded from 6.x, the STS service can still use the old STS_INTERNAL_SSL_CERT store, and Machine SSL renewal never updates it. See article 1.
• STS signing certificate. This is a separate certificate with its own failure pattern (“no healthy upstream”). See the STS signing certificate guide.
• Externally issued certificates. The VCF 9.1 vSphere blog is explicit that certificates from an external CA are not auto-renewed natively. The administrator owns them, or Fleet Management does in VCF.
• Trust problems and health checks. Stale trusted roots, mismatched trust anchors and certificates left in unused stores do not expire into an outage on a schedule, so no renewal job fixes them. A regular vCert health check (option 1) after each upgrade finds them. The vCert guide explains how to read the report.

The same blog notes that in 9.1 the VMCA root and its solution user certificates are renewed during a vCenter upgrade if the root expires in less than a year. Machine SSL and ESXi certificates are not renewed in that step.

A quick eligibility check

To see what is actually in VECS and when each entry expires, KB 396907 gives this loop. Run it on the vCenter appliance shell:

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done

Illustrative output (placeholder dates, trimmed):

STORE MACHINE_SSL_CERT
Alias : __MACHINE_CERT
            Not After : <date within 30 days>   <- alarm raised; native renewal at 5 days if mode = vmca
STORE STS_INTERNAL_SSL_CERT
Alias : __MACHINE_CERT
            Not After : <date>                   <- legacy store; not touched by renewal (see article 1)
STORE BACKUP_STORE
Alias : bkp___MACHINE_CERT
            Not After : <past date>              <- old copy; see article 3

For each certificate that expires soon, answer three questions: who issued it (VMCA or external), which mechanism owns it (native, Fleet Management or nobody), and does that mechanism’s condition hold. If the answer to the second question is “nobody”, renew it yourself with vCert, or through Fleet Management in VCF.

So, can you stop renewing manually?

For VMCA-issued Machine SSL and ESXi certificates on a vCenter in vmca mode: mostly yes, provided you accept an alarm that fires weeks before the renewal. For everything else, including VCF components with a custom CA, solution users, STS and legacy stores: no. Those need an owner, a monitor and a tested recovery procedure.

Sources

• KB 427937: SSL certificate auto-renew for Machine SSL and ESXi (VCF 9.0.2 and later / vCenter 8.0U3h and later)
• KB 396907: Machine SSL certificate replacement and auto-renewal in vCenter Server 8.x
• KB 397104: Expiring solution user certificates cause vCenter Server to throw “Certificate Status” alarm
• VMware vCenter 8.0 Update 3h Release Notes
• VCF 9.0.2 Release Notes: vCenter
• TechDocs: Set Up Automatic Renewal of Certificates in VMware Cloud Foundation
• TechDocs: ESXi Certificate Default Settings (vSphere 8.0)
• VMware blog: What’s New with vSphere in VMware Cloud Foundation 9.1?

Related on vdan.cz

• STS_INTERNAL_SSL_CERT Explained: What vCert’s LEGACY Result Means (series part 1)
• vCenter Certificate Alarm After Renewal? Check the Backup Stores (series part 3)
• VCF 9 Certificate Management: Custom CAs, Auto-Renewal, and Host Onboarding (series part 4)
• Installing and using the vCert tool
• vCenter STS signing certificate expiration: No Healthy Upstream
• ESXi Host Certificate Status alarm

Short answer: STS_INTERNAL_SSL_CERT is a legacy VECS store that older vCenter releases used for the Security Token Service (STS) TLS listener. After an upgrade, the STS service can still point at that store, so renewing the Machine SSL certificate does not change what STS presents on port 3128. vCert reports this as LEGACY. Broadcom’s fix is to switch the STS configuration to MACHINE_SSL_CERT with vCert, restart vmware-stsd, and then remove the old store (KB 423742, KB 424604).

Part of the series “vCenter and VCF Certificate Management: Diagnosis, Renewal, and Operations”: vCenter Certificate Auto-Renewal: When You Still Need vCert (start here) · vCenter Certificate Alarm After Renewal? Check the Backup Stores · VCF 9 Certificate Management: Custom CAs, Auto-Renewal, and Host Onboarding

The symptom: new Machine SSL certificate, old STS certificate

The usual sequence goes like this. You replace or renew the Machine SSL certificate, the vSphere Client shows the new certificate on 443, and everything looks healthy. Then a vulnerability scanner flags an expired or soon-to-expire certificate on port 3128, or an identity-related compliance check fails. Or you run vCert and the STS configuration check comes back as LEGACY.

Nothing went wrong with the renewal. There are simply two different certificate sources involved, and the one on 443 is not the one STS is using.

Where STS_INTERNAL_SSL_CERT comes from and why it survives upgrades

In vCenter 6.x and earlier, the vmware-stsd service took its TLS certificate from a dedicated VECS store named STS_INTERNAL_SSL_CERT. Upgrades migrate that configuration so that STS keeps working, which means the old store reference can live on for years. Broadcom’s sources describe the origin slightly differently:

• KB 423742: usually left behind after upgrading from vCenter 5.5 or 6.x to 7.x or 8.x.
• The vCert KB 385107: the legacy store is present if the system was upgraded from 5.5; greenfield 6.x/7.x deployments use MACHINE_SSL_CERT.
• KB 316625: found on appliances that were originally Windows vCenter Servers and were later migrated to the VCSA. According to that KB, new 8.x deployments do not have this store.

So the practical rule is simple: if the vCenter has a long upgrade history, check for it. KB 423742 also explains why it causes trouble. Newer vCenter versions do not refresh this store when the Machine SSL certificate is renewed, so the certificate inside it can expire without anyone noticing. On 7.x/8.x, the STS store reference is defined in /usr/lib/vmware-sso/vmware-sts/conf/server.xml (KB 316625). On vCenter 9.0 and later, STS runs on an embedded Tomcat and the reference moved to /usr/lib/vmware-sso/vmware-sts/conf/catalina.properties (KB 438122).

STS TLS certificate vs. STS signing certificate

These two are often confused because both have “STS” in the name. They are different objects with different jobs, and vCert handles them in different menus.

A LEGACY result is about the left column only. If users cannot log in and the vSphere Client shows “No Healthy Upstream”, you are looking at the signing certificate instead; see vCenter STS signing certificate expiration.

How to read the vCert configuration check

If you have not used vCert yet, start with installing and using the vCert tool. The main menu option 1. Check current certificate status checks certificate health, meaning expiry, SAN and key usage. It does not tell you which store STS reads from. That question is answered by 5. Check configurations → 2. Check STS server certificate and configuration. This check does two things (KB 385107):

1. It checks which VECS store the STS service is configured to use. Anything other than MACHINE_SSL_CERT is reported as LEGACY, and vCert offers to update the configuration and restart the service.
2. It checks the vmwSTSConnectionStrings attribute in VMware Directory. A standalone vCenter should reference its own hostname; a node in Enhanced Linked Mode should reference localhost. If the value is wrong, vCert prompts you to correct it.

Inspect what port 3128 actually presents

KB 423742 identifies the STS listener as port 3128 and gives this command to run on the appliance:

openssl s_client -connect localhost:3128 -showcerts

For a shorter comparison, you can pipe the output through standard openssl x509 and put it side by side with the Machine SSL certificate on 443. This is plain OpenSSL usage, not a Broadcom-specific procedure:

for p in 443 3128; do
  echo "== port $p"
  openssl s_client -connect localhost:$p </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -enddate
done

To see whether the legacy store exists and what its entry expires, use the VECS listing from KB 424604 / KB 316625:

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done

Broadcom’s workflow: KB 423742 and KB 424604

There are two KBs, and they cover two stages. KB 423742 (“STS_INTERNAL_SSL_CERT replacement using vCert”) switches the STS configuration interactively through the vCert menu. KB 424604 (“Removing STS_INTERNAL_SSL_CERT from vCenter Server with vCert”) removes the store, with a Lookup Service check first. Both apply to vCenter 7.x and 8.x, and both restart vmware-stsd. Combined, the procedure is:

1. Take a backup or snapshot. KB 424604 makes an offline snapshot mandatory. In Enhanced Linked Mode, take offline snapshots of all nodes.
2. Check the Lookup Service with lsdoctor: python lsdoctor.py -l. If it finds stale registrations, run python lsdoctor.py -s (stale configuration left over from 5.x) and then python lsdoctor.py -t (SSL trust mismatch). See the lsdoctor guide, and note that KB 320837 requires a service restart after -s and -t.
3. Run vCert. Either use the menu (./vCert.py → 5 → 2, then SSO administrator credentials) as in KB 423742, or run the check directly as in KB 424604: python vCert.py --run config/check_config/sts_config/op_check_sts_config.yaml
4. Answer y to “Update STS server configuration to use the MACHINE_SSL_CERT store?”
5. Answer y to “Restart service(s) vmware-stsd”.
6. Validate: port 3128 should now present the Machine SSL certificate, and the vCert check should no longer say LEGACY.

Do not delete the store by hand first. KB 316625 warns that removing STS_INTERNAL_SSL_CERT while server.xml still references it stops STS from starting, and the other vCenter services then fail with it. The configuration change always comes before the removal. That KB (and KB 438122 for vSphere 9.x) documents script and manual alternatives in case vCert is not an option.

Annotated example: before and after

Illustrative example only. Host names and dates are placeholders. The LEGACY line follows the format shown in KB 423742. Your vCert version may word the post-fix line differently.

# BEFORE - vcsa01.lab.local, upgraded from 6.x
== port 443
subject=CN = vcsa01.lab.local
issuer=CN = CA, DC = vsphere, DC = local, ...
notAfter=<new date, e.g. 2028>          <-- renewed Machine SSL, fine
== port 3128
subject=CN = vcsa01.lab.local
issuer=CN = CA, DC = vsphere, DC = local, ...
notAfter=<old date, already past>     <-- what the scanner reports

vCert: 5. Check configurations -> 2. Check STS server certificate and configuration
Checking STS Server Configuration
-----------------------------------------------------------------
Checking VECS store configuration                          LEGACY
                                    ^-- STS reads STS_INTERNAL_SSL_CERT, not MACHINE_SSL_CERT
Update STS server configuration to use the MACHINE_SSL_CERT store? [y/n]: y
Restart service(s) vmware-stsd [y/n]: y

# AFTER
== port 3128
subject=CN = vcsa01.lab.local
notAfter=<same date as port 443>       <-- 3128 and 443 now match
vCert STS check: no longer LEGACY           <-- configuration uses MACHINE_SSL_CERT
vecs-cli store list: no STS_INTERNAL_SSL_CERT entry (after KB 424604 removal)

A short checklist after the change: the 3128 and 443 certificates should have the same expiry date, vmware-stsd should be running, a vSphere Client login should work, and the scanner should be clean on its next pass. KB 423742 also says that once STS uses the Machine SSL store, later Machine SSL renewals carry over to STS as expected. For how renewals themselves are scheduled and monitored, see vCenter Certificate Auto-Renewal: When You Still Need vCert. If an expiry alarm still points at an old certificate after the fix, check the BACKUP_STORE article.

Sources

• KB 423742: STS_INTERNAL_SSL_CERT replacement using vCert
• KB 424604: Removing STS_INTERNAL_SSL_CERT from vCenter Server with vCert
• KB 385107: vCert – Scripted vCenter expired certificate replacement
• KB 316625: Remove STS_INTERNAL_SSL_CERT from VECS via shell Script and SSH
• KB 438122: Remove STS_INTERNAL_SSL_CERT from VECS via shell Script and SSH for vSphere 9.X
• KB 320837: Using the ‘lsdoctor’ Tool

Related on vdan.cz

• Installing and using the vCert tool
• vCenter STS signing certificate expiration and “No Healthy Upstream”
• A quick guide to lsdoctor for vCenter troubleshooting
• vCenter Certificate Auto-Renewal: When You Still Need vCert
• vCenter Certificate Alarm After Renewal? Check the Backup Stores
• VCF 9 Certificate Management: Custom CAs, Auto-Renewal, and Host Onboarding

Short answer: the Certificate Status alarm checks every store in VECS, not only the certificates vCenter actively uses. Replacing certificates saves the old ones in BACKUP_STORE and BACKUP_STORE_H5C so you can roll back, and when those copies expire they can raise the alarm. Find out which store and alias the alarm refers to. Confirm the active Machine SSL and solution user certificates are valid. Then use vCert to clear the expired backup entries. Your vCenter version decides which backup store can still raise the alarm.

Part of the series “vCenter and VCF Certificate Management: Diagnosis, Renewal, and Operations”: vCenter Certificate Auto-Renewal: When You Still Need vCert (start here) · STS_INTERNAL_SSL_CERT Explained: What vCert’s LEGACY Result Means · VCF 9 Certificate Management: Custom CAs, Auto-Renewal, and Host Onboarding

Why a renewal leaves certificates behind

A Machine SSL or solution user replacement on the vCenter Server Appliance doesn’t simply discard the old certificate. According to KB 326268, the replacement creates two VECS stores, BACKUP_STORE and BACKUP_STORE_H5C. The old certificates go into them so the change can be reverted. The vSphere documentation describes BACKUP_STORE as the store that supports certificate revert. It keeps only the most recent state, so you can go back one step and no further.

Nothing removes these entries automatically. They keep their original validity dates. The certificate you replaced a year ago still sits in the backup store, and its expiry date keeps getting closer. vCenter watches all VECS stores (KB 318973). When a backup copy nears or passes its expiry date, you can get a Certificate Status alarm even though every certificate vCenter actually uses is fine.

Step 1: find the store the alarm refers to

Before you change anything, find out which certificate raised the alarm. KB 318973 documents the vpxd warning that the expiry check writes. The warning includes the certificate subject and the VECS store name:

grep "will expire on" /var/log/vmware/vpxd/vpxd.log | tail -20

# Message format documented in KB 318973:
# ... opID=CheckCertificateExpiry-... Certificate [Subject: <Certificate Subject>] from store <VECS Store Name> will expire on YYYY-MM-DD HH:MM:SS

If the store is MACHINE_SSL_CERT or a solution user store, the problem is real and you need to renew. If it is BACKUP_STORE or BACKUP_STORE_H5C, read on.

Step 2: compare active and backup certificate expiry

Run this over SSH as root to list the alias and the “Not After” date for every store except TRUSTED_ROOT_CRLS. KBs 318973 and 326268 both use it:

for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

To check only the two backup stores:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store BACKUP_STORE --text | grep -ie "Alias" -ie "Not After"
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store BACKUP_STORE_H5C --text | grep -ie "Alias" -ie "Not After"

Compare the results with the active stores. MACHINE_SSL_CERT holds the __MACHINE_CERT alias. The solution user stores are machine, vpxd, vpxd-extension, vsphere-webclient and, depending on the version, hvc and wcp. Backup aliases have a bkp_ prefix, for example bkp___MACHINE_CERT or bkp_vpxd. If the active entries show future dates and only bkp_* entries have expired, the alarm is about a rollback copy.

Menu option 1 in vCert, “Check current certificate status”, gives the same view. It has its own “Checking BACKUP_STORE entries” section. The excerpt below is illustrative and follows the format shown in KB 425134:

# Illustrative example - vcsa01.lab.local, not real output
Checking Machine SSL certificate                 VALID
Checking Solution User certificates:
   machine                                       VALID
   vpxd                                          VALID
   vpxd-extension                                VALID
   vsphere-webclient                             VALID
Checking BACKUP_STORE entries:
   bkp___MACHINE_CERT                            EXPIRED   <-- old rollback copy
   bkp_machine                                   VALID
   bkp_vpxd                                      VALID

Step 3: know how your version behaves

This is where the version matters. The KBs describe it this way:

• KB 326268 (applies to vCenter 6.x, 7.x, 8.x): from vCenter Server 7.0 Update 3 and in 8.0, an expired certificate in BACKUP_STORE no longer raises the alarm. An expired certificate in BACKUP_STORE_H5C still does, and you have to delete it to clear the alert.
• KB 425734: vCert can flag BACKUP_STORE certificates that are expiring or expired. The KB says this has no impact on the system. Wait until they expire and then remove them with vCert, or open a support case if you want them removed earlier.
• KB 445656 (7.x and 8.x): the backup entries are historical, inactive copies. The KB still recommends clearing them with vCert to prevent Certificate Status alarms.
• KB 425134: expired BACKUP_STORE entries reported by vCert can also appear during vCenter upgrade prechecks. The fix is the same vCert cleanup.

So don’t assume every certificate warning is a false alarm, and don’t assume every one needs a renewal. On 7.0 U3 or 8.0, an alarm that persists while BACKUP_STORE is the only store with expired entries means you should keep looking. Check BACKUP_STORE_H5C and the other stores, and check the store name in vpxd.log. On older builds, an expired BACKUP_STORE entry alone can explain the alarm.

Troubleshooting table: store, active status, action

Step 4: clear the backup stores with vCert

vCert is documented in KB 385107. The menu numbers below are from vCert 6.1.2, the version current in that KB when this post was written. Check them against the version you run. For installation, see my vCert guide.

1. Take a VAMI file-based backup or a snapshot of the vCenter. In Enhanced Linked Mode, take powered-off snapshots of all vCenter nodes in the SSO domain.
2. Optional: export the entry you are about to remove (KB 318973):
   /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store BACKUP_STORE --alias bkp___MACHINE_CERT --output /root/old_machine.crt
3. Run ./vCert.py, then choose 3. Manage certificates and then 12. Clear expired certificates in BACKUP_STORE in VECS. KB 385107 says this option clears expired entries in BACKUP_STORE, and in BACKUP_STORE_H5C if that store exists.
4. Verify the result. Run vCert 1. Check current certificate status again, or rerun the vecs-cli loop. No expired bkp_* entries should remain.
5. If an alarm still shows after the stores are clean, reset it to green in the vSphere Client. KB 423484 notes that the alarm does not clear by itself after a certificate fix.

If vCert doesn’t clear an entry, KB 326268 gives a manual fallback. List the store, note the alias, and delete that one entry:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store BACKUP_STORE
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store BACKUP_STORE --alias <cert_alias>

According to the vecs-cli reference, entry delete removes the entry from VECS permanently. Delete only the aliases you identified as expired backup copies. Never delete entries from the active stores, and don’t touch TRUSTED_ROOT_CRLS unless a KB tells you to.

Preventing a repeat

Every Machine SSL or solution user replacement leaves another backup copy behind. Once you have confirmed that the new certificates work, add a backup-store check to your runbook. This matters most when renewal is automated or done in batches, as covered in the auto-renewal article. On VCF 9, fleet-level certificate operations are covered in the custom CA article.

Sources

• KB 326268 – Certificate Status Alarm – Clearing BACKUP_STORES certificates in the VCSA
• KB 385107 – vCert – Scripted vCenter expired certificate replacement
• KB 318973 – CertificateStatusAlarm – There are certificate that expired or about to expire
• KB 425734 – vCert tool reports expiring certificates in VECS BACKUP_STORE
• KB 445656 – Interpreting certificate status anomalies in vCenter Server 7.x and 8.x
• KB 425134 – vCenter upgrade precheck fails due to expired BACKUP_STORE certificates
• KB 423484 – MACHINE_SSL_CERT certificate in VECS is about to expire alarm
• Broadcom TechDocs – vecs-cli Command Reference
• Broadcom TechDocs – VMware Endpoint Certificate Store overview

Related on vdan.cz

• Installing and using the vCert tool
• vCenter STS signing certificate expiration and “No Healthy Upstream”
• ESXi Host Certificate Status alarm
• A quick guide to lsdoctor for vCenter troubleshooting