vCenter Certificate Alarm BACKUP

Short answer: the Certificate Status alarm checks every store in VECS, not only the certificates vCenter actively uses. Replacing certificates saves the old ones in BACKUP_STORE and BACKUP_STORE_H5C so you can roll back, and when those copies expire they can raise the alarm. Find out which store and alias the alarm refers to. Confirm the active Machine SSL and solution user certificates are valid. Then use vCert to clear the expired backup entries. Your vCenter version decides which backup store can still raise the alarm.

Part of the series “vCenter and VCF Certificate Management: Diagnosis, Renewal, and Operations”: vCenter Certificate Auto-Renewal: When You Still Need vCert (start here) · STS_INTERNAL_SSL_CERT Explained: What vCert’s LEGACY Result Means · VCF 9 Certificate Management: Custom CAs, Auto-Renewal, and Host Onboarding

Table of Contents

Why a renewal leaves certificates behind

A Machine SSL or solution user replacement on the vCenter Server Appliance doesn’t simply discard the old certificate. According to KB 326268, the replacement creates two VECS stores, BACKUP_STORE and BACKUP_STORE_H5C. The old certificates go into them so the change can be reverted. The vSphere documentation describes BACKUP_STORE as the store that supports certificate revert. It keeps only the most recent state, so you can go back one step and no further.

Nothing removes these entries automatically. They keep their original validity dates. The certificate you replaced a year ago still sits in the backup store, and its expiry date keeps getting closer. vCenter watches all VECS stores (KB 318973). When a backup copy nears or passes its expiry date, you can get a Certificate Status alarm even though every certificate vCenter actually uses is fine.

Step 1: find the store the alarm refers to

Before you change anything, find out which certificate raised the alarm. KB 318973 documents the vpxd warning that the expiry check writes. The warning includes the certificate subject and the VECS store name:

grep "will expire on" /var/log/vmware/vpxd/vpxd.log | tail -20

# Message format documented in KB 318973:
# ... opID=CheckCertificateExpiry-... Certificate [Subject: <Certificate Subject>] from store <VECS Store Name> will expire on YYYY-MM-DD HH:MM:SS

If the store is MACHINE_SSL_CERT or a solution user store, the problem is real and you need to renew. If it is BACKUP_STORE or BACKUP_STORE_H5C, read on.

Step 2: compare active and backup certificate expiry

Run this over SSH as root to list the alias and the “Not After” date for every store except TRUSTED_ROOT_CRLS. KBs 318973 and 326268 both use it:

for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

To check only the two backup stores:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store BACKUP_STORE --text | grep -ie "Alias" -ie "Not After"
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store BACKUP_STORE_H5C --text | grep -ie "Alias" -ie "Not After"

Compare the results with the active stores. MACHINE_SSL_CERT holds the __MACHINE_CERT alias. The solution user stores are machine, vpxd, vpxd-extension, vsphere-webclient and, depending on the version, hvc and wcp. Backup aliases have a bkp_ prefix, for example bkp___MACHINE_CERT or bkp_vpxd. If the active entries show future dates and only bkp_* entries have expired, the alarm is about a rollback copy.

Menu option 1 in vCert, “Check current certificate status”, gives the same view. It has its own “Checking BACKUP_STORE entries” section. The excerpt below is illustrative and follows the format shown in KB 425134:

# Illustrative example - vcsa01.lab.local, not real output
Checking Machine SSL certificate                 VALID
Checking Solution User certificates:
   machine                                       VALID
   vpxd                                          VALID
   vpxd-extension                                VALID
   vsphere-webclient                             VALID
Checking BACKUP_STORE entries:
   bkp___MACHINE_CERT                            EXPIRED   <-- old rollback copy
   bkp_machine                                   VALID
   bkp_vpxd                                      VALID

Step 3: know how your version behaves

This is where the version matters. The KBs describe it this way:

KB 326268 (applies to vCenter 6.x, 7.x, 8.x): from vCenter Server 7.0 Update 3 and in 8.0, an expired certificate in BACKUP_STORE no longer raises the alarm. An expired certificate in BACKUP_STORE_H5C still does, and you have to delete it to clear the alert.
KB 425734: vCert can flag BACKUP_STORE certificates that are expiring or expired. The KB says this has no impact on the system. Wait until they expire and then remove them with vCert, or open a support case if you want them removed earlier.
KB 445656 (7.x and 8.x): the backup entries are historical, inactive copies. The KB still recommends clearing them with vCert to prevent Certificate Status alarms.
KB 425134: expired BACKUP_STORE entries reported by vCert can also appear during vCenter upgrade prechecks. The fix is the same vCert cleanup.

So don’t assume every certificate warning is a false alarm, and don’t assume every one needs a renewal. On 7.0 U3 or 8.0, an alarm that persists while BACKUP_STORE is the only store with expired entries means you should keep looking. Check BACKUP_STORE_H5C and the other stores, and check the store name in vpxd.log. On older builds, an expired BACKUP_STORE entry alone can explain the alarm.

Troubleshooting table: store, active status, action

Store with expired/expiring entry	Active Machine SSL / solution users	Can it raise the alarm?	Action
MACHINE_SSL_CERT or solution user store	Expired or expiring	Yes. This is a real issue.	Renew first (vCert option 3 or vSphere Client). Leave the backups alone until the new certificates work.
Any backup store	Expired or expiring	Yes	Treat it as a real issue. Fix the active certificates first. The backup is your rollback path.
BACKUP_STORE	Valid	Before 7.0 U3: yes. 7.0 U3 and 8.0: not for expired entries (KB 326268).	Clear the expired entries with vCert 3 → 12. On 7.0 U3/8.0, if the alarm persists, look for another store.
BACKUP_STORE_H5C	Valid	Yes, including 7.0 U3 and 8.0 (KB 326268)	Clear the expired entries with vCert 3 → 12. vCert cleans this store too when it exists.
BACKUP_STORE, entries only approaching expiry	Valid	Reported by vCert. KB 425734 says there is no impact.	Wait until they expire and then clear them, or ask Broadcom Support to remove them early.
MACHINE_SSL_CERT, alias `__MACHINE_CSR`	Valid	Yes (KB 318973)	This is an old CSR entry. Clear it with vCert 3 → 14 or the KB 318973 command.
TRUSTED_ROOTS, SMS, STS_INTERNAL_SSL_CERT, others	Valid	Yes	Different cause. See KB 318973 and the STS_INTERNAL_SSL_CERT article.

Step 4: clear the backup stores with vCert

vCert is documented in KB 385107. The menu numbers below are from vCert 6.1.2, the version current in that KB when this post was written. Check them against the version you run. For installation, see my vCert guide.

Take a VAMI file-based backup or a snapshot of the vCenter. In Enhanced Linked Mode, take powered-off snapshots of all vCenter nodes in the SSO domain.
Optional: export the entry you are about to remove (KB 318973):
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store BACKUP_STORE --alias bkp___MACHINE_CERT --output /root/old_machine.crt
Run ./vCert.py, then choose 3. Manage certificates and then 12. Clear expired certificates in BACKUP_STORE in VECS. KB 385107 says this option clears expired entries in BACKUP_STORE, and in BACKUP_STORE_H5C if that store exists.
Verify the result. Run vCert 1. Check current certificate status again, or rerun the vecs-cli loop. No expired bkp_* entries should remain.
If an alarm still shows after the stores are clean, reset it to green in the vSphere Client. KB 423484 notes that the alarm does not clear by itself after a certificate fix.

If vCert doesn’t clear an entry, KB 326268 gives a manual fallback. List the store, note the alias, and delete that one entry:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store BACKUP_STORE
/usr/lib/vmware-vmafd/bin/vecs-cli entry delete --store BACKUP_STORE --alias <cert_alias>

According to the vecs-cli reference, entry delete removes the entry from VECS permanently. Delete only the aliases you identified as expired backup copies. Never delete entries from the active stores, and don’t touch TRUSTED_ROOT_CRLS unless a KB tells you to.

Preventing a repeat

Every Machine SSL or solution user replacement leaves another backup copy behind. Once you have confirmed that the new certificates work, add a backup-store check to your runbook. This matters most when renewal is automated or done in batches, as covered in the auto-renewal article. On VCF 9, fleet-level certificate operations are covered in the custom CA article.

Sources

Related on vdan.cz

Author: Daniel Micanek

Senior Service Architect, SAP/TDL Platform Services Team at Vivicta | vExpert ⭐⭐⭐⭐⭐⭐⭐ | Broadcom Knight: vSphere Kubernetes Service, VCF Networking (NSX), VCF Automation | Broadcom Software Knight: vSphere | Cisco Champion. Notes from running about 1,000 ESXi hosts on Cisco UCS, VCF 9 and SAP HANA on vSphere. View all posts by Daniel Micanek