Daniel Micanek virtual Blog – Like normal Dan, but virtual.
VCF 9.1 – Interactive Upgrade Planning Tool | William Lam
One of the biggest advantages of VMware Cloud Foundation (VCF) 9.1 is its flexibility and broad support for existing customer environments, meeting customers where they are at in their Private Cloud journey. From existing standalone vSphere deployments with VCF Operations to environments with […]
VCF 9.1 – Demystifying Supported Upgrade Paths to 9.1 | William Lam
The rapid advancement of frontier AI capabilities, including recent developments from Anthropic Mythos, keeping software and infrastructure up to date has become a critical priority for many IT organizations. VMware vSphere Foundation (VVF) and VMware Cloud Foundation 9.1 introduce a ton of […]
Broadcom KB 423893 remains the single source of truth for the Microsoft certificate expiration topic in UEFI Secure Boot VMs.
Important: even if the Microsoft KEK certificate expires, affected VMs will continue to boot. The impact is mainly that the VM may not be able to update Microsoft KEK, DB and DBX certificates, which can block future Secure Boot related security updates. Guest OS updates not relying on these Secure Boot databases should continue to work.
This topic only concerns VMs with UEFI Secure Boot enabled. The KB also includes PowerShell scripts to identify Secure Boot / vTPM VM status and to reboot VMs.
| Scenario | Secure Boot | vTPM | Remediation Action | ||
| ESXi 8.0 U3i (P08) and lower builds & ESX 7.x | ESXi 8.0 U3j (P09) | ESX 9.x | |||
| 1 | Disabled | Disabled | No Action | SilentPK Update Note: This is optional as Secureboot & vTPM are disabled | No Action |
| 2 | Enabled | Disabled | Manual Update from vUEFI interface | SilentPK Update | Manual Update from vUEFI interface |
| 3 | Enabled | Enabled | Manual Update from vUEFI interface | Manual Update from VMX Configuration(preferred for Non Windows VMs) For Windows VMs, wait for the automated PK update solution (to be available in an upcoming 8.x patch release) | Manual Update from VMX Configuration(preferred for Non Windows VMs) For Windows VMs, wait for the automated PK update solution (to be available in an upcoming 9.1.x patch release) |
| 4 | Disabled | Enabled | No Action | No Action | No Action |
Summary:
Certificates in vCenter are usually associated with Machine SSL, Solution Users, or trusted roots. However, there is another certificate that can have a very large operational impact: the Security Token Service signing certificate, commonly known as the STS signing certificate.
The STS certificate is not the certificate you see in your browser when opening the vSphere Client. It is part of the vCenter Single Sign-On authentication flow. When a user or internal service authenticates, STS validates the primary credentials and issues a SAML token containing user or service attributes. That token is then used by vCenter services to trust the authenticated identity.
In short: if STS cannot issue valid tokens, vCenter authentication starts to break.
Broadcom describes the STS certificate as critical for vSphere SSO because it is responsible for issuing, validating, and renewing security tokens.
The Security Token Service is part of vCenter Single Sign-On. Its job is to issue signed SAML tokens after successful authentication.
By default, the VMware Certificate Authority, or VMCA, generates the STS signing certificate. In most environments, the default VMCA-issued STS certificate is sufficient. You can refresh it with a new VMCA-issued certificate, or import a custom / third-party STS signing certificate if your company security policy requires all certificates to be replaced.
However, replacing the STS certificate should not be treated as a routine cosmetic change. Broadcom specifically warns that refreshing the STS certificate with a vCenter-issued certificate can replace third-party/custom certificates and may take the environment out of compliance if custom certificates are required.
When the STS signing certificate expires, vCenter may no longer be able to issue valid SAML tokens. This affects both users and internal solution users.
The result can be severe:
vmware-stsd service not startingvpxd authorization errorsSigning certificate is not validNo Healthy UpstreamBroadcom’s KB for "Signing certificate is not valid" or "No healthy upstream" explains that these issues occur when the STS certificate or its signing root certificate has expired, preventing internal services and solution users from acquiring valid tokens.
This is why an STS certificate problem can look much bigger than a normal certificate warning. It can make the entire vCenter feel broken.
If the STS certificate is already expired, you may see errors such as:
HTTP Status 400 – Bad Request
Signing certificate is not validCannot connect to vCenter Single Sign-On server
https://VC_FQDN/sts/STSService/vsphere.local503 Service Unavailable
No Healthy UpstreamYou may also see service startup failures after reboot. Broadcom notes that an expired STS certificate can prevent the vmware-stsd service from starting, which then causes dependent vCenter services to fail as well.
In newer vCenter versions, you may receive the following alarm in the vSphere UI:
STS Signing Certificates are about to expireThis alarm means that the STS certificate for the vCenter Server Appliance is nearing expiration. Broadcom states that if the STS signing certificates expire without being replaced, vCenter will no longer be functional.
That warning should not be ignored.
For vCenter 7.0 Update 2 and later, the STS certificate can be checked from the vSphere Client:
vSphere Client
→ Administration
→ Certificates
→ Certificate Management
→ STS signing certificateThe UI shows the Valid until date, certificate status, and certificate chain details. Broadcom recommends checking the STS certificate periodically, because the general certificate expiry alarm does not account for the STS certificate in the same way.
For command-line or advanced checks, Broadcom now recommends using vCert instead of the older checksts.py script. The checksts.py script is deprecated, and vCert is the current recommended tool for certificate management and replacement workflows.
Broadcom provides vCert.py, a menu-driven tool for certificate management on vCenter Server 7.0, 8.0, and 9.0. Its purpose is to simplify certificate replacement when certificates expire or are about to expire.
Basic usage:
unzip -q vCert-<version>.zip
cd vCert-<version>
chmod +x vCert.py
./vCert.pyUseful menu paths:
2. View certificate info
→ 8. STS signing certificates3. Manage certificates
→ 8. STS signing certificatesBroadcom’s KB also states that vCert creates logs under:
/var/log/vmware/vCertand uses a working directory under:
/root/vCert-masterfor staging and backups.
If the STS certificate is still valid and the vSphere Client is accessible, the preferred approach is usually to refresh it from the UI.
Path:
vSphere Client
→ Administration
→ Certificate Management
→ STS signing certificate
→ Actions
→ Refresh with vCenter certificateBroadcom marks Refresh with vCenter certificate as the recommended UI option. It generates a new VMCA-issued STS certificate.
There is also an Import and Replace Certificate option for custom or third-party certificates. Use this only when required by your company security policy.
Important: Before making certificate changes, create proper offline snapshots of all vCenter Server appliances in the environment. In Enhanced Linked Mode, this means all vCenter / PSC nodes in the same SSO domain. Broadcom explicitly recommends snapshots before STS certificate replacement operations.
If the STS certificate has already expired, the vSphere Client may not be usable. You may be stuck with:
No Healthy Upstreamor:
Signing certificate is not validIn that case, use vCert directly on the VCSA via SSH. Broadcom recommends vCert for checking and replacing STS signing certificates when vCenter is inaccessible.
The high-level recovery flow is:
1. Take offline snapshot / backup.
2. SSH to VCSA as root.
3. Upload and extract vCert.
4. Run ./vCert.py.
5. Check STS signing certificate.
6. Replace STS signing certificate.
7. Restart services if required.
8. Validate vSphere Client login.Never skip the snapshot step. Certificate repair can make the situation worse if performed incorrectly.
In vSphere 8.0, vCenter Single Sign-On can automatically renew a VMCA-generated STS signing certificate before it expires. However, Broadcom has documented rare cases where an STS refresh happening during long-running vSphere Lifecycle Manager operations can cause task failures due to cached STS certificate data.
So even with automatic renewal, it is still useful to know where the STS certificate is, how to check it, and how it affects authentication.
My practical checklist:
1. Check STS certificate validity regularly.
2. Do not wait until the last week before expiration.
3. Replace if the certificate expires within 6 months.
4. Use vSphere Client if vCenter is still healthy.
5. Use vCert if vCenter login or services are already broken.
6. Take offline snapshots before any certificate operation.
7. In ELM, snapshot all vCenter/PSC nodes in the SSO domain.
8. Be careful with custom/third-party STS certificates.
9. Document the certificate state before and after replacement.
10. Validate vCenter login, service health, and integrations after replacement.Broadcom recommends replacing the STS certificate if it is set to expire within six months. If expiration is more than six months away, schedule the replacement for an appropriate maintenance window.
The STS signing certificate is easy to overlook because it is not the browser-facing Machine SSL certificate. But it is one of the most important certificates in vCenter authentication.
When it expires, the impact can be dramatic: broken SSO, failed service startup, Signing certificate is not valid, and even No Healthy Upstream.
The key lesson is simple:
Do not wait for STS expiration to become an outage. Check it, plan it, snapshot first, and use the supported Broadcom workflows.
"STS Signing Certificates are about to expire" alert received in vSphere UI"Signing certificate is not valid" or "No healthy upstream" error in vCenter Server AppliancevCert - Scripted vCenter expired certificate replacementChecking the STS certificate expiration and replacing expired STS certificates on vCenter ServersHow to Export the STS Certificate from vCenter ServerESX Passthrough of AMD Ryzen Integrated Graphics (iGPU)
I recently had a conversation with Wenchao (creator of AMD Zen4/Zen5 IPMI Thermal Driver for ESX Fling and Realtek Network Driver for ESX Fling) about the incredible work he has done to support the VMware/Broadcom community. His latest contributions has really closed the capability gap when […]
VCF 9.1 – VCF Download Tool (VCFDT) Cheatsheet
The latest release of VMware Cloud Foundation (VCF) Download Tool (VCFDT), which ships as part of VCF 9.1, includes a number of enhancements for downloading VVF and VCF installation and upgrade binaries. For new users of VMware Cloud Foundation Download Tool (VCFDT), I thought it would be useful to put together a quick cheat sheet […]
VCF 9.1 – Automating VCF Single Sign-On (SSO) with OIDC-based Identity Provider
There are a number of exciting enhancements to VMware Cloud Foundation (VCF) Single Sign-On (SSO) with the release of VCF 9.1 from Generic OIDC/SAML2 Identity Provider (IdP) support, streamline way to manage component level priviledges using VCF Roles and API Client and Token support for non-interactive logins to just name a few. The process of […]
During a VMware Cloud Foundation upgrade, you may hit a situation where the upgrade workflow fails on validation of the vRNI / Aria Operations for Networks certificate thumbprint.
Even after replacing the certificate directly on the vRNI appliance, clicking Retry in SDDC Manager may continue to fail with the old certificate thumbprint.
This can be confusing because the certificate on the vRNI side is already correct, but SDDC Manager still validates against the previous thumbprint.
The root cause is that SDDC Manager caches the SSL thumbprint either in its internal database, platformdb, or in the LCM / Domain Manager service memory when the upgrade task is first initialized.
As a result, even if the certificate is replaced on the vRNI / Aria Operations for Networks appliance, the Retry button does not automatically rediscover the new certificate thumbprint.
Instead, the retry operation may continue to use the old cached value.
To resolve this, the thumbprint stored in the SDDC Manager inventory database must be updated manually.
Warning:
This procedure modifies the internal SDDC Manager database. Use it only when you fully understand the impact. Always take a backup or snapshot of the SDDC Manager appliance before making manual database changes. In production environments, validate with VMware/Broadcom support first.
Log in to the SDDC Manager appliance via SSH.
Usually this means logging in as vcf and then switching to root:
su -Now retrieve the SHA-256 fingerprint of the currently installed certificate on the vRNI / Aria Operations for Networks appliance:
echo -n | openssl s_client -connect <VRNI_FQDN_OR_IP>:443 2>/dev/null | openssl x509 -noout -fingerprint -sha256Example output:
sha256 Fingerprint=XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XXCopy only the thumbprint value, without the prefix:
XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XXConnect to the SDDC Manager PostgreSQL database:
psql -h localhost -U postgres -d platformdbNow locate the vRNI / Aria Operations for Networks resource record:
SELECT id, type, status, ssl_thumbprint
FROM resource
WHERE type LIKE '%VRNI%'
OR type LIKE '%ARIA%';Identify the row that belongs to your vRNI / Aria Operations for Networks appliance.
You should see that the ssl_thumbprint column still contains the old thumbprint, for example:
EF:0B:A2:15:...Update the resource record with the new thumbprint:
UPDATE resource
SET ssl_thumbprint='<NEW_THUMBPRINT>'
WHERE id='<COMPONENT_ID>';Example:
UPDATE resource
SET ssl_thumbprint='=XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX'
WHERE id='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx';Verify the change:
SELECT id, type, status, ssl_thumbprint
FROM resource
WHERE id='<COMPONENT_ID>';Exit PostgreSQL:
\qSDDC Manager may still cache inventory data in memory, so restart the relevant services:
systemctl restart lcm
systemctl restart domainmanagerWait a few minutes until the services are fully initialized.
You can monitor the LCM service log with:
tail -f /var/log/vmware/vcf/lcm/lcm.logIn some cases, the upgrade task may remain stuck in an IN_PROGRESS state or the Retry button may stay unavailable.
If this happens, check the active execution tasks in the SDDC Manager database.
Connect again to PostgreSQL:
psql -h localhost -U postgres -d platformdbFind tasks that are still marked as running:
SELECT id, status, action
FROM execution_task
WHERE status='IN_PROGRESS';Identify the specific stuck task related to the failed upgrade validation.
Then manually mark it as failed:
UPDATE execution_task
SET status='FAILED'
WHERE id='<TASK_ID>';Exit PostgreSQL:
\qReturn to the SDDC Manager UI and refresh the upgrade page.
The upgrade workflow should now allow you to click Retry again.
This time, SDDC Manager should read the corrected thumbprint from the database, validate it against the current vRNI / Aria Operations for Networks certificate, and continue with the VCF 9.x upgrade.
If a VCF 9.x upgrade continues to fail on vRNI / Aria Operations for Networks certificate validation even after the certificate has been replaced, the issue may not be the certificate itself.
The problem can be caused by a stale SSL thumbprint cached in SDDC Manager.
The fix is to:
ssl_thumbprint value in platformdb.lcm and domainmanager services.This is a useful recovery procedure when the UI retry mechanism continues to use stale inventory data instead of the actual certificate currently installed on the vRNI appliance.
While deploying VMware Cloud Foundation 9.0.2.0 with Holodeck 9.0.2, I hit an interesting issue during the bundle download phase.
The deployment did not fail with a clear error. Instead, it stalled indefinitely at:
Install-VcfInstallerBundlesAt first glance, everything looked fine. The VCF Installer depot UI showed all required components as downloaded successfully. However, the Holodeck deployment kept waiting forever.
The root cause turned out to be a hardcoded bundle count check inside the Holodeck PowerShell module.
The issue was observed with the following setup:
Holodeck: 9.0.2
HoloRouter OVA: 9.0.2.0424
VCF Installer: 9.0.2.0
Target VCF: 9.0.2.0
Deployment: Full VCF, ManagementOnly
Depot: Online depotImportant detail: this was full VCF, not VVF.
During deployment, the log repeatedly showed:
SddcMgmtDomain[<pid>]: [INFO] Received Bundles. Checking if all VCF 9 bundles are available
SddcMgmtDomain[<pid>]: [INFO] Didn't receive all bundles. Received 8 bundle details. Trying again after 10 secondsThis message repeated every 10 seconds.
At the same time, the VCF Installer UI showed all visible components as successfully downloaded:
SDDC Manager 9.0.2.0
VMware Cloud Foundation Automation 9.0.2.0
VMware Cloud Foundation Operations 9.0.2.0
VMware Cloud Foundation Operations Collector 9.0.2.0
VMware Cloud Foundation Operations fleet management 9.0.2.0
VMware NSX 9.0.2.0
VMware vCenter 9.0.2.0So from the UI perspective, everything looked complete. However, Holodeck was still waiting.
The problematic logic is inside the Holodeck PowerShell module on the deployed HoloRouter:
/root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1The affected function is:
Install-VcfInstallerBundlesHolodeck queries the VCF Installer API:
https://${HostName}/v1/bundles/download-status?imageType=INSTALLThen it filters bundles matching the selected VCF version:
}elseif($Version -eq "9.0.2.0"){
$vcf9_bundle_details = $bundle_details.elements | Where-Object {$_.version -match "9\.0\.2\.0\.*"}
}The problem is the final count check:
elseif($vcf9_bundle_details.count -eq 7){
Write-Log -Message "Received all Bundle Details"
$bundle_api_response = $true
}
else{
Write-Log -Message "Didn't receive all bundles. Received $($vcf9_bundle_details.count) bundle details. Trying again after 10 seconds"
Start-Sleep -Seconds 10
}For VCF 9.0.2.0, the API returns 8 matching bundle entries, not 7.
That means this condition never becomes true:
$vcf9_bundle_details.count -eq 7Holodeck receives 8 bundles, but waits for exactly 7.
Result: an infinite loop.
In VCF 9.0.2.0, the bundle structure changed compared to earlier VCF 9 versions.
The depot UI shows 7 visible rows, but the API response contains 8 entries matching:
9.0.2.0.*The 9.0.2 BOM appears to split some components more granularly, for example around VCF Operations, Operations Collector, and Fleet Management. The UI abstracts this nicely, but the API exposes one additional bundle-level entry.
The important part is this:
VCF Installer UI: 7 visible downloaded components
VCF Installer API: 8 matching bundle entries
Holodeck logic: expects exactly 7That mismatch is enough to block the deployment.
Edit the Holodeck module directly on the HoloRouter.
Change this:
}elseif($vcf9_bundle_details.count -eq 7){To this:
}elseif($vcf9_bundle_details.count -ge 7){One-liner:
sed -i 's/$vcf9_bundle_details.count -eq 7/$vcf9_bundle_details.count -ge 7/' \
/root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1This changes the check from “exactly 7 bundles” to “at least 7 bundles”.
The currently running pwsh process already has the old function loaded in memory.
After patching the file, kill the running PowerShell process:
pkill -9 -f pwshThen start a fresh PowerShell session and resume the deployment:
Import-HoloDeckConfig -ConfigID <id>
New-HoloDeckInstance -Version 9.0.2.0 -InstanceID <same-as-before> <original flags>For example:
New-HoloDeckInstance -Version 9.0.2.0 -InstanceID 1 -ManagementOnlyUse the same flags you used in the original deployment.
After applying the workaround, Holodeck resumes at the existing deployment state.
In my case, the state engine resumed at:
Install-VcfInstallerBundlesThe patched function immediately accepted the 8 returned bundle entries and logged:
Received all Bundle DetailsThe deployment then moved forward.
Some bundle download calls may return:
BUNDLE_DOWNLOAD_ALREADY_DOWNLOADEDThat is expected because the bundles are already present in the depot. The existing try/catch handling allows the phase to complete quickly.
After that, the deployment advanced to the management-domain phase.
The quick fix is:
- }elseif($vcf9_bundle_details.count -eq 7){
+ }elseif($vcf9_bundle_details.count -ge 7){A version-specific fix would also work, for example:
VCF 9.0.0.0 / 9.0.1.0 -> expect 7
VCF 9.0.2.0 -> expect 8However, that would likely reintroduce the same type of bug in a future VCF BOM revision.
A better long-term approach would be to avoid a hardcoded count completely and validate the actual bundle download state instead.
For example, Holodeck should check that all required bundles for the selected deployment type and version are present and successfully downloaded, instead of assuming that the bundle count is always static.
Still, as an immediate workaround, changing -eq 7 to -ge 7 is enough to unblock the deployment.
This is an unofficial workaround.
The affected PowerShell module is not part of the public Holodeck documentation repository. It is bundled inside the HoloRouter OVA distributed through the Broadcom Support Portal.
Before editing vendor-supplied files, it is always a good idea to make a backup:
cp /root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1 \
/root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1.bakThen apply the patch.
This is a good example of a small hardcoded assumption causing a deployment to stall without an obvious fatal error.
The VCF Installer API was returning valid data. The depot was populated. The UI showed the bundles as successfully downloaded. But Holodeck was waiting for an exact number of bundle entries that no longer matched the VCF 9.0.2.0 BOM.
For affected Holodeck 9.0.2 users deploying VCF 9.0.2.0, the key symptom is:
Didn't receive all bundles. Received 8 bundle details. Trying again after 10 secondsIf you see this message, check the bundle count logic in:
SddcMgmtDeployment.psm1The workaround is small, but it can save a lot of troubleshooting time.
cp /root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1 \
/root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1.bak
sed -i 's/$vcf9_bundle_details.count -eq 7/$vcf9_bundle_details.count -ge 7/' \
/root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1
pkill -9 -f pwshThen resume:
Import-HoloDeckConfig -ConfigID <id>
New-HoloDeckInstance -Version 9.0.2.0 -InstanceID <same-as-before> <original flags>VCF 9.1 – Additional IP allocation options for VCF Management Services (VCFMS) in VCF Installer and SDDC Manager | William Lam
One of the new components introduced in VMware vSphere Foundation (VVF) and VMware Cloud Foundation (VCF) 9.1 is the VCF Management Services (VCFMS), which provides a centralized system for unifying both existing and new capabilities for VCF Fleet management and operations. When deploying […]
