VCF 9.1 – Interactive Upgrade Planning Tool

VCF 9.1 – Interactive Upgrade Planning Tool | William Lam

VCF 9.1 – Interactive Upgrade Planning Tool

One of the biggest advantages of VMware Cloud Foundation (VCF) 9.1 is its flexibility and broad support for existing customer environments, meeting customers where they are at in their Private Cloud journey. From existing standalone vSphere deployments with VCF Operations to environments with […]


Broadcom Social Media Advocacy

VCF 9.1 – Demystifying Supported Upgrade Paths…

VCF 9.1 – Demystifying Supported Upgrade Paths to 9.1 | William Lam

VCF 9.1 – Demystifying Supported Upgrade Paths…

The rapid advancement of frontier AI capabilities, including recent developments from Anthropic Mythos, keeping software and infrastructure up to date has become a critical priority for many IT organizations. VMware vSphere Foundation (VVF) and VMware Cloud Foundation 9.1 introduce a ton of […]


Broadcom Social Media Advocacy

Microsoft UEFI Secure Boot Certificate Expiration in vSphere VMs

Broadcom KB 423893 remains the single source of truth for the Microsoft certificate expiration topic in UEFI Secure Boot VMs.

Important: even if the Microsoft KEK certificate expires, affected VMs will continue to boot. The impact is mainly that the VM may not be able to update Microsoft KEK, DB and DBX certificates, which can block future Secure Boot related security updates. Guest OS updates not relying on these Secure Boot databases should continue to work.

This topic only concerns VMs with UEFI Secure Boot enabled. The KB also includes PowerShell scripts to identify Secure Boot / vTPM VM status and to reboot VMs.

ScenarioSecure BootvTPMRemediation Action
ESXi 8.0 U3i (P08) and lower builds & ESX 7.xESXi 8.0 U3j (P09)ESX 9.x
1DisabledDisabledNo ActionSilentPK Update

Note: This is optional as Secureboot & vTPM are disabled
No Action
2EnabledDisabledManual Update from vUEFI interfaceSilentPK UpdateManual Update from vUEFI interface
3EnabledEnabledManual Update from vUEFI interfaceManual Update from VMX Configuration(preferred for Non Windows VMs)

For Windows VMs, wait for the automated PK update solution (to be available in an upcoming 8.x patch release)
Manual Update from VMX Configuration(preferred for Non Windows VMs)

For Windows VMs, wait for the automated PK update solution (to be available in an upcoming 9.1.x patch release)
4DisabledEnabledNo ActionNo ActionNo Action

Summary:

  • Secure Boot enabled without vTPM: patch ESXi hosts to 8.0 U3j and reboot the VMs to remediate PK automatically.
  • Secure Boot enabled with vTPM: follow the KB carefully. For Windows VMs, Broadcom recommends waiting for the automated remediation in a future ESXi patch.
  • After PK remediation, follow the guest OS vendor guidance to update KEK, DB and DBX certificates.

vCenter STS Signing Certificate Expiration: Small Certificate, Big Outage

Certificates in vCenter are usually associated with Machine SSL, Solution Users, or trusted roots. However, there is another certificate that can have a very large operational impact: the Security Token Service signing certificate, commonly known as the STS signing certificate.

The STS certificate is not the certificate you see in your browser when opening the vSphere Client. It is part of the vCenter Single Sign-On authentication flow. When a user or internal service authenticates, STS validates the primary credentials and issues a SAML token containing user or service attributes. That token is then used by vCenter services to trust the authenticated identity.

In short: if STS cannot issue valid tokens, vCenter authentication starts to break.

Broadcom describes the STS certificate as critical for vSphere SSO because it is responsible for issuing, validating, and renewing security tokens.


What is the STS signing certificate?

The Security Token Service is part of vCenter Single Sign-On. Its job is to issue signed SAML tokens after successful authentication.

By default, the VMware Certificate Authority, or VMCA, generates the STS signing certificate. In most environments, the default VMCA-issued STS certificate is sufficient. You can refresh it with a new VMCA-issued certificate, or import a custom / third-party STS signing certificate if your company security policy requires all certificates to be replaced.

However, replacing the STS certificate should not be treated as a routine cosmetic change. Broadcom specifically warns that refreshing the STS certificate with a vCenter-issued certificate can replace third-party/custom certificates and may take the environment out of compliance if custom certificates are required.


Why does STS expiration matter?

When the STS signing certificate expires, vCenter may no longer be able to issue valid SAML tokens. This affects both users and internal solution users.

The result can be severe:

  • vSphere Client login failures
  • vCenter services failing to start after reboot
  • vmware-stsd service not starting
  • vpxd authorization errors
  • Signing certificate is not valid
  • No Healthy Upstream
  • failures in Enhanced Linked Mode environments
  • problems exporting OVF templates
  • certificate replacement workflows failing because authentication itself is broken

Broadcom’s KB for "Signing certificate is not valid" or "No healthy upstream" explains that these issues occur when the STS certificate or its signing root certificate has expired, preventing internal services and solution users from acquiring valid tokens.

This is why an STS certificate problem can look much bigger than a normal certificate warning. It can make the entire vCenter feel broken.


Typical symptoms

If the STS certificate is already expired, you may see errors such as:

HTTP Status 400 – Bad Request
Signing certificate is not valid
Cannot connect to vCenter Single Sign-On server
https://VC_FQDN/sts/STSService/vsphere.local
503 Service Unavailable
No Healthy Upstream

You may also see service startup failures after reboot. Broadcom notes that an expired STS certificate can prevent the vmware-stsd service from starting, which then causes dependent vCenter services to fail as well.


The warning: “STS Signing Certificates are about to expire”

In newer vCenter versions, you may receive the following alarm in the vSphere UI:

STS Signing Certificates are about to expire

This alarm means that the STS certificate for the vCenter Server Appliance is nearing expiration. Broadcom states that if the STS signing certificates expire without being replaced, vCenter will no longer be functional.

That warning should not be ignored.


How to check the STS certificate

For vCenter 7.0 Update 2 and later, the STS certificate can be checked from the vSphere Client:

vSphere Client
→ Administration
→ Certificates
→ Certificate Management
→ STS signing certificate

The UI shows the Valid until date, certificate status, and certificate chain details. Broadcom recommends checking the STS certificate periodically, because the general certificate expiry alarm does not account for the STS certificate in the same way.

For command-line or advanced checks, Broadcom now recommends using vCert instead of the older checksts.py script. The checksts.py script is deprecated, and vCert is the current recommended tool for certificate management and replacement workflows.


Recommended tool: vCert

Broadcom provides vCert.py, a menu-driven tool for certificate management on vCenter Server 7.0, 8.0, and 9.0. Its purpose is to simplify certificate replacement when certificates expire or are about to expire.

Basic usage:

unzip -q vCert-<version>.zip
cd vCert-<version>
chmod +x vCert.py
./vCert.py

Useful menu paths:

2. View certificate info
→ 8. STS signing certificates
3. Manage certificates
→ 8. STS signing certificates

Broadcom’s KB also states that vCert creates logs under:

/var/log/vmware/vCert

and uses a working directory under:

/root/vCert-master

for staging and backups.


Replacing or refreshing the STS certificate before it expires

If the STS certificate is still valid and the vSphere Client is accessible, the preferred approach is usually to refresh it from the UI.

Path:

vSphere Client
→ Administration
→ Certificate Management
→ STS signing certificate
→ Actions
→ Refresh with vCenter certificate

Broadcom marks Refresh with vCenter certificate as the recommended UI option. It generates a new VMCA-issued STS certificate.

There is also an Import and Replace Certificate option for custom or third-party certificates. Use this only when required by your company security policy.

Important: Before making certificate changes, create proper offline snapshots of all vCenter Server appliances in the environment. In Enhanced Linked Mode, this means all vCenter / PSC nodes in the same SSO domain. Broadcom explicitly recommends snapshots before STS certificate replacement operations.


What if vCenter is already broken?

If the STS certificate has already expired, the vSphere Client may not be usable. You may be stuck with:

No Healthy Upstream

or:

Signing certificate is not valid

In that case, use vCert directly on the VCSA via SSH. Broadcom recommends vCert for checking and replacing STS signing certificates when vCenter is inaccessible.

The high-level recovery flow is:

1. Take offline snapshot / backup.
2. SSH to VCSA as root.
3. Upload and extract vCert.
4. Run ./vCert.py.
5. Check STS signing certificate.
6. Replace STS signing certificate.
7. Restart services if required.
8. Validate vSphere Client login.

Never skip the snapshot step. Certificate repair can make the situation worse if performed incorrectly.


Important note for vSphere 8

In vSphere 8.0, vCenter Single Sign-On can automatically renew a VMCA-generated STS signing certificate before it expires. However, Broadcom has documented rare cases where an STS refresh happening during long-running vSphere Lifecycle Manager operations can cause task failures due to cached STS certificate data.

So even with automatic renewal, it is still useful to know where the STS certificate is, how to check it, and how it affects authentication.


Best practices

My practical checklist:

1. Check STS certificate validity regularly.
2. Do not wait until the last week before expiration.
3. Replace if the certificate expires within 6 months.
4. Use vSphere Client if vCenter is still healthy.
5. Use vCert if vCenter login or services are already broken.
6. Take offline snapshots before any certificate operation.
7. In ELM, snapshot all vCenter/PSC nodes in the SSO domain.
8. Be careful with custom/third-party STS certificates.
9. Document the certificate state before and after replacement.
10. Validate vCenter login, service health, and integrations after replacement.

Broadcom recommends replacing the STS certificate if it is set to expire within six months. If expiration is more than six months away, schedule the replacement for an appropriate maintenance window.


Summary

The STS signing certificate is easy to overlook because it is not the browser-facing Machine SSL certificate. But it is one of the most important certificates in vCenter authentication.

When it expires, the impact can be dramatic: broken SSO, failed service startup, Signing certificate is not valid, and even No Healthy Upstream.

The key lesson is simple:

Do not wait for STS expiration to become an outage. Check it, plan it, snapshot first, and use the supported Broadcom workflows.


References

  • Broadcom KB 318197 — "STS Signing Certificates are about to expire" alert received in vSphere UI
  • Broadcom KB 316619 — "Signing certificate is not valid" or "No healthy upstream" error in vCenter Server Appliance
  • Broadcom KB 385107 — vCert - Scripted vCenter expired certificate replacement
  • Broadcom KB 318968 — Checking the STS certificate expiration and replacing expired STS certificates on vCenter Servers
  • Broadcom KB 423367 — How to Export the STS Certificate from vCenter Server

ESX Passthrough of AMD Ryzen Integrated…

ESX Passthrough of AMD Ryzen Integrated Graphics (iGPU)

ESX Passthrough of AMD Ryzen Integrated…

I recently had a conversation with Wenchao (creator of AMD Zen4/Zen5 IPMI Thermal Driver for ESX Fling and Realtek Network Driver for ESX Fling) about the incredible work he has done to support the VMware/Broadcom community. His latest contributions has really closed the capability gap when […]


Broadcom Social Media Advocacy