Daniel Micanek virtual Blog – Like normal Dan, but virtual.
Solve for common business challenges. Unlock big ideas to effectively predict, respond, and mitigate hurdles with innovative strategies. Don’t miss out on last chance savings. Get €100 off regular registration if you sign up before 23 September.
Common vCLS failures (MWAIT, no compatible host, embedded vCLS not deploying, missing agents) with fixes, and when Retreat Mode actually helps.
Short answer: Most vCLS problems show up as a Degraded or Unhealthy cluster service status, DRS that stops working, or VMs that refuse to power on. Fix the root cause first: BIOS MWAIT settings, invalid ESXi hostnames for embedded vCLS, or missing agent VMs. Retreat Mode is the right tool when you must remove or reset the vCLS VMs temporarily. It does not repair hardware or configuration problems, and before vSphere 9.0 it disables DRS load balancing.
Since vSphere 7.0 Update 1, DRS depends on small agent VMs called vSphere Cluster Services (vCLS). There are two implementations. Knowing which one you run matters for troubleshooting (background in my introduction to vCLS):
| External vCLS | Embedded vCLS | |
|---|---|---|
| Introduced | vSphere 7.0 U1 | vSphere 8.0 U3 |
| Managed by | ESX Agent Manager (EAM) | vpxd and ESXi (vSphere Pod technology) |
| Desired VM count | 3 | 2 |
| Host maintenance | VMs are migrated | VMs are destroyed and redeployed |
Both need at least one running vCLS VM for DRS to work, per the Embedded vCLS documentation.
The task details show Feature 'cpuid.mwait' was 0, but must be 1, or the error reads “Insufficient resources” with an MWAIT message. vCLS VMs use a per-VM EVC mode that requires the MWAIT CPU flag. The fix is in the server BIOS: update the firmware, enable Monitor/MWAIT and, where the vendor offers it, set C-States to autonomous (KB 390674, KB 326217). Retreat Mode only hides this problem; HA and DRS cannot be enabled until the vCLS VMs power on.
No deployment task appears in vCenter, and the infravisor logs complain that the name is not a valid lowercase RFC 1123 subdomain. The ESXi hostname ends with a dot or hyphen or contains trailing spaces. KB 378133 fixes it with esxcli system hostname set --host <name> --domain <domain> on each affected host in maintenance mode. Here Retreat Mode is part of the procedure: enable it, correct the hostnames, then return the cluster to System Managed mode.
vCenter blocks power-on because the cluster lacks the required number of running vCLS agents. KB 393821 suggests checking the EAM agency status and the vCLS VM count first, powering on stopped agents, and, if VMs are missing, cycling Retreat Mode on and off so vCenter redeploys them.
Past events, such as someone powering off vCLS VMs, a previous Retreat Mode period or automatic recreation, stay in the history. If the current status is Healthy, no action is needed (KB 319465).
Someone left the cluster in Retreat Mode. Switch the vCLS mode back to System Managed under Configure > vSphere Cluster Services (KB 404022).
| Situation | Retreat Mode? |
|---|---|
| vCLS VMs block an operation, such as an EVC change, and cannot simply be powered off | Yes, temporarily |
| Agent VMs are missing or stuck and must be redeployed | Yes, on then off |
| Host hostname fix for embedded vCLS (KB 378133) | Yes, as part of the procedure |
| MWAIT / BIOS or other CPU compatibility errors | No, fix the hardware configuration |
| Degraded entries only in Health History | No, nothing to fix |
| Permanently avoiding vCLS VMs on vSphere 7.0 or 8.0 | No, DRS stops balancing |
The impact depends on the version. Per KB 316514, before vSphere 9.0 a cluster in Retreat Mode has no DRS load balancing, and HA still restarts VMs, possibly on less optimal hosts. Starting with vSphere 9.0, vCLS can be disabled without affecting DRS or HA. On vSAN clusters, the Cluster Service health shows Degraded while Retreat Mode is active.
For the actual steps, the UI option in 7.0 U3o and 8.0 U2 or later, the config.vcls.clusters.domain-c<number>.enabled advanced setting, and how to exit, follow my guide How to disable vCLS on a vSphere cluster using Retreat Mode. One warning worth repeating: use only the domain-c<number> ID, because a wrong value can prevent vpxd from starting.
VMSA-2024-0019 covers vCenter heap-overflow CVE-2024-38812 and privilege escalation CVE-2024-38813. Affected versions and fixed builds (8.0 U3d, 7.0 U3t).
VMware has released an important security advisory, VMSA-2024-0019, detailing updates for VMware vCenter Server that address two significant vulnerabilities: a heap-overflow vulnerability (CVE-2024-38812) and a privilege escalation vulnerability (CVE-2024-38813). Both of these vulnerabilities could have severe implications if exploited, making it crucial for administrators to apply the necessary patches promptly.
Description: The first vulnerability, identified as CVE-2024-38812, is a heap-overflow vulnerability found in the vCenter Server’s implementation of the DCERPC protocol. This issue has been classified by VMware as Critical, with a maximum CVSSv3 base score of 9.8, indicating the potential for severe impact.
Known Attack Vectors: A malicious actor with network access to the vCenter Server can exploit this vulnerability by sending a specially crafted network packet. Successful exploitation could lead to remote code execution (RCE), allowing the attacker to execute arbitrary code on the vCenter Server with potentially full system privileges. This level of access could be used to disrupt services, exfiltrate sensitive data, or further compromise the virtual environment.
Description: The second vulnerability, CVE-2024-38813, is a privilege escalation flaw within the vCenter Server. VMware has rated this issue as Important, with a CVSSv3 base score of 7.5. While not as severe as the heap-overflow vulnerability, it still poses a significant risk.
Known Attack Vectors: An attacker with network access to the vCenter Server can exploit this vulnerability by sending a specially crafted network packet. If successful, the attacker could escalate their privileges to root, gaining full administrative control over the vCenter Server. This level of access could enable the attacker to make unauthorized changes, access sensitive information, or disrupt the entire virtual infrastructure.
Fix vCenter HTTP Status 500 – Internal Server Error: expired certificates, full disks, STS Tomcat cache, ActAsUsers, lsdoctor. Commands from Broadcom KBs.
Short answer: A vCenter “HTTP Status 500 – Internal Server Error” at the vSphere Client login is usually a backend failure, not a browser problem. Check, in this order: expired certificates (Machine SSL, solution users, STS signing), a full appliance partition, a corrupted STS Tomcat cache after patching, a missing vsphere-webclient solution user in ActAsUsers, and broken Lookup Service registrations. Take an offline snapshot first. The fixes run over SSH as root.
| Symptom | Most likely cause | Broadcom KB |
|---|---|---|
| “HTTP Status 500 – Internal Server Error” or “no healthy upstream”; services fail to start | Expired Machine SSL or solution user certificates | 324301 |
| “[500] An error occurred while fetching identity providers” | Expired STS signing, Machine SSL or solution user certificate (in ELM, one node can break the whole SSO domain) | 322178 |
| 500 after a patch or upgrade, often in Linked Mode | Corrupted Tomcat cache of the STS service | 336290 |
| 500 at login although all services run (7.0.x) | vsphere-webclient missing from ActAsUsers | 315453 |
| “500 Internal error” instead of the login page | Client registered to an old or unknown SSO domain | 374304 |
df -h # any partition at or near 100%?
service-control --status --all # which services are stopped?A full /storage/log, /storage/db or root partition stops services and produces the same 500 page. Free the space first, then continue.
This command from KB 324301 lists every VECS store with alias and expiry:
for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;The STS signing certificate is not stored in VECS, so check it separately. See vCenter STS signing certificate expiration. To replace expired certificates, Broadcom points to vCert, the fixcerts script or vSphere Certificate Manager. vCert is the quickest; see Installing and using the vCert tool. Then restart everything:
service-control --stop --all; service-control --start --allIn Enhanced Linked Mode, take offline snapshots of all linked vCenters together before changing certificates, and restart services on every node.
If the error appeared right after a patch or upgrade and certificates are valid, KB 336290 describes Tomcat failing to rebuild its cached JSP files. On vCenter 8.0.x:
service-control --stop vmware-stsd
rm -rf /var/lib/sso/webapps/ROOT/
rm -rf /var/lib/sso/workDir/ROOT/
service-control --start vmware-stsdOn vCenter 7.0.x the path is different:
service-control --stop vmware-stsd
rm -rf /usr/lib/vmware-sso/vmware-sts/work/Catalina/localhost/ROOT
service-control --start vmware-stsdvsphere-webclient logs “Access not authorized!” when acquiring a token, KB 315453 re-adds the solution user to the ActAsUsers group (LDIF or the supplied Python script; needs the SSO admin password), then restart vsphere-ui.vsphere_client_virgo.log shows “Couldn’t find any API GW registration on the local node”, rebuild the registrations with lsdoctor -r (option 2), per KB 374304. See A quick guide to lsdoctor.In this episode of the Virtually Speaking Podcast, we catch up with William Lam from VMware by Broadcom to recap VMware Explore 2024 in Las Vegas. William breaks down exciting announcements, including advancements in VMware Cloud Foundation (VCF), the introduction of the Private Cloud […]
Private AI: One Year Later with Chris Wolf
On this episode of the Virtually Speaking Podcast we welcome Chris Wolf, Global Head of AI and Advanced Services, VMware Cloud Foundation Division, Broadcom to discuss Private AI and what has changed since the announcement last year.
First Look at VMware Cloud Foundation 9
The big news from last week at VMware Explore Las Vegas was the announcement of VMware Cloud Foundation (VCF) 9! For those that attended the VCF Division Keynote 3 Transformations for the Smarter Way to Cloud, you got to hear more about VCF 9 directly from both Krish Prasad (General Manager of VCF Division) and […]
