Short answer: Most vCLS problems show up as a Degraded or Unhealthy cluster service status, DRS that stops working, or VMs that refuse to power on. Fix the root cause first: BIOS MWAIT settings, invalid ESXi hostnames for embedded vCLS, or missing agent VMs. Retreat Mode is the right tool when you must remove or reset the vCLS VMs temporarily. It does not repair hardware or configuration problems, and before vSphere 9.0 it disables DRS load balancing.
How vCLS works today
Since vSphere 7.0 Update 1, DRS depends on small agent VMs called vSphere Cluster Services (vCLS). There are two implementations. Knowing which one you run matters for troubleshooting (background in my introduction to vCLS):
| External vCLS | Embedded vCLS | |
|---|---|---|
| Introduced | vSphere 7.0 U1 | vSphere 8.0 U3 |
| Managed by | ESX Agent Manager (EAM) | vpxd and ESXi (vSphere Pod technology) |
| Desired VM count | 3 | 2 |
| Host maintenance | VMs are migrated | VMs are destroyed and redeployed |
Both need at least one running vCLS VM for DRS to work, per the Embedded vCLS documentation.
Common vCLS failures and their fixes
vCLS VMs fail to power on: “No host is compatible with the virtual machine”
The task details show Feature 'cpuid.mwait' was 0, but must be 1, or the error reads “Insufficient resources” with an MWAIT message. vCLS VMs use a per-VM EVC mode that requires the MWAIT CPU flag. The fix is in the server BIOS: update the firmware, enable Monitor/MWAIT and, where the vendor offers it, set C-States to autonomous (KB 390674, KB 326217). Retreat Mode only hides this problem; HA and DRS cannot be enabled until the vCLS VMs power on.
Embedded vCLS does not deploy after upgrading to 8.0 Update 3
No deployment task appears in vCenter, and the infravisor logs complain that the name is not a valid lowercase RFC 1123 subdomain. The ESXi hostname ends with a dot or hyphen or contains trailing spaces. KB 378133 fixes it with esxcli system hostname set --host <name> --domain <domain> on each affected host in maintenance mode. Here Retreat Mode is part of the procedure: enable it, correct the hostnames, then return the cluster to System Managed mode.
Workload VMs will not power on: “All required agent virtual machines are not currently deployed”
vCenter blocks power-on because the cluster lacks the required number of running vCLS agents. KB 393821 suggests checking the EAM agency status and the vCLS VM count first, powering on stopped agents, and, if VMs are missing, cycling Retreat Mode on and off so vCenter redeploys them.
Health History shows Degraded, but the status is Healthy
Past events, such as someone powering off vCLS VMs, a previous Retreat Mode period or automatic recreation, stay in the history. If the current status is Healthy, no action is needed (KB 319465).
Alarm: “vSphere Cluster Services are disabled by Retreat Mode”
Someone left the cluster in Retreat Mode. Switch the vCLS mode back to System Managed under Configure > vSphere Cluster Services (KB 404022).
When Retreat Mode is the right tool
| Situation | Retreat Mode? |
|---|---|
| vCLS VMs block an operation, such as an EVC change, and cannot simply be powered off | Yes, temporarily |
| Agent VMs are missing or stuck and must be redeployed | Yes, on then off |
| Host hostname fix for embedded vCLS (KB 378133) | Yes, as part of the procedure |
| MWAIT / BIOS or other CPU compatibility errors | No, fix the hardware configuration |
| Degraded entries only in Health History | No, nothing to fix |
| Permanently avoiding vCLS VMs on vSphere 7.0 or 8.0 | No, DRS stops balancing |
The impact depends on the version. Per KB 316514, before vSphere 9.0 a cluster in Retreat Mode has no DRS load balancing, and HA still restarts VMs, possibly on less optimal hosts. Starting with vSphere 9.0, vCLS can be disabled without affecting DRS or HA. On vSAN clusters, the Cluster Service health shows Degraded while Retreat Mode is active.
For the actual steps, the UI option in 7.0 U3o and 8.0 U2 or later, the config.vcls.clusters.domain-c<number>.enabled advanced setting, and how to exit, follow my guide How to disable vCLS on a vSphere cluster using Retreat Mode. One warning worth repeating: use only the domain-c<number> ID, because a wrong value can prevent vpxd from starting.
Before you exit Retreat Mode
- Confirm the root cause is fixed; otherwise the new vCLS VMs fail the same way.
- Make sure the hosts have a datastore and resources available for the agent VMs.
- Return to System Managed mode, then verify that the expected number of vCLS VMs (3 external, 2 embedded) are running and that DRS recommendations resume.
Further reading
- mikewire.com: Troubleshooting vSphere Cluster Service (vCLS) with Retreat Mode (original post)
- Broadcom KB 316514: Disable vCLS on a Cluster via Retreat Mode
- Broadcom TechDocs: Embedded vCLS
