Tag: CNS

VKS Cluster Stuck in Provisioning: “failed to create volume” with POST /vsanHealth 503 Service Unavailable

A VKS cluster stays in provisioning, control plane nodes run but workers never power on, and volume creation fails with POST /vsanHealth 503. Why the vCenter vsan-health service matters for CNS and how to fix it (KB 409491).

Short answer: When a vSphere Kubernetes Service (VKS) cluster stays in provisioning, its control plane nodes are up but the worker nodes never power on, and the events show failed to create volume. Error: POST “/vsanHealth”: 503 Service Unavailable, the volume service on vCenter is not answering. Cloud Native Storage (CNS), which creates every Kubernetes volume, runs inside the vmware-vsan-health service on the vCenter appliance. Check it with service-control --status vmware-vsan-health and start it if it is stopped. If it is running, check the connection from the Supervisor to vCenter (Broadcom KB 409491).

Symptoms

  • The VKS cluster stays in provisioning for a long time.
  • Control plane nodes are created and powered on; worker nodes are created but do not power on.
  • The cluster description or the namespace events show ProvisioningFailed from csi.vsphere.vmware.com with POST “/vsanHealth”: 503 Service Unavailable, followed by ExternalProvisioning events saying the volume is still being waited for.

Check it from the Supervisor context, in the vSphere Namespace of the cluster:

kubectl describe cluster <cluster-name> -n <namespace>
kubectl get machines -n <namespace>
kubectl get pvc -n <namespace>                     # volumes stuck in Pending
kubectl get events -n <namespace> --sort-by=.lastTimestamp | grep -iE "provision|vsanHealth"

Why a vSAN service blocks a Kubernetes cluster

The name is misleading: you do not need vSAN storage to hit this. CNS, the vCenter component that creates, attaches and deletes container volumes on any datastore, is hosted by the vmware-vsan-health service, and the CSI driver calls it through the /vsanHealth endpoint. A 503 means vCenter accepted the request but the service behind it is stopped, still starting or overloaded.

The cluster needs volumes before its workers can run, for example node volumes defined in the node pool (such as a separate disk for /var/lib/containerd) or persistent volumes that system components claim during bootstrap. Without CNS, those volumes stay Pending and the rollout stops.

Fix: check and start vmware-vsan-health

Open an SSH session to the vCenter appliance and run:

service-control --status vmware-vsan-health

# if it is stopped
service-control --start vmware-vsan-health

# while you are there: any other stopped services?
service-control --status --all

If the service does not start, look at why before you retry: a full appliance partition is a common reason for vCenter services to stop or fail to start (df -h; see VCSA root partition full), and the service log under /var/log/vmware/vsan-health/ shows the error. If it still fails, open a case with Broadcom support.

Once the service is running, the CSI provisioner retries on its own. Watch the volumes bind and the workers power on:

kubectl get pvc -n <namespace> -w
kubectl get machines -n <namespace> -w

The service is running, but the 503 stays

The latest version of the KB adds what to check when vmware-vsan-health is already started:

  • Connectivity between the Supervisor and vCenter, including the workload network VIPs and load balancer: Broadcom KB 388260, Troubleshooting vSphere Supervisor Workload Cluster VIP Connection Issues.
  • vCenter service dependencies: a service can report started while one it depends on is not healthy; Broadcom KB 328009, Check vCenter Server essential service status and dependencies.
  • The general provisioning checks in the TechDocs page Troubleshoot VKS Cluster Provisioning Errors.

It also helps to check the storage policy of the StorageClass named in the error: the policy must be compatible with a datastore that every host in the Supervisor cluster can reach, otherwise volume creation fails even with a healthy CNS.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.