Tag: CSI

vSphere CSI Node Pods in CrashLoopBackOff with “Unauthorized” on a VKS Cluster: Cause and Fix

New vsphere-csi-node pods on a VKS cluster crash with “failed to get VirtualMachines for the node … Unauthorized”? The CSI controller still uses an old token. Restart the vsphere-csi-controller deployment (KB 411871).

Short answer: When new vsphere-csi-node pods on a vSphere Kubernetes Service (VKS) cluster stay in CrashLoopBackOff and the logs end with failed to get VirtualMachines for the node … Error: Unauthorized, the vSphere CSI controller in that cluster is still using an old token to talk to the Supervisor. Restart the controller so it reads the current credentials: kubectl rollout restart deployment vsphere-csi-controller -n vmware-system-csi. The node pods recover on their own afterwards (Broadcom KB 411871).

How vSphere CSI works in a VKS cluster

A VKS cluster does not talk to vCenter directly for storage. Its vSphere CSI driver (paravirtual CSI) runs in the vmware-system-csi namespace and forwards volume requests to the Supervisor, which talks to vCenter. Two parts matter here:

  • vsphere-csi-controller (Deployment): authenticates to the Supervisor with a token stored in the secret pvcsi-provider-creds in the VKS cluster.
  • vsphere-csi-node (DaemonSet, one pod per node): registers the CSI plugin on its node. During registration it asks for the node’s topology, which needs the node’s VirtualMachine object from the Supervisor.

If the controller keeps an outdated token in memory after the secret has been updated, every call to the Supervisor returns Unauthorized. Existing node pods that registered earlier keep running, which is why the problem usually shows up only on new nodes: after a scale-out, an upgrade rollout or a node replaced by a machine health check.

Recognize the problem

In the VKS cluster context:

kubectl get pods -n vmware-system-csi -o wide

# registration error on a crashing node pod
kubectl logs <vsphere-csi-node-pod> -n vmware-system-csi -c node-driver-registrar

# Unauthorized in the controller
kubectl logs deployment/vsphere-csi-controller -n vmware-system-csi -c vsphere-csi-controller | grep -i unauthorized

# when was the secret last changed, and when did the controller start?
kubectl get secret pvcsi-provider-creds -n vmware-system-csi -o jsonpath='{.metadata.creationTimestamp}{"\n"}'
kubectl get pods -n vmware-system-csi -l app=vsphere-csi-controller

The node pods on the new nodes are in CrashLoopBackOff, and the registrar log says plugin registration failed because the topology of the node could not be retrieved: failed to get VirtualMachines for the node … Unauthorized. Pods with PersistentVolumeClaims cannot start on those nodes, because the CSI plugin is not registered there.

Fix: restart the CSI controller

# VKS cluster context
kubectl rollout restart deployment vsphere-csi-controller -n vmware-system-csi
kubectl rollout status deployment vsphere-csi-controller -n vmware-system-csi

# watch the node pods recover
kubectl get pods -n vmware-system-csi -w

The new controller pods read the current pvcsi-provider-creds secret at start. The crashing node pods retry registration with back-off, so they can need a few minutes to reach Running. To speed it up, restart the DaemonSet as well: kubectl rollout restart daemonset vsphere-csi-node -n vmware-system-csi.

If it does not help

  • Check that the pvcsi-provider-creds secret exists and is not empty. If the Supervisor cannot update it, the problem is on the Supervisor side; look at the cluster status in the Supervisor context.
  • If the error is not Unauthorized but a timeout or connection refused, the CSI controller cannot reach the Supervisor at all; check the network path from the VKS cluster to the Supervisor control plane.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.