VKS Workload Cluster Upgrade Stuck Draining a Node: Pinniped kube-cert-agent and MachineDrainRule

Short answer: On vSphere Kubernetes Service (VKS) 3.3 and later, a workload cluster upgrade can stop because one old node never finishes draining. In clusters with an identity provider (Pinniped) enabled, a kube-cert-agent pod is recreated on the draining node every few seconds, so the drain never completes and the upgrade waits forever. The fix from Broadcom KB 410900 is a MachineDrainRule in the Supervisor that tells Cluster API to skip that pod during the drain.

Why the upgrade waits

A VKS cluster upgrade is a rolling replacement driven by Cluster API in the Supervisor: a new node with the target VKr (Kubernetes release) is created, the old node is cordoned and drained, and only then deleted. Control plane nodes go first; worker nodes do not start until every control plane node runs the new version and the old ones are gone. From VKS 3.3 the drain handling changed: if a node does not drain within the cluster’s node drain timeout, the upgrade does not move on.

Pinniped’s kube-cert-agent is the problem pod. It is recreated on the node it lives on, so each time the drain evicts it, a new one appears on the same cordoned node. The drain never reaches zero pods. Deleting the node or the Machine by hand does not help; the controller is still waiting for the drain.

Recognize the problem

In the Supervisor context, one Machine of the cluster stays in Deleting on the previous VKr version, and its conditions say the drain is not finished because of a pinniped pod:

kubectl get machines -n <cluster-namespace>
kubectl describe machine <deleting-machine> -n <cluster-namespace>
#   Type: DrainingSucceeded  Status: False  Reason: Draining
#   Message: Drain not completed yet ... Pod <ns>/...pinniped-kube-cert-agent-...
#            deletionTimestamp set, but still not removed from the Node

In the workload cluster context, the old node is Ready,SchedulingDisabled on the old version, and a pinniped kube-cert-agent pod keeps appearing on it (often in Pending, ContainerCreating or ImagePullBackOff):

kubectl get nodes
kubectl get pods -A -o wide | grep -i pinniped

Note the namespace of the recreating pod. It differs by environment: pinniped-concierge for the pod named pinniped-concierge-kube-cert-agent-…, or vmware-system-tmc for cluster-auth-pinniped-kube-cert-agent-… when the cluster is attached to Tanzu Mission Control.

Fix: a MachineDrainRule in the Supervisor

A MachineDrainRule is a Cluster API object that changes how drains treat selected pods. With behavior: Skip, the drain ignores the matching pods, so the node can finish draining and the upgrade continues. Create it in the Supervisor context, in the vSphere Namespace of the workload cluster.

Save this as skip-pinniped.yaml and replace the namespace with the one you noted above:

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDrainRule
metadata:
  name: vks-pod-drain-skip-pinniped
spec:
  drain:
    behavior: Skip
  pods:
  - namespaceSelector:
      matchLabels:
        kubernetes.io/metadata.name: <pinniped-pod-namespace>
    selector:
      matchLabels:
        kube-cert-agent.pinniped.dev: v3

Apply it and check that it exists:

# Supervisor context
kubectl apply -f skip-pinniped.yaml -n <cluster-namespace>
kubectl get machinedrainrule -n <cluster-namespace>

# watch the stuck Machine finish and the rollout continue
kubectl get machines -n <cluster-namespace> -w

Within a few minutes the old node finishes draining, its Machine is deleted, and the upgrade moves on to the next node. If pinniped pods run in two namespaces, add a second entry under pods (or a second rule) for the other namespace. A MachineDrainRule without a machine selector applies to every Machine in its vSphere Namespace, so it also covers the other clusters in that namespace and later rollouts, such as the next upgrade or a node replaced by a machine health check.

Good to know

• The issue affects clusters with an identity provider (Pinniped) enabled on VKS 3.3 and later. Clusters without it drain normally.
• Do not delete nodes or Machines by hand to push the upgrade on; it does not help and can leave the cluster short of capacity.
• Check the KB before you apply the rule: Broadcom updates the namespace guidance and may fix the behavior in a later VKS release.

Further reading

• Broadcom KB 410900: Workload Cluster Upgrade Stuck on vSphere Kubernetes Service (VKS) 3.3 and higher due to Pinniped-Concierge Pods
• vSphere Supervisor in VCF 9.0: hands-on lab deep dive