Supervisor Deployment or Upgrade Stuck at “Installed and Started Kubernetes Node Agent on the ESXi Host”

Supervisor deployment or upgrade hangs in Configuring Host Nodes with repeated Apply Solution tasks? The spherelet VIB cannot install because the hosts are not compliant with the vLCM image, sometimes because of legacy BIOS boot (KB 393663).

Short answer: When a Supervisor deployment or upgrade hangs in Configuring Host Nodes with the pending task Installed and Started Kubernetes Node Agent on the ESXi Host and the vSphere Client keeps running Apply Solution tasks, the Kubernetes node agent (the spherelet VIB) cannot be installed because the ESXi hosts are not compliant with the cluster’s vSphere Lifecycle Manager (vLCM) image. Fix the image and the compliance errors, including hosts that still boot in legacy BIOS mode, and remediate the hosts; the Supervisor then installs spherelet and continues on its own (Broadcom KB 393663).

Symptoms

  • In Workload Management, the Supervisor shows Host Config Status: Configuring for a long time, during a new deployment or during a Supervisor upgrade.
  • The details list a pending task Installed and Started Kubernetes Node Agent on the ESXi Host, sometimes also Kubernetes Node Agent stopped on the ESXi Host for MaintenanceMode.
  • The vSphere Client shows the same Apply Solution task again and again on the cluster.
  • The Workload Management service log on vCenter says the solution specification in the image is incompatible with hosts and lists the ESXi hosts.

Confirm it on the vCenter appliance and on a host:

# vCenter appliance: Workload Management service log
grep -i "incompatible with hosts" /var/log/vmware/wcp/wcpsvc.log | tail -5

# ESXi host: is the spherelet VIB installed?
esxcli software vib list | grep -i spherelet

Why the Supervisor waits

Every ESXi host of a Supervisor runs spherelet, the agent that turns the host into a Kubernetes node for vSphere Pods. On clusters managed by a vLCM image, the Supervisor does not install the VIB directly: it adds spherelet to the cluster’s desired state as a solution and asks vLCM to apply it. vLCM applies a solution only to hosts that are compliant with the rest of the image. If a host is out of compliance or incompatible, the apply fails, the Supervisor retries, and you see the endless Apply Solution tasks.

Common reasons for non-compliance:

  • The image has a different ESXi version, vendor add-on, firmware add-on or component than the hosts actually run.
  • Hosts have extra VIBs that are not in the image, such as a driver installed by hand.
  • Legacy BIOS boot. The latest version of the KB adds this cause: hosts that still boot in legacy BIOS mode instead of UEFI can be reported as non-compliant, because legacy boot is deprecated in vSphere 8. See Broadcom KB 313152.

Fix: make the hosts compliant

  1. Select the Supervisor cluster, open Updates > Hosts > Image.
  2. Edit the image so it matches what the hosts should run: ESXi version and, for an OEM custom image, the right vendor add-on. Save.
  3. Run Check Compliance. Read the reason for every non-compliant host; searching the exact compliance message in the Broadcom knowledge base usually finds the cause.
  4. Fix the causes: remove or add components in the image, change the boot mode of legacy BIOS hosts to UEFI (plan this carefully; it needs a reinstall or a supported conversion), then Remediate the hosts.
  5. When every host is compliant, the next Apply Solution succeeds, spherelet is installed, and the Supervisor deployment or upgrade continues without any further action.

Tip: run a compliance check on the cluster before you enable the Supervisor or start a Supervisor upgrade. Ten minutes in the Updates tab saves hours of a stuck Workload Management wizard.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.

Author: Daniel Micanek

Senior Service Architect, SAP/TDL Platform Services Team at Vivicta | vExpert ⭐⭐⭐⭐⭐⭐⭐ | Broadcom Knight: vSphere Kubernetes Service, VCF Networking (NSX), VCF Automation | Broadcom Software Knight: vSphere | Cisco Champion. Notes from running about 1,000 ESXi hosts on Cisco UCS, VCF 9 and SAP HANA on vSphere.