Month: October 2026

How to Check VCF 9 Hardware Compatibility: Servers, CPUs, NICs and vSAN ESA Devices

The four checks that decide VCF 9 hardware support: BCG listings, CPU KB 318697, ESX 9 deprecated devices and vSAN ESA certification with a fresh HCL DB.

Short answer: VCF 9 hardware support is decided in four places. The Broadcom Compatibility Guide (BCG) must list the server model and every I/O device for ESX 9.x. The CPU must not be discontinued in KB 318697. No device may be End of Life in KB 391170. For vSAN ESA, the NVMe devices must be ESA-certified, and the vSAN HCL database in vCenter, SDDC Manager and the VCF Installer must be less than 90 days old. This page walks through the four checks in the order an upgrade fails them.

The four checks at a glance

CheckWhereWhat blocks VCF 9Details on vDan
Server model and I/O devicesBroadcom Compatibility Guide: Systems/Servers and I/O Devices, release ESX 9.0 or 9.1The model or device has no ESX 9.x listingthis page
CPU generationBroadcom KB 318697 (formerly 82794)The CPU is “Discontinued” for 9.xKB 82794 / 318697 CPU tables
Deprecated drivers and devicesBroadcom KB 391170The device is “End of Life”: ESX 9.0 no longer claims itESX 9.0 deprecated devices (searchable)
vSAN ESA devices and the HCL DBBCG vSAN ESA view, plus the local all.json in vCenter, SDDC Manager and the VCF InstallerNon-certified NVMe, or an HCL DB older than 90 daysvSAN HCL DB offline update · ESA HCL lab workaround

Collect what the host actually has

The BCG searches by model, by CPU series and, for I/O devices, by the four PCI IDs (VID, DID, SVID, SSID). Get them from the ESXi shell:

esxcli hardware platform get          # vendor and server model
esxcli hardware cpu global get        # CPU packages, cores
esxcli hardware cpu list | head -20   # Family / Model / Stepping (decimal)
vmkchdev -l | grep -E "vmnic|vmhba"   # VID:DID SVID:SSID per NIC and storage adapter
esxcli network nic list               # driver and link per vmnic
esxcli network nic get -n vmnic0      # driver and firmware version
esxcli storage core adapter list      # storage controllers and drivers
esxcli storage core device list | grep -E "Display Name|Vendor|Model|Revision"

vmkchdev -l prints the IDs as vid:did svid:ssid. Enter all four in the BCG I/O Devices search. The VID:DID pair alone matches a whole chip family, while the SVID:SSID pair identifies the vendor’s specific card, and that is what the listing certifies.

1. Server model: the BCG Systems listing

Search the BCG for the exact server model and pick release ESX 9.0 or 9.1. Two details decide most cases:

  • The listing names CPU series. A model listed for ESX 9.x with newer CPUs doesn’t cover the same chassis with an older CPU generation.
  • The OEM must keep the model listed. KB 318697 states that even a deprecated CPU stays supported only while the server OEM keeps the model in the BCG.

On Cisco UCS, check the UCS firmware bundle against the ESX release as well. The BCG entry for a UCS server lists the supported firmware, and the matching nenic/nfnic driver versions come from Cisco’s UCS hardware and software compatibility tool.

2. CPU: deprecated or discontinued in 9.x

KB 318697 has two tables. “Discontinued” CPUs block ESX 9.x installation and upgrade. “Deprecated” CPUs are fully supported in 9.x but will be blocked in the next major release. Broadwell, Kaby Lake and Skylake-W are discontinued. Skylake-SP and Skylake-D got a 9.x extension in KB 428874. Cascade Lake, EPYC 7001/7002, Coffee Lake and Denverton are deprecated. The exact rows, CPUID values and how to read esxcli hardware cpu list are in the CPU support page and the Skylake page.

3. NICs, HBAs and drivers: ESX 9 deprecated devices

KB 391170 lists 487 devices as Restricted (they work, but get no new features and only best-effort fixes) and 1,137 as End of Life. ESX 9.0 doesn’t claim End of Life devices at all, so a host can upgrade cleanly and come up without its storage controller or uplinks. Search your VID:DID in the ESX 9.0 deprecated devices table before the upgrade, not after.

The VCF Installer has a 10 GbE minimum NIC speed check. In a lab you can work around it (see the minimal VCF 9.0 lab). In production, a NIC that fails it isn’t a valid design.

4. vSAN ESA: certified NVMe and a fresh HCL database

vSAN ESA has its own BCG view. Each NVMe device must be ESA-certified for the release, or the server must be an ESA ReadyNode. Broadcom KB 326717 lists the component changes allowed in an ESA ReadyNode (CPU, storage device, memory, NIC, boot device), as long as the replacement is itself listed.

The local copy of the HCL database is a separate check, and it fails more often than the hardware does. vCenter, SDDC Manager and the VCF Installer each keep their own all.json. The Installer and SDDC Manager treat a copy older than 90 days as an error. The error text is the same as for uncertified disks (“No vSAN ESA certified disks found on the ESXi Host”), so check the database date first. The offline update for all three places is in vSAN HCL DB offline update.

Let vSphere Lifecycle Manager check the cluster

On a vSAN cluster managed by a vLCM image, the cluster’s hardware compatibility check compares the vSAN storage controllers and disks, including firmware, against the vSAN HCL. Firmware is validated only when the image includes a firmware add-on from the vendor’s Hardware Support Manager. If “Prevent remediation if hardware compatibility issues are found” is enabled, a finding blocks remediation (KB 412988). PowerCLI exposes the same check:

Get-LcmHardwareCompatibility -Cluster (Get-Cluster 'wld01-cl01')

This covers only what vSAN uses. It doesn’t replace checks 1–3 for the server, CPU and NICs.

Labs: what you can and cannot work around

BlockerSupported fixLab-only option
HCL DB older than 90 daysUpload a fresh all.json (KB 412606, KB 315556)Not needed
NVMe not ESA-certifiedCertified devices, vSAN OSA or external storagevSAN ESA hardware mock VIB (details)
NIC slower than 10 GbEFaster NICsInstaller validation bypass (minimal lab)
Discontinued CPUNew hardware, or stay on 8.x until its End of General SupportRPQ/TQ for software-only support via the account team (KB 428874)

Sources

ESXi iSCSI Port Binding with esxcli: networkportal add, remove and –force

iSCSI port binding with esxcli: list, add and remove network portals, what –force does, and how to clear active iSCSI sessions safely.

Short answer: software iSCSI port binding is managed with esxcli iscsi networkportal. Use list to see the bound VMkernel ports, add -A vmhbaNN -n vmkN to bind one, and remove -A vmhbaNN -n vmkN to unbind it. When ESX refuses because an iSCSI session is still active on the vmknic, remove --force unbinds it anyway, which logs those sessions out. Log them out deliberately first with esxcli iscsi session remove. Syntax below is from Broadcom’s ESXCLI reference for ESX 9.1.0 and is unchanged from ESXi 8.0 U3.

iSCSI port binding in one minute

Port binding ties the software iSCSI initiator to specific VMkernel adapters, so iSCSI traffic leaves only through those vmknics and each one becomes a separate path for multipathing. The usual rules: one vmknic per bound port, each vmknic with exactly one active physical uplink and no standby uplinks, all bound vmknics in the same subnet as the target portals, and unbound vmknics are never used for iSCSI. If your targets sit in a different subnet than the vmknics, read Broadcom’s routed port binding guidance before you bind anything.

esxcli iscsi networkportal options (ESX 9.1)

CommandOptionsWhat the option does
esxcli iscsi networkportal list--adapter | -ALimit the list to one iSCSI adapter
esxcli iscsi networkportal add--adapter | -A (required)The iSCSI adapter, e.g. vmhba64
--nic | -n (required)The vmknic to bind
--force | -fBind the vmknic even if it is not compatible with iSCSI multipathing
esxcli iscsi networkportal remove--adapter | -A (required)The iSCSI adapter
--nic | -n (required)The bound vmknic to remove
--force | -fRemove the vmknic even when sessions are active on it

The reference writes the flag as --force. The ESX error message suggests it as --force=true, so both spellings mean the same flag.

Find the software iSCSI adapter and the vmknics

esxcli iscsi adapter list
esxcli network ip interface ipv4 get
esxcli iscsi networkportal list

adapter list gives you the adapter name (the software initiator is usually vmhba6x). networkportal list shows each bound vmknic with its adapter, IP, uplink, compliance status and path status. A non-compliant vmknic is the first thing to fix.

Add a network portal (bind a vmknic)

esxcli iscsi networkportal add -A vmhba64 -n vmk2
esxcli iscsi networkportal add -A vmhba64 -n vmk3
esxcli storage core adapter rescan -A vmhba64
esxcli storage core path list

Replace vmhba64 and the vmknic names with your own. Rescan after binding so the new paths appear. Avoid add --force. It binds a vmknic that fails the multipathing checks (typically a port group with more than one active uplink), which leaves you with paths that do not fail over the way you expect. Fix the teaming instead: one active uplink, the rest unused.

Remove a network portal (unbind a vmknic)

esxcli iscsi networkportal remove -A vmhba64 -n vmk3
esxcli storage core adapter rescan -A vmhba64

With no active sessions on the vmknic, this is all you need.

“iSCSI session is active on the vmknic … use –force=true”

This is the message most people search for. ESX refuses to unbind because an iSCSI session still uses the vmknic, and suggests forcing it. Broadcom documents the same message when removing a static target (KB 370577). remove --force logs out every session on that vmknic. If a LUN has no other working path, that is an all-paths-down event for every VM on the datastore.

The safe order:

  1. Check that every LUN still has an active path through another vmknic: esxcli storage core path list -d naa.xxxx.
  2. List the sessions on the adapter and note the ISID of the session that uses this vmknic: esxcli iscsi session list -A vmhba64 and esxcli iscsi session connection list -A vmhba64.
  3. Log out only that session: esxcli iscsi session remove -A vmhba64 -s <isid>. Without -s or -n, session remove logs out every session on the adapter.
  4. Unbind: esxcli iscsi networkportal remove -A vmhba64 -n vmk3.
  5. Rescan the adapter and confirm the remaining paths.

Use --force only when the vmknic is already unusable (for example, a dead uplink during a NIC replacement) and you have confirmed the paths through the other vmknics.

iSCSI vmknics and netstacks

Searches for “netstack instance: iscsi” come from the output of esxcli network ip interface list, which prints the netstack every vmknic belongs to. To check:

esxcli network ip netstack list
esxcli network ip interface list

A vmknic gets its netstack when it is created. The reference documents esxcli network ip netstack add -N <name> and esxcli network ip interface add -i vmkN -p <portgroup> -N <netstack> (or -s <dvs-name> -P <dvport-id> on a distributed switch). An existing vmknic cannot be moved to another netstack; it has to be removed and created again. The main reason to give iSCSI its own netstack is a separate routing table. If your iSCSI traffic is not routed, the default netstack is the simpler choice. Before you port-bind vmknics on a custom netstack in production, confirm support with Broadcom’s vSphere Storage documentation and your array vendor, because the public documentation does not state it either way.

Verify paths after any change

esxcli iscsi networkportal list -A vmhba64
esxcli iscsi session list -A vmhba64
esxcli storage core path list
esxcli storage nmp device list

You want one active path per bound vmknic and target portal, and the path selection policy your array vendor recommends (usually Round Robin).

Common errors

  • “iSCSI session is active on the vmknic”: see the safe order above.
  • vmknic not compliant for binding: the port group has more than one active uplink, or a standby uplink. Set exactly one active uplink and move the others to unused. Don’t paper over it with add --force.
  • No new paths after add: no rescan, a target in a different subnet than the vmknic, or the array’s host access list does not include the initiator IQN.

Reference

Supervisor Stuck in Configuring: “ERROR – Configured Core Supervisor Services” with ReconcileFailed

Supervisor stays in Configuring with “ERROR – Configured Core Supervisor Services” and velero or tkg packages in ReconcileFailed because docker-registry.kube-system.svc returns Service Unavailable. The proxy is the cause (KB 411761).

Short answer: When the Supervisor stays in Configuring in Workload Management with ERROR – Configured Core Supervisor Services, and services such as velero.vsphere.vmware.com or tkg.vsphere.vmware.com show ReconcileFailed because https://docker-registry.kube-system.svc:5000/v2/ returns Service Unavailable, the Supervisor’s HTTPS proxy is blocking the package installation. Broadcom’s workaround (KB 411761): disable the HTTPS proxy on the Supervisor, restart the wcp service on vCenter, let the packages reconcile, then enable the proxy again and restart wcp once more.

Symptoms

In the vSphere Client, Workload Management shows the Supervisor in Configuring with an error that lists the core Supervisor services that failed. Each entry has Reason: ReconcileFailed and a message from vendir saying it could not fetch the package image, because the connection to the internal registry docker-registry.kube-system.svc:5000 returned Service Unavailable.

Core Supervisor services, such as the VKS (TKG) service and Velero, are installed as Carvel packages (PackageInstall objects) in the Supervisor. The latest version of the KB shows how to see the failure directly. In the Supervisor context:

# all core service packages and their state
kubectl get pkgi -n vmware-system-supervisor-services

# details of a failing one
kubectl describe pkgi -n vmware-system-supervisor-services svc-velero.vsphere.vmware.com
kubectl describe pkgi -n vmware-system-supervisor-services svc-tkg.vsphere.vmware.com

The status shows the condition ReconcileFailed, and usefulErrorMessage contains the same Unable to create round tripper … docker-registry.kube-system.svc:5000/v2/ … Service Unavailable text.

Why it happens

The package images for the core services are served from a registry inside the Supervisor, reached by its cluster-internal name docker-registry.kube-system.svc. When an HTTPS proxy is configured for the Supervisor, that internal request can be sent to the proxy instead of directly to the registry. The proxy cannot reach a Kubernetes-internal service name, answers Service Unavailable, and every package that needs an image from the internal registry stays in ReconcileFailed. The KB summarizes it simply: enabling the proxy can block package reconciliation.

Fix: proxy off, reconcile, proxy on

  1. Note the current proxy settings, then disable the HTTPS proxy: select the Supervisor, Configure > Network > Proxy Configuration.
  2. On the vCenter appliance, restart the Workload Management service so it reconciles again: service-control --restart wcp
  3. Wait until the packages reconcile; check with kubectl get pkgi -n vmware-system-supervisor-services that Velero, TKG and the others show Reconcile succeeded.
  4. Enable the HTTPS proxy again with the same settings, and restart wcp once more: service-control --restart wcp
  5. Check that all packages still reconcile and that Workload Management shows the Supervisor as Running.

Restarting wcp does not affect running workloads; it restarts the Workload Management service on vCenter, which drives the Supervisor configuration.

Prevent it: check the no-proxy list

The KB only describes the disable/re-enable cycle. To keep it from coming back after the next upgrade or package update, review the No Proxy list in the Supervisor proxy configuration: internal destinations should not go to the proxy. Typical entries are the Kubernetes service domain (.svc and .svc.cluster.local), the Supervisor service and pod CIDRs, the management and workload networks, vCenter and the internal registry. Check the exact list against the Broadcom documentation for your release before you change it.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.

Supervisor Certificates Expired? Check and Rotate vSphere Supervisor Certificates with certmgr

Supervisor control plane and spherelet certificates expire after one year. How to check them, rotate them from vCenter with Broadcom’s certmgr tool, persist the new keys, and which certificates certmgr does not cover (KB 322994).

Short answer: The Kubernetes certificates of a vSphere Supervisor (control plane VMs) and the spherelet certificates on its ESXi hosts are valid for one year. Before they expire, rotate them with Broadcom’s certmgr tool, which is attached to KB 322994 and runs on the vCenter appliance: ./certmgr certificates check-expiration, then ./certmgr certificates rotate, and finally re-encrypt the keys on each control plane VM with /usr/lib/vmware-wcp/hypercrypt.py --reencrypt. Always download the latest certmgr from the KB, because older versions miss certificates.

What expires and what breaks

Kubernetes issues its certificates with a one-year lifetime by default, and the Supervisor follows that. If nobody rotates them, a year after deployment you see the Supervisor go unhealthy: kubectl on the control plane returns Unauthorized, ESXi hosts (the Supervisor worker nodes) turn NotReady, Host Config Status stays Configuring, and vSphere Pods and VKS clusters can no longer be managed. A Supervisor upgrade renews many of the certificates, so environments that are upgraded regularly hit this less often, but do not count on it.

Check the expiration dates

With certmgr on the vCenter appliance (see the next section for how to get it), one command checks every control plane VM:

cd /root
./certmgr supervisors                                   # Supervisor cluster IDs (domain-c...)
./certmgr certificates list -c <cluster-id>            # every cert with NOTAFTER and ISEXPIRED
./certmgr certificates check-expiration --days 60       # only what expires in the next 60 days

Without the tool, SSH to each Supervisor control plane VM and read the dates directly:

# wcp/tls and kubernetes/pki certificates
find /etc -type f \( -name "*.cert" -o -name "*.crt" \) | xargs -L 1 -t -i bash -c 'openssl x509 -noout -enddate -in {}'

# kubelet
openssl x509 -noout -enddate -in /var/lib/kubelet/pki/kubelet.crt

# certificates embedded in admin.conf
grep client-certificate-data /etc/kubernetes/admin.conf | awk '{print $2}' | base64 -d | openssl x509 -noout -enddate

Before you rotate

  • Time is in sync across vCenter, the ESXi hosts and the Supervisor control plane VMs.
  • In Workload Management, both Config Status and Host Config Status of the Supervisor are Running.
  • All Supervisor nodes are Ready.
  • You have the latest wcp_cert_manager.zip from KB 322994. Older builds do not rotate spherelet certificates, which leaves the ESXi nodes NotReady after the rotation.

Rotate with certmgr

Copy the zip to /root on the vCenter appliance, unzip it, and run the tool from /root. certmgr copies itself to the control plane VMs and runs the work there.

scp wcp_cert_manager.zip root@<vcenter>:/root
# on vCenter
cd /root && unzip wcp_cert_manager.zip

# optional: back up the current certificates (directory must exist)
mkdir -p /tmp/cert-backup
./certmgr certificates backup --backup-dir /tmp/cert-backup

# rotate control plane and spherelet certificates
./certmgr certificates rotate -c <cluster-id>           # -c is needed with more than one Supervisor

# only spherelet certificates on the ESXi hosts
./certmgr certificates rotate --spherelet-only -c <cluster-id>

Then make sure the new keys survive a reboot. SSH to each Supervisor control plane VM (the KB 323407 describes how) and run:

/usr/lib/vmware-wcp/hypercrypt.py --reencrypt

On legacy vSphere 7.0 Supervisors, hypercrypt.py has no --reencrypt option; KB 322994 contains a small Python script that does the same job. Finally run ./certmgr certificates list again and check that nothing shows as expired. The tool logs to /var/log/vmware/certmgr.log on vCenter; if it returns nothing or exits with status 127, you are probably not running it from /root.

What certmgr does not cover, and known issues

The latest version of KB 322994 lists several gaps that are worth knowing before you start:

  • authproxy-client.crt and pinniped.crt in /etc/vmware/wcp/tls/ are not rotated by certmgr. They regenerate themselves at two thirds of their lifetime and cannot be replaced with custom certificates. If they have already expired, open a case with Broadcom.
  • Supervisor system pod certificates are managed by the cert-manager pod in the Supervisor and should renew on their own. If they expired, cert-manager itself has a problem; Broadcom has a separate KB for that.
  • admin.conf: some certmgr versions did not always rotate the certificates embedded in admin.conf. If they expire, kubectl on the control plane fails with You must be logged in to the server (Unauthorized), and they must be rotated by hand following the linked KB.
  • kubelet.conf: the embedded certificate was not always rotated either. A control plane node can then go NotReady with unable to load bootstrap kubeconfig; open a case with Broadcom.
  • ESXi hosts NotReady after rotation point to the spherelet certificates; Broadcom has a dedicated KB for that case.
  • --vmca (rotating the VMCA-signed certificates) is marked experimental.

VKS (guest) clusters

The same tool also rotates the control plane certificates of VKS clusters, from vCenter or from a jumpbox with kubectl (KB 323453):

./certmgr tkc list --all-namespaces
./certmgr tkc certificates list <cluster> -n <namespace>
./certmgr tkc certificates rotate <cluster> -n <namespace>

Add a reminder about eleven months after each Supervisor deployment or upgrade, and run check-expiration as part of your regular health checks.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.

Tesla P40 Passthrough GPU Disappears from the VM: “PCIe link to the device is down” on ESXi

My NVIDIA Tesla P40 in ESXi passthrough suddenly vanished from the VM: nvidia-smi could not talk to the driver and a VM reboot did not help. What the logs show, why only a host reboot fixes it, and how to make a homelab P40 stable (KB 418517).

Short answer: When a GPU in ESXi PCI passthrough suddenly disappears from the virtual machine, nvidia-smi says it couldn’t communicate with the NVIDIA driver and the host’s vmkernel.log says Skipping device reset … because PCIe link to the device is down, the card itself has locked up. Neither the guest nor ESXi can reset it, so rebooting the VM does nothing: only a reboot of the ESXi host brings it back (Broadcom KB 418517). That is exactly what happened to the NVIDIA Tesla P40 in my homelab. The lasting fix is to remove the reason the card drops off the bus: in a homelab that is usually heat, power or the PCIe riser.

What happened in my lab

My Tesla P40 runs in PCI passthrough to a single Linux VM for local LLMs. It sits on a PCIe riser and, because the P40 has no fan of its own, it is cooled by a blower fan on a 3D-printed adapter. After running fine for a long time, the GPU was suddenly gone from the VM in the middle of normal work, without any change on vSphere or in the VM:

$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
Make sure that the latest NVIDIA driver is installed and running.

Rebooting the VM did not help; the GPU stayed unavailable. The ESXi host told the real story in /var/log/vmkernel.log:

Skipping device reset on <PCIe-ID> because PCIe link to the device is down.

Why a VM reboot does not help

When a VM with a passthrough device powers on or reboots, ESXi resets the PCI device so the guest gets a clean card. That reset needs a working PCIe link. Here the GPU stopped responding at the hardware or firmware level and the link to it went down, so ESXi skips the reset and the guest driver finds nothing it can talk to. The card stays stuck until the host is restarted and the slot is initialized again during boot. Broadcom’s KB describes the same: once a passthrough GPU becomes unresponsive, neither ESXi nor the VM can reset it without a full host reboot.

Confirm it

# in the Linux VM
nvidia-smi
sudo dmesg | grep -iE "nvrm|xid"            # look for Xid errors, e.g. "GPU has fallen off the bus"

# on the ESXi host
grep -iE "PCIe link|device reset" /var/log/vmkernel.log | tail -20
lspci | grep -i nvidia                          # is the card still enumerated?
esxcli hardware pci list | grep -B2 -A12 -i nvidia

Fix: reboot the host

  1. Shut down the VM that uses the GPU (and migrate or shut down the other VMs on that host).
  2. Put the host in maintenance mode and reboot it. If the GPU is still missing after a warm reboot, do a full power cycle so the card loses power completely.
  3. Power the VM on and check nvidia-smi again.

Why it happens, and how to keep a homelab P40 stable

The KB points to GPU hardware or firmware stability and lists what to check with the hardware vendor: card stability, PCIe slot health, GPU firmware or VBIOS, thermal and power conditions, and known reset or firmware lock-up issues. For a data-center card like the P40 running outside its intended server, three of them stand out:

  • Heat. The P40 is passively cooled and expects strong server airflow. A homelab fan adapter that is fine at idle can be too weak under a long inference run. Watch the temperature and throttle reasons during load, and lower the power limit if needed:
    nvidia-smi --query-gpu=temperature.gpu,power.draw,clocks_throttle_reasons.active --format=csv -l 5
    sudo nvidia-smi -pl 200 (default limit 250 W)
  • Power. The P40 draws up to 250 W through an 8-pin CPU (EPS) style connector, not a normal PCIe 8-pin. Use the correct adapter and a power supply rail with enough headroom; a load spike on a weak rail can drop the card.
  • The PCIe path. A riser or extension cable is the first suspect for a link that goes down. Reseat it, try a shorter or better-shielded riser, and if the board allows it, force the slot to PCIe Gen3 (or Gen2 for testing) instead of Auto.

Keep the passthrough settings the large-BAR card needs (pciPassthru.use64bitMMIO = true and pciPassthru.64bitMMIOSizeGB = 64), keep the server BIOS and the NVIDIA guest driver current, and note when the drops happen: always under heavy load points to heat or power, random times at idle point more to the riser or the card itself.

Further reading

Supervisor Deployment or Upgrade Stuck at “Installed and Started Kubernetes Node Agent on the ESXi Host”

Supervisor deployment or upgrade hangs in Configuring Host Nodes with repeated Apply Solution tasks? The spherelet VIB cannot install because the hosts are not compliant with the vLCM image, sometimes because of legacy BIOS boot (KB 393663).

Short answer: When a Supervisor deployment or upgrade hangs in Configuring Host Nodes with the pending task Installed and Started Kubernetes Node Agent on the ESXi Host and the vSphere Client keeps running Apply Solution tasks, the Kubernetes node agent (the spherelet VIB) cannot be installed because the ESXi hosts are not compliant with the cluster’s vSphere Lifecycle Manager (vLCM) image. Fix the image and the compliance errors, including hosts that still boot in legacy BIOS mode, and remediate the hosts; the Supervisor then installs spherelet and continues on its own (Broadcom KB 393663).

Symptoms

  • In Workload Management, the Supervisor shows Host Config Status: Configuring for a long time, during a new deployment or during a Supervisor upgrade.
  • The details list a pending task Installed and Started Kubernetes Node Agent on the ESXi Host, sometimes also Kubernetes Node Agent stopped on the ESXi Host for MaintenanceMode.
  • The vSphere Client shows the same Apply Solution task again and again on the cluster.
  • The Workload Management service log on vCenter says the solution specification in the image is incompatible with hosts and lists the ESXi hosts.

Confirm it on the vCenter appliance and on a host:

# vCenter appliance: Workload Management service log
grep -i "incompatible with hosts" /var/log/vmware/wcp/wcpsvc.log | tail -5

# ESXi host: is the spherelet VIB installed?
esxcli software vib list | grep -i spherelet

Why the Supervisor waits

Every ESXi host of a Supervisor runs spherelet, the agent that turns the host into a Kubernetes node for vSphere Pods. On clusters managed by a vLCM image, the Supervisor does not install the VIB directly: it adds spherelet to the cluster’s desired state as a solution and asks vLCM to apply it. vLCM applies a solution only to hosts that are compliant with the rest of the image. If a host is out of compliance or incompatible, the apply fails, the Supervisor retries, and you see the endless Apply Solution tasks.

Common reasons for non-compliance:

  • The image has a different ESXi version, vendor add-on, firmware add-on or component than the hosts actually run.
  • Hosts have extra VIBs that are not in the image, such as a driver installed by hand.
  • Legacy BIOS boot. The latest version of the KB adds this cause: hosts that still boot in legacy BIOS mode instead of UEFI can be reported as non-compliant, because legacy boot is deprecated in vSphere 8. See Broadcom KB 313152.

Fix: make the hosts compliant

  1. Select the Supervisor cluster, open Updates > Hosts > Image.
  2. Edit the image so it matches what the hosts should run: ESXi version and, for an OEM custom image, the right vendor add-on. Save.
  3. Run Check Compliance. Read the reason for every non-compliant host; searching the exact compliance message in the Broadcom knowledge base usually finds the cause.
  4. Fix the causes: remove or add components in the image, change the boot mode of legacy BIOS hosts to UEFI (plan this carefully; it needs a reinstall or a supported conversion), then Remediate the hosts.
  5. When every host is compliant, the next Apply Solution succeeds, spherelet is installed, and the Supervisor deployment or upgrade continues without any further action.

Tip: run a compliance check on the cluster before you enable the Supervisor or start a Supervisor upgrade. Ten minutes in the Updates tab saves hours of a stuck Workload Management wizard.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.

TKG, Supervisor and VKS Nodes Cannot Resolve .local Hostnames: Cause and Workarounds

Kubernetes nodes on vSphere fail to resolve names ending in .local while the DNS server answers fine. Why systemd-resolved sends .local to mDNS, and the lab-only workarounds for TKG, the Supervisor and VKS clusters (KB 317072, KB 323444).

Short answer: Kubernetes nodes on vSphere (Tanzu Kubernetes Grid, Supervisor control plane VMs and VKS cluster nodes) run Linux with systemd-resolved, which by default sends names ending in .local to Multicast DNS, not to your DNS server. Lookups such as vcenter.corp.local therefore fail even though the DNS server knows the name. The workaround is to add the .local domain as a search domain on the nodes, but Broadcom supports it only for labs and proofs of concept: .local is reserved for mDNS by RFC 6762, and production environments should use a different domain (KB 317072, KB 323444).

Symptoms

  • On TKG or VKS nodes, any name that ends in .local fails to resolve: Temporary failure in name resolution. Other names resolve fine.
  • The Supervisor shows an Error state in Workload Management with a message that a host ending in .local cannot be resolved on a control plane VM and that local must be included in the management DNS search domains.
  • The DNS server is correct, reachable and answers when you ask it directly.

A quick test on a node separates the two layers:

resolvectl status                              # which DNS servers and domains each link uses
resolvectl query vcenter.corp.local            # goes through systemd-resolved: fails
dig @<dns-server-ip> vcenter.corp.local        # asks the DNS server directly: works

If the direct query works and the resolver query fails, the DNS server is fine and the node’s resolver is the problem.

Why it happens

This is not a TKG or VKS bug; it is how systemd-resolved works on any modern Linux. RFC 6762 reserves the .local top-level domain for Multicast DNS on the local link. systemd-resolved follows that: names ending in .local go to mDNS on the interfaces where it is enabled, and they are not sent to unicast DNS servers unless .local (or the specific domain, such as corp.local) is configured as a search or routing domain for that DNS server. Older systems and Windows clients simply asked the DNS server, which is why the same names work from your laptop or from older appliances.

Labs are hit most often, because many lab and nested environments still use .local domains.

Workaround for the Supervisor and VKS clusters (lab only)

KB 323444 has three parts. First, add the .local domain to the Supervisor’s management DNS search domains. On the vCenter appliance as root:

# find the Supervisor cluster ID (for example domain-c8)
dcli com vmware vcenter namespacemanagement clusters list

# current search domains
dcli com vmware vcenter namespacemanagement clusters get --cluster domain-c8 | grep -A 1 master_DNS_search_domains

# set the .local search domain
dcli com vmware vcenter namespacemanagement clusters update --cluster domain-c8 --master-dns-search-domains corp.local

Note that the update sets the list, so include any search domains you already had. Second, on each Supervisor control plane VM, make sure /etc/hosts contains the vCenter FQDN with its IP address. Third, on each VKS cluster node, set the search domain for the node’s interface:

resolvectl domain eth0 corp.local
resolvectl status

The node setting survives reboots but not node replacement: every node created by an upgrade, a scale-out or a machine health check needs it again. That alone makes it a lab-only fix.

Workaround for standalone TKG (lab only)

For standalone TKG management and workload clusters, KB 317072 uses the custom nameserver overlays from the TKG documentation (Nameservers on vSphere). Create the overlay files vsphere-overlay-dns-control-plane.yaml and vsphere-overlay-dns-workers.yaml as described there and add a search domain line at the end of both, so they finish like this:

nameservers: ["10.0.0.53"]
searchDomains: ["corp.local"]

Then deploy the clusters with that plan, so every node gets the search domain at creation, including nodes created later by upgrades.

The real fix: do not use .local

The latest versions of both KBs are clear: Broadcom does not recommend .local for any component, including vCenter, ESXi, NSX Manager, NSX Edges, Avi, Harbor and any endpoint that Kubernetes clusters use. The workarounds are strictly for proof of concept and lab use; they are not validated for production. For a new environment, use a domain you own (for example a subdomain of your company domain) or one reserved for private use such as home.arpa. For an existing production environment on .local, plan the rename before you deploy the Supervisor or VKS.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.

VKS Cluster Stuck in Provisioning: “failed to create volume” with POST /vsanHealth 503 Service Unavailable

A VKS cluster stays in provisioning, control plane nodes run but workers never power on, and volume creation fails with POST /vsanHealth 503. Why the vCenter vsan-health service matters for CNS and how to fix it (KB 409491).

Short answer: When a vSphere Kubernetes Service (VKS) cluster stays in provisioning, its control plane nodes are up but the worker nodes never power on, and the events show failed to create volume. Error: POST “/vsanHealth”: 503 Service Unavailable, the volume service on vCenter is not answering. Cloud Native Storage (CNS), which creates every Kubernetes volume, runs inside the vmware-vsan-health service on the vCenter appliance. Check it with service-control --status vmware-vsan-health and start it if it is stopped. If it is running, check the connection from the Supervisor to vCenter (Broadcom KB 409491).

Symptoms

  • The VKS cluster stays in provisioning for a long time.
  • Control plane nodes are created and powered on; worker nodes are created but do not power on.
  • The cluster description or the namespace events show ProvisioningFailed from csi.vsphere.vmware.com with POST “/vsanHealth”: 503 Service Unavailable, followed by ExternalProvisioning events saying the volume is still being waited for.

Check it from the Supervisor context, in the vSphere Namespace of the cluster:

kubectl describe cluster <cluster-name> -n <namespace>
kubectl get machines -n <namespace>
kubectl get pvc -n <namespace>                     # volumes stuck in Pending
kubectl get events -n <namespace> --sort-by=.lastTimestamp | grep -iE "provision|vsanHealth"

Why a vSAN service blocks a Kubernetes cluster

The name is misleading: you do not need vSAN storage to hit this. CNS, the vCenter component that creates, attaches and deletes container volumes on any datastore, is hosted by the vmware-vsan-health service, and the CSI driver calls it through the /vsanHealth endpoint. A 503 means vCenter accepted the request but the service behind it is stopped, still starting or overloaded.

The cluster needs volumes before its workers can run, for example node volumes defined in the node pool (such as a separate disk for /var/lib/containerd) or persistent volumes that system components claim during bootstrap. Without CNS, those volumes stay Pending and the rollout stops.

Fix: check and start vmware-vsan-health

Open an SSH session to the vCenter appliance and run:

service-control --status vmware-vsan-health

# if it is stopped
service-control --start vmware-vsan-health

# while you are there: any other stopped services?
service-control --status --all

If the service does not start, look at why before you retry: a full appliance partition is a common reason for vCenter services to stop or fail to start (df -h; see VCSA root partition full), and the service log under /var/log/vmware/vsan-health/ shows the error. If it still fails, open a case with Broadcom support.

Once the service is running, the CSI provisioner retries on its own. Watch the volumes bind and the workers power on:

kubectl get pvc -n <namespace> -w
kubectl get machines -n <namespace> -w

The service is running, but the 503 stays

The latest version of the KB adds what to check when vmware-vsan-health is already started:

  • Connectivity between the Supervisor and vCenter, including the workload network VIPs and load balancer: Broadcom KB 388260, Troubleshooting vSphere Supervisor Workload Cluster VIP Connection Issues.
  • vCenter service dependencies: a service can report started while one it depends on is not healthy; Broadcom KB 328009, Check vCenter Server essential service status and dependencies.
  • The general provisioning checks in the TechDocs page Troubleshoot VKS Cluster Provisioning Errors.

It also helps to check the storage policy of the StorageClass named in the error: the policy must be compatible with a datastore that every host in the Supervisor cluster can reach, otherwise volume creation fails even with a healthy CNS.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.

vSphere CSI Node Pods in CrashLoopBackOff with “Unauthorized” on a VKS Cluster: Cause and Fix

New vsphere-csi-node pods on a VKS cluster crash with “failed to get VirtualMachines for the node … Unauthorized”? The CSI controller still uses an old token. Restart the vsphere-csi-controller deployment (KB 411871).

Short answer: When new vsphere-csi-node pods on a vSphere Kubernetes Service (VKS) cluster stay in CrashLoopBackOff and the logs end with failed to get VirtualMachines for the node … Error: Unauthorized, the vSphere CSI controller in that cluster is still using an old token to talk to the Supervisor. Restart the controller so it reads the current credentials: kubectl rollout restart deployment vsphere-csi-controller -n vmware-system-csi. The node pods recover on their own afterwards (Broadcom KB 411871).

How vSphere CSI works in a VKS cluster

A VKS cluster does not talk to vCenter directly for storage. Its vSphere CSI driver (paravirtual CSI) runs in the vmware-system-csi namespace and forwards volume requests to the Supervisor, which talks to vCenter. Two parts matter here:

  • vsphere-csi-controller (Deployment): authenticates to the Supervisor with a token stored in the secret pvcsi-provider-creds in the VKS cluster.
  • vsphere-csi-node (DaemonSet, one pod per node): registers the CSI plugin on its node. During registration it asks for the node’s topology, which needs the node’s VirtualMachine object from the Supervisor.

If the controller keeps an outdated token in memory after the secret has been updated, every call to the Supervisor returns Unauthorized. Existing node pods that registered earlier keep running, which is why the problem usually shows up only on new nodes: after a scale-out, an upgrade rollout or a node replaced by a machine health check.

Recognize the problem

In the VKS cluster context:

kubectl get pods -n vmware-system-csi -o wide

# registration error on a crashing node pod
kubectl logs <vsphere-csi-node-pod> -n vmware-system-csi -c node-driver-registrar

# Unauthorized in the controller
kubectl logs deployment/vsphere-csi-controller -n vmware-system-csi -c vsphere-csi-controller | grep -i unauthorized

# when was the secret last changed, and when did the controller start?
kubectl get secret pvcsi-provider-creds -n vmware-system-csi -o jsonpath='{.metadata.creationTimestamp}{"\n"}'
kubectl get pods -n vmware-system-csi -l app=vsphere-csi-controller

The node pods on the new nodes are in CrashLoopBackOff, and the registrar log says plugin registration failed because the topology of the node could not be retrieved: failed to get VirtualMachines for the node … Unauthorized. Pods with PersistentVolumeClaims cannot start on those nodes, because the CSI plugin is not registered there.

Fix: restart the CSI controller

# VKS cluster context
kubectl rollout restart deployment vsphere-csi-controller -n vmware-system-csi
kubectl rollout status deployment vsphere-csi-controller -n vmware-system-csi

# watch the node pods recover
kubectl get pods -n vmware-system-csi -w

The new controller pods read the current pvcsi-provider-creds secret at start. The crashing node pods retry registration with back-off, so they can need a few minutes to reach Running. To speed it up, restart the DaemonSet as well: kubectl rollout restart daemonset vsphere-csi-node -n vmware-system-csi.

If it does not help

  • Check that the pvcsi-provider-creds secret exists and is not empty. If the Supervisor cannot update it, the problem is on the Supervisor side; look at the cluster status in the Supervisor context.
  • If the error is not Unauthorized but a timeout or connection refused, the CSI controller cannot reach the Supervisor at all; check the network path from the VKS cluster to the Supervisor control plane.

Further reading

More Supervisor and VKS problems with their fixes are listed in the vSphere Supervisor in VCF 9 overview.

vSphere Pod Traffic to ClusterIP Times Out on a VDS Supervisor: Cause and Workarounds

On a vSphere 8 Supervisor with VDS networking, vSphere Pods time out to ClusterIP services: ErrImagePull, CrashLoopBackOff or a stuck Supervisor service install. Why the gateway drops it and how to work around it (KB 312199).

Short answer: On a vSphere 8 Supervisor that uses VDS networking (not NSX), traffic from a vSphere Pod to a Kubernetes ClusterIP is sent to the gateway of the workload network, even when the backend sits in the same subnet. If that gateway or a firewall next to it drops traffic that comes back in on the same interface, or sees only half of a TCP session, the pod times out: DNS lookups fail with ErrImagePull, connections to the API fail with CrashLoopBackOff, and Supervisor services such as Harbor or Contour do not install. Allow the hairpin traffic on the gateway, use a simpler internal gateway, and allow kube-dns to reach the upstream DNS servers. Broadcom fixed the behavior in VCF 9.0 (KB 312199).

Symptoms

vSphere Pods run Supervisor services like Harbor, Contour or the image registry, so the problem usually shows up when you install a Supervisor service or upgrade the Supervisor. Typical signs:

  • ErrImagePull right after deployment. kubectl describe pod shows that the image could not be fetched because the DNS lookup of the registry name timed out (UDP), or the HTTPS connection to the registry timed out (TCP 443).
  • CrashLoopBackOff shortly after start. kubectl logs shows a timeout or a TLS handshake timeout when the pod connects to a ClusterIP, for example the Kubernetes API service.
  • ClusterIP fails, backend works. A connection from a vSphere Pod to a ClusterIP times out, while a direct connection to the backend pod IP works.
  • Supervisor upgrade stuck with an ImageRegistryUpgrade timeout, and the image disks for the service are missing in the Supervisor: kubectl get imagedisks -A -o yaml | grep <service-name> returns nothing.

Why it happens

A ClusterIP is a virtual address; it is not in the pod’s subnet. With VDS networking on vSphere 8, a vSphere Pod therefore sends every packet for a ClusterIP to its default gateway, the gateway of the Supervisor workload network. The backend behind the ClusterIP is often in the very same subnet, so the gateway has to send the packet back out of the interface it came in on. Two things can break this:

  • Hairpin routing is blocked. Routing a packet back out of the same interface can be denied by router security features such as reverse-path (RPF/uRPF) checks or a policy against same-interface forwarding. This hits UDP first, so DNS lookups during image fetching fail.
  • Asymmetric TCP. The request goes through the gateway, but the reply from the backend goes straight to the pod, because both are in the same subnet. A stateful firewall on or next to the gateway sees only the first and third packets of the TCP handshake and drops the session as invalid. The connection opens and then nothing passes.

A related variant: kube-dns forwards name lookups to the upstream DNS servers set in the workload network, and a third-party router or firewall between them does not allow that traffic. vSphere Pods then cannot resolve any external name.

Workarounds for vSphere 8

  1. Allow hairpin traffic on the gateway. Change the router and firewall configuration so that any traffic arriving from the workload network interface may be routed back out of the same interface: relax the RPF check and do not drop asymmetric TCP for that subnet.
  2. Use a simpler gateway for the workload network. Instead of the company edge firewall, point the Supervisor workload network to an internal router one hop before it, without stateful filtering on that segment.
  3. Allow kube-dns to reach the upstream DNS. Find the kube-dns address in the Supervisor context and allow DNS (UDP and TCP 53) between it and the DNS servers configured in the workload network:
# Supervisor context
kubectl get services -n kube-system | grep -i dns

After the change, delete the failing pod or retry the Supervisor service installation, and check that the image disks appear with kubectl get imagedisks -A.

Fixed in VCF 9.0

According to the KB, the underlying behavior of vSphere Pods on VDS networking is fixed in VCF 9.0. After you upgrade to 9.0 you can remove the router and firewall workarounds. The issue does not apply to Supervisors that use NSX or VPC networking.

Further reading