ESXi iSCSI Port Binding with esxcli: networkportal add, remove and –force

iSCSI port binding with esxcli: list, add and remove network portals, what –force does, and how to clear active iSCSI sessions safely.

Short answer: software iSCSI port binding is managed with esxcli iscsi networkportal. Use list to see the bound VMkernel ports, add -A vmhbaNN -n vmkN to bind one, and remove -A vmhbaNN -n vmkN to unbind it. When ESX refuses because an iSCSI session is still active on the vmknic, remove --force unbinds it anyway, which logs those sessions out. Log them out deliberately first with esxcli iscsi session remove. Syntax below is from Broadcom’s ESXCLI reference for ESX 9.1.0 and is unchanged from ESXi 8.0 U3.

iSCSI port binding in one minute

Port binding ties the software iSCSI initiator to specific VMkernel adapters, so iSCSI traffic leaves only through those vmknics and each one becomes a separate path for multipathing. The usual rules: one vmknic per bound port, each vmknic with exactly one active physical uplink and no standby uplinks, all bound vmknics in the same subnet as the target portals, and unbound vmknics are never used for iSCSI. If your targets sit in a different subnet than the vmknics, read Broadcom’s routed port binding guidance before you bind anything.

esxcli iscsi networkportal options (ESX 9.1)

CommandOptionsWhat the option does
esxcli iscsi networkportal list--adapter | -ALimit the list to one iSCSI adapter
esxcli iscsi networkportal add--adapter | -A (required)The iSCSI adapter, e.g. vmhba64
--nic | -n (required)The vmknic to bind
--force | -fBind the vmknic even if it is not compatible with iSCSI multipathing
esxcli iscsi networkportal remove--adapter | -A (required)The iSCSI adapter
--nic | -n (required)The bound vmknic to remove
--force | -fRemove the vmknic even when sessions are active on it

The reference writes the flag as --force. The ESX error message suggests it as --force=true, so both spellings mean the same flag.

Find the software iSCSI adapter and the vmknics

esxcli iscsi adapter list
esxcli network ip interface ipv4 get
esxcli iscsi networkportal list

adapter list gives you the adapter name (the software initiator is usually vmhba6x). networkportal list shows each bound vmknic with its adapter, IP, uplink, compliance status and path status. A non-compliant vmknic is the first thing to fix.

Add a network portal (bind a vmknic)

esxcli iscsi networkportal add -A vmhba64 -n vmk2
esxcli iscsi networkportal add -A vmhba64 -n vmk3
esxcli storage core adapter rescan -A vmhba64
esxcli storage core path list

Replace vmhba64 and the vmknic names with your own. Rescan after binding so the new paths appear. Avoid add --force. It binds a vmknic that fails the multipathing checks (typically a port group with more than one active uplink), which leaves you with paths that do not fail over the way you expect. Fix the teaming instead: one active uplink, the rest unused.

Remove a network portal (unbind a vmknic)

esxcli iscsi networkportal remove -A vmhba64 -n vmk3
esxcli storage core adapter rescan -A vmhba64

With no active sessions on the vmknic, this is all you need.

“iSCSI session is active on the vmknic … use –force=true”

This is the message most people search for. ESX refuses to unbind because an iSCSI session still uses the vmknic, and suggests forcing it. Broadcom documents the same message when removing a static target (KB 370577). remove --force logs out every session on that vmknic. If a LUN has no other working path, that is an all-paths-down event for every VM on the datastore.

The safe order:

  1. Check that every LUN still has an active path through another vmknic: esxcli storage core path list -d naa.xxxx.
  2. List the sessions on the adapter and note the ISID of the session that uses this vmknic: esxcli iscsi session list -A vmhba64 and esxcli iscsi session connection list -A vmhba64.
  3. Log out only that session: esxcli iscsi session remove -A vmhba64 -s <isid>. Without -s or -n, session remove logs out every session on the adapter.
  4. Unbind: esxcli iscsi networkportal remove -A vmhba64 -n vmk3.
  5. Rescan the adapter and confirm the remaining paths.

Use --force only when the vmknic is already unusable (for example, a dead uplink during a NIC replacement) and you have confirmed the paths through the other vmknics.

iSCSI vmknics and netstacks

Searches for “netstack instance: iscsi” come from the output of esxcli network ip interface list, which prints the netstack every vmknic belongs to. To check:

esxcli network ip netstack list
esxcli network ip interface list

A vmknic gets its netstack when it is created. The reference documents esxcli network ip netstack add -N <name> and esxcli network ip interface add -i vmkN -p <portgroup> -N <netstack> (or -s <dvs-name> -P <dvport-id> on a distributed switch). An existing vmknic cannot be moved to another netstack; it has to be removed and created again. The main reason to give iSCSI its own netstack is a separate routing table. If your iSCSI traffic is not routed, the default netstack is the simpler choice. Before you port-bind vmknics on a custom netstack in production, confirm support with Broadcom’s vSphere Storage documentation and your array vendor, because the public documentation does not state it either way.

Verify paths after any change

esxcli iscsi networkportal list -A vmhba64
esxcli iscsi session list -A vmhba64
esxcli storage core path list
esxcli storage nmp device list

You want one active path per bound vmknic and target portal, and the path selection policy your array vendor recommends (usually Round Robin).

Common errors

  • “iSCSI session is active on the vmknic”: see the safe order above.
  • vmknic not compliant for binding: the port group has more than one active uplink, or a standby uplink. Set exactly one active uplink and move the others to unused. Don’t paper over it with add --force.
  • No new paths after add: no rescan, a target in a different subnet than the vmknic, or the array’s host access list does not include the initiator IQN.

Reference

Tesla P40 Passthrough GPU Disappears from the VM: “PCIe link to the device is down” on ESXi

My NVIDIA Tesla P40 in ESXi passthrough suddenly vanished from the VM: nvidia-smi could not talk to the driver and a VM reboot did not help. What the logs show, why only a host reboot fixes it, and how to make a homelab P40 stable (KB 418517).

Short answer: When a GPU in ESXi PCI passthrough suddenly disappears from the virtual machine, nvidia-smi says it couldn’t communicate with the NVIDIA driver and the host’s vmkernel.log says Skipping device reset … because PCIe link to the device is down, the card itself has locked up. Neither the guest nor ESXi can reset it, so rebooting the VM does nothing: only a reboot of the ESXi host brings it back (Broadcom KB 418517). That is exactly what happened to the NVIDIA Tesla P40 in my homelab. The lasting fix is to remove the reason the card drops off the bus: in a homelab that is usually heat, power or the PCIe riser.

What happened in my lab

My Tesla P40 runs in PCI passthrough to a single Linux VM for local LLMs. It sits on a PCIe riser and, because the P40 has no fan of its own, it is cooled by a blower fan on a 3D-printed adapter. After running fine for a long time, the GPU was suddenly gone from the VM in the middle of normal work, without any change on vSphere or in the VM:

$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.
Make sure that the latest NVIDIA driver is installed and running.

Rebooting the VM did not help; the GPU stayed unavailable. The ESXi host told the real story in /var/log/vmkernel.log:

Skipping device reset on <PCIe-ID> because PCIe link to the device is down.

Why a VM reboot does not help

When a VM with a passthrough device powers on or reboots, ESXi resets the PCI device so the guest gets a clean card. That reset needs a working PCIe link. Here the GPU stopped responding at the hardware or firmware level and the link to it went down, so ESXi skips the reset and the guest driver finds nothing it can talk to. The card stays stuck until the host is restarted and the slot is initialized again during boot. Broadcom’s KB describes the same: once a passthrough GPU becomes unresponsive, neither ESXi nor the VM can reset it without a full host reboot.

Confirm it

# in the Linux VM
nvidia-smi
sudo dmesg | grep -iE "nvrm|xid"            # look for Xid errors, e.g. "GPU has fallen off the bus"

# on the ESXi host
grep -iE "PCIe link|device reset" /var/log/vmkernel.log | tail -20
lspci | grep -i nvidia                          # is the card still enumerated?
esxcli hardware pci list | grep -B2 -A12 -i nvidia

Fix: reboot the host

  1. Shut down the VM that uses the GPU (and migrate or shut down the other VMs on that host).
  2. Put the host in maintenance mode and reboot it. If the GPU is still missing after a warm reboot, do a full power cycle so the card loses power completely.
  3. Power the VM on and check nvidia-smi again.

Why it happens, and how to keep a homelab P40 stable

The KB points to GPU hardware or firmware stability and lists what to check with the hardware vendor: card stability, PCIe slot health, GPU firmware or VBIOS, thermal and power conditions, and known reset or firmware lock-up issues. For a data-center card like the P40 running outside its intended server, three of them stand out:

  • Heat. The P40 is passively cooled and expects strong server airflow. A homelab fan adapter that is fine at idle can be too weak under a long inference run. Watch the temperature and throttle reasons during load, and lower the power limit if needed:
    nvidia-smi --query-gpu=temperature.gpu,power.draw,clocks_throttle_reasons.active --format=csv -l 5
    sudo nvidia-smi -pl 200 (default limit 250 W)
  • Power. The P40 draws up to 250 W through an 8-pin CPU (EPS) style connector, not a normal PCIe 8-pin. Use the correct adapter and a power supply rail with enough headroom; a load spike on a weak rail can drop the card.
  • The PCIe path. A riser or extension cable is the first suspect for a link that goes down. Reseat it, try a shorter or better-shielded riser, and if the board allows it, force the slot to PCIe Gen3 (or Gen2 for testing) instead of Auto.

Keep the passthrough settings the large-BAR card needs (pciPassthru.use64bitMMIO = true and pciPassthru.64bitMMIOSizeGB = 64), keep the server BIOS and the NVIDIA guest driver current, and note when the drops happen: always under heavy load points to heat or power, random times at idle point more to the riser or the card itself.

Further reading

VMware Hypervisor Recovery at Boot: Causes and Fixes for ESXi 7, 8 and 9

Why an ESXi host stops at VMware Hypervisor Recovery or No hypervisor found, and how to fix it: KB 324877, Shift+R rollback, repair install.

Short answer: if an ESXi host stops at “VMware Hypervisor Recovery” or reports “No hypervisor found”, the host could not load a valid image from its boot device. First check the firmware boot mode and the boot device itself. If the hang started right after a 7.0 Update 2 upgrade on a UEFI host, apply the boot loader workaround from KB 324877. If the image is damaged, roll back with Shift+R (only if the altbootbank still holds the previous build), or reinstall with “Install ESXi, preserve VMFS datastore” and restore the configuration backup.

What the “VMware Hypervisor Recovery” screen means

ESXi 7.0 and later keep the hypervisor in two boot banks on the boot device. According to the ESXi 7.0 system storage documentation, the layout is a System Boot partition (boot loader and EFI modules), Boot-bank 0 and Boot-bank 1 (ESXi boot modules, mounted as /bootbank and /altbootbank) and ESX-OSData (scratch, VMware Tools locker, core dumps, logs). The active bank holds the running build. The other bank holds the build from before the last update.

Broadcom documents the “VMware Hypervisor Recovery” message in one specific case, KB 324877: a UEFI host hangs at the boot loader showing this message after an ESXi install or upgrade. Broadcom does not publish a general definition of the screen. Treat it, and the related “No hypervisor found” error, as a sign that the host could not boot a working image. Look at the boot loader, the boot bank contents, the boot device and the firmware settings, in that order.

VMware Hypervisor Recovery: cause and fix table

SymptomLikely causeFixSource
UEFI host hangs at “VMware Hypervisor Recovery” right after upgrading to 7.0 Update 27.0 U2 boot loader loads EFI files in a way some firmware does not supportBoot the 7.0 U1 installer and copy its boot loader files to the boot partitionKB 324877
“No hypervisor found”Missing or corrupted ESXi files, failed SD card or disk, boot LUN lost, boot order or boot mode changedCheck boot mode (UEFI vs BIOS) and boot order, check hardware and storage, then reinstallKB 404825
Host fails or misbehaves after a patch or upgradeProblem with the new image in the active bankShift+R rollback to the build in the altbootbankKB 316592
Host silently reverts to the old build after an upgradeFaulty dual SD card RAID mirror, or not enough contiguous space to repartition for the 7.0 layoutMove to supported boot media, or clean install without a datastore on the boot deviceKB 418807, KB 426659
Upgrade precheck reports a vFAT bootbank problem, or PSOD while backing up vFAT partitionsCorrupted vFAT boot bank partitionsRepair the partitions with dosfsckKB 91136 (345227)
“Out of resources” when booting a newer release in legacy BIOS modeImage too large for legacy BIOS bootSwitch to UEFI on the old release first, then upgradeKB 84233 (313152)

UEFI boot hang after ESXi 7.0 Update 2 (KB 324877)

KB 324877 covers ESXi 7.0.x. Broadcom states that the way ESXi 7.0 U2 loads EFI files is not supported by some system firmware. The workaround replaces the boot loader on the system boot partition with the one from the 7.0 U1 installer. In outline:

  1. Boot the host from the ESXi 7.0 U1 installer ISO and open the shell with Alt+F1 (log in as root, no password).
  2. Identify the boot disk with ls /dev/disks and partedUtil getptbl. Partition 1 (systemPartition) is the boot partition.
  3. Back up the existing BOOTx64.EFI, mboot64.efi and safeboot64.efi, then copy the versions from /usr/lib/vmware/bootloader-installer onto the boot partition with mcopy.
  4. Reboot from the disk.

Follow the exact commands in the KB, because the device name differs on every host. This only applies to this 7.0 U2 case. It does not fix a damaged boot bank or a failed boot device.

Shift+R: roll back to the previous bootbank

KB 316592 describes the rollback for ESXi 6.x, 7.x, 8.x and ESX 9.x. Reboot the host and press Shift+R while the hypervisor progress bar is loading, not after it finishes. The KB suggests pressing it repeatedly once the “system is preparing to boot” message appears. ESXi then warns that the current hypervisor will be permanently replaced with the older build. Press Y to confirm and Enter to boot.

Rollback only works if the altbootbank still holds a different, working build. Before you rely on it, check the build= line of both banks from the ESXi Shell:

grep build= /bootbank/boot.cfg /altbootbank/boot.cfg
esxcli software profile get

Limits documented in KB 316592, KB 376175 and KB 386377:

  • Only one previous state (n-1) is kept. Each upgrade or VIB install or removal discards the older copy.
  • After the first reboot following an upgrade, any change that needs another reboot commits the new build to both banks. After that, rollback is gone.
  • In an HA cluster, exiting maintenance mode installs the FDM VIB. After an upgrade to 8.0 this discards the previous build (KB 386377). The workaround is to disable vSphere HA before the upgrade.
  • You cannot roll back from 7.0 to 6.x because the partition layout changed. Rollback between 7.0, 8.0 and 9.0 is possible.
  • Rollback does not revert the tools-light VIB, and it will not help if the boot device itself is failing.

VMware Hypervisor Recovery: “No hypervisor found”

KB 404825 lists the causes: missing or corrupted ESXi files, a failing SD card or disk, a power outage, a boot LUN lost through network or storage problems, a changed boot order, or a changed boot mode. Work through it in this order:

  1. Confirm the firmware is still in UEFI mode and the ESXi device is first in the boot order.
  2. Check the hardware logs (iDRAC, iLO, CIMC and similar) for a failed SD module, M.2 device or disk. For boot from SAN, have the storage team check the LUN and its paths.
  3. Try Shift+R if the other bank has a valid build.
  4. If nothing else works, reinstall ESXi (next section).

ESXi repair install: reinstall without losing VMs

Broadcom does not document a separate “repair” mode. The documented path is to boot the installer and run it over the existing installation. When the installer finds ESXi and a VMFS datastore on the selected disk, KB 392956 lists three options:

  • Upgrade ESXi, preserve VMFS datastore: upgrades the installation and keeps both the settings and the datastore.
  • Install ESXi, preserve VMFS datastore: fresh ESXi install that keeps the datastore. The host configuration is lost.
  • Install ESXi, overwrite VMFS datastore: wipes the installation and the datastore on that disk.

The TechDocs page for ESX 9.0 interactive upgrade adds that if the datastore cannot be preserved, you can only overwrite it or cancel. In that case, back up the datastore first.

A configuration backup can only be restored to the same build, so reinstall from the same build you backed up (KB 313510, KB 413897). If you need a specific ISO, see the download tip linked below.

Boot device causes

SD card and USB boot media

KB 85685 (now 317631) explains why SD/USB boot devices wear out on 7.0 and later: ESX-OSData takes constant writes and the wear cannot be monitored. SD/USB boot is still supported through VCF 9.0 on previously certified servers. Broadcom recommends a persistent device for OSData, or better, an SSD/NVMe boot device (8.x: 32 GB minimum, 128 GB recommended; 9.x: 128 GB minimum). KB 317891 lists the warnings 7.0 U3 and later raise without one. KB 413897 covers the migration: back up, reinstall the same build on the new disk, fix the boot order, restore.

Corrupted vFAT boot banks

If an upgrade precheck reports a vFAT bootbank problem, see the KB 91136 walkthrough linked below. It covers identifying the boot bank partitions with esxcli storage filesystem list and vmkfstools -P, then checking and repairing them with dosfsck in maintenance mode.

Legacy BIOS vs UEFI

KB 84233 (313152) warns that newer releases may fail to boot in legacy BIOS mode, for example with “Out of resources”. Switch to UEFI while still on the old release, reboot, fix any device naming issues, and only then upgrade. An unexpected BIOS/UEFI mode change is also listed as a cause in KB 404825.

Restore the configuration afterwards

After a clean install, restore the configuration bundle as described in KB 313510. The build must match the backup, and the host UUID should match. From 7.0 U2, a TPM-encrypted configuration cannot be force-restored to different hardware. The ESXi Shell workflow is:

# before the failure (or on a healthy host, regularly)
vim-cmd hostsvc/firmware/sync_config
vim-cmd hostsvc/firmware/backup_config

# on the reinstalled host, same build (check with: vmware -vl)
vim-cmd hostsvc/maintenance_mode_enter
# reboot, copy the bundle to /tmp/configBundle.tgz, then:
vim-cmd hostsvc/firmware/restore_config 0

The restore reboots the host automatically. KB 413897 notes that the bundle does not include SSH keys, custom scripts or third-party agents.

Preventing ESXi boot recovery problems

  • Boot from an endurance-rated SSD, NVMe or M.2 RAID device, not SD/USB. At minimum, give OSData a persistent device.
  • Run hosts in UEFI mode and switch before a major upgrade, not during it.
  • Schedule regular configuration backups with vim-cmd hostsvc/firmware/backup_config and keep the matching ISO for each build.
  • Check both boot bank builds before and after an upgrade. If you need a rollback option, validate the new build before any further reboot-requiring change, and consider disabling HA during the upgrade (KB 386377).
  • Do not create extra VMFS datastores on the boot device (KB 426659).

Sources

Related on vdan.cz

ESXi Shell Commands You Actually Use (ESXi 8 and ESX 9)

The ESXi shell commands admins really run over SSH, grouped by task: esxcli, vim-cmd, vmkping, esxtop and agent restarts, checked for ESXi 8 and ESX 9.

Short answer: the ESXi Shell commands you actually use fall into a small set: esxcli for host, network, storage and software state, vim-cmd for VMs when hostd works, vmkping for VMkernel connectivity and MTU tests, esxtop for live performance, and /etc/init.d/hostd restart or /etc/init.d/vpxa restart for stuck management agents. Enable SSH only for troubleshooting. Every esxcli command below exists in ESX 9.0 and in ESXi 8.0 U3 unless noted otherwise.

This is a curated list, grouped by task, not a full reference. For the complete namespace tree see ESXCLI commands for ESXi 8.0 and ESXCLI commands for ESX 9.0. Commands were checked against Broadcom’s ESXCLI Command Reference (8.0.3, 9.0.0 and 9.1.0); non-esxcli commands are taken from the Broadcom KB articles cited in each section. Placeholders: vmk0, vmnic0, datastore1, 10.0.0.10, naa.xxx.

Enable the ESXi Shell and SSH

The ESX Shell is deactivated by default and Broadcom recommends activating it for troubleshooting only. The shell is independent of lockdown mode: if the service is running, you can still log in to it even when the host is in lockdown mode. Two services matter: ESXi Shell (local console, Alt+F1) and SSH (remote, TCP 22).

  • vSphere Client: select the host, Configure > System > Services, select SSH or ESXi Shell, Edit Startup Policy > Start and stop manually, then Start.
  • Host Client: Manage > Services, select TSM (ESXi Shell) or the SSH service, Actions > Start.
  • DCUI: F2 > Troubleshooting Options > Enable ESXi Shell / Enable SSH. The same menu has Modify ESXi Shell and SSH timeouts.

Set a timeout so the service does not stay open. The availability timeout is the advanced option UserVars.ESXiShellTimeOut (seconds, 0 = disabled). To check SSH daemon settings from the shell:

esxcli system settings advanced list -o /UserVars/ESXiShellTimeOut
esxcli system ssh server config list

Sources: KB 311213 (Using ESXi Shell in ESXi), TechDocs Using the ESX Shell (9.0).

Host info and version

TaskCommand
Version and buildvmware -vl
Version, build, update levelesxcli system version get
Hostname and FQDNesxcli system hostname get
Vendor, model, serialesxcli hardware platform get
CPU packages, cores, threadsesxcli hardware cpu global get
Physical memoryesxcli hardware memory get
Uptime (microseconds)esxcli system stats uptime get
Time and NTPesxcli system time get; esxcli system ntp get
Boot deviceesxcli system boot device get
Disk and ramdisk usagedf -h; vdf -h

vmware -v / vmware -vl come from KB 345263 (determining the build number) and KB 323110; df -h and vdf -h (ramdisk usage, useful when /, /tmp or /var/log is full) from KB 318926 (investigating disk space).

Maintenance mode and reboot

esxcli system maintenanceMode get
esxcli system maintenanceMode set --enable true
esxcli system shutdown reboot --reason "Patching"
esxcli system maintenanceMode set --enable false

esxcli system shutdown reboot and poweroff take a --reason; the reference states the host must be in maintenance mode. On vSAN hosts, --vsanmode controls data evacuation; decide it before you enter maintenance mode, not after. Take a configuration backup first (vim-cmd hostsvc/firmware/sync_config then vim-cmd hostsvc/firmware/backup_config, KB 313510; details in Backup and restore ESXi host configuration data).

Networking

TaskCommand
Physical NICs, link, speed, driveresxcli network nic list
NIC details, firmware, driver versionesxcli network nic get -n vmnic0
NIC counters and errorsesxcli network nic stats get -n vmnic0
VMkernel interfaces, MTU, netstackesxcli network ip interface list
vmk IPv4 addressesesxcli network ip interface ipv4 get
vmk service tags (Management, vMotion…)esxcli network ip interface tag get -i vmk0
Routing tableesxcli network ip route ipv4 list
DNS serversesxcli network ip dns server list
Standard vSwitches and port groupsesxcli network vswitch standard list; esxcli network vswitch standard portgroup list
Distributed switches seen by the hostesxcli network vswitch dvs vmware list
VMs with active portsesxcli network vm list
ARP tableesxcli network ip neighbor list
Open connectionsesxcli network ip connection list
Firewall rulesetsesxcli network firewall ruleset list

vmkping and the MTU test

vmkping sends ICMP from a specific VMkernel interface. For a 9000 MTU path, use -d (do not fragment) and a payload of 8972 bytes (9000 minus 28 bytes of IP and ICMP headers); for 1500 use 1472. Add -S vmotion when the vmk is on the vMotion TCP/IP stack.

vmkping -I vmk1 10.0.0.10 -d -s 8972
vmkping -I vmk1 -S vmotion 10.0.0.10 -d -s 8972
vmkping -I vmk0 10.0.0.10 -d -s 1472 -c 3
esxcli network diag ping --host 10.0.0.10 --interface vmk1 --df --size 8972 --count 3

If 1472 works and 8972 fails, something in the path (vSwitch, uplink, physical switch port) is still at 1500. Sources: KB 313061 (testing jumbo frame pings), KB 344313 (testing VMkernel connectivity with vmkping). KB 313061 also lists esxcfg-vswitch -l and esxcfg-vmknic -l for a quick MTU overview.

Storage

TaskCommand
Devices (LUNs, NVMe, local disks)esxcli storage core device list
One deviceesxcli storage core device list -d naa.xxx
Paths and their stateesxcli storage core path list
Multipathing policy per deviceesxcli storage nmp device list
HBAsesxcli storage core adapter list
HBAs, detailed (9.0+)esxcli storage core adapter listdetailed
Rescan all adaptersesxcli storage core adapter rescan --all
Mounted datastores (VMFS, NFS, vSAN)esxcli storage filesystem list
VMFS extents to device mappingesxcli storage vmfs extent list
VAAI statusesxcli storage core device vaai status get -d naa.xxx
Manual UNMAP on VMFSesxcli storage vmfs unmap --volume-label datastore1
NFS v3 / v4.1 mountsesxcli storage nfs list; esxcli storage nfs41 list

Mount an NFS v3 datastore; --connections sets the number of TCP connections (nconnect, see nconnect support in the ESXi NFS client):

esxcli storage nfs add --host 10.0.0.10 --share /export/ds01 --volume-name nfs-ds01 --connections 4
esxcli storage nfs remove --volume-name nfs-ds01

Caution: storage vmfs unmap generates I/O on the array; run it outside peak hours. Do not remove an NFS datastore that still has registered or running VMs.

VMs from the shell

vim-cmd needs a responsive hostd. esxcli vm process works on the running VM world and is the fallback for a VM that ignores power operations.

TaskCommand
List registered VMs and their vmIDsvim-cmd vmsvc/getallvms
Power statevim-cmd vmsvc/power.getstate 12
Power on / guest shutdown / hard offvim-cmd vmsvc/power.on 12; vim-cmd vmsvc/power.shutdown 12; vim-cmd vmsvc/power.off 12
Snapshotsvim-cmd vmsvc/get.snapshot 12
Hung tasks of a VMvim-cmd vmsvc/get.tasklist 12
Register a VMvim-cmd solo/registervm /vmfs/volumes/datastore1/vm01/vm01.vmx
Running VMs with World IDesxcli vm process list
Kill a stuck VMesxcli vm process kill -t soft -w 2750434

Caution: use kill -t soft first, then hard, and force only as a last resort. If esxcli itself does not respond, Broadcom suggests localcli with the same syntax. Sources: KB 308360 (common VM tasks via CLI), KB 308457 (powering off an unresponsive VM).

Services and logs

/etc/init.d/hostd restart
/etc/init.d/vpxa restart
tail -f /var/log/hostd.log
tail -f /var/log/vmkernel.log

Restart hostd or vpxa individually when the host shows as disconnected or not responding in vCenter, then wait 1–2 minutes. Caution: Broadcom warns not to use services.sh restart (or Restart Management Agents in the DCUI) on hosts running vSAN, LACP, NSX, or shared graphics (vGPU, vDGA, vSGA); restart single services instead. Source: KB 320280 (restarting management agents).

TaskCommand
Syslog configurationesxcli system syslog config get
Set remote syslog targetesxcli system syslog config set --loghost udp://10.0.0.10:514
Apply syslog changesesxcli system syslog reload
Write a marker into all logsesxcli system syslog mark --message "start of test"
Advanced settings changed from defaultesxcli system settings advanced list --delta
Core dump targetesxcli system coredump partition get; esxcli system coredump file list

Performance: esxtop basics

Run esxtop in the SSH session and switch views with single keys: c CPU, m memory, n network, d disk adapter, u disk device, v disk per VM, V show only VMs, f fields, h help, q quit.

For support cases, collect batch data to a datastore (-b batch, -a all metrics, -d interval in seconds, -n number of samples). 300 samples at 2 s = 10 minutes:

esxtop -b -a -d 2 -n 300 > /vmfs/volumes/datastore1/esxtop-$(hostname)-$(date +"%FT%H:%M").csv

Make sure the target datastore has free space and is not the one you are troubleshooting. Sources: KB 382249 (using esxtop), KB 370279 (collecting esxtop batch data).

Software and VIBs

TaskCommand
Base image (ESXi version in the image)esxcli software baseimage get
Installed componentsesxcli software component list
Installed VIBsesxcli software vib list
One VIB in detailesxcli software vib get -n <vib-name>
Image profileesxcli software profile get
Host acceptance levelesxcli software acceptance get
Verify VIB signaturesesxcli software vib signature verify
Profiles in an offline depotesxcli software sources profile list -d /vmfs/volumes/datastore1/depot.zip
Dry-run a component installesxcli software component apply -d /vmfs/volumes/datastore1/component.zip --dry-run

--dry-run exists on component apply, vib install, vib remove and profile update; use it before every change. On vLCM-managed clusters, remediate from vCenter instead. A quick check for NSX VIBs before any management-agent restart: esxcli software vib list | grep -E 'nsx|vsipfwlib'.

ESX 9 notes

ESX 9.0 adds two top-level namespaces, licensing and nsx; 9.1 adds memtier. None exist in 8.0 U3, and memtier is not in 9.0.

TaskCommand
License credentials (9.0+)esxcli licensing credential list
License entitlement assignments (9.0+)esxcli licensing entitlement list
Stop NSX services (9.0+)esxcli nsx services stop
Memory tiering status (9.1)esxcli memtier status get
Memory tiering devices (9.1)esxcli memtier device list
Memory tiering configuration (9.1)esxcli memtier config get

Caution: the reference describes esxcli nsx services stop as stopping NSX services with the tn-delete option. It is not a diagnostic command; use it only under an NSX procedure or support guidance. Other 9.0 additions worth knowing: esxcli storage core adapter listdetailed, esxcli nvme auth get and the esxcli vsan dedup namespace. The full diff is in ESXCLI changes from ESXi 8.0 U3 to ESX 9.0 and 9.1.

Full command lists

For the complete list see the full ESX 9.0 ESXCLI list and the ESXCLI 8.0 mindmap.

Sources

Related on vdan.cz

ESXCLI Changes in ESX 9.0 and 9.1: New, Removed and Changed Commands vs ESXi 8.0 U3

ESX 9.0 adds 27 ESXCLI commands and drops 11 vs 8.0 U3 (SLP, WBEM/CIM); 9.1 adds 27 more incl. memtier. Full tables from Broadcom’s reference.

Short answer: compared with ESXi 8.0 U3, the ESX 9.0 ESXCLI reference adds 27 commands and drops 11. The additions are a new licensing namespace, an nsx namespace, NVMe host authentication, duplicate iSCSI sessions and a reorganised vSAN ESA dedup tree. The removals are SLP and WBEM/CIM, which ESX 9.0 no longer ships, plus one deprecated iSER setting. ESX 9.1 adds another 27 commands, led by a new memtier namespace for NVMe memory tiering and iSNS discovery for iSCSI.

Every command and option below comes from Broadcom’s published ESXCLI Command Reference for 8.0.3, 9.0.0 and 9.1.0. I compared the three versions namespace by namespace: 814 commands in 8.0.3, 830 in 9.0.0 and 857 in 9.1.0. What a host actually offers depends on its build and installed VIBs, so check with esxcli --help before you script against a command.

Full lists: ESXCLI commands for ESX 9.0 · ESXi 8.0 · ESXi 7.0 · ESXi shell commands you actually use

ESX 9.0 at a glance

8.0.39.0.09.1.0
Top-level namespaces1618 (+ licensing, nsx)19 (+ memtier)
Commands in the reference814830857
Added vs previous–2727
Removed vs previous–110
Existing commands with changed options–1213

New ESXCLI commands in ESX 9.0

The biggest groups are licensing, which manages license credentials and entitlement assignments on the host for the VCF 9 licensing model, and vSAN ESA dedup. The dedup store commands moved from esxcli vsan dedupstore to esxcli vsan dedup dedupstore, next to new object- and space-level commands. Broadcom’s own description of vsan dedup object create says to use it only to replace a corrupted dedup hash object, following a runbook or KB, or with support. Treat that group as support tooling, not day-to-day commands.

Command (new in 9.0)What it does
esxcli iscsi session duplicate addAdd duplicate sessions on current iSCSI configuration
esxcli iscsi session duplicate listList duplicate sessions on current iSCSI configuration
esxcli iscsi session duplicate removeRemove duplicate sessions on current iSCSI configuration
esxcli licensing credential addAdd a new license credential
esxcli licensing credential listList the installed license credentials
esxcli licensing credential removeRemove an existing license credential
esxcli licensing entitlement addAdds a new license entitlement assignment
esxcli licensing entitlement listLists the license entitlement assignments
esxcli licensing entitlement removeRemoves a license entitlement assignment
esxcli licensing entitlement setUpdates an existing license entitlement assignment
esxcli nsx services stopCLI to stop NSX services with tn-delete option
esxcli nvme auth getGet NVMe host authentication information
esxcli nvme auth setSet NVMe host authentication information
esxcli nvme device log commandeffects getGet NVMe commands supported and effects log
esxcli storage core adapter listdetailedDetailed listing of all HBAs registered with PSA
esxcli vsan dedup dedupstore distribution getGet the dedup store distribution info of vSAN ESA (Express Storage Architecture) cluster
esxcli vsan dedup dedupstore getGet vSAN ESA dedup store view
esxcli vsan dedup dedupstore resync listGet detailed information of vSAN ESA resyncing dedup objects
esxcli vsan dedup dedupstore resync summary getGet summary information of vSAN ESA resyncing dedup objects
esxcli vsan dedup object createCreate a dedup object
esxcli vsan dedup object distribution getGet the chunk objects where the given regular object has data on
esxcli vsan dedup object reference getGet the regular objects whose data the given chunk object holds
esxcli vsan dedup object space getGet vSAN ESA regular object dedup space efficiency information
esxcli vsan dedup space getGet vSAN ESA dedup space efficiency information
esxcli vsan iscsi vip listList configured vSAN iSCSI VIPs
esxcli vsan witness license getGet vSAN witness license info on the host
esxcli vsan witness license setSet vSAN witness license config on the host

ESXCLI commands removed in ESX 9.0

The VCF 9.0 vSphere product support notes explain most of this list. ESX 9.0 removes CIM, the Small Footprint CIM Broker (SFCB), WSMan and the OpenSLP stack, which were deprecated in 8.0, and CIM providers for ESX 8.x and earlier aren’t supported on 9.0. Any script or monitoring check that calls esxcli system slp or esxcli system wbem fails on a 9.0 host. If hardware monitoring relied on CIM, ask the server vendor for a replacement.

Command (gone in 9.0)Replacement / reason
esxcli rdma iser params setWas already marked deprecated in 8.0.3
esxcli system slp getOpenSLP stack removed in ESX 9.0
esxcli system slp searchOpenSLP stack removed in ESX 9.0
esxcli system slp setOpenSLP stack removed in ESX 9.0
esxcli system slp stats getOpenSLP stack removed in ESX 9.0
esxcli system wbem getCIM/SFCB and WSMan removed in ESX 9.0
esxcli system wbem provider listCIM/SFCB and WSMan removed in ESX 9.0
esxcli system wbem provider setCIM/SFCB and WSMan removed in ESX 9.0
esxcli system wbem setCIM/SFCB and WSMan removed in ESX 9.0
esxcli vsan dedupstore distribution getMoved to esxcli vsan dedup dedupstore …
esxcli vsan dedupstore getMoved to esxcli vsan dedup dedupstore …

Existing commands with new or removed options (8.0 U3 → 9.0)

These commands keep their name but their options changed. The removals matter most for scripts. For example, esxcli system health report get no longer lists --all-reports in 9.0.

CommandOptions added in 9.0Options removed
esxcli iscsi adapter discovery statictarget remove--force | -f–
esxcli network ip interface set--urpf | -u–
esxcli network ip netstack set--ip-strong-es | -i–
esxcli nvme device namespace list--namespace | -n–
esxcli storage core adapter list--adapter | -a–
esxcli system health report get–--all-reports
esxcli vsan cluster join--client-network | -n–
esxcli vsan cluster new--client-network | -n–
esxcli vsan cluster unicastagent add--traffic-type | -F–
esxcli vsan debug object list--dedup-obj-type–
esxcli vsan storagepool unmount--evacuation-reason | -r–
esxcli vsan vdx create--backing-type | -b, --fs-type | -t–

New in ESX 9.1: memtier and more

The 9.1 reference adds a top-level memtier namespace to enable, configure and check NVMe memory tiering. The 9.0 reference has no memory-tiering namespace. For what memory tiering is and how it behaves, see NVMe memory tiering in vSphere 9.0. 9.1 also adds iSNS discovery for software iSCSI, NVMe flexible data placement, NFS restore commands and NFSv4.1 Kerberos credential commands. Nothing was removed between 9.0 and 9.1.

Command (new in 9.1)What it does
esxcli iscsi adapter discovery isns addAdd iSNS discovery address
esxcli iscsi adapter discovery isns listList iSNS discovery addresses
esxcli iscsi adapter discovery isns param getGet parameters on iSNS discovery address
esxcli iscsi adapter discovery isns param setSet iSNS discovery parameters
esxcli iscsi adapter discovery isns removeRemove iSNS discovery address
esxcli iscsi adapter isnsparam getGet the iSNS parameters for the iSCSI Host Bus Adapter
esxcli iscsi adapter isnsparam setSet the iSNS parameter for the iSCSI Host Bus Adapter
esxcli licensing registration getShow details about license registration
esxcli memtier config deleteDelete the memory tiering configuration
esxcli memtier config getGet the memory tiering configuration
esxcli memtier config setSet the memory tiering configuration
esxcli memtier device listList memory tiering devices
esxcli memtier disableDisable memory tiering
esxcli memtier enableEnable memory tiering
esxcli memtier status getShow the currently deployed Memory Tiering configuration
esxcli nvme device feature fdp getGet NVMe device feature flexible data placement information
esxcli nvme device feature fdp setSet NVMe device feature flexible data placement information
esxcli rdma device sniffer getGet RDMA device sniffer settings
esxcli rdma device sniffer setSet RDMA device sniffer settings
esxcli storage nfs restoreRestore previously mounted NFSv3 volumes
esxcli storage nfs41 credential getGet NFSv4.1 kerberos default user credentials
esxcli storage nfs41 credential setSet NFSv4.1 kerberos default user credentials
esxcli storage nfs41 credential unsetUnset NFSv4.1 kerberos default user credentials
esxcli storage nfs41 restoreRestore previously mounted NFS41 volumes
esxcli storage osdata getGet OSData information
esxcli vsan dedup scan getGet vSAN ESA dedup scanner information
esxcli vsan maintenancemode recommissionRecommission vSAN node, the command only set vSAN decommission state to non-decommissioned but not set host…

Option changes between 9.0 and 9.1. Note that esxcli vsan vdx create replaced three of its options and esxcli storage core claimrule add lost --if-unset:

CommandOptions added in 9.1Options removed
esxcli nvme device namespace create--anagrpid | -a, --endgid | -e, --lbstm | -l, --nvmsetid | -N, --phl | -P–
esxcli software apply--skip-reserved-vibs-caching–
esxcli software profile install--skip-reserved-vibs-caching–
esxcli software profile update--skip-reserved-vibs-caching–
esxcli storage core claimrule add–--if-unset
esxcli storage nfs param set--force | -f, --resolve-hostname | -r–
esxcli storage nfs41 param set--force | -f, --resolve-hostname | -r–
esxcli system health report get–--filename | -f
esxcli vsan health cluster list--perspective | -p–
esxcli vsan network security set--is-server-cluster | -s, --transition-state | -t–
esxcli vsan trace get--prettyjson | -p–
esxcli vsan trace set--configs | -c–
esxcli vsan vdx create--disk-path | -d--backing-type | -b, --disk-folder | -f, --disk-name | -d

Checking a host yourself

Broadcom notes that the list of commands depends on the system you run it on. To see what a specific host supports, and to compare a host before and after an upgrade, use ESXCLI’s built-in introspection:

# Every namespace and command this host knows
esxcli esxcli command list

# Help for one namespace or command
esxcli licensing --help
esxcli system health report get --help

# Save the list before and after an upgrade, then diff the two files
esxcli esxcli command list > /tmp/esxcli-before.txt

Copy the output files off the host, for example with scp, and compare them with diff. That also catches commands that come from vendor VIBs, which Broadcom’s reference doesn’t list.

What to update in your scripts

  • Remove or guard any call to esxcli system slp … and esxcli system wbem … before you upgrade hosts to 9.0.
  • Change esxcli vsan dedupstore … to esxcli vsan dedup dedupstore ….
  • Check scripts that pass --all-reports to esxcli system health report get (gone in 9.0) or --filename (gone in 9.1).
  • Scripts using esxcli vsan vdx create need the new options on 9.1.
  • On VCF 9 hosts, license through the new esxcli licensing commands (credentials and entitlements).

Sources

Related on vdan.cz

Support for nConnect Feature Added in ESXi NFS Client

Use nConnect in ESXi 8.0 U1+ for several TCP connections per NFSv3 datastore: esxcli syntax, limits (4/8/256), runtime changes in 8.0 U2 and how to check.

Short answer: since ESXi 8.0 Update 1 the NFS client supports nConnect: several TCP connections per NFSv3 datastore. Mount with esxcli storage nfs add ... -c <connections> (default 1, up to 4 per datastore, or 8 with /NFS/MaxConnectionsPerDatastore, and 256 per host in total). Since 8.0 Update 2 you can change it on a mounted datastore with esxcli storage nfs param set -v <label> -c <n>. It’s NFSv3 only and CLI only.

With the release of VMware ESXi 8.0 Update 1, VMware quietly introduced nConnect support in the NFS client — a long-awaited enhancement for environments using NFSv3 datastores.

The nConnect feature allows administrators to establish multiple parallel TCP connections per NFS datastore, significantly improving throughput, resiliency, and traffic isolation.

This guide walks you through how to configure nConnect, supported versions, and key operational tips.


🆕 What’s New in ESXi 8.0U1

From ESXi 8.0U1 onwards, a new parameter -c (number of connections) is available in esxcli when mounting NFSv3 datastores.

esxcli storage nfs add \
  -H <host> \
  -v <volume-label> \
  -s <remote_share> \
  -c <number_of_connections>

Key points:

  • Supported for NFSv3 only (NFSv4.1 not yet supported).
  • Currently available only via CLI, not via vCenter UI.
  • Host Profiles don’t officially support it yet (though there’s a workaround, see below).
  • Default max connections per datastore: 4
  • Adjustable up to 8 connections via advanced option:
esxcfg-advcfg -s 8 /NFS/MaxConnectionsPerDatastore

⚠️ The total number of connections across all mounted NFSv3 datastores is capped at 256.


Runtime Scaling in ESXi 8.0U2

The nConnect feature got a useful upgrade in VMware ESXi 8.0 Update 2.
You can now increase or decrease the number of TCP connections on an existing NFSv3 mount without unmounting the datastore:

esxcli storage nfs param set \
  -v <volume-label> \
  -c <number_of_connections>

This flexibility helps admins fine-tune NFS performance on the fly — for example, ramping up connections during heavy backup windows or scaling down during low I/O periods.

Note: In both 8.0U1 and 8.0U2, the default number of connections remains 1.


How to Check Active NFS Connections

To view the number of RPC clients currently used by mounted NFSv3 datastores, run:

vsish -e get /vmkModules/nfsclient/info

Sample output:

NFS Connections Info {
mountedVolumes:10
totalConnections 40
}

This command is particularly handy for performance troubleshooting and capacity planning.


🧪 Experimental: Host Profile Integration

Although official Host Profile support for nConnect is not yet released, VMware added an experimental field named “Number of TCP connections” in:

Host Profile → Storage Configuration → NFS

What it does:
When you set a value here, ESXi will use that number of parallel TCP sessions when mounting the NFS datastore during profile remediation.

This is ideal for consistent deployment across multiple hosts in larger clusters.


Best Practices & Tips

  • Start with 4 connections and scale up as needed after performance testing.
  • Keep the 256 connection limit in mind when mounting multiple datastores.
  • Use consistent configuration across hosts to avoid mount mismatches.
  • Monitor performance metrics via VMware vSphere or CLI.
  • If you’re using Host Profiles, experiment carefully with the TCP connection parameter.

Summary

The nConnect feature in ESXi’s NFS client marks a major step forward in improving NFSv3 performance and scalability in enterprise environments.

Whether you’re running backup workloads, large VMs, or latency-sensitive applications, multiple TCP connections per NFS datastore can make a measurable difference.

If you’re upgrading to VMware ESXi 8.0U1 or later, nConnect is worth testing and adopting in your storage strategy.


Links

NVMe Memory Tiering in vSphere 8.0 U3, ESX 9.0 and 9.1: Requirements, Configuration and ESXCLI

NVMe Memory Tiering across vSphere 8.0 U3, ESX 9.0 and 9.1: versions, requirements, DRAM:NVMe ratio, tierdevice and memtier ESXCLI commands.

Short answer: NVMe Memory Tiering lets ESX use a local, dedicated NVMe device as a second memory tier (Tier 1) behind DRAM (Tier 0). Only VM memory is tiered. It was a tech preview in vSphere 8.0 U3 (default DRAM:NVMe 4:1), became GA in ESX/VCF 9.0 (default 1:1, DRS aware, hardware RAID, encryption), and ESX 9.1 adds the esxcli memtier namespace, automatic partitioning, activation without a reboot and software mirroring. It works well when active memory stays under 50% of DRAM.

What memory tiering does

The host presents DRAM plus the NVMe tier to VMs as one contiguous memory pool. The ESX memory manager tracks how recently and how often each VM page is accessed. Frequently used pages stay in DRAM and cold pages go to NVMe. Kernel memory is never tiered. Tiering is off by default.

The 9.1 best-practices page describes the placement behaviour. Cold pages start moving to NVMe when host memory consumption reaches roughly 80% of DRAM. After that, the algorithm keeps scanning VM memory in batches and moves pages between tiers at the end of each cycle. How fast it scans depends on memory pressure. So a host whose VMs fit in DRAM will show little or no Tier 1 usage.

This is not swap. Tiering picks pages by age and activity and uses a dedicated local device, while host swap picks pages crudely and often sits on shared storage. The 9.1 guidance also says not to combine tiering with memory overcommit.

Version status: 8.0 U3 vs 9.0 vs 9.1

vSphere 8.0 U3ESX / VCF 9.0ESX / VCF 9.1
StatusTech preview (KB 311934)GA, production supportedGA
Default DRAM:NVMe4:1 (Mem.TierNvmePct = 25)1:11:1 (tier-size-pct 100)
Ratio range1–400%docs recommend 100% or lower1–400%
Tier partitionManual, esxcli system tierdevice createManual, esxcli system tierdevice createCreated automatically
EnableKernel boot option + rebootVMkernel.Boot.memoryTiering + rebootesxcli memtier enable or Configuration Manager, maintenance mode, no reboot
RedundancyNo RAID recommended, one device per hostHardware RAID (VROC, tri-mode)Software mirroring (2 devices) or hardware RAID
EncryptionNot supportedHost or VM levelHost or VM level
DRSNot recommendedTiering-aware placementTiering-aware placement
Mixed clustersNot supportedSupportedSupported

The 8.0 U3 tech preview guide attached to KB 311934 also recommended VDI-type workloads only, no vSAN in the same cluster, and disabled large pages by default. Treat 8.0 U3 as a lab feature and plan production on 9.x.

Requirements and limits

ItemRequirement (9.0 / 9.1 TechDocs)
VersionsvCenter and ESX 9.0+ (9.1 features need 9.1 on both)
Device classvSAN-cache-class NVMe from the Broadcom Compatibility Guide, vSAN SSD: Endurance Class D (≥ 7300 TBW), Performance Class F or G. The 9.1 docs add Mixed Use, 3 DWPD or higher
AttachmentLocal only. No NVMe-oF or Ethernet-attached devices
PartitionDedicated device, all other partitions deleted. Tier partition max 4 TB
SharingNot usable if it is already a vSAN cache/data disk or VMFS datastore (9.1: remove from the disk group first)
HostNo Intel Optane PMem, no NVDIMM-N. Maintenance mode for any change
Unsupported operationsSuspend to memory. NVMe hot-plug / prepare-to-remove. Quick Boot (9.1)
SizingNVMe used = the smaller of (DRAM × ratio) and the partition size. Keep active memory ≤ 50% of DRAM

VM types that are excluded

ESX 9.0: these VMs cannot power on on a tiered host: latency sensitivity high, SEV / SGX / TDX, Fault Tolerance, VMs with more than 1 TB and 128 vCPUs, and nested virtualization (vhv.enable). These are kept in DRAM automatically: 1 GB pages, preallocated memory, PCI passthrough / UPT, and vQAT.

ESX 9.1: low-latency, confidential, FT, large and nested VMs can now power on, but they are still left out of tiering and stay in DRAM. VBS guests are the exception because they can opt in with sched.mem.enableNestedTiering = TRUE. The 9.1 VCF blog says nested VMs participate in tiering, which is broader than TechDocs, so go by the documentation. For VMs above 512 GB and 64 vCPUs, the 9.1 docs recommend turning tiering off and using a full reservation.

Configure on vSphere 8.0 U3 (tech preview)

esxcli system maintenanceMode set --enable true
esxcli storage core adapter device list
# delete any existing partitions first (partedUtil, KB 1036609)
esxcli system tierdevice create -d /vmfs/devices/disks/<nvme_device>
esxcli system tierdevice list
esxcli system settings kernel set -s MemoryTiering -v TRUE
# optional ratio, default 25 (= 4:1)
esxcfg-advcfg -s 50 /Mem/TierNvmePct
reboot

The only option tierdevice create takes is -d / --nvmedevice, and tierdevice delete takes the same one. An older version of this article showed extra vendor/ID arguments, which were wrong.

Configure on ESX 9.0

  1. Put the host in maintenance mode, delete all partitions on the NVMe device (KB 323144 / KB 1036609), then run esxcli system tierdevice create -d /vmfs/devices/disks/<nvme_device> and check it with esxcli system tierdevice list. Device preparation can’t be done from the vSphere Client in 9.0.
  2. Enable tiering. Cluster: Configure → Desired State → Configuration → Draft → esx → vmkernel → options, set memory_tiering to True. Host: Advanced System Settings, set VMKernel.Boot.memoryTiering to true. Both need a cluster managed by a single image with vSphere Configuration Profiles. Broadcom’s 9.1 memtier blog confirms that 9.0 still uses the esxcli system settings kernel namespace from the CLI.
  3. Optionally set Mem.TierNvmePct for the ratio (25 = 4:1), Mem.EncryptTierNVMe = 1 for host-level encryption, or sched.mem.EncryptTierNVMe = TRUE for one VM. Reboot.

To exclude a single VM, power it off and set sched.mem.enableTiering = FALSE.

Configure on ESX 9.1: the esxcli memtier namespace

In 9.1, every tiering operation is in one namespace. The partition is created automatically, and enabling needs maintenance mode but no reboot. Commands and options below are from the ESXCLI 9.1.0 reference:

esxcli memtier device list --available        # -a; --configured / -c
esxcli memtier enable -d <device_1>
esxcli memtier enable -d <device_1>,<device_2>   # software mirror
esxcli memtier enable -d <device_1> -c t -r 50    # encryption, 2:1 ratio
esxcli memtier config get                     # -d / -e / -c / -r for single values
esxcli memtier config set --tier-size-pct 25  # 1-400, % of DRAM
esxcli memtier config set --encryption true
esxcli memtier config set -d <new_device>,<mirror_device>
esxcli memtier status get
esxcli memtier disable                        # keeps tier partition
esxcli memtier disable -d                     # --delete-devices
esxcli memtier config delete                  # removes config and partition

For a cluster, go to Configure → Desired State → Configuration → Draft → esx → memtier → nvme. Set enable and pick a device for each host (by BIOS UUID). You can also set a mirror_device, encryption and tier_size_pct. Configuration Manager puts the host into maintenance mode by itself. The mirror device has to be at least as large as the primary. If config get shows Enable true but status get reports Disabled, activation failed, so check the logs and alarms.

In 9.1, if you set sched.mem.enableTiering = FALSE on a VM, also give it a full memory reservation. Without one, its pages can end up in host swap, which is slower than the NVMe tier.

Monitor it

  • 9.0: Host → Configure → Hardware → Overview shows Tier 0 (DRAM), Tier 1 (NVMe) and Total. The cluster Hosts tab has a Memory Tiering column, which reads “Software” for tiered hosts.
  • 9.1: host and cluster Summary Memory cards show tier state and Tier 1 consumption, and there are tiering-specific performance charts. From the CLI, use esxcli memtier status get.
  • Capacity check: compare active memory with DRAM. A host whose active memory is under 50% of DRAM is a good candidate.

Operational notes

  • Expect vMotion to take roughly 1.5–2x longer for tiered VMs, because pages have to be read back from NVMe.
  • If a single-device tier fails (9.1 docs), VMs that touch a page on the failed device power off. With vSphere HA restart policies they can be restarted on another host. DRAM-only VMs keep running. This is the reason for mirroring or RAID.
  • If you turn tiering off on a host, the cluster loses that capacity, and in an overcommitted cluster some VMs may then fail to power on.

Lab notes

Sources

Related on vdan.cz

lsdoctor IndexError: list index out of range on vCenter 8.0.3 – Custom /etc/issue Banner Fix

lsdoctor fails with IndexError in _getVersion when /etc/issue holds a custom banner. Restore the default text, run lsdoctor, revert (KB 320837).

Short answer: If lsdoctor.py stops right after launch with IndexError: list index out of range in _getVersion, the vCenter Server Appliance has a custom login banner in /etc/issue. lsdoctor reads the vCenter version from that file. Broadcom KB 320837 says to put the default two-line content back for the run, with your real version number, run lsdoctor, then restore your banner. The KB does not list an lsdoctor build that fixes this.

This page covers only this one error. For download, options (-l, -t, -s, -u, -r, -p) and usage, see the lsdoctor guide: check and repair the vCenter Lookup Service.

The lsdoctor IndexError traceback

This is the error exactly as Broadcom KB 320837 shows it (vCenter Server 8.0.3.00000, lsdoctor-240201). It happens before lsdoctor asks for the SSO password, so no check or repair has run yet:

[ ~/lsdoctor-240201 ]# ./lsdoctor.py -l
Traceback (most recent call last):
  File "/root/lsdoctor-240201/./lsdoctor.py", line 547, in <module>
    main()
  File "/root/lsdoctor-240201/./lsdoctor.py", line 458, in main
    params = parameters()
  File "/root/lsdoctor-240201/./lsdoctor.py", line 136, in parameters
    params = get_params()
  File "/root/lsdoctor-240201/lib/utils.py", line 1349, in __init__
    version, build = _getVersion()
  File "/root/lsdoctor-240201/lib/utils.py", line 145, in _getVersion
    version = version.rsplit(' ',1)[1]
IndexError: list index out of range

Here is how to recognise it: the last frame is _getVersion in lib/utils.py, and the failing line is version.rsplit(' ',1)[1]. That code splits a line of text at the last space and takes the part after it. It expects something like VMware vCenter Server 8.0.3.00000. If the line it reads has no space-separated version at the end, the split returns a single element and index [1] does not exist. Line numbers differ between lsdoctor builds. The function name and the rsplit line are the parts to match.

Root cause: a custom /etc/issue login banner

KB 320837 says the script “will fail with below error” if /etc/issue has been changed to hold a custom login banner. Security teams often ask for a legal banner on the appliance console and SSH, and the easy way is to overwrite /etc/issue. The appliance keeps running normally, but lsdoctor can no longer work out which vCenter build it is on. That build decides which service registration template lsdoctor loads.

The default content the KB expects (the example is for 8.0.3.00000) is:

VMware vCenter Server 8.0.3.00000
Type: vCenter Server with an embedded Platform Services Controller

Fix: restore the default banner for the lsdoctor run

This follows the KB’s procedure. It adds a backup of your banner and uses standard shell commands on the VCSA as root:

  1. Note your exact vCenter version, for example from the VAMI Summary page (https://<vcenter>:5480). The KB warns that the version in the file must match your actual vCenter version. Do not paste 8.0.3.00000 into a different build.
  2. Back up the custom banner: cp /etc/issue /root/issue.custom
  3. Write the default content, with your own version on line 1 (see the block below).
  4. Run lsdoctor from its extracted directory, for example python lsdoctor.py -l. It should now ask for the SSO administrator password and run.
  5. Put your banner back: cp /root/issue.custom /etc/issue
cp /etc/issue /root/issue.custom
cat > /etc/issue <<'EOF'
VMware vCenter Server 8.0.3.00000
Type: vCenter Server with an embedded Platform Services Controller
EOF
cat /etc/issue
python lsdoctor.py -l
cp /root/issue.custom /etc/issue

If you plan to use a repair option (-t, -s, -u, -r) rather than the read-only -l check, leave the default banner in place until you have finished. Follow the KB warning too: take offline snapshots of every vCenter/PSC in the SSO domain at the same time before any change.

Is there an lsdoctor build that fixes the IndexError?

Not a documented one. In September 2026, KB 320837 attaches lsdoctor-260716.zip (hash added 07/16/2026). The same KB still includes the custom-banner note and the traceback above, and gives no fixed build. So download the current ZIP from the KB (the KB page is public and lists the MD5/SHA256 hashes), but do not expect a newer build alone to remove the error. Plan on the banner workaround.

ItemStatus in KB 320837
ErrorIndexError: list index out of range in _getVersion
Trigger/etc/issue changed to a custom login banner
Example version in the KBvCenter Server 8.0.3.00000 (lsdoctor-240201 in the traceback)
WorkaroundRestore default content with the correct version, run, then revert
Fixed lsdoctor buildNot documented (current attachment: lsdoctor-260716.zip)
Supported vCenter6.7 and later, including 7.x, 8.x and 9.x

Impact and how to avoid it next time

  • The custom banner does not affect vCenter services. It only blocks lsdoctor, so you cannot run Lookup Service checks or repairs until the file is back to default.
  • Add “restore default /etc/issue” to your lsdoctor runbook. Keep the approved banner text in a file (for example /root/issue.custom) so reverting takes one command.
  • After a vCenter update, the version in the default content changes. Always write the version of the build you are running now.

Sources

Related on vdan.cz

Intel Skylake and ESX 9: Skylake-SP Supported in Deprecated Mode for VCF 9.x, Not Beyond

Is Intel Skylake supported in ESX 9? Skylake-SP runs in Deprecated mode for all VCF 9.x (KB 428874), not beyond. Skylake-W stays discontinued.

Illustration of an Intel Skylake server CPU reaching end of support in vSphere

Short answer: Yes, with limits. Broadcom KB 428874 was updated on 18 September 2026. Intel Skylake-SP (Xeon Scalable 1st generation: Platinum 8100, Gold 6100/5100, Silver 4100, Bronze 3100) is now supported in Deprecated mode for all of VCF 9.x, including 9.0 GA. Support depends on BCG listing and OEM hardware support. It will not be extended beyond 9.x. Skylake-W (Xeon W-2100) remains discontinued.

This post covers Skylake specifically. For the general installer warning (“may not be supported in future ESXi releases”) and the full CPU tables, see KB 82794 explained.

Intel Skylake ESXi 9 support at a glance

Skylake variantBCG CPU seriesCPU Info / CPUIDKB 318697 status for 9.x
Skylake-SPXeon Platinum 8100, Gold 6100/5100, Silver 4100, Bronze 31006.55.4 / 0x00050654See KB 428874: Deprecated mode for VCF 9.x
Skylake-DXeon D-21006.55.4 / 0x00050654See KB 428874
Skylake-SXeon E3-1200-v5, E3-1500-v56.5E.3 / 0x000506E3See KB 428874
Skylake-WXeon W-21006.55.4 / 0x00050654Discontinued

Note: the text of KB 428874 names only Skylake-SP explicitly. KB 318697 points the Skylake-D and Skylake-S rows to it as well. If you run Xeon D-2100 or E3 v5 systems, check the BCG listing for your exact server model. The other VCF 9 hardware checks (I/O devices, deprecated drivers, vSAN ESA) are in How to check VCF 9 hardware compatibility.

How Skylake support changed from vSphere 8 to VCF 9.x

  • vSphere 8.x: Skylake was on the deprecation list. The ESXi 8 installer warned that the CPU may not be supported in future releases, and installation continued. The original version of this post (October 2024) covered that stage.
  • VCF 9.0 GA: Skylake-SP was discontinued and not listed in the BCG for 9.0. In February 2026, Broadcom’s VCF blog still said Skylake needed an RPQ, with “VCF Software Only” support on approval.
  • KB 428874, 4 July 2026: the KB added Skylake-SP support for VCF 9.1.x.
  • KB 428874, 18 September 2026: support was expanded to VCF 9.x, including 9.0 GA. No TQ/RPQ is required for certain OEM server models marked “VCF Supported. Confirm w/ Vendor”.

Advice from older posts and forums, such as “Skylake does not install on ESX 9” or “file an RPQ”, may be out of date. Check the KB’s update date before you decide.

What Deprecated mode means for an Intel Skylake server

  • Deprecated means still supported. Per KB 318697, the CPU series is supported through the life of the VCF release, its updates and patches, or until End of General Support.
  • The OEM must list the server model in the Broadcom Compatibility Guide. KB 428874 links a BCG search for Skylake systems marked “VCF Supported. Confirm w/Vendor” on ESXi 9.0 and 9.1.
  • “VCF Supported. Confirm w/Vendor” means no further system-level driver enhancements and best-effort fixes or workarounds for the VCF software only.
  • Hardware, BIOS, firmware and microcode support comes from the server OEM and Intel under your contract with them. If you have no such support, you carry the risk.
  • vSAN OSA and ESA ReadyNodes with Skylake-SP are also in Deprecated mode and still supported.

Installing or upgrading ESX 9 on Skylake-SP

KB 428874 says the specific install and upgrade paths for VCF 9.x are described in its attached document, ESX_9.x_Install_Upgrade_using_OVERRIDE_on_Skylake_CPU_based_Servers-v1.3-09182026.pdf. Download it from the KB page and follow it for your path (fresh install, ESX upgrade, or a VCF-managed upgrade). I have not reproduced its steps here because they are versioned and still changing. This is Broadcom’s supported procedure. Don’t confuse it with the unsupported homelab boot option allowLegacyCPU=true.

Is it Skylake or Cascade Lake? Check the stepping

Skylake-SP and Cascade Lake-SP use the same family and model (hex 6.55, decimal model 85). Only the stepping tells them apart: 4 is Skylake-SP, and 6 or 7 is Cascade Lake. Cascade Lake is Deprecated in 9.x and listed as Discontinued for the next major release.

esxcli hardware cpu list
# look at: Brand, Family, Model, Stepping (decimal values)
# Family 6, Model 85, Stepping 4    -> Skylake-SP (6.55.4)
# Family 6, Model 85, Stepping 6/7  -> Cascade Lake (6.55.6 / 6.55.7)

What happens after 9.x

KB 428874 is explicit: Skylake-SP support will not be extended to VCF releases beyond 9.x. In KB 318697’s definitions, a “future major release” is the release after the 9.x series. The same table lists Cascade Lake, AMD EPYC 7001 (Naples) and 7002 (Rome), Coffee Lake Xeon E-2100/E-2200 and Atom C3000 as discontinued in that release. For planning:

  1. Inventory hosts by CPU stepping, not only by server model.
  2. Treat Skylake-SP hosts as “last stop: VCF 9.x” and track the 9.x End of General Support date in the Broadcom Product Lifecycle Matrix.
  3. Confirm OEM hardware and firmware support for the remaining lifetime. KB 428874 puts this responsibility on the OEM and Intel.
  4. Plan the refresh so the cluster does not block the next major upgrade. Avoid mixing in new Cascade Lake purchases, which hit the same wall.

Sources

Related on vdan.cz

Add nConnect Support to NFS v4.1

vSphere 8.0 Update 3 adds nConnect for NFS v4.1 datastores: multiple connections per session on one IP, extending session trunking.

Starting with vSphere 8.0 Update 3, nConnect support has been added for NFS v4.1 datastores. This feature enables multiple connections using a single IP address within a session, thereby extending session trunking functionality to that IP. With nConnect, multipathing and nConnect coexist, allowing for more flexible and efficient network configurations.

Benefits of nConnect

Traditionally, vSphere NFSv4.1 implementations create a single TCP/IP connection from each host to each datastore. This setup can become a bottleneck in scenarios requiring high performance. By enabling multiple connections per IP, nConnect significantly enhances data throughput and performance. Here’s a quick overview of the benefits:

  • Increased Performance: Multiple connections can be established per session, reducing congestion and improving data transfer speeds.
  • Flexibility: Customers can configure datastores with multiple IPs to the same server and also multiple connections with the same IP.
  • Scalability: Supports up to 8 connections per IP, enhancing scalability for demanding workloads.

Configuring nConnect

Adding a New NFSv4.1 Datastore

When adding a new NFSv4.1 datastore, you can specify the number of connections at the time of the mount using the following command:

esxcli storage nfs41 add -H <host> -v <volume-label> -s <remote_share> -c <number_of_connections>

By default, the maximum number of connections per session is set to 4. However, this can be increased to 8 using advanced NFS options. Here’s how you can configure it:

  1. Set the maximum number of connections to 8: esxcfg-advcfg -s 8 /NFS41/MaxNConnectConns
  2. Verify the configuration: esxcfg-advcfg -g /NFS41/MaxNConnectConns

The total number of connections used across all mounted NFSv4.1 datastores is limited to 256.

Modifying Connections for an Existing NFSv4.1 Datastore

For an existing NFSv4.1 datastore, the number of connections can be increased or decreased at runtime using the following command:

esxcli storage nfs41 param set -v <volume-label> -c <number_of_connections>

Multipathing and nConnect Coexistence

There is no impact on multipathing when using nConnect. Both NFSv4.1 nConnect and multipaths can coexist seamlessly. Connections are created for each of the multipathing IPs, allowing for enhanced redundancy and performance.

Example Configuration with Multiple IPs

To add a datastore with multiple IPs and specify the number of connections, use:

esxcli storage nfs41 add -H <IP1,IP2> -v <volume-label> -s <remote_share> -c <number_of_connections>

This command ensures that multiple connections are created for each of the specified IPs, leveraging the full potential of nConnect.

Summary

The introduction of nConnect support in vSphere 8.0 U3 for NFS v4.1 datastores marks a significant enhancement in network performance and flexibility. By allowing multiple connections per IP, nConnect addresses the limitations of single connection setups, providing a robust solution for high-performance environments. Whether you’re configuring a new datastore or updating an existing one, nConnect offers a scalable and efficient way to manage your NFS workloads.

https://core.vmware.com/resource/whats-new-vsphere-8-core-storage