Month: October 2024

Workarounds for AVX-512 vSAN 8.0 Data…

In some older releases of vSAN there is an issue where if applications are using AVX-512 CPU instructions data corruption can occur. This was first reported by customers using DB2 but can occur with other applications as well. Here is the KB describing the problem: […]

---

Broadcom Social Media Advocacy

Short answer: Zabbix monitors vSphere agentlessly through the vCenter (or ESXi) SDK. First enable VMware collectors on the Zabbix server with StartVMwareCollectors and restart it. Then create one host linked to the VMware or VMware FQDN template and set {$VMWARE.URL} to https://<vcenter>/sdk, plus the username and password macros. Low-level discovery then creates hosts for hypervisors and VMs automatically.

How Zabbix VMware monitoring works

Zabbix needs no agent on ESXi hosts. Dedicated vmware collector processes in the Zabbix server (or proxy) log in to the vSphere Web Services API, cache configuration and performance counters in shared memory, and serve that cache to vmware.* items. According to the Zabbix VMware monitoring manual, the server must be built with --with-libcurl --with-libxml2. Official Zabbix packages already include both.

Step 1: enable VMware collectors on the Zabbix server

By default no collectors run. Edit /etc/zabbix/zabbix_server.conf (or the proxy config if a proxy monitors vCenter):

StartVMwareCollectors=2
# optional tuning, see the zabbix_server.conf reference
# VMwareCacheSize=
# VMwareFrequency=
# VMwarePerfFrequency=
# VMwareTimeout=

systemctl restart zabbix-server

Zabbix recommends running more collectors than monitored VMware services, and no more than twice as many. With too few, slow configuration retrieval on large vCenters delays performance data. VMwareCacheSize holds the shared memory for VMware data, so size it to the number of hosts and VMs you monitor. Use Zabbix internal monitoring to see how full the cache is and how busy the collectors are.

Parameter	Meaning
StartVMwareCollectors	Number of pre-forked VMware collector processes
VMwareCacheSize	Shared memory for VMware data
VMwareFrequency	Seconds between configuration data gathering from one VMware service
VMwarePerfFrequency	Seconds between performance counter retrievals
VMwareTimeout	Maximum wait for a response from the VMware service

Step 2: prepare a vCenter service account

The Zabbix guide uses administrator@vsphere.local as an example. In production, use a dedicated SSO user with the Read-only role, propagated from the vCenter root object. Monitoring only reads inventory, events and counters. Zabbix does not document a minimum privilege set, so if some items stay unsupported, check permissions before anything else.

For datastore capacity metrics, the Zabbix docs require the vCenter advanced setting vpxd.stats.maxQueryMetrics to be at least 64.

Step 3: create the vCenter host and macros in Zabbix

1. Data collection > Hosts > Create host, for example VMware environment, host group VMware. No interface is needed.
2. Link the template VMware FQDN or VMware (see below).
3. On the Macros tab set the three macros, using a secret text type for the password.

{$VMWARE.URL}      = https://vcenter.example.local/sdk
{$VMWARE.USERNAME} = svc-zabbix@vsphere.local
{$VMWARE.PASSWORD} = ********

Discovery rules such as vmware.hv.discovery[{$VMWARE.URL}] then create hypervisor hosts (template VMware Hypervisor) and VM hosts (template VMware Guest) from host prototypes. Check the results in Monitoring > Hosts > Latest data.

VMware vs VMware FQDN template

The template documentation explains that the difference is naming. VMware names discovered VM hosts from vCenter data (UUID-based macros, VM names as shown in vCenter). VMware FQDN uses the guest FQDN. That requires VMware Tools and a unique, FQDN-compliant OS name in each VM. It also avoids duplicate hosts when the same VMs also run a Zabbix agent that auto-registers under its hostname. Do not link VMware Guest manually: it is meant only for discovery.

Troubleshooting Zabbix vCenter monitoring

• “No vmware collector processes started”: StartVMwareCollectors is 0 on the server or proxy that actually polls the host.
• Items “Not supported” on datastores: the performance counter is not enabled or collected in vCenter, or maxQueryMetrics is too low.
• Login errors after certificate or password changes: check the URL ends in /sdk and the service account has not expired. See vCenter password expiration fix.
• Deep debugging: raise the log level for vmware collectors only (debug level 5), as described in the Zabbix manual.

Further reading

• Original post: Monitoring VMware vSphere with Zabbix (vMattroman)
• Zabbix manual: Monitor VMware with Zabbix
• Zabbix manual: VMware monitoring
• Zabbix manual: zabbix_server.conf parameters

Short answer: Broadcom cannot recover a lost ESXi root password, only replace it. If the host is still connected to vCenter, reset root with a Host Profile, vSphere Configuration Profiles or PowerCLI (esxcli system account set). If it is joined to Active Directory, an “ESX Admins” member can reset it. A standalone host with no other admin access must be reinstalled with VMFS preserved. Editing the configuration store offline is an unsupported lab workaround.

Supported ways to reset the ESXi root password

Broadcom lists the options in KB 393496 (ESXi 7.0 and 8.0). Choose one based on what still has access to the host:

Method	Requirement	Support status
Host Profile	Host connected to vCenter; vSphere Enterprise Plus license	Supported (KB 323617)
vSphere Configuration Profiles	Clusters managed by configuration profiles instead of Host Profiles	Supported (KB 393496)
PowerCLI / esxcli via vCenter	Host connected to vCenter; on 8.x shell access for vpxuser	Supported (KB 376979)
AD account in “ESX Admins”	Host joined to AD; access to a domain controller	Supported (KB 390021)
Reinstall, preserve VMFS	Standalone host with no other admin path	Supported (KB 393496)
Offline edit of state.tgz / configstore	Physical access, Linux VM, nested ESXi	Unsupported, lab only

Reset root with a Host Profile (host managed by vCenter)

Here is a summary of KB 323617, which also covers ESX 9.x:

1. In Policies and Profiles > Host Profiles, extract a profile from a healthy host of the same version.
2. Edit the profile, clear all sections, search for root and set Fixed password configuration with the new password.
3. Attach the profile to the affected host, edit the host customization, then Remediate. No reboot is required, so you can clear the automatic reboot option.
4. Check compliance, then delete the temporary profile.

Reset the ESXi root password with PowerCLI

If Host Profiles are not licensed, KB 376979 runs esxcli system account set through vCenter. The host must not be Disconnected or Not Responding. On ESXi 8.x, shell access for the vpxuser account must be enabled, and the KB says the method fails if root is lost and that shell access is disabled.

Connect-VIServer vcenter.example.local
$creds = Get-Credential -UserName "root" -Message "Enter root as username and new password"
$esxiserver = Get-VMHost -Name esx01.example.local
$esxcli = Get-EsxCli -VMHost $esxiserver -V2
$userarg = $esxcli.system.account.set.CreateArgs()
$userarg.id = $creds.UserName
$userarg.password = $creds.GetNetworkCredential().Password
$userarg.passwordconfirmation = $creds.GetNetworkCredential().Password
$esxcli.system.account.set.invoke($userarg)   # returns true on success

Reset via Active Directory (“ESX Admins”)

For an AD-joined host, KB 390021 uses the default admin group. Create an AD group named exactly ESX Admins and add a user to it. Sign in to the VMware Host Client with that account, then edit root under Manage > Security & users > Users. Caveat: from ESXi 8.0 Update 3, Config.HostAgent.plugins.hostsvc.esxAdminsGroupAutoAdd defaults to false (KB 369707, the CVE-2024-37085 hardening). On those hosts, this only works if an AD group was already given admin rights on the host.

Standalone host: reinstall ESXi and keep VMFS

With no vCenter, AD or other admin account, the supported path in KB 393496 is a reinstall. Evacuate or power off the VMs. Reinstall the same ESXi version on the boot device and choose to preserve the VMFS datastore. Then restore the configuration if you have a backup, and re-register the VMs from the datastore. A configuration bundle only helps if you took it earlier. See how to back up and restore ESXi host configuration.

Unsupported: editing the ESXi configuration store offline

Older guides replace the password hash in /etc/shadow inside state.tgz from a Linux live CD. William Lam points out that this no longer works from ESXi 7.0 Update 1 onward, because host configuration moved into an encrypted configuration store. His workaround copies the host’s encrypted state from the boot device. It decrypts that state in a nested ESXi VM, edits the password hash in the configstore SQLite database, re-encrypts it and writes it back.

Treat this as lab only. It is not a Broadcom-documented procedure, and it is not in any Broadcom KB. It modifies the boot device directly, and a mistake can leave the host unbootable. On production hosts use one of the supported methods above, or open a support case.

Prevent the next lockout

• Store root passwords in a vault and rotate them on a schedule (VCF password management does this for managed hosts).
• Keep a second local admin, or an AD group, with the Administrator role.
• Back up host configuration after every change. Useful esxcli commands are in esxcli commands for ESXi 8.0.

Further reading

• Original post: Recovering ESXi 7.x & 8.x host after forgetting or losing root password (William Lam)
• Broadcom KB 393496: Methods to recover the ESXi host root password
• Broadcom KB 323617: Reset host root password with Host Profile
• Broadcom KB 376979: Reset root password of ESXi using PowerCLI

ESXi on ASUS NUC 14 Performance (Scorpion Canyon)

In addition to the ASUS NUC 14 Pro (which I recently reviewed), ASUS has also released the ASUS NUC 14 Performance (formally known as Scorpion Canyon) as part of their Intel 14th Generation (Meteor Lake) lineup. Compared to the ASUS NUC 14 Pro and Pro+, the ASUS NUC 14 Performance offers […]

---

Broadcom Social Media Advocacy

Updated Dashboard for VMware Community Homelabs…

While working on some data analysis for an internal project, I was looking for a better way to summarize and provide some visualizations of the raw data for better consumption. I also wanted to automate this process, so that I could easily build reports or dashboards regardless of the […]

---

Broadcom Social Media Advocacy

Quick Tip – Retrieving vSAN File Share Network…

When creating a new vSAN File Share, which is powered by vSAN File Services, additional network access controls (no access, allow access from any IP or custom) can be configured. To view the configured network permissions, users must expand each file share to get the relevant information. For […]

---

Broadcom Social Media Advocacy

Short answer: On the ESXi shell run vim-cmd hostsvc/firmware/sync_config and then vim-cmd hostsvc/firmware/backup_config, and download the configBundle-*.tgz from the URL it prints. With PowerCLI use Get-VMHostFirmware -BackupConfiguration. To restore, the host must run the same ESXi build and, by default, keep the same UUID. Put it in maintenance mode and use restore_config or Set-VMHostFirmware -Restore. The host reboots.

Which ESXi backup do you need?

You want to protect	Use	Covered here
The host’s own settings: standard switches, VMkernel ports, storage adapters, users, services, advanced settings	ESXi configuration bundle (backup_config / Get-VMHostFirmware)	Yes, this page
Virtual machines and their data	A VM backup product that uses the vSphere data protection APIs	No. The config bundle contains no VMs and no VM inventory.
The ESXi image: base image, vendor add-on, drivers and VIBs	The cluster’s vSphere Lifecycle Manager image. Reinstall the same build (bootable ESXi installer USB), then restore the bundle.	Only the “same build” rule
One standard configuration for every host in a cluster	Host Profiles, or vSphere Configuration Profiles on vSphere 8 and later	No. It is complementary to the bundle.
Distributed switch configuration	vCenter (export the VDS configuration from vCenter)	No. Per KB 313510 the restore brings back standard-switch networking. The VDS comes from vCenter.

What the ESXi configuration backup contains

The bundle holds the host’s configuration: networking (vSwitches, VMkernel ports), storage settings, users, services, advanced settings and similar. According to Broadcom KB 313510 it does not contain:

• Virtual machine inventory. After a restore you re-register VMs from the datastores.
• Bootbank contents (the ESXi image and VIBs themselves).

The same KB says the procedure is not supported for Distributed Services Engine (DPU) server configurations or for Auto Deploy-provisioned hosts. VM data and datastores need a real backup product; this covers only the host.

ESX 9: KB 313510 lists no 9.x-specific exceptions, so the same vim-cmd hostsvc/firmware and Get-VMHostFirmware procedure applies. On VCF hosts the restore only brings the host back to its own last state. Reconnecting it to vCenter and the distributed switch is still a vCenter task. Test a restore on one lab host before relying on it in production.

Back up ESXi configuration from the command line (vim-cmd)

Enable SSH or the ESXi Shell, log in as root and first record the two values the restore will depend on:

vmware -vl          # ESXi version and build number
esxcfg-info -u      # host UUID

vim-cmd hostsvc/firmware/sync_config
vim-cmd hostsvc/firmware/backup_config
# Bundle can be downloaded at : http://*/downloads/<id>/configBundle-<host>.tgz

Replace * with the host FQDN or IP and download the file with a browser or wget. The file sits under /scratch/downloads only temporarily, so copy it off the host immediately. Store the build number and UUID next to the bundle.

Back up ESXi configuration with PowerCLI (including via vCenter)

Connect-VIServer vcenter.lab.local      # or connect directly to the host
Get-VMHostFirmware -VMHost esx01.lab.local -BackupConfiguration -DestinationPath C:\Backup\ESXi

# all hosts managed by vCenter
Get-VMHost | Get-VMHostFirmware -BackupConfiguration -DestinationPath C:\Backup\ESXi

PowerCLI downloads the bundle from the host itself, so the machine running it needs network access to each host on TCP 443 and TCP 80, per the KB. A missing port 80 rule is the usual reason this fails when you back up ESXi configuration from vCenter.

Schedule ESXi configuration backups for a whole vCenter

A one-time backup goes stale after the next VMkernel or storage change. This PowerCLI script backs up every connected host through vCenter into a dated folder. It writes the build number and host UUID next to each bundle (the two values a restore depends on) and deletes folders older than the retention period. Run it from Windows Task Scheduler or cron (PowerShell 7) on a management VM that can reach every host on TCP 443 and 80.

# Backup-EsxiConfig.ps1 – requires VMware.PowerCLI (VCF.PowerCLI)
param(
    [string]$VCenter    = 'vcenter.lab.local',
    [string]$Root       = 'D:\Backup\ESXi',
    [int]$RetentionDays = 30
)
Connect-VIServer -Server $VCenter | Out-Null          # use a stored credential, not a password in the script
$target = Join-Path $Root (Get-Date -Format 'yyyy-MM-dd')
New-Item -ItemType Directory -Path $target -Force | Out-Null

foreach ($h in Get-VMHost | Where-Object { $_.ConnectionState -in 'Connected','Maintenance' }) {
    try {
        Get-VMHostFirmware -VMHost $h -BackupConfiguration -DestinationPath $target -ErrorAction Stop | Out-Null
        $esxcli = Get-EsxCli -VMHost $h -V2
        [pscustomobject]@{
            Host    = $h.Name
            Version = $h.Version
            Build   = $h.Build
            Uuid    = $esxcli.system.uuid.get.Invoke()
        } | Export-Csv (Join-Path $target 'hosts.csv') -Append -NoTypeInformation
    } catch {
        Write-Warning "$($h.Name): $($_.Exception.Message)"
    }
}

Get-ChildItem $Root -Directory |
    Where-Object { $_.CreationTime -lt (Get-Date).AddDays(-$RetentionDays) } |
    Remove-Item -Recurse -Force

Disconnect-VIServer -Confirm:$false

• Run it after every change window, not only nightly. A restore needs a bundle from the same build, so take a fresh backup after each ESXi patch.
• Bundles of TPM-protected hosts restore only onto the same TPM. Keep the encryption recovery keys (esxcli system settings encryption recovery list) in your password vault next to the bundles.
• The bundle contains password hashes and host certificates. Store it with the same access controls as other credentials.

Restore rules you cannot skip

Requirement	Why it matters
Same ESXi build as the backup	Install the exact build first (vmware -vl), then restore.
Same host UUID	Default restore refuses a mismatch; restore_config 1 forces it.
Maintenance mode	Required before the restore.
Reboot before restore	The KB lists a reboot as part of the workflow.
Automatic reboot after restore	Plan the outage.

Restore ESXi configuration with vim-cmd

1. Rename the file to configBundle.tgz and copy it to a datastore on the host.
2. Enter maintenance mode: vim-cmd hostsvc/maintenance_mode_enter
3. Reboot the host.
4. Move the bundle to /tmp/configBundle.tgz.
5. Run vim-cmd hostsvc/firmware/restore_config 0 (or 1 to override a UUID mismatch). The host reboots when done.

Restore with PowerCLI Set-VMHostFirmware

Set-VMHost -VMHost esx01.lab.local -State Maintenance
Restart-VMHost -VMHost esx01.lab.local -Confirm:$false
# wait for the host to come back
Set-VMHostFirmware -VMHost esx01.lab.local -Restore -SourcePath C:\Backup\ESXi\configBundle-esx01.lab.local.tgz -HostUser root -HostPassword '<password>'

The old vSphere CLI command vicfg-cfgbackup is still in the KB, but only for vSphere 6.7 and earlier. On 7.x, 8.x and 9.x use vim-cmd or PowerCLI.

TPM and encrypted configuration caveats

Since vSphere 7.0 U2 the ESXi configuration can be sealed to the host’s TPM. Broadcom’s KB states that in that case the force option does not work if the host has been changed. In other words, restore_config 1 will not move a TPM-protected config to replacement hardware. What protects you is the recovery key. Record it on every TPM-equipped host and store it off-host, as described in Manage a Secure ESXi Configuration:

esxcli system settings encryption recovery list

If a recovered host keeps asking for the recovery key at every boot and backup_config fails with “Internal error”, check the boot options. KB 410638 traces this to a missing or malformed bootUUID or recovery-key entry.

For other host-level commands see esxcli commands for ESXi 8.0. If you run the free hypervisor, see Free VMware ESXi 8.

Further reading

• Broadcom KB 313510: How to back up and restore the ESXi host configuration
• Broadcom KB 410638: ESXi settings are not persisted after restoring an encrypted ESXi with its recovery key
• Broadcom TechDocs: Manage a Secure ESXi Configuration
• Mateusz Romaniuk (vMattroman): Backup and restore ESXi host configuration data using command line (original post)