Daniel Micanek virtual Blog – Like normal Dan, but virtual.
I am currently working a lot with VMware Aria Operations and the matching Dell Management Packs that are available. One of the most interesting is the integration of PowerEdge Servers, from which you can view various alarms about: This is exactly the information I want to integrate into VMware […]
Why VMware Workstation can’t expose VT-x/EPT or AMD-V/RVI when Windows VBS/Hyper-V is on, and how to disable it to run nested ESXi.
Short answer: The error means Windows is running its own hypervisor, usually because Virtualization-Based Security (VBS) and memory integrity are on, which is the default on many Windows 10/11 installs. VMware Workstation then runs on the Hyper-V APIs and cannot pass VT-x/EPT or AMD-V/RVI to a guest, so nested ESXi will not power on. Turn off memory integrity, remove the Hyper-V platform features, set hypervisorlaunchtype off, disable VBS, reboot, and confirm VBS shows “Not enabled”.
A VM with Virtualize Intel VT-x/EPT or AMD-V/RVI ticked asks Workstation to expose hardware virtualization to the guest. That is required for nested ESXi or Hyper-V inside the VM. Workstation can only do this when it owns the CPU virtualization extensions itself. When Windows boots the Windows hypervisor (for VBS, memory integrity/HVCI, Credential Guard, Hyper-V, WSL2, Windows Sandbox and similar features), Workstation falls back to user-mode operation on top of Hyper-V. In that mode nested virtualization is not available. Broadcom KB 389469 describes the cause and fix.
Typical messages:
VirtualizationBasedSecurityStatus 0 = not enabled, 1 = enabled but not running, 2 = running.Monitor Mode. CPL0 means Workstation owns VT-x. ULM means it is running on the Hyper-V APIs.Get-CimInstance -ClassName Win32_DeviceGuard -Namespace root\Microsoft\Windows\DeviceGuard |
Select-Object VirtualizationBasedSecurityStatus, SecurityServicesRunningBefore you start, save your BitLocker recovery key. Changes to boot and security configuration can trigger a recovery prompt.
bcdedit /set hypervisorlaunchtype off
reg add "HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard" /v EnableVirtualizationBasedSecurity /t REG_DWORD /d 0 /f
reg add "HKLM\SYSTEM\CurrentControlSet\Control\DeviceGuard\Scenarios\HypervisorEnforcedCodeIntegrity" /v Enabled /t REG_DWORD /d 0 /fIf Workstation (or a nested ESXi) itself runs inside a VM on ESXi, the outer VM needs hardware virtualization exposed. Edit the VM, expand CPU and tick Expose hardware assisted virtualization to the guest OS. Nested ESXi is unsupported by Broadcom and belongs in labs only. For lab sizing ideas, see deploying a minimal VCF 9.0 lab.
Disabling VBS and memory integrity lowers the host’s protection against kernel-level malware. When you no longer need nested labs, run bcdedit /set hypervisorlaunchtype auto, turn Memory integrity back on in Windows Security, re-add the Windows features you removed, and reboot.
With the introduction of Log Assist in Version 4.0.7 last week and the killing of the Skyline Advisor, the Skyline Health Diagnostic Appliance has come back into focus. In this article, I would like to show you how it works, what settings can be made and what functions the appliance offers. […]
In this episode of the Virtually Speaking Podcast, we delve into the world of AI with Justin Murray, Product Marketing Engineer, and Frank Denneman, Chief Technologist for AI at Broadcom. We discuss retrieval augmented generation (RAG), a powerful approach that combines large language models […]
ESXi upgrade precheck fails with VFAT_CORRUPTION (KB 91136)? Find the bootbank partitions and repair them with dosfsck, following Broadcom’s procedure.
Short answer: The precheck error “A problem with one or more vFAT bootbank partitions was detected. Please refer to KB 91136 and run dosfsck on bootbank partitions.” means one of the ESXi host’s FAT bootbank partitions is dirty or corrupt. Put the host in maintenance mode and find the vFAT partitions with esxcli storage filesystem list and vmkfstools -P. Check each one with dosfsck -Vv, repair with dosfsck -a -w, re-check, then reboot and retry the upgrade.
vLCM prints this when the hardware precheck of an image fails during an ESXi upgrade, for example 7.0 to 8.0:
Hardware precheck of the profile <ProfileName> failed with errors:
<VFAT_CORRUPTION ERROR: A problem with one or more vFAT bootbank partitions was detected.
Please refer to KB 91136 and run dosfsck on bootbank partitions.KB 91136 is the legacy number. The current article is Broadcom KB 345227. On the host, /var/run/log/lifecycle.log shows that the precheck runs /bin/dosfsck -V -n read-only against each bootbank partition. On older upgrade paths the same corruption can cause worse failures: a purple diagnostic screen with An error occurred while backing up VFAT partition files before re-partitioning: Failed to calculate size for temporary Ramdisk (or Failed to copy files to Ramdisk), a host that falls back to 6.5/6.7, or missing /bootbank links after the upgrade.
ESXi 7.x and later have two vFAT bootbank partitions. ESXi 6.5/6.7 had four or five: two bootbanks plus scratch and locker. Broadcom explains that the FAT “dirty bit” is never set by ESXi itself. It means the partition was mounted somewhere without being unmounted cleanly, which is common on hosts that have been through many upgrades since 6.x. The KB states that the issue is permanently fixed in ESXi 8.0 Update 3b.
# esxcli storage filesystem list
# vmkfstools -P /vmfs/volumes/<vfat-volume-UUID>
...
Partitions spanned (on "disks"):
mpx.vmhba0:C0:T0:L0:5# dosfsck -Vv "/dev/disks/<disk and partition id>"# dosfsck -a -w "/dev/disks/<disk and partition id>"dosfsck -Vv again until it is clean, then reboot the host and re-run the vLCM remediation.Dirty bit prompt: if the interactive check asks 0x25: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt. 1) Remove dirty bit 2) No action, the KB says to choose No action and then repair with dosfsck -a -w. On NVMe boot devices (t10.NVMe____…), dosfsck may print CP850: Invalid argument and report orphaned long file names. In that case choose Delete and then keep the changes.
Solution 2 in the KB repairs the partition while it is not mounted. Mount the matching ESXi ISO through iLO, iDRAC or CIMC and reset the host. At the “Loading VMware ESXi” screen press Shift+O, replace the boot options with cdromBoot, and once the ISO environment is up, run:
# dosfsck -v -a /dev/disks/<disk and partition id>Then unmount the ISO and reboot. If the partition still cannot be repaired, follow the KB section on re-creating it. For other host-level commands, see the esxcli commands for ESXi 8.0 reference.
VMware Cloud Foundation offers a wide array of features and capabilities to help organizations be and stay secure. In the short time we have we’ll talk about recent improvements aimed at making hard security tasks easy or non-existent (ESXi Live Patch, Image-Based Lifecycle Management, audit & […]
VMware Cloud Foundation (VCF) Homelab Hardware Options
I have been getting a ton of inquiries lately, both internally as well as externally for hardware recommendations for building a new or updating an existing homelab/test environment to be able to deploy the full VMware Cloud Foundation (VCF) solution. I typically receive at least half a dozen inquiries per week about general VMware Homelabs, […]
