“A problem with one or more vFAT bootbank partitions was detected” (KB 91136): Fix with dosfsck

Short answer: The precheck error “A problem with one or more vFAT bootbank partitions was detected. Please refer to KB 91136 and run dosfsck on bootbank partitions.” means one of the ESXi host’s FAT bootbank partitions is dirty or corrupt. Put the host in maintenance mode and find the vFAT partitions with esxcli storage filesystem list and vmkfstools -P. Check each one with dosfsck -Vv, repair with dosfsck -a -w, re-check, then reboot and retry the upgrade.

The exact error

vLCM prints this when the hardware precheck of an image fails during an ESXi upgrade, for example 7.0 to 8.0:

Hardware precheck of the profile <ProfileName> failed with errors:
<VFAT_CORRUPTION ERROR: A problem with one or more vFAT bootbank partitions was detected.
Please refer to KB 91136 and run dosfsck on bootbank partitions.

KB 91136 is the legacy number. The current article is Broadcom KB 345227. On the host, /var/run/log/lifecycle.log shows that the precheck runs /bin/dosfsck -V -n read-only against each bootbank partition. On older upgrade paths the same corruption can cause worse failures: a purple diagnostic screen with An error occurred while backing up VFAT partition files before re-partitioning: Failed to calculate size for temporary Ramdisk (or Failed to copy files to Ramdisk), a host that falls back to 6.5/6.7, or missing /bootbank links after the upgrade.

Why vFAT bootbanks get corrupted

ESXi 7.x and later have two vFAT bootbank partitions. ESXi 6.5/6.7 had four or five: two bootbanks plus scratch and locker. Broadcom explains that the FAT “dirty bit” is never set by ESXi itself. It means the partition was mounted somewhere without being unmounted cleanly, which is common on hosts that have been through many upgrades since 6.x. The KB states that the issue is permanently fixed in ESXi 8.0 Update 3b.

Fix: check and repair with dosfsck (KB 91136 / 345227)

1. Put the host into maintenance mode. Broadcom requires this before either solution.
2. List the vFAT volumes and map each to its disk and partition:

# esxcli storage filesystem list
# vmkfstools -P /vmfs/volumes/<vfat-volume-UUID>
...
Partitions spanned (on "disks"):
        mpx.vmhba0:C0:T0:L0:5

3. Check every vFAT partition. A healthy one ends with a file and cluster count and no errors:

# dosfsck -Vv "/dev/disks/<disk and partition id>"

4. If the check reports errors or hangs, repair it. When it reports orphaned files, delete them and write the changes:

# dosfsck -a -w "/dev/disks/<disk and partition id>"

5. Run dosfsck -Vv again until it is clean, then reboot the host and re-run the vLCM remediation.

Dirty bit prompt: if the interactive check asks 0x25: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt. 1) Remove dirty bit 2) No action, the KB says to choose No action and then repair with dosfsck -a -w. On NVMe boot devices (t10.NVMe____…), dosfsck may print CP850: Invalid argument and report orphaned long file names. In that case choose Delete and then keep the changes.

If dosfsck cannot repair the partition

Solution 2 in the KB repairs the partition while it is not mounted. Mount the matching ESXi ISO through iLO, iDRAC or CIMC and reset the host. At the “Loading VMware ESXi” screen press Shift+O, replace the boot options with cdromBoot, and once the ISO environment is up, run:

# dosfsck -v -a /dev/disks/<disk and partition id>

Then unmount the ISO and reboot. If the partition still cannot be repaired, follow the KB section on re-creating it. For other host-level commands, see the esxcli commands for ESXi 8.0 reference.

Further reading

• IRG-Net: A problem with one or more vFAT bootbank partitions was detected – PSOD alarm (original post)
• Broadcom KB 345227 (legacy KB 91136): “A problem with one or more vFAT bootbank partitions was detected”