vSphere 8U1 – Deep dive on configuring NVMe-oF (Non-Volatile Memory Express over Fabrics) for VMware vSphere datastores.
What’s new
With vSphere 8.0 update 1, VMware has completed their journey to a completely native end-to-end NVMe storage stack. Prior to 8.0U1, there was a SCSI translation layer which added some complexity to the stack and slightly decreased some of the efficiencies inherent in the NVMe protocol.
/usr/lib/vmware/secureboot/bin/secureBoot.py -h
usage: secureBoot.py [-h] [-a | -c | -s]
optional arguments:
-h, --help show this help message and exit
-a, --acceptance-level-check
Validate acceptance levels for installed vibs
-c, --check-capability
Check if the host is ready to enable secure boot
-s, --check-status Check if UEFI secure boot is enabled
Check if the host is ready to enable secure boot
/usr/lib/vmware/secureboot/bin/secureBoot.py -c
Secure boot can be enabled: All vib signatures verified. All tardisks validated. All acceptance levels validated
Hardware machine is configured to boot in legacy BIOS mode.
Booting stops early in the boot process with messages displayed in red on black with wording similar to “Error 10 (Out of resources) while loading module”, “Requested malloc size failed”, or “No free memory”.
“Error 10 (Out of resources) while loading module”, “Requested malloc size failed”, or “No free memory”
VMware’s recommended workaround is to transition the machine to UEFI boot mode permanently, as discussed in KB article 84233 . There will not be a future ESXi change to allow legacy BIOS to work on this machine again.
VMware’s plans to deprecate support for legacy BIOS in server platforms.
If you upgrade a server that was certified and running successfully with legacy BIOS to a newer release of ESXi, it is possible the server will no longer function with that release. For example, some servers may fail to boot with an “Out of resources” message because the newer ESXi release is too large to boot in legacy BIOS mode. Generally, VMware will not provide any fix or workaround for such issues besides either switching the server to UEFI
Motivation
UEFI provides several advantages over legacy BIOS and aligns with VMware goals for being “secure by default”. UEFI
UEFI Secure Boot, a security standard that helps ensure that the server boots using only software that is trusted by the server manufacturer.
Automatic update of the system boot order during ESXi installation.
After installing Windows Server 2022 update KB5022842 (OS Build 20348.1547), guest OS can not boot up when virtual machine(s) configured with secure boot enabled running on vSphere ESXi 6.7 U2/U3 or vSphere ESXi 7.0.x.
In VM vmware.log, there is ‘Image DENIED’ info like the below:
2023-02-15T05:34:31.379Z In(05) vcpu-0 - SECUREBOOT: Signature: 0 in db, 0 in dbx, 1 unrecognized, 0 unsupported alg.
2023-02-15T05:34:31.379Z In(05) vcpu-0 - Hash: 0 in db, 0 in dbx.
2023-02-15T05:34:31.379Z In(05) vcpu-0 - SECUREBOOT: Image DENIED.
To identify the location of vmware.log files:
Establish an SSH session to your host. For ESXi hosts
Log in to the ESXi Host CLI using root account.
To list the locations of the configuration files for the virtual machines registered on the host, run the below command:
#vim-cmd vmsvc/getallvms | grep -i "VM_Name"
The vmware.log file is located in virtual machine folder along with the vmx file.
Record the location of the .vmx configuration file for the virtual machine you are troubleshooting. For example:
Currently there is no resolution for virtual machines running on vSphere ESXi 6.7 U2/U3 and vSphere ESXi 7.0.x. However the issue doesn’t exist with virtual machines running on vSphere ESXi 8.0.x.
How to fix network after adding to vDS. When you add NX6412 to vDS and reboot ESXi. I don’t have uplink for vDS. You could check it with:
# esxcfg-vswitch -l
DVS Name Num Ports Used Ports Configured Ports MTU Uplinks
vDS 2560 6 512 9000 vusb0
--cut
DVPort ID In Use Client
468 0
469 0
470 0
471 0
We will have to note DVPort ID 468 – example. vDS is name of your vDS switch.
esxcfg-vswitch -P vusb0 -V 468 vDS
It is necessary add it to /etc/rc.local.d/local.sh before exit 0. You could have similar script from source Persisting USB NIC Bindings
vusb0_status=$(esxcli network nic get -n vusb0 | grep 'Link Status' | awk '{print $NF}')
count=0
while [[ $count -lt 20 && "${vusb0_status}" != "Up" ]]
do
sleep 10
count=$(( $count + 1 ))
vusb0_status=$(esxcli network nic get -n vusb0 | grep 'Link Status' | awk '{print $NF}')
done
esxcfg-vswitch -R
esxcfg-vswitch -P vusb0 -V 468 vDS
exit 0
TPM_VERSION WARNING: Support for TPM version 1.2 is discontinued. With Apply –no-hardware-warning option to ignore the warnings and proceed with the transaction.
esxcli software profile update -d /vmfs/volumes/datastore1/_ISO/ESXi-8.0.1-20842819-USBNIC.zip -p ESXi-8.0.1-20842819-USBNIC --no-hardware-warning
Update Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
During VMware Explore 2022 Barcelona, I’ve been given a gift as a vExpert. You could read it in my previous article. NX6412 doesn’t support onboard NICs. We will need Custom ISO with USB Network Native Driver for ESXi. Because of problem using latest PowerCLI 13 release Nov 25, 2022 with export ISO. I decided to install Custom ISO ESXi 7u2e and than upgrade to ESXi 8.0 with depot zip.
Thank You Cohesity. Power consumpion is only 10 Watts …
How to prepare ESXi Custom ISO image 7U2e for NX6412 NUC?
Currently there is a limitation in ESXi where USB NIC bindings are picked up much later in the boot process and to ensure settings are preserved upon a reboot, the following needs to be added to /etc/rc.local.d/local.sh based on your configurations.
vusb0_status=$(esxcli network nic get -n vusb0 | grep 'Link Status' | awk '{print $NF}')
count=0
while [[ $count -lt 20 && "${vusb0_status}" != "Up" ]]
do
sleep 10
count=$(( $count + 1 ))
vusb0_status=$(esxcli network nic get -n vusb0 | grep 'Link Status' | awk '{print $NF}')
done
esxcfg-vswitch -R
esxcli software profile update -d /vmfs/volumes/datastore1/_ISO/ESXi-8.0.0-20513097-USBNIC.zip -p ESXi-8.0.0-20513097-USBNIC
Hardware precheck of profile ESXi-8.0.0-20513097-USBNIC failed with warnings: <TPM_VERSION WARNING: TPM 1.2 device detected. Support for TPM version 1.2 is discontinued. Installation may proceed, but may cause the system to behave unexpectedly.>
You could fix TPM_VERSION WARNING: Support for TPM version 1.2 is discontinued. With Apply –no-hardware-warning option to ignore the warnings and proceed with the transaction.
