Cisco Custom ISO MISSING_DEPENDENCY_VIBS ERROR during upgrade ESXi 6.7 -> 7.0

I found a problem and workaround to fix Cisco Custom ISO MISSING_DEPENDENCY_VIBS ERROR during upgrade 6.7 -> 7.0.

It was during these type of upgrade from VMware_ESXi_6.7.0_13006603_Custom_Cisco_6.7.2.1.iso to VMware_ESXi_7.0.0_15843807_Custom_Cisco_4.1.1a.iso

Workaround is to remove VIBs with dependency collision:

# esxcli software vib list | grep QLC
qcnic                          1.0.22.0-1OEM.670.0.0.8169922         QLC 
qedentv                        3.9.31.0-1OEM.670.0.0.8169922         QLC 
qedrntv                        3.9.31.1-1OEM.670.0.0.8169922         QLC 
qfle3                          1.0.77.2-1OEM.670.0.0.8169922         QLC 
qfle3f                         1.0.63.0-1OEM.670.0.0.8169922         QLC 
qfle3i                         1.0.20.0-1OEM.670.0.0.8169922         QLC 
scsi-qedil                     1.2.13.0-1OEM.600.0.0.2494585         QLC 

# esxcli software vib remove -f -n scsi-qedil
# esxcli software vib remove -f -n qfle3f
# reboot

allowLegacyCPU=true: Fix “The CPU in this host is not supported by ESXi” (7.0, 8.0 and 9.x Installer)

CPU_SUPPORT ERROR in the ESXi installer? Press Shift+O and add allowLegacyCPU=true to turn it into a warning. Versions, limits (XSAVE), lab only.

Short answer: When the ESXi installer stops with CPU_SUPPORT ERROR: The CPU in this host is not supported by ESXi, reboot the installer and press Shift+O at the boot prompt. Append allowLegacyCPU=true and press Enter. The error becomes a CPU_SUPPORT WARNING that you can accept. This works on 7.0 and 8.0, and lab users report it on 9.x. It is not supported by Broadcom, so use it only in a homelab.

The error: “The CPU in this host is not supported by ESXi”

I first hit this on an older homelab server with the ESXi 7.0.0 installer (VMware-VMvisor-Installer-7.0.0-15843807.x86_64.iso):

CPU_SUPPORT ERROR:
The CPU in this host is not supported by ESXi
7.0.0. Please refer to the VMware Compatibility Guide (VCG) for
the list of supported CPUs.
ESXi 7.0.0 installer showing CPU_SUPPORT ERROR: the CPU in this host is not supported, with only F11 Reboot available
Without the boot option, the only way forward is F11 Reboot.

The installer offers only Back or Reboot. Newer installers add a pointer to KB 82794, which is now Broadcom KB 318697. For what that KB says and which CPUs it lists, see KB 82794 explained.

Why the installer blocks the CPU

Broadcom KB 318697 defines two stages. A deprecated CPU series gets a warning and the installation continues. A discontinued series is blocked with an error. The check is part of the installer precheck. On the 7.0.0 ISO it lives in \UPGRADE\PRECHECK.PY, and when I looked at that file (around line 1720) it also contained the allowLegacyCPU override. If you only see a warning, your CPU is deprecated, not discontinued, and you need no override.

How to use allowLegacyCPU=true (Shift+O)

  1. Boot the ESXi installer ISO or USB.
  2. When the installer boot screen shows Shift+O: Edit boot options, press Shift+O. The prompt is only shown for a few seconds. Shift+O is the documented way to edit installer boot options.
  3. Leave the existing text (for example runweasel cdromBoot) in place and append a space plus the option.
  4. Press Enter. The system scan now reports a warning instead of an error. Press Enter to continue.
allowLegacyCPU=true
ESXi installer boot screen with the Shift+O edit boot options prompt
Shift+O opens the boot options line.
allowLegacyCPU boot option appended to the ESXi installer boot options line
Append allowLegacyCPU=true to the existing boot options.
ESXi 7.0.0 installer showing CPU_SUPPORT WARNING instead of an error after using allowLegacyCPU
Now it is only a warning, and Enter continues the installation.
CPU_SUPPORT WARNING:
The CPU in this host is not supported by ESXi
7.0.0. Please refer to the VMware Compatibility Guide (VCG) for
the list of supported CPUs.

Using allowLegacyCPU=true in boot.cfg and kickstart

Shift+O applies only to the current boot. To avoid typing it, put it on the kernelopt= line of boot.cfg on your installer media or PXE files. TechDocs documents that kernelopt= appends its string to the kernel boot options. William Lam also noted (for 7.0) that the option is not persistent after installation. For an installed host that needs it on every boot, he suggests adding it to kernelopt in both /bootbank/boot.cfg and /altbootbank/boot.cfg.

For unattended installs, the warning prompt still waits for Enter, so you also need a kickstart flag. I cover that separately in Automating an ESXi 8.0 install with allowLegacyCPU=true (kickstart).

Which ESXi versions honour allowLegacyCPU=true

VersionBehaviourBasis
ESXi 7.0Error becomes a warning; install continuesMy own test on 7.0.0 (build 15843807); William Lam
ESXi 8.0 GA to 8.0 U1Error becomes a warningWilliam Lam
ESXi 8.0 U2 and laterStill honoured, but the CPU must support XSAVE (Intel Sandy Bridge / AMD Bulldozer or newer). Older CPUs such as Westmere stop with VMB: 716: Unsupported CPU.William Lam
ESX 9.0 / 9.1Lab users report the same error-to-warning behaviourCommunity reports only; not documented by Broadcom

allowLegacyCPU is not listed in Broadcom’s installation documentation or in KB 313264 (additional kernel boot options). It is an engineering override that has never been guaranteed and may disappear in any build.

What allowLegacyCPU=true does not give you

  • No support. The host runs on a CPU that is not in the Broadcom Compatibility Guide for that release.
  • No guarantee for updates. Any patch or update can add a new hard requirement, as 8.0 U2 did with XSAVE. Keep the previous boot bank so you can roll back.
  • Not the fix for a warning. For a deprecated CPU, patching with esxcli software profile update only needs --no-hardware-warning (Broadcom KB 391610). That host is still supported.
  • Not a production path. For production hosts with discontinued CPUs, KB 428874 describes RPQ/TQ requests for “VCF Software Only” support through your account team. Broadcom has also published a supported override procedure for Intel Skylake-SP on VCF 9.x. See Intel Skylake and ESX 9 support.

Disclaimer: this is not officially supported by VMware by Broadcom. You run it at your own risk.

Sources

Further reading (William Lam, who first published the tip): Allow unsupported CPUs when upgrading to ESXi 7.0, Automating ESXi 8.0 install using allowLegacyCPU=true, ESXi 8.0 Update 2 requires XSAVE even with allowLegacyCPU=true.

Related on vdan.cz

Convert vLCM Baselines to an Image: vSphere Lifecycle Manager Cluster Transition (vSphere 8 and 9)

Baselines are deprecated in vSphere 8 and can’t manage ESX 9. How to convert a cluster from vLCM baselines to an image, prerequisites and blockers.

Short answer: In the vSphere Client, select the cluster and open Updates > Image. vCenter runs an eligibility check first. Then you either use the image from a host in the cluster (vCenter 8.0 U3+), import a JSON image, or set one up manually. Finish with Finish image setup and remediate. The switch can’t be reversed. Baselines were deprecated in vSphere 8.0, and in vCenter 9.0 they can only patch ESXi 8.x hosts. You must convert before you move to ESX 9.

vLCM baselines status: deprecated in 8.0, removed for ESX 9

I wrote the first version of this post in 2020, right after upgrading my lab to ESXi 7.0. Back then images were new and optional. Here is what the Broadcom documentation says today:

ReleaseBaselines (VUM workflows)Source
vSphere 7.0Supported. vLCM images introduced as the declarative alternative.KB 322186
vSphere 8.0Deprecated. Still supported in vSphere 8, with removal planned for the next major release.vSphere 8.0 release notes, KB 322186
vCenter 9.0Removed as a way to manage clusters. Baselines only patch ESXi 8.x hosts. ESX 9 hosts can’t be patched or upgraded with them, ISO import and ISO-based upgrades are blocked, and new clusters and standalone hosts are image-managed by default.VCF 9.0 release notes, KB 379388

So an existing baseline-managed cluster with 8.x hosts keeps working under vCenter 9.0, but it can’t reach ESX 9. According to KB 379388, the way around those restrictions is to move from VUM to vLCM images.

Prerequisites and what blocks the conversion

When you start the switch, vCenter runs the Check cluster’s eligibility to be managed with a single image task. Its results come in three types.

Errors (block the conversion)

  • A host is not stateful, for example an Auto Deploy stateless host. It must boot from a disk.
  • A host runs a version earlier than ESXi 7.0. Upgrade it with an upgrade baseline first.
  • VIBs from an unintegrated solution are present. Deactivate the solution and retry.
  • The cluster is currently being remediated against a baseline.

Warnings (need action)

  • A standalone VIB for which a component exists in the depot. Add that component to the image, or the VIB is deleted during remediation.
  • An unknown VIB, with no component in the depot. Import a component that contains it, or it is deleted during remediation.

Other prerequisites

  • All hosts from the same hardware vendor.
  • The vLCM image privileges.
  • For third-party solutions, confirmation from the vendor that they work with vLCM.
  • For the “use an image from a host” option: hosts on 7.0 U2 or later and vCenter on 8.0 U3 or later.
  • For VCF environments: do this only through SDDC Manager, on VCF 5.2.2 or later or VCF 9 (KB 385617). Don’t convert VCF clusters directly in vCenter.

The depot also needs the content you plan to use in the image. On current releases, vLCM no longer downloads from the old public VMware URLs; each customer gets a unique download URL. See KB 390098 if the Software Depot doesn’t show recent releases. Offline, you can import a depot ZIP through Actions > Import Updates. These are my original vSphere 7.0 screenshots:

vSphere Lifecycle Manager Actions menu with Import Updates for adding an offline depot
Lifecycle Manager: Actions > Import Updates (vSphere 7.0 UI)
Importing the VMware-ESXi-7.0.0-15843807-depot.zip offline bundle into the vLCM depot
Importing VMware-ESXi-7.0.0-15843807-depot.zip
Lifecycle Manager Image Depot listing ESXi 7.0 GA build 15843807 after importing the offline bundle
Image Depot: ESXi 7.0 GA – 15843807 available after the import

How to convert vLCM baselines to an image (step by step)

These steps follow the TechDocs procedure for Option 3: Set Up a New Image Manually. Options 1 and 2 start the same way.

  1. Open the image setup. In the vSphere Client, select the baseline-managed cluster and go to Updates > Image.
  2. Choose how to build the image. Pick one of the three options. Set up image manually builds it by hand. Import image takes a JSON file. On vCenter 8.0 U3+, you can also choose one of the “Currently running images in the cluster” and click Proceed with selected image. The eligibility check runs; fix any errors and start again.
  3. Define the image. In the Convert to an Image pane, select the ESXi Version. Optionally add a Vendor Addon, a Firmware and Drivers Addon (this needs a hardware support manager) and extra Components.
  4. Validate and save. Click Validate to catch missing dependencies and conflicts, then Save. Saving triggers a compliance check of all hosts against the image.
  5. Finish the setup. Click Finish image setup and confirm with Yes, finish image setup. From now on the cluster is managed by a single image.
  6. Remediate. Run the pre-check and then Remediate All. Switching the management method alone does not change the hosts; only remediation applies the image.
Cluster Updates tab in vSphere Client with the Setup Image button for converting from baselines to a vLCM image
Cluster > Updates: start with SETUP IMAGE (vSphere 7.0 UI)
Defining a vLCM image: selecting the ESXi version and adding a component (USB NIC Fling driver)
Select the ESXi version and add components (example: USB NIC Fling driver)
vLCM Step 2 Check Image Compliance: 3 of 3 hosts out of compliance with the cluster image, host will be rebooted during remediation
Step 2: image compliance check (3 of 3 hosts out of compliance before remediation)
Finish image setup confirmation dialog in vSphere Lifecycle Manager
Finish image setup: confirm with YES
Cluster Updates tab after conversion: the Baselines menu is gone and the cluster is image-managed
After the conversion the Baselines menu disappears
Remediate All dialog for applying the vLCM image to all hosts in the cluster
REMEDIATE ALL: start dialog
vLCM remediation installing the added component on the cluster hosts
START REMEDIATION installs the example USB NIC Fling driver

What changes after the switch

  • There is no way back. A cluster can’t return to baselines. You can only move hosts into another cluster that still uses baselines.
  • Remediation removes what the image doesn’t include. Standalone VIBs and agents of non-integrated solutions are deleted. Put every driver and tool you need into the image first.
  • Standalone hosts keep the image only under certain conditions. A host removed from an image-managed cluster keeps the image only when vCenter and ESXi are 8.0 U3+ and NSX, if present, is 4.2 or later. Otherwise it reverts to baselines.
  • Unsupported configurations are still possible. If you enable a solution that doesn’t work with vLCM on an empty cluster (Dell EMC VxRail is the documented example), the transition succeeds but the configuration is unsupported.

Sources

Related on vdan.cz

vSphere 6 Update Manager -> vSphere 7 Lifecycle Manager : upgrade ESXi 6.7 to ESXi 7

Move from vSphere 6 Update Manager to vSphere 7 Lifecycle Manager and upgrade ESXi 6.7 hosts to ESXi 7.0 step by step.

vSphere Update Manager – vSphere 6

In vSphere 6 we can use various methods and tools to deploy ESXi hosts and maintain their software lifecycle.

To deploy and boot an ESXi host, you can use an ESXi installer image or VMware vSphere® Auto Deploy™. The availability of choice options results in two different underlying ESXi platforms:

  • Using vSphere Auto Deploy – stateless mode
  • Using an installer ESXi image – statefull mode

vSphere Lifecycle Manager Images: A Single Platform, Single Tool, Single Workflow

By introducing the concept of images, vSphere Lifecycle Manager provides a unified platform for ESXi lifecycle management.
You can use vSphere Lifecycle Manager for stateful hosts only, but starting with vSphere 7.0, you can convert the Auto Deploy-based stateless hosts into stateful hosts, which you can add to clusters that you manage with vSphere Lifecycle Manager images.

How to Upgrade ESXi 6.7 to 7 with vSphere Lifecycle Manager?

After upgrade VCSA 7.0, We prepare upgrade for ESXi 6.7. It is simular logic like in vSphere Update Manager:

IMPORT ISO – We can upload ISO for example VMware-VMvisor-Installer-7.0.0-15843807.x86_64.iso
Step 1 of 2 – Uploading file to server
Step 2 of 2 – Adding to repository
After ISO upload We could check ISO image context
Next step is Create Baseline
Create Baseline – for ISO image
Select uploaded ISO image
Check summary
On Targer Cluster We attach our Baseline
Select ESXi 7 Baseline
– Check Compliance
– We can see Non-compliant for 3x ESXi host
REMEDIATE will start upgrade dialoge
It is necessary accept EULA
REMEDIATE will start ESXi 7 upgrade
In Recent Task We caould chek progress.
We can check ESXi 7.0 upgrade result.