Bug VCF 9.0.2.0 with Holodeck 9.0.2

While deploying VMware Cloud Foundation 9.0.2.0 with Holodeck 9.0.2, I hit an interesting issue during the bundle download phase.

The deployment did not fail with a clear error. Instead, it stalled indefinitely at:

Install-VcfInstallerBundles

At first glance, everything looked fine. The VCF Installer depot UI showed all required components as downloaded successfully. However, the Holodeck deployment kept waiting forever.

The root cause turned out to be a hardcoded bundle count check inside the Holodeck PowerShell module.

Environment

The issue was observed with the following setup:

Holodeck:        9.0.2
HoloRouter OVA: 9.0.2.0424
VCF Installer:  9.0.2.0
Target VCF:     9.0.2.0
Deployment:     Full VCF, ManagementOnly
Depot:          Online depot

Important detail: this was full VCF, not VVF.

Symptom

During deployment, the log repeatedly showed:

SddcMgmtDomain[<pid>]: [INFO] Received Bundles. Checking if all VCF 9 bundles are available
SddcMgmtDomain[<pid>]: [INFO] Didn't receive all bundles. Received 8 bundle details. Trying again after 10 seconds

This message repeated every 10 seconds.

At the same time, the VCF Installer UI showed all visible components as successfully downloaded:

SDDC Manager 9.0.2.0
VMware Cloud Foundation Automation 9.0.2.0
VMware Cloud Foundation Operations 9.0.2.0
VMware Cloud Foundation Operations Collector 9.0.2.0
VMware Cloud Foundation Operations fleet management 9.0.2.0
VMware NSX 9.0.2.0
VMware vCenter 9.0.2.0

So from the UI perspective, everything looked complete. However, Holodeck was still waiting.

Root Cause

The problematic logic is inside the Holodeck PowerShell module on the deployed HoloRouter:

/root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1

The affected function is:

Install-VcfInstallerBundles

Holodeck queries the VCF Installer API:

https://${HostName}/v1/bundles/download-status?imageType=INSTALL

Then it filters bundles matching the selected VCF version:

}elseif($Version -eq "9.0.2.0"){
    $vcf9_bundle_details = $bundle_details.elements | Where-Object {$_.version -match "9\.0\.2\.0\.*"}
}

The problem is the final count check:

elseif($vcf9_bundle_details.count -eq 7){
    Write-Log -Message "Received all Bundle Details"
    $bundle_api_response = $true
}
else{
    Write-Log -Message "Didn't receive all bundles. Received $($vcf9_bundle_details.count) bundle details. Trying again after 10 seconds"
    Start-Sleep -Seconds 10
}

For VCF 9.0.2.0, the API returns 8 matching bundle entries, not 7.

That means this condition never becomes true:

$vcf9_bundle_details.count -eq 7

Holodeck receives 8 bundles, but waits for exactly 7.

Result: an infinite loop.

Why Does the API Return 8 Bundles?

In VCF 9.0.2.0, the bundle structure changed compared to earlier VCF 9 versions.

The depot UI shows 7 visible rows, but the API response contains 8 entries matching:

9.0.2.0.*

The 9.0.2 BOM appears to split some components more granularly, for example around VCF Operations, Operations Collector, and Fleet Management. The UI abstracts this nicely, but the API exposes one additional bundle-level entry.

The important part is this:

VCF Installer UI: 7 visible downloaded components
VCF Installer API: 8 matching bundle entries
Holodeck logic: expects exactly 7

That mismatch is enough to block the deployment.

Workaround

Edit the Holodeck module directly on the HoloRouter.

Change this:

}elseif($vcf9_bundle_details.count -eq 7){

To this:

}elseif($vcf9_bundle_details.count -ge 7){

One-liner:

sed -i 's/$vcf9_bundle_details.count -eq 7/$vcf9_bundle_details.count -ge 7/' \
  /root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1

This changes the check from “exactly 7 bundles” to “at least 7 bundles”.

Restart the Running PowerShell Session

The currently running pwsh process already has the old function loaded in memory.

After patching the file, kill the running PowerShell process:

pkill -9 -f pwsh

Then start a fresh PowerShell session and resume the deployment:

Import-HoloDeckConfig -ConfigID <id>

New-HoloDeckInstance -Version 9.0.2.0 -InstanceID <same-as-before> <original flags>

For example:

New-HoloDeckInstance -Version 9.0.2.0 -InstanceID 1 -ManagementOnly

Use the same flags you used in the original deployment.

What Happens After the Patch?

After applying the workaround, Holodeck resumes at the existing deployment state.

In my case, the state engine resumed at:

Install-VcfInstallerBundles

The patched function immediately accepted the 8 returned bundle entries and logged:

Received all Bundle Details

The deployment then moved forward.

Some bundle download calls may return:

BUNDLE_DOWNLOAD_ALREADY_DOWNLOADED

That is expected because the bundles are already present in the depot. The existing try/catch handling allows the phase to complete quickly.

After that, the deployment advanced to the management-domain phase.

Suggested Proper Fix

The quick fix is:

- }elseif($vcf9_bundle_details.count -eq 7){
+ }elseif($vcf9_bundle_details.count -ge 7){

A version-specific fix would also work, for example:

VCF 9.0.0.0 / 9.0.1.0 -> expect 7
VCF 9.0.2.0          -> expect 8

However, that would likely reintroduce the same type of bug in a future VCF BOM revision.

A better long-term approach would be to avoid a hardcoded count completely and validate the actual bundle download state instead.

For example, Holodeck should check that all required bundles for the selected deployment type and version are present and successfully downloaded, instead of assuming that the bundle count is always static.

Still, as an immediate workaround, changing -eq 7 to -ge 7 is enough to unblock the deployment.

Important Notes

This is an unofficial workaround.

The affected PowerShell module is not part of the public Holodeck documentation repository. It is bundled inside the HoloRouter OVA distributed through the Broadcom Support Portal.

Before editing vendor-supplied files, it is always a good idea to make a backup:

cp /root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1 \
   /root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1.bak

Then apply the patch.

Summary

This is a good example of a small hardcoded assumption causing a deployment to stall without an obvious fatal error.

The VCF Installer API was returning valid data. The depot was populated. The UI showed the bundles as successfully downloaded. But Holodeck was waiting for an exact number of bundle entries that no longer matched the VCF 9.0.2.0 BOM.

For affected Holodeck 9.0.2 users deploying VCF 9.0.2.0, the key symptom is:

Didn't receive all bundles. Received 8 bundle details. Trying again after 10 seconds

If you see this message, check the bundle count logic in:

SddcMgmtDeployment.psm1

The workaround is small, but it can save a lot of troubleshooting time.

Quick Reference

cp /root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1 \
   /root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1.bak

sed -i 's/$vcf9_bundle_details.count -eq 7/$vcf9_bundle_details.count -ge 7/' \
  /root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1

pkill -9 -f pwsh

Then resume:

Import-HoloDeckConfig -ConfigID <id>
New-HoloDeckInstance -Version 9.0.2.0 -InstanceID <same-as-before> <original flags>

Tag: holodeck

Holodeck 9.0.2 with VCF 9.0.2.0 Stuck at Install-VcfInstallerBundles