Bug VCF 9.0.2.0 with Holodeck 9.0.2
While deploying VMware Cloud Foundation 9.0.2.0 with Holodeck 9.0.2, I hit an interesting issue during the bundle download phase.
The deployment did not fail with a clear error. Instead, it stalled indefinitely at:
Install-VcfInstallerBundlesAt first glance, everything looked fine. The VCF Installer depot UI showed all required components as downloaded successfully. However, the Holodeck deployment kept waiting forever.
The root cause turned out to be a hardcoded bundle count check inside the Holodeck PowerShell module.
Environment
The issue was observed with the following setup:
Holodeck: 9.0.2
HoloRouter OVA: 9.0.2.0424
VCF Installer: 9.0.2.0
Target VCF: 9.0.2.0
Deployment: Full VCF, ManagementOnly
Depot: Online depotImportant detail: this was full VCF, not VVF.
Symptom
During deployment, the log repeatedly showed:
SddcMgmtDomain[<pid>]: [INFO] Received Bundles. Checking if all VCF 9 bundles are available
SddcMgmtDomain[<pid>]: [INFO] Didn't receive all bundles. Received 8 bundle details. Trying again after 10 secondsThis message repeated every 10 seconds.
At the same time, the VCF Installer UI showed all visible components as successfully downloaded:
SDDC Manager 9.0.2.0
VMware Cloud Foundation Automation 9.0.2.0
VMware Cloud Foundation Operations 9.0.2.0
VMware Cloud Foundation Operations Collector 9.0.2.0
VMware Cloud Foundation Operations fleet management 9.0.2.0
VMware NSX 9.0.2.0
VMware vCenter 9.0.2.0So from the UI perspective, everything looked complete. However, Holodeck was still waiting.
Root Cause
The problematic logic is inside the Holodeck PowerShell module on the deployed HoloRouter:
/root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1The affected function is:
Install-VcfInstallerBundlesHolodeck queries the VCF Installer API:
https://${HostName}/v1/bundles/download-status?imageType=INSTALLThen it filters bundles matching the selected VCF version:
}elseif($Version -eq "9.0.2.0"){
$vcf9_bundle_details = $bundle_details.elements | Where-Object {$_.version -match "9\.0\.2\.0\.*"}
}The problem is the final count check:
elseif($vcf9_bundle_details.count -eq 7){
Write-Log -Message "Received all Bundle Details"
$bundle_api_response = $true
}
else{
Write-Log -Message "Didn't receive all bundles. Received $($vcf9_bundle_details.count) bundle details. Trying again after 10 seconds"
Start-Sleep -Seconds 10
}For VCF 9.0.2.0, the API returns 8 matching bundle entries, not 7.
That means this condition never becomes true:
$vcf9_bundle_details.count -eq 7Holodeck receives 8 bundles, but waits for exactly 7.
Result: an infinite loop.
Why Does the API Return 8 Bundles?
In VCF 9.0.2.0, the bundle structure changed compared to earlier VCF 9 versions.
The depot UI shows 7 visible rows, but the API response contains 8 entries matching:
9.0.2.0.*The 9.0.2 BOM appears to split some components more granularly, for example around VCF Operations, Operations Collector, and Fleet Management. The UI abstracts this nicely, but the API exposes one additional bundle-level entry.
The important part is this:
VCF Installer UI: 7 visible downloaded components
VCF Installer API: 8 matching bundle entries
Holodeck logic: expects exactly 7That mismatch is enough to block the deployment.
Workaround
Edit the Holodeck module directly on the HoloRouter.
Change this:
}elseif($vcf9_bundle_details.count -eq 7){To this:
}elseif($vcf9_bundle_details.count -ge 7){One-liner:
sed -i 's/$vcf9_bundle_details.count -eq 7/$vcf9_bundle_details.count -ge 7/' \
/root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1This changes the check from “exactly 7 bundles” to “at least 7 bundles”.
Restart the Running PowerShell Session
The currently running pwsh process already has the old function loaded in memory.
After patching the file, kill the running PowerShell process:
pkill -9 -f pwshThen start a fresh PowerShell session and resume the deployment:
Import-HoloDeckConfig -ConfigID <id>
New-HoloDeckInstance -Version 9.0.2.0 -InstanceID <same-as-before> <original flags>For example:
New-HoloDeckInstance -Version 9.0.2.0 -InstanceID 1 -ManagementOnlyUse the same flags you used in the original deployment.
What Happens After the Patch?
After applying the workaround, Holodeck resumes at the existing deployment state.
In my case, the state engine resumed at:
Install-VcfInstallerBundlesThe patched function immediately accepted the 8 returned bundle entries and logged:
Received all Bundle DetailsThe deployment then moved forward.
Some bundle download calls may return:
BUNDLE_DOWNLOAD_ALREADY_DOWNLOADEDThat is expected because the bundles are already present in the depot. The existing try/catch handling allows the phase to complete quickly.
After that, the deployment advanced to the management-domain phase.
Suggested Proper Fix
The quick fix is:
- }elseif($vcf9_bundle_details.count -eq 7){
+ }elseif($vcf9_bundle_details.count -ge 7){A version-specific fix would also work, for example:
VCF 9.0.0.0 / 9.0.1.0 -> expect 7
VCF 9.0.2.0 -> expect 8However, that would likely reintroduce the same type of bug in a future VCF BOM revision.
A better long-term approach would be to avoid a hardcoded count completely and validate the actual bundle download state instead.
For example, Holodeck should check that all required bundles for the selected deployment type and version are present and successfully downloaded, instead of assuming that the bundle count is always static.
Still, as an immediate workaround, changing -eq 7 to -ge 7 is enough to unblock the deployment.
Important Notes
This is an unofficial workaround.
The affected PowerShell module is not part of the public Holodeck documentation repository. It is bundled inside the HoloRouter OVA distributed through the Broadcom Support Portal.
Before editing vendor-supplied files, it is always a good idea to make a backup:
cp /root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1 \
/root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1.bakThen apply the patch.
Summary
This is a good example of a small hardcoded assumption causing a deployment to stall without an obvious fatal error.
The VCF Installer API was returning valid data. The depot was populated. The UI showed the bundles as successfully downloaded. But Holodeck was waiting for an exact number of bundle entries that no longer matched the VCF 9.0.2.0 BOM.
For affected Holodeck 9.0.2 users deploying VCF 9.0.2.0, the key symptom is:
Didn't receive all bundles. Received 8 bundle details. Trying again after 10 secondsIf you see this message, check the bundle count logic in:
SddcMgmtDeployment.psm1The workaround is small, but it can save a lot of troubleshooting time.
Quick Reference
cp /root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1 \
/root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1.bak
sed -i 's/$vcf9_bundle_details.count -eq 7/$vcf9_bundle_details.count -ge 7/' \
/root/.local/share/powershell/Modules/HoloDeck/Modules/SddcMgmtDeployment.psm1
pkill -9 -f pwshThen resume:
Import-HoloDeckConfig -ConfigID <id>
New-HoloDeckInstance -Version 9.0.2.0 -InstanceID <same-as-before> <original flags>
