Month: March 2020

Short answer: On vCenter Server Appliance 6.7, 7.0, 8.0 and 9.0 you reset the VUM database in four steps. Stop the service with service-control --stop vmware-updatemgr, run python /usr/lib/vmware-updatemgr/bin/updatemgr-utility.py reset-db, empty /storage/updatemgr/patch-store/ and start the service again. On 6.5 the tool is updatemgr-util reset-db. From vCenter 9.1, plain reset-db only resets online depots, and a full reset needs reset-db --force. Take an offline snapshot first.

The procedure is Broadcom KB 316581 (formerly VMware KB 2147284; the old number redirects). Its current title is Reset VMware Lifecycle Manager depot in vCenter Server Appliance 6.5/6.7/7.0/8.0/9.x. The service is still called vmware-updatemgr even though the product is now vSphere Lifecycle Manager (vLCM).

When to reset the VUM database

I first needed this after upgrading a VCSA from 6.5 to 6.7. Every Scan for Updates failed with:

There are errors during the scan operation. Check the events and log files for details.

Broadcom KB 345496 (formerly 67771) documents this exact case. The cause is missing dependency VIBs, and /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log shows MISSING_DEPENDENCY_VIBS precheck entries. The fix in that KB is the database reset described below. More generally, reset the database only as a last resort, when the depot or patch store is inconsistent and normal troubleshooting has not helped.

Before you reset: what gets deleted

KB 316581 calls a full reset a destructive task. Plan for the following:

• Custom baselines, custom download settings and manually imported patches and ISOs are removed. Cluster images are kept.
• On vSAN clusters the vSAN default baselines are removed as well. vCenter re-creates them automatically on the next vSAN configuration change, such as adding a host or disk or an HCL DB update.
• The new “download token” based depot URLs count as custom URLs, so you have to add them again after the reset.
• Write down proxy settings, third-party download URLs and custom baselines before you start.
• Take a backup or an offline snapshot (powered off) of the VCSA. In Enhanced Linked Mode, do it for all replicating nodes.

NSX warning: the KB says not to reset the database without VMware Support if NSX-T is installed on a cluster that uses a vLCM image. After a reset, those clusters cannot save or validate their image until the NSX depot is uploaded again (see the NSX section below).

Reset VUM database commands by vCenter version

vCenter Server Appliance	Reset command	Notes
6.5	/usr/lib/vmware-updatemgr/bin/updatemgr-util reset-db	Full reset
6.7, 7.0, 8.0	python /usr/lib/vmware-updatemgr/bin/updatemgr-utility.py reset-db	Full reset
9.0 without VKS	python /usr/lib/vmware-updatemgr/bin/updatemgr-utility.py reset-db	Full reset
9.0 with VKS	Same command, then the SQL fix from KB 316581	The reset removes a column used by VKS upgrades
9.1 and later	reset-db = online depot reset reset-db --force = full reset	Default behaviour changed in 9.1

Full reset on vCenter 6.5, 6.7, 7.0 and 8.0

1. SSH to the VCSA as root and type shell to get a Bash shell.
2. Stop the Update Manager service.
3. Run the reset command for your version (see the table).
4. Empty the patch store.
5. Start the service, then log out of the vSphere Client and back in.

shell
service-control --stop vmware-updatemgr

# 6.7 / 7.0 / 8.0 (and 9.0 without VKS)
python /usr/lib/vmware-updatemgr/bin/updatemgr-utility.py reset-db
# 6.5 only:
# /usr/lib/vmware-updatemgr/bin/updatemgr-util reset-db

rm -rf /storage/updatemgr/patch-store/*
service-control --start vmware-updatemgr

Double-check the rm -rf path before you press Enter. It has to be exactly /storage/updatemgr/patch-store/*.

vCenter 9.0 with vSphere Kubernetes Service (VKS)

KB 316581 lists a known issue: on vCenter 9.0, reset-db removes a database column that VKS upgrades use. When VKS is in use, take an offline snapshot, run the reset, and then apply the SQL script from the KB to the VCDB. The KB names the file flb-vcdb-fix.sql and runs it with psql -U postgres -d VCDB -f flb-vcdb-fix.sql. The expected output is SET followed by six ALTER TABLE lines. Copy the SQL from the KB itself rather than from a blog. If your output does not match, open a support request with Broadcom and reference the KB.

vCenter 9.1 and later: online depot reset vs reset-db --force

From vCenter 9.1, updatemgr-utility.py reset-db no longer wipes everything by default. It cleans all online depots (built-in and custom) and syncs them with the latest content. It does not restart the vLCM service, so running vLCM operations are not interrupted. The tool asks for SSO administrator credentials and returns a task that you can follow in the UI or with dcli.

shell
python /usr/lib/vmware-updatemgr/bin/updatemgr-utility.py reset-db
# monitor the SyncDepots task
dcli com vmware cis tasks get --task <task id>

If the online reset does not fix the depot, you have two options:

1. Full reset with reset-db --force. This removes all online and offline depot content, including custom online depots and the NSX and IOFilter depots. Only the built-in online depot content comes back. The command restarts the vLCM service. Use the same stop / reset / empty patch store / start sequence as above, with reset-db --force as the reset command.
2. Delete only the offline depot that causes the problem:

dcli com vmware esx settings depots offline list
dcli com vmware esx settings depots offline delete-task --depot <depot UUID>
# if the depot is in use:
dcli com vmware esx settings depots offline delete-task --depot <depot UUID> --force true

NSX in vLCM-managed clusters: re-upload the NSX depot

This applies to vCenter 7.0 U1 and later with NSX-T. It is not needed for NSX-V. NSX Manager uploads its depot to vCenter only when NSX is configured on a vLCM cluster and the depot is missing. The KB therefore uses an empty helper cluster to trigger the upload again:

1. In vCenter, create a new, empty cluster that is managed with a vLCM image.
2. In NSX Manager, open the Hosts and Clusters view, select the new cluster, click Add NSX and pick a Transport Node Profile. Live traffic is not affected because the cluster has no hosts.
3. When the related vCenter tasks complete, delete the helper cluster.

After the reset

• Log out of the vSphere Client and back in.
• Re-add the download-token depot URLs, the proxy and any third-party depots.
• Re-import ISOs and offline bundles. Re-create and re-attach custom baselines on clusters that still use baselines.
• Let the depot sync finish, then run a new compliance check.

Sources

• Broadcom KB 316581: Reset VMware Lifecycle Manager depot in vCenter Server Appliance 6.5/6.7/7.0/8.0/9.x (legacy KB 2147284)
• Broadcom KB 345496: “There are errors during the scan operation” after upgrading vCenter Server Appliance from 6.5 to 6.7 (legacy KB 67771)

Related on vdan.cz

• Cisco Custom ISO MISSING_DEPENDENCY_VIBS: ESXi 6.7 to 7.0
• vLCM: Convert Update Manager Baselines to an Image
• Retrieving detailed vSphere Lifecycle Manager information
• Upgrade ESXi 6.7 to 7.0 with vSphere Lifecycle Manager