Month: March 2024

Short answer: When the vCenter “PostgreSQL Archiver Service Health Alarm” fires or vmware-postgres-archiver shows as stopped or dead, read the archiver logs first. Broadcom documents three causes, each with its own fix. A wal_sender_timeout that is too low for a busy appliance needs 600s. A broken vpg_archiver replication slot needs to be dropped along with the archived segments. NTP drift makes the service health file go stale. Snapshot the VCSA before you touch vPostgres.

What the Postgres Archiver service does

vmware-postgres-archiver streams write-ahead log (WAL) segments out of vCenter’s embedded vPostgres database and stores them under /storage/archive/vpostgres/. It connects as a replication client through a replication slot called vpg_archiver. When it stops, vCenter raises the PostgreSQL Archiver Service Health Alarm, and the service health can turn red. File-based backup through the VAMI also depends on all required services running, so a dead archiver often shows up first as a failing backup job.

Step 1: identify which failure you have

service-control --status vmware-postgres-archiver
ls -l /var/log/vmware/vpostgres/
tail -n 50 /var/log/vmware/vpostgres/pg_archiver.log*.stderr
grep -i "stale" /var/log/vmon/vmon.log

Log evidence	Cause	Broadcom KB
pg_archiver could not receive data from WAL stream: server closed the connection unexpectedly plus terminating walsender process due to replication timeout in postgresql-*.log	wal_sender_timeout too low while the VCSA is under CPU load	KB 318472
pg_archiver could not seek compressed file "/storage/archive/vpostgres/….gz": Invalid argument and replication slot "vpg_archiver" already exists	Defunct replication slot / unreadable archived segments, typical after a crash or dirty shutdown	KB 388731
Service health xml file is stale in vmon.log, clock jumps in system logs	NTP not synchronising	KB 383704

Fix A: raise wal_sender_timeout (KB 318472)

Broadcom lists vCenter 6.7, 7.x and 8.x as affected. Edit /storage/db/vpostgres/postgresql.conf, replace the commented default #wal_sender_timeout = 60s with an active value, and restart both services:

cd /storage/db/vpostgres/
vi postgresql.conf
#   wal_sender_timeout = 600s

service-control --stop vmware-postgres-archiver vmware-vpostgres
service-control --start vmware-postgres-archiver vmware-vpostgres

Fix B: drop the dead vpg_archiver replication slot (KB 388731)

This is the usual “archiver dead after a homelab crash” case. Take an offline snapshot first, and in Enhanced Linked Mode that means all linked vCenters. Then:

1. List the replication slots. In the active column, f means the slot is not in use:
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "select * from pg_replication_slots;"
2. Drop the slot:
   /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "select pg_drop_replication_slot('vpg_archiver');"
3. Run the query from step 1 again and confirm it returns (0 rows).
4. Delete the old archived segments: rm /storage/archive/vpostgres/*
5. Start the service: service-control --start vmware-postgres-archiver, then check the slot list again.

Deleting /storage/archive/vpostgres/* removes only the WAL archive and leaves the live database alone. Even so, once you run it you can no longer roll back from those segments, so make sure the snapshot exists.

Fix C: repair time synchronisation (KB 383704)

When the clock jumps backwards, vmon marks the archiver’s health file as stale and the alarm goes red even though PostgreSQL itself is fine. Configure a reachable NTP source in the VAMI and check with ntpq -p that the appliance is actually receiving time. On nested lab vCenters, make sure the host itself is not drifting.

After the fix

Run service-control --status --all, check that the alarm clears in the vSphere Client, and start a manual VAMI file-based backup to confirm the whole chain works. If other services fail after a crash too, look at STS certificate expiry and expired root/administrator passwords, both of which cause similar symptoms.

Further reading

• Robert Guske: Fixing vCenter Postgres Archiver Service – Dead Postgres Replication Slot (original post)
• Broadcom KB 388731: vCenter PostgreSQL Archiver Service Won’t Start – Dead Postgres Replication Slot
• Broadcom KB 318472: VMware Postgres Archiver Service changes status to “stopped”
• Broadcom KB 383704: vmware-postgres-archiver status changed from green to red

Short answer: vSphere vMotion live-migrates a running VM between ESXi hosts by copying its memory and device state over a dedicated vMotion VMkernel network while the VM keeps running, then switching execution to the destination host in a brief stun. It needs a vMotion license, compatible CPUs (or EVC), storage both hosts can see (or Storage vMotion), and a vMotion network with at least 250 Mbps per concurrent migration and up to 150 ms round-trip time. vSphere 8 can also notify applications before a migration.

What vMotion transfers: cold data and hot data

KB 342209 splits a migration into two kinds of traffic:

Data	Content	Protocol	VMkernel port used
Cold	VM files and disks (when storage also moves)	NFC, TLS-encrypted by default	Provisioning vmk if configured, otherwise management
Hot	Memory and device state of the running VM	vMotion protocol	vMotion vmk

For a compute-only vMotion on shared storage only hot data moves. For vMotion without shared storage (compute plus storage in one operation) both move, so a provisioning VMkernel adapter keeps disk copy traffic off the management network.

How a live migration runs

1. Compatibility check: vCenter validates CPU compatibility, network and storage visibility, and device backing (connected ISO images or physical devices can block the move).
2. Memory pre-copy: the destination host creates a shadow VM and memory is copied while the VM keeps running. Pages the guest changes during the copy are tracked and sent again in further passes.
3. Switch-over: once the remaining dirty memory is small enough, the VM is briefly stunned, the last pages and device state are sent, and execution resumes on the destination.
4. Network notification: the destination host announces the VM’s MAC addresses on the physical network so switches update their tables, and the source VM is cleaned up.

The busier the guest writes to memory, the more pre-copy passes are needed. Bandwidth on the vMotion network therefore directly affects how long a migration takes and how short the stun is.

vMotion requirements for ESXi hosts

Area	Requirement
License	Each host licensed for vMotion; long-distance vMotion needs a license that covers it
CPU	Same or compatible CPU family, or an EVC baseline on the cluster
Storage	Shared storage visible to both hosts, or vMotion without shared storage
Bandwidth	At least 250 Mbps dedicated per concurrent vMotion session
Latency	Round-trip time up to 150 ms between hosts
VMkernel	vMotion-enabled vmk on both hosts; use the vMotion TCP/IP stack to route vMotion across L3 subnets
VM network	The VM port group must exist at L2 on the destination

Sources: Host Configuration for vSphere vMotion and KB 342209. To check which VMkernel adapters carry vMotion on a host, esxcli network ip interface tag get -i vmk1 shows the tags per adapter; more commands are in my esxcli cheat sheet for ESXi 8.

vMotion application notifications in vSphere 8

Some applications (clustered databases, trading, telco and real-time workloads) do not tolerate even a sub-second stun without preparation. Starting with vSphere 8.0, ESXi can tell an application inside the guest when a vMotion is about to start and when it has finished, so the application can drain work, fail over a role, or delay the migration. See vSphere vMotion Notifications for Latency Sensitive Applications.

• The VM needs hardware version 20 or later and current VMware Tools, which delivers the notifications to the guest.
• Enable it per VM with the advanced setting vmOpNotificationToAppEnabled = TRUE (VM Options > Advanced > Edit Configuration).
• The application registers through VMware Tools and must acknowledge the start event; vMotion pauses until it does or until the timeout expires.
• The host timeout vmOpNotificationToApp.Timeout (seconds, set via the vSphere API) defaults to 0: notifications are sent but migrations are not delayed. If both host and VM timeouts are set, the smaller one wins.

Keep the timeout short. A long delay also holds up DRS balancing and host maintenance mode, which is exactly when you need vMotion to finish.

Troubleshooting slow or failing vMotion

• Test the vMotion path with vmkping -I vmkX -S vmotion <dest-vmotion-IP> when using the vMotion TCP/IP stack; add -d -s 8972 to validate jumbo frames end to end.
• A migration stuck on the storage phase usually points to the provisioning/management network, not the vMotion vmk.
• CPU compatibility errors mean the cluster needs an EVC baseline or the VM needs a power cycle to pick one up.

Further reading

• Davoud Teimouri: A Deep Dive into VMware vSphere vMotion Application Notification
• Broadcom TechDocs: vSphere vMotion Notifications for Latency Sensitive Applications
• Broadcom TechDocs: Host Configuration for vSphere vMotion
• KB 342209: Understand vMotion networking requirements

Quick Tip – Using ESXCLI to upgrade ESXi 8.x…

Several users in the community have reported running into a memory error issue when attempting to upgrade their standalone ESXi hosts using this ESXCLI trick, which has been possible since ESXi 5.1 back in 2012. [MemoryError] Please refer to the log file for more details. Initially, I was not […]

---

VMware Social Media Advocacy

VMware vSphere 8.0 U2b Released

There is a new patch you’ll have to apply to your virtual infrastructure. The patch is quite important as it fixes A LOT, especially the CBT fix that occurred after the U2a.

---

VMware Social Media Advocacy

Aria Operations for Logs Expansion

You’re running a single node Aria Operations for Logs deployment and would like to expand it, let’s make it a 3-node cluster, supported configurations can be found here. To do that, deploy a new Aria Operations for Logs appliance and join it to an existing deployment, documentation here. […]

---

VMware Social Media Advocacy