PostgreSQL Archiver Service Health Alarm on vCenter: Fix

Short answer: When the vCenter “PostgreSQL Archiver Service Health Alarm” fires or vmware-postgres-archiver shows as stopped or dead, read the archiver logs first. Broadcom documents three causes, each with its own fix. A wal_sender_timeout that is too low for a busy appliance needs 600s. A broken vpg_archiver replication slot needs to be dropped along with the archived segments. NTP drift makes the service health file go stale. Snapshot the VCSA before you touch vPostgres.

Table of Contents

What the Postgres Archiver service does

vmware-postgres-archiver streams write-ahead log (WAL) segments out of vCenter’s embedded vPostgres database and stores them under /storage/archive/vpostgres/. It connects as a replication client through a replication slot called vpg_archiver. When it stops, vCenter raises the PostgreSQL Archiver Service Health Alarm, and the service health can turn red. File-based backup through the VAMI also depends on all required services running, so a dead archiver often shows up first as a failing backup job.

Step 1: identify which failure you have

service-control --status vmware-postgres-archiver
ls -l /var/log/vmware/vpostgres/
tail -n 50 /var/log/vmware/vpostgres/pg_archiver.log*.stderr
grep -i "stale" /var/log/vmon/vmon.log

Log evidence	Cause	Broadcom KB
`pg_archiver could not receive data from WAL stream: server closed the connection unexpectedly` plus `terminating walsender process due to replication timeout` in `postgresql-*.log`	`wal_sender_timeout` too low while the VCSA is under CPU load	KB 318472
`pg_archiver could not seek compressed file "/storage/archive/vpostgres/….gz": Invalid argument` and `replication slot "vpg_archiver" already exists`	Defunct replication slot / unreadable archived segments, typical after a crash or dirty shutdown	KB 388731
`Service health xml file is stale` in `vmon.log`, clock jumps in system logs	NTP not synchronising	KB 383704

Fix A: raise wal_sender_timeout (KB 318472)

Broadcom lists vCenter 6.7, 7.x and 8.x as affected. Edit /storage/db/vpostgres/postgresql.conf, replace the commented default #wal_sender_timeout = 60s with an active value, and restart both services:

cd /storage/db/vpostgres/
vi postgresql.conf
#   wal_sender_timeout = 600s

service-control --stop vmware-postgres-archiver vmware-vpostgres
service-control --start vmware-postgres-archiver vmware-vpostgres

Fix B: drop the dead vpg_archiver replication slot (KB 388731)

This is the usual “archiver dead after a homelab crash” case. Take an offline snapshot first, and in Enhanced Linked Mode that means all linked vCenters. Then:

List the replication slots. In the active column, f means the slot is not in use:
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "select * from pg_replication_slots;"
Drop the slot:
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "select pg_drop_replication_slot('vpg_archiver');"
Run the query from step 1 again and confirm it returns (0 rows).
Delete the old archived segments: rm /storage/archive/vpostgres/*
Start the service: service-control --start vmware-postgres-archiver, then check the slot list again.

Deleting /storage/archive/vpostgres/* removes only the WAL archive and leaves the live database alone. Even so, once you run it you can no longer roll back from those segments, so make sure the snapshot exists.

Fix C: repair time synchronisation (KB 383704)

When the clock jumps backwards, vmon marks the archiver’s health file as stale and the alarm goes red even though PostgreSQL itself is fine. Configure a reachable NTP source in the VAMI and check with ntpq -p that the appliance is actually receiving time. On nested lab vCenters, make sure the host itself is not drifting.

After the fix

Run service-control --status --all, check that the alarm clears in the vSphere Client, and start a manual VAMI file-based backup to confirm the whole chain works. If other services fail after a crash too, look at STS certificate expiry and expired root/administrator passwords, both of which cause similar symptoms.

Author: Daniel Micanek

Senior Service Architect, SAP/TDL Platform Services Team at Vivicta | vExpert ⭐⭐⭐⭐⭐⭐⭐ | Broadcom Knight: vSphere Kubernetes Service, VCF Networking (NSX), VCF Automation | Broadcom Software Knight: vSphere | Cisco Champion. Notes from running about 1,000 ESXi hosts on Cisco UCS, VCF 9 and SAP HANA on vSphere. View all posts by Daniel Micanek

Fixing a Dead vCenter Postgres Archiver Service (PostgreSQL Archiver Service Health Alarm)