Month: March 2020

Short answer: When the VCSA root partition (/) is full, find the culprit first: df -h and df -i to confirm the partition, then du to find the directory. The usual causes documented by Broadcom are a huge /var/log/audit/audit.log, a stuck mail queue in /var/spool/clientmqueue, the vSphere Client work cache, journald and old support bundles. Truncate or delete only what the KB says. Resizing the root disk is not supported on vCenter 7.0 and later.

Symptoms of a full VCSA root partition

• VAMI (https://<vcenter>:5480) shows File system /root has run out of storage space, or vCenter raises a disk exhaustion alarm. Warnings start at 80%.
• At 95%, vCenter stops vmware-vpxd to protect the database. The vSphere Client then returns 503 Service Unavailable, services do not start, and patching or upgrades fail.

Step 1: Confirm which partition is full

SSH to the appliance as root, type shell, and check both space and inodes. VAMI Monitor > Disks shows the same data. Output from one of my appliances:

root@vcsa [ ~ ]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        11G  8.2G  1.9G  82% /

root@vcsa [ ~ ]# df -i
Filesystem     Inodes IUsed  IFree IUse% Mounted on
/dev/sda3      712704 97100 615604   14% /

# only partitions at 78% or more (Broadcom KB 318953)
root@vcsa [ ~ ]# df -h | awk '0+$5 >= 78 {print}'

Here the inodes are fine and the problem is space. Note which mount point is actually full. /storage/log, /storage/core, /storage/db and /storage/seat are separate virtual disks with their own KBs (see the table below). /storage/archive near 100% is expected by design on 6.7 and later. The root filesystem shows as /dev/sda3 on many builds. Broadcom KBs also show /dev/mapper/vg_root_0-lv_root_0 as the root device on some 7.0/8.0 systems.

If you cannot log in at all because / is full, Broadcom KB 318953 points to KB 322247 (steps 1 to 7) to get a single-user-mode shell.

Step 2: Find what fills the root partition

# largest areas on / (KB 422898)
du -sh /usr /var /opt /tmp /etc /home 2>/dev/null | sort -h
du -sh /var/log/* 2>/dev/null | sort -h | tail -10
journalctl --disk-usage

# top 20 entries under a path (KB 318953)
du -a /var | sort -n -r | head -n 20

# files larger than 100 MB in the current directory tree
find ./ -type f -size +100M

In my case, /var/spool was the problem:

root@vcsa [ /var ]# du -sh *
2.1G    log
5.2G    spool

Common culprits and the matching Broadcom KB

Where	Cause	Versions in KB	KB
/var/log/audit/audit.log	audit.log not rotated (often cron failing because the root password expired)	5.5 to 6.7	318911
/var/spool/clientmqueue	Mail queued to an unreachable SMTP relay; sendmail may also crash and fill /storage/core with core.sendmail.*	8.x	318475
/usr/lib/vmware-vsphere-ui/server/work	vSphere Client (Tomcat) work cache growth (normal about 100 MB to 1 GB, abnormal 10 GB+)	8.x	422898
/usr/lib/vmware-vsphere-ui/server/work/tmp/*.esa	Failed plug-in redeploys; fixed in vCenter 8.0.3 build 24262322	7.0, 8.0	319442
journald	System journal growth	8.x	422898, 316602
/opt/vmware/var/log/lighttpd	VAMI access logs	6.7	318172
/etc/vmware-vpx/docRoot/diagnostics, /etc/vmware-vpx/bundle-stage	Old support bundles not deleted	all	319886
/root (and other non-default files on /)	Files copied there by admins (ISOs, bundles, backups)	8.x	422898

Not the root partition? For /storage/log use KB 313077, and for /storage/core use KB 318750.

Step 3: Safe cleanup

Before you delete anything, take a backup or snapshot of the vCenter VM. With Enhanced Linked Mode, follow Broadcom’s snapshot best practices. vCenter HA must be removed first. For active log files, the KBs truncate the file instead of deleting it. Do the same.

audit.log

cd /var/log/audit
ls -lh
truncate -s 0 audit.log

# check that the daily cron actually runs
ls -l /var/spool/cron/lastrun/
# an expired root password breaks cron
chage -l root

If chage shows the root password as expired, change it, then restart services (service-control --stop --all and service-control --start --all). Otherwise the log fills up again.

clientmqueue (sendmail queue)

systemctl stop sendmail.service
find /var/spool/clientmqueue/* -exec rm {} \;
ls -ltrh /var/spool/clientmqueue/
# if sendmail was crashing, also remove its cores
rm -rf /storage/core/core.sendmail.*
# fix vCenter > Configure > General > Edit > Mail, then:
systemctl start sendmail.service

In my case, the space in /var/spool was this mail queue. KB 318475 gives the typical reason: vCenter’s mail settings point to an SMTP relay it cannot reach, so every alarm email stays queued. Fix the mail settings, or the queue grows again.

vSphere Client work cache and journald

du -sh /usr/lib/vmware-vsphere-ui/server/work
service-control --stop vsphere-ui
rm -rf /usr/lib/vmware-vsphere-ui/server/work/*
service-control --start vsphere-ui

journalctl --vacuum-time=7d
df -h /

Clearing the work cache only restarts the vSphere Client (HTML5); running VMs, hosts and core vCenter services are not affected. The cache is regenerated when the service starts.

Can you resize the VCSA root partition?

Mostly not. Broadcom KB 316602 says:

vCenter version	Resize root disk (VMDK1)
6.5	Not supported
6.7	Added in 6.7
7.0 and later (incl. 8.0)	Not supported

In the vCenter 8.0 reference layout (Tiny, default storage), VMDK1 holds / (48 GB) and /boot (KB 316438). Check your real size with df -h, because Broadcom’s own 8.x KB examples also show much smaller root filesystems. On 7.0 and 8.0, cleanup is the only supported fix for /. KB 316602 even recommends journalctl --vacuum-time when an upgrade pre-check complains about free space on root.

The other disks (/storage/log, /storage/seat, /storage/db and so on) are LVM and can be grown online:

1. Take a file-based backup. Remove all snapshots from the vCenter VM, because a disk with snapshots cannot be extended.
2. Map the mount point to the virtual disk with df -h; lsblk; lsscsi (for example /storage/log is sde, SCSI(0:4)).
3. Increase that hard disk in the VM settings.
4. Run /usr/lib/applmgmt/support/scripts/autogrow.sh and confirm with df -h.

The KBs above list versions up to 8.0. For vCenter 9.0, check the current Broadcom KBs before you assume the same limits.

Prevent it from happening again

• Monitor VCSA disks and act on the 80% alarm, not the 95% outage (KB 318930).
• Keep the root password from expiring silently, so log rotation keeps working.
• Keep the vCenter SMTP settings valid, delete support bundles after upload, and never use /root as a download folder.
• Stay on current patches. Several root-partition bugs (e.g. the .esa files) are fixed in later builds.

Sources

• KB 318953: vCenter Server Appliance disk space is full
• KB 422898: The /root partition on the vCenter Server is full
• KB 318911 (formerly 2149278): root partition 100% full due to audit.log files not being rotated
• KB 318475: large files queued for mails in /var/spool/clientmqueue
• KB 319442: VCSA root partition full due to .esa extension files
• KB 318172: root partition consumed by lighttpd logs
• KB 319886: vCenter Server partition full due to support bundle
• KB 316602: Increasing the disk space for the vCenter Server Appliance in vSphere 6.5, 6.7, 7.0 and 8.0
• KB 316438: List of VMDKs/Partitions for vCenter Server 8.0
• KB 322247: Change or Reset the root account password in vCenter Appliance
• KB 318930: Monitor and prevent vCenter Appliance disks from filling

Related on vdan.cz

• Reset vCenter root password without restarting
• Fixing vCenter Postgres archiver service dead
• A quick guide to installing and using lsdoctor for vCenter troubleshooting