Short answer: When the VCSA root partition (/) is full, find the culprit first: df -h and df -i to confirm the partition, then du to find the directory. The usual causes documented by Broadcom are a huge /var/log/audit/audit.log, a stuck mail queue in /var/spool/clientmqueue, the vSphere Client work cache, journald and old support bundles. Truncate or delete only what the KB says. Resizing the root disk is not supported on vCenter 7.0 and later.
Symptoms of a full VCSA root partition
- VAMI (
https://<vcenter>:5480) shows File system /root has run out of storage space, or vCenter raises a disk exhaustion alarm. Warnings start at 80%. - At 95%, vCenter stops
vmware-vpxdto protect the database. The vSphere Client then returns 503 Service Unavailable, services do not start, and patching or upgrades fail.
Step 1: Confirm which partition is full
SSH to the appliance as root, type shell, and check both space and inodes. VAMI Monitor > Disks shows the same data. Output from one of my appliances:
root@vcsa [ ~ ]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 11G 8.2G 1.9G 82% /
root@vcsa [ ~ ]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda3 712704 97100 615604 14% /
# only partitions at 78% or more (Broadcom KB 318953)
root@vcsa [ ~ ]# df -h | awk '0+$5 >= 78 {print}'Here the inodes are fine and the problem is space. Note which mount point is actually full. /storage/log, /storage/core, /storage/db and /storage/seat are separate virtual disks with their own KBs (see the table below). /storage/archive near 100% is expected by design on 6.7 and later. The root filesystem shows as /dev/sda3 on many builds. Broadcom KBs also show /dev/mapper/vg_root_0-lv_root_0 as the root device on some 7.0/8.0 systems.
If you cannot log in at all because / is full, Broadcom KB 318953 points to KB 322247 (steps 1 to 7) to get a single-user-mode shell.
Step 2: Find what fills the root partition
# largest areas on / (KB 422898)
du -sh /usr /var /opt /tmp /etc /home 2>/dev/null | sort -h
du -sh /var/log/* 2>/dev/null | sort -h | tail -10
journalctl --disk-usage
# top 20 entries under a path (KB 318953)
du -a /var | sort -n -r | head -n 20
# files larger than 100 MB in the current directory tree
find ./ -type f -size +100MIn my case, /var/spool was the problem:
root@vcsa [ /var ]# du -sh *
2.1G log
5.2G spoolCommon culprits and the matching Broadcom KB
| Where | Cause | Versions in KB | KB |
|---|---|---|---|
/var/log/audit/audit.log | audit.log not rotated (often cron failing because the root password expired) | 5.5 to 6.7 | 318911 |
/var/spool/clientmqueue | Mail queued to an unreachable SMTP relay; sendmail may also crash and fill /storage/core with core.sendmail.* | 8.x | 318475 |
/usr/lib/vmware-vsphere-ui/server/work | vSphere Client (Tomcat) work cache growth (normal about 100 MB to 1 GB, abnormal 10 GB+) | 8.x | 422898 |
/usr/lib/vmware-vsphere-ui/server/work/tmp/*.esa | Failed plug-in redeploys; fixed in vCenter 8.0.3 build 24262322 | 7.0, 8.0 | 319442 |
| journald | System journal growth | 8.x | 422898, 316602 |
/opt/vmware/var/log/lighttpd | VAMI access logs | 6.7 | 318172 |
/etc/vmware-vpx/docRoot/diagnostics, /etc/vmware-vpx/bundle-stage | Old support bundles not deleted | all | 319886 |
/root (and other non-default files on /) | Files copied there by admins (ISOs, bundles, backups) | 8.x | 422898 |
Not the root partition? For /storage/log use KB 313077, and for /storage/core use KB 318750.
Step 3: Safe cleanup
Before you delete anything, take a backup or snapshot of the vCenter VM. With Enhanced Linked Mode, follow Broadcom’s snapshot best practices. vCenter HA must be removed first. For active log files, the KBs truncate the file instead of deleting it. Do the same.
audit.log
cd /var/log/audit
ls -lh
truncate -s 0 audit.log
# check that the daily cron actually runs
ls -l /var/spool/cron/lastrun/
# an expired root password breaks cron
chage -l rootIf chage shows the root password as expired, change it, then restart services (service-control --stop --all and service-control --start --all). Otherwise the log fills up again.
clientmqueue (sendmail queue)
systemctl stop sendmail.service
find /var/spool/clientmqueue/* -exec rm {} \;
ls -ltrh /var/spool/clientmqueue/
# if sendmail was crashing, also remove its cores
rm -rf /storage/core/core.sendmail.*
# fix vCenter > Configure > General > Edit > Mail, then:
systemctl start sendmail.serviceIn my case, the space in /var/spool was this mail queue. KB 318475 gives the typical reason: vCenter’s mail settings point to an SMTP relay it cannot reach, so every alarm email stays queued. Fix the mail settings, or the queue grows again.
vSphere Client work cache and journald
du -sh /usr/lib/vmware-vsphere-ui/server/work
service-control --stop vsphere-ui
rm -rf /usr/lib/vmware-vsphere-ui/server/work/*
service-control --start vsphere-ui
journalctl --vacuum-time=7d
df -h /Clearing the work cache only restarts the vSphere Client (HTML5); running VMs, hosts and core vCenter services are not affected. The cache is regenerated when the service starts.
Can you resize the VCSA root partition?
Mostly not. Broadcom KB 316602 says:
| vCenter version | Resize root disk (VMDK1) |
|---|---|
| 6.5 | Not supported |
| 6.7 | Added in 6.7 |
| 7.0 and later (incl. 8.0) | Not supported |
In the vCenter 8.0 reference layout (Tiny, default storage), VMDK1 holds / (48 GB) and /boot (KB 316438). Check your real size with df -h, because Broadcom’s own 8.x KB examples also show much smaller root filesystems. On 7.0 and 8.0, cleanup is the only supported fix for /. KB 316602 even recommends journalctl --vacuum-time when an upgrade pre-check complains about free space on root.
The other disks (/storage/log, /storage/seat, /storage/db and so on) are LVM and can be grown online:
- Take a file-based backup. Remove all snapshots from the vCenter VM, because a disk with snapshots cannot be extended.
- Map the mount point to the virtual disk with
df -h; lsblk; lsscsi(for example/storage/logissde, SCSI(0:4)). - Increase that hard disk in the VM settings.
- Run
/usr/lib/applmgmt/support/scripts/autogrow.shand confirm withdf -h.
The KBs above list versions up to 8.0. For vCenter 9.0, check the current Broadcom KBs before you assume the same limits.
Prevent it from happening again
- Monitor VCSA disks and act on the 80% alarm, not the 95% outage (KB 318930).
- Keep the root password from expiring silently, so log rotation keeps working.
- Keep the vCenter SMTP settings valid, delete support bundles after upload, and never use
/rootas a download folder. - Stay on current patches. Several root-partition bugs (e.g. the
.esafiles) are fixed in later builds.
Sources
- KB 318953: vCenter Server Appliance disk space is full
- KB 422898: The /root partition on the vCenter Server is full
- KB 318911 (formerly 2149278): root partition 100% full due to audit.log files not being rotated
- KB 318475: large files queued for mails in /var/spool/clientmqueue
- KB 319442: VCSA root partition full due to .esa extension files
- KB 318172: root partition consumed by lighttpd logs
- KB 319886: vCenter Server partition full due to support bundle
- KB 316602: Increasing the disk space for the vCenter Server Appliance in vSphere 6.5, 6.7, 7.0 and 8.0
- KB 316438: List of VMDKs/Partitions for vCenter Server 8.0
- KB 322247: Change or Reset the root account password in vCenter Appliance
- KB 318930: Monitor and prevent vCenter Appliance disks from filling
