vSphere 8.0 Update 2 adds federated authentication with Azure Active Directory (Microsoft Entra ID) for vCenter, as presented at VMware Explore 2023.
At the 2023 VMware Explore event in Barcelona. I was on presentation with Viviana Miranda relates to the way users authenticate to vCenter.
vSphere 8.0 Update 2 update heralds a number of groundbreaking features, with one standout enhancement in user authentication methods for vCenter. vCenter Server 8.0 Update 2 has now incorporated federated authentication capabilities with Azure Active Directory.
Dive into the details of this integration and discover how to activate it.
The Advantages of External Identity Providers
Organizations that leverage external identity providers can expect to reap substantial benefits:
- Integration with existing identity provider infrastructure to streamline processes.
- Implementation of Single Sign-On to simplify access across services.
- Adherence to the best practices of role separation between infrastructure management and identity administration.
- Utilization of robust multi-factor authentication options that come with their chosen identity providers.
Supported Identity Providers in vSphere 8.0 U2
While our focus here is the Azure Active Directory integration, it’s essential to highlight the comprehensive range of authentication methods now supported with vCenter Server 8.0 U2.
Configure NVMe/TCP between vSphere 8.0 Update 1 and NetApp ONTAP 9.13.1 for VMFS datastores: ESXi adapters, ONTAP subsystem and namespaces.
vSphere 8U1 – Deep dive on configuring NVMe-oF (Non-Volatile Memory Express over Fabrics) for VMware vSphere datastores.
What’s new
With vSphere 8.0 update 1, VMware has completed their journey to a completely native end-to-end NVMe storage stack. Prior to 8.0U1, there was a SCSI translation layer which added some complexity to the stack and slightly decreased some of the efficiencies inherent in the NVMe protocol.
VMware vCenter Server heap-overflow vulnerability (CVE-2023-20892)
Description: The vCenter Server contains a heap overflow vulnerability due to the usage of uninitialized memory in the implementation of the DCERPC protocol. VMware has evaluated the severity of this issue to be in the Important severity range with a maximum CVSSv3 base score of 8.1.
Known Attack Vectors: A malicious actor with network access to vCenter Server may exploit this issue to execute arbitrary code on the underlying operating system that hosts vCenter Server.
VMware vCenter Server use-after-free vulnerability (CVE-2023-20893)
Description: The vCenter Server contains a use-after-free vulnerability in the implementation of the DCERPC protocol. VMware has evaluated the severity of this issue to be in the Important severity range with a maximum CVSSv3 base score of 8.1.
Known Attack Vectors: A malicious actor with network access to vCenter Server may exploit this issue to execute arbitrary code on the underlying operating system that hosts vCenter Server.
VMware vCenter Server out-of-bounds write vulnerability (CVE-2023-20894)
Description: The vCenter Server contains an out-of-bounds write vulnerability in the implementation of the DCERPC protocol. VMware has evaluated the severity of this issue to be in the Important severity range with a maximum CVSSv3 base score of 8.1.
Known Attack Vectors: A malicious actor with network access to vCenter Server may trigger an out-of-bound write by sending a specially crafted packet leading to memory corruption.
VMware vCenter Server out-of-bounds read vulnerability (CVE-2023-20895)
Description: The vCenter Server contains a memory corruption vulnerability in the implementation of the DCERPC protocol. VMware has evaluated the severity of this issue to be in the Important severity range with a maximum CVSSv3 base score of 8.1
Known Attack Vectors: A malicious actor with network access to vCenter Server may trigger a memory corruption vulnerability which may bypass authentication.
VMware vCenter Server out-of-bounds read vulnerability (CVE-2023-20896)
Description: The vCenter Server contains an out-of-bounds read vulnerability in the implementation of the DCERPC protocol. VMware has evaluated the severity of this issue to be in the Moderate severity range with a maximum CVSSv3 base score of 5.9.
Known Attack Vectors: A malicious actor with network access to vCenter Server may trigger an out-of-bounds read by sending a specially crafted packet leading to denial-of-service of certain services (vmcad, vmdird, and vmafdd).
Cisco UCS / Proactive HA Registration / vCenter server credentials / Register
Cisco UCS / Proactive HA Registration / Register
How Could I check it? Edit Proactive HA / Providers
It is better use Name “ADDDC_Memory_Rank_VLS” without spaces. On my picture I used “My F1705 Alerts”
Adding Custom Alert is only possible with unregistered Cisco UCS Provider, it is better to do it immediatly after Cisco UCS Manager Plugin instalation.
Now I can deceided If I will block F1705 or NOT. I personaly preffer to have F1705 Alert under Proactive HA. Then I only restart Blades with F1705. During reboot Hard-PPRpermanently remaps accesses from a designated faulty row to a designated spare row.
The fastest workaround for Log4j CVE-2021-44228 (VMSA-2021-0028) on vCenter Server 7.0, 6.7 and 6.5, and why patching is the real fix.
https://logging.apache.org/log4j/2.x/
Apache Log4j open source component has security bug (CVE-2021-44228 – VMSA-2021-0028). It is neccesary to fix vCenter Server 7.0.x, vCenter 6.7.x & vCenter 6.5.x.
Connected to service
* List APIs: "help api list"
* List Plugins: "help pi list"
* Launch BASH: "shell"
Command> shell
Shell access is granted to root
root@localhost [ ~ ]# cd /tmp
root@localhost [ /tmp ]# vim vc_log4j_mitigator.py
Run script python vc_log4j_mitigator.py
root@localhost [ /tmp ]# python vc_log4j_mitigator.py
2021-12-21T10:38:20 INFO main: Script version: 1.6.0
2021-12-21T10:38:20 INFO main: vCenter type: Version: 7.0.2.00500; Build: 18455184; Deployment type: embedded; Gateway: False; VCHA: False; Windows: False;
A service stop and start is required to complete this operation. Continue?[y]y
2021-12-21T10:38:23 INFO stop: stopping services
2021-12-21T10:38:46 INFO process_jar: Found a VULNERABLE FILE: /opt/vmware/lib64/log4j-core-2.13.0.jar
2021-12-21T10:38:46 INFO backup_file: VULNERABLE FILE: /opt/vmware/lib64/log4j-core-2.13.0.jar backed up to /tmp/tmpxi89fco8/opt/vmware/lib64/log4j-core-2.13.0.jar.bak
2021-12-21T10:38:47 INFO process_jar: VULNERABLE FILE: /opt/vmware/lib64/log4j-core-2.13.0.jar backed up to /tmp/tmpxi89fco8/opt/vmware/lib64/log4j-core-2.13.0.jar.bak
2021-12-21T10:39:03 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.13.1.jar
2021-12-21T10:39:03 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:04 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:04 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar
2021-12-21T10:39:04 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak
2021-12-21T10:39:04 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.8.2.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.8.2.jar.bak
2021-12-21T10:39:06 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.11.0.jar
2021-12-21T10:39:06 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.11.0.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.11.0.jar.bak
2021-12-21T10:39:06 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.11.0.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.11.0.jar.bak
2021-12-21T10:39:07 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.11.2.jar
2021-12-21T10:39:07 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.11.2.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.11.2.jar.bak
2021-12-21T10:39:07 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware/common-jars/log4j-core-2.11.2.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/common-jars/log4j-core-2.11.2.jar.bak
2021-12-21T10:39:08 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/sso/lstool/lib/log4j-core-2.13.1.jar
2021-12-21T10:39:08 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/sso/lstool/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/sso/lstool/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:08 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/sso/lstool/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/sso/lstool/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:14 INFO process_jar: Found a VULNERABLE FILE: /tmp/tmpn2a_0ql2/WEB-INF/lib/log4j-core-2.13.3.jar
2021-12-21T10:39:14 INFO backup_file: VULNERABLE FILE: /tmp/tmpn2a_0ql2/WEB-INF/lib/log4j-core-2.13.3.jar backed up to /tmp/tmpxi89fco8/tmp/tmpn2a_0ql2/WEB-INF/lib/log4j-core-2.13.3.jar.bak
2021-12-21T10:39:15 INFO process_war: Found a VULNERABLE WAR file with: /usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-fileupload.war
2021-12-21T10:39:15 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-fileupload.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-fileupload.war.bak
2021-12-21T10:39:15 INFO process_war: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-fileupload.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-fileupload.war.bak
2021-12-21T10:39:15 INFO process_jar: Found a VULNERABLE FILE: /tmp/tmpxn5_4ah_/WEB-INF/lib/log4j-core-2.13.3.jar
2021-12-21T10:39:15 INFO backup_file: VULNERABLE FILE: /tmp/tmpxn5_4ah_/WEB-INF/lib/log4j-core-2.13.3.jar backed up to /tmp/tmpxi89fco8/tmp/tmpxn5_4ah_/WEB-INF/lib/log4j-core-2.13.3.jar.bak
2021-12-21T10:39:16 INFO process_war: Found a VULNERABLE WAR file with: /usr/lib/vmware-updatemgr/bin/jetty/webapps/root.war
2021-12-21T10:39:16 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/webapps/root.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/webapps/root.war.bak
2021-12-21T10:39:16 INFO process_war: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/webapps/root.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/webapps/root.war.bak
2021-12-21T10:39:16 INFO process_jar: Found a VULNERABLE FILE: /tmp/tmpa4w275ot/WEB-INF/lib/log4j-core-2.13.3.jar
2021-12-21T10:39:16 INFO backup_file: VULNERABLE FILE: /tmp/tmpa4w275ot/WEB-INF/lib/log4j-core-2.13.3.jar backed up to /tmp/tmpxi89fco8/tmp/tmpa4w275ot/WEB-INF/lib/log4j-core-2.13.3.jar.bak
2021-12-21T10:39:17 INFO process_war: Found a VULNERABLE WAR file with: /usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-filedownload.war
2021-12-21T10:39:17 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-filedownload.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-filedownload.war.bak
2021-12-21T10:39:18 INFO process_war: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-filedownload.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-filedownload.war.bak
2021-12-21T10:39:21 INFO process_jar: Found a VULNERABLE FILE: /tmp/tmpxv_znca3/WEB-INF/lib/log4j-core-2.13.1.jar
2021-12-21T10:39:21 INFO backup_file: VULNERABLE FILE: /tmp/tmpxv_znca3/WEB-INF/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/tmp/tmpxv_znca3/WEB-INF/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:22 INFO process_war: Found a VULNERABLE WAR file with: /usr/lib/vmware-sso/vmware-sts/webapps/ROOT.war
2021-12-21T10:39:22 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-sso/vmware-sts/webapps/ROOT.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-sso/vmware-sts/webapps/ROOT.war.bak
2021-12-21T10:39:24 INFO process_war: VULNERABLE FILE: /usr/lib/vmware-sso/vmware-sts/webapps/ROOT.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-sso/vmware-sts/webapps/ROOT.war.bak
2021-12-21T10:39:25 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar
2021-12-21T10:39:25 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:26 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:28 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar
2021-12-21T10:39:28 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar.bak
2021-12-21T10:39:29 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar.bak
2021-12-21T10:39:32 INFO process_jar: Found a VULNERABLE FILE: /tmp/tmprq0yfnd1/WEB-INF/lib/log4j-core-2.13.1.jar
2021-12-21T10:39:32 INFO backup_file: VULNERABLE FILE: /tmp/tmprq0yfnd1/WEB-INF/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/tmp/tmprq0yfnd1/WEB-INF/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:33 INFO process_war: Found a VULNERABLE WAR file with: /usr/lib/vmware-lookupsvc/webapps/ROOT.war
2021-12-21T10:39:33 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-lookupsvc/webapps/ROOT.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-lookupsvc/webapps/ROOT.war.bak
2021-12-21T10:39:34 INFO process_war: VULNERABLE FILE: /usr/lib/vmware-lookupsvc/webapps/ROOT.war backed up to /tmp/tmpxi89fco8/usr/lib/vmware-lookupsvc/webapps/ROOT.war.bak
2021-12-21T10:39:34 INFO process_jar: Found a VULNERABLE FILE: /usr/lib/vmware-lookupsvc/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar
2021-12-21T10:39:35 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-lookupsvc/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware-lookupsvc/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:35 INFO process_jar: VULNERABLE FILE: /usr/lib/vmware-lookupsvc/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar backed up to /tmp/tmpxi89fco8/usr/lib/vmware-lookupsvc/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar.bak
2021-12-21T10:39:37 INFO _patch_file: Found VULNERABLE FILE: /usr/lib/vmware-vmon/java-wrapper-vmon
2021-12-21T10:39:37 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-vmon/java-wrapper-vmon backed up to /tmp/tmpxi89fco8/usr/lib/vmware-vmon/java-wrapper-vmon.bak
2021-12-21T10:39:37 INFO patch_vum: Found a VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/start.ini
2021-12-21T10:39:37 INFO backup_file: VULNERABLE FILE: /usr/lib/vmware-updatemgr/bin/jetty/start.ini backed up to /tmp/tmpxi89fco8/usr/lib/vmware-updatemgr/bin/jetty/start.ini.bak
2021-12-21T10:39:37 INFO print_summary:
===== Summary =====
Backup Directory: /tmp/tmpxi89fco8
List of processed java archive files:
/opt/vmware/lib64/log4j-core-2.13.0.jar
/usr/lib/vmware/common-jars/log4j-core-2.13.1.jar
/usr/lib/vmware/common-jars/log4j-core-2.8.2.jar
/usr/lib/vmware/common-jars/log4j-core-2.11.0.jar
/usr/lib/vmware/common-jars/log4j-core-2.11.2.jar
/usr/lib/vmware/cis_upgrade_runner/payload/component-scripts/sso/lstool/lib/log4j-core-2.13.1.jar
/usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-fileupload.war
/usr/lib/vmware-updatemgr/bin/jetty/webapps/root.war
/usr/lib/vmware-updatemgr/bin/jetty/webapps/vum-filedownload.war
/usr/lib/vmware-sso/vmware-sts/webapps/ROOT.war
/usr/lib/vmware-sso/vmware-sts/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar
/usr/lib/vmware-dbcc/lib/log4j-core-2.8.2.jar
/usr/lib/vmware-lookupsvc/webapps/ROOT.war
/usr/lib/vmware-lookupsvc/webapps/ROOT/WEB-INF/lib/log4j-core-2.13.1.jar
List of processed configuration files:
/usr/lib/vmware-vmon/java-wrapper-vmon
/usr/lib/vmware-updatemgr/bin/jetty/start.ini
Total fixed: 16
NOTE: Running this script again with the --dryrun
flag should now yield 0 vulnerable files.
Log file: /var/log/vmsa-2021-0028_2021_12_21_10_38_20.log
===========================
2021-12-21T10:39:37 INFO start: starting services
2021-12-21T10:52:47 INFO main: Done.
Verify python vc_log4j_mitigator.py -r
root@localhost [ /tmp ]# python vc_log4j_mitigator.py -r
2021-12-21T11:10:01 INFO main: Script version: 1.6.0
2021-12-21T11:10:01 INFO main: vCenter type: Version: 7.0.2.00500; Build: 18455184; Deployment type: embedded; Gateway: False; VCHA: False; Windows: False;
2021-12-21T11:10:01 INFO main: Running in dryrun mode.
2021-12-21T11:11:01 INFO print_summary:
===== Summary =====
No vulnerable files found!
Total found: 0
Log file: /var/log/vmsa-2021-0028_2021_12_21_11_10_01.log
===========================
2021-12-21T11:11:01 INFO main: Done.
vc_log4j_mitigator.py [-h] – helps and more options
root@localhost [ /tmp ]# python vc_log4j_mitigator.py -h
usage: vc_log4j_mitigator.py [-h] [-d dirnames [dirnames ...]] [-a] [-r] [-b BACKUP_DIR] [-l LOG_DIR]
VMSA-2021-0028 vCenter tool; Version: 1.6.0 This tool deletes the JndiLookup.class file from *.jar and *.war files. On Windows systems the tool will by default traverse the folders identified by the VMWARE_CIS_HOME, VMWARE_CFG_DIR, VMWARE_DATA_DIR and VMWARE_RUNTIME_DATA_DIR
variables. On vCenter Appliances the tool will search by default from the root of the filesystem. All modified files are backed up if the process needs to be reversed due to an error.
optional arguments:
-h, --help show this help message and exit
-d dirnames [dirnames ...], --directories dirnames [dirnames ...]
space separated list of directories to check recursively for CVE-2021-44228 vulnerable java archive files.
-a, --accept-services-restart
accept the restart of the services without having manual prompt confirmation for the same
-r, --dryrun Run the script and log vulnerable files without mitigating them. The vCenter services are not restarted with this option.
-b BACKUP_DIR, --backup-dir BACKUP_DIR
Specify a backup directory to store original files.
-l LOG_DIR, --log-dir LOG_DIR
Specify a directory to store log files.
VAMI login fails with “Exception in invoking authentication handler User password expired”? The vCenter root password expired. Here is the 2-minute fix.
If logging in to the vCenter Server Appliance Management Interface (VAMI, https://<vcenter>:5480) fails with “Exception in invoking authentication handler User password expired”, the appliance’s root password has expired. Since vSphere 6.5 the root password expires after 90 days by default. SSH and the VM console still let you in, so the fix takes two minutes: log in to the shell, run passwd root, and then set an expiry policy that fits your environment.
The appliance was deployed more than 90 days ago with default settings.
VAMI login fails with Exception in invoking authentication handler User password expired.
Login to the appliance shell (SSH or VM console) still works, and may ask you to change the password immediately.
1. Check the password expiration
root@localhost [ ~ ]# chage -l root You are required to change your password immediately (password expired) chage: PAM: Authentication token is no longer valid; new one required
2. Change the root password
root@localhost [ ~ ]# passwd root New password: Retype new password: passwd: password updated successfully
VAMI login works again straight away.
3. Set an expiry policy
After logging in to VAMI, you can change the root password settings under Administration: the expiry period, “never expire”, and an email address for the expiration warning. Keep expiry enabled in production and set the warning email so this doesn’t catch you again.
For labs only, you can disable expiry from the shell:
root@localhost [ ~ ]# chage -M -1 root
Verify the result:
root@localhost [ ~ ]# chage -l root Last password change : Dec 21, 2021 Password expires : never Password inactive : never Account expires : never Minimum number of days between password change : 0 Maximum number of days between password change : -1 Number of days of warning before password expires : 7
vCenter Appliance root partition full? Find what fills / with df and du, clean audit.log, mail queue, UI cache and journald per Broadcom KBs, and know the resize limits.
Short answer: When the VCSA root partition (/) is full, find the culprit first: df -h and df -i to confirm the partition, then du to find the directory. The usual causes documented by Broadcom are a huge /var/log/audit/audit.log, a stuck mail queue in /var/spool/clientmqueue, the vSphere Client work cache, journald and old support bundles. Truncate or delete only what the KB says. Resizing the root disk is not supported on vCenter 7.0 and later.
Symptoms of a full VCSA root partition
VAMI (https://<vcenter>:5480) shows File system /root has run out of storage space, or vCenter raises a disk exhaustion alarm. Warnings start at 80%.
At 95%, vCenter stops vmware-vpxd to protect the database. The vSphere Client then returns 503 Service Unavailable, services do not start, and patching or upgrades fail.
Step 1: Confirm which partition is full
SSH to the appliance as root, type shell, and check both space and inodes. VAMI Monitor > Disks shows the same data. Output from one of my appliances:
root@vcsa [ ~ ]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 11G 8.2G 1.9G 82% /
root@vcsa [ ~ ]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda3 712704 97100 615604 14% /
# only partitions at 78% or more (Broadcom KB 318953)
root@vcsa [ ~ ]# df -h | awk '0+$5 >= 78 {print}'
Here the inodes are fine and the problem is space. Note which mount point is actually full. /storage/log, /storage/core, /storage/db and /storage/seat are separate virtual disks with their own KBs (see the table below). /storage/archive near 100% is expected by design on 6.7 and later. The root filesystem shows as /dev/sda3 on many builds. Broadcom KBs also show /dev/mapper/vg_root_0-lv_root_0 as the root device on some 7.0/8.0 systems.
If you cannot log in at all because / is full, Broadcom KB 318953 points to KB 322247 (steps 1 to 7) to get a single-user-mode shell.
Step 2: Find what fills the root partition
# largest areas on / (KB 422898)
du -sh /usr /var /opt /tmp /etc /home 2>/dev/null | sort -h
du -sh /var/log/* 2>/dev/null | sort -h | tail -10
journalctl --disk-usage
# top 20 entries under a path (KB 318953)
du -a /var | sort -n -r | head -n 20
# files larger than 100 MB in the current directory tree
find ./ -type f -size +100M
In my case, /var/spool was the problem:
root@vcsa [ /var ]# du -sh *
2.1G log
5.2G spool
Common culprits and the matching Broadcom KB
Where
Cause
Versions in KB
KB
/var/log/audit/audit.log
audit.log not rotated (often cron failing because the root password expired)
5.5 to 6.7
318911
/var/spool/clientmqueue
Mail queued to an unreachable SMTP relay; sendmail may also crash and fill /storage/core with core.sendmail.*
8.x
318475
/usr/lib/vmware-vsphere-ui/server/work
vSphere Client (Tomcat) work cache growth (normal about 100 MB to 1 GB, abnormal 10 GB+)
8.x
422898
/usr/lib/vmware-vsphere-ui/server/work/tmp/*.esa
Failed plug-in redeploys; fixed in vCenter 8.0.3 build 24262322
Files copied there by admins (ISOs, bundles, backups)
8.x
422898
Not the root partition? For /storage/log use KB 313077, and for /storage/core use KB 318750.
Step 3: Safe cleanup
Before you delete anything, take a backup or snapshot of the vCenter VM. With Enhanced Linked Mode, follow Broadcom’s snapshot best practices. vCenter HA must be removed first. For active log files, the KBs truncate the file instead of deleting it. Do the same.
audit.log
cd /var/log/audit
ls -lh
truncate -s 0 audit.log
# check that the daily cron actually runs
ls -l /var/spool/cron/lastrun/
# an expired root password breaks cron
chage -l root
If chage shows the root password as expired, change it, then restart services (service-control --stop --all and service-control --start --all). Otherwise the log fills up again.
clientmqueue (sendmail queue)
systemctl stop sendmail.service
find /var/spool/clientmqueue/* -exec rm {} \;
ls -ltrh /var/spool/clientmqueue/
# if sendmail was crashing, also remove its cores
rm -rf /storage/core/core.sendmail.*
# fix vCenter > Configure > General > Edit > Mail, then:
systemctl start sendmail.service
In my case, the space in /var/spool was this mail queue. KB 318475 gives the typical reason: vCenter’s mail settings point to an SMTP relay it cannot reach, so every alarm email stays queued. Fix the mail settings, or the queue grows again.
Clearing the work cache only restarts the vSphere Client (HTML5); running VMs, hosts and core vCenter services are not affected. The cache is regenerated when the service starts.
Can you resize the VCSA root partition?
Mostly not. Broadcom KB 316602 says:
vCenter version
Resize root disk (VMDK1)
6.5
Not supported
6.7
Added in 6.7
7.0 and later (incl. 8.0)
Not supported
In the vCenter 8.0 reference layout (Tiny, default storage), VMDK1 holds / (48 GB) and /boot (KB 316438). Check your real size with df -h, because Broadcom’s own 8.x KB examples also show much smaller root filesystems. On 7.0 and 8.0, cleanup is the only supported fix for /. KB 316602 even recommends journalctl --vacuum-time when an upgrade pre-check complains about free space on root.
The other disks (/storage/log, /storage/seat, /storage/db and so on) are LVM and can be grown online:
Take a file-based backup. Remove all snapshots from the vCenter VM, because a disk with snapshots cannot be extended.
Map the mount point to the virtual disk with df -h; lsblk; lsscsi (for example /storage/log is sde, SCSI(0:4)).
Increase that hard disk in the VM settings.
Run /usr/lib/applmgmt/support/scripts/autogrow.sh and confirm with df -h.
The KBs above list versions up to 8.0. For vCenter 9.0, check the current Broadcom KBs before you assume the same limits.
Prevent it from happening again
Monitor VCSA disks and act on the 80% alarm, not the 95% outage (KB 318930).
Keep the root password from expiring silently, so log rotation keeps working.
Keep the vCenter SMTP settings valid, delete support bundles after upload, and never use /root as a download folder.
Stay on current patches. Several root-partition bugs (e.g. the .esa files) are fixed in later builds.
