VAMI login fails with “Exception in invoking authentication handler User password expired”? The vCenter root password expired. Here is the 2-minute fix.
If logging in to the vCenter Server Appliance Management Interface (VAMI, https://<vcenter>:5480) fails with “Exception in invoking authentication handler User password expired”, the appliance’s root password has expired. Since vSphere 6.5 the root password expires after 90 days by default. SSH and the VM console still let you in, so the fix takes two minutes: log in to the shell, run passwd root, and then set an expiry policy that fits your environment.
The appliance was deployed more than 90 days ago with default settings.
VAMI login fails with Exception in invoking authentication handler User password expired.
Login to the appliance shell (SSH or VM console) still works, and may ask you to change the password immediately.
1. Check the password expiration
root@localhost [ ~ ]# chage -l root You are required to change your password immediately (password expired) chage: PAM: Authentication token is no longer valid; new one required
2. Change the root password
root@localhost [ ~ ]# passwd root New password: Retype new password: passwd: password updated successfully
VAMI login works again straight away.
3. Set an expiry policy
After logging in to VAMI, you can change the root password settings under Administration: the expiry period, “never expire”, and an email address for the expiration warning. Keep expiry enabled in production and set the warning email so this doesn’t catch you again.
For labs only, you can disable expiry from the shell:
root@localhost [ ~ ]# chage -M -1 root
Verify the result:
root@localhost [ ~ ]# chage -l root Last password change : Dec 21, 2021 Password expires : never Password inactive : never Account expires : never Minimum number of days between password change : 0 Maximum number of days between password change : -1 Number of days of warning before password expires : 7
ADDDC Sparing and PPR on Cisco UCS M5: how they work, the Memory RAS and PPR Type BIOS tokens, FN-70432 firmware, and handling F1705/F1706 events.
Short answer: ADDDC Sparing (Adaptive Double Device Data Correction) is the Memory RAS mode that Cisco recommends, and uses as the default, on UCS M5 servers with 1st and 2nd Gen Intel Xeon Scalable CPUs. It watches for correctable errors and moves a failing region into virtual lockstep before the errors become uncorrectable. Post Package Repair (PPR) then repairs the DIMM permanently on the next reboot. Field Notice FN-70432 recommends server firmware 4.1(3d) or later, or 4.1(3c) for standalone C-Series.
Why FN-70432 exists
Intel changed how Single Device Data Correction (SDDC) works in the Xeon Scalable (“Skylake Server”) and 2nd Gen Xeon Scalable (“Cascade Lake Server”) processors. As a result, Cisco UCS M5 servers with DIMM errors can see a higher rate of runtime uncorrectable memory errors than older generations did with the default SDDC RAS mode. An uncorrectable error at runtime means a crash or hang and an outage. Cisco’s answer was new microcode plus BIOS changes that turn on ADDDC Sparing. Cisco also changed the default Memory RAS option to ADDDC Sparing (defect CSCvq38078). FN-70432 was last updated on 19 November 2021.
What ADDDC sparing does
ADDDC can correct two successive DRAM device failures in the same region. It tracks correctable errors and spare-copies the failing bits into a “buddy” cache line.
It uses virtual lockstep (VLS) within a memory channel, at DRAM bank level (bank VLS) or device level (rank VLS). Platinum and Gold CPUs support bank and rank VLS. Silver and Bronze CPUs support bank VLS only.
It does not reserve spare DIMM capacity and does not reduce the memory the OS sees.
Once a region is in lockstep, the server keeps running at slightly reduced memory performance until you can schedule maintenance.
Cisco’s measured overhead with ADDDC enabled, before any sparing event:
Test
Bandwidth change
Memory access, all reads
-4.5%
3:1 reads-writes
-1.3%
2:1 reads-writes
-1.2%
1:1 reads-writes
+1.7%
STREAM-triad-like
-4.4%
SPECrate2017_int_base
-2.3%
The penalty grows as more bank or rank regions are spared. For comparison, full memory mirroring halves usable capacity. Partial Mirror leaves the non-mirrored memory with baseline ECC only, with no ADDDC protection. Maximum Performance turns off all firmware RAS beyond ECC.
What PPR (Post Package Repair) adds
PPR permanently remaps a failing DRAM row to a spare row inside the DIMM. For a repair, the server must first have an ADDDC event and then go through at least one reboot. It has no effect on performance or capacity. Cisco analysed 273 field-returned DDR4 DIMMs and found that PPR repairs 70% of the faults that ADDDC detected. PPR does not start on a DIMM that has had an uncorrectable error. ADDDC Sparing must be enabled for PPR to work. The platform default PPR type is Hard PPR, and Cisco does not recommend disabling it.
BIOS tokens: Memory RAS configuration and PPR Type
From the Cisco UCS Server BIOS Tokens guide (UCS Manager 4.1, table “Server BIOS Tokens in Release 4.1(1a)”):
UCS Manager: if the BIOS policy’s Memory RAS configuration is Platform Default, nothing needs to change. If it is set to anything else, change it to ADDDC Sparing or Platform Default, or ADDDC will not be active. Check this before the firmware upgrade so you do not need an extra reboot.
Standalone C-Series (IMC): no change is needed.
Since 4.1(3), the BIOS policy also has Enhanced Memory Test (EMT). It adds boot time (about 8 minutes for 768 GB and 30 minutes for 3 TB, worst case) and repairs weak rows with PPR. Cisco recommends it for burn-in and after memory changes.
FN-70432 recommended firmware
Platform
Firmware that supports ADDDC Sparing
Recommended firmware
UCS M5 blades and UCSM-integrated M5 rack servers
3.2(3p)+, 4.0(4i)+, 4.1(1d)+
4.1(3d) or later
Standalone UCS M5 rack servers
3.1(3k)+, 4.0(4l)+, 4.1(1g)+
4.1(3c) or later
ADDDC Sparing first shipped in UCS Manager 4.0(4c) and 4.1(1a) and in IMC 4.0(4e) and 4.1(1c). Critical fixes came later, including the ADDDC virtual lockstep hang. As a minimum, Cisco recommends 4.1(1) for ADDDC event monitoring and PPR, and 4.1(3d) or later for ADDDC, PPR and Enhanced Memory Test together. For HyperFlex, FN-70432 points to the HyperFlex compatibility matrix and lists minimum HXDP releases. In October 2021 I recommended 4.2(1i), which already met the notice. Today, pick the current Cisco-recommended release on your train, not the FN minimum.
Handling ADDDC and PPR events (F1705 / F1706)
Bank-level and rank-level RAS events are non-critical. They mean a bad memory region was swapped out at runtime. Cisco’s procedure:
Check that there are no other DIMM faults, such as an uncorrectable error.
Schedule a maintenance window. Put the ESXi host in maintenance mode and reboot so PPR can repair the DIMM permanently.
If there are no new errors, the repair worked. If new ADDDC events appear, reboot again for another PPR pass.
If an uncorrectable error occurs after the reboot, replace the DIMM.
Firmware
CPU SKU
Bank event
Rank event
4.0(4)
Standard / Advanced
N/A / N/A
N/A / F1706
4.1(1)
All
F1706
F1706
4.1(2), 4.1(3)
Standard / Advanced
F1706 / F1705
N/A / F1706
4.2(1)
Standard / Advanced
F1705 / F1705
N/A / F1706
The Cisco UCS Manager plugin for vSphere can pass these faults to vSphere Proactive HA, so a host can be evacuated before the DIMM fails. The related posts below cover the setup.
Today, VMware announced Project Capitola as a technology preview. Project Capitola is a software-defined memory implementation that will aggregate tiers of different memory types such as DRAM, PMEM, NVMe and […]
This week at VMworld 2021 VMware announced Project Arctic and Project Cascade. Together these projects will address the needs of two distinct groups in a multi-cloud world: Project Arctic will address the needs of IT operators who need to deliver enterprise IT resources, and Project Cascade […]
Change is inevitable, it’s the only constant we have in life. Growth is optional. It is not necessary to change. Survival is not mandatory~ W Edwards Demming. To be able to handle change is the great differentiator. A company/person with the agility to see the change coming can adapt the quickest, one that has the change impact thrust upon it must respond, and that which can’t adapt will not survive. Change happens gradually, gradually – then suddenly. There is always a catalyst for change. Sometimes it’s a nice to have business need, other times it’s a hurricane that wipes out your entire business. The catalyst only determines the speed. Preparedness is essential in either situation. In this series, Time for Change, Kit Colbert talks…Read More
Join us online for VMworld 2021, the industry’s premier multi-cloud event. Where the people & organizations creating the digital foundation for technology and business transformation gather.
Join us online for VMworld 2021, the industry’s premier multi-cloud event. Where the people & organizations creating the digital foundation for technology and business transformation gather.
✔️Corrected vSphere 7.0 Update 3 Release Note URLs 🔸vCenter:https://t.co/pQxSj5ltxk 🔹ESXihttps://t.co/qrKrtmo87c https://t.co/oYMMWVWmWy — William Lam (@lamw) October 5, 2021 vSphere 7.0 Update 3 officially GA’ed this morning and I am happy to share a quick that my Nested ESXi Virtual Appliance (OVA and Content Library) has also been updated to support the latest release. In addition, […]
Today is a big day for the Tanzu team at VMware. It’s a day we’ve long been looking forward to, and a day that we’d like to celebrate with you. Just this morning we released VMware Tanzu Community Edition, a freely available, community-supported, open source distribution of VMware Tanzu that […]
