Short answer: On Cisco UCS M5 servers with ADDDC Sparing, the memory controller could hang after ADDDC activated virtual lockstep (VLS). The result was #IERR, an M2M (mesh-to-memory) timeout and a reboot. Intel documents the root cause as erratum SKX108 (CLX37 on Cascade Lake). Cisco fixed it in the BIOS: CSCvr79388 and CSCvr79396 in UCS Manager 4.1(1c), and CSCvt64871 for C480 M5 / C480 M5 ML in 4.1(1d). Run at least 4.1(1d). Field Notice FN-70432 recommends 4.1(3d) or later.
This page covers only the lockstep hang. For what ADDDC Sparing and PPR are, the BIOS tokens and FN-70432, see ADDDC Sparing and PPR on Cisco UCS M5.
ADDDC virtual lockstep hang: symptom
ADDDC Sparing corrects failing DRAM by pairing cache lines in virtual lockstep once correctable errors build up on a bank or rank. That is when this bug appears. A server that ran fine for months logs an ADDDC sparing event (UCS Manager fault F1706 or F1705, depending on firmware and CPU SKU). Later it stops responding and reboots with an internal error (#IERR) and an M2M timeout in the memory subsystem. For memory-heavy workloads such as SAP HANA on vSphere, this is exactly the outage ADDDC was supposed to prevent.
Cisco’s release notes point to the Intel specification updates: SKX108 in the Intel Xeon Processor Scalable Family Specification Update, and CLX37 in the 2nd Gen Intel Xeon Scalable Specification Update. The erratum title is “Memory Controller May Hang While in Virtual Lockstep”. Under complex microarchitectural conditions, a memory controller in VLS may hang on a partial write, and the BIOS can contain a workaround. Cisco’s fix is that BIOS workaround.
Cisco bugs and fixed releases
| Defect | Symptom (Cisco release notes) | First affected | Resolved in |
|---|---|---|---|
| CSCvr79388 | UCS servers stop responding and reboot after ADDDC virtual lockstep is activated. #IERR and M2M timeout in the memory system. | 4.0(4e)B | 4.1(1c)B |
| CSCvr79396 | On UCS M5, the VLS sparing copy finishes early, leaving incorrect values in the lockstep region. | 4.0(4e)B | 4.1(1c)B |
| CSCvt64871 | In rare cases, C480 M5 and C480 M5 ML stop responding and reboot after ADDDC virtual lockstep is activated (SKX108 / CLX37). | 4.0(4h)C | 4.1(1d)C |
The “B” and “C” suffixes are the UCS Manager bundle types (B-Series and C-Series server bundles). CSCvt64871 was still an open caveat in 4.1(1c). Cisco’s workaround until 4.1(1d): “If the server crashes many times after activating ADDDC virtual lockstep, disable ADDDC.”
Why I no longer recommend stopping at 4.1(1c)
When I first wrote this post in May 2020, 4.1(1c) was the fix. The UCS Manager 4.1 release notes later listed two regressions in the 4.1(1c) M5 BIOS, both resolved in 4.1(1d) (July 2020):
- CSCvu14656: after the BIOS upgrade to the 4.1(1c) versions (for example
B200M5.4.1.1c.0.0404202345,C240M5.4.1.1c.0.0405200025), M5 servers could hang at the memory testing step during boot. - CSCvu11155: possible performance degradation on B-, C- and S-Series M5 servers with 2nd Gen Xeon Scalable CPUs on the same BIOS versions.
FN-70432 also moved the minimum up. ADDDC support begins at 4.1(1d) for UCSM-managed servers (4.1(1g) for standalone). The recommended releases are 4.1(3d) or later for blades and integrated racks and 4.1(3c) or later for standalone M5 racks. So the practical rule is to treat 4.1(1c) as history. Run a current Cisco-recommended release for M5, and never lower than 4.1(3d)/4.1(3c).
What to check on your M5 servers
- Find the server firmware (host firmware package) version in UCS Manager or IMC. If it is older than 4.1(1c) and ADDDC is active, check it against CSCvr79388/CSCvr79396. The release notes list 4.0(4e) as first affected, so check the Bug Search Tool for fixed 4.0(4) builds.
- Look for past F1705/F1706 faults and unexpected reboots with #IERR or M2M timeout in the SEL. The hang can only happen after ADDDC has actually put a region into lockstep.
- Check the BIOS policy. Select Memory RAS configuration should be Platform Default or ADDDC Sparing. Do not switch ADDDC off as a permanent fix. That brings back the higher uncorrectable-error risk that FN-70432 describes.
- Upgrade the server firmware bundle. Put the ESXi host in maintenance mode first, because the BIOS update needs a reboot.
- After the upgrade, follow Cisco’s ADDDC event procedure. Reboot in a maintenance window so PPR can repair the DIMM, and replace the DIMM only if an uncorrectable error follows.
Sources
- Cisco – Release Notes for Cisco UCS Manager, Release 4.1 (Resolved Caveats in 4.1(1c) and 4.1(1d), Open Caveats in 4.1(1c))
- Cisco Field Notice FN-70432 – Improved Memory RAS Features for UCS M5 Platforms
- Cisco UCS HX M5 Memory Technical Overview – Memory RAS Features
- Intel Xeon Processor Scalable Family Specification Update (erratum SKX108) and 2nd Gen Intel Xeon Scalable Processors Specification Update (erratum CLX37), both referenced by Cisco in CSCvt64871. Search Intel’s Resource and Documentation Center for the current revision.