Short answer: ADDDC Sparing (Adaptive Double Device Data Correction) is the Memory RAS mode that Cisco recommends, and uses as the default, on UCS M5 servers with 1st and 2nd Gen Intel Xeon Scalable CPUs. It watches for correctable errors and moves a failing region into virtual lockstep before the errors become uncorrectable. Post Package Repair (PPR) then repairs the DIMM permanently on the next reboot. Field Notice FN-70432 recommends server firmware 4.1(3d) or later, or 4.1(3c) for standalone C-Series.
Why FN-70432 exists
Intel changed how Single Device Data Correction (SDDC) works in the Xeon Scalable (“Skylake Server”) and 2nd Gen Xeon Scalable (“Cascade Lake Server”) processors. As a result, Cisco UCS M5 servers with DIMM errors can see a higher rate of runtime uncorrectable memory errors than older generations did with the default SDDC RAS mode. An uncorrectable error at runtime means a crash or hang and an outage. Cisco’s answer was new microcode plus BIOS changes that turn on ADDDC Sparing. Cisco also changed the default Memory RAS option to ADDDC Sparing (defect CSCvq38078). FN-70432 was last updated on 19 November 2021.
What ADDDC sparing does
- ADDDC can correct two successive DRAM device failures in the same region. It tracks correctable errors and spare-copies the failing bits into a “buddy” cache line.
- It uses virtual lockstep (VLS) within a memory channel, at DRAM bank level (bank VLS) or device level (rank VLS). Platinum and Gold CPUs support bank and rank VLS. Silver and Bronze CPUs support bank VLS only.
- It does not reserve spare DIMM capacity and does not reduce the memory the OS sees.
- Once a region is in lockstep, the server keeps running at slightly reduced memory performance until you can schedule maintenance.
Cisco’s measured overhead with ADDDC enabled, before any sparing event:
| Test | Bandwidth change |
|---|---|
| Memory access, all reads | -4.5% |
| 3:1 reads-writes | -1.3% |
| 2:1 reads-writes | -1.2% |
| 1:1 reads-writes | +1.7% |
| STREAM-triad-like | -4.4% |
| SPECrate2017_int_base | -2.3% |
The penalty grows as more bank or rank regions are spared. For comparison, full memory mirroring halves usable capacity. Partial Mirror leaves the non-mirrored memory with baseline ECC only, with no ADDDC protection. Maximum Performance turns off all firmware RAS beyond ECC.
What PPR (Post Package Repair) adds
PPR permanently remaps a failing DRAM row to a spare row inside the DIMM. For a repair, the server must first have an ADDDC event and then go through at least one reboot. It has no effect on performance or capacity. Cisco analysed 273 field-returned DDR4 DIMMs and found that PPR repairs 70% of the faults that ADDDC detected. PPR does not start on a DIMM that has had an uncorrectable error. ADDDC Sparing must be enabled for PPR to work. The platform default PPR type is Hard PPR, and Cisco does not recommend disabling it.
BIOS tokens: Memory RAS configuration and PPR Type
From the Cisco UCS Server BIOS Tokens guide (UCS Manager 4.1, table “Server BIOS Tokens in Release 4.1(1a)”):
| BIOS token | Default | Values | Platforms |
|---|---|---|---|
| Select Memory RAS configuration | ADDDC Sparing | Mirror Mode 1LM, Partial Mirror mode 1LM, Maximum Performance, ADDDC Sparing | B200 M5, B480 M5, C220 M5, C240 M5, C480 M5, S3260 M5 |
| Select PPR Type | Hard PPR | Hard PPR, Disabled | B200 M5, B480 M5, C220 M5, C240 M5, C480 M5, S3260 M5 |
- UCS Manager: if the BIOS policy’s Memory RAS configuration is Platform Default, nothing needs to change. If it is set to anything else, change it to ADDDC Sparing or Platform Default, or ADDDC will not be active. Check this before the firmware upgrade so you do not need an extra reboot.
- Standalone C-Series (IMC): no change is needed.
- Since 4.1(3), the BIOS policy also has Enhanced Memory Test (EMT). It adds boot time (about 8 minutes for 768 GB and 30 minutes for 3 TB, worst case) and repairs weak rows with PPR. Cisco recommends it for burn-in and after memory changes.
FN-70432 recommended firmware
| Platform | Firmware that supports ADDDC Sparing | Recommended firmware |
|---|---|---|
| UCS M5 blades and UCSM-integrated M5 rack servers | 3.2(3p)+, 4.0(4i)+, 4.1(1d)+ | 4.1(3d) or later |
| Standalone UCS M5 rack servers | 3.1(3k)+, 4.0(4l)+, 4.1(1g)+ | 4.1(3c) or later |
ADDDC Sparing first shipped in UCS Manager 4.0(4c) and 4.1(1a) and in IMC 4.0(4e) and 4.1(1c). Critical fixes came later, including the ADDDC virtual lockstep hang. As a minimum, Cisco recommends 4.1(1) for ADDDC event monitoring and PPR, and 4.1(3d) or later for ADDDC, PPR and Enhanced Memory Test together. For HyperFlex, FN-70432 points to the HyperFlex compatibility matrix and lists minimum HXDP releases. In October 2021 I recommended 4.2(1i), which already met the notice. Today, pick the current Cisco-recommended release on your train, not the FN minimum.
Handling ADDDC and PPR events (F1705 / F1706)
Bank-level and rank-level RAS events are non-critical. They mean a bad memory region was swapped out at runtime. Cisco’s procedure:
- Check that there are no other DIMM faults, such as an uncorrectable error.
- Schedule a maintenance window. Put the ESXi host in maintenance mode and reboot so PPR can repair the DIMM permanently.
- If there are no new errors, the repair worked. If new ADDDC events appear, reboot again for another PPR pass.
- If an uncorrectable error occurs after the reboot, replace the DIMM.
| Firmware | CPU SKU | Bank event | Rank event |
|---|---|---|---|
| 4.0(4) | Standard / Advanced | N/A / N/A | N/A / F1706 |
| 4.1(1) | All | F1706 | F1706 |
| 4.1(2), 4.1(3) | Standard / Advanced | F1706 / F1705 | N/A / F1706 |
| 4.2(1) | Standard / Advanced | F1705 / F1705 | N/A / F1706 |
The Cisco UCS Manager plugin for vSphere can pass these faults to vSphere Proactive HA, so a host can be evacuated before the DIMM fails. The related posts below cover the setup.
Sources
- Cisco Field Notice FN-70432 – Improved Memory RAS Features for UCS M5 Platforms – Software Upgrade Recommended
- Cisco UCS HX M5 Memory Technical Overview – Memory RAS Features
- Cisco UCS Server BIOS Tokens, Release 4.1
