Tag: ADDDC

Adaptive Double DRAM Device Correction (ADDDC) or Adaptive Double Device Data Correction (ADDDC)

ADDDC Sparing—System reliability is optimized by holding memory in reserve so that it can be used in case other DIMMs fail. This mode provides some memory redundancy, but does not provide as much redundancy as mirroring.

Cisco UCS Manager Plugin for vSphere: Remote Plugin 4.0(0) and 4.0(1) for vCenter 8.0 U2, 8.0 U3 and 9.0

Cisco UCS Manager remote plugin for the vSphere HTML client: 4.0(0) vs 4.0(1), supported vCenter 7.0 U1 to 9.0, appliance sizing, install and Proactive HA.

Short answer: The Cisco UCS Manager plugin for vSphere is a remote plugin for the vSphere HTML client. It runs from its own OVA appliance. Version 4.0(0) was released in November 2023 and added support for vSphere 8.0 U2. The current release is 4.0(1). Cisco’s release notes list both versions for vSphere 7.0 U1 to U3, 8.0 GA to 8.0 U3, and 9.0, with UCS Manager 3.2 or later. The appliance needs 4 vCPU and 8 GB RAM.

I first wrote this post in November 2023 after rolling out 4.0(0) across about 13 UCS Manager domains and 411 ESXi 8.0 U2 hosts. This September 2026 update adds 4.0(1) and the current compatibility matrix from Cisco’s release notes, and keeps my hands-on notes.

Cisco UCS Manager plugin vSphere compatibility

vSphere HTML clientCisco UCS Manager plugin versions
7.03.0(4), 3.0(5), 3.0(6)
7.0 U1, 7.0 U23.0(5), 3.0(6), 4.0(0), 4.0(1)
7.0 U33.0(5), 3.0(6), 4.0(0), 4.0(1)
8.0, 8.0 U1, 8.0 U2, 8.0 U34.0(0), 4.0(1)
9.04.0(0), 4.0(1)

UCS Manager support is listed as “Release 3.2 and later”. Two known limitations are still open: CSCwi01415, where vCenter 7.0 GA is not supported (use 7.0 U1 or later), and CSCwi01422, where vSphere 8.0.1 does not show some appliance network fields (IP address, gateway, netmask).

Cisco UCS Manager remote plugin 4.0(0) vs 4.0(1)

ReleaseDateNew in this release
4.0(0)6 November 2023Plugin registration through the Cisco UCS Manager plugin appliance, registration of multiple vCenter Servers, support bundle logs (vCenter and appliance), adding and deleting custom faults, list of Proactive HA faults
4.0(1)Release notes created March 2024, last updated December 2025Plugin upgrade from the CLI, adding a certificate, changing the appliance IP and password, and actions removed for unmanaged servers. Fixes CSCwi01416, CSCwi01417, CSCwi01420, CSCwi01423, CSCwi03516 and CSCwj65650 (including FQDN registration and upper-case hostname issues).

If you are still on 4.0(0), move to 4.0(1). It fixes several registration bugs, and the CLI upgrade makes later updates easier.

What the plugin does in vCenter

  1. Physical hierarchy view of the UCS domain: chassis, blades, rack servers.
  2. Inventory: installed firmware, faults, and power and temperature statistics.
  3. Mapping of ESXi hosts to physical servers.
  4. Firmware management for B-Series and C-Series servers.
  5. Launch of the UCS Manager GUI and the KVM console from vCenter.
  6. Locator LED control.
  7. Cisco UCS provider for vSphere Proactive HA, with configurable faults.
Cisco UCS Manager plugin menu in the vSphere HTML client

Install the plugin and register vCenter

  1. Download the plugin OVA from Cisco Software Download (Cisco.com login required).
  2. Deploy the OVA with 4 vCPU and 8 GB RAM, on a 4 GB thin-provisioned or 100 GB thick-provisioned datastore.
  3. Log in to the appliance and register one or more vCenter Servers. The steps are in the chapter “Installing the Plug-in OVA and Registering the VMware vCenter Servers” of the user guide.
  4. In the vSphere client, open the plugin and register your UCS Manager domains.
  5. Optional: enable the Cisco UCS provider for Proactive HA on the cluster and choose which faults it should act on.
Cisco UCS Manager plugin login screen

Proactive HA experience from production

Proactive HA worked as intended with fault F1236. When it was raised, the plugin’s provider put the blade’s ESXi host into Quarantine mode automatically.

I had trouble with a custom fault, F1705: Proactive HA did not trigger on it. It was probably a configuration issue on my side, but I would like Cisco to include F1705 in the default fault list. I describe the setup in Add F1705 alert to Cisco UCS Manager plugin 4.0(0).

Sources

Related on vdan.cz

Add F1705 Alert to Cisco UCS Manager Plugin 4.0(0)

Add Cisco UCS fault F1705 (Rank VLS) as a custom fault in the UCS Manager Plugin 4.0(0) for the vSphere HTML client, so Proactive HA can react to it.

New Cisco UCS firmware brings possibility to have notification about F1705 Alerts – Rank VLS.

In latest version of Cisco UCS Manager Plugin for VMware vSphere HTML Client (Version 4.0(0)) we could add Custom fault addition for proactive HA monitoring. How to do it?

Cisco UCS / Proactive HA Registration / Registered Fault / Add / ADDDC_Memory_Rank_VLS

If You can’t Add, it is necessary to Unregister UCSM Manager Plugin.

Cisco UCS / Proactive HA Registration / Registered Fault / Add
Cisco UCS / Proactive HA Registration / vCenter server credentials / Register
Cisco UCS / Proactive HA Registration / Register
How Could I check it? Edit Proactive HA / Providers
It is better use Name “ADDDC_Memory_Rank_VLS” without spaces. On my picture I used “My F1705 Alerts”

Adding Custom Alert is only possible with unregistered Cisco UCS Provider, it is better to do it immediatly after Cisco UCS Manager Plugin instalation.

Now I can deceided If I will block F1705 or NOT. I personaly preffer to have F1705 Alert under Proactive HA. Then I only restart Blades with F1705. During reboot Hard-PPR permanently remaps accesses from a designated faulty row to a designated spare row.

Links:

Field Notice: FN – 72368 – Some DIMMs Might Fail Prematurely Due to a Manufacturing Deviation – Hardware Upgrade Available

Cisco Field Notice FN-72368: some DIMMs might fail prematurely due to a manufacturing deviation. What’s affected, and why ADDDC and PPR help.

Cisco announced Field Notice: FN – 72368 – Some DIMMs Might Fail Prematurely Due to a Manufacturing Deviation – Hardware Upgrade Available

My personal recommendation please use ADDDC and PPR – It could prevent hardware failures … UCS-ML-128G4RT-H is in 2nd revision from 28-Oct-22.

Problem Description

A limited number of DIMMs shipped from Cisco are impacted by a known deviation in the memory supplier’s manufacturing process. This deviation might result in a higher rate of failure.

Background

DIMM manufacturers compose their DIMMs of multiple memory modules to reach the desired capacity. A 16GB DIMM might be composed of the same modules that a 32GB DIMM is composed of. In this case, a manufacturing deviation in specific modules impacts 16GB, 32GB, 64GB, and 128GB DIMMs. This deviation was contained to a specific date range, and the DIMMs which use these chips were manufactured during the middle to end of 2020. Since the discovery of this deviation, additional limits have been imposed on the manufacturing process to ensure that future DIMMs are not exposed to this process variation.

Problem Symptom

Most DIMMs with this manufacturing deviation will exhibit persistent correctable memory errors. If left untreated, the DIMMs might eventually encounter an uncorrectable memory event. If encountered during runtime, uncorrectable errors will cause a sudden unexpected server reset. If encountered during Power-On Self-Test (POST), the DIMM will be mapped out and the total available memory reduced. In some cases a boot error might be seen.

Various DIMM Reliability, Availability, and Serviceability (RAS) features or even operating system features might mask the extent of these correctable errors. It is recommended to check your DIMMs for exposure using the Serial Number Validation Tool described in the Serial Number Validation section of this field notice. Only specific DIMMs are impacted by this issue, so do not rely solely on the DIMM error count to judge exposure.

Workaround/Solution

This is a hardware failure. A replacement is strongly recommended in order to avoid potential for unexpected server failure.

ADDDC Sparing and PPR on Cisco UCS M5: BIOS Tokens, Field Notice FN-70432 and Recommended Firmware

ADDDC Sparing and PPR on Cisco UCS M5: how they work, the Memory RAS and PPR Type BIOS tokens, FN-70432 firmware, and handling F1705/F1706 events.

Short answer: ADDDC Sparing (Adaptive Double Device Data Correction) is the Memory RAS mode that Cisco recommends, and uses as the default, on UCS M5 servers with 1st and 2nd Gen Intel Xeon Scalable CPUs. It watches for correctable errors and moves a failing region into virtual lockstep before the errors become uncorrectable. Post Package Repair (PPR) then repairs the DIMM permanently on the next reboot. Field Notice FN-70432 recommends server firmware 4.1(3d) or later, or 4.1(3c) for standalone C-Series.

Why FN-70432 exists

Intel changed how Single Device Data Correction (SDDC) works in the Xeon Scalable (“Skylake Server”) and 2nd Gen Xeon Scalable (“Cascade Lake Server”) processors. As a result, Cisco UCS M5 servers with DIMM errors can see a higher rate of runtime uncorrectable memory errors than older generations did with the default SDDC RAS mode. An uncorrectable error at runtime means a crash or hang and an outage. Cisco’s answer was new microcode plus BIOS changes that turn on ADDDC Sparing. Cisco also changed the default Memory RAS option to ADDDC Sparing (defect CSCvq38078). FN-70432 was last updated on 19 November 2021.

What ADDDC sparing does

  • ADDDC can correct two successive DRAM device failures in the same region. It tracks correctable errors and spare-copies the failing bits into a “buddy” cache line.
  • It uses virtual lockstep (VLS) within a memory channel, at DRAM bank level (bank VLS) or device level (rank VLS). Platinum and Gold CPUs support bank and rank VLS. Silver and Bronze CPUs support bank VLS only.
  • It does not reserve spare DIMM capacity and does not reduce the memory the OS sees.
  • Once a region is in lockstep, the server keeps running at slightly reduced memory performance until you can schedule maintenance.

Cisco’s measured overhead with ADDDC enabled, before any sparing event:

TestBandwidth change
Memory access, all reads-4.5%
3:1 reads-writes-1.3%
2:1 reads-writes-1.2%
1:1 reads-writes+1.7%
STREAM-triad-like-4.4%
SPECrate2017_int_base-2.3%

The penalty grows as more bank or rank regions are spared. For comparison, full memory mirroring halves usable capacity. Partial Mirror leaves the non-mirrored memory with baseline ECC only, with no ADDDC protection. Maximum Performance turns off all firmware RAS beyond ECC.

What PPR (Post Package Repair) adds

PPR permanently remaps a failing DRAM row to a spare row inside the DIMM. For a repair, the server must first have an ADDDC event and then go through at least one reboot. It has no effect on performance or capacity. Cisco analysed 273 field-returned DDR4 DIMMs and found that PPR repairs 70% of the faults that ADDDC detected. PPR does not start on a DIMM that has had an uncorrectable error. ADDDC Sparing must be enabled for PPR to work. The platform default PPR type is Hard PPR, and Cisco does not recommend disabling it.

BIOS tokens: Memory RAS configuration and PPR Type

From the Cisco UCS Server BIOS Tokens guide (UCS Manager 4.1, table “Server BIOS Tokens in Release 4.1(1a)”):

BIOS tokenDefaultValuesPlatforms
Select Memory RAS configurationADDDC SparingMirror Mode 1LM, Partial Mirror mode 1LM, Maximum Performance, ADDDC SparingB200 M5, B480 M5, C220 M5, C240 M5, C480 M5, S3260 M5
Select PPR TypeHard PPRHard PPR, DisabledB200 M5, B480 M5, C220 M5, C240 M5, C480 M5, S3260 M5
  • UCS Manager: if the BIOS policy’s Memory RAS configuration is Platform Default, nothing needs to change. If it is set to anything else, change it to ADDDC Sparing or Platform Default, or ADDDC will not be active. Check this before the firmware upgrade so you do not need an extra reboot.
  • Standalone C-Series (IMC): no change is needed.
  • Since 4.1(3), the BIOS policy also has Enhanced Memory Test (EMT). It adds boot time (about 8 minutes for 768 GB and 30 minutes for 3 TB, worst case) and repairs weak rows with PPR. Cisco recommends it for burn-in and after memory changes.

FN-70432 recommended firmware

PlatformFirmware that supports ADDDC SparingRecommended firmware
UCS M5 blades and UCSM-integrated M5 rack servers3.2(3p)+, 4.0(4i)+, 4.1(1d)+4.1(3d) or later
Standalone UCS M5 rack servers3.1(3k)+, 4.0(4l)+, 4.1(1g)+4.1(3c) or later

ADDDC Sparing first shipped in UCS Manager 4.0(4c) and 4.1(1a) and in IMC 4.0(4e) and 4.1(1c). Critical fixes came later, including the ADDDC virtual lockstep hang. As a minimum, Cisco recommends 4.1(1) for ADDDC event monitoring and PPR, and 4.1(3d) or later for ADDDC, PPR and Enhanced Memory Test together. For HyperFlex, FN-70432 points to the HyperFlex compatibility matrix and lists minimum HXDP releases. In October 2021 I recommended 4.2(1i), which already met the notice. Today, pick the current Cisco-recommended release on your train, not the FN minimum.

Handling ADDDC and PPR events (F1705 / F1706)

Bank-level and rank-level RAS events are non-critical. They mean a bad memory region was swapped out at runtime. Cisco’s procedure:

  1. Check that there are no other DIMM faults, such as an uncorrectable error.
  2. Schedule a maintenance window. Put the ESXi host in maintenance mode and reboot so PPR can repair the DIMM permanently.
  3. If there are no new errors, the repair worked. If new ADDDC events appear, reboot again for another PPR pass.
  4. If an uncorrectable error occurs after the reboot, replace the DIMM.
FirmwareCPU SKUBank eventRank event
4.0(4)Standard / AdvancedN/A / N/AN/A / F1706
4.1(1)AllF1706F1706
4.1(2), 4.1(3)Standard / AdvancedF1706 / F1705N/A / F1706
4.2(1)Standard / AdvancedF1705 / F1705N/A / F1706

The Cisco UCS Manager plugin for vSphere can pass these faults to vSphere Proactive HA, so a host can be evacuated before the DIMM fails. The related posts below cover the setup.

Sources

Related on vdan.cz

Fault Resilient Memory (FRM) for Cisco UCS

Use Fault Resilient Memory (reliable memory) on Cisco UCS with ESXi to protect against uncorrectable memory errors: how to enable it in UCS.

We can see annual incidence of uncorrectable errors is rissing. Here is one possibility – How to solved it with FRM.

ESXi supports reliable memory.

Some systems have reliable memory, which is a part of memory that is less likely to have hardware memory errors than other parts of the memory in the system. If the hardware exposes information about the different levels of reliability, ESXi might be able to achieve higher system reliability.

How to enable in Cisco UCS

Configuration is in BIOS policy / Advanced / RAS Memory

8GB Could be enough for ESXi hypervisor …

This forces the Hypervisor and some core kernel processes to be mirrored between DIMMs so ESXi itself can survive the complete and total failure of a memory DIMM.

# esxcli hardware memory get
    Physical Memory: 540800864256 Bytes
    Reliable Memory: 8589934592 Bytes
    NUMA Node Count: 2 
#  esxcli system settings kernel list | grep useReliableMem
 useReliableMem Bool TRUE TRUE TRUE System is aware of reliable memory.

Configuring Reliable Memory in Per-virtual machine basis (2146595)

I can decided to configure more Reliable Memory for VM – not only 8GB for hypervisor.

To turn on the feature per VM:

  1. Edit the .vmx file using a text editor
  2. Add the parameter:
    sched.mem.reliable = "True"
  3. Save and close the file

Conclusion:

  • For enable Fault Resilient Memory (FRM) I had to disable ADDDC Sparing in BIOS policy / Advanced / RAS Memory / Memory RAS configuration
  • With ADDDC and Proactive HA I can save about 95% failures – Personaly I prefer to use ADDDC
  • The Best possibility is to have both in future firmware …

Interesting links:

VMware vSphere Reliable Memory – A few thoughts

Field Notice: FN – 70432 – Improved Memory RAS Features for UCS M5 Platforms – Software Upgrade Recommended

Memory Errors and Dell EMC PowerEdge YX4X Server Memory RAS Features

ADDDC Virtual Lockstep Hang on Cisco UCS M5 (CSCvr79388, SKX108): Fixed in UCSM 4.1(1c), C480 M5 in 4.1(1d)

UCS M5 servers could hang and reboot (#IERR, M2M timeout) once ADDDC virtual lockstep kicked in. The Cisco bugs, fixed releases and what to run now.

Short answer: On Cisco UCS M5 servers with ADDDC Sparing, the memory controller could hang after ADDDC activated virtual lockstep (VLS). The result was #IERR, an M2M (mesh-to-memory) timeout and a reboot. Intel documents the root cause as erratum SKX108 (CLX37 on Cascade Lake). Cisco fixed it in the BIOS: CSCvr79388 and CSCvr79396 in UCS Manager 4.1(1c), and CSCvt64871 for C480 M5 / C480 M5 ML in 4.1(1d). Run at least 4.1(1d). Field Notice FN-70432 recommends 4.1(3d) or later.

This page covers only the lockstep hang. For what ADDDC Sparing and PPR are, the BIOS tokens and FN-70432, see ADDDC Sparing and PPR on Cisco UCS M5.

ADDDC virtual lockstep hang: symptom

ADDDC Sparing corrects failing DRAM by pairing cache lines in virtual lockstep once correctable errors build up on a bank or rank. That is when this bug appears. A server that ran fine for months logs an ADDDC sparing event (UCS Manager fault F1706 or F1705, depending on firmware and CPU SKU). Later it stops responding and reboots with an internal error (#IERR) and an M2M timeout in the memory subsystem. For memory-heavy workloads such as SAP HANA on vSphere, this is exactly the outage ADDDC was supposed to prevent.

Cisco’s release notes point to the Intel specification updates: SKX108 in the Intel Xeon Processor Scalable Family Specification Update, and CLX37 in the 2nd Gen Intel Xeon Scalable Specification Update. The erratum title is “Memory Controller May Hang While in Virtual Lockstep”. Under complex microarchitectural conditions, a memory controller in VLS may hang on a partial write, and the BIOS can contain a workaround. Cisco’s fix is that BIOS workaround.

Cisco bugs and fixed releases

DefectSymptom (Cisco release notes)First affectedResolved in
CSCvr79388UCS servers stop responding and reboot after ADDDC virtual lockstep is activated. #IERR and M2M timeout in the memory system.4.0(4e)B4.1(1c)B
CSCvr79396On UCS M5, the VLS sparing copy finishes early, leaving incorrect values in the lockstep region.4.0(4e)B4.1(1c)B
CSCvt64871In rare cases, C480 M5 and C480 M5 ML stop responding and reboot after ADDDC virtual lockstep is activated (SKX108 / CLX37).4.0(4h)C4.1(1d)C

The “B” and “C” suffixes are the UCS Manager bundle types (B-Series and C-Series server bundles). CSCvt64871 was still an open caveat in 4.1(1c). Cisco’s workaround until 4.1(1d): “If the server crashes many times after activating ADDDC virtual lockstep, disable ADDDC.”

Why I no longer recommend stopping at 4.1(1c)

When I first wrote this post in May 2020, 4.1(1c) was the fix. The UCS Manager 4.1 release notes later listed two regressions in the 4.1(1c) M5 BIOS, both resolved in 4.1(1d) (July 2020):

  • CSCvu14656: after the BIOS upgrade to the 4.1(1c) versions (for example B200M5.4.1.1c.0.0404202345, C240M5.4.1.1c.0.0405200025), M5 servers could hang at the memory testing step during boot.
  • CSCvu11155: possible performance degradation on B-, C- and S-Series M5 servers with 2nd Gen Xeon Scalable CPUs on the same BIOS versions.

FN-70432 also moved the minimum up. ADDDC support begins at 4.1(1d) for UCSM-managed servers (4.1(1g) for standalone). The recommended releases are 4.1(3d) or later for blades and integrated racks and 4.1(3c) or later for standalone M5 racks. So the practical rule is to treat 4.1(1c) as history. Run a current Cisco-recommended release for M5, and never lower than 4.1(3d)/4.1(3c).

What to check on your M5 servers

  1. Find the server firmware (host firmware package) version in UCS Manager or IMC. If it is older than 4.1(1c) and ADDDC is active, check it against CSCvr79388/CSCvr79396. The release notes list 4.0(4e) as first affected, so check the Bug Search Tool for fixed 4.0(4) builds.
  2. Look for past F1705/F1706 faults and unexpected reboots with #IERR or M2M timeout in the SEL. The hang can only happen after ADDDC has actually put a region into lockstep.
  3. Check the BIOS policy. Select Memory RAS configuration should be Platform Default or ADDDC Sparing. Do not switch ADDDC off as a permanent fix. That brings back the higher uncorrectable-error risk that FN-70432 describes.
  4. Upgrade the server firmware bundle. Put the ESXi host in maintenance mode first, because the BIOS update needs a reboot.
  5. After the upgrade, follow Cisco’s ADDDC event procedure. Reboot in a maintenance window so PPR can repair the DIMM, and replace the DIMM only if an uncorrectable error follows.

Sources

Related on vdan.cz

Proactive HA is working in VCSA 6.7 with Cisco UCS Manager Plugin for VMware vSphere HTML Client (beta Version 3.0(2))

Proactive HA working on vCenter 6.7 with the Cisco UCS Manager Plugin 3.0(2) beta for the vSphere HTML client, tested on 9 clusters and 290 hosts.

Cisco has released the 3.0(2) beta version of the the Cisco UCS Manager VMware vSphere HTML client plugin. These version is working with vSphere 6.7. It’s currently running and enabled on 9 different clusters – 290 hosts. It works great so far.

Here are the new and changed features in Release3.0(2):

  • Included defect fixes
  • Added a new fault (F1706)to the Cisco UCS Provider failure conditions list
  • Added support for proactive High Availability for more than 100 hosts in vCenter

It is great to combine it with new Cisco UCS 4.1.1 because of Intel Post Package Repair (PPR).

  • Intel Post Package Repair (PPR) uses additional spare capacity within the DDR4 DRAM to remap and replace faulty cell areas detected during system boot time. Remapping is permanent and persists through power-down and reboot.
  • Newer memories, such as double data ram version 4 (DDR4) include so-called post-package repair (PPR) capabilities. PPR capabilities enable a compatible memory controller to remap accesses from a faulty row of a memory module to a spare row of the memory module that is not faulty.
    • Hard-PPR permanently remaps accesses from a designated faulty row to a designated spare row. A Hard-PPR row remapping survives power cycles.
    • Soft-PPR remapping temporarily maps accesses from a faulty row to a designated spare row. A Soft-PPR row remapping will survive a “warm” reboot,but does not survive a powercycle.
  • You can enabled it in BIOS policy / Memory RAS configuration – Select PPR type configuration – Hard PPR

  • To support “Alert F1706 – ADDDC Memory RAS Problem” is necessary
    ADDDC Sparing—System reliability is optimized by holding memory in reserve so that it can be used in case other DIMMs fail. This mode provides some memory redundancy, but does not provide as much redundancy as mirroring.
  • Cisco recommends upgrading to 4.0(4c) or later to expand memory fault coverage. Beginning with 4.0(4c) an additional RAS feature, Adaptive Double Device Data Correction (ADDDC Sparing) is available. It will be enabled and configured as “Platform Default” for Memory RAS configuration.