Tag: Proactive HA

Cisco UCS Manager Plugin for vSphere: Remote Plugin 4.0(0) and 4.0(1) for vCenter 8.0 U2, 8.0 U3 and 9.0

Cisco UCS Manager remote plugin for the vSphere HTML client: 4.0(0) vs 4.0(1), supported vCenter 7.0 U1 to 9.0, appliance sizing, install and Proactive HA.

Short answer: The Cisco UCS Manager plugin for vSphere is a remote plugin for the vSphere HTML client. It runs from its own OVA appliance. Version 4.0(0) was released in November 2023 and added support for vSphere 8.0 U2. The current release is 4.0(1). Cisco’s release notes list both versions for vSphere 7.0 U1 to U3, 8.0 GA to 8.0 U3, and 9.0, with UCS Manager 3.2 or later. The appliance needs 4 vCPU and 8 GB RAM.

I first wrote this post in November 2023 after rolling out 4.0(0) across about 13 UCS Manager domains and 411 ESXi 8.0 U2 hosts. This September 2026 update adds 4.0(1) and the current compatibility matrix from Cisco’s release notes, and keeps my hands-on notes.

Cisco UCS Manager plugin vSphere compatibility

vSphere HTML clientCisco UCS Manager plugin versions
7.03.0(4), 3.0(5), 3.0(6)
7.0 U1, 7.0 U23.0(5), 3.0(6), 4.0(0), 4.0(1)
7.0 U33.0(5), 3.0(6), 4.0(0), 4.0(1)
8.0, 8.0 U1, 8.0 U2, 8.0 U34.0(0), 4.0(1)
9.04.0(0), 4.0(1)

UCS Manager support is listed as “Release 3.2 and later”. Two known limitations are still open: CSCwi01415, where vCenter 7.0 GA is not supported (use 7.0 U1 or later), and CSCwi01422, where vSphere 8.0.1 does not show some appliance network fields (IP address, gateway, netmask).

Cisco UCS Manager remote plugin 4.0(0) vs 4.0(1)

ReleaseDateNew in this release
4.0(0)6 November 2023Plugin registration through the Cisco UCS Manager plugin appliance, registration of multiple vCenter Servers, support bundle logs (vCenter and appliance), adding and deleting custom faults, list of Proactive HA faults
4.0(1)Release notes created March 2024, last updated December 2025Plugin upgrade from the CLI, adding a certificate, changing the appliance IP and password, and actions removed for unmanaged servers. Fixes CSCwi01416, CSCwi01417, CSCwi01420, CSCwi01423, CSCwi03516 and CSCwj65650 (including FQDN registration and upper-case hostname issues).

If you are still on 4.0(0), move to 4.0(1). It fixes several registration bugs, and the CLI upgrade makes later updates easier.

What the plugin does in vCenter

  1. Physical hierarchy view of the UCS domain: chassis, blades, rack servers.
  2. Inventory: installed firmware, faults, and power and temperature statistics.
  3. Mapping of ESXi hosts to physical servers.
  4. Firmware management for B-Series and C-Series servers.
  5. Launch of the UCS Manager GUI and the KVM console from vCenter.
  6. Locator LED control.
  7. Cisco UCS provider for vSphere Proactive HA, with configurable faults.
Cisco UCS Manager plugin menu in the vSphere HTML client

Install the plugin and register vCenter

  1. Download the plugin OVA from Cisco Software Download (Cisco.com login required).
  2. Deploy the OVA with 4 vCPU and 8 GB RAM, on a 4 GB thin-provisioned or 100 GB thick-provisioned datastore.
  3. Log in to the appliance and register one or more vCenter Servers. The steps are in the chapter “Installing the Plug-in OVA and Registering the VMware vCenter Servers” of the user guide.
  4. In the vSphere client, open the plugin and register your UCS Manager domains.
  5. Optional: enable the Cisco UCS provider for Proactive HA on the cluster and choose which faults it should act on.
Cisco UCS Manager plugin login screen

Proactive HA experience from production

Proactive HA worked as intended with fault F1236. When it was raised, the plugin’s provider put the blade’s ESXi host into Quarantine mode automatically.

I had trouble with a custom fault, F1705: Proactive HA did not trigger on it. It was probably a configuration issue on my side, but I would like Cisco to include F1705 in the default fault list. I describe the setup in Add F1705 alert to Cisco UCS Manager plugin 4.0(0).

Sources

Related on vdan.cz

Add F1705 Alert to Cisco UCS Manager Plugin 4.0(0)

Add Cisco UCS fault F1705 (Rank VLS) as a custom fault in the UCS Manager Plugin 4.0(0) for the vSphere HTML client, so Proactive HA can react to it.

New Cisco UCS firmware brings possibility to have notification about F1705 Alerts – Rank VLS.

In latest version of Cisco UCS Manager Plugin for VMware vSphere HTML Client (Version 4.0(0)) we could add Custom fault addition for proactive HA monitoring. How to do it?

Cisco UCS / Proactive HA Registration / Registered Fault / Add / ADDDC_Memory_Rank_VLS

If You can’t Add, it is necessary to Unregister UCSM Manager Plugin.

Cisco UCS / Proactive HA Registration / Registered Fault / Add
Cisco UCS / Proactive HA Registration / vCenter server credentials / Register
Cisco UCS / Proactive HA Registration / Register
How Could I check it? Edit Proactive HA / Providers
It is better use Name “ADDDC_Memory_Rank_VLS” without spaces. On my picture I used “My F1705 Alerts”

Adding Custom Alert is only possible with unregistered Cisco UCS Provider, it is better to do it immediatly after Cisco UCS Manager Plugin instalation.

Now I can deceided If I will block F1705 or NOT. I personaly preffer to have F1705 Alert under Proactive HA. Then I only restart Blades with F1705. During reboot Hard-PPR permanently remaps accesses from a designated faulty row to a designated spare row.

Links:

Field Notice: FN – 72368 – Some DIMMs Might Fail Prematurely Due to a Manufacturing Deviation – Hardware Upgrade Available

Cisco Field Notice FN-72368: some DIMMs might fail prematurely due to a manufacturing deviation. What’s affected, and why ADDDC and PPR help.

Cisco announced Field Notice: FN – 72368 – Some DIMMs Might Fail Prematurely Due to a Manufacturing Deviation – Hardware Upgrade Available

My personal recommendation please use ADDDC and PPR – It could prevent hardware failures … UCS-ML-128G4RT-H is in 2nd revision from 28-Oct-22.

Problem Description

A limited number of DIMMs shipped from Cisco are impacted by a known deviation in the memory supplier’s manufacturing process. This deviation might result in a higher rate of failure.

Background

DIMM manufacturers compose their DIMMs of multiple memory modules to reach the desired capacity. A 16GB DIMM might be composed of the same modules that a 32GB DIMM is composed of. In this case, a manufacturing deviation in specific modules impacts 16GB, 32GB, 64GB, and 128GB DIMMs. This deviation was contained to a specific date range, and the DIMMs which use these chips were manufactured during the middle to end of 2020. Since the discovery of this deviation, additional limits have been imposed on the manufacturing process to ensure that future DIMMs are not exposed to this process variation.

Problem Symptom

Most DIMMs with this manufacturing deviation will exhibit persistent correctable memory errors. If left untreated, the DIMMs might eventually encounter an uncorrectable memory event. If encountered during runtime, uncorrectable errors will cause a sudden unexpected server reset. If encountered during Power-On Self-Test (POST), the DIMM will be mapped out and the total available memory reduced. In some cases a boot error might be seen.

Various DIMM Reliability, Availability, and Serviceability (RAS) features or even operating system features might mask the extent of these correctable errors. It is recommended to check your DIMMs for exposure using the Serial Number Validation Tool described in the Serial Number Validation section of this field notice. Only specific DIMMs are impacted by this issue, so do not rely solely on the DIMM error count to judge exposure.

Workaround/Solution

This is a hardware failure. A replacement is strongly recommended in order to avoid potential for unexpected server failure.

Cisco UCS Manager Plugin for VMware vSphere HTML Client (Version 3.0(6))

What’s in version 3.0(6) of the Cisco UCS Manager plugin for the vSphere HTML client: view, manage and monitor UCS hardware from vCenter.

Cisco has released the 3.0(6) version of the Cisco UCS Manager VMware vSphere HTML client plugin. The UCS Manager vSphere HTML client plugin enables a virtualization administrator to view, manage, and monitor the Cisco UCS physical infrastructure. The plugin provides a physical view of the UCS hardware inventory on the HTML client.

I notify BUG “Host not going into monitoring state vCenter restart”. Thank You for fix.

Release 3.0(6)

Here are the new features in Release 3.0(6):

  • Custom fault addition for proactive HA monitoring
  • Resolved host not going into monitoring state vCenter restart
  • Included defect fixes

VMware vSphere HTML Client Releases

Cisco UCS Manager plug-in is compatible with the following vSphere HTML Client releases:

VMware vSphere HTML Client VersionCisco UCS Manager Plugin for VMware vSphere Version
6.73.0(1), 3.0(2), 3.0(3), 3.0(4), 3.0(5), 3.0(6)
7.03.0(4), 3.0(5), 3.0(6)
7.0u1, 7.0u23.0(5), 3.0(6)

Note		VMware vSphere HTML Client Version 7.0u3 is not supported.
More info here.

ADDDC Sparing and PPR on Cisco UCS M5: BIOS Tokens, Field Notice FN-70432 and Recommended Firmware

ADDDC Sparing and PPR on Cisco UCS M5: how they work, the Memory RAS and PPR Type BIOS tokens, FN-70432 firmware, and handling F1705/F1706 events.

Short answer: ADDDC Sparing (Adaptive Double Device Data Correction) is the Memory RAS mode that Cisco recommends, and uses as the default, on UCS M5 servers with 1st and 2nd Gen Intel Xeon Scalable CPUs. It watches for correctable errors and moves a failing region into virtual lockstep before the errors become uncorrectable. Post Package Repair (PPR) then repairs the DIMM permanently on the next reboot. Field Notice FN-70432 recommends server firmware 4.1(3d) or later, or 4.1(3c) for standalone C-Series.

Why FN-70432 exists

Intel changed how Single Device Data Correction (SDDC) works in the Xeon Scalable (“Skylake Server”) and 2nd Gen Xeon Scalable (“Cascade Lake Server”) processors. As a result, Cisco UCS M5 servers with DIMM errors can see a higher rate of runtime uncorrectable memory errors than older generations did with the default SDDC RAS mode. An uncorrectable error at runtime means a crash or hang and an outage. Cisco’s answer was new microcode plus BIOS changes that turn on ADDDC Sparing. Cisco also changed the default Memory RAS option to ADDDC Sparing (defect CSCvq38078). FN-70432 was last updated on 19 November 2021.

What ADDDC sparing does

  • ADDDC can correct two successive DRAM device failures in the same region. It tracks correctable errors and spare-copies the failing bits into a “buddy” cache line.
  • It uses virtual lockstep (VLS) within a memory channel, at DRAM bank level (bank VLS) or device level (rank VLS). Platinum and Gold CPUs support bank and rank VLS. Silver and Bronze CPUs support bank VLS only.
  • It does not reserve spare DIMM capacity and does not reduce the memory the OS sees.
  • Once a region is in lockstep, the server keeps running at slightly reduced memory performance until you can schedule maintenance.

Cisco’s measured overhead with ADDDC enabled, before any sparing event:

TestBandwidth change
Memory access, all reads-4.5%
3:1 reads-writes-1.3%
2:1 reads-writes-1.2%
1:1 reads-writes+1.7%
STREAM-triad-like-4.4%
SPECrate2017_int_base-2.3%

The penalty grows as more bank or rank regions are spared. For comparison, full memory mirroring halves usable capacity. Partial Mirror leaves the non-mirrored memory with baseline ECC only, with no ADDDC protection. Maximum Performance turns off all firmware RAS beyond ECC.

What PPR (Post Package Repair) adds

PPR permanently remaps a failing DRAM row to a spare row inside the DIMM. For a repair, the server must first have an ADDDC event and then go through at least one reboot. It has no effect on performance or capacity. Cisco analysed 273 field-returned DDR4 DIMMs and found that PPR repairs 70% of the faults that ADDDC detected. PPR does not start on a DIMM that has had an uncorrectable error. ADDDC Sparing must be enabled for PPR to work. The platform default PPR type is Hard PPR, and Cisco does not recommend disabling it.

BIOS tokens: Memory RAS configuration and PPR Type

From the Cisco UCS Server BIOS Tokens guide (UCS Manager 4.1, table “Server BIOS Tokens in Release 4.1(1a)”):

BIOS tokenDefaultValuesPlatforms
Select Memory RAS configurationADDDC SparingMirror Mode 1LM, Partial Mirror mode 1LM, Maximum Performance, ADDDC SparingB200 M5, B480 M5, C220 M5, C240 M5, C480 M5, S3260 M5
Select PPR TypeHard PPRHard PPR, DisabledB200 M5, B480 M5, C220 M5, C240 M5, C480 M5, S3260 M5
  • UCS Manager: if the BIOS policy’s Memory RAS configuration is Platform Default, nothing needs to change. If it is set to anything else, change it to ADDDC Sparing or Platform Default, or ADDDC will not be active. Check this before the firmware upgrade so you do not need an extra reboot.
  • Standalone C-Series (IMC): no change is needed.
  • Since 4.1(3), the BIOS policy also has Enhanced Memory Test (EMT). It adds boot time (about 8 minutes for 768 GB and 30 minutes for 3 TB, worst case) and repairs weak rows with PPR. Cisco recommends it for burn-in and after memory changes.

FN-70432 recommended firmware

PlatformFirmware that supports ADDDC SparingRecommended firmware
UCS M5 blades and UCSM-integrated M5 rack servers3.2(3p)+, 4.0(4i)+, 4.1(1d)+4.1(3d) or later
Standalone UCS M5 rack servers3.1(3k)+, 4.0(4l)+, 4.1(1g)+4.1(3c) or later

ADDDC Sparing first shipped in UCS Manager 4.0(4c) and 4.1(1a) and in IMC 4.0(4e) and 4.1(1c). Critical fixes came later, including the ADDDC virtual lockstep hang. As a minimum, Cisco recommends 4.1(1) for ADDDC event monitoring and PPR, and 4.1(3d) or later for ADDDC, PPR and Enhanced Memory Test together. For HyperFlex, FN-70432 points to the HyperFlex compatibility matrix and lists minimum HXDP releases. In October 2021 I recommended 4.2(1i), which already met the notice. Today, pick the current Cisco-recommended release on your train, not the FN minimum.

Handling ADDDC and PPR events (F1705 / F1706)

Bank-level and rank-level RAS events are non-critical. They mean a bad memory region was swapped out at runtime. Cisco’s procedure:

  1. Check that there are no other DIMM faults, such as an uncorrectable error.
  2. Schedule a maintenance window. Put the ESXi host in maintenance mode and reboot so PPR can repair the DIMM permanently.
  3. If there are no new errors, the repair worked. If new ADDDC events appear, reboot again for another PPR pass.
  4. If an uncorrectable error occurs after the reboot, replace the DIMM.
FirmwareCPU SKUBank eventRank event
4.0(4)Standard / AdvancedN/A / N/AN/A / F1706
4.1(1)AllF1706F1706
4.1(2), 4.1(3)Standard / AdvancedF1706 / F1705N/A / F1706
4.2(1)Standard / AdvancedF1705 / F1705N/A / F1706

The Cisco UCS Manager plugin for vSphere can pass these faults to vSphere Proactive HA, so a host can be evacuated before the DIMM fails. The related posts below cover the setup.

Sources

Related on vdan.cz

Cisco UCS Manager Plugin for VMware vSphere HTML Client (Version 3.0(4))

What’s in version 3.0(4) of the Cisco UCS Manager plugin for the vSphere HTML client: view, manage and monitor Cisco UCS hardware from vCenter.

Cisco has released the 3.0(4) version of the Cisco UCS Manager VMware vSphere HTML client plugin. The UCS Manager vSphere HTML client plugin enables a virtualization administrator to view, manage, and monitor the Cisco UCS physical infrastructure. The plugin provides a physical view of the UCS hardware inventory on the HTML client.

Here are the new features in Release 3.0(4):

  • Support for VMware vSphere HTML Client 7.0
  • Included defect fix

Resolved Bugs

Defect IDSymptomFirst Affected in ReleaseResolved in Release
CSCvv01618UCSM domain not visible in HTML plug-in 3.0(2).3.0(2)3.0(4)
More info here.

ADDDC Virtual Lockstep Hang on Cisco UCS M5 (CSCvr79388, SKX108): Fixed in UCSM 4.1(1c), C480 M5 in 4.1(1d)

UCS M5 servers could hang and reboot (#IERR, M2M timeout) once ADDDC virtual lockstep kicked in. The Cisco bugs, fixed releases and what to run now.

Short answer: On Cisco UCS M5 servers with ADDDC Sparing, the memory controller could hang after ADDDC activated virtual lockstep (VLS). The result was #IERR, an M2M (mesh-to-memory) timeout and a reboot. Intel documents the root cause as erratum SKX108 (CLX37 on Cascade Lake). Cisco fixed it in the BIOS: CSCvr79388 and CSCvr79396 in UCS Manager 4.1(1c), and CSCvt64871 for C480 M5 / C480 M5 ML in 4.1(1d). Run at least 4.1(1d). Field Notice FN-70432 recommends 4.1(3d) or later.

This page covers only the lockstep hang. For what ADDDC Sparing and PPR are, the BIOS tokens and FN-70432, see ADDDC Sparing and PPR on Cisco UCS M5.

ADDDC virtual lockstep hang: symptom

ADDDC Sparing corrects failing DRAM by pairing cache lines in virtual lockstep once correctable errors build up on a bank or rank. That is when this bug appears. A server that ran fine for months logs an ADDDC sparing event (UCS Manager fault F1706 or F1705, depending on firmware and CPU SKU). Later it stops responding and reboots with an internal error (#IERR) and an M2M timeout in the memory subsystem. For memory-heavy workloads such as SAP HANA on vSphere, this is exactly the outage ADDDC was supposed to prevent.

Cisco’s release notes point to the Intel specification updates: SKX108 in the Intel Xeon Processor Scalable Family Specification Update, and CLX37 in the 2nd Gen Intel Xeon Scalable Specification Update. The erratum title is “Memory Controller May Hang While in Virtual Lockstep”. Under complex microarchitectural conditions, a memory controller in VLS may hang on a partial write, and the BIOS can contain a workaround. Cisco’s fix is that BIOS workaround.

Cisco bugs and fixed releases

DefectSymptom (Cisco release notes)First affectedResolved in
CSCvr79388UCS servers stop responding and reboot after ADDDC virtual lockstep is activated. #IERR and M2M timeout in the memory system.4.0(4e)B4.1(1c)B
CSCvr79396On UCS M5, the VLS sparing copy finishes early, leaving incorrect values in the lockstep region.4.0(4e)B4.1(1c)B
CSCvt64871In rare cases, C480 M5 and C480 M5 ML stop responding and reboot after ADDDC virtual lockstep is activated (SKX108 / CLX37).4.0(4h)C4.1(1d)C

The “B” and “C” suffixes are the UCS Manager bundle types (B-Series and C-Series server bundles). CSCvt64871 was still an open caveat in 4.1(1c). Cisco’s workaround until 4.1(1d): “If the server crashes many times after activating ADDDC virtual lockstep, disable ADDDC.”

Why I no longer recommend stopping at 4.1(1c)

When I first wrote this post in May 2020, 4.1(1c) was the fix. The UCS Manager 4.1 release notes later listed two regressions in the 4.1(1c) M5 BIOS, both resolved in 4.1(1d) (July 2020):

  • CSCvu14656: after the BIOS upgrade to the 4.1(1c) versions (for example B200M5.4.1.1c.0.0404202345, C240M5.4.1.1c.0.0405200025), M5 servers could hang at the memory testing step during boot.
  • CSCvu11155: possible performance degradation on B-, C- and S-Series M5 servers with 2nd Gen Xeon Scalable CPUs on the same BIOS versions.

FN-70432 also moved the minimum up. ADDDC support begins at 4.1(1d) for UCSM-managed servers (4.1(1g) for standalone). The recommended releases are 4.1(3d) or later for blades and integrated racks and 4.1(3c) or later for standalone M5 racks. So the practical rule is to treat 4.1(1c) as history. Run a current Cisco-recommended release for M5, and never lower than 4.1(3d)/4.1(3c).

What to check on your M5 servers

  1. Find the server firmware (host firmware package) version in UCS Manager or IMC. If it is older than 4.1(1c) and ADDDC is active, check it against CSCvr79388/CSCvr79396. The release notes list 4.0(4e) as first affected, so check the Bug Search Tool for fixed 4.0(4) builds.
  2. Look for past F1705/F1706 faults and unexpected reboots with #IERR or M2M timeout in the SEL. The hang can only happen after ADDDC has actually put a region into lockstep.
  3. Check the BIOS policy. Select Memory RAS configuration should be Platform Default or ADDDC Sparing. Do not switch ADDDC off as a permanent fix. That brings back the higher uncorrectable-error risk that FN-70432 describes.
  4. Upgrade the server firmware bundle. Put the ESXi host in maintenance mode first, because the BIOS update needs a reboot.
  5. After the upgrade, follow Cisco’s ADDDC event procedure. Reboot in a maintenance window so PPR can repair the DIMM, and replace the DIMM only if an uncorrectable error follows.

Sources

Related on vdan.cz

How to Configure vSphere 6.7 Proactive HA with Cisco UCS Manager Plugin for VMware vSphere?

How to configure vSphere 6.7 Proactive HA with the Cisco UCS Manager Plugin for the vSphere HTML client, so hosts are evacuated on UCS hardware faults.

Proactive HA is working in VCSA 6.7 with Cisco UCS Manager Plugin for VMware vSphere HTML Client (beta Version 3.0(2))
I wrote in previous blog latest Cisco UCS Manager Plugin is working with vCenter 6.7 U3b.

Install Cisco UCS Manager Plugin

vSphere Web Client – Enable Proactive HA

From vSphere Web Client -> Cluster Properties -> Configure -> vSphere Availability -> Proactive HA is Turned OFF – Click on Edit. You can notice vSphere Proactive HA is disabled by default.

  • Automation Level – Determine whether host quarantine or maintenance mode and VM migrations are recommendations or automatic.
    • Manual – vCenter Server suggests migration recommendations for virtual machines.
    • Automated – Virtual machines are migrated to healthy hosts and degraded hosts are entered into quarantine or maintenance mode depending on the configured Proactive HA automation level.
  • Remediation – Determine what happens to partially degraded hosts.
    • Quarantine mode – for all failures. Balances performance and availability, by avoiding the usage of partially degraded hosts provided that virtual machine performance is unaffected.
    • Mixed mode – Quarantine mode for moderate and Maintenance mode for severe failure (Mixed). Balances performance and availability, by avoiding the usage of moderately degraded hosts provided that virtual machine performance is unaffected. Ensures that virtual machines do not run on severely failed hosts.
    • Maintenance mode – for all failures. Ensures that virtual machines do not run on partially failed hosts.
Best options is Automated + Mixed Mode
Select Cisco UCS Provider – NOT Block Failure Conditions

How is Proactive HA working?

With settings Automatic Level – Automated and Remediation – Mixed Mode after HW Failure. Proactive HA is Entering Host Into Quarantine Mode and Migrate all VMs from ESXi with HW Failure:

After 4:10 mintes Proactive HA migrated all VMs from ESXi host with failure.