Month: October 2022

Field Notice: FN – 72368 – Some DIMMs Might Fail Prematurely Due to a Manufacturing Deviation – Hardware Upgrade Available

Cisco Field Notice FN-72368: some DIMMs might fail prematurely due to a manufacturing deviation. What’s affected, and why ADDDC and PPR help.

Cisco announced Field Notice: FN – 72368 – Some DIMMs Might Fail Prematurely Due to a Manufacturing Deviation – Hardware Upgrade Available

My personal recommendation please use ADDDC and PPR – It could prevent hardware failures … UCS-ML-128G4RT-H is in 2nd revision from 28-Oct-22.

Problem Description

A limited number of DIMMs shipped from Cisco are impacted by a known deviation in the memory supplier’s manufacturing process. This deviation might result in a higher rate of failure.

Background

DIMM manufacturers compose their DIMMs of multiple memory modules to reach the desired capacity. A 16GB DIMM might be composed of the same modules that a 32GB DIMM is composed of. In this case, a manufacturing deviation in specific modules impacts 16GB, 32GB, 64GB, and 128GB DIMMs. This deviation was contained to a specific date range, and the DIMMs which use these chips were manufactured during the middle to end of 2020. Since the discovery of this deviation, additional limits have been imposed on the manufacturing process to ensure that future DIMMs are not exposed to this process variation.

Problem Symptom

Most DIMMs with this manufacturing deviation will exhibit persistent correctable memory errors. If left untreated, the DIMMs might eventually encounter an uncorrectable memory event. If encountered during runtime, uncorrectable errors will cause a sudden unexpected server reset. If encountered during Power-On Self-Test (POST), the DIMM will be mapped out and the total available memory reduced. In some cases a boot error might be seen.

Various DIMM Reliability, Availability, and Serviceability (RAS) features or even operating system features might mask the extent of these correctable errors. It is recommended to check your DIMMs for exposure using the Serial Number Validation Tool described in the Serial Number Validation section of this field notice. Only specific DIMMs are impacted by this issue, so do not rely solely on the DIMM error count to judge exposure.

Workaround/Solution

This is a hardware failure. A replacement is strongly recommended in order to avoid potential for unexpected server failure.

Install ESXi 8.0 on an Unsupported CPU: allowLegacyCPU=true and Kickstart Automation (Lab Only)

Install ESXi 8.0 on an unsupported CPU in a lab: allowLegacyCPU=true boot option, boot.cfg kernelopt and kickstart –forceunsupportedinstall, plus limits.

Short answer: For a lab install of ESXi 8.0 on an unsupported CPU, add the kernel boot option allowLegacyCPU=true, either interactively with Shift+O or on the kernelopt= line of boot.cfg. This turns the CPU error into a warning you must accept. To avoid that prompt in a scripted install, add --forceunsupportedinstall to the kickstart install line. This is unsupported, and it is not for production.

Why the ESXi 8 installer stops on older CPUs

The ESXi installer checks the processor against Broadcom’s support list. KB 318697 (CPU Support Deprecation and Discontinuation) defines two states. For a deprecated CPU the installer shows a warning and continues, because it is still supported but will be dropped in a future release. For a discontinued CPU the installation is halted with an error. allowLegacyCPU=true targets the second case: it downgrades the hard stop to a warning that you have to acknowledge.

Important: allowLegacyCPU does not appear in Broadcom’s ESXi installation documentation. It is a community-known option (popularised by William Lam). A host installed this way runs on hardware outside the compatibility guide, so no Broadcom support applies. Future updates can also stop booting on it.

Interactive install: add allowLegacyCPU=true with Shift+O

  1. Boot the ESXi 8.0 installer ISO or USB.
  2. When the installer window appears, press Shift+O to edit boot options, as documented for scripted installs in TechDocs.
  3. Append allowLegacyCPU=true to the existing line and press Enter.
  4. Accept the unsupported CPU warning when the installer displays it.

Automating the ESXi 8.0 install with kickstart

In a scripted install, the warning prompt is what breaks automation: the installer waits for a keypress. You need two things. The first is the boot option, so the installer does not halt. The second is a kickstart flag, so it does not prompt.

1. boot.cfg: kickstart location plus allowLegacyCPU

According to About the boot.cfg File, kernelopt= appends its string to the kernel boot options. Put both options there, in /EFI/BOOT/BOOT.CFG (UEFI) and /BOOT.CFG (BIOS) on your custom ISO, USB or PXE boot files:

kernelopt=ks=http://192.168.30.10/kickstart/KS.CFG allowLegacyCPU=true

2. KS.CFG: suppress the CPU prompt

vmaccepteula
install --firstdisk=local --overwritevmfs --forceunsupportedinstall
rootpw <your-password>
network --bootproto=static --ip=192.168.30.13 --netmask=255.255.255.0 --gateway=192.168.30.1 --nameserver=192.168.30.1 --hostname=esx-lab01.example.local
reboot

--forceunsupportedinstall is listed under the install command in the ESXi 8.0 script command reference, which describes it only briefly in relation to deprecated CPUs. William Lam’s original tip also adds --ignoreprereqwarnings and --ignoreprereqerrors, noting that some systems need all three. Those two flags are not in the TechDocs command reference, so treat them as undocumented. Also note that --overwritevmfs erases an existing datastore on the target disk.

Limits: ESXi 8.0 Update 2 and later, and ESX 9

  • William Lam reported that ESXi 8.0 Update 2 requires the XSAVE CPU instruction even with allowLegacyCPU=true. In practice that means Intel Sandy Bridge / AMD Bulldozer or newer. Older CPUs cannot be forced past this.
  • Each update is a new risk. An unsupported host may fail the precheck or not boot after patching, so keep the previous boot bank.
  • For ESX 9.x, KB 318697 lists additional discontinued CPUs. Check it before you plan a lab refresh. It is not documented whether this boot option still works with the 9.x installer.

Related kickstart notes on vDan: failed to locate kickstart on a nested ESXi VM and disabling IPv6 in an ESXi kickstart. For post-install host configuration, see esxcli commands for ESXi 8.0.

Further reading

Quick Tip – Accessing new custom theme editor…

Quick Tip – Accessing new custom theme editor…

While looking over the vSphere 8 release notes, I had noticed there was also release notes for new version of the ESXi Embedded Host Client, which is an HTML 5 UI for accessing standalone ESXi host. The really interesting feature that stood out to me was the following: Ability to change the […]

VMware Social Media Advocacy

What’s New in vSphere 8 for vMotion: vMotion Notifications, Unified Data Transport and vGPU Improvements

vSphere 8 vMotion features explained: application notifications (HW v20), Unified Data Transport for cold migration, and vGPU changes in 8.0 U2/U3.

Short answer: vSphere 8.0 brought two vMotion features. vMotion notifications tell an application inside the guest that a migration is about to happen, so it can prepare. Unified Data Transport (UDT) speeds up cold migration and cloning of powered-off VMs by sending the data over the vMotion protocol instead of NFC. Later 8.0 updates added vGPU improvements: a per-VM stun time limit in 8.0 U2 and zero-copy vGPU migration in 8.0 U3.

vSphere 8 vMotion features at a glance

FeatureReleaseWhat it solves
vMotion notifications8.0 (VM hardware version 20)Latency-sensitive or clustered apps that can’t tolerate an unannounced stun
Unified Data Transport8.0Slow cold migration and clone of powered-off VMs over NFC
vGPU stun time limit8.0 Update 2Stops a vGPU VM from being migrated to a host whose estimated stun time exceeds the limit
vGPU zero-copy8.0 Update 3Faster vGPU migrations, using up to 100 Gbps

Sources: vSphere 8.0 release notes (hardware version 20 includes “vMotion App Notification”), vCenter 8.0 U2 release notes, and ESXi 8.0 U3 release notes.

vMotion notifications for latency-sensitive applications

Most workloads never notice a vMotion. Some do: trading or real-time apps that break on a short stun, and clustered databases that would rather fail over first. With notifications, the application registers through VMware Tools, gets a start event, does its preparation (drains, quiesces or fails over) and acknowledges. The migration then runs, and the app gets an end event. The app can delay the migration up to a timeout, but it can’t refuse it.

Requirements and configuration

  1. vSphere 8, VM hardware version 20, and VMware Tools or open-vm-tools 11.0 or later in the guest.
  2. Per VM, set the advanced setting vmOpNotificationToAppEnabled to TRUE. It is off by default.
  3. Per host, set vmOpNotificationToApp.Timeout (in seconds, greater than 0) through the vSphere API. For example, 300 means 5 minutes.
  4. In the guest, the application uses vmtoolsd to register and handle events.
vmtoolsd --cmd "vm-operation-notification.register {\"appName\": \"demo\", \"notificationTypes\": [\"sla-miss\"]}"

The other vmtoolsd commands are vm-operation-notification.unregister, .list, .check-for-event and .ack-event. The application polls for events and acknowledges them. One side effect matters for design. DRS excludes VMs with notifications enabled from automatic load-balancing migrations, but still moves them when a host enters maintenance mode. Details are in the Broadcom paper vSphere vMotion Notifications.

Unified Data Transport (UDT) for cold migration

Live vMotion and Storage vMotion of running VMs use the optimized vMotion protocol. Before vSphere 8, a powered-off VM, or the non-active snapshot disks of a running VM, was copied with Network File Copy (NFC), which is much slower. UDT keeps NFC as the control channel and moves the data itself over the vMotion protocol.

  1. Enable the Provisioning service on a VMkernel adapter on the source and destination hosts. You can reuse the vMotion VMkernel adapter.
  2. If vMotion uses the dedicated vMotion TCP/IP stack, you can’t add Provisioning to that adapter. Create a separate VMkernel adapter for it.
  3. Cold migrations and clones then use UDT automatically. Running VMs keep using standard vMotion.

If you expect UDT and don’t see a speed-up, check whether the VM was actually powered off. Broadcom KB 442756 confirms that UDT isn’t used for migrations of powered-on VMs. See the UDT technical paper for the protocol comparison. For host-side networking commands, see esxcli commands for ESXi 8.0.

vGPU vMotion improvements in 8.0 U2 and U3

Since 8.0 U2 you can set a stun time limit per NVIDIA vGPU VM. vCenter then won’t place or migrate the VM on a host or network whose estimated maximum stun time is higher than the limit. 8.0 U3 added zero-copy for vGPU migrations, using up to 100 Gbps. Both are aimed at GPU-accelerated workloads, where a long stun hurts the most.

Further reading

vSphere with Tanzu – new TKG 2.0 ClusterClass…

vSphere with Tanzu – new TKG 2.0 ClusterClass Preview – CormacHogan.com

vSphere with Tanzu – new TKG 2.0 ClusterClass…

One of they key features of the TKG 2.0 on vSphere 8 announcement at VMware Explore 2022 is the consolidation of our the Tanzu Kubernetes offerings into a single unified Kubernetes runtime. This can be considered the second edition of VMware Tanzu Kubernetes Grid. It will still come in two flavors.

VMware Social Media Advocacy