Category: vSphere

Critical Security Alert: Update to Fixed Version 8.0 U3d Immediately

Urgent Notice from VMware by Broadcom

VMware has announced that the security patches released on September 17, 2024, intended to address CVE-2024-38812, did not fully mitigate the vulnerability. As a result, all customers are strongly advised to update to the latest version, 8.0 U3d, immediately. Patches for the 8.0 U2 line are also available to address this issue.

This urgent advisory applies to all vCenter Server users, as the newly identified vulnerabilities pose a significant security risk. Notably, two critical vulnerabilities were reported in vCenter Server, including a heap-overflow vulnerability and a privilege escalation vulnerability. These have been responsibly reported to VMware, which has now provided updates to address them.

Key Vulnerability: Heap-Overflow in vCenter Server (CVE-2024-38812)

Vulnerability Description:

A critical heap-overflow vulnerability was discovered in vCenter Server, specifically in its implementation of the DCERPC protocol. The issue has been assigned CVE-2024-38812 and carries a CVSSv3 base score of 9.8, placing it in the Critical severity range.

Known Attack Vectors:

This vulnerability can be exploited by a malicious actor who has network access to the vCenter Server. By sending a specially crafted network packet, the attacker could trigger the heap-overflow, potentially leading to remote code execution (RCE) on the affected system.

Why You Should Patch Now

This vulnerability could allow attackers to gain control over your vCenter Server environment, putting your infrastructure at risk for unauthorized access, data breaches, or service disruptions. Given the widespread use of vCenter Server for managing virtual environments, this threat is extremely serious, especially for businesses relying on VMware for critical operations.

Patch Availability

The new patches, which can be found in the Response Matrix, have been made available for both the 8.0 U3, 8.0 U2 and 7.0 U3 versions of vCenter Server. Customers should apply the new patches immediately to ensure their systems are protected.

What to Do:

  1. Check your version: Identify if your vCenter Server version is affected.
  2. Apply the patches: Use the Response Matrix provided by VMware to download and install the necessary updates.
  3. Follow VMware’s best practices: Regularly update your systems, review security advisories, and apply patches as soon as they are released to minimize security risks.

Mount VMware-vCenter-Server-Appliance-8.0.3.00400-24322831-patch-FP.iso to VCSA VM. Log in to the appliance shell as a user with super administrative privileges (for example, root) and run the following commands:

# To stage the ISO:
software-packages stage --iso

# To see the staged content:
software-packages list --staged

# To install the staged rpms:
software-packages install --staged

vCenter Server 8.0 U2 Issue with Edit Settings Virtual Machine Hardware 9 or older

Introduction

I was unable to manage Virtual Machines with virtual Hardware Version 9 or older via the vSphere Client while they are in a powered on state.

Symptoms

  1. vCenter Server Version: The problem is specific to vCenter Server version 8.0 U2 – 22385739.
  2. Virtual Machine Hardware Version: Affected VMs are those with hardware version 9 or below.
  3. VM State: The issue occurs when the Virtual Machine is in a powered-on state.
  4. UI Glitches: In the vSphere Client, when attempting to open the ‘Edit Settings’ for the affected VMs, users notice red exclamation marks next to the Virtual Hardware and VM Options tabs. Additionally, the rest of the window appears empty, hindering any further action.

Impact and Risks:

The primary impact of this issue is a significant management challenge:

  • Users are unable to manage Virtual Machines with Virtual Hardware Version 9 or older through the vSphere Client while they remain powered on. This limitation can affect routine operations, maintenance, and potentially urgent modifications needed for these VMs.

Workarounds:

In the meantime, users can employ either of the following workarounds to manage their older VMs effectively:

  1. Power Off the VM: By powering off the VM, the ‘Edit Settings’ window should function correctly. While this is not ideal for VMs that need to remain operational, it can be a temporary solution for making necessary changes.
  2. Use ESXi Host Client: Alternatively, users can connect directly to the ESXi Host Client to perform the ‘Edit Settings’ operations. This method allows the VM to remain powered on, which is beneficial for critical systems that cannot afford downtime.

Resolution:

Keep an eye on updates from VMware for a permanent resolution to this issue Edit settings window fails to load on Virtual Machines with virtual hardware version 9 or older on vCenter Server 8.0U2 (94979).

List of vSphere 8.0 Knowledge base articles and Important Links (89756)

List of Knowledge base articles for vSphere 8.0 – [Main KB] – List of vSphere 8.0 Knowledge base articles and Important Links (89756)

“SECUREBOOT: Image DENIED.” – Virtual Machine with Windows Server 2022 KB5022842 (OS Build 20348.1547) configured with secure boot enabled not booting up (90947)

“SECUREBOOT: Image DENIED.” – Linux VMs created with Hardware version 20 will fail to start installation when Secure Boot is enabled (88737)
Reference error “SECUREBOOT: Image DENIED.” for Linux VMs

Important KB90947 Symptoms

After installing Windows Server 2022 update KB5022842 (OS Build 20348.1547), guest OS can not boot up when virtual machine(s) configured with secure boot enabled running on vSphere ESXi 6.7 U2/U3 or vSphere ESXi 7.0.x.

In VM vmware.log, there is ‘Image DENIED’ info like the below:

2023-02-15T05:34:31.379Z In(05) vcpu-0 - SECUREBOOT: Signature: 0 in db, 0 in dbx, 1 unrecognized, 0 unsupported alg.
2023-02-15T05:34:31.379Z In(05) vcpu-0 - Hash: 0 in db, 0 in dbx.
2023-02-15T05:34:31.379Z In(05) vcpu-0 - SECUREBOOT: Image DENIED.
To identify the location of vmware.log files:
  1. Establish an SSH session to your host. For ESXi hosts
  2. Log in to the ESXi Host CLI using root account.
  3. To list the locations of the configuration files for the virtual machines registered on the host, run the below command:
#vim-cmd vmsvc/getallvms | grep -i "VM_Name"
  1. The vmware.log file is located in virtual machine folder along with the vmx file.
  2. Record the location of the .vmx configuration file for the virtual machine you are troubleshooting. For example:
/vmfs/volumes/xxxxxxxx-xxxxxxx-c1d2-111122223333/vm1/vm1.vmx
/vmfs/volumes/xxxxxxxx-xxxxxxx-c1d2-111122223333/vm1/vmware.log

Resolution

Currently there is no resolution for virtual machines running on vSphere ESXi 6.7 U2/U3 and vSphere ESXi 7.0.x. However the issue doesn’t exist with virtual machines running on vSphere ESXi 8.0.x.

Note: vSphere ESXi 6.7 is End of general Support. For more information, see The End of General Support for vSphere 6.5 and vSphere 6.7 is October 15, 2022.

Workaround

There are three methods to avoid this issue

  1. Upgrade the ESXi Host where the virtual machine in question is running to vSphere ESXi 8.0
  2. Disable “Secure Boot” on the VMs.
  3. Do not install the KB5022842 patch on any Windows 2022 Server virtual machine until the issue is resolved.

See the Microsoft article for details on the updates within the patch release

To disable virtual machine “Secure Boot “option, please follow the below steps:

  1. Power off the VM.
  2. Right-click the virtual machine and click Edit Settings.
  3. Click the VM Options tab.
  4. Under Boot Option, uncheck the “Secure Boot enabled“

Related Information

Uninstalling the KB5022842 patch will not resolve the issue. If the Virtual machine has already been updated, then the only available options are
 

  1. Upgrade the ESXi Host where the virtual machine in question is running to vSphere ESXi 8.0
  2. Disable “Secure Boot” on the VMs.

“SECUREBOOT: Image DENIED.” – Linux VMs created with Hardware version 20 will fail to start installation when Secure Boot is enabled (88737)

“SECUREBOOT: Image DENIED.” – Virtual Machine with Windows Server 2022 KB5022842 (OS Build 20348.1547) configured with secure boot enabled not booting up (90947)
Reference “SECUREBOOT: Image DENIED.” for Windows Server 2022
How ESXi Uses UEFI Secure Boot

Important 88737 Symptoms

The installation of the Operating System image will be denied and “SECUREBOOT: Image DENIED.” will be reported in vmware.log.

Below goes the list of the impacted Linux Operating Systems.

  • RHEL 8.0~8.4, 7.x 
  • CentOS 8.0~8.5, 7.x
  • Oracle Linux 8.0~8.3, 7.x
  • AlmaLinux 8.4    
  • Rocky Linux 8.4    
  • Photon OS 4.0GA, 3.0 GA & Rev 2 & Rev 3, 2.0    
  • Ubuntu LTS 20.04~20.04.4, 18.04~18.04.5 and earlier
  • Ubuntu Non-LTS 21.04, 20.10, 19.10, 19.04, 18.10 and earlier     
  • Debian 10.9 and earlier     
  • SLE 12SP0~SP5, 15SP0-SP2

Cause

This is caused due to the Secure Boot deny list (dbx) is updated to prevent vulnerable bootloaders from being used. For more information, refer to VMware response to GRUB2 security vulnerability CVE-2020-10713 (80181)

Resolution

  1. Create the SecureBoot Virtual Machine with Hardware version 19 (or earlier).
  2. After the installation is completed, update the vulnerable bootloader of the VM to a newer and fixed version, refer to VMware response to GRUB2 security vulnerability CVE-2020-10713 (80181)
  3. Upgrade the Virtual Machine’s Hardware version to 20.

Workaround

Create the Virtual Machine with Secureboot disabled instead.

VMworld 2021 – My Top 10 Sessions

Well here we are again – another VMworld has come around. As most of you will know, VMworld 2021 is going to be fully virtual event. Here are My Top 10 Sessions:

Niels Hagoort

10 things You Need to Know About Project Monterey [MCL1833] 

Curious how Project Monterey and SmartNICs will redefine the data center for VMware ESXi hosts and bare-metal systems?

  • Learn the 10 things you need to know about Project Monterey to accelerate your business
  • Find out how Project Monterey helps to increase performance, security, and manageability by offloading to a DPU
  • Session includes demo of the overall architecture and use cases

Automate and Improve Day 2 Operations with vSphere Lifecycle Manager [MCL1274] 

Reducing the amount of time required to update and upgrade your systems is imperative as the number of systems and environments grow

  • Learn the features and capabilities of vSphere Lifecycle Manager, including newly added support for NSX-T and Tanzu
  • Information and examples to help you get started further automating vSphere Lifecycle Manager with PowerCLI
  • Creating consistent vSphere infrastructures has never been easier with vLCM!

William Lam

App Modernization Deep Dive with VMware Cloud on AWS and VMware Tanzu [MCL2290] 

Is app modernization top of mind for your business? You’re not alone, but many are struggling to begin

  • Learn to leverage Tanzu + VMware Cloud on AWS to discover, analyze and map dependencies
  • How to convert to containers and ultimately deploy a modernized app on an API driven infrastructure
  • Realize the TCO benefits that come with VMware Cloud on AWS

Unleashing the Power of Event-Driven Automation [VI2622] 

Got VEBA? The VMware Event Broker Appliance has come a long way since its release 3 years ago

  • Reflect on key milestones achieved by our great community over the past years
  • Explore real-world use cases and share how users have leveraged VEBA
  • Join and be a part of the event-driven revolution!

Martijn Smit

A Guide to Application Migration Nirvana [MCL1264]

Ever experience application migration paralysis? Fear not!

  • We spill the beans on how our customers solved their migration headaches
  • See technical demos of vRealize Network Insight, vRealize Operations and VMware HCX
  • Find out how to migrate apps to VMWonAWS, Azure VMware Solution, and Google Cloud VMware Engine

Automated Problem Resolution in Modern Networks [NET2160]

Legacy network operations & management solutions have been primarily reactive. Enter the self-healing network

  • Learn how VMware’s self-healing network capabilities solve problems straight away using telemetry-driven and event-driven automation
  • Get an overview of VMware’s networking portfolio and approach to self-healing networks in the data center, branch, and cloud
  • Demo-centric session featuring a special guest from Intel!

Raymond de Jong

Simplify Network Consumption and Automation for Day 1 and Day 2 Operations [NET2185]

Apps are the lifeblood of the business in today’s digital economy. Can you envision seamless connectivity between your apps & networking technologies?

  • Learn why apps are the lifeblood of the business in today’s digital economy
  • See how seamless connectivity between your apps and networking technologies for end-to-end network configuration, compliance and automation with a single API call can be possible
  • Find out how to simplify network consumption with network automation for Day 1 and Day 2 operations

Keith Lee

Manage Kubernetes Across Multiple Clouds [APP1187]

Managing Kubernetes clusters across teams and clouds can be difficult. Not with Tanzu Mission Control.

  • Learn how to centrally operate and manage Kubernetes clusters across multiple clouds with Tanzu Mission Control
  • Also hear about policy management, data protection, security and configuration inspections
  • Jump-start and scale your modern apps practice across the enterprise

Eric Shanks

Meet the Expert: Tanzu Kubernetes Grid with Tom Schwaller [APP2436]

Questions on how VMware Tanzu Kubernetes Grid works? Our experts are here for you!

  • Get the answers to all of your VMware Tanzu Kubernetes Grid questions
  • Open, informal roundtable session with VMware experts
  • Get master insights to ensure you get the best Kubernetes experience possible

Chris McCain

Enterprise Multi-Cloud Security [SEC2445]

With the expansion of VMware’s security portfolio, you may be wondering how all of the pieces fit together into a cohesive multi-cloud enterprise design

  • Learn about VMware’s expanding security portfolio and how all the tools fit together
  • See how cyberattacks occur, including a look into the most famous attack of all time!
  • Gain a better understanding of how to design a policy-driven, defense-in-depth strategy for the modern enterprise

vRops – Data Retriever is not initialized yet. Please wait … -> Replace expired internal certificate in vRealize Operations (71018)

Data Retriever is not initialized yet. Please wait … What is WRONG?

OK, I tries to login with vRops admin login but another message “Incorrect user name/password”.

No luck with How to reset the admin password in vRealize Operations (2078313)

My problem was with expired internal certificate in vRealize Operations.

# /bin/grep -E --color=always -B1 'java.security.cert.CertPathValidatorException: validity check failed|java.security.cert.CertificateExpiredException' $ALIVE_BASE/user/log/*.log | /usr/bin/tail -20

/usr/lib/vmware-vcops/user/log/web.log- at java.lang.Thread.run(Thread.java:748)
/usr/lib/vmware-vcops/user/log/web.log:Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
--
/usr/lib/vmware-vcops/user/log/web.log- ... 27 more
/usr/lib/vmware-vcops/user/log/web.log:Caused by: java.security.cert.CertPathValidatorException: validity check failed
--
/usr/lib/vmware-vcops/user/log/web.log- ... 35 more
/usr/lib/vmware-vcops/user/log/web.log:Caused by: java.security.cert.CertificateExpiredException: NotAfter: Sun Jan 1 10:42:20 EET 2021
--
/usr/lib/vmware-vcops/user/log/web.log-2021-01-10 14:45:19,988 ERROR [pool-2-thread-1] com.vmware.vcops.util.admin.HTTPSRequester.doHttpRequest - Sending 'GET' request to URL : https://vrops/casa/deployment/slice failed
/usr/lib/vmware-vcops/user/log/web.log:javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
--
/usr/lib/vmware-vcops/user/log/web.log- at java.lang.Thread.run(Thread.java:748)
/usr/lib/vmware-vcops/user/log/web.log:Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
--
/usr/lib/vmware-vcops/user/log/web.log- ... 27 more
/usr/lib/vmware-vcops/user/log/web.log:Caused by: java.security.cert.CertPathValidatorException: validity check failed
--
/usr/lib/vmware-vcops/user/log/web.log- ... 35 more
/usr/lib/vmware-vcops/user/log/web.log:Caused by: java.security.cert.CertificateExpiredException: NotAfter: Sun Jan 1 10:42:20 EET 2021

How to fix it? Internal Certificate Expired

  1. Snapshot the vRealize Operations nodes
  2. Download the Certificate Renewal PAK file for your version of vRealize Operations:

vRealize Operations 6.3 – 8.1.1
vRealize Operations 8.2 and later

Note: The file name indicates 8.0.0 but will work for vRealize Operations 6.x and 7.x.

  1. Copy the vRealize Operations Certificate Renewal PAK file to the /tmp/ directory on all nodes in the vRealize Operations cluster using an SCP utility.
  2. Log into all nodes in the vRealize Operations cluster as root via SSH or Console.
  3. Run the following command on all nodes in the vRealize Operations cluster to make the necessary directories:
mkdir -p /data/db/pakRepoLocal/vRealize_Operations_Manager_Enterprise_Certificate_Renewal/extracted
  1. Unzip the vRealize Operations Certificate Renewal PAK file by running the following command on all nodes in the vRealize Operations cluster:
unzip /tmp/vRealize_Operations_Manager_Enterprise_Certificate_Renewal-8.0.0.15217416.pak -d /data/db/pakRepoLocal/vRealize_Operations_Manager_Enterprise_Certificate_Renewal/extracted
  1. The following command needs to be run in a particular order.  Follow each sub-step carefully.
$VMWARE_PYTHON_BIN /data/db/pakRepoLocal/vRealize_Operations_Manager_Enterprise_Certificate_Renewal/extracted/updateCoordinator.py EXPIRED
  • First, run the command on all Remote Collector nodes (if present) in the cluster, and wait for the task to complete.  Continue to step 7.2.
  • Next, run the command on all Data nodes, the Witness node (if present), and the Primary Replica node (if present) in the cluster; do not wait for each node to complete, just start the command on all nodes.  Once Waiting for certificate generation to complete appears on the last node, wait roughly 60 seconds, and continue to step 7.3.
  • Finally, run the command on the Primary node.

The expected behavior is for the command to finish, then shortly afterwards the pending tasks on the Data nodes and Primary Replica node (if present) will complete. To ensure that the command completes successfully check:

ls -l /var/vmware/_cert_generation_completed
  1. Run the following commands on all nodes in the vRealize Operations cluster:
chown admin:admin -R /storage/vcops/user/conf/ssl/ /storage/vcops/user/conf/ssl_bak/ /storage/db/casa/webapp/hsqldb/
chmod guo+r -R /storage/vcops/user/conf/ssl/
service vmware-casa restart
service vmware-vcops stop
sed -i 's/sliceonline\ \=\ true/sliceonline\ \=\ false/g' /usr/lib/vmware-vcopssuite/utilities/sliceConfiguration/data/roleState.properties
  1. Run the following commands on the Primary node, and Primary Replica node (if present):
service vmware-casa stop
sed -i -e 's/\"onlineState\"\:\"GOING\_OFFLINE\"/\"onlineState\"\:\"OFFLINE\"/g' -e 's/\"online\_state\"\:\"GOING\_OFFLINE\"/\"online\_state\"\:\"OFFLINE\"/g' -e 's/\"onlineState\"\:\"GOING\_ONLINE\"/\"onlineState\"\:\"OFFLINE\"/g' -e 's/\"online\_state\"\:\"GOING\_ONLINE\"/\"online\_state\"\:\"OFFLINE\"/g' -e 's/\"onlineState\"\:\"ONLINE\"/\"onlineState\"\:\"OFFLINE\"/g' -e 's/\"online\_state\"\:\"ONLINE\"/\"online\_state\"\:\"OFFLINE\"/g' -e 's/\"onlineState\"\:\"FAILURE\"/\"onlineState\"\:\"OFFLINE\"/g' -e 's/\"online\_state\"\:\"FAILURE\"/\"online\_state\"\:\"OFFLINE\"/g' /data/db/casa/webapp/hsqldb/casa.db.script
service vmware-casa start
service vmware-vcops-web restart
/etc/init.d/apache2 restart
  1. Log into the vRealize Operations Admin UI as the local admin user.
  2. Click Bring Online under Cluster Status.

Summary:

When I can see message “Data Retriever is not initialized yet. Please wait …” or “Incorrect user name/password”.

Try to check and fix for expired internal certificate in vRealize Operations.

Fast check command is here:

/bin/grep -E --color=always -B1 'java.security.cert.CertPathValidatorException: validity check failed|java.security.cert.CertificateExpiredException' $ALIVE_BASE/user/log/*.log | /usr/bin/tail -20

More info:

How to reset the admin password in vRealize Operations (2078313)

One day I reach vRops admin login message “Incorrect user name/password”. Here is a short info How to reset the admin password in vRealize Operations according KB 2078313. But I was unable to reset. My problem was with expired internal certificate in vRealize Operations.

Reset the OS admin Password

Note: Before resetting the admin password, ensure that the account is not locked out by this command: 

pam_tally2 --user admin --reset

Note: If the admin password expires for the local OS, update the local OS version of the password before completing the below steps to reset the password across the Application:

  1. Log into the Master node as root via SSH or Console.
  2. Run the following command to reset the password, and password age of the local admin account:
passwd admin
  1. Repeat steps 1 and 2 on all nodes in the vRealize Operations Manager cluster, including Remote Collectors.

Reset the vRops admin Password

To reset the admin password, follow the steps below:

  1. Log in to the Master node as root via SSH or Console.
  2. Run this command and follow the prompts:

$VMWARE_PYTHON_BIN $VCOPS_BASE/../vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsSetAdminPassword.py --reset

Unable to reset – problem with expired internal certificate in vRealize Operations

localhost:~ # $VMWARE_PYTHON_BIN $VCOPS_BASE/../vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsSetAdminPassword.py --reset
Password:
Re-enter Password:
Saved existing admin user credentials into backup file
Call failed: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)>
Original admin credentials restored from backup

More info:

FIX: The virtual machine is configured for too much PMEM. 6.0 TB main memory and x PMEM exceeds the maximum allowed 6.0 TB

VMware supports Intel Optane PMem with vSphere 6.7 and 7.0. VMware and Intel worked with SAP to complete the validation of SAP HANA with Optane PMem enabled VMs.

I configured PoC testing VM RAM 1 TB and 1 TB PMem. I was unable to power on. Error – The virtual machine is configured for too much PMem. 6.0 TB main memory and 1024 GB PMem exceeds the maximum allowed 6.0 TB.

Problem was with enable Memory Hot Plug, because there is a limit calculation:

With enable Memory Hot Plug
* Limit 6TB <= size of RAM x 16 + size of PMEM

With disable Memory Hot Plug
* Limit 6TB <= size of RAM + size of PMEM

SAP HANA does not support hot-add memory. Because of this, hot-add memory was not validated by SAP and VMware with SAP HANA and is therefore not supported. According SAP HANA on VMware vSphere

Example how to reproduce an error:

6 TB limit
1 TB PMem
5 * 1024 / 16 = 320 GB

With enable Memory Hot Plug I can't start VM with more than 321 GB RAM.

How to Configure vSphere 6.7 Proactive HA with Cisco UCS Manager Plugin for VMware vSphere?

Proactive HA is working in VCSA 6.7 with Cisco UCS Manager Plugin for VMware vSphere HTML Client (beta Version 3.0(2))
I wrote in previous blog latest Cisco UCS Manager Plugin is working with vCenter 6.7 U3b.

Install Cisco UCS Manager Plugin

vSphere Web Client – Enable Proactive HA

From vSphere Web Client -> Cluster Properties -> Configure -> vSphere Availability -> Proactive HA is Turned OFF – Click on Edit. You can notice vSphere Proactive HA is disabled by default.

  • Automation Level – Determine whether host quarantine or maintenance mode and VM migrations are recommendations or automatic.
    • Manual – vCenter Server suggests migration recommendations for virtual machines.
    • Automated – Virtual machines are migrated to healthy hosts and degraded hosts are entered into quarantine or maintenance mode depending on the configured Proactive HA automation level.
  • Remediation – Determine what happens to partially degraded hosts.
    • Quarantine mode – for all failures. Balances performance and availability, by avoiding the usage of partially degraded hosts provided that virtual machine performance is unaffected.
    • Mixed mode – Quarantine mode for moderate and Maintenance mode for severe failure (Mixed). Balances performance and availability, by avoiding the usage of moderately degraded hosts provided that virtual machine performance is unaffected. Ensures that virtual machines do not run on severely failed hosts.
    • Maintenance mode – for all failures. Ensures that virtual machines do not run on partially failed hosts.
Best options is Automated + Mixed Mode
Select Cisco UCS Provider – NOT Block Failure Conditions

How is Proactive HA working?

With settings Automatic Level – Automated and Remediation – Mixed Mode after HW Failure. Proactive HA is Entering Host Into Quarantine Mode and Migrate all VMs from ESXi with HW Failure:

After 4:10 mintes Proactive HA migrated all VMs from ESXi host with failure.