Critical vCenter vmdir information-disclosure vulnerability CVE-2020-3952 (VMSA-2020-0006): affected versions, how to check, and patching.
On April 9th, 2020 VMSA-2020-0006 was published. This advisory documents a critical severity sensitive information disclosure vulnerability identified by CVE-2020-3952.
Affected versions
The vulnerability received a CVSSv3 score of 10 out of 10. Which means this is a very serious security issue. Response matrix is VMSA-2020-0006.
How I can check it?
Additional Documentation for VMSA-2020-0006: Determining if a vCenter 6.7 deployment w/embedded or external Platform Services Controller (PSC) is affected by CVE-2020-3952 (78543)
Virtual Appliance Log File Location: /var/log/vmware/vmdird/vmdird-syslog.log or in /var/log/vmware/vmdird/vmdird-syslog.log.*.gz
zgrep "ACL" /var/log/vmware/vmdird/*.gz
/var/log/vmware/vmdird/vmdird-syslog.log.x.gz:2020-xx-xxTxxxxxx+00:00 info vmdird t@xxxxxx: ACL MODE: Legacy
Notes from KB:
In order to be affected by CVE-2020-3952, a deployment must meet 2 criteria. First, it must be a 6.7 deployment prior to 6.7u3f. Second, it must be running in legacy ACL mode.
Because the ACL MODE: Legacy log entry is only thrown at vmdir startup, it is possible that it will be absent due to log file rollover even on affected deployments.
The ACL MODE: Legacy log entry will still be thrown after upgrading to 6.7u3f and/or 7.0 even though CVE-2020-3952 is resolved in these releases.
Path it NOW ! – PoC was published !
It is recommended to block any access over the LDAP port (389) except for administrative use.
Clean installations of vCenter Server 6.7 (embedded or external PSC) are not affected.
vCenter Server 6.7 (embedded or external PSC) prior to 6.7u3f is affected by CVE-2020-3952 if it was upgraded from a previous release line such as 6.0 or 6.5.
Path it ASAP because:
On April 15th, 2020 was relased information about How to reconstructed the faulty code flow that led to this vulnerability.
CPU_SUPPORT ERROR in the ESXi installer? Press Shift+O and add allowLegacyCPU=true to turn it into a warning. Versions, limits (XSAVE), lab only.
Short answer: When the ESXi installer stops with CPU_SUPPORT ERROR: The CPU in this host is not supported by ESXi, reboot the installer and press Shift+O at the boot prompt. Append allowLegacyCPU=true and press Enter. The error becomes a CPU_SUPPORT WARNING that you can accept. This works on 7.0 and 8.0, and lab users report it on 9.x. It is not supported by Broadcom, so use it only in a homelab.
The error: “The CPU in this host is not supported by ESXi”
I first hit this on an older homelab server with the ESXi 7.0.0 installer (VMware-VMvisor-Installer-7.0.0-15843807.x86_64.iso):
CPU_SUPPORT ERROR:
The CPU in this host is not supported by ESXi
7.0.0. Please refer to the VMware Compatibility Guide (VCG) for
the list of supported CPUs.
Without the boot option, the only way forward is F11 Reboot.
The installer offers only Back or Reboot. Newer installers add a pointer to KB 82794, which is now Broadcom KB 318697. For what that KB says and which CPUs it lists, see KB 82794 explained.
Why the installer blocks the CPU
Broadcom KB 318697 defines two stages. A deprecated CPU series gets a warning and the installation continues. A discontinued series is blocked with an error. The check is part of the installer precheck. On the 7.0.0 ISO it lives in \UPGRADE\PRECHECK.PY, and when I looked at that file (around line 1720) it also contained the allowLegacyCPU override. If you only see a warning, your CPU is deprecated, not discontinued, and you need no override.
How to use allowLegacyCPU=true (Shift+O)
Boot the ESXi installer ISO or USB.
When the installer boot screen shows Shift+O: Edit boot options, press Shift+O. The prompt is only shown for a few seconds. Shift+O is the documented way to edit installer boot options.
Leave the existing text (for example runweasel cdromBoot) in place and append a space plus the option.
Press Enter. The system scan now reports a warning instead of an error. Press Enter to continue.
allowLegacyCPU=true
Shift+O opens the boot options line.Append allowLegacyCPU=true to the existing boot options.Now it is only a warning, and Enter continues the installation.
CPU_SUPPORT WARNING:
The CPU in this host is not supported by ESXi
7.0.0. Please refer to the VMware Compatibility Guide (VCG) for
the list of supported CPUs.
Using allowLegacyCPU=true in boot.cfg and kickstart
Shift+O applies only to the current boot. To avoid typing it, put it on the kernelopt= line of boot.cfg on your installer media or PXE files. TechDocs documents that kernelopt= appends its string to the kernel boot options. William Lam also noted (for 7.0) that the option is not persistent after installation. For an installed host that needs it on every boot, he suggests adding it to kernelopt in both /bootbank/boot.cfg and /altbootbank/boot.cfg.
My own test on 7.0.0 (build 15843807); William Lam
ESXi 8.0 GA to 8.0 U1
Error becomes a warning
William Lam
ESXi 8.0 U2 and later
Still honoured, but the CPU must support XSAVE (Intel Sandy Bridge / AMD Bulldozer or newer). Older CPUs such as Westmere stop with VMB: 716: Unsupported CPU.
William Lam
ESX 9.0 / 9.1
Lab users report the same error-to-warning behaviour
Community reports only; not documented by Broadcom
allowLegacyCPU is not listed in Broadcom’s installation documentation or in KB 313264 (additional kernel boot options). It is an engineering override that has never been guaranteed and may disappear in any build.
What allowLegacyCPU=true does not give you
No support. The host runs on a CPU that is not in the Broadcom Compatibility Guide for that release.
No guarantee for updates. Any patch or update can add a new hard requirement, as 8.0 U2 did with XSAVE. Keep the previous boot bank so you can roll back.
Not the fix for a warning. For a deprecated CPU, patching with esxcli software profile update only needs --no-hardware-warning (Broadcom KB 391610). That host is still supported.
Not a production path. For production hosts with discontinued CPUs, KB 428874 describes RPQ/TQ requests for “VCF Software Only” support through your account team. Broadcom has also published a supported override procedure for Intel Skylake-SP on VCF 9.x. See Intel Skylake and ESX 9 support.
Disclaimer: this is not officially supported by VMware by Broadcom. You run it at your own risk.
I know many of you have been asking me about my vSphere with Kubernetes automation script which I had been sharing snippets of on Twitter. For the past couple of weeks, I have been hard at work making the required changes between the vSphere 7 Beta and GA workflows, some additional testing and of course […]
Baselines are deprecated in vSphere 8 and can’t manage ESX 9. How to convert a cluster from vLCM baselines to an image, prerequisites and blockers.
Short answer: In the vSphere Client, select the cluster and open Updates > Image. vCenter runs an eligibility check first. Then you either use the image from a host in the cluster (vCenter 8.0 U3+), import a JSON image, or set one up manually. Finish with Finish image setup and remediate. The switch can’t be reversed. Baselines were deprecated in vSphere 8.0, and in vCenter 9.0 they can only patch ESXi 8.x hosts. You must convert before you move to ESX 9.
vLCM baselines status: deprecated in 8.0, removed for ESX 9
I wrote the first version of this post in 2020, right after upgrading my lab to ESXi 7.0. Back then images were new and optional. Here is what the Broadcom documentation says today:
Release
Baselines (VUM workflows)
Source
vSphere 7.0
Supported. vLCM images introduced as the declarative alternative.
KB 322186
vSphere 8.0
Deprecated. Still supported in vSphere 8, with removal planned for the next major release.
vSphere 8.0 release notes, KB 322186
vCenter 9.0
Removed as a way to manage clusters. Baselines only patch ESXi 8.x hosts. ESX 9 hosts can’t be patched or upgraded with them, ISO import and ISO-based upgrades are blocked, and new clusters and standalone hosts are image-managed by default.
VCF 9.0 release notes, KB 379388
So an existing baseline-managed cluster with 8.x hosts keeps working under vCenter 9.0, but it can’t reach ESX 9. According to KB 379388, the way around those restrictions is to move from VUM to vLCM images.
Prerequisites and what blocks the conversion
When you start the switch, vCenter runs the Check cluster’s eligibility to be managed with a single image task. Its results come in three types.
Errors (block the conversion)
A host is not stateful, for example an Auto Deploy stateless host. It must boot from a disk.
A host runs a version earlier than ESXi 7.0. Upgrade it with an upgrade baseline first.
VIBs from an unintegrated solution are present. Deactivate the solution and retry.
The cluster is currently being remediated against a baseline.
Warnings (need action)
A standalone VIB for which a component exists in the depot. Add that component to the image, or the VIB is deleted during remediation.
An unknown VIB, with no component in the depot. Import a component that contains it, or it is deleted during remediation.
Other prerequisites
All hosts from the same hardware vendor.
The vLCM image privileges.
For third-party solutions, confirmation from the vendor that they work with vLCM.
For the “use an image from a host” option: hosts on 7.0 U2 or later and vCenter on 8.0 U3 or later.
For VCF environments: do this only through SDDC Manager, on VCF 5.2.2 or later or VCF 9 (KB 385617). Don’t convert VCF clusters directly in vCenter.
The depot also needs the content you plan to use in the image. On current releases, vLCM no longer downloads from the old public VMware URLs; each customer gets a unique download URL. See KB 390098 if the Software Depot doesn’t show recent releases. Offline, you can import a depot ZIP through Actions > Import Updates. These are my original vSphere 7.0 screenshots:
Lifecycle Manager: Actions > Import Updates (vSphere 7.0 UI)Importing VMware-ESXi-7.0.0-15843807-depot.zipImage Depot: ESXi 7.0 GA – 15843807 available after the import
How to convert vLCM baselines to an image (step by step)
These steps follow the TechDocs procedure for Option 3: Set Up a New Image Manually. Options 1 and 2 start the same way.
Open the image setup. In the vSphere Client, select the baseline-managed cluster and go to Updates > Image.
Choose how to build the image. Pick one of the three options. Set up image manually builds it by hand. Import image takes a JSON file. On vCenter 8.0 U3+, you can also choose one of the “Currently running images in the cluster” and click Proceed with selected image. The eligibility check runs; fix any errors and start again.
Define the image. In the Convert to an Image pane, select the ESXi Version. Optionally add a Vendor Addon, a Firmware and Drivers Addon (this needs a hardware support manager) and extra Components.
Validate and save. Click Validate to catch missing dependencies and conflicts, then Save. Saving triggers a compliance check of all hosts against the image.
Finish the setup. Click Finish image setup and confirm with Yes, finish image setup. From now on the cluster is managed by a single image.
Remediate. Run the pre-check and then Remediate All. Switching the management method alone does not change the hosts; only remediation applies the image.
Cluster > Updates: start with SETUP IMAGE (vSphere 7.0 UI)Select the ESXi version and add components (example: USB NIC Fling driver)Step 2: image compliance check (3 of 3 hosts out of compliance before remediation)Finish image setup: confirm with YESAfter the conversion the Baselines menu disappearsREMEDIATE ALL: start dialogSTART REMEDIATION installs the example USB NIC Fling driver
What changes after the switch
There is no way back. A cluster can’t return to baselines. You can only move hosts into another cluster that still uses baselines.
Remediation removes what the image doesn’t include. Standalone VIBs and agents of non-integrated solutions are deleted. Put every driver and tool you need into the image first.
Standalone hosts keep the image only under certain conditions. A host removed from an image-managed cluster keeps the image only when vCenter and ESXi are 8.0 U3+ and NSX, if present, is 4.2 or later. Otherwise it reverts to baselines.
Unsupported configurations are still possible. If you enable a solution that doesn’t work with vLCM on an empty cluster (Dell EMC VxRail is the documented example), the transition succeeds but the configuration is unsupported.
We are excited to announce the general availability of VMware NSX-T™ 3.0, a major release of our full stack Layer 2 to Layer 7 networking platform that offers virtual networking, security, load balancing, visibility, and analytics in a single platform. NSX-T 3.0 includes key innovations across cloud-scale networking, security, containers, and operations that help enterprises achieve one-click public cloud experience wherever their workloads are deployed. As enterprises adopt cloud,…Read More
How to configure vSphere 6.7 Proactive HA with the Cisco UCS Manager Plugin for the vSphere HTML client, so hosts are evacuated on UCS hardware faults.
From vSphere Web Client -> Cluster Properties -> Configure -> vSphere Availability -> Proactive HA is Turned OFF – Click on Edit. You can notice vSphere Proactive HA is disabled by default.
Automation Level – Determine whether host quarantine or maintenance mode and VM migrations are recommendations or automatic.
Manual – vCenter Server suggests migration recommendations for virtual machines.
Automated – Virtual machines are migrated to healthy hosts and degraded hosts are entered into quarantine or maintenance mode depending on the configured Proactive HA automation level.
Remediation – Determine what happens to partially degraded hosts.
Quarantine mode – for all failures. Balances performance and availability, by avoiding the usage of partially degraded hosts provided that virtual machine performance is unaffected.
Mixed mode – Quarantine mode for moderate and Maintenance mode for severe failure (Mixed). Balances performance and availability, by avoiding the usage of moderately degraded hosts provided that virtual machine performance is unaffected. Ensures that virtual machines do not run on severely failed hosts.
Maintenance mode – for all failures. Ensures that virtual machines do not run on partially failed hosts.
Best options is Automated + Mixed ModeSelect Cisco UCS Provider – NOT Block Failure Conditions
How is Proactive HA working?
With settings Automatic Level – Automated and Remediation – Mixed Mode after HW Failure. Proactive HA is Entering Host Into Quarantine Mode and Migrate all VMs from ESXi with HW Failure:
After 4:10 mintes Proactive HA migrated all VMs from ESXi host with failure.
Move from vSphere 6 Update Manager to vSphere 7 Lifecycle Manager and upgrade ESXi 6.7 hosts to ESXi 7.0 step by step.
vSphere Update Manager – vSphere 6
In vSphere 6 we can use various methods and tools to deploy ESXi hosts and maintain their software lifecycle.
To deploy and boot an ESXi host, you can use an ESXi installer image or VMware vSphere® Auto Deploy™. The availability of choice options results in two different underlying ESXi platforms:
Using vSphere Auto Deploy – stateless mode
Using an installer ESXi image – statefull mode
vSphere Lifecycle Manager Images: A Single Platform, Single Tool, Single Workflow
By introducing the concept of images, vSphere Lifecycle Manager provides a unified platform for ESXi lifecycle management. You can use vSphere Lifecycle Manager for stateful hosts only, but starting with vSphere 7.0, you can convert the Auto Deploy-based stateless hosts into stateful hosts, which you can add to clusters that you manage with vSphere Lifecycle Manager images.
How to Upgrade ESXi 6.7 to 7 with vSphere Lifecycle Manager?
After upgrade VCSA 7.0, We prepare upgrade for ESXi 6.7. It is simular logic like in vSphere Update Manager:
IMPORT ISO – We can upload ISO for example VMware-VMvisor-Installer-7.0.0-15843807.x86_64.isoStep 1 of 2 – Uploading file to serverStep 2 of 2 – Adding to repositoryAfter ISO upload We could check ISO image contextNext step is Create BaselineCreate Baseline – for ISO imageSelect uploaded ISO imageCheck summaryOn Targer Cluster We attach our BaselineSelect ESXi 7 Baseline– Check Compliance – We can see Non-compliant for 3x ESXi hostREMEDIATE will start upgrade dialogeIt is necessary accept EULAREMEDIATE will start ESXi 7 upgradeIn Recent Task We caould chek progress.We can check ESXi 7.0 upgrade result.
We’re very excited to announce the general availability of vSphere 7 today! It caps off a massive across-the-board effort by the many engineering teams within VMware. We have built a ton of new capabilities into vSphere 7, including drastically improved lifecycle management, many new security features, and broader application focus and support. But of course, The post How to Get vSphere with Kubernetes appeared first on VMware vSphere Blog.
Short answer: Three BIOS tokens cut Cisco UCS M5 boot time. Adaptive Memory Training = Enabled reuses saved memory training results if the server reboots within 24 hours. OptionROM Launch Optimization = Enabled launches Option ROMs only for controllers in the boot policy. BIOS Techlog Level = Minimum logs only critical messages. From UCS Manager 4.0(2a) these are the M5 defaults, so check that nothing overrides them. On a B480 M5 with 3 TB RAM, each boot after the first was about 2 minutes faster.
Cisco UCS M5 boot time BIOS tokens
Token (UCS Manager BIOS policy)
Supported values on M5
Default on M5 (4.0(2a) and later)
Value for faster boot
Adaptive Memory Training
Enabled, Disabled
Enabled
Enabled
OptionROM Launch Optimization
Enabled, Disabled
Enabled
Enabled
BIOS Techlog Level
Maximum, Normal, Minimum
Minimum
Minimum
Cisco’s BIOS token guide lists Adaptive Memory Training for B200 M5, B480 M5, C220 M5, C240 M5, C480 M5, C480 M5 ML and S3260 M5. It lists the other two tokens for all M5 servers. The tokens appear from the 4.0(2a) section of the guide onwards. Older firmware may not have them, or may use different defaults.
Adaptive Memory Training
From the UCS Manager Server Management Guide (Boot Options BIOS Settings): when this token is enabled, the BIOS saves the memory training results (optimized timing and voltage values) together with the CPU and memory configuration, and reuses them on later reboots to save boot time. The saved results are used only if the reboot happens within 24 hours of the last save. In my experience it has the biggest effect on large-memory hosts.
OptionROM Launch Optimization
Option ROM launch is controlled per PCI slot and is enabled by default. With many network controllers and storage HBAs, every Option ROM may launch even though only a few controllers take part in booting. With this token enabled, Option ROMs launch only for the controllers listed in the boot policy. So keep the boot policy accurate: a device that is missing from it will not get its Option ROM.
BIOS Techlog Level
This sets which messages go to the BIOS tech log. Minimum logs critical messages only and is the default. Normal adds warning and loading messages, and Maximum adds informational messages. Keep Minimum in production and raise it only when Cisco TAC asks for verbose BIOS logs.
Correction to my 2020 version: I described the “BIOS Techlog Level” token as an Enabled/Disabled switch that makes the tech log more granular. That description belongs to a different token, BIOS Tech Message Level Control, which the guide lists for the C125 M5.
How to set the tokens in UCS Manager
Open the BIOS policy that your service profiles or templates use (Servers > Policies > root > BIOS Policies).
On the Boot Options settings, set Adaptive Memory Training and OptionROM Launch Optimization to Enabled, and BIOS Techlog Level to Minimum. “Platform Default” gives the same result on 4.0(2a) and later firmware. Set them explicitly if you want the policy to show the intent.
Check the boot policy. It should contain every device you really boot from (for example the SAN or M.2 boot device and any PXE vNIC).
BIOS changes take effect at the next server reboot. Put the ESXi host into maintenance mode and reboot it according to the service profile’s maintenance policy.
Results on a B480 M5 with 3 TB RAM
The first boot after the change took about 1 to 2 minutes longer.
From the second boot on, each boot was about 2 minutes shorter.
After 24 hours without a reboot, the saved training data is no longer used, so the next boot is a full-training boot again.
Other tokens that affect boot time
Quiet Boot controls only what is displayed during POST (logo vs messages and Option ROM information).
Enhanced Memory Test (the M6 name; Cisco notes it was renamed from Advanced Memory Test) runs extended memory tests during boot and “increases the boot time based on the memory”. Cisco recommends leaving it at Auto.
UCS Manager CLI interactive mindmap: the BIOS policy scopes (/org/bios-policy, token settings) and every other UCSM CLI command from Cisco’s CLI guides
In my previous post on VCF 4.0, we looked at the steps involved in deploying vSphere with Kubernetes in a Workload Domain (WLD). When we completed that step, we had rolled out the Supervisor Control Plane VMs, and installed the Spherelet components which allows our ESXi hosts to behave as Kubernetes worker nodes. Let’s now take a closer look at that configuration, and I will show you a few simple Kubernetes operations to get you started on the Supervisor Cluster in vSphere with Kubernetes….Read More
