Category: vSphere

FIX: The virtual machine is configured for too much PMEM. 6.0 TB main memory and x PMEM exceeds the maximum allowed 6.0 TB

VMware supports Intel Optane PMem with vSphere 6.7 and 7.0. VMware and Intel worked with SAP to complete the validation of SAP HANA with Optane PMem enabled VMs.

I configured PoC testing VM RAM 1 TB and 1 TB PMem. I was unable to power on. Error – The virtual machine is configured for too much PMem. 6.0 TB main memory and 1024 GB PMem exceeds the maximum allowed 6.0 TB.

Problem was with enable Memory Hot Plug, because there is a limit calculation:

With enable Memory Hot Plug
* Limit 6TB <= size of RAM x 16 + size of PMEM

With disable Memory Hot Plug
* Limit 6TB <= size of RAM + size of PMEM

SAP HANA does not support hot-add memory. Because of this, hot-add memory was not validated by SAP and VMware with SAP HANA and is therefore not supported. According SAP HANA on VMware vSphere

Example how to reproduce an error:

6 TB limit
1 TB PMem
5 * 1024 / 16 = 320 GB

With enable Memory Hot Plug I can't start VM with more than 321 GB RAM.

How to Configure vSphere 6.7 Proactive HA with Cisco UCS Manager Plugin for VMware vSphere?

Proactive HA is working in VCSA 6.7 with Cisco UCS Manager Plugin for VMware vSphere HTML Client (beta Version 3.0(2))
I wrote in previous blog latest Cisco UCS Manager Plugin is working with vCenter 6.7 U3b.

Install Cisco UCS Manager Plugin

vSphere Web Client – Enable Proactive HA

From vSphere Web Client -> Cluster Properties -> Configure -> vSphere Availability -> Proactive HA is Turned OFF – Click on Edit. You can notice vSphere Proactive HA is disabled by default.

  • Automation Level – Determine whether host quarantine or maintenance mode and VM migrations are recommendations or automatic.
    • Manual – vCenter Server suggests migration recommendations for virtual machines.
    • Automated – Virtual machines are migrated to healthy hosts and degraded hosts are entered into quarantine or maintenance mode depending on the configured Proactive HA automation level.
  • Remediation – Determine what happens to partially degraded hosts.
    • Quarantine mode – for all failures. Balances performance and availability, by avoiding the usage of partially degraded hosts provided that virtual machine performance is unaffected.
    • Mixed mode – Quarantine mode for moderate and Maintenance mode for severe failure (Mixed). Balances performance and availability, by avoiding the usage of moderately degraded hosts provided that virtual machine performance is unaffected. Ensures that virtual machines do not run on severely failed hosts.
    • Maintenance mode – for all failures. Ensures that virtual machines do not run on partially failed hosts.
Best options is Automated + Mixed Mode
Select Cisco UCS Provider – NOT Block Failure Conditions

How is Proactive HA working?

With settings Automatic Level – Automated and Remediation – Mixed Mode after HW Failure. Proactive HA is Entering Host Into Quarantine Mode and Migrate all VMs from ESXi with HW Failure:

After 4:10 mintes Proactive HA migrated all VMs from ESXi host with failure.

Multiple-NIC vMotion tunning 2x 40Gbps

Because for Monster SAP HANA VM (1-3 TB RAM) I tuned several AdvSystemSettings.

In the end I was able to speedup vMotion 4x times and utilize 2x flow with 40 Gbps – VIC 1340 with PE.

Inspiration was:

It is in production from 04/2018, My tuned final settings is:

AdvSystemSettingsDefaultTunningDesc
Migrate.VMotionStreamHelpers08Number of helpers to allocate for VMotion streams
Net.NetNetqTxPackKpps300600Max TX queue load (in thousand packet per second) to allow packing on the corresponding RX queue
Net.NetNetqTxUnpackKpps6001200Threshold (in thousand packet per second) for TX queue load to trigger unpacking of the corresponding RX queue
Net.MaxNetifTxQueueLen200010000Maximum length of the Tx queue for the physical NICs – toto stačí pro urychlení VM komunikace

VMware vNIC placement order not adhered to Cisco UCS configuration – How to fix it?

It is better to use Cisco UCS Consistent Device Naming CDN + ESXi 6.7, but in same casses. It is necessary fix manualy according KB – How VMware ESXi determines the order in which names are assigned to devices (2091560) .

Here is an example How to fix it:

Check current mapping

[~] esxcfg-nics -l
 Name    PCI           MAC Address       
 vmnic0  0000:67:00.0  00:25:b5:00:a0:0e 
 vmnic1  0000:67:00.1  00:25:b5:00:b2:2f 
 vmnic2  0000:62:00.0  00:25:b5:00:a0:2e 
 vmnic3  0000:62:00.1  00:25:b5:00:b2:4f 
 vmnic4  0000:62:00.2  00:25:b5:00:a0:3e 

[~] localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias list
 Bus type  Bus address            Alias
 pci       s00000002:03.02        vmnic4
 pci       s00000002:03.01        vmnic3
 pci       s0000000b:03.00        vmnic0
 pci       s0000000b:03.01        vmnic1
 pci       p0000:00:11.5          vmhba0
 pci       s00000002:03.00        vmnic2
 logical   pci#s0000000b:03.00#0  vmnic0
 logical   pci#s0000000b:03.01#0  vmnic1
 logical   pci#s00000002:03.01#0  vmnic3
 logical   pci#s00000002:03.02#0  vmnic4
 logical   pci#p0000:00:11.5#0    vmhba0
 logical   pci#s00000002:03.00#0  vmnic2

Fix transfer table for physical devices 

Bus type  Bus address            Alias
 pci       s0000000b:03.00        vmnic0 --> vmnic3
 pci       s0000000b:03.01        vmnic1 --> vmnic4
 pci       s00000002:03.00        vmnic2 --> vmnic0
 pci       s00000002:03.01        vmnic3 --> vmnic1
 pci       s00000002:03.02        vmnic4 --> vmnic2

Fix commands for physical devices 

localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic0 --bus-address s00000002:03.00 --bus-type pci

 localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic1 --bus-address s00000002:03.01 --bus-type pci

 localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic2 --bus-address s00000002:03.02 --bus-type pci

 localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic3 --bus-address s0000000b:03.00 --bus-type pci

 localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic4 --bus-address s0000000b:03.01 --bus-type pci

Fix transfer table for logical devices 

[~] localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias list
 Bus type  Bus address            Alias
 logical   pci#s0000000b:03.00#0  vmnic0 --> vmnic3
 logical   pci#s0000000b:03.01#0  vmnic1 --> vmnic4
 logical   pci#s00000002:03.00#0  vmnic2 --> vmnic0
 logical   pci#s00000002:03.01#0  vmnic3 --> vmnic1
 logical   pci#s00000002:03.02#0  vmnic4 --> vmnic2

Fix commands for logical devices 

localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic0 --bus-address pci#s00000002:03.00#0 --bus-type logical

localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic1 --bus-address pci#s00000002:03.01#0 --bus-type logical

localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic2 --bus-address pci#s00000002:03.02#0 --bus-type logical

localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic3 --bus-address pci#s0000000b:03.00#0 --bus-type logical

localcli --plugin-dir /usr/lib/vmware/esxcli/int/ deviceInternal alias store --alias vmnic4 --bus-address pci#s0000000b:03.01#0 --bus-type logical

Reboot

reboot

Crosscheck – Now we have target order

[~] esxcfg-nics -l
 Name    PCI           MAC Address       
 vmnic0  0000:62:00.0  00:25:b5:00:a0:2e 
 vmnic1  0000:62:00.1  00:25:b5:00:b2:4f 
 vmnic2  0000:62:00.2  00:25:b5:00:a0:3e 
 vmnic3  0000:67:00.0  00:25:b5:00:a0:0e 
 vmnic4  0000:67:00.1  00:25:b5:00:b2:2f

Cisco UCS supports Consistent Device Naming CDN in ESXi 6.7

Cisco introduced Consistent Device Naming in Cisco UCS Manager Release 2.2(4).

In the past I could saw that VMware vNIC placement order not adhered to Cisco UCS configuration. But his issue will not be seen in the latest ESXi updates – ESXi 6.5 U2 and ESXi 6.7 U1.

VMware vNIC placement order not adhered to Cisco UCS configuration – How to fix it?

How VMware ESXi determines the order in which names are assigned to devices (2091560)

When there is no mechanism for the Operating System to label Ethernet interfaces in a consistent manner. It becomes difficult to manage network connections with server configuration changes.

Allows Ethernet interfaces to be named in a consistent manner. This makes Ethernet interface names more persistent when adapter or other configuration changes are made.

To configure CDN for a vNIC, do the following:

This makes Ethernet interface names more uniform, easy to identify, and persistent when adapter or other configuration changes are made.

 set consistent-device-name-control cdn-name

Whether consistent device naming is enabled or not. This can be one of the following:

  • enabled—Consistent device naming is enabled for the BIOS policy. This enables Ethernet interfaces to be named consistently.
  • disabled—Consistent device naming is disabled for the BIOS policy.
  • platform-default—The BIOS uses the value for this attribute contained in the BIOS defaults for the server type and vendor.

Homelab considerations for vSphere 7

Homelab considerations for vSphere 7

Homelab considerations for vSphere 7

With the vSphere 7 Launch Event just a few days away, I know many of you are eager to get your hands on this latest release of vSphere and start playing with it in you homelab. A number of folks in the VMware community have already started covering some of the amazing capabilities that will […]

VMware Social Media Advocacy

vSpeaking Podcast Ep 150: What’s New in vSphere 7

vSpeaking Podcast Ep 150: What’s New in vSphere 7

This month VMware announced vSphere 7 touting it as the biggest innovation since the launch of ESXi. This is a prettty signifigant release. So far the virtually speaking podcast covered part of the release in two previous episodes (vSphere with Kubernetes and vSphere Lifecycle Manager in the …Read More

VMware Social Media Advocacy