Because for Monster SAP HANA VM (1-3 TB RAM) I tuned several AdvSystemSettings.
In the end I was able to speedup vMotion 4x times and utilize 2x flow with 40 Gbps – VIC 1340 with PE.
Inspiration was:
- Multiple-NIC vMotion in vSphere (2007467)
- VMworld 2017 – SER1494BU Encrypted vMotion
- vMotion multi-threading and other tuning settings
- How to Tune vMotion for Lower Migration Times?
It is in production from 04/2018, My tuned final settings is:
| AdvSystemSettings | Default | Tunning | Desc |
|---|---|---|---|
| Migrate.VMotionStreamHelpers | 0 | 8 | Number of helpers to allocate for VMotion streams |
| Net.NetNetqTxPackKpps | 300 | 600 | Max TX queue load (in thousand packet per second) to allow packing on the corresponding RX queue |
| Net.NetNetqTxUnpackKpps | 600 | 1200 | Threshold (in thousand packet per second) for TX queue load to trigger unpacking of the corresponding RX queue |
| Net.MaxNetifTxQueueLen | 2000 | 10000 | Maximum length of the Tx queue for the physical NICs – this alone speeds up VM traffic |
Explorer note: each setting in the table links to the ESXi Advanced and Kernel Settings Explorer, which shows its description, default, allowed range, the esxcli and PowerCLI commands and the history of the default across 42 ESXi releases. All four exist unchanged from ESXi 6.0 Update 3 to 9.0.1 with the defaults above; Migrate.VMotionStreamHelpers accepts 0 to 112, the three Net.* values 1 to 10000, and esxcli can set all of them. To see what else moved between your source and target release, use the compare view.
ESX 9.1 note: the default Migrate.VMotionStreamHelpers = 0 lets ESX 9.1 scale up to 7 helpers per uplink, and more than 16 helpers can crash the host during vMotion. A fixed value of 16 or lower avoids it; details in ESX 9.1 vMotion PSOD (KB 455306).
VCF 9 note: Multi-NIC vMotion needs more than one vmk_vmotion per host, and SDDC Manager in VCF 9 accepts only one (KB 407383). On VCF 9 a HANA vMotion is capped by one uplink, so the vNIC layout decides how much it disturbs the NFS log flow. Compare the layouts in SAP HANA on VCF 9: TDI network design and vNIC simulator and why 6 NICs matter for vMotion and NFS.
