What’s New in vSphere 8 for vMotion: vMotion Notifications, Unified Data Transport and vGPU Improvements

vSphere 8 vMotion features explained: application notifications (HW v20), Unified Data Transport for cold migration, and vGPU changes in 8.0 U2/U3.

Short answer: vSphere 8.0 brought two vMotion features. vMotion notifications tell an application inside the guest that a migration is about to happen, so it can prepare. Unified Data Transport (UDT) speeds up cold migration and cloning of powered-off VMs by sending the data over the vMotion protocol instead of NFC. Later 8.0 updates added vGPU improvements: a per-VM stun time limit in 8.0 U2 and zero-copy vGPU migration in 8.0 U3.

vSphere 8 vMotion features at a glance

FeatureReleaseWhat it solves
vMotion notifications8.0 (VM hardware version 20)Latency-sensitive or clustered apps that can’t tolerate an unannounced stun
Unified Data Transport8.0Slow cold migration and clone of powered-off VMs over NFC
vGPU stun time limit8.0 Update 2Stops a vGPU VM from being migrated to a host whose estimated stun time exceeds the limit
vGPU zero-copy8.0 Update 3Faster vGPU migrations, using up to 100 Gbps

Sources: vSphere 8.0 release notes (hardware version 20 includes “vMotion App Notification”), vCenter 8.0 U2 release notes, and ESXi 8.0 U3 release notes.

vMotion notifications for latency-sensitive applications

Most workloads never notice a vMotion. Some do: trading or real-time apps that break on a short stun, and clustered databases that would rather fail over first. With notifications, the application registers through VMware Tools, gets a start event, does its preparation (drains, quiesces or fails over) and acknowledges. The migration then runs, and the app gets an end event. The app can delay the migration up to a timeout, but it can’t refuse it.

Requirements and configuration

  1. vSphere 8, VM hardware version 20, and VMware Tools or open-vm-tools 11.0 or later in the guest.
  2. Per VM, set the advanced setting vmOpNotificationToAppEnabled to TRUE. It is off by default.
  3. Per host, set vmOpNotificationToApp.Timeout (in seconds, greater than 0) through the vSphere API. For example, 300 means 5 minutes.
  4. In the guest, the application uses vmtoolsd to register and handle events.
vmtoolsd --cmd "vm-operation-notification.register {\"appName\": \"demo\", \"notificationTypes\": [\"sla-miss\"]}"

The other vmtoolsd commands are vm-operation-notification.unregister, .list, .check-for-event and .ack-event. The application polls for events and acknowledges them. One side effect matters for design. DRS excludes VMs with notifications enabled from automatic load-balancing migrations, but still moves them when a host enters maintenance mode. Details are in the Broadcom paper vSphere vMotion Notifications.

Unified Data Transport (UDT) for cold migration

Live vMotion and Storage vMotion of running VMs use the optimized vMotion protocol. Before vSphere 8, a powered-off VM, or the non-active snapshot disks of a running VM, was copied with Network File Copy (NFC), which is much slower. UDT keeps NFC as the control channel and moves the data itself over the vMotion protocol.

  1. Enable the Provisioning service on a VMkernel adapter on the source and destination hosts. You can reuse the vMotion VMkernel adapter.
  2. If vMotion uses the dedicated vMotion TCP/IP stack, you can’t add Provisioning to that adapter. Create a separate VMkernel adapter for it.
  3. Cold migrations and clones then use UDT automatically. Running VMs keep using standard vMotion.

If you expect UDT and don’t see a speed-up, check whether the VM was actually powered off. Broadcom KB 442756 confirms that UDT isn’t used for migrations of powered-on VMs. See the UDT technical paper for the protocol comparison. For host-side networking commands, see esxcli commands for ESXi 8.0.

vGPU vMotion improvements in 8.0 U2 and U3

Since 8.0 U2 you can set a stun time limit per NVIDIA vGPU VM. vCenter then won’t place or migrate the VM on a host or network whose estimated maximum stun time is higher than the limit. 8.0 U3 added zero-copy for vGPU migrations, using up to 100 Gbps. Both are aimed at GPU-accelerated workloads, where a long stun hurts the most.

Further reading

Author: Daniel Micanek

Senior Service Architect, SAP/TDL Platform Services Team at Vivicta | vExpert ⭐⭐⭐⭐⭐⭐⭐ | Broadcom Knight: vSphere Kubernetes Service, VCF Networking (NSX), VCF Automation | Broadcom Software Knight: vSphere | Cisco Champion. Notes from running about 1,000 ESXi hosts on Cisco UCS, VCF 9 and SAP HANA on vSphere.