NVMe Memory Tiering: vSphere 8.0 U3, ESX 9.0 and 9.1 Guide

Short answer: NVMe Memory Tiering lets ESX use a local, dedicated NVMe device as a second memory tier (Tier 1) behind DRAM (Tier 0). Only VM memory is tiered. It was a tech preview in vSphere 8.0 U3 (default DRAM:NVMe 4:1), became GA in ESX/VCF 9.0 (default 1:1, DRS aware, hardware RAID, encryption), and ESX 9.1 adds the esxcli memtier namespace, automatic partitioning, activation without a reboot and software mirroring. It works well when active memory stays under 50% of DRAM.

Table of Contents

What memory tiering does

The host presents DRAM plus the NVMe tier to VMs as one contiguous memory pool. The ESX memory manager tracks how recently and how often each VM page is accessed. Frequently used pages stay in DRAM and cold pages go to NVMe. Kernel memory is never tiered. Tiering is off by default.

The 9.1 best-practices page describes the placement behaviour. Cold pages start moving to NVMe when host memory consumption reaches roughly 80% of DRAM. After that, the algorithm keeps scanning VM memory in batches and moves pages between tiers at the end of each cycle. How fast it scans depends on memory pressure. So a host whose VMs fit in DRAM will show little or no Tier 1 usage.

This is not swap. Tiering picks pages by age and activity and uses a dedicated local device, while host swap picks pages crudely and often sits on shared storage. The 9.1 guidance also says not to combine tiering with memory overcommit.

Version status: 8.0 U3 vs 9.0 vs 9.1

	vSphere 8.0 U3	ESX / VCF 9.0	ESX / VCF 9.1
Status	Tech preview (KB 311934)	GA, production supported	GA
Default DRAM:NVMe	4:1 (`Mem.TierNvmePct` = 25)	1:1	1:1 (`tier-size-pct` 100)
Ratio range	1–400%	docs recommend 100% or lower	1–400%
Tier partition	Manual, `esxcli system tierdevice create`	Manual, `esxcli system tierdevice create`	Created automatically
Enable	Kernel boot option + reboot	`VMkernel.Boot.memoryTiering` + reboot	`esxcli memtier enable` or Configuration Manager, maintenance mode, no reboot
Redundancy	No RAID recommended, one device per host	Hardware RAID (VROC, tri-mode)	Software mirroring (2 devices) or hardware RAID
Encryption	Not supported	Host or VM level	Host or VM level
DRS	Not recommended	Tiering-aware placement	Tiering-aware placement
Mixed clusters	Not supported	Supported	Supported

The 8.0 U3 tech preview guide attached to KB 311934 also recommended VDI-type workloads only, no vSAN in the same cluster, and disabled large pages by default. Treat 8.0 U3 as a lab feature and plan production on 9.x.

Requirements and limits

Item	Requirement (9.0 / 9.1 TechDocs)
Versions	vCenter and ESX 9.0+ (9.1 features need 9.1 on both)
Device class	vSAN-cache-class NVMe from the Broadcom Compatibility Guide, vSAN SSD: Endurance Class D (≥ 7300 TBW), Performance Class F or G. The 9.1 docs add Mixed Use, 3 DWPD or higher
Attachment	Local only. No NVMe-oF or Ethernet-attached devices
Partition	Dedicated device, all other partitions deleted. Tier partition max 4 TB
Sharing	Not usable if it is already a vSAN cache/data disk or VMFS datastore (9.1: remove from the disk group first)
Host	No Intel Optane PMem, no NVDIMM-N. Maintenance mode for any change
Unsupported operations	Suspend to memory. NVMe hot-plug / prepare-to-remove. Quick Boot (9.1)
Sizing	NVMe used = the smaller of (DRAM × ratio) and the partition size. Keep active memory ≤ 50% of DRAM

VM types that are excluded

ESX 9.0: these VMs cannot power on on a tiered host: latency sensitivity high, SEV / SGX / TDX, Fault Tolerance, VMs with more than 1 TB and 128 vCPUs, and nested virtualization (vhv.enable). These are kept in DRAM automatically: 1 GB pages, preallocated memory, PCI passthrough / UPT, and vQAT.

ESX 9.1: low-latency, confidential, FT, large and nested VMs can now power on, but they are still left out of tiering and stay in DRAM. VBS guests are the exception because they can opt in with sched.mem.enableNestedTiering = TRUE. The 9.1 VCF blog says nested VMs participate in tiering, which is broader than TechDocs, so go by the documentation. For VMs above 512 GB and 64 vCPUs, the 9.1 docs recommend turning tiering off and using a full reservation.

Configure on vSphere 8.0 U3 (tech preview)

esxcli system maintenanceMode set --enable true
esxcli storage core adapter device list
# delete any existing partitions first (partedUtil, KB 1036609)
esxcli system tierdevice create -d /vmfs/devices/disks/<nvme_device>
esxcli system tierdevice list
esxcli system settings kernel set -s MemoryTiering -v TRUE
# optional ratio, default 25 (= 4:1)
esxcfg-advcfg -s 50 /Mem/TierNvmePct
reboot

The only option tierdevice create takes is -d / --nvmedevice, and tierdevice delete takes the same one. An older version of this article showed extra vendor/ID arguments, which were wrong.

Configure on ESX 9.0

Put the host in maintenance mode, delete all partitions on the NVMe device (KB 323144 / KB 1036609), then run esxcli system tierdevice create -d /vmfs/devices/disks/<nvme_device> and check it with esxcli system tierdevice list. Device preparation can’t be done from the vSphere Client in 9.0.
Enable tiering. Cluster: Configure → Desired State → Configuration → Draft → esx → vmkernel → options, set memory_tiering to True. Host: Advanced System Settings, set VMKernel.Boot.memoryTiering to true. Both need a cluster managed by a single image with vSphere Configuration Profiles. Broadcom’s 9.1 memtier blog confirms that 9.0 still uses the esxcli system settings kernel namespace from the CLI.
Optionally set Mem.TierNvmePct for the ratio (25 = 4:1), Mem.EncryptTierNVMe = 1 for host-level encryption, or sched.mem.EncryptTierNVMe = TRUE for one VM. Reboot.

To exclude a single VM, power it off and set sched.mem.enableTiering = FALSE.

Configure on ESX 9.1: the esxcli memtier namespace

In 9.1, every tiering operation is in one namespace. The partition is created automatically, and enabling needs maintenance mode but no reboot. Commands and options below are from the ESXCLI 9.1.0 reference:

esxcli memtier device list --available        # -a; --configured / -c
esxcli memtier enable -d <device_1>
esxcli memtier enable -d <device_1>,<device_2>   # software mirror
esxcli memtier enable -d <device_1> -c t -r 50    # encryption, 2:1 ratio
esxcli memtier config get                     # -d / -e / -c / -r for single values
esxcli memtier config set --tier-size-pct 25  # 1-400, % of DRAM
esxcli memtier config set --encryption true
esxcli memtier config set -d <new_device>,<mirror_device>
esxcli memtier status get
esxcli memtier disable                        # keeps tier partition
esxcli memtier disable -d                     # --delete-devices
esxcli memtier config delete                  # removes config and partition

For a cluster, go to Configure → Desired State → Configuration → Draft → esx → memtier → nvme. Set enable and pick a device for each host (by BIOS UUID). You can also set a mirror_device, encryption and tier_size_pct. Configuration Manager puts the host into maintenance mode by itself. The mirror device has to be at least as large as the primary. If config get shows Enable true but status get reports Disabled, activation failed, so check the logs and alarms.

In 9.1, if you set sched.mem.enableTiering = FALSE on a VM, also give it a full memory reservation. Without one, its pages can end up in host swap, which is slower than the NVMe tier.

Monitor it

9.0: Host → Configure → Hardware → Overview shows Tier 0 (DRAM), Tier 1 (NVMe) and Total. The cluster Hosts tab has a Memory Tiering column, which reads “Software” for tiered hosts.
9.1: host and cluster Summary Memory cards show tier state and Tier 1 consumption, and there are tiering-specific performance charts. From the CLI, use esxcli memtier status get.
Capacity check: compare active memory with DRAM. A host whose active memory is under 50% of DRAM is a good candidate.

Operational notes

Expect vMotion to take roughly 1.5–2x longer for tiered VMs, because pages have to be read back from NVMe.
If a single-device tier fails (9.1 docs), VMs that touch a page on the failed device power off. With vSphere HA restart policies they can be restarted on another host. DRAM-only VMs keep running. This is the reason for mirroring or RAID.
If you turn tiering off on a host, the cluster loses that capacity, and in an overcommitted cluster some VMs may then fail to power on.

Lab notes

NVMe Tiering with AMD Ryzen CPU workaround (lab-only, unsupported)
NVMe Tiering with Nested Virtualization in VCF 9.0 (lab-only; nested VMs were blocked on tiered hosts in 9.0)
Further reading (lab tricks, not supported limits): William Lam, NVMe Tiering in vSphere 8.0 Update 3

Sources

Broadcom KB 311934, Memory Tiering over NVMe in vSphere 8.0 U3 (and the attached Tech Preview Technical Guide rev. 1.2)
TechDocs 9.0: Memory Tiering over NVMe, Configuration, VM profiles, Host memory size, Troubleshooting
TechDocs 9.1: Memory Tiering over NVMe, Configuration, Best practices, NVMe failure scenarios
VCF 9.1 release notes, What’s new in vSphere
ESXCLI reference: esxcli memtier (9.1.0), esxcli system (9.0.0), esxcli system (8.0.3)
VCF blog: Memory Tiering in VCF 9.0, enhancements in VCF 9.1, the esxcli memtier namespace

Related on vdan.cz

Author: Daniel Micanek

Senior Service Architect, SAP/TDL Platform Services Team at Vivicta | vExpert ⭐⭐⭐⭐⭐⭐⭐ | Broadcom Knight: vSphere Kubernetes Service, VCF Networking (NSX), VCF Automation | Broadcom Software Knight: vSphere | Cisco Champion. Notes from running about 1,000 ESXi hosts on Cisco UCS, VCF 9 and SAP HANA on vSphere. View all posts by Daniel Micanek