When moving a virtual machine from one server to another in a data center, the mechanism for transferring CPU and memory state has been in place for nearly two decades. Ever since VMware announced VMotion in 2003, live migration has become a foundational technology of cloud operations. But when a locally attached NVMe SSD is directly assigned to a VM, the story changes entirely. There has never been a standard solution to the problem of how to migrate storage state.
According to research published at USENIX NSDI 2026, live migration of storage-optimized VMs requires statically reserving dozens of CPU cores, and latency fluctuations during migration are unavoidable. For this reason, major cloud providers have disabled live migration for storage-optimized VMs. Without being able to stop a VM, neither server maintenance nor load balancing is possible. A paradox had emerged in which the more you extract NVMe SSD performance, the more operational flexibility you lose.
On August 4, 2026, NVM Express published a specification that has the potential to change this situation. It revised all eleven specification documents at once, centered on the NVMe 2.4 Base Specification, and placed "PCIe Exported NVM Subsystem Migration" at its core.
Fifteen Years From "Fast Interface" to "Data Center Foundational Standard"
Looking back at the history of NVMe, we can trace a trajectory in which this specification, which started out as a storage protocol, has transformed into a foundational layer of data center infrastructure.
In 2011, NVMe Revision 1.0, formulated primarily by Intel's Amber Huffman and others, was born to break through the limitations of SATA/AHCI. AHCI was a protocol designed for the HDD era, constrained to a queue depth of 32 and a single queue. NVMe connected directly to the PCIe bus and adopted a design capable of handling 65,535 queues, each with a command depth of 65,535. It was the optimal solution for drawing out the parallelism of NAND flash.
In 2016, NVMe over Fabrics (NVMe-oF) was added, making it possible to bring NVMe's low latency to storage accessed over a network as well. In 2021, NVMe 2.0 was released, with the specification split into 9 documents, creating a structure in which command sets and transports could evolve independently. With this NVMe 2.4, the number of specification documents has grown to 11, extending scope to include using storage as a computational resource and as memory, via the Computational Programs Command Set and the Subsystem Local Memory Command Set.
| Period | Specification | Number of Documents | Key Additions |
|---|---|---|---|
| 2011 | NVMe 1.0 | 1 | Direct PCIe connection, 65,535 queues |
| 2016 | NVMe-oF 1.0 | 2 | NVMe over network |
| 2021 | NVMe 2.0 | 9 | Command set separation (ZNS, KV, etc.) |
| August 2026 | NVMe 2.4 | 11 | PCIe Exported NVM Subsystem Migration, PQC, Rate Limiting |
The direction of this evolution is clear. NVMe has continued to expand from "a communication procedure between the host and the SSD" into "an architectural standard that defines the storage layer of the data center."
The Burden of Virtualization That Hypervisors Have Continued to Carry
When directly assigning an NVMe SSD to a VM, the conventional choice has been either to use PCIe's SR-IOV (Single Root I/O Virtualization) or to dedicate the entire SSD to a single VM via device passthrough. SR-IOV splits a single physical SSD into multiple Virtual Functions (VFs), each of which can be assigned to a different VM. The performance overhead is nearly zero.
The problem lies in compatibility with live migration. The state of a VF assigned via SR-IOV is deeply tied to the physical hardware, and there is no standard means of reproducing that same state on a different SSD on a different server. With device passthrough, since an entire SSD is occupied by a single VM, neither sharing nor migration is possible in principle.
Another approach is for the hypervisor to intercept and emulate NVMe management commands. The hypervisor receives Admin Queue commands issued by the VM and synthesizes responses based on virtualization rules. It works, but the hypervisor's code becomes complex, and storage behavior becomes tightly coupled to the hypervisor's logic.
What NVMe 2.4 attempts is to restructure this arrangement itself.
How the "Exported NVM Subsystem" Lets the SSD Itself Handle Virtualization
The core of PCIe Exported NVM Subsystem Migration lies in shifting the responsibility for virtualization from the hypervisor to the SSD controller. Mike Allison, Senior Director at Samsung and an NVM Express Board Member, explained this design philosophy on the official blog as follows:
"With NVM Subsystem Migration, NVMe SSDs can present exported NVM subsystems that hide the complexity of the underlying hardware. Instead of interacting with physical controllers and namespaces, the host only sees exported controllers and namespaces."
The specific migration procedure is defined in seven steps:
- Create an Exported NVM Subsystem on the destination server
- Recreate the Exported Controller and Exported Namespace on the destination with the same identifiers as the source
- Apply the same configuration template to both systems
- Capture the runtime state of the source's Exported Subsystem
- Transfer and apply that state to the destination
- Migrate the VM itself
- Clean up the source's Exported resources and underlying resources
Since none of the storage identifiers change from the VM's perspective, the OS and applications can continue operating without any reconfiguration.
The "Guarantee of Being the Same SSD" Provided by Templates
The linchpin of this mechanism is the template (Exported NVM Subsystem Template). A template is a blueprint that defines which features, identifiers, and behaviors an Exported NVM Subsystem exposes to a VM. It is identified by a 128-bit UUID, and an SSD reports which UUID it supports. Between two SSDs that support the same UUID, it is guaranteed that the configuration and state of an Exported NVM Subsystem using that template can be fully migrated.
The NVMe specification defines a "Reference Exported NVM Subsystem Template." This template is designed to maximize interoperability across vendors and targets a minimal Exported Subsystem configuration consisting of a single controller and a single namespace. Vendors can extend this to create custom templates that define support for multiple controllers and namespaces, additional optional features, the content of specific log pages, and so on.
However, the richer the templates become, the lower the migration compatibility becomes. The fact that this trade-off is explicitly documented in the specification is important for ensuring implementation-level transparency.
Comparison with Conventional Approaches
| Item | SR-IOV | Device Passthrough | Hypervisor Emulation | PCIe Exported NVM Subsystem Migration |
|---|---|---|---|---|
| SSD sharing per VM | Possible (per VF) | Not possible (1 VM = 1 SSD) | Possible | Possible |
| Live migration | No standard support | Not possible | Possible (high cost) | Standardized in specification |
| Virtualization burden | SSD side (VF splitting) | None | Hypervisor side | SSD side (Exported Subsystem) |
| Direct VM Admin Queue access | Via VF | Possible | Intercepted by hypervisor | Possible (via Exported Controller) |
| Identifier portability | Depends on physical hardware | N/A | Depends on hypervisor implementation | Host-controlled (UUID template) |
Preparing for the Quantum Computing Era and Strengthened Operational Features
NVMe 2.4 also includes several notable features beyond virtualization.
Post-Quantum Cryptography (PQC) support incorporates the quantum-resistant algorithms FIPS 203 (ML-KEM), FIPS 204 (ML-DSA), and FIPS 205 (SLH-DSA), finalized by NIST in August 2024, into the NVMe protocol. It strengthens end-to-end encryption between the host and SSD to withstand decryption by quantum computers. While no practical quantum computer capable of breaking cryptography exists at present, building this into the standard prepares for future threats.
Rate Limiting is a QoS feature that lets the NVMe controller itself enforce bandwidth and IOPS caps. This enables cloud providers to define storage service tiers based on IOPS and bandwidth thresholds. Unlike conventional software-based throttling, the limits are enforced at the controller's hardware level, eliminating polling and software interrupt overhead.
Voltage Monitoring is a feature that sets overvoltage and undervoltage thresholds and records real-time alerts and historical logs. It allows power anomalies to be detected before they affect performance or reliability. Restore Manufacturing Default Settings is a feature that restores an NVMe subsystem to its factory-shipped state, used for troubleshooting and for consistent configuration management across an entire server fleet.
Amber Huffman, Chair of NVM Express, stated at the announcement: "The latest NVMe Specifications introduce foundational enablers to remain secure in the post-quantum era while streamlining device manageability and improving sustainability."
The Distance Remaining From Standardization to Implementation
Publishing the specification is only a starting point. There remain several unresolved challenges before PCIe Exported NVM Subsystem Migration is actually put to use in data centers.
First, there is SSD vendor implementation. Even if defined in the standard, if no SSDs ship with Exported NVM Subsystem functionality built into the controller firmware and reporting template UUIDs, the feature is effectively nonexistent. Support is also needed on the Linux kernel's VFIO driver side; as of August 2025, RFC patches for live migration commands have been posted to the mailing list, but they have not yet been merged.
Second, there is the matter of verifying template interoperability. The reference template is limited to a minimal configuration of a single controller and a single namespace. In actual operation, scenarios using multiple namespaces are common, and how to verify compatibility between vendor-specific extended templates remains a challenge.
Third, there is integration with Confidential Computing. NVM Express's blog states that this technology is positioned to support confidential computing, cloud-scale virtualization, and next-generation data center architectures, but concrete interoperability specifications for how VM memory encryption and storage Exported Subsystems should be combined have not yet been presented.
All of the NVMe 2.4 specification documents are available for download from the NVM Express website. The direction presented by this consortium, which has more than 100 member companies, is an attempt to shift storage virtualization from "a problem handled by host software" to "a feature natively provided by the SSD." Whether it succeeds will be judged by the controller silicon that ships over the next year or two.