On July 21, 2026, NVIDIA published details of the inference-focused architecture behind its next-generation GPU, "Rubin." While the 336 billion transistors and up to 50 PFLOPS of NVFP4 compute performance grab attention, what stands out most in this deep dive is the mechanism for reducing the time compute units spend waiting on data or synchronization. NVIDIA has reexamined everything from Mixture-of-Experts (MoE) weight transfers to Attention, dependent kernels, and inter-GPU communication over NVLink as a single, continuous pathway. For agentic AI handling long contexts, the proportion of raw GPU speed that actually converts into token generation speed is what determines deployment cost.
Cutting Wait Time Takes Priority Over 50 PFLOPS
Rubin integrates two compute dies into a single package via the NVIDIA High-Bandwidth Interface (NV-HBI), packing in 224 Streaming Multiprocessors (SMs) and 896 Tensor Cores. The third-generation Transformer Engine handles sparse NVFP4 inference performance of up to 50 PFLOPS. Memory consists of up to 288GB of 12-layer HBM4, delivering 22TB/s of bandwidth — 2.8 times the 8TB/s found in Blackwell and Blackwell Ultra.
The reason bandwidth and capacity have both increased comes down to how bottlenecks shift depending on the inference stage. Context processing, which handles input in bulk, is compute-intensive. Decode, which generates tokens one at a time, repeatedly reads model weights and the Key-Value (KV) cache, so whether memory bandwidth can be fully utilized tends to determine speed. The 288GB capacity is used to keep larger models and KV caches resident on the GPU, reducing the need to offload to external memory.
NVIDIA states that on its internal 2-trillion-parameter MoE workload, the Vera Rubin platform showed up to 10 times the agentic throughput per unit of energy compared to Blackwell. This is not a measurement showing that every workload on Rubin alone becomes 10 times faster. It's a platform-level comparison combining the Vera CPU with the Rubin NVL72, and the gap will shift depending on model configuration, parallelization strategy, and response-time requirements.
Compute Units That Don't Stall on MoE and Long Context
In MoE, only a subset of Experts selected per input token are activated. As the number of Experts grows and weights are distributed across multiple GPUs, managing which weights need to be moved from where into shared memory also becomes more complex. Blackwell's Tensor Memory Accelerator (TMA) held a separate descriptor in memory for each Expert. In Rubin, Tensors with the same layout share a single common descriptor, and the memory pointer and stride can be overwritten at runtime within the TMA instruction itself.
This inline descriptor update doesn't speed up the transfer of the weights themselves. Rather, it's a mechanism for reducing the metadata management and address calculation overhead that accumulates as the number of Experts grows, lightening the load before work is handed off to the Tensor Cores. Even though MoE keeps computation in check by activating only a small number of Experts, that advantage erodes if time is spent tracking down where weights reside. Rubin closes that gap at the instruction level.
For matrix multiplication, NVIDIA doubled the amount of K-dimension work a Tensor Core can process per clock. K is the inner dimension shared between the two matrices being multiplied, and in the GEMM example NVIDIA presented, a K-direction loop that required 4 iterations on Blackwell now takes only 2 on Rubin. In Tensor Parallel processing — where outputs are split across many GPUs but the K dimension remains large — this reduces loop-control overhead, making it easier to keep the Tensor Cores busy during both context processing and decode.
For long-context Attention, two countermeasures have been introduced. Rubin compresses intermediate Attention scores into a structured 2:4 sparse format, passing only the non-zero values and metadata downstream. This reduces the data that Softmax and the subsequent matrix multiplication need to handle, while still delivering a dense output to the rest of the model's processing pipeline as before. Additionally, the exponential computation used by Softmax has been sped up by 2x for FP32 and 4x for BF16/FP16 compared to Blackwell. For BF16/FP16, that's also a 2x improvement over Blackwell Ultra.
The Kernel Gap Extends All the Way to NVLink
In inference, one kernel produces an Activation, and the next kernel picks it up to continue computation. Blackwell's programmatic dependent launch made it possible to start downstream work earlier, but a waiting period remained until the needed data was fully ready. Rubin resolves dependencies at the tile or thread-block level, launching downstream work as soon as the upstream portion has finished writing — rather than waiting for all blocks to complete. This design tightens the GPU's timeline and shrinks the gaps between kernels.
The same problem shows up within the rack, inside the Vera Rubin NVL72, which connects 72 Rubin GPUs. Since model weights, the KV cache, and synchronization messages all travel back and forth over NVLink 6, confirming completion after sending data also lands on inference's critical path. NVLink 6 bandwidth has grown to 3,600GB/s per GPU, and NVLink-C2C connecting CPU and GPU has increased to 1,800GB/s — but wider bandwidth doesn't automatically eliminate the synchronization steps themselves.
Counted writes, introduced in Rubin, simplify synchronization for NVLink communication initiated directly from a GPU kernel. On Blackwell, a data write had to be followed by a memory barrier, a receive acknowledgment, and an atomic flag. In Rubin, that entire sequence of synchronization steps is replaced with a single counter update on the receiving GPU, cutting down on coordination traffic and wait time. This change is less about improving peak throughput on the send path and more about keeping compute from stalling in scenarios where all 72 GPUs are exchanging frequent, small messages.
The Conditions Behind the Up-to-10x Figure in Real Hardware
Rubin's improvements also extend to power design at the rack level. The Vera Rubin NVL72 is built from 72 Rubin GPUs and 36 Vera CPUs, unifying liquid cooling, power control, and NVLink switches into a single system. According to NVIDIA, Intelligent Power Smoothing reduces average power consumption by roughly 10% and 50-millisecond peak power by roughly 20%, compared to conventional approaches. With DSX MaxLPS, which controls power across an entire data center, up to 40% more GPUs can be housed within the same power envelope, with minimal impact on workload performance, according to NVIDIA.
Here too, the various multipliers need to be considered separately. The 4x figure applies to BF16/FP16 exponential-computation throughput in Softmax; the 2.8x figure applies to peak HBM4 bandwidth; and the up-to-10x figure comes from a platform-level measurement using an internal 2-trillion-parameter MoE workload. These numbers cannot simply be multiplied together to estimate application-level performance. Real-world differences will first depend on the number of Experts, context length, and batch size. How Tensor Parallel work is partitioned and how frequently GPUs communicate with each other will also shape the outcome.
NVIDIA says Rubin is now in mass production, with partner products expected in the second half of 2026. The number worth verifying once it's in the wild isn't the 50 PFLOPS figure itself, but how well per-user token speed can be maintained within a given response-time and power budget. If independent measurements across a range of real-world models confirm these reductions in wait time, Rubin's architectural changes will translate directly into lower per-token costs for inference infrastructure.