On August 24, 2026, NVIDIA announced at Hot Chips 2026 that its rack-scale inference accelerator, the NVIDIA Groq 3 LPX, has entered full-scale mass production. The product's prototype had already been shown at GTC in March, when NVIDIA said it was mass-producing all seven chips that make up the Vera Rubin platform. What August added was the first third-party benchmark confirming the LPX rack is commercially ready, along with news that Nebius will be the first AI cloud provider to adopt it. NVIDIA hasn't chosen to replace GPUs with LPUs outright. Instead, its design splits the work of reading long inputs from the work of rapidly generating the next token, assigning each to a different processor.
How much can this division of labor cut latency for AI agents? The figure of 3,431 output tokens per second is eye-catching, but it doesn't account for price or power consumption. To gauge what mass production actually means, we need to separate the mechanics behind that number from the conditions the benchmark leaves out.
3,431 tokens per second isn't the speed of a full agent workflow
Artificial Analysis ran Gemma 4 31B on an NVIDIA-hosted Groq 3 LPX and measured a median of 3,431 output tokens per second with a 100,000-token input. Rather than each model's own token-counting method, the test used the shared o200k_base tokenizer. NVIDIA rounded this to 3,400 tokens per second in its announcement, framing it as roughly 4x faster than the closest public endpoint, which ran at 870 tokens per second.
A 100,000-token input reflects scenarios like coding agents reading hundreds of files or research agents carrying forward long conversation histories. The same test with a 10,000-token input produced 3,382 output tokens per second. At least for Gemma 4 31B, extending the input context tenfold barely dented output speed.
But output token speed isn't the same as task completion time. Agents read files, run external tools, and verify results between model calls. Network, storage, and tool-side processing aren't captured in the 3,431-tokens-per-second figure. In workflows where multiple inference calls run in sequence, the savings from faster generation do add up—but NVIDIA's claim that "hours of work become minutes" isn't a number verified through an end-to-end task benchmark.
GPUs and LPUs split the inference pipeline
Large language model inference has two phases with different workload characteristics: "prefill," which processes the input and builds the KV cache, and "decode," which generates output one token at a time. Prefill, which processes long inputs all at once, favors the Rubin GPU's parallel compute power and large HBM capacity. Decode, which must respond quickly to small requests, benefits more from memory bandwidth that keeps feeding data to compute units without idle time, along with control mechanisms that reduce timing variance.
The Vera Rubin NVL72 and LPX can exploit this difference in three ways. In the simplest configuration, the GPU finishes prefill and hands off the KV cache, while the LPU handles decode entirely. A more granular approach, Attention-FFN Disaggregation (AFD), has the GPU hold the KV cache and compute attention while the LPU processes the feed-forward layers or MoE experts. The two exchange intermediate data every time a token is produced.
A third option is speculative decoding, where a small draft model on the LPX generates candidate tokens first, and a larger model on the Rubin GPU verifies them in batches. NVIDIA Dynamo classifies each request and coordinates the movement of KV caches and intermediate data, so developers see it as a single inference path. The LPX design unveiled in March split inference internals into GPU-suited and LPU-suited work; the August measurements are the first to show output speed for one of these configurations.
Why 256 LPUs are bundled into a single rack
A single Groq 3 LPX rack holds 32 liquid-cooled 1U trays, each carrying 8 LP30 chips. The 256 total LPUs deliver 315 PFLOPS of FP8 compute, 128GB of on-chip SRAM, and 40PB per second of SRAM bandwidth. Scale-up bandwidth linking the chips reaches 640TB per second across the rack.
| Metric | Per rack | Per LPU |
|---|---|---|
| LPU count | 256 | 1 |
| SRAM capacity | 128GB | 500MB |
| SRAM bandwidth | 40PB/s | 150TB/s |
| Scale-up bandwidth | 640TB/s | 2.5TB/s |
The key detail here is bandwidth, not capacity. A single LPU has only 500MB of SRAM—not enough to hold a 31B model on its own. The Register estimates that storing Gemma 4 31B's weights in FP8 would require just over 31GB, meaning at least roughly 64 chips' worth of SRAM. Rather than fitting a large model onto fewer chips, LPX splits large models across many chips and relies on SRAM speed and inter-chip communication to accelerate the next token.
To run 256 chips with low latency, the LPU minimizes dynamic runtime arbitration. A compiler pre-schedules the timing of compute, memory access, and communication in units of 320-byte vectors. Each LPU has 96 inter-chip links running at 112Gbps, with data timed to arrive on a planned clock cycle. This is less an architecture built to win on peak compute and more one designed to suppress latency variance across the entire rack.
What lies beyond the "4x faster" claim
The 3,431 tokens-per-second figure is a real measurement, but it was verified on just one dense 31B model. The full configuration—what precision was used, how many chips were allocated, how many concurrent users were served—hasn't been disclosed. Price, rack power consumption, and P95 or P99 latency also remain unknown. Single-user output speed alone can't tell you whether a cloud provider's economics work out.
The "4x" figure also isn't an apples-to-apples chip comparison. In NVIDIA's chart, the competing public endpoint ran at 870 tokens per second, but The Register points out that the Cerebras-based comparison may have used only one or two chips, while the LPX side likely used at least about 64. The perceived speed as a service is comparable, but the silicon area, power, and infrastructure costs required are not accounted for.
That doesn't mean Gemma 4 31B is a meaningless small model for this purpose. Since dense models use all 31 billion parameters for every token, the workload can resemble the active-parameter count that a massive MoE model actually invokes per call. Still, MoE models select different experts for each token and must keep the full set of weights in memory. NVIDIA's claim of "up to 35x throughput per megawatt" is a projection based on placing a 2-trillion-parameter-class model on the system—not a result measured in this Gemma test.
Nebius, for its part, hasn't announced a launch date or pricing yet. The company says it will make LPX accessible through its existing Token Factory API, initially supporting a subset of models. A mass-production announcement is evidence that the hardware has advanced through manufacturing, but it doesn't yet mean developers have a cloud product they can choose today.
From a $17 billion deal to a mass-production announcement in eight months
Groq and NVIDIA signed a non-exclusive license agreement for inference technology on December 24, 2025. Groq founder Jonathan Ross, president Sunny Madra, and others moved to NVIDIA, while Groq continued to operate independently and keep running GroqCloud. The following March, NVIDIA revealed that the Groq 3 LPU and LPX would be built into Vera Rubin, and in August it announced mass production along with the first cloud adoption. Eight months passed between the deal and the mass-production announcement.
The deal was initially reported at roughly $20 billion. However, NVIDIA's annual report filed with the U.S. Securities and Exchange Commission shows a payment of $13 billion at signing, plus $4 billion payable within a year including accrued interest, for total consideration of $17 billion. NVIDIA did not acquire Groq's equity, existing products, or customer contracts.
The accounting treatment also reveals what NVIDIA was racing toward. Of the $17 billion, $14.4 billion was allocated to goodwill, primarily attributed to talent and the future development of licensed technology. The intangible assets recorded as developed technology totaled $2.5 billion, with a useful life of five years. Rather than a purchase of a finished product, the accounting disclosures suggest this was payment for quickly integrating Groq's design philosophy and talent into NVIDIA's own GPUs, networking, and Dynamo software.
Nebius is deploying LPX within its Token Factory, and Groq itself is partnering with Dell Technologies to bring Vera Rubin NVL72 and LPX to the cloud. NVIDIA's plan is to distribute LPU speed through existing cloud channels rather than keeping it locked inside its own products alone. This lets NVIDIA fold inference—where response speed per user is the product's value—into its platform alongside GPUs, which handle training and bulk processing.
What will ultimately determine how Groq 3 LPX is judged isn't the next peak-speed number. It's how far P99 latency and per-token pricing drop when large MoE models run on Nebius's live service, and how many different models the system extends to. Once those three figures are public, we'll be able to measure whether LPX—which went from a $17 billion deal to a mass-production announcement in eight months—actually shortens the practical wait times AI agents impose on their users.