On August 26, 2026, Alibaba's Qwen team previewed the design it plans to use for the next-generation Qwen4, releasing it as "Qwen3.8-Flash-Next." The "3.8" in the name denotes a generation, not an 8-billion-parameter-class small model. The base model is a 125-billion-parameter Mixture-of-Experts (MoE) system, further equipped with a 51-billion-parameter N-gram embedding table. Even so, only 6 billion parameters are activated to process a single token. What the Qwen team aimed for was not to uniformly shrink the model, but to separate where computation happens, where memory is stored, and how long histories are searched.
One-Ninth Is the Product of Active Parameters and Training Tokens
The technical report uses Qwen3.7-Plus-Base as its comparison point, a model that activates 17 billion parameters per token out of a total of 397 billion. Qwen3.8-Flash-Next-Base reduces the active parameter count to 6 billion and also cuts the number of training tokens to roughly one-third. Multiplying the roughly one-third reduction in active parameters by the roughly one-third reduction in training tokens yields a training FLOPs reduction of approximately one-ninth.
It should be noted that this is not a controlled comparison where only the architecture was swapped under otherwise identical conditions. Even so, across 14 pretraining benchmarks, Qwen3.8-Flash-Next-Base outperformed Qwen3.7-Plus-Base on 8, and kept the gap on the remaining 6 to within 2.6 points at most. The key takeaway from these results is that the drop in capability was kept small relative to the scale of the computational reduction.
What directly constitutes the ninefold FLOPs comparison is the reduced active parameter count and training token count enabled by the MoE. Beyond that, the Qwen team also addressed three additional cost categories: long-context inference computation, memory access, and training stability.
| Change | Cost reduced, or capability enhanced |
|---|---|
| Hybrid of GDN and QSA | Compresses long histories and searches only the necessary regions |
| Gated Residual | Dynamically reads from and writes to four residual pathways |
| N-gram embedding | Increases local pattern memory capacity with almost no increase in matrix operations |
| Selective use of Muon and AdamW | Stabilizes training according to each parameter's role |
These four changes are not a breakdown of the ninefold reduction in training FLOPs. While the MoE reduces the matrix operations per token, QSA and the N-gram embedding reduce post-training inference computation and memory access. The revisions to Gated Residual (GR) and the optimization method serve to keep massive training runs stable to completion.
GDN Memorizes, QSA Searches Long Histories
In Qwen3.8-Flash-Next, three out of every four layers use Gated DeltaNet (GDN). Because GDN sequentially compresses past information into a fixed-size state, it does not need to retain a Key-Value Cache for every token even as the context grows longer. However, compressing into a finite-size state weakens the ability to directly recall specific points from the past as-is. To compensate, the remaining one layer handles Attention, which can access the entire context.
The predecessor, Qwen3-Next, also mixed GDN with regular Attention at a 3-to-1 ratio. What's different this time is that this Attention has been replaced with Qwen Sparse Attention (QSA). Rather than computing the importance of each token individually, QSA groups multiple tokens into micro-blocks. A lightweight indexer selects candidates at the block level, and Attention concentrates its computation on the regions of high relevance.
In kernel tests at 1M tokens, prefill was 7.6 times faster and decode 4.9 times faster compared to dense Grouped Query Attention. However, these are figures for the Attention module alone. Prefill was measured with 16K-token chunks and a batch size of 1, while decode was measured with a batch size of 4 and 3-stage Multi-Token Prediction. There is no guarantee that shorter inputs, different GPUs, or the overall model's response time would speed up by the same factor.
Nor can it be said that QSA sacrificed retrieval accuracy for speed. According to the technical report, RULER scores in the 512K-1M range rose from 90.08 to 93.00 compared to regular Attention. MRCR, which searches for 8 pieces of information, also improved from 30.66 to 40.53 at 512K, and from 20.71 to 26.44 at 1M. That said, switching directly from dense Attention causes performance to drop, so the model is adapted to sparse retrieval through a brief period of joint training. QSA is not a drop-in replacement part for existing models.
Placing 51 Billion Parameters Outside the GPU
The N-gram embedding, which accounts for roughly 30% of the total capacity, behaves differently from a typical MoE expert. Whereas a standard embedding looks up a table based on the current single token, the N-gram version determines what to reference based on the local context including several preceding tokens. It's easiest to think of this as "local pattern memory," giving dedicated representations to frequent phrases and short patterns.
Since the reference target can be computed in advance, the 51-billion-parameter table does not need to be kept resident in GPU memory and can instead be prefetched asynchronously from Host Memory. The model uses this table only once, near the second layer. Rather than multiplying by a huge matrix every time, it simply retrieves the necessary rows, allowing capacity to be added with almost no increase in per-token FLOPs.
On the other hand, increasing the parameter count did not scale proportionally with capability gains. In experiments where the N-gram vocabulary was increased from 20 times to 200 times the baseline, training loss decreased monotonically, but downstream benchmarks saturated partway through. Moreover, an architecture that places components outside the GPU also requires sufficient RAM, adequate PCIe or NVLink bandwidth, and an inference implementation that can hide the prefetching. This is not a claim that a model with the equivalent of 176 billion parameters in weights can be easily run on a typical single GPU.
Four Residual Pathways and Muon Push the Limits of Training
In deep neural networks, each layer keeps adding information to the same residual stream. As the number of layers increases, early features become mixed with information from later stages, and extremely large activation values can sometimes disrupt training. The Qwen team's Gated Residual (GR) widens the residual stream into four pathways and reads from them using an element-wise gate that depends on the current content. The amount written back is also controlled separately for each branch, and the residual state can be kept in FP8.
Muon was adopted for optimization. However, this does not mean all parameters are updated with Muon. Muon is used for weights that function as two-dimensional linear mappings, such as those in Attention, GDN, and MoE experts, while AdamW is retained for embeddings, the MoE router, and the low-rank components of GR. Matrices that are implementation-wise combined, such as QKV or SwiGLU, are also split into semantically distinct parts before orthogonalization.
When the architecture and optimizer change, the appropriate learning rate and batch size change as well. The Qwen team re-fit the scaling law and chose larger values than before. A warmup that gradually increases from a small batch size did not improve final performance and required 18.8% more optimizer steps for the same token budget, so it was removed. Even under a stress test with a 4x learning rate, the new configuration remained stable, and the team reported that no loss spikes occurred during the actual large-scale training run. In addition to inference costs, designs that keep massive training runs from stalling are also part of the cost reduction.
Open Weights and the 1M-Capable API Are Not the Same Product
The publicly released Qwen3.8-Flash-Next handles 262,144 tokens by default and can be extended to 1M tokens using YaRN. The official SGLang and vLLM self-hosting examples also assume a distributed setup using tensor parallelism. The license on Hugging Face is the Qwen Community License 1.0, not Apache 2.0. The ability to download the weights and being distributed under a conventional open-source license need to be considered separately.
"Qwen3.8-Flash," offered through QwenCloud, is the production version that standardizes a 1M context and built-in tools. The current product page lists a maximum input of 991K, a maximum output of 131K, and a maximum reasoning length of 262K. The price per million input tokens is listed as $0.16 in the announcement blog post, but as $0.15 on the product page as of August 27. The output price is $0.47 in both places. No reason has been given for the discrepancy in input pricing.
The published benchmarks also have their own comparative boundaries. CoWorkBench and RecreationBench are Qwen's own proprietary evaluations, and the figures for competing models include blanks or officially published values from those companies. This is not an apples-to-apples leaderboard measured under exactly the same harness as models like Claude Opus. When choosing between the self-hosted and API versions, it is more reliable to measure end-to-end throughput on the target hardware, Host Memory occupancy, and long-context accuracy, rather than relying on top benchmark scores.
If Qwen4 inherits this design, the yardstick for evaluating model competition will move even further away from total parameter count. Whether the 6-billion active parameter count, the amount of context QSA selects, and the bandwidth needed to hide the 51-billion-parameter table outside the GPU all mesh well together in real-world operation will determine how far this "one-ninth" reduction actually translates into product cost savings.