DeepSeek has open-sourced DeepInfer, an inference framework that supports its large-scale AI services. It supports DeepSeek-V4.1-Flash and provides an inference engine that optimizes model execution and workload distribution for NVIDIA GPUs and Huawei Ascend NPUs. DeepSeek had previously released individual compute kernels, but this release extends to the framework that manages model execution as a whole. However, the public version centers on offline inference, and building an online service that handles requests from many users requires additional components. Key questions for anyone considering adoption are how much of the released optimization technology can actually be used, and how to evaluate the performance figures DeepSeek has presented in real operating environments.
What the Inference Engine Includes, and What Production Operation Requires
DeepInfer is software that runs trained AI models and generates outputs in response to inputs. It has been released under the Apache License 2.0 in the official GitHub repository. In the official README, DeepSeek describes it as the inference framework behind its own large-scale services, while positioning the public version as an offline inference engine with the core functions needed for large-scale service operation.
The released code includes optimized compute kernels and model execution processing, as well as parallelism features that distribute computation across multiple devices. It also supports "speculative decoding," which predicts candidate tokens in advance and verifies their correctness in batches, and "structured output," which constrains output to follow JSON Schema, regular expressions, or grammar rules. According to the README, DeepInfer was developed primarily on top of vLLM, and also draws on the designs of SGLang and TensorRT-LLM.
DeepInfer alone, however, is not enough to build an online service. DeepSeek has not included in this release the KV cache management system used across its services or the load-balancing function among multiple data-parallel inference engines. A KV cache stores part of the computation results for past inputs, reducing the amount of computation needed when the same context is processed again. It is important to distinguish between holding a KV cache inside an inference engine and a system that stores and shares caches across multiple servers.
What DeepInfer's public release covers, and what production deployment requires
| Item | What the public release provides | Work required for deployment |
|---|---|---|
| Model execution | Offline inference for V4.1-Flash and various optimizations | Prepare supported hardware and a runtime environment, and verify operation |
| Service architecture | Most engine-side functions needed to separate processing stages | Build an operational platform that separates image processing, input processing, and output generation |
| Caching and request routing | Production system design published in the official Q&A | Implement service-level KV cache management and load balancing across data-parallel replicas |
| Performance evaluation | Benchmark scripts and measurement conditions | Measure performance reflecting real token acceptance rates and load conditions |
This table was compiled by cross-checking the scope of the official README with the official Q&A explaining the inference system architecture and the benchmark measurement methods. The Q&A also describes in detail the design of the production system DeepSeek actually uses, but not every feature introduced there is included in the public code. The technology and equipment required differ between validating the inference engine itself and building a service that responds reliably to many users.
More Efficient Speculative Decoding and Less GPU Idle Time
One of DeepInfer's distinctive optimizations is more efficient verification in speculative decoding. DeepSeek's "DSpark" prioritizes verifying those pre-generated candidate tokens that are most likely to be adopted. Generating many candidates wastes verification compute if they are unlikely to be accepted. A prediction mechanism called the "Confidence Head" therefore estimates the acceptance probability of each candidate and dynamically adjusts the verification length to maximize processing efficiency.
According to the official Q&A, DeepSeek's production system generates 5 candidate tokens at a time. Rather than verifying all five every time, it selects the range to verify based on each candidate's acceptance probability and the hardware's processing capability. This is not a technique for improving the quality or capability of the text the model can generate; it is a mechanism for using the compute resources needed for token verification efficiently.
Another key feature is asynchronous scheduling, which runs CPU and GPU processing in parallel. In ordinary inference, the CPU receives the GPU's results, then performs post-processing and assigns the next computation, during which the GPU may sit idle. In DeepInfer, the CPU prepares the next task while the GPU is working on the current one, reducing GPU wait time. According to the official README, this feature is enabled by default.
An optimization called "SWA bounded replay" has also been introduced for reusing past context. It limits recomputation to the most recent 128 tokens and approximately reconstructs the KV cache for Sliding Window Attention (SWA). DeepSeek says that using this method on the decoder side, which handles output generation, can cut the total computation for input processing by roughly half. Ordinarily, a long past context would need to be recomputed to restore the cache accurately, but computing only a limited recent range reduces the load. Because the reconstructed cache differs from one produced by exact recomputation, users should understand that the method involves approximation, not only reduced computation.
Published Performance: Watch Token Acceptance Rates and Load Conditions
DeepSeek publishes DeepInfer's performance measurements in the official README. The "Prefill" stage, which reads the input, used 4 NVIDIA SM100 GPUs, and the "Decode" stage, which generates output, used 64, each measured within a single domain connected by a high-speed interconnect. When comparing against environments with different GPU counts or interconnect configurations, these measurement conditions must be taken into account.
Particularly important is how output throughput is calculated. In the official benchmark example, throughput is computed on the assumption that all candidate tokens generated by speculative decoding are accepted. The number of tokens counted as generated in one step is "1 + candidate token length" per request. Because this calculation does not reflect actual acceptance rates, the published tokens-per-second-per-GPU figure cannot be equated with the text generation speed users actually experience.
This difference also matters when evaluating the effect of DSpark's optimization. DeepSeek estimates the performance gain from dynamically changing the verification length, based on benchmark data with expert parallelism (EP) and data parallelism (DP) both set to 64. According to the company, under conditions close to production operation, where each parallel unit handles about 60 sequences simultaneously, throughput improves by roughly 4–7% compared with a fixed-length approach that always verifies 5 tokens.
The size of the gain, however, varies with the number of concurrent sequences. In the comparison table DeepSeek published, throughput under the dynamic-length method falls 1.5% at 16 concurrent sequences, but rises 5.4% at 60. In other words, verifying fewer tokens does not necessarily speed things up under every load condition. These values are also estimates calculated from the company's measurement data, and do not directly prove superior performance over other inference frameworks.
In actual deployment, you need to confirm whether the required response speed can be maintained, taking into account the number of concurrent requests and the candidate-token acceptance rate. The published benchmarks offer clues to the performance of the inference engine itself, but the throughput of an entire online service also depends on KV cache reuse efficiency and load balancing across servers. It is therefore important to re-measure under conditions close to your own usage.
Support for NVIDIA and Huawei Ascend, but Production Needs Additional Build-Out
The hardware officially supported by DeepInfer is NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The runtime requires Linux, GCC 13.0 or later, and Python 3.12, with CUDA 13 for NVIDIA environments and CANN 9.2 for Ascend environments. Because the supported platforms are clearly limited, it may not run as-is on existing GPU environments.
There are also constraints on features that ensure reproducible inference results. The function that keeps model computation results consistent even when batch composition changes is limited to NVIDIA environments and must be explicitly enabled. Furthermore, using deterministic sampling, which yields identical output from the same input tokens and sampling settings, requires disabling dynamic-length verification and specifying a fixed seed. In other words, not all of DeepInfer's features can be used simultaneously, and the settings for reproducibility and optimization must be checked.
The scale of hardware needed for production operation also differs greatly from ordinary functional testing. The official Q&A cites, as the smallest efficiency-oriented configuration, one using 64 GPUs for input processing and 64 for output generation. This assumes the company's recommended configuration of 64 or more GPUs for output generation, and is not the minimum requirement for running offline inference. Trying the model in a small environment and ensuring high processing efficiency in a large-scale user-facing service should be considered separately.
Alongside DeepInfer, DeepSeek also released "deepseek-recipe," which converts between API request formats and the token sequences the model processes. It is software that converts requests from different API formats for DeepSeek models and returns generated results in the corresponding format. Although it is a useful building block for constructing services, it does not include request load balancing or shared KV cache management. With this release, outside developers can now consult both the inference optimization techniques DeepSeek uses and concrete implementations for incorporating them into services.
The release of DeepInfer is an important step toward letting third parties verify DeepSeek's inference technology and adopt it in their own systems. However, the performance of the released inference engine does not translate directly into the performance of an entire online service. Measuring token acceptance rates and response times under real traffic, and evaluating the efficiency of the whole system including KV caching and load balancing, will likely shape future adoption decisions.