DeepInfer: what the inference engine does—and what production serving still needs.
DeepInfer is DeepSeek’s open-source inference engine for DeepSeek-V4.1-Flash. It exposes production-oriented execution and scheduling techniques, but it is not by itself a complete public API service.
An engine for efficient model execution, not the whole serving control plane.
DeepInfer combines optimized kernels, model and sampler execution, attention data parallelism, mixture-of-experts expert parallelism and load balancing, speculative decoding, sliding-window-attention replay, asynchronous scheduling, deterministic modes, and structured outputs. The initial release supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.
The important boundary is explicit: the release provides offline inference and core serving-engine capabilities. DeepSeek says a complete service also needs a KV-cache system, vision-encoder/prefill/decode disaggregation, and load balancing across data-parallel replicas. That boundary is more useful for system-design preparation than treating the repository as a turnkey endpoint.
Where DeepInfer fits in a production inference system
Request and protocol layer
Tokenization, API translation, admission control, routing, and observability sit around the engine. DeepSeek describes DeepSeek Recipe as a separate protocol and token-processing component.
Vision encoder and prefill
Disaggregation lets each stage scale for different work. Prefill is compute-heavy and determines time to first token; prefix-cache hits can avoid repeated prompt work.
Decode
Decode is bandwidth- and synchronization-sensitive. Wide expert parallelism, continuous scheduling, and speculative decoding target higher output throughput without losing the latency budget.
Cache, balancing, and fleet control
A complete online service still needs KV-cache storage, prefill/decode disaggregation, replica load balancing, autoscaling, failure handling, and service-level controls.
Four questions that expose the real design trade-offs
- Why separate prefill and decode?
- Their compute, memory, latency, and scaling profiles differ. Independent pools make time-to-first-token and time-between-tokens separately controllable, but add routing and cache-transfer complexity.
- What does asynchronous scheduling remove?
- It overlaps CPU scheduling and post-processing with device execution so the accelerator does not wait between decode steps.
- Why combine expert and data parallelism?
- Expert parallelism spreads mixture-of-experts computation; data-parallel replicas add capacity. The hard part becomes keeping experts and requests balanced as traffic changes.
- Why is the open-source engine not a complete serving stack?
- The released engine covers model execution and scheduling, while production service components such as durable KV-cache management and fleet-wide load balancing remain deployment responsibilities.
Benchmarks are evidence about one deployment shape
DeepSeek publishes prefill and decode Pareto curves measured on its stated hardware and workload. Use them to reason about throughput-versus-latency trade-offs, not as a universal capacity promise. Before comparing systems, hold constant the model, quantization, accelerator, batch and sequence distribution, cache-hit rate, latency target, and whether the result is offline or online.
The deployment Q&A also describes a much larger internal system—including long-lived distributed KV cache, stage-specific load balancers, autoscaling, and fault handling. Those descriptions explain the production architecture; they do not mean every component is included in the open-source repository.
Official DeepSeek sources
- DeepInfer repository ↗Implementation, supported hardware, prerequisites, examples, benchmarks, and release boundaries.
- DeepSeek inference deployment Q&A ↗Production architecture, expert and data parallelism, prefill/decode disaggregation, KV-cache storage, and load balancing.