DeepInfer README

项目的 README,来自 github.com/deepseek-ai/DeepInfer。合并为一个 markdown 文件:README.md。

DeepInfer

DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale.

Supported models:

Supported features:

Feature details and configuration
  • Optimized kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels.
  • DSpark and confidence-scheduled verification: accelerate decoding with draft tokens and use confidence estimates to schedule verification. Set DEEPINFER_ENABLE_MTP_VARLEN=1; DEEPINFER_USE_NEXTN_DECODE controls the maximum draft length (default: 5).
  • Decoder SWA bounded replay: reduce decoder prefill work by replaying only a bounded suffix. Set DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact and DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140 for the default V4.1 settings; the retained suffix must cover the SWA window plus the maximum backtracking length.
  • Asynchronous scheduling: overlap CPU batch scheduling with device execution. Enabled by default.
  • Batch-invariant inference: keep model results identical across different batch compositions. Enable with DEEPINFER_BATCH_INVARIANT=1 (NVIDIA only).
  • Deterministic sampling: produce identical outputs for identical input tokens and sampling parameters. Set DEEPINFER_BATCH_INVARIANT=1 DEEPINFER_ENABLE_MTP_VARLEN=0 and pass a fixed seq_seed to SamplingParams (NVIDIA only).
  • Structured outputs: constrain generation to a regex, JSON schema, or EBNF grammar. Pass so_args={"schema_type": ..., "schema": ...} to SamplingParams; see examples.

Supported hardware:

Required for a complete inference service (not included in this release):

Getting started

Prerequisites

Build from source

Before building DeepInfer, please create and activate an empty Python 3.12 environment.

# Use `ascend` instead for the Ascend platform
# Keep it exported in the environment where you run DeepInfer
export DEEPINFER_ACCELERATOR=nvidia
./build.sh

Offline inference

The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, is downloaded into the Hugging Face cache on the first run; set DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 to run from a local copy instead.

python examples/offline_inference.py

See examples for details.

Performance benchmarks

Prefill Decode
Prefill: non-cached input throughput vs. latency Decode: output throughput vs. per-user speed

Measured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain. To reproduce the benchmarks, see run_sweep.sh.

Evaluation benchmarks

See eval/README.md.

License

DeepInfer is released under the Apache License 2.0. See LICENSE for details.

Acknowledgments

DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. We thank their developers.

Citation

@misc{deepinfer2026,
  title = {DeepInfer: High-performance, large-scale inference framework for DeepSeek models},
  author = {DeepSeek-AI},
  year = {2026},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/deepseek-ai/DeepInfer}},
}

DeepInfer (deepseek-ai/DeepInfer)

正在搜索数据库…

26.2K ·GitHub
这是什么?DeepInfer 是 DeepSeek 用于运行其大规模生产推理的高性能推理框架的开源版本,目前支持 DeepSeek-V4.1-Flash 模型。
DeepInfer 是完整的服务栈,还是只是推理引擎?它只是离线推理引擎本身,不是开箱即用的服务。完整的生产部署仍需要 KV 缓存系统和负载均衡等组件,而本次发布并不包含这些。
我该怎么让它运行起来?创建并激活一个空的 Python 3.12 环境,从源码构建 DeepInfer,然后运行离线推理;默认检查点(DeepSeek-V4.1-Flash-1017)会在首次运行时从 Hugging Face 缓存中拉取。
构建之前需要先安装什么?需要 Linux、GCC 13 或更高版本以及 Python 3.12。如果使用 NVIDIA GPU,还需要 CUDA 13;如果使用华为昇腾 NPU,则需要 CANN 9.2。
要把它变成完整的生产服务,还缺什么?本次发布提供的是引擎本身,不包含 KV 缓存系统、预填充/解码分离,也不包含跨数据并行副本的负载均衡,因此你需要自行开发或添加这些组件。
它实际能在什么硬件上运行?DeepInfer 只针对两类特定芯片:NVIDIA SM100 级别的 GPU 和华为昇腾 950 NPU,并不支持通用 GPU。
目前我能用 DeepInfer 运行哪些模型?目前只正式支持 DeepSeek-V4.1-Flash,默认下载的是 deepseek-ai/DeepSeek-V4.1-Flash-1017 检查点。
DSpark 是什么?我为什么要关心它?DSpark 是一种推测解码方案,它会生成草稿 token,并利用置信度估计来安排这些 token 的验证时机,从而加快解码速度;其最大草稿长度可以通过环境变量进行调整。
我为什么需要批次不变推理或确定性采样?这两种仅限 NVIDIA 的模式可以确保无论请求如何分批,或使用相同随机种子重新运行,输出结果都完全一致,这对调试、测试和可复现的评估非常有价值。
DeepInfer 是基于什么构建的?它主要基于 vLLM 构建,并借鉴了 SGLang 和 TensorRT-LLM 的部分设计思路。
大小
3.1 MB
星标
26.2K
更新
提交Nov 03
时长
1个月
提交 · 90 天