tokenspeed README

项目的 README,来自 github.com/lightseekorg/tokenspeed。合并为一个 markdown 文件:README.md。

TokenSpeed: Tokens at the speed of light

TokenSpeed is a speed-of-light LLM inference engine designed for agentic workloads, with TensorRT-LLM-level performance and vLLM-level usability. The goal is to be the most performant inference engine for production agentic workloads.

Core components:

Differentiation

TokenSpeed takes a fundamentally different architectural approach from existing LLM inference engines. The PyTorch Foundation highlighted these architectural differentiators when welcoming TokenSpeed to the PyTorch Ecosystem Landscape:

TokenSpeed is an open source LLM inference engine and the first to separate the control plane from the execution plane. The control plane is implemented in C++ as a finite-state machine, using the type system to enforce safe resource management, including request lifecycles and KV cache state, at compile time rather than runtime. The execution plane is implemented in Python, enabling fast iteration and lowering the cognitive load for researchers and engineers. This architecture combines strong correctness guarantees in the core scheduling system with the development velocity of a high-level execution layer.

TokenSpeed also treats kernels as a first-class, modular subsystem, separating them from the core engine through a portable public API, centralized registry and selection model, and an extensible plugin mechanism for heterogeneous accelerators.

More importantly, TokenSpeed is the Switzerland of open-source LLM inference engines: neutral because it's backed by the non-profit LightSeek Foundation rather than a commercial company, with serious hardware and engineering support from vendors such as NVIDIA and AMD.

News

Blogs and talks

For technical blogs, conference talks, and engineering articles from the LightSeek Foundation, visit the LightSeek Blog.

Sponsors and partners

LightSeek's work is advanced by the support of sponsors and partners across the AI ecosystem.

Performance comparison

TokenSpeed vs. TensorRT-LLM Pareto curves on agentic workloads (Kimi K2.5, B200)

TokenSpeed vs. TensorRT-LLM Pareto curves on agentic workloads (Kimi K2.5, B200)

Documentation

Start here:

Citation

@misc{tokenspeed2026,
  author       = {{TokenSpeed Team}},
  title        = {{TokenSpeed}: A Speed-of-Light {LLM} Inference Engine},
  year         = {2026},
  howpublished = {\url{https://github.com/lightseekorg/tokenspeed}}
}

tokenspeed (lightseekorg/tokenspeed)

正在搜索数据库…

这是什么?TokenSpeed 是一个速度极快的大语言模型推理引擎,专为智能体(agentic)工作负载设计,目标是在性能上达到 TensorRT-LLM 的水平,同时保持 vLLM 的易用性。
这是我要导入的库,还是我要运行的服务器?它以服务器的形式运行(通过 AsyncLLM 入口启动),用来提供模型服务;它不是一个可以直接用 pip 导入的简单推理库。
我该如何用 TokenSpeed 把模型跑起来?文档提供了“Getting Started”指南和“Launching a Server”页面,会一步步讲解如何结合特定的模型配置方案,搭建起推理服务器。
运行它需要特定的硬件吗?需要。它面向高端加速卡,比如 NVIDIA Blackwell/Rubin(B200/B300/GB200/GB300)和 AMD Instinct(MI355X/MI455X)GPU,并不适用于普通消费级硬件。
它在任何 GPU 上都能良好运行,还是只支持最新的芯片?文档中的主要基准测试和优化数据(每用户 585–685 tokens/s,FP4 推理)都是在最新的 NVIDIA 和 AMD 数据中心 GPU 上测得的;README 中没有提及在旧款或消费级 GPU 上的表现。
这个引擎是为谁设计的?它面向运行大规模生产级智能体大语言模型工作负载的机器学习系统研究人员和工程师,既需要极高的原始吞吐量,也需要能够快速迭代的灵活性。
TokenSpeed 的架构与 vLLM 等引擎有什么不同?它将控制平面和执行平面分开:控制平面是一个 C++ 有限状态机,在编译期就保证 KV 缓存和请求生命周期管理的安全性;执行平面使用 Python,便于快速迭代。此外,它把内核作为可插拔的、集中注册的子系统来处理。
什么是 MLA?它在这里为什么重要?MLA 指 Multi-head Latent Attention(多头潜在注意力)。TokenSpeed 在 NVIDIA Blackwell 上包含了最快的 MLA 内核实现之一,并专门针对智能体工作负载做了优化。
TokenSpeed 开箱即支持哪些模型?README 重点介绍了对 Kimi K3/K2.5、Qwen3.5/3.8、GLM 5.3、DeepSeek、GPT-OSS 和 MiniMax 等模型的首日(Day-0)支持和优化。
为什么项目自称是推理引擎中的“瑞士”?因为它由非营利组织 LightSeek Foundation 运营,而不是由单一公司主导,并且获得了 NVIDIA 和 AMD 在硬件和工程方面的支持,因此保持了厂商中立。
大小
63.6 MB
星标
2.9K
时长
6个月
提交 · 90 天