vllm-torchtpu README

项目的 README,来自 github.com/vllm-project/vllm-torchtpu。合并为一个 markdown 文件:README.md。

vLLM TorchTPU vLLM TorchTPU

| Documentation | Blog | User Forum | Developer Slack (#sig-tpu) |


About

vLLM TorchTPU is the PyTorch-native TPU backend for vLLM. It is an out-of-tree vLLM platform plugin, packaged as vllm-torchtpu, that runs vLLM's standard PyTorch model definitions on Google Cloud TPUs through TorchTPU (torch_tpu, via torch.compile(backend="tpu")) and a library of custom JAX/Pallas kernels for attention, Mixture-of-Experts, and quantized matmul.

The goal is simple: the same vLLM you already use — same CLI, same OpenAI-compatible server, same model definitions — running at TPU speed with no model code changes.

[!NOTE] Looking for the JAX backend? The original TPU backend for vLLM, tpu-inference, has been rebranded as vLLM JAX. It remains available for serving pure JAX model implementations on TPU. vllm-torchtpu is the recommended path for serving PyTorch model definitions on TPU.


Supported hardware

TPU generation Status
TPU v7x (Ironwood) Recommended — validated in CI on every pull request and nightly
TPU v6e (Trillium) Supported — not covered by CI
TPU v5e / v5p Experimental — not covered by CI

See Cloud TPU Setup for project setup, quota, and provisioning commands.


Get started

Install

vllm-torchtpu requires Python 3.12 on a Cloud TPU VM. Install vllm-torchtpu from PyPI and a matching vLLM with a single command:

pip install \
  --extra-index-url https://download.pytorch.org/whl/cpu \
  vllm-torchtpu

Or pull the pre-built container from Docker Hub (vllm/vllm-torchtpu):

docker pull vllm/vllm-torchtpu:latest

For uv, editable source installs, cross-repo development against a local vllm checkout, and Docker images, see the full Installation guide and Docker guide.

Serve a model

Start an OpenAI-compatible server exactly as you would on any other vLLM platform:

vllm serve meta-llama/Llama-3.2-1B-Instruct \
  --tensor-parallel-size 1 \
  --max-model-len 4096

Then send a request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.2-1B-Instruct",
    "messages": [{"role": "user", "content": "Explain how TPUs accelerate matrix multiplication."}],
    "max_tokens": 128
  }'

[!TIP] The first startup compiles TPU graphs for the configured shapes and can take several minutes. Set VLLM_XLA_CACHE_PATH to persist compiled artifacts across restarts — see Compilation & Warmup Cache.

The Quickstart covers offline batch inference with vllm.LLM, multi-device serving with TP/EP, and multimodal examples.


Supported models and features

vllm-torchtpu can serve most PyTorch models supported by vLLM, with first-class TPU kernel, quantization, and parallelism support for Qwen3 / Qwen3.5, DeepSeek-V3 / R1 / V4, Kimi-K2.5 / K3, GLM-5, Llama 3 / 4, GPT-OSS, Gemma 3 / 4, Mixtral, Qwen-VL and Gemma VL families, and diffusion models via vllm-omni.


Recipes and deployment


🤝 Contribute

We welcome contributions of all kinds — bug reports, model and feature requests, documentation fixes, and code.

Developer references: Testing & CI · Profiling · Benchmarking · Coding guidance · Optimization best practices


💬 Contact us

vllm-torchtpu (vllm-project/vllm-torchtpu)

正在搜索数据库…

这是什么?vllm-torchtpu 是面向 vLLM 的、原生基于 PyTorch 的 TPU 后端插件。它让同一套 vLLM 命令行、兼容 OpenAI 的服务器和模型定义无需修改模型代码,就能通过 torch.compile 和自定义的 JAX/Pallas 内核在 Google Cloud TPU 上运行。
这是独立的应用,还是 vLLM 的插件?它是 vLLM 的树外(out-of-tree)平台插件,以 vllm-torchtpu 包的形式发布,需要与 vLLM 一起安装,而不是一个独立的应用程序。
我该如何安装和运行?使用一条 pip 命令,从 PyPI 安装 vllm-torchtpu 以及与之匹配的 vLLM 版本;或者从 Docker Hub 拉取预构建的 vllm/vllm-torchtpu 镜像。然后启动 vLLM 常规的兼容 OpenAI 的服务器。
使用前我需要准备什么?你需要在 Google Cloud TPU 虚拟机上运行 Python 3.12;不需要 GPU,但必须先完成 TPU 项目访问、配额和资源配置。
运行时有什么需要注意的地方?首次启动服务器时,会针对已配置的输入形状编译 TPU 计算图,可能需要几分钟(你可以通过 VLLMXLACACHEPATH 保存编译缓存)。另外,只有 TPU v7x 通过了 CI 验证;v6e 没有 CI 覆盖,v5e/v5p 仍处于实验阶段。
它与项目的 JAX 后端(vLLM JAX)有什么不同?最初的 TPU 后端(原名 tpu-inference,现已更名为 vLLM JAX)只提供纯 JAX 模型实现,而 vllm-torchtpu 是在 TPU 上运行 vLLM 原生 PyTorch 模型定义的推荐途径。
支持哪些模型和功能?它可以运行 vLLM 已支持的大多数 PyTorch 模型,并为 Qwen3/3.5、DeepSeek-V3/R1/V4、Kimi-K2.5/K3、GLM-5、Llama 3/4、GPT-OSS、Gemma 3/4、Mixtral、Qwen-VL/Gemma-VL 提供一流的 TPU 内核和量化支持,此外还可通过 vllm-omni 支持扩散模型。
它为 TPU 性能带来了哪些自定义内核?它附带了针对 TPU 调优的 Pallas 内核,包括 Ragged Paged Attention、稠密与稀疏 MLA、Gated DeltaNet、带 SparseCore 卸载的融合 MoE,以及 FP8/INT8/MXFP4 量化矩阵乘法。
它是否支持跨多个 TPU 主机扩展?支持。张量、数据、流水线和专家并行,以及预填充/解码上下文并行,可以在单主机和多主机 TPU 切片上组合使用,包括解耦式的预填充/解码服务。
大小
220.3 MB
星标
142
时长
5个月
提交 · 90 天