semantic-router README

项目的 README,来自 github.com/vllm-project/semantic-router。合并为一个 markdown 文件:README.md。

vLLM Semantic Router

An open, programmable decision layer for models and compute.

Documentation | System Duo | Blog | Publications | Hugging Face | Slack

vllm-project%2Fsemantic-router | Trendshift Decision 3.0 — #1 on Hugging Face Trending Collections, October 14, 2026

Main GitHub Release Go Ask DeepWiki


About

Intelligence beyond any one model.

Give your agent harness one API for many models. vLLM Semantic Router selects or combines models for each call, guided by your policy.

Your harness keeps the agent loop, tools, and task state. The Router chooses among configured backends across local, private, and cloud compute.

Dimension Fragmented today With vLLM SR
Models Different models excel at different tasks. Select or combine models.
Compute Hardware varies in speed and capacity. Choose among configured backends.
Location Edge, private, and cloud. Keep calls within approved locations.
Preference Priorities change by task. Set quality, latency, and cost priorities.

Explore how it works →

Getting Started

Install

curl -fsSL https://vllm-sr.ai/install.sh | bash -s -- --channel stable

For pip, uv, or agent-driven installation, see the Installation Guide.

Try it online

Try System Duo: pick an example and it runs live in your browser, with a free key on first run.

Latest News

Earlier announcements

More announcements are available on the Blog and Publications pages.

Community

For questions, feedback, or to contribute, please join the #semantic-router channel in vLLM Slack. Track contributors, workgroups, and weekly activity at community.vllm-sr.ai.

Community Meetings

We host two monthly community meetings across APAC and the Americas:

Contributing

If you want to contribute, start with CONTRIBUTING.md.

For repository-native development workflow and validation commands, use AGENTS.md as the entrypoint and tools/agent/docs/README.md as the canonical index.

Citation

If you find Semantic Router helpful in your research or projects, please consider citing it:

@misc{semanticrouter2025,
  title={vLLM Semantic Router},
  author={vLLM Semantic Router Team},
  year={2025},
  howpublished={\url{https://github.com/vllm-project/semantic-router}},
}

Ecosystem & partnerships

An open ecosystem spanning research, infrastructure, and enterprise adoption.

vLLM Semantic Router's growing ecosystem: AMD, Hugging Face, Microsoft, Intel, NVIDIA, Red Hat, IBM, Liquid, DaoCloud, Delta, MBZUAI, McGill, KR Labs, University of Chicago, UC Berkeley, UMass Boston, University of Illinois Chicago, National Taiwan University, New York University, UBS, AI21, Bayer, Dell, and Nutanix.

semantic-router (vllm-project/semantic-router)

正在搜索数据库…

这是什么?这是一个开放、可编程的决策层,它会根据你自己的策略,为每次调用挑选或组合大语言模型,让同一个 API 能统一调度本地、私有和云端算力上的多个模型。
它是模型、库,还是我要自己运行的服务?它是一个服务端路由服务:你需要部署它,并让你的智能体框架指向它的推理端点,然后由它在你配置的后端中做选择。
我要怎么把它跑起来?通过 pip 或 uv 安装(也可以让智能体代为安装),具体参照安装指南;然后让你的框架指向路由端点,并使用公开模型 ID 'vllm-sr/auto'。
我需要先装 Python 或 Kubernetes 吗?安装方式是 pip 或 uv,所以需要 Python;文档还提到了 Kubernetes 和基于 Docker 的部署,但 README 没有写明必须搭建的集群配置。
它是为谁设计的?面向同时运行多个大语言模型或后端的团队,包括本地、私有和云端算力,希望用一个 API 并基于策略选择模型。
它是否要作为服务器运行?是的,它位于你的模型后端前面,作为智能体框架调用的端点,适合自托管或 Kubernetes 部署。
我能把调用限制在经批准的位置内吗?可以,README 写明你可以通过在边缘、私有和云端后端上设置策略,把调用限制在经批准的位置内。
这里的多模型混合是什么意思?不是把每次调用都发给某一个模型,路由器可以根据你设定的质量、延迟和成本优先级,为每个请求选择或组合多个模型。
装之前能先试试吗?可以,README 中附有一个在线体验场,你可以用它看看路由的实际效果。
'vllm-sr/auto' 这个模型 ID 是什么?它是你在请求中发送的公开模型 ID,路由器会自动挑选后端,你无需指定某个具体模型。
最近的版本有哪些变化?更新日志列出了 v0.3 Themis(面向有状态生产路由)和 v0.2 Athena,另外还有关于信号驱动路由和类别感知语义缓存的研究论文。
大小
573 MB
星标
6.5K
时长
1年
提交 · 90 天