semantic-router README

The project's README from github.com/vllm-project/semantic-router. As one markdown file: README.md.

vLLM Semantic Router

An open, programmable decision layer for models and compute.

Documentation | System Duo | Blog | Publications | Hugging Face | Slack

vllm-project%2Fsemantic-router | Trendshift Decision 3.0 — #1 on Hugging Face Trending Collections, October 14, 2026

Main GitHub Release Go Ask DeepWiki


About

Intelligence beyond any one model.

Give your agent harness one API for many models. vLLM Semantic Router selects or combines models for each call, guided by your policy.

Your harness keeps the agent loop, tools, and task state. The Router chooses among configured backends across local, private, and cloud compute.

Dimension Fragmented today With vLLM SR
Models Different models excel at different tasks. Select or combine models.
Compute Hardware varies in speed and capacity. Choose among configured backends.
Location Edge, private, and cloud. Keep calls within approved locations.
Preference Priorities change by task. Set quality, latency, and cost priorities.

Explore how it works →

Getting Started

Install

curl -fsSL https://vllm-sr.ai/install.sh | bash -s -- --channel stable

For pip, uv, or agent-driven installation, see the Installation Guide.

Try it online

Try System Duo: pick an example and it runs live in your browser, with a free key on first run.

Latest News

Earlier announcements

More announcements are available on the Blog and Publications pages.

Community

For questions, feedback, or to contribute, please join the #semantic-router channel in vLLM Slack. Track contributors, workgroups, and weekly activity at community.vllm-sr.ai.

Community Meetings

We host two monthly community meetings across APAC and the Americas:

Contributing

If you want to contribute, start with CONTRIBUTING.md.

For repository-native development workflow and validation commands, use AGENTS.md as the entrypoint and tools/agent/docs/README.md as the canonical index.

Citation

If you find Semantic Router helpful in your research or projects, please consider citing it:

@misc{semanticrouter2025,
  title={vLLM Semantic Router},
  author={vLLM Semantic Router Team},
  year={2025},
  howpublished={\url{https://github.com/vllm-project/semantic-router}},
}

Ecosystem & partnerships

An open ecosystem spanning research, infrastructure, and enterprise adoption.

vLLM Semantic Router's growing ecosystem: AMD, Hugging Face, Microsoft, Intel, NVIDIA, Red Hat, IBM, Liquid, DaoCloud, Delta, MBZUAI, McGill, KR Labs, University of Chicago, UC Berkeley, UMass Boston, University of Illinois Chicago, National Taiwan University, New York University, UBS, AI21, Bayer, Dell, and Nutanix.

semantic-router (vllm-project/semantic-router)

Searching Database…

What is this?An open, programmable decision layer that picks or combines LLMs for each call, guided by your own policy, so one API can front many models across local, private, and cloud compute.
Is it a model, a library, or a service I run?It is a server-side routing service: you deploy it and point your agent harness at its inference endpoint, which then chooses among your configured backends.
How do I get it running?Install it via pip or uv (or let an agent install it) following the Installation Guide, then point your harness at the router endpoint and use the public model ID 'vllm-sr/auto'.
Do I need Python or Kubernetes first?The install path is pip or uv, so Python is needed; the topics also point to Kubernetes and Docker-based deployment, but the README does not spell out a required cluster setup.
Who is it built for?Teams running several LLMs or backends, including local, private, and cloud compute, who want one API and policy-based model selection.
Is it meant to run as a server?Yes, it sits in front of your model backends as an endpoint your agent harness calls, which fits a self-hosted or Kubernetes deployment.
Can I keep calls inside approved locations?Yes, the README says you can keep calls within approved locations by setting policy across edge, private, and cloud backends.
What is a mixture-of-models setup here?Instead of sending each call to one model, the router can select or combine several models per request, guided by quality, latency, and cost priorities you set.
Can I try it before installing?Yes, the README links an online playground you can use to see how routing behaves.
What is the 'vllm-sr/auto' model ID?It is the public model ID you send in requests so the router picks the backend automatically instead of you naming a specific model.
What changed in the recent releases?The changelog lists v0.3 Themis for stateful production routing and v0.2 Athena, plus research papers on signal-driven routing and category-aware semantic caching.
Size
573 MB
Stars
6.5K
Age
1y
Commits · 90 Days