OrcaRouter-Lite README

项目的 README,来自 github.com/Continuum-AI-Corp/OrcaRouter-Lite。合并为一个 markdown 文件:README.md。

OrcaRouter Lite

Self-hosted LLM router with a managed safety net. OpenAI-compatible. BYOK. Single-workspace. Streaming. model="auto".

OrcaRouter Lite Logo

tests models license

OrcaRouter Lite failover demo

model="auto" absorbing a provider outage in real time — no code change. How it was recorded: DEMO.md.

Languages

Find us: All model APIs · OrcaCode Review · OrcaReplay · OrcaPromptVault

OrcaRouter Lite is the open-source single-workspace edition of OrcaRouter. Run it on your laptop, ship it in your product, or use hosted api.orcarouter.ai directly for the long tail of models you don't want to manage keys for.

Why us? LiteLLM is a library; OpenRouter is closed-source hosted; Ollama is local-only. We're the self-hosted server with a managed fallback — a sentence none of those can say.

60-second quickstart

Two ways to use OrcaRouter:

Path A — Self-hosted (BYOK)

Run Lite on your own machine; bring your own provider keys.

git clone https://github.com/Continuum-AI-Corp/OrcaRouter-Lite.git
cd OrcaRouter-Lite
cp .env.example .env
# add at least one: OPENAI_API_KEY=sk-...  (or ORCAROUTER_API_KEY=...)

docker compose up
# logs: ✓ orcarouter-lite ready. API key: sk-orca-abc123...

Base URL: http://localhost:8000/v1. Use the sk-orca-* key printed at startup.

Path B — Hosted (account required)

No clone, no docker. Register, get a key, point any OpenAI SDK at hosted.

# 1. Register at https://www.orcarouter.ai and copy your sk-orca-* key
# 2. Use https://api.orcarouter.ai/v1 as the base URL

Account required. Hosted handles routing, billing, and the long tail of providers — billed per-token on your OrcaRouter account. See docs.orcarouter.ai/introduction.

Then call it from any OpenAI SDK

Examples below use Path A's localhost base URL — swap for https://api.orcarouter.ai/v1 if you're on Path B.

Python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="sk-orca-abc123...",
)
r = client.chat.completions.create(
    model="auto",  # or "gpt-4o-mini", "claude-3-5-sonnet-latest", ...
    messages=[{"role": "user", "content": "Hello!"}],
)
print(r.choices[0].message.content)
Node.js
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:8000/v1",
  apiKey: "sk-orca-abc123...",
});

const r = await client.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "Hello!" }],
});
console.log(r.choices[0].message.content);
curl
curl http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer sk-orca-abc123..." \
  -H "Content-Type: application/json" \
  -d '{"model":"auto","messages":[{"role":"user","content":"Hello!"}]}'

Open http://localhost:8000/ for the dashboard — providers, routing, analytics, keys (Path A only).

Why?

OrcaRouter Lite LiteLLM library OpenRouter Ollama
Self-hosted server ✓ as a library ✗ ✓
OpenAI-compatible ✓ ✓ ✓ ✓
Multi-provider (OpenAI/Anthropic/Google/…) ✓ ✓ ✓ ✗
Built-in dashboard ✓ ✗ ✓ ✗
model="auto" (cheapest capable) ✓ ✗ ✗ n/a
Streaming ✓ ✓ ✓ ✓
BYOK ✓ ✓ ✗ n/a
Hosted-as-fallback ✓ ✗ n/a ✗
No Postgres / no Redis required ✓ n/a n/a ✓

model="auto" — the headline feature

Send model="auto" and OrcaRouter picks the cheapest model in your configured providers that meets the request's capability requirements (tools, vision, JSON mode). No manual routing rules; no rate-limit gymnastics; no if x: ... cost optimization in your code.

client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "What's in this image?"},
        {"type": "image_url", "image_url": {"url": "data:..."}},
    ]}],
)
# → routes to the cheapest VISION-capable model your keys cover

The resolved model is exposed back to callers via the x-orca-resolved-model response header so you can log/display what was actually used.

Hosted as upstream (Lite + hosted)

Already running Lite? Set ORCAROUTER_API_KEY to your sk-orca-* from www.orcarouter.ai, and hosted becomes one more provider in the routing chain — covering models your local keys don't:

# .env
ORCAROUTER_API_KEY=sk-orca-hosted-abc...

Use cases:

Streaming

OpenAI-compatible SSE format with the standard data: ... \n\n framing and a terminal [DONE] sentinel — drop-in for any SDK that already streams from OpenAI.

for chunk in client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Tell me a story"}],
    stream=True,
):
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Native protocol endpoints (Anthropic + Gemini)

Lite speaks three inbound protocols against one routing pipeline. Clients that only talk Anthropic or Gemini wire formats connect directly — no OpenAI SDK required:

# Claude Code, pointed at Lite (no /v1 suffix in the base URL)
export ANTHROPIC_BASE_URL=http://localhost:8000
export ANTHROPIC_API_KEY=sk-orca-...
claude
# google-genai SDK, pointed at Lite
from google import genai
from google.genai.types import HttpOptions
client = genai.Client(api_key="sk-orca-...",
                      http_options=HttpOptions(base_url="http://localhost:8000"))
client.models.generate_content(model="auto", contents="Hello!")

Requests are translated at the edge into the same internal pipeline, so model="auto", the cross-provider prompt cache (shared across protocols), routing strategies, and the analytics dashboard all work identically. Guides: integrations/claude-code.md, integrations/gemini-sdk.md.

Model catalog

100+ chat models are loaded at startup from LiteLLM's community-maintained pricing database — no model list to maintain manually. Each entry exposes:

GET /v1/models returns the OpenAI-format catalogue.

Deploy somewhere else

Platform One-click
Railway Deploy on Railway
Fly.io fly launch --dockerfile Dockerfile
Render Connect repo, root dir = .
Bare Docker docker run -p 8000:8000 -e OPENAI_API_KEY=... ghcr.io/... (image coming soon)

What's in the box

Rotating CREDENTIAL_ENCRYPTION_KEY

Dashboard-stored provider keys are sealed with AES-256-GCM using CREDENTIAL_ENCRYPTION_KEY. Changing that value without a migrate step leaves the ciphertext intact but unreadable: build_deployments skips the row, chat returns 503 if nothing else is configured, and the providers list now reports decryptable: false instead of a green "Enabled".

To rotate without re-entering every key:

# 1. Keep the old key, set the new one
CREDENTIAL_ENCRYPTION_KEY=<new 64-hex>
CREDENTIAL_ENCRYPTION_PREVIOUS_KEY=<old 64-hex>

# 2. Restart. Startup re-encrypts stored keys and logs
#    reencrypted_provider_keys / undecryptable_provider_keys.

# 3. Confirm the dashboard shows Enabled, then unset the previous key
#    and restart once more.

If you already rotated without CREDENTIAL_ENCRYPTION_PREVIOUS_KEY, re-save each provider key in the dashboard (or fall back to env vars). Do not reuse a previous key as the current key after you have already re-saved — that would reseal new ciphertext with the old material.

Cross-provider prompt cache

Deterministic requests (temperature=0 or pinned seed) are served from cache on repeat — works across every provider, not just Anthropic. Backend is Redis when REDIS_URL is set, in-process LRU otherwise. Cache hits return instantly with x-orca-cache: HIT and cost $0.

$ curl ... -d '{"model":"auto","messages":[...], "temperature": 0}' -i
HTTP/1.1 200 OK
x-orca-cache: MISS
x-orca-resolved-model: gpt-4o-mini

$ curl ...  # same payload again
HTTP/1.1 200 OK
x-orca-cache: HIT          ← served from cache, no upstream call

Savings widget

GET /v1/analytics/savings?baseline=gpt-4o&days=7 reports what your traffic would have cost on always-GPT-4 vs what it actually cost. The dashboard shows it as a tile.

Integrations

Drop-in configs for Claude Code, Gemini SDK, Continue.dev, Aider, Cursor, LangChain, LlamaIndex, Vercel AI SDK, and any tool that speaks the OpenAI Chat Completions protocol — plus native Anthropic and Gemini wire formats. See integrations/.

What's deliberately not

This is the single-workspace edition. By design, no:

For those, see the hosted product or the (forthcoming) Teams edition.

Testing

Built test-first. Every behaviour shipped here had a failing test first.

pip install -e ".[dev]"
PYTHONPATH=. pytest -v
# 403 passed
Slice Tests What
1. Config 5 env loading, defaults, env_provider_keys()
2. Seed 3 bootstrap workspace + API key + RoutingConfig, idempotent
3. Auth middleware 4 bearer-token validation, 401 on missing/invalid
4. App factory 3 /health, error envelope, /v1/* gating
5. Provider keys CRUD 5 encrypted at rest, plaintext never round-trips
6. Router cache 13 env+DB+hosted deployment assembly with precedence
7. Chat completion 5 OpenAI format, RequestLog, validation
8. Analytics 4 recent / spend / latency p50/p99
9. /v1/{models,keys,routing} 8 list/create/revoke + strategy update
10. Streaming 4 SSE format, [DONE] sentinel, log writeback
11. Catalog 7 100+ models, capability flags, pricing
12. model="auto" 21 capability detection, cheapest-meeting-needs (unit + integration)
13. Cost savings 9 savings vs always-GPT-4 baseline + hosted-auto comparison
14. Prompt cache 15 cross-provider exact-match cache + chat integration
15. Benchmark 4 summarize() + render_markdown() aggregation
16. Hosted status 7 /v1/hosted config-source + signup-URL surface
17. Hosted-auto savings 3 _hosted_auto_savings edge cases on synthetic catalogs
18. Unreachable models 7 "models you can't reach" tile clears when hosted is on
19. Multi-protocol auth 6 x-api-key / x-goog-api-key / ?key= scoping, /v1beta guard, per-protocol 401 envelopes
20. Anthropic /v1/messages 53 request/response/stream translation + ingress integration
21. Gemini /v1beta 40 translation incl. schema-enum normalization + generateContent/stream ingress
Total 403

Slice rows show the tests added when each slice shipped; the total is the current full suite.

Architecture

app/
├── main.py             FastAPI factory + lifespan + SPA mount
├── config.py           Settings (~15 fields)
├── deps.py             DI helpers
├── seed.py             First-run bootstrap
├── auto_routing.py     model="auto" capability + cost scoring
├── router_cache.py     Single-workspace router
├── prompt_cache.py     Cross-provider exact-match cache (Redis or in-memory LRU)
├── schemas.py          OpenAI-compatible request schema
├── middleware/auth.py  sk-orca-* validation
└── routes/
    ├── chat.py         /v1/chat/completions  (blocking + streaming)
    ├── models.py       /v1/models
    ├── providers.py    BYOK CRUD
    ├── routing.py      strategy config
    ├── analytics.py    recent / spend / latency / savings / unreachable
    ├── keys.py         list / rotate / revoke API keys
    ├── hosted.py       /v1/hosted — hosted-fallback status for the dashboard
    └── health.py

packages/
├── litellm_adapter/    Router wrapper + 100+ model catalog
├── auth/               hashing + AES-256-GCM
└── db/                 models + engine + session

Roadmap

See DEMO.md for the failover demo.

License

MIT. See LICENSE.

OrcaRouter-Lite (Continuum-AI-Corp/OrcaRouter-Lite)

正在搜索数据库…

这是什么?一个自托管的服务器,位于你的应用和大模型服务商之间。它提供兼容 OpenAI 的 API,将每个请求路由到你添加了密钥的服务商,并且在本地密钥失效时可以回退到托管的 OrcaRouter。
它和 LiteLLM、OpenRouter 或 Ollama 有什么不同?LiteLLM 是一个嵌入到代码里的库,OpenRouter 是闭源的托管服务,Ollama 只能运行本地模型。Lite 则是一个你自己运行的服务器,可以路由到多个服务商,并提供托管的回退功能。
怎样让它运行起来?在自己的机器上用自己的服务商密钥运行 Lite,然后让任意 OpenAI SDK 指向本地 base URL,并使用启动时打印的 sk-orca- 密钥。或者跳过安装,改用注册账号后的托管服务 api.orcarouter.ai。
我需要先安装 Postgres 或 Redis 吗?不需要。默认使用 SQLite。Postgres 可通过 DATABASE_URL 选择启用,而 Redis 只有在运行多个 worker 或副本、需要让配置变更在它们之间同步时才需要。
这个版本缺少哪些功能?不支持多租户、RBAC 或 SSO,没有计费、管理控制台或审计日志,不支持 Kubernetes 或多 Pod 部署,也没有告警集成。这些功能属于托管产品或即将推出的 Teams 版。
能部署到服务器或云平台上吗?可以。README 列出了 Railway、Fly.io 和 Render,Fly 部署使用 'fly launch --dockerfile Dockerfile',也可以直接用 Docker 运行。预构建的 Docker 镜像即将推出。
我的使用数据会离开我的机器吗?花费、延迟和近期请求等分析数据都存储在本地,不会发送任何遥测数据。你的提示词仍会发送给你配置的上游服务商,只有在你启用的情况下才会发送给托管的回退服务。
某个服务商宕机时会发生什么?设置 model='auto' 时,请求会被实时重新路由到另一个有能力的服务商。如果你设置了 ORCAROUTER_API_KEY,托管服务会作为调用链中的另一个服务商,起到安全兜底的作用。
它会缓存重复的请求吗?会。使用 temperature=0 或固定 seed 的确定性请求,在重复时会从缓存中返回,且适用于所有服务商。配置了 Redis 时使用 Redis,否则使用进程内 LRU。缓存命中时会返回 x-orca-cache: HIT,且不产生费用。
Claude Code 或 Gemini SDK 能直接连接吗?可以。服务器接受原生的 Anthropic Messages 和 Gemini generateContent 请求,并将其转换到同一条处理流程中,因此 model='auto'、缓存、路由策略和分析功能对这些客户端同样适用。
我如何查看自己节省了多少?调用 GET /v1/analytics/savings,并指定一个基准模型,例如 gpt-4o。它会将你的实际花费与同样的流量在该模型上的花费进行对比,仪表盘会以卡片形式显示结果。
有哪些类似的项目?experiential, busbar, new-api, gpt-load, router, pxpipe, bifrost, litellm
大小
1.7 MB
星标
3.3K
时长
6个月
提交 · 90 天