gateway README

The project's README from github.com/Portkey-AI/gateway. As one markdown file: README.md.

English | 中文 | 日本語

[!IMPORTANT] :rocket: Gateway 2.0 (Pre-Release) Portkey's core enterprise gateway is merging into open-source with our 2.0 release. You can try the pre-release branch here. Read more about what's next for Portkey in our Series A announcement.

🆕 Portkey Models - Open-source LLM pricing for 2,300+ models across 40+ providers. Explore →

AI Gateway

Route to 250+ LLMs with 1 fast & friendly API

Portkey AI Gateway Demo showing LLM routing capabilities

Docs | Enterprise | Hosted Gateway | Changelog | API Reference

License Discord Twitter npm version Better Stack Badge

Deploy to AWS EC2 Ask DeepWiki


The AI Gateway is designed for fast, reliable & secure routing to 1600+ language, vision, audio, and image models. It is a lightweight, open-source, and enterprise-ready solution that allows you to integrate with any language model in under 2 minutes.


What can you do with the AI Gateway?



[!TIP] Starring this repo helps more developers discover the AI Gateway 🙏🏻

star-2



Quickstart (2 mins)

1. Setup your AI Gateway

# Run the gateway locally (needs Node.js and npm)
npx @portkey-ai/gateway

The Gateway is running on http://localhost:8787/v1

The Gateway Console is running on http://localhost:8787/public/

Deployment guides:   Portkey Cloud (Recommended)   Docker   Node.js   Cloudflare   Replit   Others...

2. Make your first request

# pip install -qU portkey-ai

from portkey_ai import Portkey

# OpenAI compatible client
client = Portkey(
    provider="openai", # or 'anthropic', 'bedrock', 'groq', etc
    Authorization="sk-***" # the provider API key
)

# Make a request through your AI Gateway
client.chat.completions.create(
    messages=[{"role": "user", "content": "What's the weather like?"}],
    model="gpt-4o-mini"
)

Supported Libraries:   JS   Python   REST   OpenAI SDKs   Langchain   LlamaIndex   Autogen   CrewAI   More..

On the Gateway Console (http://localhost:8787/public/) you can see all of your local logs in one place.

3. Routing & Guardrails

Configs in the LLM gateway allow you to create routing rules, add reliability and setup guardrails.

config = {
  "retry": {"attempts": 5},

  "output_guardrails": [{
    "default.contains": {"operator": "none", "words": ["Apple"]},
    "deny": True
  }]
}

# Attach the config to the client
client = client.with_options(config=config)

client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Reply randomly with Apple or Bat"}]
)

# This would always response with "Bat" as the guardrail denies all replies containing "Apple". The retry config would retry 5 times before giving up.
Request flow through Portkey's AI gateway with retries and guardrails

You can do a lot more stuff with configs in your AI gateway. Jump to examples →


Enterprise Version (Private deployments)

AWS   Azure   GCP   OpenShift   Kubernetes

The LLM Gateway's enterprise version offers advanced capabilities for org management, governance, security and more out of the box. View Feature Comparison →

The enterprise deployment architecture for supported platforms is available here - Enterprise Private Cloud Deployments

Book an enterprise AI gateway demo


MCP Gateway

MCP Gateway provides a centralized control plane for managing MCP (Model Context Protocol) servers across your organization.

Works with Claude Desktop, Cursor, VS Code, and any MCP-compatible client. Get started →


Core Features

Reliable Routing

Security & Accuracy

Cost Management

Collaboration & Workflows



* Available in hosted and enterprise versions

Portkey Models

Open-source LLM pricing database for 40+ providers - used by the Gateway for cost tracking.

GitHub | Model Explorer


Cookbooks

🚨 Latest

View all cookbooks →

Supported Providers

Explore Gateway integrations with 45+ providers and 8+ agent frameworks.

Provider Support Stream
OpenAI ✅ ✅
Azure OpenAI ✅ ✅
Anyscale ✅ ✅
Google Gemini ✅ ✅
Anthropic ✅ ✅
Cohere ✅ ✅
Together AI ✅ ✅
Perplexity ✅ ✅
Mistral ✅ ✅
Nomic ✅ ✅
AI21 ✅ ✅
Stability AI ✅ ✅
DeepInfra ✅ ✅
Ollama ✅ ✅
Novita AI ✅ ✅

View the complete list of 200+ supported models here



Agents

Gateway seamlessly integrates with popular agent frameworks. Read the documentation here.

Framework Call 200+ LLMs Advanced Routing Caching Logging & Tracing* Observability* Prompt Management*
Autogen ✅ ✅ ✅ ✅ ✅ ✅
CrewAI ✅ ✅ ✅ ✅ ✅ ✅
LangChain ✅ ✅ ✅ ✅ ✅ ✅
Phidata ✅ ✅ ✅ ✅ ✅ ✅
Llama Index ✅ ✅ ✅ ✅ ✅ ✅
Control Flow ✅ ✅ ✅ ✅ ✅ ✅
Build Your Own Agents ✅ ✅ ✅ ✅ ✅ ✅
IO Intelligence ✅ ✅

*Available on the hosted app. For detailed documentation click here.

Gateway Enterprise Version

Make your AI app more reliable and forward compatible, while ensuring complete data security and privacy.

✅  Secure Key Management - for role-based access control and tracking
✅  Simple & Semantic Caching - to serve repeat queries faster & save costs
✅  Access Control & Inbound Rules - to control which IPs and Geos can connect to your deployments
✅  PII Redaction - to automatically remove sensitive data from your requests to prevent indavertent exposure
✅  SOC2, ISO, HIPAA, GDPR Compliances - for best security practices
✅  Professional Support - along with feature prioritization

Schedule a call to discuss enterprise deployments


Contributing

The easiest way to contribute is to pick an issue

gateway (Portkey-AI/gateway)

Searching Database…

13.2K ·GitHub
Gateway
这是什么？一个开源的 AI 网关，通过一个快速、易用的 API 把你的请求转发到 45+ 家服务商的 1600 多个语言、视觉、音频和图像模型，内置防护规则、缓存、重试和故障转移。
它是要加进我代码里的库，还是我单独运行的一个服务？它是一个服务，不是库：它位于你的应用和模型服务商之间，把 OpenAI 风格的请求转发到你选定的模型。你可以用 Docker、Node.js、Cloudflare Workers 或 Replit 自行部署，也可以直接使用托管的 Portkey Cloud。
我怎么把它跑起来？先部署网关——推荐用 Portkey Cloud，Docker、Node.js、Cloudflare 和 Replit 也可以——然后通过 JS 或 Python SDK、原生 REST 或 OpenAI SDK 发出你的第一个请求。README 声称不到 2 分钟就能完成接入。
我需要每家服务商的 API 密钥吗，还是网关帮我搞定？你需要自带各服务商的密钥，也可以在网关上随时生成虚拟密钥。客户端方面，它支持原生 REST，以及 JS、Python、OpenAI SDK、LangChain、LlamaIndex、Autogen 和 CrewAI。
它是完全免费开源的吗？核心网关基于 MIT 协议开源，但部分功能是收费的：提示词模板管理只在托管版和企业版中提供，企业版还增加了组织管理、治理和 PII 脱敏。Gateway 2.0 目前仍是预发布版本。
我可以把敏感或受监管的流量指向它吗？可以——它符合 SOC2、HIPAA、GDPR 和 CCPA 合规要求，企业版还提供 PII 脱敏、IP/地理位置入站规则和基于角色的访问控制。自行部署的话，请求流量完全留在你自己的基础设施上。
如果某家服务商宕机或变慢了会怎样？请求会自动重试最多 5 次，并采用指数退避来避免网络过载；你可以设置细粒度的超时来终止挂起的请求；负载均衡还能按权重把流量分配到多个 API 密钥或服务商，并带有保障可用性的故障转移。
MCP Gateway 那部分是干什么的？它是一个用于管理 MCP 服务器的集中控制平面：用户只需在网关认证一次，管理员可以控制哪些团队能访问哪些服务器和工具，每次工具调用都会记录调用者、参数和延迟。它兼容 Claude Desktop、Cursor、VS Code 以及任何支持 MCP 的客户端。
它能帮我省 LLM 的账单吗？能。智能缓存（简单缓存和语义缓存）让重复查询更快、更省；用量分析可以跟踪请求量、延迟、成本和错误率；服务商优化还能根据使用模式自动切换到性价比最高的服务商。
它接入了哪些服务商和智能体框架？45+ 家服务商，包括 OpenAI、Azure OpenAI、Google Gemini、Anthropic、Cohere、Mistral、Together AI、DeepInfra 和 Ollama；以及 8+ 个智能体框架：Autogen、CrewAI、LangChain、LlamaIndex、Phidata、Control Flow、IO Intelligence 和自定义的智能体。
What are similar projects?bifrost, litellm, busbar, casbin-gateway, ThinkWatch-Lite, OmniRoute, langfuse, claude-code-router
Which topics is it in?mcp-servers, mcp-gateway, llm-proxy, model-router, gateway, mcp-client, ai-gateway, llm-gateway, llmops, api-gateway, llms, caching
Creator
Portkey-AI
Size
66.1 MB
Stars
13.2K
Updated
CommitMay 25Releasev1.15.2Jan 12
Age
3y
Commits · 90 Days