gateway README

The project's README from github.com/Portkey-AI/gateway. As one markdown file: README.md.

English | 中文 | 日本語

[!IMPORTANT] :rocket: Gateway 2.0 (Pre-Release) Portkey's core enterprise gateway is merging into open-source with our 2.0 release. You can try the pre-release branch here. Read more about what's next for Portkey in our Series A announcement.

🆕 Portkey Models - Open-source LLM pricing for 2,300+ models across 40+ providers. Explore →

AI Gateway

Route to 250+ LLMs with 1 fast & friendly API

Portkey AI Gateway Demo showing LLM routing capabilities

Docs | Enterprise | Hosted Gateway | Changelog | API Reference

License Discord Twitter npm version Better Stack Badge

Deploy to AWS EC2 Ask DeepWiki


The AI Gateway is designed for fast, reliable & secure routing to 1600+ language, vision, audio, and image models. It is a lightweight, open-source, and enterprise-ready solution that allows you to integrate with any language model in under 2 minutes.


What can you do with the AI Gateway?



[!TIP] Starring this repo helps more developers discover the AI Gateway 🙏🏻

star-2



Quickstart (2 mins)

1. Setup your AI Gateway

# Run the gateway locally (needs Node.js and npm)
npx @portkey-ai/gateway

The Gateway is running on http://localhost:8787/v1

The Gateway Console is running on http://localhost:8787/public/

Deployment guides:   Portkey Cloud (Recommended)   Docker   Node.js   Cloudflare   Replit   Others...

2. Make your first request

# pip install -qU portkey-ai

from portkey_ai import Portkey

# OpenAI compatible client
client = Portkey(
    provider="openai", # or 'anthropic', 'bedrock', 'groq', etc
    Authorization="sk-***" # the provider API key
)

# Make a request through your AI Gateway
client.chat.completions.create(
    messages=[{"role": "user", "content": "What's the weather like?"}],
    model="gpt-4o-mini"
)

Supported Libraries:   JS   Python   REST   OpenAI SDKs   Langchain   LlamaIndex   Autogen   CrewAI   More..

On the Gateway Console (http://localhost:8787/public/) you can see all of your local logs in one place.

3. Routing & Guardrails

Configs in the LLM gateway allow you to create routing rules, add reliability and setup guardrails.

config = {
  "retry": {"attempts": 5},

  "output_guardrails": [{
    "default.contains": {"operator": "none", "words": ["Apple"]},
    "deny": True
  }]
}

# Attach the config to the client
client = client.with_options(config=config)

client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Reply randomly with Apple or Bat"}]
)

# This would always response with "Bat" as the guardrail denies all replies containing "Apple". The retry config would retry 5 times before giving up.
Request flow through Portkey's AI gateway with retries and guardrails

You can do a lot more stuff with configs in your AI gateway. Jump to examples →


Enterprise Version (Private deployments)

AWS   Azure   GCP   OpenShift   Kubernetes

The LLM Gateway's enterprise version offers advanced capabilities for org management, governance, security and more out of the box. View Feature Comparison →

The enterprise deployment architecture for supported platforms is available here - Enterprise Private Cloud Deployments

Book an enterprise AI gateway demo


MCP Gateway

MCP Gateway provides a centralized control plane for managing MCP (Model Context Protocol) servers across your organization.

Works with Claude Desktop, Cursor, VS Code, and any MCP-compatible client. Get started →


Core Features

Reliable Routing

Security & Accuracy

Cost Management

Collaboration & Workflows



* Available in hosted and enterprise versions

Portkey Models

Open-source LLM pricing database for 40+ providers - used by the Gateway for cost tracking.

GitHub | Model Explorer


Cookbooks

🚨 Latest

View all cookbooks →

Supported Providers

Explore Gateway integrations with 45+ providers and 8+ agent frameworks.

Provider Support Stream
OpenAI ✅ ✅
Azure OpenAI ✅ ✅
Anyscale ✅ ✅
Google Gemini ✅ ✅
Anthropic ✅ ✅
Cohere ✅ ✅
Together AI ✅ ✅
Perplexity ✅ ✅
Mistral ✅ ✅
Nomic ✅ ✅
AI21 ✅ ✅
Stability AI ✅ ✅
DeepInfra ✅ ✅
Ollama ✅ ✅
Novita AI ✅ ✅

View the complete list of 200+ supported models here



Agents

Gateway seamlessly integrates with popular agent frameworks. Read the documentation here.

Framework Call 200+ LLMs Advanced Routing Caching Logging & Tracing* Observability* Prompt Management*
Autogen ✅ ✅ ✅ ✅ ✅ ✅
CrewAI ✅ ✅ ✅ ✅ ✅ ✅
LangChain ✅ ✅ ✅ ✅ ✅ ✅
Phidata ✅ ✅ ✅ ✅ ✅ ✅
Llama Index ✅ ✅ ✅ ✅ ✅ ✅
Control Flow ✅ ✅ ✅ ✅ ✅ ✅
Build Your Own Agents ✅ ✅ ✅ ✅ ✅ ✅
IO Intelligence ✅ ✅

*Available on the hosted app. For detailed documentation click here.

Gateway Enterprise Version

Make your AI app more reliable and forward compatible, while ensuring complete data security and privacy.

✅  Secure Key Management - for role-based access control and tracking
✅  Simple & Semantic Caching - to serve repeat queries faster & save costs
✅  Access Control & Inbound Rules - to control which IPs and Geos can connect to your deployments
✅  PII Redaction - to automatically remove sensitive data from your requests to prevent indavertent exposure
✅  SOC2, ISO, HIPAA, GDPR Compliances - for best security practices
✅  Professional Support - along with feature prioritization

Schedule a call to discuss enterprise deployments


Contributing

The easiest way to contribute is to pick an issue

gateway (Portkey-AI/gateway)

mcp-gateway
Searching Database…

13.2K ·GitHub
Gateway
What is this?An open-source AI gateway that routes your requests to 1,600+ language, vision, audio and image models across 45+ providers through one fast, friendly API, with guardrails, caching, retries and fallbacks built in.
Is this a library I add to my code, or a separate server I run?It's a server, not a library: it sits between your app and the model providers, forwarding OpenAI-style requests to whichever model you pick. You can self-host it via Docker, Node.js, Cloudflare Workers or Replit, or use the hosted Portkey Cloud.
How do I get it running?Set up the gateway first — Portkey Cloud is the recommended path, with Docker, Node.js, Cloudflare and Replit as alternatives — then make your first request with the JS or Python SDK, plain REST, or OpenAI SDKs. The README claims you can integrate in under 2 minutes.
Do I need API keys from each provider, or does the gateway handle that?You bring your own provider keys, or generate virtual keys on the gateway on the fly. On the client side it accepts plain REST plus JS, Python, OpenAI SDKs, LangChain, LlamaIndex, Autogen and CrewAI.
Is the whole thing free and open source?The core gateway is MIT-licensed open source, but some features are gated: prompt template management ships only in the hosted and enterprise versions, and the enterprise tier adds org management, governance and PII redaction. Gateway 2.0 is still pre-release.
Can I point sensitive or regulated traffic at it?Yes — it's SOC2, HIPAA, GDPR and CCPA compliant, and the enterprise tier adds PII redaction, IP/geo inbound rules and role-based access control. Self-hosting keeps request traffic on your own infrastructure.
What happens if a provider goes down or gets slow?Requests automatically retry up to 5 times with exponential backoff to avoid network overload, you can set granular timeouts that kill hanging requests, and load balancing spreads traffic across multiple API keys or providers with weights, with fallbacks for availability.
What is the MCP Gateway part for?It's a centralized control plane for managing MCP servers: users authenticate once at the gateway, admins control which teams can reach which servers and tools, and every tool call is logged with caller, parameters and latency. It works with Claude Desktop, Cursor, VS Code and any MCP-compatible client.
Can it help cut my LLM bill?Yes. Smart caching (simple and semantic) serves repeat queries faster and cheaper, usage analytics track request volume, latency, costs and error rates, and provider optimization can automatically switch to the most cost-effective provider based on usage patterns.
Which providers and agent frameworks does it plug into?45+ providers including OpenAI, Azure OpenAI, Google Gemini, Anthropic, Cohere, Mistral, Together AI, DeepInfra and Ollama, and 8+ agent frameworks: Autogen, CrewAI, LangChain, LlamaIndex, Phidata, Control Flow, IO Intelligence and custom-built agents.
What are similar projects?bifrost, litellm, busbar, casbin-gateway, ThinkWatch-Lite, OmniRoute, langfuse, claude-code-router
Which topics is it in?mcp-servers, mcp-gateway, llm-proxy, model-router, gateway, mcp-client, ai-gateway, llm-gateway, llmops, api-gateway, llms, caching
Creator
Portkey-AI
Size
66.1 MB
Stars
13.2K
Updated
CommitMay 25Releasev1.15.2Jan 12
Age
3y
Commits · 90 Days