litellm README

The project's README from github.com/BerriAI/litellm. As one markdown file: README.md.

🚅 LiteLLM

LiteLLM AI Gateway

Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready. Call any LLM in OpenAI format.

Deploy to Render Deploy on Railway Deploy on AWS Deploy on GCP

LiteLLM Proxy Server (AI Gateway) | Hosted Proxy | Enterprise Tier | Website

PyPI Version GitHub Stars Y Combinator W23 Whatsapp Discord Slack CodSpeed

LiteLLM AI Gateway

What is LiteLLM

LiteLLM is an open source AI Gateway that gives you a single, unified interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and more — using the OpenAI format.

Use it as a Python SDK for direct library integration, or deploy the AI Gateway (Proxy Server) as a centralized service for your team or organization.

Jump to LiteLLM Proxy (LLM Gateway) Docs
Jump to Supported LLM Providers

Why LiteLLM

Managing LLM calls across providers gets complicated fast — different SDKs, auth patterns, request formats, and error types for every model. LiteLLM removes that friction:

OSS Adopters

Stripe image Google ADK Greptile OpenHands

Netflix

 OpenAI Agents SDK

Features

LLMs - Call 100+ LLMs (Python SDK + AI Gateway)

All Supported Endpoints - /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a, /messages and more.

Python SDK

uv add litellm

from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"

# OpenAI
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])

# Anthropic  
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}])

AI Gateway (Proxy Server)

Getting Started - E2E Tutorial - Setup virtual keys, make your first request

uv tool install 'litellm[proxy]'
litellm --model gpt-4o

import openai

client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)

Docs: LLM Providers

Agents - Invoke A2A Agents (Python SDK + AI Gateway)

Supported Providers - LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI

Python SDK - A2A Protocol

from litellm.a2a_protocol import A2AClient
from a2a.types import SendMessageRequest, MessageSendParams
from uuid import uuid4

client = A2AClient(base_url="http://localhost:10001")

request = SendMessageRequest(
    id=str(uuid4()),
    params=MessageSendParams(
        message={
            "role": "user",
            "parts": [{"kind": "text", "text": "Hello!"}],
            "messageId": uuid4().hex,
        }
    )
)
response = await client.send_message(request)

AI Gateway (Proxy Server)

Step 1. Add your Agent to the AI Gateway — set protocolVersion to 1.0 or 0.3 per agent

Step 2. Call Agent via A2A SDK (requires a2a-sdk>=1.1.0)

import httpx
from a2a.client import A2ACardResolver, ClientConfig, ClientFactory
from a2a.types import Message, Part, Role, SendMessageRequest
from a2a.utils.constants import TransportProtocol
from uuid import uuid4

base_url = "http://localhost:4000/a2a/my-agent"  # LiteLLM proxy + agent name
headers = {"Authorization": "Bearer sk-1234"}    # LiteLLM Virtual Key

async with httpx.AsyncClient(headers=headers, timeout=60.0) as http_client:
    resolver = A2ACardResolver(httpx_client=http_client, base_url=base_url)
    agent_card = await resolver.get_agent_card()
    config = ClientConfig(
        httpx_client=http_client,
        streaming=False,
        supported_protocol_bindings=[TransportProtocol.JSONRPC, TransportProtocol.HTTP_JSON],
    )
    client = ClientFactory(config).create(agent_card)

    request = SendMessageRequest(
        message=Message(
            message_id=uuid4().hex,
            role=Role.ROLE_USER,
            parts=[Part(text="Hello!")],
        )
    )
    async for event in client.send_message(request):
        populated = event.ListFields()
        if populated and populated[0][0].name in ("message", "msg"):
            print("".join(getattr(p, "text", "") or "" for p in populated[0][1].parts))

Docs: A2A Agent Gateway

MCP Tools - Connect MCP servers to any LLM (Python SDK + AI Gateway)

Python SDK - MCP Bridge

from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from litellm import experimental_mcp_client
import litellm

server_params = StdioServerParameters(command="python", args=["mcp_server.py"])

async with stdio_client(server_params) as (read, write):
    async with ClientSession(read, write) as session:
        await session.initialize()

        # Load MCP tools in OpenAI format
        tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai")

        # Use with any LiteLLM model
        response = await litellm.acompletion(
            model="gpt-4o",
            messages=[{"role": "user", "content": "What's 3 + 5?"}],
            tools=tools
        )

AI Gateway - MCP Gateway

Step 1. Add your MCP Server to the AI Gateway

Step 2. Call MCP tools via /chat/completions

curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Summarize the latest open PR"}],
    "tools": [{
      "type": "mcp",
      "server_url": "litellm_proxy/mcp/github",
      "server_label": "github_mcp",
      "require_approval": "never"
    }]
  }'

Use with Cursor IDE

{
  "mcpServers": {
    "LiteLLM": {
      "url": "http://localhost:4000/mcp/",
      "headers": {
        "x-litellm-api-key": "Bearer sk-1234"
      }
    }
  }
}

Docs: MCP Gateway

Supported Providers (Website Supported Models | Docs)

Provider /chat/completions /messages /responses /embeddings /image/generations /audio/transcriptions /audio/speech /moderations /batches /rerank
Abliteration (abliteration) ✅
AI/ML API (aiml) ✅ ✅ ✅ ✅ ✅
AI21 (ai21) ✅ ✅ ✅
AI21 Chat (ai21_chat) ✅ ✅ ✅
Aleph Alpha ✅ ✅ ✅
Amazon Nova ✅ ✅ ✅
Anthropic (anthropic) ✅ ✅ ✅ ✅
Anthropic Text (anthropic_text) ✅ ✅ ✅ ✅
Anyscale ✅ ✅ ✅
AssemblyAI (assemblyai) ✅ ✅ ✅ ✅
Auto Router (auto_router) ✅ ✅ ✅
AWS - Bedrock (bedrock) ✅ ✅ ✅ ✅ ✅
AWS - Sagemaker (sagemaker) ✅ ✅ ✅ ✅
Azure (azure) ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
Azure AI (azure_ai) ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
Azure Text (azure_text) ✅ ✅ ✅ ✅ ✅ ✅ ✅
Baseten (baseten) ✅ ✅ ✅
Bytez (bytez) ✅ ✅ ✅
Cerebras (cerebras) ✅ ✅ ✅
Clarifai (clarifai) ✅ ✅ ✅
Cloudflare AI Workers (cloudflare) ✅ ✅ ✅
Codestral (codestral) ✅ ✅ ✅
Cognition (cognition) ✅ ✅ ✅
Cohere (cohere) ✅ ✅ ✅ ✅ ✅
Cohere Chat (cohere_chat) ✅ ✅ ✅
CometAPI (cometapi) ✅ ✅ ✅ ✅
CompactifAI (compactifai) ✅ ✅ ✅
Custom (custom) ✅ ✅ ✅
Custom OpenAI (custom_openai) ✅ ✅ ✅ ✅ ✅ ✅ ✅
Dashscope (dashscope) ✅ ✅ ✅ ✅ ✅
Databricks (databricks) ✅ ✅ ✅
DataRobot (datarobot) ✅ ✅ ✅
Deepgram (deepgram) ✅ ✅ ✅ ✅
DeepInfra (deepinfra) ✅ ✅ ✅
Deepseek (deepseek) ✅ ✅ ✅
ElevenLabs (elevenlabs) ✅ ✅ ✅ ✅ ✅
Empower (empower) ✅ ✅ ✅
Fal AI (fal_ai) ✅ ✅ ✅ ✅
Featherless AI (featherless_ai) ✅ ✅ ✅
Fireworks AI (fireworks_ai) ✅ ✅ ✅
FriendliAI (friendliai) ✅ ✅ ✅
Galadriel (galadriel) ✅ ✅ ✅
GitHub Copilot (github_copilot) ✅ ✅ ✅ ✅
GitHub Models (github) ✅ ✅ ✅
Google - PaLM ✅ ✅ ✅
Google - Vertex AI (vertex_ai) ✅ ✅ ✅ ✅ ✅
Google AI Studio - Gemini (gemini) ✅ ✅ ✅
GradientAI (gradient_ai) ✅ ✅ ✅
Groq AI (groq) ✅ ✅ ✅
Heroku (heroku) ✅ ✅ ✅
Hosted VLLM (hosted_vllm) ✅ ✅ ✅
Huggingface (huggingface) ✅ ✅ ✅ ✅ ✅
Hyperbolic (hyperbolic) ✅ ✅ ✅
IBM - Watsonx.ai (watsonx) ✅ ✅ ✅ ✅
Infinity (infinity) ✅
Jina AI (jina_ai) ✅
Lambda AI (lambda_ai) ✅ ✅ ✅
Lemonade (lemonade) ✅ ✅ ✅
LiteLLM Proxy (litellm_proxy) ✅ ✅ ✅ ✅ ✅
Llamafile (llamafile) ✅ ✅ ✅
LM Studio (lm_studio) ✅ ✅ ✅
Maritalk (maritalk) ✅ ✅ ✅
Meta - Llama API (meta_llama) ✅ ✅ ✅
Mistral AI API (mistral) ✅ ✅ ✅ ✅
ModelScope (modelscope) ✅ ✅ ✅ ✅
Moonshot (moonshot) ✅ ✅ ✅
Morph (morph) ✅ ✅ ✅
Nebius AI Studio (nebius) ✅ ✅ ✅ ✅
NLP Cloud (nlp_cloud) ✅ ✅ ✅
Novita AI (novita) ✅ ✅ ✅
Nscale (nscale) ✅ ✅ ✅
Nvidia NIM (nvidia_nim) ✅ ✅ ✅
[

litellm (BerriAI/litellm)

Searching Database…

61.4K ·GitHub
Litellm
What is this?LiteLLM is an open source AI Gateway that gives you one unified interface to call 100+ LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Azure, and more) using the OpenAI API format.
Do I use this as a Python library, or run it as a standalone server?Both: install it as a Python SDK for direct in-code calls, or deploy the AI Gateway Proxy Server as a centralized self-hosted service for your whole team.
How do I get the gateway running?Follow the E2E Tutorial to set up virtual keys and make your first request; Docker and docker-compose configs are included for self-hosted deployment.
Do I need API keys from the LLM providers I want to route to?Yes, you supply credentials for whichever providers you want to call (OpenAI, Anthropic, Azure, Bedrock, Vertex AI, etc.) and LiteLLM routes requests to them in the OpenAI format.
Do all providers support the same features through the gateway?No, the supported-providers table shows endpoint support differs per provider — most support chat completions, but only some also support embeddings, image generation, audio, or rerank.
What does the MCP Gateway do?It lets you register MCP (Model Context Protocol) servers with LiteLLM and call their tools through any LLM via the standard /chat/completions endpoint.
What is the A2A Agent Gateway for?It lets you register Agent-to-Agent (A2A) protocol agents — like LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, or Pydantic AI — and invoke them through the gateway with the a2a-sdk.
How fast is the proxy?Thanks to its Rust core, LiteLLM benchmarks at 8ms P95 latency at 1,000 requests per second.
What do I get out of the box with the Proxy Server beyond routing?Virtual keys, spend tracking, guardrails, load balancing across providers, and an admin dashboard all ship as part of the AI Gateway.
What are similar projects?busbar, bifrost, gateway, OmniRoute, ThinkWatch-Lite, casbin-gateway, kong, 9router
Which topics is it in?bedrock, vertex-ai, mcp-gateway, gateway, ai-gateway, llm-gateway, llmops, load-balancer, api-gateway, langchain, openai-compatible, observability
Creator
BerriAI
Size
2.4 GB
Stars
61.4K
Updated
CommitOct 18Releasev1.105.1Oct 14
Age
3y
Commits · 90 Days