The Agent Framework is designed for building realtime, programmable participants
that run on servers. Use it to create conversational, multi-modal voice
agents that can see, hear, and understand.
Features
Flexible integrations: A comprehensive ecosystem to mix and match the right STT, LLM, TTS, and Realtime API to suit your use case.
Integrated job scheduling: Built-in task scheduling and distribution with dispatch APIs to connect end users to agents.
Extensive WebRTC clients: Build client applications using LiveKit's open-source SDK ecosystem, supporting all major platforms.
Telephony integration: Works seamlessly with LiveKit's telephony stack, allowing your agent to make calls to or receive calls from phones.
Exchange data with clients: Use RPCs and other Data APIs to seamlessly exchange data with clients.
Semantic turn detection: Uses a transformer model to detect when a user is done with their turn, helps to reduce interruptions.
MCP support: Native support for MCP. Integrate tools provided by MCP servers with one line of code.
Builtin test framework: Write tests and use judges to ensure your agent is performing as expected.
Open-source: Fully open-source, allowing you to run the entire stack on your own servers, including LiveKit server, one of the most widely used WebRTC media servers.
Installation
To install the core Agents library, along with plugins for popular model providers:
Documentation on the framework and how to use it can be found here
Building with AI coding agents
If you're using an AI coding assistant to build with LiveKit Agents, we recommend the following setup for the best results:
Install the LiveKit Docs MCP server — Gives your coding agent access to up-to-date LiveKit documentation, code search across LiveKit repositories, and working examples.
Install the LiveKit Agent Skill — Provides your coding agent with architectural guidance and best practices for building voice AI applications, including workflow design, handoffs, tasks, and testing patterns.
The Agent Skill works best alongside the MCP server: the skill teaches your agent how to approach building with LiveKit, while the MCP server provides the current API details to implement it correctly.
Core concepts
Agent: An LLM-based application with defined instructions.
AgentSession: A container for agents that manages interactions with end users.
entrypoint: The starting point for an interactive session, similar to a request handler in a web server.
AgentServer: The main process that coordinates job scheduling and launches agents for user sessions.
Usage
Simple voice agent
from livekit.agents import (
Agent,
AgentServer,
AgentSession,
JobContext,
RunContext,
cli,
function_tool,
inference,
)
@function_tool
async def lookup_weather(
context: RunContext,
location: str,
):
"""Used to look up weather information."""
return {"weather": "sunny", "temperature": 70}
server = AgentServer()
@server.rtc_session()
async def entrypoint(ctx: JobContext):
session = AgentSession(
vad=inference.VAD(),
# any combination of STT, LLM, TTS, or realtime API can be used
# this example shows LiveKit Inference, a unified API to access different models via LiveKit Cloud
# to use model provider keys directly, replace with the following:
# from livekit.plugins import deepgram, openai, cartesia
# stt=deepgram.STT(model="nova-3"),
# llm=openai.LLM(model="gpt-4.1-mini"),
# tts=cartesia.TTS(model="sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
stt=inference.STT("deepgram/nova-3", language="multi"),
llm=inference.LLM("google/gemma-4-31b-it"), # low-latency gemma, hosted on LiveKit
tts=inference.TTS("cartesia/sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
)
agent = Agent(
instructions="You are a friendly voice assistant built by LiveKit.",
tools=[lookup_weather],
)
await session.start(agent=agent, room=ctx.room)
await session.generate_reply(instructions="greet the user and ask about their day")
if __name__ == "__main__":
cli.run_app(server)
You'll need the following environment variables for this example:
LIVEKIT_URL
LIVEKIT_API_KEY
LIVEKIT_API_SECRET
Multi-agent handoff
This code snippet is abbreviated. For the full example, see the LiveKit docs
...
class IntroAgent(Agent):
def __init__(self) -> None:
super().__init__(
instructions=f"You are a story teller. Your goal is to gather a few pieces of information from the user to make the story personalized and engaging."
"Ask the user for their name and where they are from"
)
async def on_enter(self):
self.session.generate_reply(instructions="greet the user and gather information")
@function_tool
async def information_gathered(
self,
context: RunContext,
name: str,
location: str,
):
"""Called when the user has provided the information needed to make the story personalized and engaging.
Args:
name: The name of the user
location: The location of the user
"""
context.userdata.name = name
context.userdata.location = location
story_agent = StoryAgent(name, location)
return story_agent, "Let's start the story!"
class StoryAgent(Agent):
def __init__(self, name: str, location: str) -> None:
super().__init__(
instructions=f"You are a storyteller. Use the user's information in order to make the story personalized."
f"The user's name is {name}, from {location}",
# override the default model, switching to Realtime API from standard LLMs
llm=openai.realtime.RealtimeModel(voice="echo"),
chat_ctx=chat_ctx,
)
async def on_enter(self):
self.session.generate_reply()
@server.rtc_session()
async def entrypoint(ctx: JobContext):
userdata = StoryData()
session = AgentSession[StoryData](
vad=inference.VAD(),
stt="deepgram/nova-3",
llm="google/gemma-4-31b-it", # low-latency gemma, hosted on LiveKit
tts="cartesia/sonic-3:9626c31c-bec5-4cca-baa8-f8ba9e84c8bc",
userdata=userdata,
)
await session.start(
agent=IntroAgent(),
room=ctx.room,
)
...
Testing
Automated tests are essential for building reliable agents, especially with the non-deterministic behavior of LLMs. LiveKit Agents include native test integration to help you create dependable agents.
@pytest.mark.asyncio
async def test_no_availability() -> None:
llm = google.LLM()
async with AgentSession(llm=llm) as sess:
await sess.start(MyAgent())
result = await sess.run(
user_input="Hello, I need to place an order."
)
result.expect.skip_next_event_if(type="message", role="assistant")
result.expect.next_event().is_function_call(name="start_order")
result.expect.next_event().is_function_call_output()
await (
result.expect.next_event()
.is_message(role="assistant")
.judge(llm, intent="assistant should be asking the user what they would like")
)
Runs your agent in terminal mode, enabling local audio input and output for testing.
This mode doesn't require external servers or dependencies and is useful for quickly validating behavior.
Developing with LiveKit clients
python myagent.py dev
Starts the agent server and enables hot reloading when files change. This mode allows each process to host multiple concurrent agents efficiently.
The agent connects to LiveKit Cloud or your self-hosted server. Set the following environment variables:
LIVEKIT_URL
LIVEKIT_API_KEY
LIVEKIT_API_SECRET
You can connect using any LiveKit client SDK or telephony integration.
To get started quickly, try the Agents Playground.
Running for production
python myagent.py start
Runs the agent with production-ready optimizations.
License
The Agents framework is licensed under Apache-2.0. The LiveKit turn detection models are licensed under the LiveKit Model License.
Contributing
The Agents framework is under active development in a rapidly evolving field. We welcome and appreciate contributions of any kind, be it feedback, bugfixes, features, new plugins and tools, or better documentation. You can file issues under this repo, open a PR, or chat with us in the LiveKit community.
Development setup
This project uses uv for package management. To install dependencies for development:
uv sync --all-extras --dev
Examples
This project includes many examples in the examples directory. To run them, create the file examples/.env with credentials for LiveKit Server and any necessary model providers (see examples/.env.example), then run:
Unit tests are in the tests directory and can be run with:
uv run pytest --unit
Integration tests for each plugin require various API credentials and run automatically in GitHub CI for PRs submitted by project maintainers. See the tests workflow for details.
Formatting
This project uses ruff for formatting and linting:
What is this?LiveKit's open-source Python framework for building realtime voice AI agents: programmable participants that run on servers, can see, hear, and understand, and talk to users over WebRTC.
Is it a hosted API I call, or code I run myself?A code framework, not a hosted service: you write agent logic in Python and run the agent processes yourself, either self-hosting the whole stack (including the LiveKit WebRTC server) or connecting to LiveKit Cloud.
How do I get a first agent running?Install the core Agents library plus plugins for your model providers, set the LIVEKIT_URL, LIVEKIT_API_KEY and LIVEKIT_API_SECRET environment variables, then start it in terminal mode to talk to your agent through the local mic and speakers.
Do I need a LiveKit server or paid API keys just to try it?Terminal-mode testing needs no external servers or dependencies at all. Real sessions with clients or phone calls need a LiveKit deployment — LiveKit Cloud or your own self-hosted server — plus credentials for the model providers you picked.
Does it lock me into LiveKit's paid cloud?No — the framework and the LiveKit server are fully open-source, so you can run the entire stack on your own machines; LiveKit Cloud is just the faster way to start.
Is everything under a permissive license, including the models?The Agents framework is Apache-2.0, but the bundled turn detection models ship under a separate LiveKit Model License — worth checking before commercial redistribution of the models.
Which speech and LLM providers can I plug in?It ships a plugin ecosystem of STT, LLM, TTS and Realtime API integrations that you mix and match per use case — examples cover OpenAI and Google's Gemini Live, among others.
Can my agent make and answer real phone calls?Yes — it integrates with LiveKit's telephony stack, so the same agent can place outbound calls and receive calls from regular phones.
How does it stop the agent from interrupting people mid-sentence?A built-in semantic turn detection model, based on a transformer, decides when the user has actually finished their turn, which reduces premature interruptions.
Can I write automated tests for a voice agent?Yes — the framework has native test integration: you write tests and use judges to keep agents dependable despite the non-deterministic behavior of LLMs.
LiveKit's open-source Python framework for building realtime voice AI agents: programmable participants that run on servers, can see, hear, and understand, and talk to users over WebRTC.