OpenLLM README

The project's README from github.com/bentoml/OpenLLM. As one markdown file: README.md.

🦾 OpenLLM: Self-Hosting LLMs Made Easy

License: Apache-2.0 Releases CI X Community

OpenLLM allows developers to run any open-source LLMs (Llama 3.3, Qwen2.5, Phi3 and more) or custom models as OpenAI-compatible APIs with a single command. It features a built-in chat UI, state-of-the-art inference backends, and a simplified workflow for creating enterprise-grade cloud deployment with Docker, Kubernetes, and BentoCloud.

Understand the design philosophy of OpenLLM.

Get Started

Run the following commands to install OpenLLM and explore it interactively.

pip install openllm  # or pip3 install openllm
openllm hello

hello

Supported models

OpenLLM supports a wide range of state-of-the-art open-source LLMs. You can also add a model repository to run custom models with OpenLLM.

Model Parameters Required GPU Start a Server
deepseek r1-671b 80Gx16 openllm serve deepseek:r1-671b
gemma2 2b 12G openllm serve gemma2:2b
gemma3 3b 12G openllm serve gemma3:3b
jamba1.5 mini-ff0a 80Gx2 openllm serve jamba1.5:mini-ff0a
llama3.1 8b 24G openllm serve llama3.1:8b
llama3.2 1b 24G openllm serve llama3.2:1b
llama3.3 70b 80Gx2 openllm serve llama3.3:70b
llama4 17b16e 80Gx8 openllm serve llama4:17b16e
mistral 8b-2410 24G openllm serve mistral:8b-2410
mistral-large 123b-2407 80Gx4 openllm serve mistral-large:123b-2407
phi4 14b 80G openllm serve phi4:14b
pixtral 12b-2409 80G openllm serve pixtral:12b-2409
qwen2.5 7b 24G openllm serve qwen2.5:7b
qwen2.5-coder 3b 24G openllm serve qwen2.5-coder:3b
qwq 32b 80G openllm serve qwq:32b

For the full model list, see the OpenLLM models repository.

Start an LLM server

To start an LLM server locally, use the openllm serve command and specify the model version.

[!NOTE] OpenLLM does not store model weights. A Hugging Face token (HF_TOKEN) is required for gated models.

  1. Create your Hugging Face token here.
  2. Request access to the gated model, such as meta-llama/Llama-3.2-1B-Instruct.
  3. Set your token as an environment variable by running:
    export HF_TOKEN=<your token>

openllm serve llama3.2:1b

The server will be accessible at http://localhost:3000, providing OpenAI-compatible APIs for interaction. You can call the endpoints with different frameworks and tools that support OpenAI-compatible APIs. Typically, you may need to specify the following:

Here are some examples:

OpenAI Python client 
from openai import OpenAI

client = OpenAI(base_url='http://localhost:3000/v1', api_key='na')

# Use the following func to get the available models
# model_list = client.models.list()
# print(model_list)

chat_completion = client.chat.completions.create(
    model="meta-llama/Llama-3.2-1B-Instruct",
    messages=[
        {
            "role": "user",
            "content": "Explain superconductors like I'm five years old"
        }
    ],
    stream=True,
)
for chunk in chat_completion:
    print(chunk.choices[0].delta.content or "", end="")
LlamaIndex 
from llama_index.llms.openai import OpenAI

llm = OpenAI(api_bese="http://localhost:3000/v1", model="meta-llama/Llama-3.2-1B-Instruct", api_key="dummy")
...

Chat UI

OpenLLM provides a chat UI at the /chat endpoint for the launched LLM server at http://localhost:3000/chat.

openllm_ui

Chat with a model in the CLI

To start a chat conversation in the CLI, use the openllm run command and specify the model version.

openllm run llama3:8b

Model repository

A model repository in OpenLLM represents a catalog of available LLMs that you can run. OpenLLM provides a default model repository that includes the latest open-source LLMs like Llama 3, Mistral, and Qwen2, hosted at this GitHub repository. To see all available models from the default and any added repository, use:

openllm model list

To ensure your local list of models is synchronized with the latest updates from all connected repositories, run:

openllm repo update

To review a model’s information, run:

openllm model get llama3.2:1b

Add a model to the default model repository

You can contribute to the default model repository by adding new models that others can use. This involves creating and submitting a Bento of the LLM. For more information, check out this example pull request.

Set up a custom repository

You can add your own repository to OpenLLM with custom models. To do so, follow the format in the default OpenLLM model repository with a bentos directory to store custom LLMs. You need to build your Bentos with BentoML and submit them to your model repository.

First, prepare your custom models in a bentos directory following the guidelines provided by BentoML to build Bentos. Check out the default model repository for an example and read the Developer Guide for details.

Then, register your custom model repository with OpenLLM:

openllm repo add <repo-name> <repo-url>

Note: Currently, OpenLLM only supports adding public repositories.

Deploy to BentoCloud

OpenLLM supports LLM cloud deployment via BentoML, the unified model serving framework, and BentoCloud, an AI inference platform for enterprise AI teams. BentoCloud provides fully-managed infrastructure optimized for LLM inference with autoscaling, model orchestration, observability, and many more, allowing you to run any AI model in the cloud.

Sign up for BentoCloud for free and log in. Then, run openllm deploy to deploy a model to BentoCloud:

openllm deploy llama3.2:1b --env HF_TOKEN

[!NOTE] If you are deploying a gated model, make sure to set HF_TOKEN in enviroment variables.

Once the deployment is complete, you can run model inference on the BentoCloud console:

bentocloud_ui

Community

OpenLLM is actively maintained by the BentoML team. Feel free to reach out and join us in our pursuit to make LLMs more accessible and easy to use 👉 Join our Slack community!

Contributing

As an open-source project, we welcome contributions of all kinds, such as new features, bug fixes, and documentation. Here are some of the ways to contribute:

Acknowledgements

This project uses the following open-source projects:

We are grateful to the developers and contributors of these projects for their hard work and dedication.

OpenLLM (bentoml/OpenLLM)

Searching Database…

12.6K ·GitHub
OpenLLM
What is this?OpenLLM lets you run open-source LLMs like Llama, DeepSeek, Mistral and Qwen as OpenAI-compatible API endpoints with a single command, including a built-in chat UI and cloud deployment support.
Is this a library I import, or a server I run?It is a CLI tool/server: you run 'openllm serve' to start an inference server exposing OpenAI-compatible endpoints, plus a chat UI.
How do I get a model running?Install OpenLLM, then run 'openllm serve : ' (e.g. 'openllm serve llama3.1:8b') to start a local server with OpenAI-compatible APIs.
Do I need a Hugging Face token to use any model?Only for gated models: you need to create a Hugging Face token, request access to the gated model, and set it as the HFTOKEN environment variable.
How much GPU memory do I actually need?It varies hugely by model, from 12G for small models like Gemma2 2B up to 80Gx16 for DeepSeek R1 671B, so picking the right model for your hardware matters.
Can I use custom models not in the default list?Yes, you can add a model repository to run custom models with OpenLLM beyond its default supported model list.
Does it support cloud deployment beyond local serving?Yes, it simplifies enterprise-grade cloud deployment with Docker, Kubernetes, and BentoCloud, and you can run model inference via the BentoCloud console once deployed.
What inference backend does OpenLLM use under the hood?It uses vLLM as the production-level LLM backend and BentoML for model serving, plus astral-sh/uv for fast dependency installation.
Can I use the API with existing OpenAI client libraries?Yes, the server exposes OpenAI-compatible endpoints that work with tools like the OpenAI Python client and LlamaIndex.
What are similar projects?ludwig, serve, petals, GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold, Soup, llama-gpt, skypilot, Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install
Which topics is it in?llm-ops, llama2, mistral, llm-serving, llmops, mlops, llama, llm-inference, chat, fine-tuning, openai-compatible, local-llm
Creator
bentoml
Size
41.8 MB
Stars
12.6K
Updated
CommitMay 29Releasev0.6.30Apr 21
Age
3y
Commits · 90 Days