llm-council README

The project's README from github.com/karpathy/llm-council. As one markdown file: README.md.

LLM Council

llmcouncil

The idea of this repo is that instead of asking a question to your favorite LLM provider (e.g. OpenAI GPT 5.1, Google Gemini 3.0 Pro, Anthropic Claude Sonnet 4.5, xAI Grok 4, eg.c), you can group them into your "LLM Council". This repo is a simple, local web app that essentially looks like ChatGPT except it uses OpenRouter to send your query to multiple LLMs, it then asks them to review and rank each other's work, and finally a Chairman LLM produces the final response.

In a bit more detail, here is what happens when you submit a query:

  1. Stage 1: First opinions. The user query is given to all LLMs individually, and the responses are collected. The individual responses are shown in a "tab view", so that the user can inspect them all one by one.
  2. Stage 2: Review. Each individual LLM is given the responses of the other LLMs. Under the hood, the LLM identities are anonymized so that the LLM can't play favorites when judging their outputs. The LLM is asked to rank them in accuracy and insight.
  3. Stage 3: Final response. The designated Chairman of the LLM Council takes all of the model's responses and compiles them into a single final answer that is presented to the user.

Vibe Code Alert

This project was 99% vibe coded as a fun Saturday hack because I wanted to explore and evaluate a number of LLMs side by side in the process of reading books together with LLMs. It's nice and useful to see multiple responses side by side, and also the cross-opinions of all LLMs on each other's outputs. I'm not going to support it in any way, it's provided here as is for other people's inspiration and I don't intend to improve it. Code is ephemeral now and libraries are over, ask your LLM to change it in whatever way you like.

Setup

1. Install Dependencies

The project uses uv for project management.

Backend:

uv sync

Frontend:

cd frontend
npm install
cd ..

2. Configure API Key

Create a .env file in the project root:

OPENROUTER_API_KEY=sk-or-v1-...

Get your API key at openrouter.ai. Make sure to purchase the credits you need, or sign up for automatic top up.

3. Configure Models (Optional)

Edit backend/config.py to customize the council:

COUNCIL_MODELS = [
    "openai/gpt-5.1",
    "google/gemini-3-pro-preview",
    "anthropic/claude-sonnet-4.5",
    "x-ai/grok-4",
]

CHAIRMAN_MODEL = "google/gemini-3-pro-preview"

Running the Application

Option 1: Use the start script

./start.sh

Option 2: Run manually

Terminal 1 (Backend):

uv run python -m backend.main

Terminal 2 (Frontend):

cd frontend
npm run dev

Then open http://localhost:5173 in your browser.

Tech Stack

llm-council (karpathy/llm-council)

multi-model
Searching Database…

25.5K ·GitHub
Llm-council
What is this?LLM Council is a local web app that sends your question to a group of LLMs through OpenRouter, has each of them anonymously review and rank the others' answers, then a designated Chairman LLM compiles everything into one final response.
Is this a hosted service or something I run myself?It's a self-hosted local web app, not a hosted product: a FastAPI backend plus a React/Vite frontend that you run on your own machine, looking and feeling like ChatGPT.
How do I get it running?Install backend dependencies with uv and frontend dependencies with npm, then either use the provided start script or run the FastAPI backend and the Vite frontend in two separate terminals and open the app in your browser.
Do I need an OpenRouter API key?Yes, you put your OpenRouter API key in a .env file, and since OpenRouter bills per model call across the whole council you need to buy credits or enable auto top-up.
Is this actively maintained?No, the author built it as a 99% vibe-coded weekend hack and explicitly says he won't support or improve it further, suggesting you ask an LLM to modify it yourself if you want changes.
Where does my conversation data go?Nothing goes to a database or external service for storage; conversations are saved as plain JSON files locally in data/conversations/.
How do I change which models are in the council?Edit backend/config.py, which lets you add, remove, or swap the LLMs and the Chairman model used for the council.
How does the council actually produce one final answer?After every LLM answers independently and anonymously ranks the others' answers, the designated Chairman LLM takes all the responses and compiles a single final answer shown to the user.
Why did karpathy build this?He wanted a quick way to compare several LLMs side by side, including their opinions of each other's outputs, while reading books together with LLMs on a weekend.
What are similar projects?whirl, searchmirror, DB-GPT, chatgpt-web, jevbox, gin-vue-admin, screenshot-to-code, comfyEz
Which topics is it in?multi-model, openrouter, fastapi, vite, json, chatbot, backend, web-app, markdown, react, frontend, api
Creator
karpathy
Size
0.3 MB
Stars
25.5K
Updated
CommitNov 22
Age
10mo
Commits · 90 Days