llm-council README

The project's README from github.com/karpathy/llm-council. As one markdown file: README.md.

LLM Council

llmcouncil

The idea of this repo is that instead of asking a question to your favorite LLM provider (e.g. OpenAI GPT 5.1, Google Gemini 3.0 Pro, Anthropic Claude Sonnet 4.5, xAI Grok 4, eg.c), you can group them into your "LLM Council". This repo is a simple, local web app that essentially looks like ChatGPT except it uses OpenRouter to send your query to multiple LLMs, it then asks them to review and rank each other's work, and finally a Chairman LLM produces the final response.

In a bit more detail, here is what happens when you submit a query:

  1. Stage 1: First opinions. The user query is given to all LLMs individually, and the responses are collected. The individual responses are shown in a "tab view", so that the user can inspect them all one by one.
  2. Stage 2: Review. Each individual LLM is given the responses of the other LLMs. Under the hood, the LLM identities are anonymized so that the LLM can't play favorites when judging their outputs. The LLM is asked to rank them in accuracy and insight.
  3. Stage 3: Final response. The designated Chairman of the LLM Council takes all of the model's responses and compiles them into a single final answer that is presented to the user.

Vibe Code Alert

This project was 99% vibe coded as a fun Saturday hack because I wanted to explore and evaluate a number of LLMs side by side in the process of reading books together with LLMs. It's nice and useful to see multiple responses side by side, and also the cross-opinions of all LLMs on each other's outputs. I'm not going to support it in any way, it's provided here as is for other people's inspiration and I don't intend to improve it. Code is ephemeral now and libraries are over, ask your LLM to change it in whatever way you like.

Setup

1. Install Dependencies

The project uses uv for project management.

Backend:

uv sync

Frontend:

cd frontend
npm install
cd ..

2. Configure API Key

Create a .env file in the project root:

OPENROUTER_API_KEY=sk-or-v1-...

Get your API key at openrouter.ai. Make sure to purchase the credits you need, or sign up for automatic top up.

3. Configure Models (Optional)

Edit backend/config.py to customize the council:

COUNCIL_MODELS = [
    "openai/gpt-5.1",
    "google/gemini-3-pro-preview",
    "anthropic/claude-sonnet-4.5",
    "x-ai/grok-4",
]

CHAIRMAN_MODEL = "google/gemini-3-pro-preview"

Running the Application

Option 1: Use the start script

./start.sh

Option 2: Run manually

Terminal 1 (Backend):

uv run python -m backend.main

Terminal 2 (Frontend):

cd frontend
npm run dev

Then open http://localhost:5173 in your browser.

Tech Stack

llm-council (karpathy/llm-council)

Searching Database…

这是什么?LLM Council 是一个本地网页应用,它通过 OpenRouter 把你的问题发给一组 LLM。每个 LLM 匿名评审并给其他模型的回答排名,然后由指定的 Chairman 模型把所有内容整合成一个最终回答。
这是托管服务,还是需要自己运行的程序?它是一个自托管的本地网页应用,不是托管产品:由 FastAPI 后端和 React/Vite 前端组成,在你自己的电脑上运行,界面和使用感受都类似 ChatGPT。
怎么让它跑起来?先用 uv 安装后端依赖,用 npm 安装前端依赖,然后可以使用提供的启动脚本,或者在两个独立的终端里分别运行 FastAPI 后端和 Vite 前端,最后在浏览器中打开应用。
我需要 OpenRouter API 密钥吗?需要。把你的 OpenRouter API 密钥写进 .env 文件。由于 OpenRouter 按每次模型调用计费,而整个委员会会调用多个模型,所以你需要购买额度或开启自动充值。
这个项目还在维护吗?不再维护。作者称它是一个周末随手做的 hack,其中 99% 的代码是用“氛围编程”生成的,并明确表示不会再提供支持或改进。如果你想修改,建议自己让 LLM 帮忙改。
我的对话数据会去哪里?不会存入数据库或外部服务进行存储;对话以普通 JSON 文件的形式保存在本地的 data/conversations/ 目录中。
怎样更换委员会中的模型?编辑 backend/config.py,你可以在其中添加、删除或替换委员会使用的 LLM,以及 Chairman 模型。
委员会究竟是怎样产生一个最终答案的?每个 LLM 先独立作答,并匿名给其他模型的回答排名;随后,指定的 Chairman 模型会汇总所有回答,生成一个展示给用户的最终答案。
为什么 karpathy 要做这个项目?他想要一个快速的方法,能并排比较几个 LLM,还能看到它们对彼此输出的看法;那是他在周末用 LLM 一起读书时的需求。
Creator
karpathy
Size
0.3 MB
Stars
25.5K
Updated
CommitNov 22
Age
10mo
Commits · 90 Days