GVS5H README

The project's README from github.com/slee-persis/GVS5H. As one markdown file: README.md.

GVS5H: Five Qwen3.8-27B Models Match Claude Fable 5 on LiveCodeBench Hard

GVS5H: Gorilla vs. 5 Humans

Five small monkeys stacked one on another, together as tall as a single gorilla beside them
slee-persis%2FGVS5H | Trendshift

Results

Manager vs single call, six models — LCB-100, 5 passes, 128k max tokens, reasoning ON

What one pass costs — LCB-100, 5 passes, single call vs manager, against Fable 5

Abstract. Frontier coding performance is typically bought with larger proprietary models at high cost. We introduce ledger-based zero-shot self-orchestration, a training-free method in which fresh instances of one model decompose problems and coordinate through a shared filesystem holding a plan, notes and current solution. Across eleven open and closed-weight models on the 100 latest hard LiveCodeBench problems, the method yields gains of up to 25.6 percentage points on pinned backends and offers two routes to frontier-level accuracy. Orchestrated Qwen3.8 Flash Next reaches 93.0% pass@1 against Fable 5's 90.4% at 9% of the cost, and locally served, open-weight Qwen3.8-27B rises from 66.8% to 92.4%. Gains are not universal: some models are unchanged or worse. Transcript analysis attributes the gain to decomposition and persistent context. Inference-time organization can approach frontier coding accuracy at a fraction of the cost, or slightly exceed it on self-hostable weights.

— the paper

Running the code

Needs uv and an API key for the model you want to test.

cd codebase/v2-current
export OPENAI_API_KEY=...

LCB_RELEASE=release_v6 \
ESCALATION_CLOUD_MAX_TOKENS=128000 \
ESCALATION_CLOUD_TIMEOUT=7200 \
MULTIAGENT_MODEL=openai:gpt-5.6-terra \
uv run --no-project --python 3.12 --with 'datasets<4' --with numpy --with anthropic \
  python escalation/run_bench.py --engine multiagent --only lcb --lcb 100 --parallel 8

License

Code is under the MIT License. The paper, figures and run data are under CC BY 4.0. The LiveCodeBench fork, the benchmark problem statements and the LaTeX template files keep their own licenses. See NOTICE.md for which license covers which path.

GVS5H (slee-persis/GVS5H)

Searching Database…

752 ·GitHub
GVS5H
这是什么？GVS5H 是一篇研究论文的代码和实验数据，该论文提出了「基于账本的零样本自编排」：同一个模型的新实例分解一个编码问题，并通过一个存放计划、笔记和当前解法的共享文件系统进行协调，让多个廉价的开源模型能够达到或接近一个大得多的专有模型的准确率。
这是通用的智能体框架，还是某篇特定论文的实验代码？它是论文背后的参考实现，不是通用框架：它针对 LiveCodeBench 题目运行多智能体编排器或普通的单次调用基线，并输出 pass@1 分数。
怎么运行这些实验？安装 uv，获取你想测试的后端的 API 密钥，然后用 --engine multiagent 或 --engine single 运行脚本；pass@1 分数最后会打印出来，完整结果写入 runs/results.json。
支持哪些模型后端？Anthropic、DashScope（Qwen）和 OpenRouter 开箱即受支持，各自需要自己的 API 密钥。
这种编排技巧对每个模型的帮助都一样吗？不一样 —— 论文明确指出收益并非普适；与单次调用基线相比，一些被测模型没有提升，甚至出现退步。
用什么基准来评估这些模型？LiveCodeBench 最新的 100 道难题，按 pass@1 准确率评分。
在对比中，Claude Fable 5 是谁或是什么？Fable 5 是论文对前沿专有 Claude 模型的称呼，用作编排后的 Qwen 配置所对照的准确率和成本基线。
这种编排方法为什么省钱？它不调用一个大型专有模型，而是让多个更便宜的模型实例通过共享笔记和计划进行协调，使编排后的最佳 Qwen3.8 配置以约 9% 于 Fable 5 的成本达到 93.0% 的 pass@1。
运行输出保存在哪里？最终 pass@1 分数保存在 runs/results.json，每次运行的工作文件（计划、笔记、解法）保存在 runs/ws/。
What are similar projects?context-language-models, ruflo, hive, Internal-Safety-Collapse, agentcraft, PocketFlow, Agent-Reflex, ASPIRE
Which topics is it in?benchmarking, llm-evaluation, openrouter, multi-agent-systems, paper, code-generation, qwen, large-language-models, academic-research, local-llm, orchestration, agent-orchestration
Creator
slee-persis
Size
93.7 MB
Stars
752
Updated
CommitSep 23
Age
1mo
Commits · 90 Days