What is this?
GVS5H is the code and experiment data for a research paper introducing 'ledger-based zero-shot self-orchestration': fresh instances of one model decompose a coding problem and coordinate through a shared filesystem holding a plan, notes and the current solution, letting several cheap open models match or approach the accuracy of a much larger proprietary model.
GVS5H is the code and experiment data for a research paper introducing 'ledger-based zero-shot self-orchestration': fresh instances of one model decompose a coding problem and coordinate through a shared filesystem holding a plan, notes and the current solution, letting several cheap open models match or approach the accuracy of a much larger proprietary model.Is this a general agent framework or a specific paper's experiment code?It is the reference implementation behind the paper, not a general-purpose framework: it runs either the multi-agent orchestrator or a plain single-call baseline against LiveCodeBench problems and prints a pass@1 score.
It is the reference implementation behind the paper, not a general-purpose framework: it runs either the multi-agent orchestrator or a plain single-call baseline against LiveCodeBench problems and prints a pass@1 score.How do I run the experiments?Install uv, get an API key for the backend you want to test, then run the script with --engine multiagent or --engine single; the pass@1 score prints at the end and full results go to runs/results.json.
Install uv, get an API key for the backend you want to test, then run the script with --engine multiagent or --engine single; the pass@1 score prints at the end and full results go to runs/results.json.Which model backends are supported?Anthropic, DashScope (Qwen) and OpenRouter are supported out of the box, each needing its own API key.
Anthropic, DashScope (Qwen) and OpenRouter are supported out of the box, each needing its own API key.Does the orchestration trick help every model equally?No - the paper explicitly notes the gains aren't universal; some tested models show no improvement or even regress compared to the single-call baseline.
No - the paper explicitly notes the gains aren't universal; some tested models show no improvement or even regress compared to the single-call baseline.What benchmark is used to evaluate the models?The 100 latest hard problems from LiveCodeBench, scored by pass@1 accuracy.
The 100 latest hard problems from LiveCodeBench, scored by pass@1 accuracy.Who or what is Claude Fable 5 in the comparison?Fable 5 is the paper's name for the frontier proprietary Claude model used as the accuracy and cost baseline that the orchestrated Qwen setups are measured against.
Fable 5 is the paper's name for the frontier proprietary Claude model used as the accuracy and cost baseline that the orchestrated Qwen setups are measured against.Why does the orchestration method save money?Instead of calling one large proprietary model, several cheaper model instances coordinate through shared notes and a plan, letting the best orchestrated Qwen3.8 setup hit 93.0% pass@1 at roughly 9% of Fable 5's cost.
Instead of calling one large proprietary model, several cheaper model instances coordinate through shared notes and a plan, letting the best orchestrated Qwen3.8 setup hit 93.0% pass@1 at roughly 9% of Fable 5's cost.Where are run outputs saved?Final pass@1 scores land in runs/results.json, while each run's working files (plan, notes, solution) are kept in runs/ws/.
Final pass@1 scores land in runs/results.json, while each run's working files (plan, notes, solution) are kept in runs/ws/.What are similar projects?context-language-models, ruflo, hive, agentcraft, PocketFlow, open-multi-agent, decider, ai-agents-the-definitive-guide
context-language-models, ruflo, hive, agentcraft, PocketFlow, open-multi-agent, decider, ai-agents-the-definitive-guideWhich topics is it in?benchmarking, llm-evaluation, openrouter, multi-agent-systems, paper, code-generation, qwen, academic-research, large-language-models, local-llm, orchestration, agent-orchestration
benchmarking, llm-evaluation, openrouter, multi-agent-systems, paper, code-generation, qwen, academic-research, large-language-models, local-llm, orchestration, agent-orchestration