harness2 README

项目的 README,来自 github.com/google-research/harness2。合并为一个 markdown 文件:README.md。

Harness²
Recursive Agent Harnessing for an Open World

Paper Website Benchmark guides License

Install, set up, and run JobBench: uv pip install -e '.[render]', harness2-setup jb, then harness2 --bench jb --k 1 --mode parallel. Click for prerequisites and commands.

A harness is the system around a model that enables it to act as an agent: prompts, tools, memory, skills, orchestration, and more. Harnessing is the process of optimizing that system.

Harness² is a plug-and-play framework for recursive agent harnessing in an open world, where agents encounter tasks unseen during harness design. It connects two levels of recursion while keeping model weights fixed:

Harness² pipeline. Task-level recursion: an agent executor runs the initial harness, a harness editor proposes invariant and domain edits, candidate probes execute them, and a harness composer combines them into the final harness. Harnessing-level recursion: an experience reflector reflects on the step and updates the improvement harness used for the next task.

Across three professional benchmarks and four model–harness configurations, a single Harness² step improves GLM-5 and Gemini 3.5 Flash over their base harnesses, with gains of up to 11.5 points on WorkBuddy-Bench, 4.5 points on JobBench, and 11.8 percentage points on LAB's partial pass rate. Increasing the task-level recursion budget brings further gains.

The release supports parallel and sequential recursion, OpenCode and Codex executors, and JobBench, LAB, and WorkBuddy-Bench.

Video overview

https://github.com/user-attachments/assets/76b10548-b14b-4c8f-9aa5-a93662df4d56

Contents

Harness components

Harness² exposes eight editable components around the model: system prompt, guardrails, memory, tool guidance, subagents, skills, scripts, and plugins. They control context construction, the tool interface, delegation, and behavior inside the tool loop.

Eight harness components grouped by runtime role: system prompt, guardrails, memory, and skills construct context; tool guidance and scripts define the tool interface; subagents orchestrate delegated work; plugins control middleware.

Install

Use Linux, Python 3.12, and uv. From the repository root:

uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -e '.[render]'
harness2-demo --output runs/demo --k 2

The local demo runs both recursion modes with deterministic editors and real Python executions. It requires no model credentials.

For model-backed benchmarks, install Node.js 20+, the Google Cloud CLI, and the benchmark dependencies. LAB needs Pandoc and Poppler; WorkBuddy-Bench needs Docker. Then configure your models:

cp .env.example .env
# Edit .env with your Vertex AI project and model IDs.
gcloud auth application-default login

See the setup guide for model settings, authentication alternatives, and the Codex executor.

Run a benchmark

The following example installs LAB and runs one task, then evaluates and reports its result. Model-backed runs incur API costs; start with a small task selection.

harness2-setup lab
harness2-check --bench lab
harness2 --bench lab --k 1 --mode parallel \
  --only-domains antitrust-competition --max-tasks 1
harness2-judge --bench lab --k 1 --mode parallel
harness2-report --bench lab --k 1 --mode parallel

Sequential recursion runs the same commands with --mode sequential. --k is the task-level recursion budget; see Experiments. Judging and reporting must use the same benchmark, executor, subset, --k, --mode, and seed as the refinement run. Add --dry-run to harness2 to preview task selection without model calls.

Benchmark Tasks Install Guide
JobBench Professional work harness2-setup jb JobBench
LAB Legal work harness2-setup lab LAB
WorkBuddy-Bench Office, web, and code harness2-setup wb WorkBuddy

Choose one benchmark per command. WorkBuddy also takes --subset office, --subset web, or --subset code on every command after setup. The benchmark guides cover each one's data, domains, and grading.

Experiments

Scale task-level recursion in two ways: parallel recursion explores multiple candidate harnesses and composes refinements from their execution evidence; sequential recursion carries the refined harness and accumulated evidence through successive Propose–Probe–Compose steps.

Two ways to scale task-level recursion. Parallel: probe k general and k domain-specific candidate harnesses, then produce k compositions from the shared evidence. Sequential: carry the refined harness and accumulated evidence through k Propose–Probe–Compose steps.

Parallel recursion explores wider; sequential recursion refines deeper.

Option Behavior
--k 1 One Propose–Probe–Compose step
--k K --mode parallel Probe K general and K domain-specific candidates, then produce K compositions
--k K --mode sequential Refine the harness through K successive steps
--substrate cx Use Codex instead of the default OpenCode executor

After each task, harnessing-level recursion retains the adopted harness and evidence-supported lessons to guide future refinement. See the experiment guide for task selection, concurrency, reproducibility, and baseline comparisons.

Web access

Executor network access is unrestricted, and Codex runs with its sandbox disabled. Public benchmark rubrics may be accessible online. Before evaluation, configure network restrictions to prevent access to benchmark answers and grading criteria.

Results and resume

Results are saved under runs/, organized by benchmark and subset where applicable. They include task summaries, deliverables, adopted harnesses, reflection notes, and evaluation results.

Repeat the same command to resume. Use a new HARNESS2_RUNS_DIR when changing models or task selection. See results and resume for details; each command also supports --help.

Citation

@article{xu2026harness2,
  title   = {Harness$^2$: Recursive Agent Harnessing for an Open World},
  author  = {Xu, Ruiyao and Chen, Yanfei and CuiZhu, Zhongying and Dalvi Mishra, Bhavana and Ming, Yifei and Yu, Han and Han, Rujun and Lee, Chen-Yu and Pfister, Tomas},
  journal = {arXiv preprint arXiv:2610.14224},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.14224}
}

Contributing

See CONTRIBUTING.md.

License

Harness² is licensed under Apache 2.0. Third-party code under third_party/, benchmarks, and datasets retain their upstream licenses; see third-party notices.

Disclaimer

This is not an officially supported Google product. This project is not eligible for the Google Open Source Software Vulnerability Rewards Program.

harness2 (google-research/harness2)

正在搜索数据库…

24.3K ·GitHub
这是什么?Harness2 是一个框架,通过两层递归优化 AI 模型周围的系统(提示词、工具、记忆、技能、编排),同时保持模型权重不变,使智能体能够处理在 harness 设计阶段未曾见过的任务。
这是某个具体的智能体,还是围绕智能体的框架?它是一个框架或工具包,提供八个可编辑的 harness 组件(系统提示词、防护规则、记忆、工具指引、子智能体、技能、脚本、插件),以及基准测试运行器和递归引擎,本身并不是一个单独的智能体。
怎样才能把它运行起来?在 Linux 系统上使用 Python 3.12 和 uv,克隆仓库后在仓库根目录运行安装步骤;本地演示使用确定性编辑器,无需任何模型凭证即可直接运行。
要运行基于真实模型的基准测试,需要安装什么?需要 Node.js 20+、Google Cloud CLI,以及各个基准测试自身的依赖项;LAB 还需要 Pandoc 和 Poppler,WorkBuddy-Bench 需要 Docker。
运行评测之前有没有需要注意的安全事项?执行器的网络访问不受限制,Codex 的沙箱也处于禁用状态,因此在评测前必须自行设置网络限制,防止智能体在线获取基准测试答案或评分标准。
Harness2 目前取得了哪些结果?在三个专业基准测试和四种模型与 harness 组合的配置中,单次 Harness2 迭代使 GLM-5 和 Gemini 3.5 Flash 相比其基础 harness 有所提升:在 WorkBuddy-Bench 上最多提高 11.5 分,在 JobBench 上提高 4.5 分,在 LAB 的部分通过率上提高 11.8 个百分点。
我可以用它运行哪些基准测试?它支持 JobBench(专业工作)、LAB(法律工作)和 WorkBuddy-Bench(办公、网页和代码任务),每个基准测试都可以通过各自的 harness2-setup 命令进行安装。
并行递归和顺序递归有什么区别?并行递归会探测多个通用和领域专用的候选 harness,并根据它们综合得到的证据组合出改进方案,探索范围更广;顺序递归则让一个 harness 依次经历多个“提议-探测-组合”步骤,优化得更深入。
它可以配合哪些智能体执行器使用?Harness2 默认使用 OpenCode 执行器,也可以通过 --substrate cx 参数切换为 Codex。
有哪些类似的项目?rrsi, FlashRSI, hermit, Recuris, Agensh, ASI-Bench, RSIGame, DeLM
大小
7.1 MB
星标
24.3K
更新
提交Nov 02
时长
1个月
提交 · 90 天