HarnessEval-W
Agentifying the Evaluation of
Visual Worlds
The era of harnessed benchmarking: evaluation defines the taste of evolution.
Overview
A benchmark should deliver more than a scalar score: what makes an evaluation trustworthy is the reasoning that justifies the score. This is especially critical for world models, where judging a rollout requires understanding whether physics, causality, and world state evolve correctly. Humans spot such violations naturally, yet no existing benchmark automates this capability: metrics are computed brute-force, leaving no reasoning chain that can be examined or verified.
HarnessEval is an agentified evaluation pipeline that brings the harness paradigm from the LLM ecosystem to world model benchmarking. Rather than applying a fixed rubric, HarnessEval interprets the context of each evaluation case, decomposes the evaluation question into measurable sub-questions, and spawns specialized sub-agents, each equipped with tailored context and diagnostic tools to reason over its own sub-question. The parent agent then validates the gathered evidence and aggregates it into the final verdict. Every evaluation becomes a transparent evidence tree whose complete reasoning chain justifies the result.
News
- [2026/09/02] 🤝 Launched the Contribute page for case, skill, model, and concrete suggestion submissions.
- [2026/09/02] 🏆 Released the 100-case public set and its leaderboard.
- [2026/08/18] 📄 Paper now available.
- [2026/08/18] 🌐 Homepage with leaderboard-ready reports and benchmark resources is live.
- [2026/08/18] 🚀 Released the full HarnessEval benchmark, evaluation code, and metric backends.
How It Works
- Case-specific skill routing. Given a case (initial world, action, probe intent), the planner routes it to the skills that can legitimately evaluate it — and records an evidence-grounded reason for every skill it skips.
- Sub-agent reasoning. Each skill decomposes its evaluation into measurable sub-questions, each answered by a dedicated sub-agent against rollout evidence.
- Validated aggregation. The parent agent validates the collected evidence and aggregates it into the case score. The full trace — every question, answer, score, and supporting frame — is saved as an auditable case card.
Routing depends only on the case context, never on the model being evaluated, so every model faces the same questions on the same cases.
Installation
Create three environments:
harnesseval-main: launcher / CLIharnesseval-metrics: metric backendsharnesseval-pavrm: physical-plausibility backend
git clone --branch main --single-branch https://github.com/mirros-lab/harnesseval-w.git
cd harnesseval-w
conda env create -f docs/installation/main.environment.yml
conda env create -f docs/installation/metrics.environment.yml
conda env create -f docs/installation/pavrm.environment.yml
conda activate harnesseval-main
Configure credentials and paths:
cp config/example.env harnesseval.env
set -a; . ./harnesseval.env; set +a
The bundled demo in runs/example/results_example can be evaluated immediately after setup.
Usage
Evaluate a model's generated results:
# Evaluate generated videos and write run outputs.
harnesseval eval \
--results runs/example/results_example/generation \
--model-id seedance-2.0-standard \
--run-root runs/example/results_example/run \
--manifest runs/example/results_example/manifest.json \
--plan-root benchmark/plans
Check a completed run:
# Verify that an existing run has all expected scores.
harnesseval verify run \
--eval-root runs/example/results_example/run/harnesseval/models/seedance-2.0-standard/evaluation \
--manifest runs/example/results_example/manifest.json \
--model seedance-2.0-standard
Or use the bundled demo end to end:
# Run the bundled demo from the example directory.
cd runs/example/results_example
# Evaluate the demo outputs.
harnesseval eval --results generation --model-id seedance-2.0-standard --run-root run --manifest manifest.json --plan-root ../../benchmark/plans
# Verify the demo run.
harnesseval verify run --eval-root run/harnesseval/models/seedance-2.0-standard/evaluation --manifest manifest.json --model seedance-2.0-standard
What You Get
The bundled example already produces scores. A completed evaluation writes:
runs/example/results_example/run/harnesseval/models/<model-id>/evaluation/
├── summary.json # overall and per-family scores
├── leaderboard_latest.json
├── leaderboard_latest.csv
└── LEADERBOARD.md
Per-case artifacts and caches live under run/harnesseval/metric_cache/ and run/harnesseval/models/<model-id>/.
Extending HarnessEval
HarnessEval is a living benchmark, and contributions are welcome !
- Submit a new case with a new world, action, or probe family.
- Submit a new skill when a case needs a new kind of evaluation.
Keep submissions aligned with the existing benchmark format so they can plug into the bundled flow.
TODO
- Hosted submission & evaluation service (submit videos, get scores)
- Public contribution page for cases, skills, model additions, and concrete suggestions
- public set & model weights on HuggingFace
- Model generation adapters for text/image-, action-, and camera-conditioned models
- Evaluation code & example release
- ArXiv paper, homepage with interactive leaderboard, blog release
Acknowledge
This project builds upon the following excellent works:
- VBench — Video quality metrics
- WBench — World model benchmark
- WorldScore - World model benchmark
- Cosmos - Open source world model
- Lingbot World - Open source world model
- MiniMax H3 - Open source world model
- ... and many other excellent open-source projects
Citation
If you find HarnessEval useful, please cite:
@article{Mirros2026harnessevalw,
title = {HarnessEval-W: Agentifying the Evaluation of Visual Worlds},
author = {MirroS Team},
journal = {arXiv preprint arXiv:2608.16859},
year = {2026}
}
License
We release our code with Apache 2.0 License.
HarnessEval-W 是一条智能体化的评估流水线，用于给世界模型/视频生成模型做基准测试：它不采用暴力的单一标量分数，而是把每个评估案例拆成若干子问题，交给专门的子智能体回答，再把证据汇总成透明、可审计的结论。