308 ·GitHub
What is this?
HarnessEval-W is an agentified evaluation pipeline for benchmarking world/video-generation models: instead of a brute-force scalar score, it decomposes each evaluation case into sub-questions answered by specialized sub-agents, then aggregates the evidence into a transparent, auditable verdict.
HarnessEval-W is an agentified evaluation pipeline for benchmarking world/video-generation models: instead of a brute-force scalar score, it decomposes each evaluation case into sub-questions answered by specialized sub-agents, then aggregates the evidence into a transparent, auditable verdict.Is this a single metric tool or a full agentic pipeline?It is a multi-agent pipeline: a planner routes each case to the skills that can legitimately judge it, dedicated sub-agents answer measurable sub-questions against rollout evidence, and a parent agent validates and aggregates that evidence into the final case score.
It is a multi-agent pipeline: a planner routes each case to the skills that can legitimately judge it, dedicated sub-agents answer measurable sub-questions against rollout evidence, and a parent agent validates and aggregates that evidence into the final case score.How do I get it running?Create three separate environments - harnesseval-main (launcher/CLI), harnesseval-metrics (metric backends), and harnesseval-pavrm (physical-plausibility backend) - configure credentials and paths, then run the bundled demo in runs/example/resultsexample.
Create three separate environments - harnesseval-main (launcher/CLI), harnesseval-metrics (metric backends), and harnesseval-pavrm (physical-plausibility backend) - configure credentials and paths, then run the bundled demo in runs/example/resultsexample.What do I need before running it?Python plus three separately managed environments for the CLI launcher, the metric backends, and the physical-plausibility backend, each with its own credentials and paths configured first.
Python plus three separately managed environments for the CLI launcher, the metric backends, and the physical-plausibility backend, each with its own credentials and paths configured first.Any install pitfalls I should expect?Setup requires juggling three separate environments instead of one, and there's no hosted service yet - the 'submit videos, get scores' option is still listed as an unfinished TODO, so everything runs locally for now.
Setup requires juggling three separate environments instead of one, and there's no hosted service yet - the 'submit videos, get scores' option is still listed as an unfinished TODO, so everything runs locally for now.What is a case card?It is the auditable artifact HarnessEval saves per evaluation: every sub-question, sub-agent answer, score, and supporting frame that justified the final verdict.
It is the auditable artifact HarnessEval saves per evaluation: every sub-question, sub-agent answer, score, and supporting frame that justified the final verdict.Which world models can it evaluate?It supports text-, image-, action-, and camera-conditioned world models, including Kling, MiniMax H3, Seedance, WAN, and Cosmos, via model generation adapters.
It supports text-, image-, action-, and camera-conditioned world models, including Kling, MiniMax H3, Seedance, WAN, and Cosmos, via model generation adapters.Is there a public leaderboard?Yes, a 100-case public set with its leaderboard was released alongside the full benchmark and metric code in September 2026.
Yes, a 100-case public set with its leaderboard was released alongside the full benchmark and metric code in September 2026.Why not just use a fixed scoring rubric?Fixed metrics are computed brute-force with no inspectable reasoning chain, so HarnessEval instead routes cases to specialized sub-agents that reason over evidence and produce a justifiable, traceable score.
Fixed metrics are computed brute-force with no inspectable reasoning chain, so HarnessEval instead routes cases to specialized sub-agents that reason over evidence and produce a justifiable, traceable score.What are similar projects?CodeVideoModel, object-permanence, RSIGame, context-language-models, Proteus, ParseBench, WanGP, Zero-WAM
CodeVideoModel, object-permanence, RSIGame, context-language-models, Proteus, ParseBench, WanGP, Zero-WAMWhich topics is it in?seedance, world-model, wan, minimax-h3, harness-engineering, arxiv, dataset, harness, evaluation, video-generation, generative-ai, huggingface
seedance, world-model, wan, minimax-h3, harness-engineering, arxiv, dataset, harness, evaluation, video-generation, generative-ai, huggingface