GamePhanes README

The project's README from github.com/GamePhanesStudio/GamePhanes. As one markdown file: README.md.

GameForgeBench

Terminal-Bench for interactive game-engine development agents.

A Harbor-compatible benchmark for coding agents that build, debug, and repair games through terminal workspaces and executable runtime feedback.

Homepage · Benchmark · Task Registry · Issues · Discussions

What Is GameForgeBench?

Terminal benchmarks often stop at files, commands, and exit codes. Interactive software has a second truth: the project must import, launch, accept controlled input, change runtime state, and produce the intended behavior. GameForgeBench evaluates coding agents on that complete loop.

instruction -> inspect -> edit -> run -> observe -> diagnose -> repair -> verify

The benchmark target is a coding agent, not a player bot. Evaluator-controlled probes provide runtime evidence; the agent is judged on engineering work, debugging, behavior, and regression resistance.

Open Release

The release corpus contains 81 completed executable tasks spanning Godot, Unity, Roblox, Minecraft, Unreal, Web, and generic engine projects. The tasks cover gameplay systems, engine/runtime work, UI and interaction, plugins, persistence, callbacks, build workflows, and delivery quality.

Each task is a self-contained contract with a starter project, a natural-language request, a reproducible environment, protected files, a reference solution, and executable acceptance checks. The public upload includes one representative task so the complete workflow can be inspected without downloading the full corpus.

Task Role Runtime acceptance boundary
godot-battle-status-bars-gloss Reference task Replace flat HP/MP fills with glossy native textures; preserve layout, bounded ratios, and lifecycle behavior.

The task starts with a compact Godot battle HUD whose HP and MP fills disappear into the background. The requested change sounds visual, but the acceptance boundary is engineering-focused:

The directory contains the starter project, instruction, rubric, native probe, Docker verifier, redacted provenance record, and reference solution.

Task Format

Every contributed task follows the same portable directory shape:

task-name/
├── task.toml
├── instruction.md
├── environment/
│   ├── Dockerfile
│   └── project files
├── tests/
│   ├── test.sh
│   └── runtime probes and fixtures
└── solution/
    └── reference implementation

File Responsibilities

Tasks must not depend on a contributor's local absolute path, private package registry, undisclosed asset, network download at evaluation time, or interactive GUI step.

How To Run The Example

Requires Godot 4.6.1 for local inspection and Docker for the hermetic verifier.

cd release/tasks/godot-battle-status-bars-gloss
godot --headless --path . --editor --quit

The task README contains the reference-workspace command and the Docker invocation. The verifier observes the running Godot project and writes a machine-readable result with per-check diagnostics.

Evaluation Model

GameForgeBench separates four layers of evidence:

  1. Artifact: required files, schemas, bindings, and protected-workspace integrity.
  2. Engine: import, parsing, scene loading, and native startup.
  3. Behavior: ordered events, state transitions, boundaries, persistence, and lifecycle behavior.
  4. Diagnostics: invariant-level results explaining the outcome.

This separation distinguishes a plausible file edit from a project that actually behaves correctly when the engine runs it.

Task Quality Bar

A strong task has a narrow user-visible objective, a believable engineering problem, enough surface area for inspection and repair, deterministic evidence, and a meaningful regression boundary. It should reward understanding the project rather than string matching a known patch.

Reviewers reject tasks that can pass by deleting the gameplay loop, replacing the project with a stub, weakening the verifier, hard-coding the probe sequence, or relying on screenshots alone.

How To Contribute

  1. Fork the repository and create a branch for one task.
  2. Add a directory under release/tasks/<task-name>/ using the format above.
  3. Make tests/test.sh executable and include a reference solution.
  4. Run a clean Docker build, the verifier, and the reference solution from a clean checkout.
  5. Open a pull request describing the runtime behavior and regression boundary.

Public Scope

The public repository is the inspectable trust layer: task contracts, starter projects, verifier design, reference examples, and reproducible reports. Private evaluation variants, collected agent trajectories, and service credentials are not published as benchmark answers.

Contributors

GamePhanesStudio maintains the benchmark, task format, runtime adapters, and evaluation tooling. Contributions are welcome through issues, discussions, and pull requests.

License

MIT.

Citation

@software{gameforgebench2026,
  author  = {GamePhanesStudio},
  title   = {GameForgeBench},
  year    = {2026},
  url     = {https://github.com/GamePhanesStudio/GamePhanes},
  license = {MIT}
}

GamePhanes (GamePhanesStudio/GamePhanes)

godot
Searching Database…

462 ·GitHub
GamePhanes
What is this?GameForgeBench is a benchmark that evaluates coding agents on building, debugging, and repairing real game projects (Godot and other engines) through terminal workflows, judged by actual runtime behavior rather than just file diffs.
Is this a tool I run, or a benchmark suite of tasks?It's a Harbor-compatible benchmark suite (similar to Terminal-Bench) made of self-contained task directories, each with a starter project, instructions, and an executable verifier; it targets coding agents, not player bots.
How do I run a task?Open the included example task, then run it locally against Godot 4.6.1 or through the Docker-based hermetic verifier, following the exact commands given in the task's own README.
What do I need installed to inspect a task?You need Godot 4.6.1 to open and run the starter project locally, and Docker to execute the official hermetic verifier that produces the machine-readable result.
Does the public repo include the full 81-task corpus?No, only one representative task is public so the workflow can be inspected; the remaining tasks, collected agent trajectories, and service credentials stay private.
Which game engines do the benchmark tasks cover?The release corpus spans Godot, Unity, Roblox, Minecraft, Unreal, Web, and generic engine projects, covering gameplay systems, UI, plugins, persistence, and build workflows.
How does the verifier decide if an agent's fix is correct?A Docker-based verifier observes the running project across four evidence layers: artifact integrity, engine import/startup, behavior/state transitions, and diagnostics, rather than just diffing files.
What stops a task from being gamed with a trivial patch?Reviewers reject tasks that can be passed by deleting the gameplay loop, swapping in a stub, weakening the verifier, hard-coding the probe sequence, or relying only on screenshots.
What are similar projects?Tracely-ai, Recuris, ASI-Bench, RSIGame, EvoTrace, iFixAi, eval, godogen
Which topics is it in?agent-evaluation, godot, unity, llm-evaluation, harness, game-development, evaluation, benchmark, coding-agent, ai-agents, llm, nodejs
Creator
GamePhanesStudio
Size
20.4 MB
Stars
462
Updated
CommitSep 12
Age
1mo
Commits · 90 Days