arc-code
Claude Opus 5 scores 30.2%
on the 25 public ARC-AGI-3 games in ARC's model-only evaluation. I ran the
same model through stock Claude Code, with a shell, a filesystem, and one
command for taking actions. It won 24 of 25 games and scored 96.2% at the
same high reasoning effort, for about $540. A second attempt at the one
missed game brings pass@2 to 99.3, with all 25 games won.
The harness has no solver, planner, world model, or grid tooling. The model
writes those while it plays. In one pass it produced 269 programs—about
12,700 lines, or ten times the size of the harness—and discarded all of them
when their games ended.
This repo contains the harness and six complete session records. The full
post-broker record will be released shortly. The score by itself is not the
interesting part: several systems built specifically for ARC score this high
or higher. What is unusual is where the ARC-specific machinery comes from. It
is not built into the harness. Opus builds it during each game—a parser, a
model of the rules, a simulator, and a search procedure—and throws it away
afterward.
ARC-AGI-3 is designed to measure learning
on first contact: an agent has to explore a new environment, infer its goal
and mechanics, build a world model, and revise it from experience. That is
what Opus is doing here through a general coding harness. It builds the
task-specific harness it needs to learn each game. This test-time
meta-learning, rather than the raw score, is the result this repo is meant to
show: the capability is now strong enough to carry nearly the whole benchmark.
Every game runs in its own sandbox with no access to the public internet.
That is enforced rather than left to the model; some of the Codex runs did in
fact try to find published solutions. See sandboxing and
an agent tried to break out.
The harness
The agent is stock Claude Code, running headless with six tools pre-approved:
Bash, Read, Write, Edit, Grep, and Glob. There are no MCP servers, subagents,
custom tools, plugins, or hooks. Nothing is fine-tuned, and the agent never
sees an example of a solved game.
The harness itself is three files:
PROMPT.md gives the method: parse the log with code, keep
findings in files, test a guess with 1–2 actions before committing 10–20.
It does not name ARC or contain facts about any game. An eleven-line brief,
ARC.md, describes the generic interface; the actuator documents
the board and log format in the log itself. The same prompt is used for
Claude and Codex. Its portability across tasks is a design claim, not yet
a demonstrated result.act.py is the only way the agent touches the game.
./act do ACTION1 ACTION6:30,40 --plan "..." plays actions in order, stops
when one scores, and appends every action and resulting board to logs.txt.run.py launches one agent session per game and records its full
event stream. Codex swaps in with one flag; the rest of the harness is the
same.
logs.txt, rather than the context window, is the agent's complete memory.
The model reads it back with grep and Python and keeps longer-lived findings
in files of its own.
Everything supporting those three files lives in rig/: sandboxes,
the broker that holds the game key, the database record, auditing, scoring,
and export. rig/README.md lists the pieces.
Results
|
Claude Code (Opus 5) |
Codex (gpt-5.6-sol) |
| thinking effort |
high (the default) |
xhigh |
| ARC score |
96.2% |
73.7% |
| games won |
24 / 25 |
19 / 25 |
| total actions |
8,789 |
26,971 |
| cost |
~$540 |
~$424 |
| internet access |
none |
none |
ARC scores action efficiency against human baselines, capped at 100 per game.
Claude's billed cost includes two sessions that crashed or stopped early and
were resumed. Codex reports tokens rather than cost, so its number is computed
from published prices and is a floor. Each column is one pass over the same 25
games with the same prompt and actuator.
These are preliminary runs, intended to test Opus 5's meta-learning through a
general coding harness rather than maximize the score. Claude ran at its
default high, and Codex at xhigh; higher-effort runs are still to do.
The Claude pass lost one game, lf52. Retrying that game once gives
pass@2 of 99.3 and 25/25 games won. The first failure is discussed below:
when uncertainty does not lead to action.
Running it
You need Python 3.13+, uv, an
E2B account, a Neon database, and a
model key. Copy .env.example to .env and fill it in.
uv sync
uv run pytest -q # 117 tests, no network
uv run rig/db.py init # create the tables
uv run --with e2b rig/cloud.py build # bake the sandbox image
uv run --with e2b rig/cloud.py start ft09-0d8bbf25 --broker --run smoke
uv run --with e2b rig/cloud.py watch smoke # progress from the database
Add --agent codex to run Codex, or pass $(cat docs/games.txt) to run all
25 games. docs/REPRODUCE.md covers setup, verification,
and the parts of the result that cannot be established from the stored record
alone.
The agent builds the machinery
Opus starts each game with an empty folder. It usually begins by inspecting a
few boards and writing a parser. As it learns the rules it adds notes, a
simulator, and often a search program. The prompt permits shell and Python
loops but does not ask for any of these things. Across the pass, all 25 games
ended up with their own collection of programs.
Those collections differed substantially. Every game got a parser and 23 of
25 got a search, but only nine got a simulator. sc25 needed one 79-line
file; lf52 needed 80 files and 2,288 lines—29 times as much code. A fixed
harness supplies the same machinery to every game in advance. Here the model
decides what to build after it has seen the board.
One game, start to finish
In re86, the player steers shapes around a board to cover colored boxes. The
agent was not told that; the session had to infer it.
Its first file, parse.py, turns each 64×64 board into a list of objects. From
the first few actions it records that arrows move the active shape three cells
at a time, ACTION5 changes which shape is active, and a purple bar at the
bottom appears to count down the remaining moves. It also records a tentative
goal: four boxes of one color form a plus, and the cross belongs at their
center.
Once those rules seem stable, the agent writes sim.py to predict the result
of an action without spending one in the real game. Later levels add a
rectangular ring. The first version, rect.py, assumes the ring slides like
the other shapes. The board disproves that: against an obstacle the ring
squashes, holding one edge fixed and expanding sideways. rect2.py replaces
the sliding rule; rect3.py corrects the order in which the edges grow.
When the simulator matches the observed boards, route.py and game.py
search it for short winning sequences, which the agent then plays for real.
Eight groups of model files went through at least three versions during the
pass.
Different games, different machinery
tu93 (session) is a maze encoded as a 21×21
grid of blocks on the 64×64 screen. The agent also infers that the magenta
bar along the bottom tracks a 50-move budget. A later level adds a pursuer,
so it writes chase.py to search for paths that remain safe against its
movement.m0r0 (session) has two characters that move
as mirror images. Four solver versions lead to a search state containing
both positions, the target dot, and the current mode.lf52 (session) scrolls through a world wider
than the screen. Six files measure the camera shift, stitch views into a
map, locate walls, track the camera, and correct clicks for scrolling.ka59 becomes a Sokoban search: steer a box around the board and push
objects into pens. The agent derives the mechanics without being given the
name.
The agents' notes.md files distinguish observations from guesses with labels
such as "hypothesis, strong," "confirmed so far," "presumed," and "untested."
Six complete workspaces are checked into data/runs/, including
the notes, programs, action log, and full event stream. The full post-broker
record will be released shortly on the
releases page.
Other harnesses
Tycho (100), Retrodict (99.9), and baseline1 (99) report higher scores on
ARC's community leaderboard,
which is separate from the official model-only evaluation. Those systems
include ARC-specific machinery such as grid tools, planners, world-model
contracts, and replay verification. They are stronger results; I am not
claiming a rank over them.
Two general-purpose harnesses make useful comparisons:
- PRO-LONG
(paper) also runs Claude Code and is the
source of the log-as-memory idea used here. It reports 97.4% best@2 with
Fable 5. Its agent writes a batch of actions and exits; a runner plays the
batch, so the interaction loop remains in the harness.
- Prime Agent is built
on
pi, with recursive context,
subagents as function calls, and harness state the agent may rewrite during
a task. It reports 95.5% RHAE Best@1 with Opus 5.
The cleanest comparison is within Opus 5: 30.2% model-only, 96.2 here, and
95.5 through Prime Agent. ARC-specific machinery still buys the last few
points. Most of the gap, though, appears when the model is allowed to build
its own machinery during the run.
When uncertainty does not lead to action
The clearest recurring failure was a gap between uncertainty and action. Opus
would record an unresolved question or an incomplete part of its model, then
keep searching within the model—or stop—instead of using its remaining actions
to investigate the gap.
lf52 is a peg-solitaire game with ten levels. Level 6 adds a red piece and a
world wider than the screen. Two agents mapped it, built simulators, searched
them, and concluded that the level was impossible. Both stopped with more than
1,700 of their 2,500 actions unused.
One had marked the red piece cannot board a shuttle as verified after testing
one way of moving it aboard. A winning run tried a different transition. The
red is a mobile wall: a red-green pair can leapfrog across the board without
consuming the green, and a shuttle can carry the red to where it is needed.
Level 6 then takes about 130 actions, and the game can be completed 10/10.
Across four attempts, two solved it.
Across 191 Claude sessions, 14 did not win. Six were voluntary early exits:
the agent concluded that the current level was impossible and stopped with
1,391–2,218 actions unused. Three of the six had spent less on the level that
beat them than winning sessions spent on average. One declared level 8 of
bp35 to need an undiscovered mechanic after 52 actions—inside the 20–64 it
had spent on each level it solved—and stopped with 2,162 left.
Five of the six had already written down an unresolved question but did not run
the corresponding experiment. The uncertainty was present in their notes. One
conclusion was headed IMPOSSIBILITY PROOF (within the verified model) and
ended "unless a mechanic outside this model exists." One lf52 agent had also
written this rule after level 4:
Don't conclude "unsolvable/isolated" until the whole world has been swept.
Two levels later it concluded unsolvable, with 1,726 actions unspent and that
line still sitting in the notes file it re-reads whenever its memory is
trimmed. It had recorded the uncertainty but did not act on it.
docs/failure-modes.md works through all fourteen
non-wins, including the failures that do not fit this pattern.
Sandboxing
Game source and other harnesses' traces are public, so a run with ordinary
internet access could find an answer key. Each game therefore runs inside an
E2B sandbox with default-deny egress.
- Three hosts are allowed—the model API, broker, and Postgres—plus DNS. The
rules match SNI and the Host header.
tests/verify_fence.py
checks the fence live from inside a sandbox.
- The ARC game key never enters the agent's machine.
broker.py
holds it, owns the game sessions, and writes the canonical log. The agent
receives a forwarding client with no credentials.
- A provider can browse on the caller's behalf, which the network fence cannot
prevent. Claude's API reports server-side web use; across 191 Claude
sessions it reported zero searches and zero fetches.
audit.py scans each complete event stream for network
clients, DNS probes, package installation, encoded payloads, credential
reads, second agents, and attempts to use the model API as a browser.
An agent tried to break out
In the latest 25-game Codex pass, 7 sessions tried to reach the internet for
help. The targets included game source on Hugging Face and GitHub, published
traces for the game being played, trajectory datasets, another harness's site,
and ARC scorecard pages. The agents tried HTTPS, HTTP, a mirror, a proxy,
git clone, wget, urllib, and curl. Across all Codex passes, 11 of 60
sessions tried it.
The fence blocked every attempt, usually during the TLS handshake:
urlopen error TLS/SSL connection has been closed (EOF)
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL
fatal: unable to access '…': gnutls_handshake() failed
None of the 191 Claude sessions did this; the provider's web-request counter
also remained at zero.
The contrast between the models is notable: GPT-5.6-sol tried to look outside
the game for an answer, while Opus 5 did not. Because the attempts took several
forms, the result depends on prevention rather than detection after the fact.
The record
The database holds all 251 sessions: every command, every result, each action
log, the agent's notes and programs, and per-level action counts. Under the
current audit rules, 236 are clean. The other 15 consist of the 11 Codex
answer-seeking attempts above, two agents probing their own broker while
debugging a failed connection, and two early Claude runs that used the game
key before the broker existed.
The full post-broker record will be released shortly on the
releases page. It will include
the headline Claude pass, the Codex comparison, all four lf52 attempts, the
Codex breakout attempts, and earlier passes. One earlier prompt still included
interface facts such as action names, board size, and the color map. It scored
97.3, about one point higher and within the observed variance, while using a
third more actions. Each session contains the exact prompt it ran.
The pre-broker workspaces will not be included because they contain live ARC
session cookies—the exposure the broker was written to remove.
ARC only discloses scores on closed scorecards and removes idle ones, so
score.py contains a reconstruction of the metric. It
reproduces 106 of the 111 scores ARC disclosed; the five misses are documented
in the code rather than fitted away. export.py builds and
verifies the release archive, and docs/REPRODUCE.md
shows how to load it into a database once released, rerun the audit and
scorer, and check the known uncertainties.
Credit
PRO-LONG is where I got the idea to
use a log written by code and read back with grep and Python as the agent's
memory.
The human baselines in baselines.json are ARC's own
numbers from the closed scorecards for these runs.
License
MIT
arc-code 是一个研究测试框架,让 Claude Code(或 Codex)通过 shell 和文件系统去玩 ARC-AGI-3 基准游戏,为每个游戏自行编写解析器、模拟器和搜索代码,而不是依赖内置的 ARC 专用工具。