ASI-Bench: At the Dawn of Artificial Superintelligence
Evaluating General Intelligence, Innovation, and Autonomous Execution in Scientific Research
Help build ASI-Bench: we are actively seeking high-quality scientific task contributions.
Core Authors
Junwei Zhou5,†, Zhen Sun1,†, Binyu Li1, Jiangyu Zhou1, Yuexi Pan1, Hengyu Wang1, Honghe Ren1, Xiaohan Jia1, Xueyang Zhou1, Xiaoyu Cao1, Yongchao Chen1,*
† Equal contribution · * Corresponding author
Contributors
Yuanning Feng1, Junhao Wu1, Cheng Zhang13, Sijia Chen10, Haoyu Xue1, Chengsong You1, Huan Wang1, Koutian Wu13, Peigan Gao9, Jiakun Wu1, Wenzhe Li1, Ergan Shang4, Qingyuan Zheng1, Jingjing Zhou1, Ruixuan Jia1, Yan Xu2, Hongrui Zhang7, Xiao-Han Ma9, Zhengxiang Cheng1, Yuexing Hao2, Liting Mai6, Xianglin Ji2, Wenjun Zhang8, Zhuofan Chen1, Yixiao Huang1, Chi Wang12, Wenyue Hua11, Yilun Hao2, Yuantao Zhai1, Ziyan Zhao1, Jingyan Xie3
1 Tsinghua University ·
2 Massachusetts Institute of Technology ·
3 Harvard University ·
4 Carnegie Mellon University ·
5 University of Michigan ·
6 University of Illinois Urbana–Champaign ·
7 Boston University ·
8 University of Queensland ·
9 University of Science and Technology of China ·
10 Flatiron Institute ·
11 Microsoft Research ·
12 AG2 AI ·
13 Independent Researcher
Paper
ASI-Bench: At the Dawn of Artificial Superintelligence
arXiv:2608.17271
Figure 1: Overview of ASI-Bench. Left: B3 performance across agents. Right: scores from B1 to B4, where B1 provides full methods, B2 only the method name, B3 only the research goal and data, and B4 further adds distractors.
Overview
ASI-Bench is the first benchmark to jointly evaluate general intelligence,
innovation, and autonomous execution, and the first to progressively withdraw
human methodological guidance within the same research project. It tests how
far an AI agent can independently select methods, conduct end-to-end research,
and produce verifiable scientific results instead of merely following a
human-specified procedure.
| Benchmark scope |
Description |
| Tasks |
60 project-level tasks |
| Scientific coverage |
11 domains |
| Information levels |
Four matched B1–B4 conditions with progressively less methodological guidance |
| Evaluation |
Expert cross-review, AI-assisted auditing, sandbox execution, and scorer validation |
| Evaluated systems |
18 agent–model configurations |
The scientific objective, input data, required artifacts, and scoring criteria
remain fixed across four matched guidance conditions:
| Level |
Information provided |
What the agent must do |
| B1 |
Scientific background, method, equations, and full procedure |
Implement and execute the prescribed approach |
| B2 |
Intended method and relevant constraints, without the full procedure |
Turn the method into a working research workflow |
| B3 |
Objective, data, constraints, and required outputs only |
Select the method and construct and validate the workflow independently |
| B4 |
B3 plus factually correct but non-essential information |
Conduct the same autonomous research while resisting distraction |
These are long-horizon investigations rather than isolated questions. Across
the 60 tasks, complete research trajectories involve more than 2,600 interaction
turns and 2,400 execution steps, spanning over 35 hours of agent execution.
The benchmark was distilled from more than 1,300 candidate research ideas
through five review rounds, over 1,100 review assignments, more than 2,000 task
revisions, and over 1,500 sandbox runs. Its construction and validation involved
more than 31,000 human-hours.
Resources
Quick start
1. Install ASI-Bench
pip install asibench
asibench --help
Requirements: Python 3.11 or later. The default installation is lightweight
and includes everything needed to download public benchmark instances. For
local agent execution or task authoring, install the optional full stack:
asibench is the canonical CLI. The legacy ai4sci-bench command remains only
as a backwards-compatibility alias.
pip install 'asibench[full]'
2. Download a benchmark set
Each fixed-seed dataset contains its own prompts and input data. seed31415
also publishes references for local scoring; seed42 keeps references private.
Choose one seed for each run:
mkdir asi-bench-run
cd asi-bench-run
asibench task pull \
--repo seed31415 \
--output-dir hf_instances_seed31415/
3. Run an agent
asibench run \
--instances-dir hf_instances_seed31415/ \
--agent-cmd 'python my_agent.py --workspace {workspace}' \
--sandbox linux_ns \
--output-dir out_seed31415/
The command above evaluates all four prompt levels (b1,b2,b3,b4). Use
--prompt-levels only when intentionally running a subset.
4. Score locally or submit
seed31415 can be scored locally with its public references and the scoring
contracts in this GitHub checkout:
asibench score \
--repo seed31415 \
--results-dir out_seed31415/ \
--instances-dir hf_instances_seed31415/ \
--tasks-dir /path/to/ASI-Bench/tasks/
The command writes a separate local_score_seed31415.json and never overwrites
the produce-only result. Local scores are reproducible but non-official.
BenchFlow integration
BenchFlow owns scheduling and agent execution, while
asibench benchflow-score scores one already-materialized seed31415 attempt.
The score is local and non-official; seed42 is not accepted by this interface.
See the BenchFlow integration guide for the
end-to-end run, schema-v2 manifest, scoring command, status fields, retries,
and Judge API configuration.
Some seed31415 scoring contracts use an LLM or VLM Judge. Judge credentials and
transport overrides are runtime operator settings: keep the secret in an
ignored .env file or process environment, never in task_eval.yaml or a
command-line argument. score, run-score, and benchflow-score expose the
same options:
| Option |
Environment equivalent |
Meaning |
--judge-api-base |
ASIBENCH_JUDGE_API_BASE |
Optional custom Judge endpoint |
--judge-api-key-env |
ASIBENCH_JUDGE_API_KEY_ENV |
Name of the variable containing the key, not the key itself |
--judge-api-protocol |
ASIBENCH_JUDGE_API_PROTOCOL |
native or openai for an OpenAI-compatible endpoint |
For native Google Gemini, put GEMINI_API_KEY=... in .env. Existing
google/gemini-* task models then use the Gemini API directly. To select and
validate that credential explicitly, add only its variable name:
asibench score \
--repo seed31415 \
--results-dir out_seed31415/ \
--instances-dir hf_instances_seed31415/ \
--tasks-dir /path/to/ASI-Bench/tasks/ \
--judge-api-key-env GEMINI_API_KEY
For TokenRouter or another OpenAI-compatible gateway, put the secret in .env
as TOKENROUTER_API_KEY=..., then provide all three transport settings:
asibench score \
--repo seed31415 \
--results-dir out_seed31415/ \
--instances-dir hf_instances_seed31415/ \
--tasks-dir /path/to/ASI-Bench/tasks/ \
--judge-api-base https://api.tokenrouter.com/v1 \
--judge-api-key-env TOKENROUTER_API_KEY \
--judge-api-protocol openai
The OpenAI-compatible route automatically sends a task model such as
google/gemini-3.5-flash as openai/google/gemini-3.5-flash; the public task
configuration does not need to be copied or edited. Alternatively, place the
three ASIBENCH_JUDGE_* settings shown in .env.example in .env and run any
of the three scoring commands without repeating the flags. Public score details
record only the endpoint, protocol, and key-variable name—not the secret.
During run-score, a dedicated nonstandard Judge variable such as
TOKENROUTER_API_KEY and all ASIBENCH_JUDGE_* selectors are removed from the
agent-run subprocess; they remain available to the scoring subprocess. Standard
provider variables such as GEMINI_API_KEY remain shared for backward
compatibility. Use a dedicated Judge variable when the evaluated agent must not
receive the scoring credential. ASI-Bench also prevents its run child from
reloading the removed key from the project .env, then restores normal dotenv
behavior before starting the evaluated agent.
For seed42, repeat the pull and run with separate directories using the
required Docker OS sandbox, then submit for private-reference scoring. Because
generic --agent-cmd does not support os, use an OS-compatible built-in
adapter:
asibench run \
--instances-dir hf_instances_seed42/ \
--agent codex_cli \
--agent-config '{"model": "gpt-5.6-sol"}' \
--sandbox os \
--output-dir out_seed42/
asibench login
asibench submit --results-dir out_seed42/ \
--benchmark-repo Apexintelligence-AI/ASI-Bench-seed42
submit creates an authenticated draft on the ASI-Bench website. Review the
completeness summary and confirm the draft to enter the official scoring queue.
Only seed42 result directories produced with --sandbox os are accepted: the
CLI checks every instance ID and each result's verified Docker provenance before
building a bundle or authenticating. It rejects non-OS runs, seed31415, unknown,
or mixed-seed results. If --benchmark-repo is supplied, it must identify the
official seed42 dataset.
asibench score --repo seed42 is rejected before reading local inputs because
seed42 GT is not public. Local benchmark runs never calculate official scores.
Run configuration
| Setting |
Behavior |
| Dataset seed |
seed31415 supports public local scoring; seed42 is private-reference and Portal-scored |
| Downloaded instances |
--instances-dir is read-only; run-specific framework metadata is written under the output directory |
| Prompt levels |
All four levels run by default; select a subset with --prompt-levels |
| Timeout |
--timeout defaults to 10,800 seconds and applies uniformly to every task |
| Produce-only reports |
Unscored placeholders are never displayed as 0.0; all-unscored per-task score tables are omitted |
| Submission |
seed42 requires verified --sandbox os results; submit uploads a draft by default and --no-upload creates a local bundle only |
| Local scoring |
score --repo seed31415 writes a separate non-official JSON report; seed42 is rejected |
| Judge API |
score, run-score, and benchflow-score accept credential-safe runtime overrides for native providers and OpenAI-compatible gateways |
| Custom agents |
--agent-cmd supports the none and linux_ns sandboxes for local runs; official seed42 submission requires an os-compatible built-in adapter |
| Built-in agents |
Use --agent with --agent-config; compatible adapters can use Docker-based os isolation |
Platform note: The examples use Linux Bash, and the linux_ns sandbox
requires Linux. Windows users should run it through WSL2. PowerShell and
Command Prompt use different line-continuation and JSON-escaping syntax.
See Getting Started for agent configuration,
sandbox selection, and platform-specific commands.
If a run completes but scores below expectations, follow the
result troubleshooting guide. It
explains why direct_llm is a single-turn, no-tool baseline, when to consider
multi-turn CLI agents, and how to diagnose execution, evaluation, artifact,
environment, timeout, and score-variance issues.
Package and CLI compatibility
asibench is the only published Python distribution and the canonical CLI.- Existing Python integrations may continue to import
ai4sci_bench.
ai4sci-bench remains available as a legacy CLI alias.- Benchmark instances are downloaded separately with
asibench task pull.
- Runtime and declared-output metadata come from
tasks/ or --tasks-dir.
- Retired internal orchestration commands are not part of the public CLI. Use
run, report, batch-report, review, and the Portal workflows.
Built-in CLI agents — installation
Install the CLI agent you want to evaluate, then select its ASI-Bench adapter
with --agent.
- New model, existing harness. When a supported harness such as Codex CLI,
Claude Code, or Kimi Code adds a model, select the new model through
--agent-config. If the model is served through a third-party endpoint and
the adapter supports that route, also set api_base, api_key, and
api_protocol (openai or anthropic). No ASI-Bench code change is normally
needed while the harness accepts the model and the endpoint remains protocol
compatible.
- New agent harness. A new CLI or runtime can be used immediately through
--agent-cmd when it can read the prepared workspace and write result files;
this generic mode supports none and linux_ns. First-class built-in
integration—its own --agent name, harness-specific authentication and
configuration, proxy wiring, and Docker-based os support—requires ASI-Bench
maintainers to implement and test a new adapter.
| Agent |
Install command |
Notes |
| Claude Code |
npm install -g @anthropic-ai/claude-code |
Requires Anthropic API key or local login (claude /login) |
| Codex CLI |
npm install -g @openai/codex |
Requires OpenAI API key |
| Kimi Code |
npm install -g @moonshot-ai/kimi-code |
Use the npm package. The PyPI package (kimi-cli) has different CLI arguments and is not compatible with this adapter. Requires Moonshot API key or local login (kimi /login) |
| CodeWhale |
See CodeWhale docs |
Requires DeepSeek API key by default |
| AntiGravity |
curl -fsSL https://antigravity.google/cli/install.sh | bash |
Experimental |
| pi |
npm install -g @earendil-works/[email protected] |
Native JSONL trajectories + cost extraction. Requires local login (pi + /login) or provider API key; model uses provider/model syntax |
| opencode |
npm install -g [email protected] |
Native JSONL trajectories + cost extraction. Requires opencode auth login or provider API key; model uses provider/model syntax |
Example with Kimi Code:
asibench run --agent kimi_code_cli \
--agent-config '{"model": "kimi-k2.7"}' \
--output-dir out/
Example with pi (local login) and opencode (explicit endpoint):
# pi with harness-local login state
asibench run --agent pi_cli \
--agent-config '{"model": "anthropic/claude-opus-4-6"}' \
--output-dir out/
# opencode against a third-party OpenAI-compatible endpoint
asibench run --agent opencode_cli \
--agent-config '{"model": "deepseek-chat", "api_base": "https://api.deepseek.com/v1", "api_key": "sk-...", "api_protocol": "openai"}' \
--output-dir out/
Framework-managed model proxies use text-only input by default. For endpoints
that accept images, set "supports_image_input": true in --agent-config.
Without this flag, image attachments are replaced with a short notice before
the next model request. Direct endpoints outside the framework proxy remain the
responsibility of the connected agent.
Harness session state (transcripts, history, auto-memory) never carries over
between instances or repeated executions of the same instance: the OS sandbox
gives each run a one-shot container with a fresh HOME, and host-side runs
build execution-scoped, isolated per-run homes for
claude_code_cli / codex_cli and a per-instance KIMI_CODE_HOME for
kimi_code_cli (an explicitly configured kimi_home stays shared by choice).
See TEST.md (“Per-run harness home isolation”) for details.
Contribute a task
We welcome project-level scientific tasks from the research community. Authors
can draft a task locally or in the Portal, validate it, and submit a frozen
revision for review.
Quick start for task authors
# 1. Create a task scaffold
asib
ASI-Bench is a benchmark for evaluating AI agents on long-horizon scientific research: it jointly tests general intelligence, innovation, and autonomous execution across 60 project-level tasks in 11 domains.