Code-as-World README

The project's README from github.com/MirroS-Lab/Code-as-World. As one markdown file: README.md.

MirroS logo

Code-as-World

Agentic Discovery of Executable World Representations for Physical Reasoning

A world becomes intelligible to evolving intelligence when it can be represented, executed, and verified.

Paper Project Page Blog Code

Code-as-World overview

Overview

Pixels are evidence of the physical world, not its ontology. A pixel-level observation records how the world appears at a particular moment and from a particular viewpoint, but does not directly specify what exists within it, how it is structured, or what governs its evolution.

Code-as-World introduces code as executable representations for the physical world and an agentic process for discovering them through iterative simulation and verification. In doing so, it turns raw abundant observations into reusable physical data: explicit states, dynamics, and mechanisms that capture not only what was seen, but the underlying world that could have produced it. This provides scalable physical supervision, enabling our models to achieve state-of-the-art performance on quantitative physical reasoning.

News

Get started

The release includes local inference and QuantiPhy evaluation for Code-as-World-VL-4B and Code-as-World-VL-9B, plus a video-driven abstraction example.

Installation

Use Python 3.10 or 3.11 on a CUDA host.

git clone https://github.com/MirroS-Lab/Code-as-World.git
cd Code-as-World
python -m venv .venv
source .venv/bin/activate
pip install -r requirements/inference.txt

hf download MirroS-Lab/Code-as-World-VL-4B --local-dir weights/4b
hf download MirroS-Lab/Code-as-World-VL-9B --local-dir weights/9b

Clone QuantiPhy and download its validation videos:

git clone https://github.com/Paulineli/QuantiPhy.git /path/to/QuantiPhy
hf download PaulineLi/QuantiPhy-validation \
  --repo-type dataset \
  --local-dir /path/to/QuantiPhy-validation

QuantiPhy evaluation

Run the released evaluation directly from the Code-as-World repository:

python -m code_as_world.evaluation 4b \
  --input-csv /path/to/QuantiPhy/quantiphy_validation.csv \
  --video-dir /path/to/QuantiPhy-validation/validation_videos

python -m code_as_world.evaluation 9b \
  --input-csv /path/to/QuantiPhy/quantiphy_validation.csv \
  --video-dir /path/to/QuantiPhy-validation/validation_videos

Each run writes an evaluator-compatible prediction CSV, a single-run metric summary, and the raw generations to outputs/quantiphy/. To evaluate both CSV files with the official QuantiPhy evaluator:

pip install pandas
python /path/to/QuantiPhy/evaluator.py \
  outputs/quantiphy \
  outputs/quantiphy_metrics \
  --gt_file /path/to/QuantiPhy/quantiphy_validation.csv

OpenAI-compatible serving

The checkpoints can also be exposed through the standard vLLM API:

CUDA_VISIBLE_DEVICES=0 vllm serve weights/4b \
  --served-model-name code-as-world-4b \
  --chat-template code_as_world/templates/qwen3_5_no_think.jinja \
  --chat-template-content-format openai \
  --default-chat-template-kwargs '{"enable_thinking":false}' \
  --max-model-len 4608 \
  --gpu-memory-utilization 0.90 \
  --media-io-kwargs '{"video":{"num_frames":16,"fps":-1,"video_backend":"openpangu"}}' \
  --mm-processor-kwargs '{"do_sample_frames":false}' \
  --mm-processor-cache-gb 0 \
  --generation-config vllm

For the 9B checkpoint, replace weights/4b and code-as-world-4b with weights/9b and code-as-world-9b.

Video-driven abstraction example

Install MuJoCo and run the bundled ballistic soccer case:

pip install -r requirements/simulation.txt
python -m code_as_world.simulation

The rendered video and trajectory are written to outputs/simulations/. Use python -m code_as_world.simulation --no-render when only the trajectory is needed.

TODO

Acknowledgements

We sincerely thank the teams behind the following projects for making their work available to the community:

Component Projects
Physical simulation MuJoCo
Visual perception and reconstruction SAM 3, DA3, VGGT-Omega, SAM 3D
Realistic video generation Wan, VACE
Quantitative evaluation QuantiPhy

... and many other excellent open-source projects.

Citation

If you find Code-as-World useful, please cite:

@article{mirros2026codeasworld,
  title   = {Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning},
  author  = {{MirroS Team}},
  journal = {arXiv preprint arXiv:2608.27549},
  year    = {2026}
}

Code-as-World (MirroS-Lab/Code-as-World)

Searching Database…

419 ·GitHub
Code-as-World
What is this?Code-as-World is a research project that represents the physical world as executable code, discovered through an agentic loop of simulation and verification, so models get reusable physical data (states, dynamics, mechanisms) instead of just raw pixels.
Is this a trained model or a full research pipeline?It is a full research release: two pretrained vision-language checkpoints (4B and 9B), inference and evaluation code, and a video-driven abstraction example that turns a clip into a MuJoCo simulation.
How do I get this running?Use Python 3.10 or 3.11 on a CUDA host, then clone the QuantiPhy benchmark repo and download its validation videos before running evaluation or inference scripts from this repository.
What do I need besides Python to run the checkpoints?You need a CUDA GPU host, MuJoCo for the video-driven simulation example, and vLLM if you want to serve the 4B or 9B checkpoint through its OpenAI-compatible API.
Can this run without a GPU?No, the setup instructions require a CUDA host, and there is no mention of a CPU-only path for inference or evaluation.
What are Code-as-World-VL-4B and 9B?They are the two released vision-language checkpoints trained to produce executable code representations of physical scenes for quantitative physical reasoning, served locally or through vLLM.
What is QuantiPhy?QuantiPhy is the official benchmark and evaluator the project uses to score predictions on quantitative physical reasoning, with its own repo and validation videos.
What does the video-driven abstraction example do?It uses MuJoCo to turn a bundled ballistic soccer video into an executable simulation, writing a rendered video and trajectory to outputs/simulations/ (trajectory-only mode is available via --no-render).
Why use code instead of raw pixel observations?The authors argue pixels only record how a scene looks, not its underlying structure, so executable code is used to capture the explicit states, dynamics, and mechanisms that could have produced that observation.
What are similar projects?Zero-WAM, OraRL, CodeVideoModel, ZDTaichu5.0-9B, Embodied-AI-Start, le-wm, EvoVLM, EasyWAM
Which topics is it in?vision-language-models, video-understanding, physics-simulation, world-models, paper, simulation, code-generation, vllm, evaluation, multimodal, academic-research, openai-compatible
Creator
MirroS-Lab
Size
11.3 MB
Stars
419
Updated
CommitAug 31
Age
1mo
Commits · 90 Days