OpenWAM README

The project's README from github.com/OpenWAM-Official/OpenWAM. As one markdown file: README.md.

OpenWAM

An Open, Modular Exploration Towards Systematic World–Action Model Pretraining

Project Page Paper on arXiv Model & Data on Hugging Face

OpenWAM teaser

What is OpenWAM

OpenWAM is an open research stack for systematically developing World-Action Models (WAMs). It turns tightly coupled design choices into modular components and controlled experiments. It consists of:

OpenWAM infrastructure overview

News

Installation

Create an environment with conda:

# Requires Python >= 3.10
conda create -n openwam python=3.10
conda activate openwam

or with venv:

# Requires Python >= 3.10 (check with `python3 --version`)
python3 -m venv .venv
source .venv/bin/activate

We recommend using PyTorch 2.7.1 with CUDA 12.8 (others may also work):

pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu128

Then install OpenWAM:

pip install -e .

deepspeed ships as a source distribution and is compiled during pip install, so a C compiler (gcc) must be on the PATH.

Cosmos-Predict2.5 Extras (Optional) — needed only for experiments with the cosmos_predict25_2b video backbone

With your environment activated:

git submodule update --init third_party/cosmos-predict2.5
bash scripts/install_cosmos_predict25.sh

The script installs the upstream cosmos packages into the active environment and compiles transformer-engine (CUDA toolkit with nvcc required), then automatically restores the package versions OpenWAM pins.

Assets Preparation

The downloaders are interactive by default; every menu step also has a flag (--name, --source, --root, --yes, see --help) so they can run unattended, and the default storage location is resolved relative to the repository regardless of the working directory. Component downloaders store assets under assets/ and update the matching YAML path; the released-checkpoint downloader keeps each checkpoint's self-contained config unchanged.

1. Video Backbone

python scripts/download_assets/download_video_backbone.py

Supported video backbones

Wan2.2-TI2V-5B ✅ Wan2.1-VACE-1.3B ✅ Wan2.1-I2V-14B-480P ✅
Cosmos-Predict2.5-2B ✅ Cosmos3-Edge ✅

Weights are saved under assets/video_backbone_ckpt/ and the selected configs/model/video_backbone/ file is updated with the downloaded path.

2. Benchmark Data

python scripts/download_assets/download_benchmark_data.py

Supported benchmarks

RoboTwin2.0 ✅ RoboDojo ✅ RoboDojo-Real ✅
LIBERO ✅ VLABench ✅ EBench ✅
RoboCasa365 ✅ RoboCasa_GR1 ✅

Data is saved under assets/benchmark_data/<benchmark>/. Normalization statistics are prepared when needed, and the selected dataloader configuration is updated.

3. VLM Backbone (Optional)

Required only by tri_system:

python scripts/download_assets/download_vlm_backbone.py

Supported VLM backbones

Qwen3-VL-2B-Instruct ✅

Weights are saved under assets/vlm_backbone_ckpt/, and the selected configuration is updated.

4. Visual Encoders (Optional)

Required only for video backbones that use an external encoder:

python scripts/download_assets/download_visual_encoder.py

Supported visual encoders

DINOv3 ViT-B/16 ✅ V-JEPA 2.1 ViT-G/16 ✅
Wan2.2 VAE ✅ FLUX.2 VAE ✅

Weights are saved under assets/visual_encoder_ckpt/, and the selected encoder configuration is updated.

5. Released OpenWAM Checkpoints

Use this downloader to obtain OpenWAM-Alpha releases or OpenWAM-Study checkpoints from the OpenWAM collection:

python scripts/download_assets/download_openwam_checkpoints.py

Checkpoints are saved under assets/openwam_ckpt/openwam_alpha/ or assets/openwam_ckpt/openwam_study/<type>/. Each checkpoint directory contains its own config and can be deployed directly with:

bash scripts/deploy.sh <ckpt_dir_path>

For fine-tuning, set training.finetune_ckpt_path in configs/train.yaml to the downloaded checkpoint directory. Benchmark data is still required.

Quick Start

Quick Start provides a minimal end-to-end example: prepare the assets, train or fine-tune a policy, deploy its checkpoint, and run a first inference check.

The example uses the DualSystem JointSelfAttention architecture, the Wan2.2-TI2V-5B video backbone, and the Mutual attention mask.

Component Selection Configuration
Architecture dual_system / joint_self_attn configs/model/dual_system.yaml
Video backbone wan22_ti2v_5b configs/model/video_backbone/wan22_ti2v_5b.yaml
Attention mask mutual model.architecture.attention_mask_mode
Dataset libero configs/dataloader/libero.yaml

Resource recommendation: We recommend 8 GPUs with 80 GB VRAM each for training. This configuration supports normal training for all architectures using Wan2.2-5B or smaller video backbones. More GPUs are better when available and can further improve training throughput.

From Scratch Training

  1. Download LIBERO and let the downloader update its dataloader configuration:

    python scripts/download_assets/download_benchmark_data.py

    Select LIBERO in the interactive menu.

  2. Download Wan2.2-TI2V-5B:

    python scripts/download_assets/download_video_backbone.py

    Select Wan2.2-TI2V-5B and the desired model source.

  3. Start a debug run with the complete model selection:

    bash scripts/train.sh \
  dataloader=libero \
  model=dual_system \
  model/video_backbone=wan22_ti2v_5b \
  model.architecture.variant=joint_self_attn \
  model.architecture.attention_mask_mode=mutual \
  training.debug=true

    training.debug=true runs 20 steps, saves at steps 10 and 20, and uses a constant learning rate. Check the run output, then set training.debug=false for normal training. Training defaults and CLI overrides are defined in configs/train.yaml. Debug outputs use training.output_path, whose default is outputs/openwam_checkpoints.

  4. Deploy the debug checkpoint and inspect one input-output cycle:

    bash scripts/deploy.sh <debug_ckpt_dir_path>

    Keep the server running, then open another terminal and run the two inference helpers. Deployment enables compile by default, so the first inference may take longer while compilation warms up; later requests are typically faster:

    python scripts/inference_test/inference_single_test.py \
  --server ws://127.0.0.1:8848 --test --state-dim 10

python scripts/inference_test/inference_continuous_test.py \
  --server ws://127.0.0.1:8848 --test --state-dim 10

    The single-request helper checks ping, one prediction, and reset. The continuous helper sends repeated predictions over one connection and reports the returned action dimension and latency. Stop the deployment process after the checks.

OpenWAM-α Fine-Tuning

  1. Download LIBERO as shown above.

  2. Download the OpenWAM-Alpha foundation checkpoint:

    python scripts/download_assets/download_openwam_checkpoints.py

    Select OpenWAM_Alpha and OpenWAM-Alpha-Pretrain-Foundation-Model. Keep the resulting directory as .

  3. Start fine-tuning from that directory:

    bash scripts/train.sh \
  dataloader=libero \
  training.finetune_ckpt_path=<foundation_ckpt_dir_path>

    The default model configuration already matches the required setup above. You can set the same field in configs/train.yaml instead of passing it on the command line. num_frames=33 and video_stride=4 in configs/dataloader/libero.yaml produce the 32-step action horizon expected by the sampler; no separate action_chunk override is needed.

  4. Deploy the resulting checkpoint directory:

    bash scripts/deploy.sh <ckpt_dir_path>

    Install and run the LIBERO client according to the LIBERO evaluation guide.

OpenWAM Usage Guidance

OpenWAM is configured through composable Hydra YAML files. Select an architecture, backbone, dataloader, and runtime behavior by changing configuration values or overriding them on the command line. The task guides below are the maintained entry points for using and extending the repository:

Guide Use it when you want to…
Training and deployment choose a model/dataloader, prepare assets, train, fine-tune, resume, or deploy a policy
Architecture extension extend a video, visual, VLM, action backbone, or WAM architecture
Benchmark integration extend a dataloader and connect a benchmark client to the WebSocket protocol
OpenWAM-α fine-tuning fine-tune the released foundation checkpoint to execute downstream task

💡 Agent tip: Pick the guide that matches your task and feed it directly to your agent — less explaining, more building.

Installation and Assets Preparation above cover environment setup and model or dataset downloads. Benchmark-specific environment and evaluation details remain in the benchmarks directory.

Star History

Star history of OpenWAM-Official/OpenWAM

License

OpenWAM is released under the Apache License 2.0.

Citation

If you use OpenWAM, please cite:

@article{wang2026openwam,
  title   = {OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining},
  author  = {Yuran Wang and Siqiao Huang and Mingleyang Li and Chenhao Zhang and Jiaqi Liang and Weiyang Jin and Yue Chen and Xuemin Chi and Donghao Zhou and Qize Yu and Yu-Kai Wang and Yuhan Rui and Shenzhe Yao and Zhen Yuan and Zhenhao Shen and Kefei Zhu and Zijie Zhu and Ning Gao and Xiaowei Chi and Guanqi He and Shanghang Zhang and Hao Dong and Lin Shao and Hang Zhao},
  year    = {2026},
  journal = {arXiv preprint arXiv: 2609.07398}
}

Contact

Have a question, want to share results, or just follow development? Join us:

OpenWAM WeChat group QR code

WeChat Group
Scan to join 		OpenWAM Discord QR code

Discord
discord.gg/yRSqhpNXu

OpenWAM (OpenWAM-Official/OpenWAM)

embodied-ai
Searching Database…

1.1K ·GitHub
OpenWAM
What is this?OpenWAM is an open research stack for systematically building World-Action Models (WAMs) for robots, breaking the usual tightly-coupled design choices into modular, swappable components you can compare in controlled experiments.
Is this a ready-made model or a research framework?It is a modular research framework (OpenWAM-Infra) for composing model, backbone, training, inference, deployment and evaluation choices, bundled with controlled-study results (OpenWAM-Study) and a pretrained model (OpenWAM-alpha) built on top of it.
How do I get it running?Create a conda or venv environment, install PyTorch with CUDA, then pip install OpenWAM; deepspeed is compiled from source during this step so a C compiler (gcc) must be on the PATH. Interactive downloader scripts then fetch video backbones, benchmark data and checkpoints into assets/ before you train or deploy.
Do I need a specific PyTorch and CUDA version?Yes, the authors recommend PyTorch 2.7.1 with CUDA 12.8 (other versions may work but aren't guaranteed), plus make and gcc for building deepspeed.
What hardware do I actually need to train with this?The recommended setup is 8 GPUs with 80 GB VRAM each for normal training with Wan2.2-5B-scale or smaller video backbones; larger backbones like the Cosmos-Predict2.5-2B extras need extra CUDA toolkit/nvcc compilation of transformer-engine.
Does this need Linux, or will it run on Mac/Windows too?The install and GPU/CUDA/deepspeed toolchain assumes a Linux environment with NVIDIA GPUs; there's no mention of macOS or Windows support.
What is OpenWAM-alpha?OpenWAM-alpha is the open pretrained World-Action Model released by the project, trained on 518.5M frames (about 6,400 hours) of egocentric human and robot data, and it can be fine-tuned on downstream tasks like LIBERO.
Which video backbones can I plug in?Supported backbones include Wan2.2-TI2V-5B, Wan2.1-VACE-1.3B, Wan2.1-I2V-14B-480P, Cosmos-Predict2.5-2B and Cosmos3-Edge, each downloadable via the included asset scripts.
Which robotics benchmarks does it support?The repo ships downloaders and configs for RoboTwin2.0, RoboDojo, RoboDojo-Real, LIBERO, VLABench, EBench, RoboCasa365 and RoboCasaGR1.
How is a model actually deployed and queried?After training or downloading a checkpoint, you start a deployment server from the checkpoint directory and query it over a WebSocket-based inference protocol, with helper scripts for single and continuous prediction requests.
What are similar projects?Zero-WAM, flex-pi, video-prediction-policy-2, S-Space, EasyWAM, FluxVLA, ZDTaichu5.0-9B, tapnet
Which topics is it in?world-action-model, vla, pretraining, world-models, embodied-ai, arxiv, vision-language-model, attention-mechanism, robotics, dataset, academic-research, artificial-intelligence
Creator
OpenWAM-Official
Size
16.2 MB
Stars
1.1K
Updated
CommitOct 14
Age
1mo
Commits · 90 Days