OpenWAM README

The project's README from github.com/OpenWAM-Official/OpenWAM. As one markdown file: README.md.

OpenWAM

An Open, Modular Exploration Towards Systematic World–Action Model Pretraining

Project Page Paper on arXiv Model & Data on Hugging Face

OpenWAM teaser

What is OpenWAM

OpenWAM is an open research stack for systematically developing World-Action Models (WAMs). It turns tightly coupled design choices into modular components and controlled experiments. It consists of:

OpenWAM infrastructure overview

News

Installation

Create an environment with conda:

# Requires Python >= 3.10
conda create -n openwam python=3.10
conda activate openwam

or with venv:

# Requires Python >= 3.10 (check with `python3 --version`)
python3 -m venv .venv
source .venv/bin/activate

We recommend using PyTorch 2.7.1 with CUDA 12.8 (others may also work):

pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu128

Then install OpenWAM:

pip install -e .

deepspeed ships as a source distribution and is compiled during pip install, so a C compiler (gcc) must be on the PATH.

Cosmos-Predict2.5 Extras (Optional) — needed only for experiments with the cosmos_predict25_2b video backbone

With your environment activated:

git submodule update --init third_party/cosmos-predict2.5
bash scripts/install_cosmos_predict25.sh

The script installs the upstream cosmos packages into the active environment and compiles transformer-engine (CUDA toolkit with nvcc required), then automatically restores the package versions OpenWAM pins.

Assets Preparation

The downloaders are interactive by default; every menu step also has a flag (--name, --source, --root, --yes, see --help) so they can run unattended, and the default storage location is resolved relative to the repository regardless of the working directory. Component downloaders store assets under assets/ and update the matching YAML path; the released-checkpoint downloader keeps each checkpoint's self-contained config unchanged.

1. Video Backbone

python scripts/download_assets/download_video_backbone.py

Supported video backbones

Wan2.2-TI2V-5B ✅ Wan2.1-VACE-1.3B ✅ Wan2.1-I2V-14B-480P ✅
Cosmos-Predict2.5-2B ✅ Cosmos3-Edge ✅

Weights are saved under assets/video_backbone_ckpt/ and the selected configs/model/video_backbone/ file is updated with the downloaded path.

2. Benchmark Data

python scripts/download_assets/download_benchmark_data.py

Supported benchmarks

RoboTwin2.0 ✅ RoboDojo ✅ RoboDojo-Real ✅
LIBERO ✅ VLABench ✅ EBench ✅
RoboCasa365 ✅ RoboCasa_GR1 ✅

Data is saved under assets/benchmark_data/<benchmark>/. Normalization statistics are prepared when needed, and the selected dataloader configuration is updated.

3. VLM Backbone (Optional)

Required only by tri_system:

python scripts/download_assets/download_vlm_backbone.py

Supported VLM backbones

Qwen3-VL-2B-Instruct ✅

Weights are saved under assets/vlm_backbone_ckpt/, and the selected configuration is updated.

4. Visual Encoders (Optional)

Required only for video backbones that use an external encoder:

python scripts/download_assets/download_visual_encoder.py

Supported visual encoders

DINOv3 ViT-B/16 ✅ V-JEPA 2.1 ViT-G/16 ✅
Wan2.2 VAE ✅ FLUX.2 VAE ✅

Weights are saved under assets/visual_encoder_ckpt/, and the selected encoder configuration is updated.

5. Released OpenWAM Checkpoints

Use this downloader to obtain OpenWAM-Alpha releases or OpenWAM-Study checkpoints from the OpenWAM collection:

python scripts/download_assets/download_openwam_checkpoints.py

Checkpoints are saved under assets/openwam_ckpt/openwam_alpha/ or assets/openwam_ckpt/openwam_study/<type>/. Each checkpoint directory contains its own config and can be deployed directly with:

bash scripts/deploy.sh <ckpt_dir_path>

For fine-tuning, set training.finetune_ckpt_path in configs/train.yaml to the downloaded checkpoint directory. Benchmark data is still required.

Quick Start

Quick Start provides a minimal end-to-end example: prepare the assets, train or fine-tune a policy, deploy its checkpoint, and run a first inference check.

The example uses the DualSystem JointSelfAttention architecture, the Wan2.2-TI2V-5B video backbone, and the Mutual attention mask.

Component Selection Configuration
Architecture dual_system / joint_self_attn configs/model/dual_system.yaml
Video backbone wan22_ti2v_5b configs/model/video_backbone/wan22_ti2v_5b.yaml
Attention mask mutual model.architecture.attention_mask_mode
Dataset libero configs/dataloader/libero.yaml

Resource recommendation: We recommend 8 GPUs with 80 GB VRAM each for training. This configuration supports normal training for all architectures using Wan2.2-5B or smaller video backbones. More GPUs are better when available and can further improve training throughput.

From Scratch Training

  1. Download LIBERO and let the downloader update its dataloader configuration:

    python scripts/download_assets/download_benchmark_data.py

    Select LIBERO in the interactive menu.

  2. Download Wan2.2-TI2V-5B:

    python scripts/download_assets/download_video_backbone.py

    Select Wan2.2-TI2V-5B and the desired model source.

  3. Start a debug run with the complete model selection:

    bash scripts/train.sh \
  dataloader=libero \
  model=dual_system \
  model/video_backbone=wan22_ti2v_5b \
  model.architecture.variant=joint_self_attn \
  model.architecture.attention_mask_mode=mutual \
  training.debug=true

    training.debug=true runs 20 steps, saves at steps 10 and 20, and uses a constant learning rate. Check the run output, then set training.debug=false for normal training. Training defaults and CLI overrides are defined in configs/train.yaml. Debug outputs use training.output_path, whose default is outputs/openwam_checkpoints.

  4. Deploy the debug checkpoint and inspect one input-output cycle:

    bash scripts/deploy.sh <debug_ckpt_dir_path>

    Keep the server running, then open another terminal and run the two inference helpers. Deployment enables compile by default, so the first inference may take longer while compilation warms up; later requests are typically faster:

    python scripts/inference_test/inference_single_test.py \
  --server ws://127.0.0.1:8848 --test --state-dim 10

python scripts/inference_test/inference_continuous_test.py \
  --server ws://127.0.0.1:8848 --test --state-dim 10

    The single-request helper checks ping, one prediction, and reset. The continuous helper sends repeated predictions over one connection and reports the returned action dimension and latency. Stop the deployment process after the checks.

OpenWAM-α Fine-Tuning

  1. Download LIBERO as shown above.

  2. Download the OpenWAM-Alpha foundation checkpoint:

    python scripts/download_assets/download_openwam_checkpoints.py

    Select OpenWAM_Alpha and OpenWAM-Alpha-Pretrain-Foundation-Model. Keep the resulting directory as .

  3. Start fine-tuning from that directory:

    bash scripts/train.sh \
  dataloader=libero \
  training.finetune_ckpt_path=<foundation_ckpt_dir_path>

    The default model configuration already matches the required setup above. You can set the same field in configs/train.yaml instead of passing it on the command line. num_frames=33 and video_stride=4 in configs/dataloader/libero.yaml produce the 32-step action horizon expected by the sampler; no separate action_chunk override is needed.

  4. Deploy the resulting checkpoint directory:

    bash scripts/deploy.sh <ckpt_dir_path>

    Install and run the LIBERO client according to the LIBERO evaluation guide.

OpenWAM Usage Guidance

OpenWAM is configured through composable Hydra YAML files. Select an architecture, backbone, dataloader, and runtime behavior by changing configuration values or overriding them on the command line. The task guides below are the maintained entry points for using and extending the repository:

Guide Use it when you want to…
Training and deployment choose a model/dataloader, prepare assets, train, fine-tune, resume, or deploy a policy
Architecture extension extend a video, visual, VLM, action backbone, or WAM architecture
Benchmark integration extend a dataloader and connect a benchmark client to the WebSocket protocol
OpenWAM-α fine-tuning fine-tune the released foundation checkpoint to execute downstream task

💡 Agent tip: Pick the guide that matches your task and feed it directly to your agent — less explaining, more building.

Installation and Assets Preparation above cover environment setup and model or dataset downloads. Benchmark-specific environment and evaluation details remain in the benchmarks directory.

Star History

Star history of OpenWAM-Official/OpenWAM

License

OpenWAM is released under the Apache License 2.0.

Citation

If you use OpenWAM, please cite:

@article{wang2026openwam,
  title   = {OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining},
  author  = {Yuran Wang and Siqiao Huang and Mingleyang Li and Chenhao Zhang and Jiaqi Liang and Weiyang Jin and Yue Chen and Xuemin Chi and Donghao Zhou and Qize Yu and Yu-Kai Wang and Yuhan Rui and Shenzhe Yao and Zhen Yuan and Zhenhao Shen and Kefei Zhu and Zijie Zhu and Ning Gao and Xiaowei Chi and Guanqi He and Shanghang Zhang and Hao Dong and Lin Shao and Hang Zhao},
  year    = {2026},
  journal = {arXiv preprint arXiv: 2609.07398}
}

Contact

Have a question, want to share results, or just follow development? Join us:

OpenWAM WeChat group QR code

WeChat Group
Scan to join 		OpenWAM Discord QR code

Discord
discord.gg/yRSqhpNXu

OpenWAM (OpenWAM-Official/OpenWAM)

Searching Database…

1.1K ·GitHub
OpenWAM
Что это?OpenWAM — открытый исследовательский стек для систематического построения моделей «мир—действие» (World-Action Models, WAM) для роботов: привычные жёстко связанные проектные решения разбиты на модульные взаимозаменяемые компоненты, которые можно сравнивать в контролируемых экспериментах.
Это готовая модель или исследовательский фреймворк?Это модульный исследовательский фреймворк (OpenWAM-Infra) для комбинирования решений по модели, бэкбону, обучению, инференсу, деплою и оценке, в комплекте с результатами контролируемых исследований (OpenWAM-Study) и предобученной моделью (OpenWAM-alpha), построенной на его основе.
Как это запустить?Создайте окружение conda или venv, установите PyTorch с CUDA, затем pip install OpenWAM; deepspeed на этом шаге компилируется из исходников, поэтому C-компилятор (gcc) должен быть в PATH. Дальше интерактивные скрипты-загрузчики подтягивают видео-бэкбоны, данные бенчмарков и чекпоинты в assets/, прежде чем вы начнёте обучение или деплой.
Нужны ли конкретные версии PyTorch и CUDA?Да, авторы рекомендуют PyTorch 2.7.1 с CUDA 12.8 (другие версии могут сработать, но не гарантированы), плюс make и gcc для сборки deepspeed.
Какое железо реально нужно для обучения?Рекомендуемая конфигурация — 8 GPU по 80 ГБ VRAM каждая для обычного обучения с видео-бэкбонами масштаба Wan2.2-5B или меньше; более крупные бэкбоны вроде дополнений Cosmos-Predict2.5-2B требуют дополнительной компиляции transformer-engine с CUDA toolkit/nvcc.
Нужен Linux или будет работать и на Mac/Windows?Установка и инструментарий GPU/CUDA/deepspeed рассчитаны на Linux-окружение с GPU NVIDIA; о поддержке macOS или Windows ничего не говорится.
Что такое OpenWAM-alpha?OpenWAM-alpha — открытая предобученная модель «мир—действие», выпущенная проектом: обучена на 518,5 млн кадров (около 6400 часов) эгоцентричных данных людей и роботов и может дообучаться на прикладных задачах вроде LIBERO.
Какие видео-бэкбоны можно подключить?Среди поддерживаемых бэкбонов Wan2.2-TI2V-5B, Wan2.1-VACE-1.3B, Wan2.1-I2V-14B-480P, Cosmos-Predict2.5-2B и Cosmos3-Edge, каждый скачивается прилагаемыми скриптами загрузки ресурсов.
Какие робототехнические бенчмарки он поддерживает?В репозитории есть загрузчики и конфиги для RoboTwin2.0, RoboDojo, RoboDojo-Real, LIBERO, VLABench, EBench, RoboCasa365 и RoboCasaGR1.
Как модель реально разворачивают и опрашивают?После обучения или скачивания чекпоинта вы запускаете сервер деплоя из каталога чекпоинта и опрашиваете его по протоколу инференса на базе WebSocket; есть вспомогательные скрипты для одиночных и непрерывных запросов предсказаний.
What are similar projects?Zero-WAM, flex-pi, video-prediction-policy-2, S-Space, EasyWAM, FluxVLA, ZDTaichu5.0-9B, tapnet
Which topics is it in?world-action-model, vla, pretraining, world-models, embodied-ai, arxiv, vision-language-model, attention-mechanism, robotics, dataset, academic-research, artificial-intelligence
Creator
OpenWAM-Official
Size
16.2 MB
Stars
1.1K
Updated
CommitOct 14
Age
1mo
Commits · 90 Days