video-prediction-policy-2 README

The project's README from github.com/roboterax/video-prediction-policy-2. As one markdown file: README.md.

Video Prediction Policy 2: Predict Better, Act Better

Official implementation of "Video Prediction Policy 2: Predict Better, Act Better".

arXiv: 2610.10270 Project Page ModelScope — Models Hugging Face — Models

News

Overview

We introduce Video Prediction Policy 2 (VPP2), a world-action model (WAM) with strong zero-shot generalization in both video prediction and action generation.

Our pipeline has three stages:

  1. Event-level video pretraining. Continue pretraining the base video model on a large, diverse dataset of manipulation videos with detailed captions to improve generalization across open-ended tasks.
  2. Single-step visual planning. Post-train and distill the video model into a single-step visual planner with a fixed prediction horizon.
  3. Action learning. Add an action module through a mixture-of-transformers (MoT) architecture to learn an implicit inverse dynamics model.

VPP2 paper overview: event-level video prediction, instruction following, and robot policy learning.

Installation

git clone https://github.com/roboterax/video-prediction-policy-2.git
cd video-prediction-policy-2
conda create -n vpp2 python=3.10 -y
conda activate vpp2

Install the CUDA/PyTorch dependencies using the installation guide, then install the dependencies from the repository root:

python -m pip install -r requirements-train.txt -c environment-reference.txt

The scripts load VPP2 directly from this checkout; installing the vpp2 package is unnecessary. For inference and data processing only, use requirements.txt.

Get started

Benchmark What is included Guide
RoboDojo Joint video–action training, checkpoint export, policy serving, and closed-loop evaluation RoboDojo guide
LIBERO Video-stage training, action training, and standard benchmark evaluation LIBERO guide
LIBERO-OOD / PRO Generalization evaluation with the LIBERO policy checkpoint OOD and PRO setup

RoboDojo configurations live in configs/robodojo/; LIBERO configurations live in configs/libero/. Follow the guide for the benchmark you want to use.

Model Weights

Weights are available on Hugging Face and ModelScope. The Hugging Face repository is public; ModelScope requires an account with access.

Checkpoint Use Hugging Face ModelScope
RoboDojo history-conditioned Video-10k Starting Video model for joint + Action2B training Download ModelScope
RoboDojo joint Video-100k Video model paired with Action2B-100k for evaluation Download ModelScope
RoboDojo Action2B-100k RoboDojo policy evaluation Download ModelScope
LIBERO Video-10k Video model for Action training and evaluation Download ModelScope
LIBERO Action-30k LIBERO, LIBERO-OOD and LIBERO-PRO evaluation Download ModelScope

Evaluation uses the matching Video and Action checkpoints together. Shared Wan2.1 VAE, CLIP, UMT5 and tokenizer assets are also available in the model repositories under checkpoints/Wan2.1-I2V-14B-480P/. Follow the benchmark guides to download the complete assets and normalization files:

TODO

Resource Link
Paper arXiv:2610.10270
Project page VPP2 Project Page
XPolicyLab adapter VPP2 RoboDojo adapter PR
ModelScope haodong123/VPP2_preview
Hugging Face Haodong082399/VPP2

Documentation

Citation

If you use VPP2 in your research, please cite our paper (BibTeX):

@misc{guo2026videopredictionpolicy2,
  title = {Video Prediction Policy 2: Predict Better, Act Better},
  author = {Yanjiang Guo and Haodong Yan and Zhide Zhong and Zhongru Zhang and
            Qingyuan Yang and Qingzhou Lu and Xiaoyu Chen and Yen-Jen Wang and
            Shuying Deng and Chenghan Yang and Puzhen Yuan and Chenxin Liu and
            Tun Ban and Xiang Zhu and Yichen Liu and Kun Feng and Haoang Li and
            Jianyu Chen},
  year = {2026},
  eprint = {2610.10270},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url = {https://arxiv.org/abs/2610.10270}
}

License

The code is released under the MIT License. Third-party components retain their original licenses; see third-party notices. Model weights, datasets, and simulator assets have their own terms.

video-prediction-policy-2 (roboterax/video-prediction-policy-2)

Searching Database…

257 ·GitHub
Video-prediction-policy-2
这是什么？Video Prediction Policy 2(VPP2)是一个面向机器人操作的世界-动作模型(WAM),它预测未来的视频帧并将其转化为机器人动作,在跨任务上具有很强的零样本泛化能力。
这是拿来即用的机器人控制器,还是要我自己训练的研究代码库？它是一个围绕三阶段训练管线(视频预训练、视觉规划蒸馏、动作学习)构建的研究代码库,不是即插即用的控制器。
怎么让它跑起来？按照安装指南安装 CUDA 和 PyTorch,然后在仓库根目录安装项目依赖;脚本直接从代码检出目录加载 VPP2,因此 vpp2 包本身无需单独安装。
评估它需要预训练检查点吗？需要,匹配的 Video 和 Action 检查点必须配套使用,另外还要共享的 Wan2.1 VAE、CLIP、UMT5 和分词器资源;权重托管在 Hugging Face(公开)和 ModelScope(需要已审批的账号)上。
论文里的所有内容都已经放出了吗？还没有——机器人视频预训练的骨干检查点,以及大规模事件级视频预训练的代码和配置仍未发布。
它开箱支持哪些基准测试？RoboDojo(视频-动作联合训练、检查点导出、策略服务、闭环评估)和 LIBERO,包括其 LIBERO-OOD 和 LIBERO-PRO 泛化变体。
VPP2 在基准测试中表现如何？它在官方 RoboDojo-Sim 排行榜上总排名第一,以 39.26 的分数和 32.26% 的成功率,在 agent 与非 agent 两个赛道都取得了最先进的结果。
基于 mixture-of-transformers 的动作模块是做什么的？它是加在单步视觉规划器之后的动作模块,学习一个隐式逆动力学模型,把预测出的视频帧转化为可执行的机器人动作。
提供了哪些模型检查点？RoboDojo(Video-10k、Video-100k、Action2B-100k)和 LIBERO(Video-10k、Action-30k)各有独立的 Video 和 Action 检查点,外加共享的 Wan2.1 VAE、CLIP、UMT5 和分词器资源,全部托管在 Hugging Face 和 ModelScope 上。
VPP2 能泛化到自带基准之外吗？得益于在大量带字幕、多样化的操作视频上预训练,它的设计目标是对开放式操作任务具备很强的零样本泛化能力,不过 RoboDojo 和 LIBERO 目前是仅有的官方支持评估基准。
What are similar projects?flex-pi, EasyWAM, OpenWAM, Zero-WAM, SolarWM, pointdit, RoboICL, VisionHOPE
Which topics is it in?world-action-model, model-training, world-models, embodied-ai, arxiv, mixture-of-experts, pretrained-models, paper, robotics, video-generation, academic-research, nvidia
Creator
roboterax
Size
6.7 MB
Stars
257
Updated
CommitOct 12
Age
10d
Commits · 90 Days