What is this?
Video Prediction Policy 2 (VPP2) is a world-action model (WAM) for robot manipulation that predicts future video frames and converts them into robot actions, with strong zero-shot generalization across tasks.
Video Prediction Policy 2 (VPP2) is a world-action model (WAM) for robot manipulation that predicts future video frames and converts them into robot actions, with strong zero-shot generalization across tasks.Is this a ready-to-use robot controller or a research codebase I train myself?It's a research codebase built around a three-stage training pipeline (video pretraining, visual-planning distillation, action learning), not a plug-and-play controller.
It's a research codebase built around a three-stage training pipeline (video pretraining, visual-planning distillation, action learning), not a plug-and-play controller.How do I get it running?Install CUDA and PyTorch via the installation guide, then install the repo's dependencies from the repository root; the scripts load VPP2 directly from the checkout so the vpp2 package itself doesn't need separate installation.
Install CUDA and PyTorch via the installation guide, then install the repo's dependencies from the repository root; the scripts load VPP2 directly from the checkout so the vpp2 package itself doesn't need separate installation.Do I need the pretrained checkpoints to evaluate it?Yes, matching Video and Action checkpoints are required together, plus shared Wan2.1 VAE, CLIP, UMT5 and tokenizer assets; weights are hosted on Hugging Face (public) and ModelScope (needs an approved account).
Yes, matching Video and Action checkpoints are required together, plus shared Wan2.1 VAE, CLIP, UMT5 and tokenizer assets; weights are hosted on Hugging Face (public) and ModelScope (needs an approved account).Is everything from the paper already released?Not yet - the robot-video pretrained backbone checkpoint and the large-scale event-level video pretraining code and configs are still pending.
Not yet - the robot-video pretrained backbone checkpoint and the large-scale event-level video pretraining code and configs are still pending.Which benchmarks does it support out of the box?RoboDojo (joint video-action training, checkpoint export, policy serving, closed-loop evaluation) and LIBERO, including its LIBERO-OOD and LIBERO-PRO generalization variants.
RoboDojo (joint video-action training, checkpoint export, policy serving, closed-loop evaluation) and LIBERO, including its LIBERO-OOD and LIBERO-PRO generalization variants.How does VPP2 perform on benchmarks?It ranks #1 overall on the official RoboDojo-Sim leaderboard, achieving state-of-the-art results with a 39.26 Score and 32.26% success rate across agent and non-agent tracks.
It ranks #1 overall on the official RoboDojo-Sim leaderboard, achieving state-of-the-art results with a 39.26 Score and 32.26% success rate across agent and non-agent tracks.What does the mixture-of-transformers action module do?It's an action module added after the single-step visual planner that learns an implicit inverse dynamics model, turning predicted video frames into executable robot actions.
It's an action module added after the single-step visual planner that learns an implicit inverse dynamics model, turning predicted video frames into executable robot actions.What model checkpoints are provided?Separate Video and Action checkpoints for RoboDojo (Video-10k, Video-100k, Action2B-100k) and LIBERO (Video-10k, Action-30k), plus shared Wan2.1 VAE, CLIP, UMT5 and tokenizer assets, all hosted on Hugging Face and ModelScope.
Separate Video and Action checkpoints for RoboDojo (Video-10k, Video-100k, Action2B-100k) and LIBERO (Video-10k, Action-30k), plus shared Wan2.1 VAE, CLIP, UMT5 and tokenizer assets, all hosted on Hugging Face and ModelScope.Can VPP2 generalize beyond the benchmarks it ships with?It's designed for strong zero-shot generalization to open-ended manipulation tasks thanks to pretraining on a large, diverse set of captioned manipulation videos, though RoboDojo and LIBERO are currently the only officially supported evaluation benchmarks.
It's designed for strong zero-shot generalization to open-ended manipulation tasks thanks to pretraining on a large, diverse set of captioned manipulation videos, though RoboDojo and LIBERO are currently the only officially supported evaluation benchmarks.What are similar projects?flex-pi, EasyWAM, OpenWAM, Zero-WAM, SolarWM, pointdit, RoboICL, VisionHOPE
flex-pi, EasyWAM, OpenWAM, Zero-WAM, SolarWM, pointdit, RoboICL, VisionHOPEWhich topics is it in?world-action-model, model-training, world-models, embodied-ai, arxiv, mixture-of-experts, pretrained-models, paper, robotics, video-generation, academic-research, nvidia
world-action-model, model-training, world-models, embodied-ai, arxiv, mixture-of-experts, pretrained-models, paper, robotics, video-generation, academic-research, nvidia