GR00T-WholeBodyControl
This is the codebase for the GR00T Whole-Body Control (WBC) projects. It hosts model checkpoints and scripts for training, evaluating, and deploying advanced whole-body controllers for humanoid robots. We currently support:
- Decoupled WBC: the decoupled controller (RL for lower body, and IK for upper body) used in NVIDIA GR00T N1.5 and N1.6 models;
- GEAR-SONIC Series: our latest iteration of generalist humanoid whole-body controllers (see our whitepaper);
- MotionBricks: a real-time latent generative model for interactive motion control in animation and robotics (see the project page).
News
- [2026-08-31] SONIC deployment update — added per-motor Kp/Kd scaling to reduce stumbling.
- [2026-07-23] SONIC v1.1 checkpoint — released a robot-heading-normalized SONIC controller trained with wrist-pose augmentation for whole-body teleoperation and SONIC-backed VLA execution. See the Model Card and Download Models.
- [06/16] Isaac Teleop Setup (CloudXR / DeviceIO, in-process) — added bring-up docs for the in-process CloudXR path via
isaacteleop[cloudxr], with no separate publisher container. See Isaac Teleop Setup.
- [2026-06-16] Low-latency teleoperation checkpoint — released a SONIC checkpoint with 4-frame SMPL reference lookahead for more responsive whole-body teleoperation. See the Model Card, Download Models, and VLA Inference.
- [2026-05-07] 🤖 End-to-end VLA workflow on G1 — collect teleop data, fine-tune Isaac-GR00T N1.7, and deploy with SONIC whole-body control. See Data Collection, VLA Workflow, and VLA Inference.
- [2026-04-27] 🧩 MotionBricks preview — interactive G1 demo, pretrained checkpoints (VQVAE · pose · root), synthetic training code, and motion-representation docs. See
motionbricks/ and the project page.
- [2026-04-14] 🌐 Live web demo — try SONIC interactively in your browser. Features Kimodo text-to-motion generation.
- [2026-04-10] 🚀 Released SONIC training code and checkpoint on HuggingFace. Train from scratch or finetune. Additional embodiment support and VLA data collection pipeline. See Training Guide.
- [2026-03-24] 🔧 C++ inference stack update: motor error monitoring, TTS alerts, ZMQ protocol v4, idle-mode readaptation. ZMQ header size changed to 1280 bytes.
- [2026-03-16] 📦 BONES-SEED open-sourced — 142K+ human motions (~288 hours) with G1 MuJoCo trajectories.
- [2026-02-19] 🎉 Released GEAR-SONIC: pretrained checkpoints, C++ inference, VR teleoperation, and documentation.
- [2025-11-12] 🏁 Initial release with Decoupled WBC for GR00T N1.5 and N1.6.
Table of Contents
GEAR-SONIC
Website |
Model |
Paper |
Docs
SONIC is a humanoid behavior foundation model that gives robots a core set of motor skills learned from large-scale human motion data. Rather than building separate controllers for predefined motions, SONIC uses motion tracking as a scalable training task, enabling a single unified policy to produce natural, whole-body movement and support a wide range of behaviors — from walking and crawling to teleoperation and multi-modal control. It is designed to generalize beyond the motions it has seen during training and to serve as a foundation for higher-level planning and interaction.
In this repo, we release SONIC's training code, deployment framework, model checkpoints, and teleoperation stack for data collection.
Model Card
SONIC provides three released Unitree G1 checkpoints. Choose the model based on its reference representation and intended deployment.
Available Models
| Model |
Hugging Face location |
SMPL reference input |
Intended use and comments |
| Default SONIC (original release) |
Top-level model_encoder.onnx, model_decoder.onnx, and observation_config.yaml; training checkpoint at sonic_release/last.pt |
10 future frames at 20 ms spacing, approximately 200 ms of reference lookahead |
Default general-purpose SONIC controller for motion tracking, planning, teleoperation, and compatibility with existing deployments. G1 and teleoperation future-reference observations use step5. |
| Low-latency teleoperation |
low_latency/ |
4 future frames at 20 ms spacing, approximately 80 ms of reference lookahead |
Intended for more responsive whole-body teleoperation and VLA execution. G1 and teleoperation future-reference observations use step1. Use its encoder, decoder, and observation config together. |
| SONIC v1.1 |
sonic_v1_1/ |
10 future frames at 20 ms spacing, approximately 200 ms of reference lookahead |
Uses robot-heading-normalized target orientation and was trained with wrist-pose augmentation. Intended for heading-stable whole-body teleoperation and SONIC-backed VLA policies that use this controller. G1 and teleoperation future-reference observations use step5; this is not the low-latency model. |
All three models use the SONIC universal-token controller, produce 64-dimensional latent motion tokens, run the controller at 50 Hz, and support SMPL pose, G1 motion reference, and teleoperation inputs. Deployment uses C++ and TensorRT; the PyTorch checkpoints support Isaac Lab evaluation and continued training.
The lookahead values describe the reference horizon presented to the controller. They are not measurements of total end-to-end teleoperation latency, which also includes sensing, networking, preprocessing, and inference. Model weights are covered by the NVIDIA Open Model License.
Released Files
| Model |
Deployment files |
PyTorch and configuration files |
| Default SONIC |
model_encoder.onnx, model_decoder.onnx, observation_config.yaml |
sonic_release/last.pt, sonic_release/config.yaml |
| Low-latency teleoperation |
low_latency/model_encoder.onnx, low_latency/model_decoder.onnx, low_latency/observation_config.yaml |
low_latency/last.pt, low_latency/config.yaml, low_latency/model_config.yaml |
| SONIC v1.1 |
sonic_v1_1/model_encoder.onnx, sonic_v1_1/model_decoder.onnx, sonic_v1_1/observation_config.yaml |
sonic_v1_1/last.pt, sonic_v1_1/config.yaml, sonic_v1_1/model_config.yaml |
The root config.json
is the canonical release manifest for these variants and shared artifacts. The
official downloader fetches and validates it before downloading model files.
Usage
Download the default model and planner:
python download_from_hf.py
Download the low-latency teleoperation model and planner:
python download_from_hf.py --low-latency
Download SONIC v1.1 and the planner:
python download_from_hf.py --sonic-v1-1
Run the default C++ deployment stack:
cd gear_sonic_deploy
./deploy.sh --input-type zmq_manager real
Run the low-latency C++ deployment stack:
cd gear_sonic_deploy
./deploy.sh \
--cp policy/low_latency/model \
--obs-config policy/low_latency/observation_config.yaml \
--input-type zmq_manager \
real
Run the SONIC v1.1 C++ deployment stack:
cd gear_sonic_deploy
./deploy.sh \
--cp policy/sonic_v1_1/model \
--obs-config policy/sonic_v1_1/observation_config.yaml \
--input-type zmq_manager \
--motor-kp-scale 4,10=1.5 \
--motor-kd-scale 4,10=1.5 \
real
This is the tested SONIC v1.1 deployment tuning. Hardware indices 4 and
10 are the left and right ankle-pitch motors; increasing their control gains
improves whole-body stability and, in practice, wrist tracking. The flags are
opt-in and do not alter other checkpoints.
Run the default Python VLA launcher, which orchestrates the C++ controller and Python inference client:
python gear_sonic/scripts/launch_inference.py \
--camera-host 192.168.123.164 \
--prompt "pick up the cup"
For the low-latency model, add the matching deployment files:
python gear_sonic/scripts/launch_inference.py \
--deploy-checkpoint policy/low_latency/model \
--deploy-obs-config policy/low_latency/observation_config.yaml \
--camera-host 192.168.123.164 \
--prompt "pick up the cup"
For SONIC v1.1, use the matching model/config and gain tuning:
python gear_sonic/scripts/launch_inference.py \
--deploy-checkpoint policy/sonic_v1_1/model \
--deploy-obs-config policy/sonic_v1_1/observation_config.yaml \
--deploy-motor-kp-scale 4,10=1.5 \
--deploy-motor-kd-scale 4,10=1.5 \
--camera-host 192.168.123.164 \
--prompt "pick up the cup"
See Downloading Model Checkpoints for Python checkpoint evaluation and additional deployment options. Test in simulation before using the checkpoint on a physical robot.
VR Whole-Body Teleoperation
SONIC supports real-time whole-body teleoperation via PICO VR headset, enabling natural human-to-robot motion transfer for data collection and interactive control.
SONIC v1.1: Whole-body teleoperation mode with expressive wrist motion and dynamic movement.
SONIC Low Latency: Whole-body teleoperation mode with a successful ground pickup.
This repo can also drive the headset over Isaac Teleop / CloudXR by launching gear_sonic/scripts/pico_manager_thread_server.py --input-source isaac-teleop. The streamer hosts the CloudXR runtime in-process via isaacteleop[cloudxr] — no separate publisher container required. That path is currently documented and supported only for G1 with a Thor backpack. The Isaac Teleop bring-up steps are documented in docs/source/tutorials/isaac_teleop_publisher_setup.md.
| Walking |
Running |
 |
 |
| Sideways Movement |
Kneeling |
 |
 |
| Getting Up |
Jumping |
 |
 |
| Bimanual Manipulation |
Object Hand-off |
 |
 |
Kinematic Planner
SONIC includes a kinematic planner for real-time locomotion generation — choose a movement style, steer with keyboard/gamepad, and adjust speed and height on the fly.
| In-the-Wild Navigation |
 |
| Run |
Happy |
 |
 |
| Stealth |
Injured |
 |
 |
| Kneeling |
Hand Crawling |
 |
 |
| Elbow Crawling |
Boxing |
 |
 |
SONIC Training
SONIC can be trained from scratch on the Bones-SEED
motion capture dataset (142K+ motions, ~288 hours, Unitree G1 retargeted), or finetuned
from the released checkpoint on Hugging Face.
Quick start
# Install training dependencies (Isaac Lab must be installed separately — see docs)
pip install -e "gear_sonic/[training]"
# Download checkpoint + SMPL data from Hugging Face
pip install huggingface_hub
python download_from_hf.py --training
# Download Bones-SEED G1 CSVs from huggingface.co/datasets/bones-studio/seed, then convert and filter
python gear_sonic/data_process/convert_soma_csv_to_motion_lib.py \
--input /path/to/bones_seed/g1/csv/ \
--output data/motion_lib_bones_seed/robot --fps 30 --fps_source 120 --individual --num_workers 16
python gear_sonic/data_process/filter_and_copy_bones_data.py \
--source data/motion_lib_bones_seed/robot --dest data/motion_lib_bones_seed/robot_filtered
# Finetune from released checkpoint (64+ GPUs recommended)
accelerate launch --num_processes=8 gear_sonic/train_agent_trl.py \
+exp=manager/universal_token/all_modes/sonic_release \
+checkpoint=sonic_release/last.pt \
num_envs=4096 headless=True \
++manager_env.commands.motion.motion_lib_cfg.motion_file=data/motion_lib_bones_seed/robot_filtered \
++manager_env.commands.motion.motion_lib_cfg.smpl_motion_file=data/smpl_filtered
Adaptive sampling is enabled by default and attributes failures to the motion
cursor tracked before environment reset.
For the full guide including multi-node training, evaluation, ONNX export, and SOMA encoder setup:
📖 Installation (Training) |
Training Guide
TODOs
- Release pretrained SONIC policy checkpoints
- Open source C++ inference stack
- Setup documentation
- Open source teleoperation stack and demonstration scripts
- Release training scripts and recipes for motion imitation and fine-tuning
- Open source large-scale data collection workflows and fine-tuning VLA scripts.
- Publish additional preprocessed large-scale human motion datasets
What's Included
This release includes:
gear_sonic_deploy: C++ inference stack for deploying SONIC policies on real hardwaregear_sonic: Full SONIC training stack — PPO training, data processing pipeline, and configuration system for training on Bones-SEED and custom motion datasetsmotionbricks: Preview release of the MotionBricks real-time latent generative stack — interactive G1 demo, pretrained checkpoints, synthetic training code, and motion-representation docs
Setup
Git LFS required. This repo contains large binary assets (meshes, ONNX
models). Without Git LFS, you will get small pointer files instead of actual
data, causing silent failures. Install Git LFS first if you don't have it:
sudo apt install git-lfs && git lfs install
MotionBricks pretrained checkpoints are skipped by default to avoid an extra
~2.2 GiB download during normal monorepo setup. MotionBricks GIFs and meshes
still download normally. Fetch the checkpoints explicitly if you plan to run
the MotionBricks demo.
git clone https://github.com/NVlabs/GR00T-WholeBodyControl.git
cd GR00T-WholeBodyControl
git lfs pull
# Optional: fetch MotionBricks pretrained checkpoints.
git lfs pull --include="motionbricks/out/**" --exclude=""
# Verify your environment
python check_environment.py
Which environment do I need?
| I want to... |
Environment |
How to install |
| Train / finetune SONIC |
Isaac Lab's Python env |
Install Isaac Lab, then pip install -e "gear_sonic/[training]" |
| Run MuJoCo simulation |
.venv_sim (auto-created) |
bash install_scripts/install_mujoco_sim.sh |
| VR teleoperation |
.venv_teleop (auto-created) |
bash install_scripts/install_pico.sh |
| Collect data |
.venv_data_collection (auto-created) |
bash install_scripts/install_data_collection.sh |
| **Deploy on real ro |
|
|
NVIDIA GR00T Whole-Body Control — a codebase hosting pretrained checkpoints plus training, evaluation, and deployment code for advanced humanoid robot controllers: the Decoupled WBC used in GR00T N1.5/N1.6, the GEAR-SONIC series, and MotionBricks.