openlambda README

项目的 README,来自 github.com/LambdaHumanoid/openlambda。合并为一个 markdown 文件:README.md。

Lambda-humanoid Team

Openlambda for Generalist Humanoid Learning

arXiv Paper Project Page Hugging Face Paper Video Demo
Stage II checkpoint HumanVerse viewer License: Apache 2.0 Python 3.11 | 3.12

Highlights · Overview · Results · Get started · Hardware · Alignment · TODO · Citation

$\lambda_{\mathit{0}}$ learns whole-body humanoid loco-manipulation from everyday human experience.

λ₀ demo: human data capture transitions to humanoid object transport

News

Highlights

Overview

λ₀ overview: HumanVerse, three-stage learning, and humanoid loco-manipulation results

λ₀ couples a shared vision–language backbone with an action expert. It takes a robot camera image, a language instruction and proprioception, and outputs whole-body motion tokens executed by SONIC together with hand commands. Training runs in three stages:

Stage Data Purpose
I · Egocentric hand-motion pretraining Seven existing egocentric human datasets Learn object interaction and articulated hand motion
II · Robot-aligned whole-body human mid-training HumanVerse, aligned to the G1 observation and action spaces Learn whole-body coordination from human experience
III · Robot post-training 600 task-aligned human demonstrations (150 per task, held out from HumanVerse) plus native robot demonstrations Ground the model in executable robot actions
Full pipeline figure

λ₀ pipeline: robot-free human demonstrations, visual alignment, coarse-to-fine action alignment, policy learning and zero-shot whole-body loco-manipulation

Results

Real-world evaluation on Unitree G1 across four tasks (Close Laptop, Open Fridge, Pull Curtain, Close Drawer), 40 trials per method.

Method Success (%) Progress (%)
GR00T N1.6 7.5 21.3
Ψ₀ 10.0 31.8
StarVLA 27.5 54.6
π₀.₅ 40.0 54.0
λ₀ (ours) 62.5 80.4
Training recipe Success (%) Progress (%)
No Stage I or II 15.0 54.0
Stages I + III 22.5 59.9
Stages II + III 50.0 75.9
Full recipe (I + II + III) 62.5 80.4

See the project page and the report for the full evaluation.

Releases

Resource Description Link
Stage II checkpoint λ₀ after 100k mid-training updates, used to initialize the Stage III example 🤗 lambda0-midtrain-100k
HumanVerse viewer Browse 20 HumanVerse excerpts with egocentric RGB video and SMPL body and hand motion 🤗 HumanVerse-500-Viewer
Alignment examples Four trajectories with data, reports and replay videos examples · v0.1.1 release
Visual alignment examples Three recordings with four-stage previews and synchronized motion Examples
Hardware mounts STL files for the G1–Revo2 adapters and the head-mounted GoPro bracket hardware/assets

Get started

git clone https://github.com/LambdaHumanoid/openlambda.git
cd openlambda
Task Guide
Train λ₀ Model setup
Collect real-robot demonstrations G1 recorder and tutorial
Collect human data Hardware → Environment → Recording
Align and replay motion Environment → Alignment
Align egocentric video Environment → Alignment

λ₀ foundation model

Install and run λ₀

Requires Python 3.11 or 3.12. From the repository root:

PYTHON=python3.11 bash scripts/install.sh
source .venv/bin/activate
python scripts/test.py -v

scripts/test.py checks package installation and CLI compatibility.

Optional extras: bash scripts/install.sh serve,tracking enables HTTP inference and WandB; fast adds platform-dependent GPU kernels. Existing environments can still use python -m pip install -e . (or .[serve], .[tracking], .[fast]) from the repository root.

Training

Stage Data selection Training config
I · Pretraining configs/lambda0/pretrain/data.json configs/lambda0/pretrain/train.json
II · Mid-training configs/lambda0/midtrain/data.example.json configs/lambda0/midtrain/train.json
III · Post-training configs/lambda0/posttrain/data.example.json configs/lambda0/posttrain/train.json

The Stage III example initializes from the raw weights in the Stage II checkpoint. Set STAGE2_CHECKPOINT_DIR to its downloaded directory.

From the repository root, set the paths required by the selected config, then prepare data and validate it (Stage I example). Relative paths in JSON configs resolve from the config file; command-line paths resolve from the current working directory:

python scripts/prepare.py --source-config configs/lambda0/pretrain/data.json \
  --data-root "$DATA_ROOT" --output "$BUNDLE_DIR"
python scripts/train.py --config configs/lambda0/pretrain/train.json --check-only
Training data format

Stage I expects first-person video and Parquet observation.state and observation.valid_mask fields, each with 138 coordinates per frame. Stage II and III examples expect canonical observation.state, observation.valid_mask, action, and action.valid_mask fields plus aligned video and metadata; the action and state dimensions follow the selected contract in src/openlambda/models/action_specs.py. Defaults are in src/openlambda/recipes/; editable examples are in configs/lambda0/.

Human collection outputs (observation.pico.*, observation.images.gopro) and alignment outputs (64-dimensional action.motion_token) require conversion before training. Stage II uses a 76-dimensional SONIC/Revo2 action contract, including hand commands. Conversion must supply aligned video, actions, proprioception, timestamps, task text, and validity masks. lambda0 prepare only bundles episodes already in the canonical format.

Hardware setup

Human Hardware

Animated human hardware tour: PICO headset, five body trackers and neck-mounted GoPro
🔴 PICO 4 Ultra
  • Tracking: PICO 4 Ultra + trackers
    for whole-body and hand tracking
  • Software: XRobo​Toolkit
    Setup guide · Python SDK
  • Connec​tion: same LAN as the PC
    enable Head, Hand and Send

🔵 5× PICO Trackers
  • Placement: five trackers
    both wrists, waist and both ankles
  • Calibration: PICO Motion Tracker
    pair and calibrate in the app (guide)
  • Mode: Full Body in XRobo​Toolkit
    enable Hand tracking (setup guide)

🟠 GoPro
  • Mount: neck-mounted bracket
    Example: JD listing
  • Alignment: match height, pitch and framing to the robot camera
  • Connec​tion: Bluetooth pairing and WiFi setup;
    WiFi / COHN recording and download (API)

Robot Hardware

Animated robot hardware tour: Unitree G1, BrainCo Revo2 hands and head-mounted GoPro
🟢 Unitree G1

🔵 BrainCo Revo2
  • Hand: 6 active DOF per hand
    11 including passive joints (specs)
  • Software: official SDK
    hand control and state feedback
  • Mount: custom G1–Revo2 adapters
    STL: part 1 · part 2

🟠 GoPro
  • Mount: head-mounted bracket
    STL: part 1 · part 2
  • Alignment: match height, pitch and framing to the human camera
  • Connec​tion: USB control and video to robot;
    wired / WiFi forwarding to host (API)

Environment setup

Use separate environments for robot control, human collection, visual alignment, and model training.

🤖 Robot Control & Data Collection

Host

Ubuntu 22.04 · NVIDIA driver 580+.

1. Install dependencies.

sudo apt-get update
sudo apt-get install -y --no-install-recommends ca-certificates curl build-essential cmake git git-lfs libgl1 libegl1 libegl-mesa0 libglib2.0-0 ffmpeg libyaml-cpp-dev libeigen3-dev libmsgpack-dev libzmq3-dev nlohmann-json3-dev libssl-dev zlib1g-dev
curl -LsSf https://astral.sh/uv/0.11.14/install.sh | sh
source "$HOME/.local/bin/env"
curl -fL https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb -o /tmp/cuda-keyring.deb
sudo dpkg -i /tmp/cuda-keyring.deb
sudo apt-get update
sudo apt-get install -y --no-install-recommends cuda-minimal-build-12-4

2. Install TensorRT and ONNX Runtime. Download TensorRT-10.13.3.9.Linux.x86_64-gnu.cuda-13.0.tar.gz from NVIDIA to the repository root, then run:

sudo tar -xzf TensorRT-10.13.3.9.Linux.x86_64-gnu.cuda-13.0.tar.gz -C /opt
curl -fL https://github.com/microsoft/onnxruntime/releases/download/v1.16.3/onnxruntime-linux-x64-1.16.3.tgz -o /tmp/onnxruntime.tgz
sudo tar -xzf /tmp/onnxruntime.tgz -C /opt

3. Install environment. Repeat the exports when opening a new terminal.

export TensorRT_ROOT=/opt/TensorRT-10.13.3.9
export onnxruntime_ROOT=/opt/onnxruntime-linux-x64-1.16.3
export CUDAToolkit_ROOT=/usr/local/cuda-12.4
export LD_LIBRARY_PATH="$TensorRT_ROOT/lib:$onnxruntime_ROOT/lib:$CUDAToolkit_ROOT/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
bash data_collection/robot/setup/install_host.sh

4. Activate environment.

source sonic/.venv/bin/activate

Robot

ARM64 · Ubuntu 20.04 · ROS 2 Foxy · Python 3.8.

1. Install environment.

sudo apt-get update
sudo apt-get install -y python3-dev python3-venv build-essential cmake git python3-colcon-common-extensions ros-foxy-rmw-cyclonedds-cpp ros-foxy-rosidl-generator-dds-idl libyaml-cpp-dev libssl-dev libgl1 libglib2.0-0
bash data_collection/robot/setup/install_robot.sh

2. Activate environment.

source /opt/ros/foxy/setup.bash
source .venv-robot/ros_ws/install/setup.bash
source .venv-robot/bin/activate

👤 Human Data Collection

Ubuntu 22.04 · Python 3.10.

1. Install environment.

sudo apt-get update
sudo apt-get install -y ca-certificates python3.10-dev python3.10-venv build-essential cmake git ffmpeg bluez network-manager
bash data_collection/human/setup/install.sh

2. Activate environment.

source .venv-human/bin/activate

🎨 Vision Alignment

Tested on Linux with a 48 GB NVIDIA GPU. Allow about 35 GB of disk space. Requires uv, FFmpeg, and an NVIDIA driver compatible with CUDA 13.0; reuse these if already installed.

1. Obtain model access. Request access to SAM 3.1, then authenticate:

uvx --from huggingface_hub hf auth login

2. Install the environments and models.

bash data_collection/human/vision_alignment/setup/install.sh

The installer creates two isolated environments under runtime/vision: Python 3.12 for segmentation/rendering and Python 3.10 for inpainting. It downloads pinned models and verifies their checksums. The alignment command calls both environments automatically; no manual switching is needed.

Installation scripts, dependency pins, model manifests, and the SAM compatibility patch are in data_collection/human/vision_alignment/setup/. Processing code, tests, and example motion files are in data_collection/human/vision_alignment/.


Human data collection

1. Prepare devices.

Connect the host and PICO headset to the same network. Start the XRoboToolkit PC service on the host and the XRoboToolkit app on the headset, then enable body and hand tracking. Turn on the GoPro, enable Bluetooth pairing and select Video mode; the collector connects it to the host's WiFi network in the next step.

2. Start recording.

lambda-collect-human --task-name "{task_name}" --task "{task_instruction}"

Open http://127.0.0.1:8080. Enter the host's WiFi credentials if needed, then click Prepare collection → Start recording → Save episode / Discard episode. Wait for saving before recording again. Episodes are saved in LeRobot format to data/{task_name}/; new recordings append to the dataset.

3. Check and synchronize data.

lambda-sync-human --root "data/{task_name}"

Synchronizes motion and video at 50 Hz. Output: data/{task_name}_synced/; synchronization and quality reports: meta/.

Real-robot data collection

Record G1/SONIC/Revo2 demonstrations with a wired GoPro. Video, robot states and hand targets are saved on G1 during teleoperation, then aligned and exported as Parquet and MP4 on the workstation.

Follow the collection guide or 中文教程 to record your first episode. The data format reference covers the 44D state and 76D action layout for training adapters.

Data alignment

Action alignment

Action alignment: human motion, SONIC rollout initialization, kinematic reference correction with offline IK, dynamics-aware refinement with an MLP response model, aligned robot motion

Four example trajectories and their data, reports, and videos are available. Each replay compares motion before (left) and after (right) alignment.

https://github.com/user-attachments/assets/c4b43c5d-199a-447b-b0a3-afe6afe76d3e

https://github.com/user-attachments/assets/a7d8f8d4-a22a-40b6-816a-bf562f691720

https://github.com/user-attachments/assets/403659cc-2f87-4d60-bf84-14bf831a65d7

https://github.com/user-attachments/assets/71c1de99-8f2e-4ee1-875c-fe4be36db618

1. Align motion.

The pipeline performs human-reference alignment, robot-reference alignment, and dynamics-aware optimization. For hand tasks, H first corrects SMPL collar, shoulder, and elbow poses using a uniformly scaled virtual body while preserving the original end-effector targets. Foot tasks encode the input SMPL directly without this correction; K and D retain their joint-space and motion-token optimization.

Place the input recording at {task_name}/input.parquet. Set {endpoints} to hands for hand alignment or feet for foot alignment.

lambda-align \
  --input "{task_name}/input.parquet" \
  --endpoints {endpoints} \
  --output "results/{task_name}"

Results are saved under results/{task_name}/: data.parquet contains the original human SMPL motion and aligned tokens; run.json records per-round metrics and the selected round.

2. Replay motion.

SMPL replay: encode the original human motion online, without human-reference correction, to simulate before alignment.

lambda-replay \
  --input "{task_name}/data.parquet" \
  --mode smpl \
  --output "results/{task_name}/before"

Token replay: replay the aligned motion tokens to simulate after alignment.

lambda-replay \
  --input "{task_name}/data.parquet" \
  --mode token \
  --output "results/{task_name}/after"

Each command saves replay.mp4 in its output directory. Results may vary slightly between runs.

Visual alignment

The pipeline replaces human arms with rendered G1 arms while preserving the original gloves and hand–object contact. SAM 3.1 segments human arms and wrist trackers. DiffuEraser reconstructs the background.

Synchronized joint trajectories drive the G1 model through an approximate virtual camera. Wrist locations and segmentation masks then refine the rendered arms' placement in the image without changing the joint angles.

The examples below demonstrate visual alignment for closing a laptop, opening a fridge, and pulling a curtain. Each row shows RGB → Mask → Inpainting → Robot-arm rendering. Download the video and motion files to reproduce the results.

Close Laptop · Video · Motion

Human RGB SAM 3.1 Mask Inpainting Robot-arm Rendering
Close Laptop animation Close Laptop animation Close Laptop animation Close Laptop animation

Open Fridge · Video · Motion

Human RGB SAM 3.1 Mask Inpainting Robot-arm Rendering
Open Fridge animation Open Fridge animation Open Fridge animation Open Fridge animation

Pull Curtain · Video · Motion

Human RGB SAM 3.1 Mask Inpainting Robot-arm Rendering
Pull Curtain animation Pull Curtain animation Pull Curtain animation Pull Curtain animation

Align a recording.

Save the video as {task_name}/rgb.mp4 and its synchronized motion as {task_name}/motion.npz. The NPZ contains body_q (29 G1 joint angles in radians per frame), joint_names, and fps matching the video.

runtime/vision/.venv-sam/bin/lambda-visual-align \
  --video "{task_name}/rgb.mp4" \
  --motion "{task_name}/motion.npz" \
  --runtime "runtime/vision" \
  --output "results/{task_name}"

Results are saved under results/{task_name}/: aligned.mp4 contains the aligned video; run.json records settings and completed stages.

Inference

python scripts/serve.py (also lambda0 serve) takes camera observations, language instructions, and proprioceptive state, and returns actions for the checkpoint's configured action space. It does not send actuator commands. Motion-level simulation replay is available under Action alignment.

For the six original AMO-based SIMPLE core tasks, use scripts/eval_simple.py in the simulation environment with the existing HTTP server. The SIMPLE evaluation guide covers checkpoint requirements, task-specific normalization, the AMO adapter, evaluation settings and result validation.

TODO

[!NOTE] Due to data licensing, privacy and institutional policy constraints, the full pretraining and HumanVerse mid-training datasets are not publicly released at this time. The Stage I and II code and configurations are provided as a reference for the training recipe.

Community

Join our WeChat community to discuss humanoid intelligence, human data, and λ₀.

For bug reports and feature requests, please open a GitHub issue.

Lambda0 WeChat community QR code
微信扫码加入 Lambda0 交流群

Citation

If you find λ₀ or openlambda useful in your research, please cite our report:

@misc{lambdahumanoidteam2026lambda0scalingopenworldegocentric,
  title         = {$\lambda_0$: Scaling Open-World Egocentric Human Data for Humanoid Loco-Manipulation},
  author        = {{Lambda-humanoid Team}},
  year          = {2026},
  eprint        = {2610.18950},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2610.18950},
}
Paper arXiv
Towards a General Humanoid Loco-Manipulation Model via Egocentric Whole-Body Human Data Pretraining 2610.00438
EgoHumanoid-V2: Human-to-Humanoid Transfer of Coordinated Whole-Body Skills for Loco-Manipulation 2609.37181
EgoAlign: Bridging the Human-Humanoid Gap for Long-Range Loco-Manipulation 2609.38046
@article{xu2026generalhumanoidlocomanipulationmodel,
  title   = {Towards a General Humanoid Loco-Manipulation Model via Egocentric Whole-Body Human Data Pretraining},
  author  = {Xu, Chongyang and Wu, Zhao and Chen, Jin and Jiang, Yiming and Ye, Jinhui and Jiang, Yuming and Zhang, Shifeng and Feng, Ziliang and Xu, Mu and Chen, Yilun and Lu, Li and Hoi, Steven C. H.},
  journal = {arXiv preprint arXiv:2610.00438},
  year    = {2026}
}

@article{chen2026egohumanoidv2,
  title   = {{EgoHumanoid-V2}: Human-to-Humanoid Transfer of Coordinated Whole-Body Skills for Loco-Manipulation},
  author  = {Chen, Jin and Jiang, Yiming and Xu, Chongyang and Shi, Modi and Peng, Shijia and Chen, Li and Li, Tianyu and Xu, Mu and Chen, Yilun and Hoi, Steven and Li, Hongyang},
  journal = {arXiv preprint arXiv:2609.37181},
  year    = {2026}
}

@article{jiang2026egoalign,
  title   = {{EgoAlign}: Bridging the Human-Humanoid Gap for Long-Range Loco-Manipulation},
  author  = {Jiang, Yiming and Chen, Jin and Xu, Chongyang and Chen, Yilun and Hao, Aimin and He, Yisheng},
  journal = {arXiv preprint arXiv:2609.38046},
  year    = {2026}
}

Acknowledgments

openlambda builds on the following open-source projects and prior work. We thank their authors for making them available.

Area Projects
Policy learning X-VLA, StarVLA
Whole-body control SONIC
Human data collection XRoboToolkit, Open GoPro
Robot data collection Unitree G1 model files
Motion alignment EgoHumanoid-V2, EgoAlign

See NOTICE for detailed source and asset attributions.

License

openlambda is released under the Apache License 2.0.

Third-party components keep their original licenses:

openlambda (LambdaHumanoid/openlambda)

正在搜索数据库…

这是什么?λ0 是一个人形视觉-语言-动作模型,它从日常的人类第一人称数据中学习全身移动与操作能力,然后将其迁移到 Unitree G1 机器人上。
这是现成的机器人应用,还是模型代码库?这是一个研究代码库,不是开箱即用的机器人应用。它包括训练代码、人类数据采集和动作对齐工具,以及用于推理的 serve 脚本。推理输出的是动作 token 和手部指令,而不是执行器指令。
我该怎么让它跑起来?请使用 Python 3.11 或 3.12。在仓库根目录运行 python -m pip install -e .,然后运行 scripts/test.py 检查软件包和命令行工具。可选扩展包有 serve(HTTP 推理)、tracking(WandB)和 fast(GPU 内核)。
我需要哪些运行环境和硬件?你需要三套独立的环境:机器人控制和数据采集使用 Ubuntu 22.04,搭配 NVIDIA 驱动 580 及以上版本、TensorRT 和 ONNX Runtime;机器人的 ARM64 Ubuntu 20.04 使用 ROS 2 Foxy 和 Python 3.8;人类数据采集使用 Ubuntu 22.04 和 Python 3.10。真实世界数据还需要 PICO 4 Ultra、动作追踪器、GoPro 以及配备 BrainCo Revo2 灵巧手的 Unitree G1。
它能直接用在任何机器人上吗?不能。它只针对配备 Revo2 灵巧手的 Unitree G1。还有几项内容列在待办清单里:HumanVerse 示例数据、硬件指南、数据转换、第三阶段评估,以及带安全检查的部署。自定义数据必须先转换为标准格式。
新手友好吗?暂时不友好。部署涉及机器人硬件、动作捕捉设备、ROS 2 和多阶段训练。新手教程仍在待办清单中。
它支持哪些操作系统?它只为 Linux 设计。主机使用 Ubuntu 22.04,机器人的板载计算机使用 ARM64 Ubuntu 20.04,训练则使用它自己的环境。
什么是 HumanVerse?它是一组 500 小时的无机器人采集数据,涵盖开放世界场景,将第一人称视频与同步的全身及手部动作配对。它是中期训练的主要数据来源。
采集人类数据比遥操作便宜多少?README 显示,在多类代表性任务中,其采集效率约为机器人遥操作的 3.4 倍,每次演示约 29 秒,而遥操作约为 100 秒。
它在真实的 Unitree G1 上表现如何?在四项任务上,它的真实世界成功率为 62.5%,任务进度为 80.4%,分别比最强的外部基线高出 22.5 和 25.8 个百分点。
我可以从预训练检查点开始吗?可以。第二阶段检查点在 100k 次中期训练更新后发布在 Hugging Face 上,并用于初始化第三阶段示例。
大小
158 MB
星标
166
时长
1个月
提交 · 90 天