# wuji-ego-mint (ZZJ-Asher/wuji-ego-mint) > MINT is a unified deep learning model that takes ordinary egocentric RGB video and directly predicts world-space camera motion and two-hand (MANO) motion, replacing the usual calibration + depth + SLAM + hand-reconstruction pipeline with a single pass. - GitHub: https://github.com/ZZJ-Asher/wuji-ego-mint - GitHub.Locker page: https://github.locker/ZZJ-Asher/wuji-ego-mint - Stars: 268 - Language: Python - License: MIT - Last push: 2026-09-14 ## Questions and answers Written by GitHub.Locker from the README and the repository; check the README below for details. **Is this a trained model I run, or a pipeline I have to assemble myself?** It's a single trained model (1.139B parameters) with ready inference/training code and a Web Viewer; the older multi-stage EgoPipeline is kept only as the offline supervision generator, not as something you deploy. **How do I start using the Web Viewer?** Install the inference environment, run scripts/downloadassets.sh to fetch the public MINT checkpoint into checkpoints/model.safetensors, then open the Viewer, load the checkpoint, pick a LeRobot episode or an egocentric video, and click Start Inference. **Do I need to sign up anywhere before running it?** Yes — besides the public MINT checkpoint, you must register on the official MANO website, accept the MANO license, and manually place the MANO left/right hand model files under assets/mano/. **What hardware do I need, and is the released dataset ready for benchmarking?** Inference requires an NVIDIA GPU with at least 24GB of VRAM, and the released dataset's camera trajectories are currently scale-enlarged — usable for pretraining but not yet for metric evaluation. **How much faster is MINT than the old multi-stage pipeline?** On one GPU MINT is 17.4x faster than the VITRA-based pipeline (72.4ms vs 1260ms per frame), and 12.5x faster on four GPUs, by replacing calibration, depth, SLAM and hand reconstruction with one model. **What's inside the released dataset?** Structured, non-video annotations covering 1,021 hours and 560K episodes of egocentric footage, generated by EgoPipeline, alongside the MINT checkpoint and training code. **Can I test it on my own video?** Yes — drop an .mp4, .mov, .avi, .mkv or .webm egocentric clip into the Viewer's input picker. Without ground truth it just runs in prediction-only mode, hiding the GT panels. **How is MINT evaluated?** It's tested zero-shot on HOT3D and ARCTIC, datasets never used in training, with a coverage-aware protocol scoring hand detection, articulated pose, wrist orientation and temporal jitter. --- ## README (checked against GitHub 2026-11-03) Source: https://github.com/ZZJ-Asher/wuji-ego-mint/blob/main/README.md
108 MINT renders playing at once, one tile per clip

MINT: A Unified Model for World-Space Camera and Hand Motion
Estimation from Scalable Egocentric Pipeline Supervision

Zijie Zhu1,3,4  ·  Weiren Cai3  ·  Yizhou Wang1,3  ·  Zhenjie Yang4  ·  Yide Liu3,5  ·  Jiahao Chen3,\*  ·  Guanqi He2,3,\* 1ShanghaiTech University   2Tsinghua University   3Wuji Technology   4The University of Hong Kong   5Zhejiang University   \*Corresponding authors [![Project Page](https://img.shields.io/badge/Project-Page-2f855a)](https://zzj-asher.github.io/mint-project-page/) [![arXiv](https://img.shields.io/badge/arXiv-2609.04958-b31b1b)](https://arxiv.org/pdf/2609.04958) [![Model](https://img.shields.io/badge/%F0%9F%A4%97_Model-mint__v1-ff9d00)](https://huggingface.co/ZZJAsher/mint_v1) [![Model](https://img.shields.io/badge/ModelScope_Model-mint__v1-624aff)](https://www.modelscope.cn/models/AsherZhu/mint_v1) [![Dataset](https://img.shields.io/badge/%F0%9F%A4%97_Dataset-1,021_hours-ff9d00)](https://huggingface.co/datasets/ZZJAsher/wuji_ego_mint) [![Dataset](https://img.shields.io/badge/ModelScope_Dataset-1,021_hours-624aff)](https://www.modelscope.cn/datasets/AsherZhu/wuji_ego_mint) [![License](https://img.shields.io/badge/License-MIT-3fa03f)](LICENSE) [中文说明](README_ZH.md)
Eight panels: device ground truth beside MINT prediction in the ego view, in world space, and retargeted to the Wuji hand in MuJoCo --- ### 🤲 Meet MINT — making world-space egocentric motion data more accessible Egocentric motion reconstruction estimates how a camera moves and how both hands are positioned, oriented, and moving in the surrounding space. Producing this data often requires capture hardware with built-in tracking or a pipeline that combines calibration, monocular depth, SLAM, hand reconstruction, and trajectory cleanup. Deploying multiple models, managing errors between stages, and working with different licenses add to the cost of producing data at scale. **MINT takes ordinary egocentric RGB video and uses a unified model to predict camera and two-hand motion, then composes them into world-space outputs.** The Web Viewer supports inference and inspection on an NVIDIA GPU with at least 24 GB of VRAM. Long videos are processed in windows, with camera trajectories joined across windows. - **Unified camera and hand modeling.** A shared spatiotemporal representation feeds four prediction heads: camera extrinsics, field of view, camera-frame MANO, and per-frame hand presence. Explicit rigid composition produces world-space hand motion without depth maps or point clouds at inference. - **Structured pipeline amortization.** EgoPipeline generates structured camera and hand supervision offline. MINT learns from these labels, reducing the need to deploy multiple independent models when processing new videos. - **Open models and research resources.** The release provides model weights, training and inference code, structured non-video annotations covering **1,021 hours** of egocentric footage, and reference code for EgoPipeline orchestration, cleaning, and export. Source videos and separately licensed assets must be obtained under their own access terms. Same frames through the conventional ego pipeline and through MINT: the pipeline places both hands away from the real hands, MINT keeps them on the hands
Same frames, same overlay renderer. Left: the input. Middle: the conventional ego pipeline, whose output is pseudo-label, not ground truth. Right: MINT predictions.
--- ## 📑 Table of Contents
Collapse - [📋 Release status](#-release-status) - [🚀 Quick Start: Web Viewer](#-quick-start-web-viewer) - [🧠 How MINT works](#-how-mint-works) - [📊 What MINT is measured on](#-what-mint-is-measured-on) - [📦 What is included](#-what-is-included) - [🏋️ Training and optional pipeline reconstruction](#️-training-and-optional-pipeline-reconstruction) - [🗂️ Public Ego pretraining data](#️-public-ego-pretraining-data) - [⚠️ Limitations and future work](#️-limitations-and-future-work) - [🧾 Repository layout](#-repository-layout) - [📚 Documentation](#-documentation) - [✨ Acknowledgements](#-acknowledgements) - [📜 License](#-license) - [📖 Citation](#-citation)
--- ## 📋 Release status | | Item | Where | | :-- | :-- | :-- | | ✅ | MINT checkpoint, 1.139B total parameters | [Hugging Face](https://huggingface.co/ZZJAsher/mint_v1) · [ModelScope](https://www.modelscope.cn/models/AsherZhu/mint_v1) | | ✅ | Inference, Web Viewer, training code | this repository | | ✅ | 1,021-hour structured egocentric dataset (non-video portion) | [Hugging Face](https://huggingface.co/datasets/ZZJAsher/wuji_ego_mint) · [ModelScope](https://www.modelscope.cn/datasets/AsherZhu/wuji_ego_mint) | | ✅ | Benchmark implementation and CLI, integrated into the Viewer | `eval/model_effect/benchmark/` | | ✅ | EgoPipeline orchestration, cleaning and LeRobot export reference | `ego_pipeline/` | | ✅ | Video normalisation and hand-filtering scripts | [`ego_pipeline/preprocessing/`](ego_pipeline/preprocessing/) | | ✅ | Wuji hand URDF/MJCF/STL and retargeting | `eval/simulate/wuji-retargeting/` | | ✅ | Zero-shot HOT3D / ARCTIC result table (paper Table 1) | [Camera-frame bimanual reconstruction](#camera-frame-bimanual-reconstruction) | | ⏳ | Scale-corrected camera trajectories — the released dataset's trajectories are scale-enlarged | The current ones are usable for pretraining, not for metric evaluation. Cause in [Public Ego pretraining data](#️-public-ego-pretraining-data) | | ❌ | License-restricted pipeline adaptations (adapted HaWoR source, weights, MANO) | cannot be redistributed; see [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md) | ## 🚀 Quick Start: Web Viewer The Web Viewer is the primary MINT entry point. It lets you select a checkpoint, load the MINT model, run inference on a LeRobot episode or an egocentric MP4 video, and inspect GT/prediction overlays, camera trajectories, hand motion, frame metrics, and benchmark results in one interface. > **Hardware requirement: MINT model inference requires an NVIDIA GPU with at least 24 GB of VRAM.** Install the inference environment and download the public MINT model. The Viewer also requires the separately licensed MANO hand models. Create an account on the [official MANO website](https://mano.is.tue.mpg.de/), accept the MANO license, and download the MANO release before running the asset check below. ```bash git clone https://github.com/wuji-technology/wuji-ego-mint.git cd wuji-ego-mint bash scripts/create_env.sh inference conda activate mint-inference bash scripts/download_assets.sh # Copy the MANO files from the downloaded release into the project. mkdir -p assets/mano/mano_right assets/mano/mano_left cp /path/to/MANO_RIGHT.pkl assets/mano/mano_right/MANO_RIGHT.pkl cp /path/to/MANO_LEFT.pkl assets/mano/mano_left/MANO_LEFT.pkl python -m mint doctor --profile inference python -m mint viewer ``` ### Model and asset locations Place the models and assets used by Quick Start at the paths below. `scripts/download_assets.sh` downloads the public MINT checkpoint automatically; MANO must be downloaded from its official website and copied manually. | Model or asset | Download source | Path in this repository | Notes | | --- | --- | --- | --- | | MINT checkpoint | [ModelScope](https://www.modelscope.cn/models/AsherZhu/mint_v1) or [Hugging Face](https://huggingface.co/ZZJAsher/mint_v1) | `checkpoints/model.safetensors` | The download script places and verifies it automatically; use the same path for a manual download. | | MANO left- and right-hand models | [MANO website](https://mano.is.tue.mpg.de/) | `assets/mano/mano_right/MANO_RIGHT.pkl`
`assets/mano/mano_left/MANO_LEFT.pkl` | Registration and acceptance of the MANO License are required. | | LingBot-Map pretrained backbone | [LingBot-Map](https://github.com/robbyant/lingbot-map) | `assets/models/lingbot-map.pt` | Optional; download only when required by the selected configuration. | | Wuji Hand URDF, MJCF, and STL | Included in this repository | `eval/simulate/wuji-retargeting/wuji_retargeting/wuji-description/hand/body/` | No additional download is required. | ### Using the Viewer The Viewer automatically opens `http://127.0.0.1:8011` in the default browser. Then: 1. In **Model and Sample**, keep `checkpoints/model.safetensors` or select another compatible checkpoint. 2. Click **Load Model** and wait until the model status is ready. 3. Select a LeRobot episode or an egocentric MP4 video, then choose the camera-inference, hand-window, and geometry settings. 4. Click **Start Inference**. 5. Inspect the synchronized GT/Pred 2D view, fixed-world and camera-frame 3D panels, per-frame values, losses, exports, and optional benchmark tools. The export menu can either compose the selected views into one grid MP4 or download 2D, fixed-world 3D, MuJoCo, and Wuji Hand GT/Pred renders as eight individual MP4 files in a ZIP. ![MINT Web Viewer after loading the model and running inference](docs/asset/mint-web-viewer.png) **Testing your own video** needs no configuration change and no command line: browse to the video in the input-directory picker at the bottom right of the Viewer and click it. Supported formats are `.mp4`, `.mov`, `.avi`, `.mkv` and `.webm`. A plain video carries no ground truth, so the Viewer runs in prediction-only mode — GT panels, the GT/Pred side-by-side layout and the loss readout are hidden, and everything shown is prediction. All visualization operations live in the Viewer panel — there is no separate command-line visualization step. See [Installation](docs/installation.md) for installation, CUDA, MANO, and offline deployment details. ## 🧠 How MINT works Overview: large-scale egocentric video on the left, MINT and dataset diversity in the middle, zero-shot world-space outputs on the right
Left: the released supervision — 1,021 h, 560 K episodes of egocentric video, and how its diversity compares with EgoDex, Ego4D and EPIC-KITCHENS. Right: MINT predictions for an unseen video — world-space camera and hand trajectory, MANO hands, and the same motion retargeted to a robot hand.
Within each window, four prediction heads share a spatiotemporal representation. Explicit rigid transforms compose the predicted camera and hand states into world-space motion. ### Model at a glance | | | | --- | --- | | **Input** | monocular egocentric RGB, 378 × 518 frames, patch 14, 999 tokens per frame | | **Backbone** | LingBot-Map / GCT — 24 alternating frame-wise / global attention pairs | | **Clip** | T = 32 frames, re-anchored to the window's own first frame | | **Total parameters** | 1.139B | | **Head 01** | camera extrinsics, 7-D `[t, q]`, iterative causal refinement, 4 steps | | **Head 02** | field of view, `f_h, f_w`, independent temporal branch, Softplus | | **Head 03** | camera-frame hand MANO, 218-D, left + right, component queries, 2 refinements | | **Head 04** | per-frame hand presence, logits L / R, patch cross-attention | | **Composition** | `x_c = R x_w + t`, `p_w = Rᵀ(p_c − t)`, `Q_w = Rᵀ Q_c` — explicit and differentiable | | **At inference** | no depth map, no point cloud, no dense 3D intermediate | | **Training** | Stage 1 pretrain on 1,021 h of pipeline supervision → Stage 2 correct the camera trajectory on a small high-precision trajectory set, with the geometric encoder and the hand, presence and field-of-view modules frozen | EgoPipeline stages: hand detection and frame filtering, camera pose estimation through GeoCalib, MoGe-2 and MegaSaM, HaWoR hand reconstruction, then outlier rejection, interpolation, temporal smoothing and the world-coordinate transform
EgoPipeline stages.
### Where the supervision comes from **EgoPipeline** is our open-source implementation of the conventional multi-stage route, and it stays in the project as the *supervision generator*, not as the deployment path: | Stage | Component | Produces | | --- | --- | --- | | 00 | [preprocessing and filtering](ego_pipeline/preprocessing/) | normalise fps/resolution/duration; cut out stretches with no hands or more than two; validate metadata, split overlapping clips | | 01 | GeoCalib | camera intrinsics | | 02 | MoGe-2 | monocular depth | | 03 | MegaSaM / DROID-SLAM | metric camera track | | 04 | HaWoR | camera-frame MANO | | 05 | cleanup | outlier rejection, gap interpolation, temporal filters | Its output is **pseudo-label, not ground truth** — that distinction is load-bearing everywhere in this repository. ### Why one model instead of the chain Three views of MINT predictions: ego view with reprojected MANO, the world-space camera and two-hand trajectory, and that trajectory retargeted to the Wuji hand in MuJoCo A multi-stage pipeline typically combines several visual models and post-processing components. Different models may extract features from the same video independently, intermediate estimates can propagate errors across stages, and deployment requires coordinating multiple dependencies and interfaces. MINT shares a spatiotemporal representation across camera and hand prediction, reducing repeated feature extraction and integration work when processing new videos. Inference throughput measured on **RTX 4090D**, at 512 × 384 and 30 fps under identical conditions; time is the **marginal cost per frame in steady state**. Speedups are against **VITRA**, which **EgoPipeline**, this project's data pipeline, was built by optimising. The three methods are points on one line of work: VITRA to start, EgoPipeline as the pipeline-level optimisation, MINT replacing the chain with a single model.
Method One GPU Four GPUs
Time ↓
ms/frame		 fps ↑ Speedup ↑
vs VITRA		 Time ↓
ms/frame		 fps ↑ Speedup ↑
vs VITRA
VITRA1260.00.8—283.33.5—
EgoPipeline———83.412.03.4×
MINT72.413.817.4×22.744.112.5×
Against the conventional pipeline, MINT is **17.4×** faster on one GPU (VITRA 1260.0 → 72.4 ms/frame) and **12.5×** on four (283.3 → 22.7 ms/frame). The four-GPU columns show where that comes from: optimising the conventional pipeline itself (EgoPipeline) accounts for 3.4× (283.3 → 83.4), and the unified model takes it to 22.7. EgoPipeline is itself distributed-optimised (Ray multi-GPU operators, persistent workers, asynchronous CPU stages): rewriting its serial single-GPU execution as a scheduled 4-GPU worker pool took 179.0 s down to 63.9 s per 270 frames, **2.8×** internal to the pipeline. That is whole-clip wall-clock including decode and write-out, a different accounting from the steady-state cost above; the two do not multiply. ## 📊 What MINT is measured on ### Camera-frame bimanual reconstruction **Result sources: MINT and MINT + UKF use the latest coverage-aware evaluation; all other results are taken from the ViDiHand evaluation.** Following the coverage-aware protocol in Sec. V-A of the paper, missed hands receive the error of a canonical MANO placeholder instead of being excluded from pose evaluation. FAcc, recall, and F1 measure detection; MPJPE-p and PA-MPJPE-p measure articulated pose; GO-p and CT-p measure wrist orientation and hand placement; Jitter measures temporal smoothness. MINT is evaluated on HOT3D and ARCTIC in a zero-shot setting; neither dataset is used in either of its two training stages. `MINT + UKF` uses the same model weights as MINT and applies an unscented Kalman filter (UKF) at inference time, with all other evaluation settings held constant. Bold marks the best displayed value in each column; ties are all bolded.
Method Detection 3D pose Orient. & pos. Temporal
FAcc ↑ Recall ↑ F1 ↑ MPJPE-p ↓
mm		 PA-MPJPE-p ↓
mm		 GO-p ↓
deg		 CT-p ↓
m		 Jitter ↓
mm/frame²
ARCTIC
InterWild0.8780.9430.95930.8215.9525.390.09746.58
HaMeR0.8750.9430.95729.2014.6024.910.09518.28
Hamba0.8330.9120.94131.2317.1727.820.11015.36
WildHands0.8790.9460.96025.7013.9422.320.05812.97
OmniHands0.8660.9490.95429.6714.2024.580.08745.31
WiLoR0.9190.9510.97422.0111.8717.360.07524.09
Dyn-HaMR0.8420.9180.95127.9017.0225.950.12112.84
HaWoR0.7000.8170.89545.3626.3843.330.14919.79
MINT0.9180.9570.97829.5516.5720.880.0912.77
MINT + UKF0.9180.9570.97829.5816.5620.890.092.67
HOT3D
InterWild0.6690.8810.86877.1724.8158.500.213101.16
HaMeR0.6920.9040.88368.3121.4649.640.10223.63
Hamba0.6320.8280.85371.7329.6256.530.12818.51
WildHands0.6550.8630.84452.7928.9553.930.15722.89
OmniHands0.6490.8950.86863.2822.6849.120.13369.51
WiLoR0.8270.8970.93730.9719.9825.750.09817.98
Dyn-HaMR0.6140.8110.80274.2138.2043.850.57144.94
HaWoR0.3480.4990.65471.4066.0379.350.26223.87
MINT0.9450.9830.95329.92613.65621.0990.19612.057
MINT + UKF0.9450.9830.95329.91813.64621.0910.1962.373
In the latest local evaluation, applying UKF reduces Jitter from 12.057 to 2.373 mm/frame² on **HOT3D** and from 12.77 to 2.67 mm/frame² on **ARCTIC**, while leaving detection metrics unchanged. The hand prediction heads have not yet been fine-tuned on high-precision data; see [Limitations and future work](#️-limitations-and-future-work). **HaWoR supplies hand pseudo-labels to EgoPipeline.** It is used at stage 04 to estimate camera-frame MANO parameters. These labels provide supervision for MINT's hand branch, but they retain reconstruction errors and do not constitute high-precision ground truth. ### World-frame camera trajectory The error metrics are transcribed from **Table 2 of the [paper](https://arxiv.org/pdf/2609.04958)**: HOT3D (27 sequences, 94,978 frames) and the ARCTIC P2 validation split (34 sequences, 25,883 frames). Sequences are evaluated at **full length, with SE(3)-only alignment and no fitted scale** — so scale error is charged rather than absorbed. The arc-length ratio is **GT path length / predicted path length**, averaged equally across sequences: above 1 means the predicted path is too short; below 1 means it is too long. `MegaSaM†` runs without depth refinement. `MINT w/o stage 2` never sees metric ground truth. Bold marks follow the paper; for the arc-length ratio, the target is 1.
Method ATE ↓ (mm) RPE-T ↓ (mm) RPE-R ↓ (deg) Arc len.
GT / Pred → 1		 ATE ↓
%
mean med. mean med. mean med.
HOT3D
DROID-SLAM49.139.65.363.520.2270.1460.7780.36
HaWoR200.3179.28.607.331.0980.9280.9501.28
InfiniteVGGT124.8100.013.529.671.4920.3920.5560.80
LingBot-Map85.551.07.566.180.6840.2530.7120.55
MegaSaM†94.465.33.182.130.0820.0630.7160.69
MINT w/o stage 2524.7434.68.758.370.2340.2290.4663.29
MINT181.7155.54.694.780.2840.2591.0941.15
ARCTIC
DROID-SLAM181.549.633.8414.251.0060.4230.9648.07
HaWoR66.229.424.084.160.2980.1510.7592.87
InfiniteVGGT79.069.116.2112.631.2650.6550.2843.23
LingBot-Map59.662.09.178.470.9800.7170.5912.46
MegaSaM†51.450.48.735.580.7790.7251.9562.15
MINT w/o stage 263.759.33.533.620.2510.2430.7552.63
MINT81.983.23.393.370.2560.2511.4123.40
Following Sec. V-A of the paper, **ATE and ATE% measure global trajectory accuracy, while RPE-T and RPE-R measure relative motion accuracy**. RPE reflects local motion consistency; full-sequence ATE captures errors that accumulate over longer trajectories. Both are relevant to the quality of the reconstructed motion. ATE and RPE are computed as an RMSE for each sequence and summarized by a mean and a median. The paper reports mean RPE-T of **3.39 mm on ARCTIC** and **4.69 mm on HOT3D** for MINT. Dedicated trajectory systems retain an advantage in absolute accuracy: MINT uses 32-frame windows without loop closure or global optimization, so long sequences can accumulate drift, as discussed in Sec. V-C. **ATE is not where this checkpoint wins**: 181.7 mm on HOT3D against DROID-SLAM's 49.1 mm. The arc-length ratios of 1.094 on HOT3D and 1.412 on ARCTIC indicate predicted paths that are too short under the GT / Pred convention. These benchmark predictions should be distinguished from the scale-enlarged pipeline trajectories in the released pretraining dataset. On HOT3D, Stage 2 moves the ratio from 0.466 to 1.094, correcting an overlong path towards the target length, and lowers ATE from 524.7 mm to 181.7 mm. On ARCTIC, the ratio moves from 0.755 to 1.412 and ATE increases from 63.7 mm to 81.9 mm. ### Evaluation protocol and result sources The hand-reconstruction and camera-trajectory tables follow Tables 1 and 2 of the paper. Non-MINT hand results retain their source in the ViDiHand evaluation; camera-trajectory results follow the full-sequence protocol stated in Table 2. Shared metric definitions alone do not establish that different evaluation splits or manifests are directly comparable. - **Hand evaluation.** The coverage-aware protocol includes missed ground-truth hands through a canonical MANO placeholder penalty. Pose metrics therefore include true positives and false negatives. - **Camera evaluation.** Each complete sequence is aligned with SE(3), without fitting scale. ATE and RPE are computed per sequence; the reported mean and median aggregate the successfully evaluated sequences. - **Reproduction.** Compare local results with a reference only when the checkpoint, evaluation split and manifest, preprocessing, alignment, and aggregation settings match the intended experiment. MINT's metric definitions and reporting code are available in [`eval/model_effect/benchmark/`](eval/model_effect/benchmark/). **Integrity statement.** Our EgoPipeline and MINT training code support reproduction. We commit that every metric reported by this project is an authentic result produced under the stated evaluation protocol; we do not alter the original numeric results. To reproduce another method's or baseline's exact values, use that method's official repository and environment. MINT's metric definitions, alignment rules, aggregation logic, and reporting code are available in `eval/model_effect/benchmark/` for inspection. The complete implementation and tests live in `eval/model_effect/benchmark/` and are integrated into the Viewer's Benchmark panel. Before use, download HOT3D and ARCTIC, organize the benchmark data as required by each adapter, and install the optional runtime. The CLI remains available: ```bash python eval/model_effect/benchmark/run.py \ --ckpt /path/to/checkpoint \ --config configs/training/mint_step2.yaml \ --data-root /path/to/benchmark-data ``` Set `CAMERA_TRAJECTORY_ROOT` for camera-trajectory exports when needed. Aliyun defaults are placeholders; users must configure the workspace, resource, image, CPFS, credentials, and environment. This project does not provision or maintain benchmark environments. ## 📦 What is included | Area | Entry point | Purpose | | --- | --- | --- | | Infer and view | `python -m mint viewer` | Web UI for LeRobot GT, predictions, 2D/3D trajectories, and frame metrics. | | Train | `python -m mint train` | Train the camera-and-hand MINT model with Accelerate/DDP. | | Benchmark | `python eval/model_effect/benchmark/run.py` | Open benchmark CLI with user-provided data and environment. | | Audit | `python -m mint doctor` | Check package dependencies, model imports, backend source, and asset files. | | Pipeline reference | `ego_pipeline/` | Reuse the open Ray orchestration, interfaces, cleaning, and LeRobot export code after integrating the required upstream backends locally. | ## 🏋️ Training and optional pipeline reconstruction Training or local pipeline-development work requires the full environment: ```bash bash scripts/create_env.sh full conda activate mint python -m mint doctor --profile full ``` The public release uses the Viewer as the unified entry point for MINT inference, visualization, and artifact export. This project publishes only code that third-party licenses permit us to distribute; license-restricted adaptations and internal integrations are excluded, so **this repository does not contain the complete production data pipeline.** GeoCalib, MoGe, and Mega-SAM source snapshots are distributed under `third_party/`, but some production adaptations cannot be published under their upstream license terms. In particular, the locally adapted HaWoR source is Git-ignored because CC BY-NC-ND prohibits redistribution of modifications. Weights, MANO files, and other separately licensed assets are also excluded. If you need to reconstruct the data-generation pipeline, first read [`THIRD_PARTY_NOTICES.md`](THIRD_PARTY_NOTICES.md), obtain and install every required upstream library and asset under its own terms, and implement the necessary compatibility adapters locally. The open code in `ego_pipeline/` documents the orchestration, interfaces, data flow, cleaning, manifests, and LeRobot export contracts. An AI coding assistant may help reconcile upstream API differences, but the resulting integration remains the user's responsibility and must comply with all licenses. Only after completing that local integration should the data-profile doctor and `python -m mint pipeline` be treated as usable entry points. The public release provides an implementation reference, not a one-command reproduction of the production data generator. Use MINT directly for inference; for pipeline reconstruction, supply the licensed upstream code and assets and complete the local integration described in [Data pipeline](docs/data-pipeline.md). If you do go down this road, the least painful order is: install each upstream project separately by its own instructions and get it running on its own first, then wire them together against the source in `ego_pipeline/`. That wiring step suits an AI coding assistant well — point it at the upstream APIs and at this repository's calling conventions and have it write the compatibility layer. The fixed in-repository paths for each asset are: | Data-pipeline asset | Path in this repository | | --- | --- | | GeoCalib weights | `model/geocalib/pinhole.tar` | | MoGe weights | `model/moge2/model.pt` | | Mega-SAM weights | `model/megasam/megasam_final.pth` | | HaWoR weights | `model/hawor/hawor.ckpt` | | HaWoR configuration | `model/hawor/model_config.yaml` | | HaWoR detector | `model/hawor/detector.pt` | | DROID-SLAM weights | `third_party/HaWoR/weights/external/droid.pth` | | Metric3D weights | `third_party/HaWoR/thirdparty/Metric3D/weights/metric_depth_vit_large_800k.pth` | | HaWoR right-hand MANO | `third_party/HaWoR/_DATA/data/mano/MANO_RIGHT.pkl` | | HaWoR left-hand MANO | `third_party/HaWoR/_DATA/data_left/mano_left/MANO_LEFT.pkl` | ### Train One configuration per stage is kept. Stage 1 pretrains on the pipeline supervision; Stage 2 initialises from the Stage 1 weights and trains only the camera head on high-precision camera-trajectory data, and its output is the checkpoint released here. Run from the repository root. Before Stage 1, prepare the backbone weights at `assets/models/lingbot-map.pt` and the three LeRobot datasets at `data/lerobot/{ego4d,egodex,epickitchen}/lerobot_v3`, or update `model.pretrained` and `data.root` to your own paths. Both configurations use offline W&B logging by default; see [Training](docs/training.md) for logging and path settings. ```bash python -m mint train --config configs/training/mint_step1.yaml python -m mint train --config configs/training/mint_step2.yaml ``` Set the Stage 2 `data.root` to your prepared camera-trajectory LeRobot dataset; `data/lerobot/stage2/lerobot_v3` is a placeholder. Set `train.init_from` to the `model.safetensors` produced by your own Stage 1 run, or copy it to the default location `checkpoints/stage1/model.safetensors`. `mint train` consumes a compatible, separately prepared LeRobot dataset and writes training checkpoints. After training, select the new checkpoint directly in the Viewer panel for interactive inspection. ## 🗂️ Public Ego pretraining data We provide the non-video portions of the `ego4d`, `egodex`, and `epickitchen` data processed by this repository's Ego data-production pipeline: - [Hugging Face: ZZJAsher/wuji_ego_mint](https://huggingface.co/datasets/ZZJAsher/wuji_ego_mint) - [ModelScope: AsherZhu/wuji_ego_mint](https://www.modelscope.cn/datasets/AsherZhu/wuji_ego_mint) | | | | --- | --- | | Structured supervision | camera trajectory, two-hand MANO, per-frame presence, world-space motion | | Kept after filtering and trajectory cleaning | **1,021.514 h** · 560,649 episodes · 110,323,558 frames | | Sources | Ego4D 332.874 h · EPIC-KITCHENS-100 55.194 h · EgoDex 633.446 h | | Yield | 59.1 % of 1,729 raw hours | That figure is the whole chain accumulated, not any single rule: [normalisation, cutting stretches with no hands or more than two, and dropping segments that end up too short](ego_pipeline/preprocessing/), per-stage failures and timeouts, and outlier rejection during trajectory cleaning all reduce what survives. The final step — packing the cleaned results into the training LeRobot dataset — runs in an internal repository and is not part of this release, so only the overall figure is given here; it cannot be broken down per rule. > **Important: the camera trajectories in this data were generated by this repository's Ego data pipeline and currently exhibit obvious scale enlargement. We recommend using the data only to pretrain this project's Ego hand reconstruction model. Do not use it for real-scale evaluation, precise camera trajectory evaluation, or as real-scale ground truth.** The public dataset does not include `.mp4` videos. The source videos belong to Ego4D, EgoDex, and EPIC-KITCHENS and are governed by their respective dataset licenses and access terms, so this project cannot redistribute them. Users who need the videos must apply for and download them through the corresponding official dataset channels and verify their own usage and redistribution rights. ## ⚠️ Limitations and future work Camera-frame hand estimation remains constrained by pseudo-label quality, and the hand prediction heads have not yet been fine-tuned on high-precision data. Although the stage-two supervision used to refine camera trajectories is more accurate than data generated by traditional pipelines, residual errors remain relative to ground truth. The 32-frame training window may lead to accumulated drift when processing long sequences. Future work will prioritize more accurate, diverse, metric-scale supervision to improve generalization and data-generation quality, bring predictions closer to ground truth, and address trajectory drift in long videos. For limitations of the released data and pipeline code, see [Public Ego pretraining data](#️-public-ego-pretraining-data) and [Training and optional pipeline reconstruction](#️-training-and-optional-pipeline-reconstruction). ## 🧾 Repository layout ```text mint/ |-- configs/ Two-stage training recipes and inference settings |-- data/samples/ Approved Hot3D LeRobot v3 sample |-- eval/model_effect Original visualization, inference adapters, and benchmarks |-- docs/ Architecture and operational guides | `-- asset/ README and documentation images, animations, and videos |-- environments/ Full and inference-only dependency specifications |-- mint/ CLI, inference engine, renderer, and web viewer |-- model_train/ Training engine, model, losses, and LeRobot loader |-- ego_pipeline/ Ray scheduling, actors, model backends, trajectory cleanup, manifests, and export | `-- preprocessing/ video normalisation and hand filtering, ahead of the pipeline stages |-- scripts/ Reproducible setup, asset, privacy, and sample tools `-- third_party/ Redistributable source snapshot; adapted HaWoR is local-only, assets excluded ``` ## 📚 Documentation | | | | --- | --- | | [Architecture](docs/architecture.md) | how the model and the repository fit together | | [Installation](docs/installation.md) | profiles, CUDA, MANO, offline deployment | | [Data pipeline](docs/data-pipeline.md) | EgoPipeline stages and local reconstruction | | [Training](docs/training.md) | the two-stage recipe | | [LeRobot training data format](docs/lerobot-training-data.md) | what a compatible dataset must contain | | [Inference and viewer](docs/inference.md) | Viewer panels, exports, benchmark tools | | [Privacy and release checklist](docs/privacy.md) | consent, review, redistribution | | [Security policy](SECURITY.md) | how to report a vulnerability | | [Third-party notices](THIRD_PARTY_NOTICES.md) | upstream licenses — read before distributing | ## ✨ Acknowledgements MINT is made possible by the following research projects, models, and datasets. - **[VITRA](https://microsoft.github.io/VITRA/)** — MINT's data-processing architecture, egocentric reconstruction workflow, world-space camera/hand annotations, and LeRobot conversion conventions evolved from the VITRA and VITRA-1M data engine. - **[LingBot-Map](https://github.com/robbyant/lingbot-map)** — provides the core model architecture and the upstream source adapted for MINT camera-and-hand training and inference. - **[HaWoR](https://github.com/ThunderVVV/HaWoR)** — provides monocular hand motion reconstruction, MANO estimation, tracking, and world-space hand-processing components used by the optional data pipeline. Its use remains subject to the upstream non-commercial, no-derivatives license. - **Camera, depth, and tracking research** — [GeoCalib](https://github.com/cvg/GeoCalib), [MoGe](https://github.com/microsoft/MoGe), [Mega-SAM](https://github.com/mega-sam/mega-sam), [DROID-SLAM](https://github.com/princeton-vl/DROID-SLAM), [UniDepth](https://github.com/lpiccinelli-eth/UniDepth), [Metric3D](https://github.com/YvanYin/Metric3D), [DeepCalib](https://github.com/alexvbogdan/DeepCalib), [DINOv2](https://github.com/facebookresearch/dinov2), [VGGT](https://github.com/facebookresearch/vggt), InfiniteVGGT, and [PyTorch3D](https://github.com/facebookresearch/pytorch3d). - **Hand models, simulation, and retargeting** — [MANO](https://mano.is.tue.mpg.de), [SMPL-X](https://smpl-x.is.tue.mpg.de), [MuJoCo](https://mujoco.org), and the Wuji hand description and retargeting components used by the optional Viewer panels. - **Datasets and benchmarks** — [HOT3D](https://github.com/facebookresearch/hot3d), [ARCTIC](https://arctic.is.tue.mpg.de), [Ego4D](https://ego4d-data.org), [EPIC-KITCHENS](https://epic-kitchens.github.io), and [EgoDex](https://github.com/apple/ml-egodex). Dataset access and redistribution remain governed by each dataset's own terms. We thank all upstream authors and maintainers. This acknowledgement does not replace their citation or license requirements; see [Third-party notices](THIRD_PARTY_NOTICES.md) before use or distribution. ## 📜 License wuji-ego-mint's original code is released under the MIT License. Upstream models, datasets, MANO assets, vendored LingBot-Map files, and optional research backends retain their own licenses. Review [Third-party notices](THIRD_PARTY_NOTICES.md) before distribution. ## 📖 Citation The paper is under review; this entry will be replaced with the published reference. ```bibtex @misc{zhu2026mint, title = {MINT: A Unified Model for World-Space Camera and Hand Motion Estimation from Scalable Egocentric Pipeline Supervision}, author = {Zhu, Zijie and Cai, Weiren and Wang, Yizhou and Yang, Zhenjie and Liu, Yide and Chen, Jiahao and He, Guanqi}, year = {2026}, note = {Manuscript under review} } ``` ## 📮 Contact We are glad to share what we have learned and to help move egocentric data forward. If you run into anything — installation, data generation, model training, inference, or the Viewer — please contact Zijie Zhu. WeChat: `z3132544408`; email: `3132544408@qq.com`.