PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimation
Haofei Xu1,2,3 ·
Rundi Wu1 ·
Philipp Henzler1 ·
Nikolai Kalischek1 ·
Michael Oechsle1
Fabian Manhardt1 ·
Marc Pollefeys2,4 ·
Andreas Geiger3,5 ·
Federico Tombari1,6 ·
Michael Niemeyer1
1Google 2ETH Zurich 3University of Tübingen, Tübingen AI Center 4Microsoft 5KE:SAI 6TUM
ICML 2026
PointDiT is a minimalist pixel-space Diffusion Transformer for monocular geometry estimation. It is a plain ViT that denoises raw 3D point map patches directly, conditioned on image tokens from a frozen pre-trained DINOv3: no point map tokenizer, no latent diffusion, no hybrid architecture, and no intricate loss formulation.
Installation
This codebase is developed with Python 3.12, PyTorch 2.7.0, and CUDA 12.8.
We recommend setting up a virtual environment (e.g., conda or venv) before installation:
# conda
conda create -y -n pointdit python=3.12
conda activate pointdit
# or venv
# python -m venv /path/to/venv/pointdit
# source /path/to/venv/pointdit/bin/activate
# torch 2.7.0, cuda 12.8
pip install torch==2.7.0 torchvision==0.22.0 --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt
# DINOv3 encoder code (not vendored; built with torch.hub from a local checkout)
git clone https://github.com/facebookresearch/dinov3.git third_party/dinov3
All commands are run from the repository root, and the launch scripts in scripts/.
The DINOv3 weights are gated and cannot be redistributed, so they are not part of the released
checkpoints. Request access on the
official DINOv3 repository and place the weights
under pretrained/dinov3/ with their exact upstream filenames, or point DINOV3_WEIGHTS_DIR at
the directory holding them. They are needed for evaluation and the demo as well as for training;
see MODELS.md for the file list.
Model Zoo
Pre-trained models are available in the Model Zoo, at 256×256 and 512×512 for
each of PointDiT-B / L / H. All weights are hosted on
Hugging Face.
Download the weights and place (or symlink) them in the pretrained directory:
ln -s YOUR_MODEL_PATH pretrained
The checkpoints ship without the frozen DINOv3 encoder, which is downloaded separately (see
Installation).
Demo
Check scripts/demo_l_512.sh for running our pre-trained models on your own images, with no ground truth and no camera intrinsics required. Set IMAGE_DIR at the top of the script and run it:
bash scripts/demo_l_512.sh
Any .jpg/.png files work, and nested folders are fine, since the loader globs recursively. Results are written to generation/.
Dataset Preparation
See DATASETS.md for detailed instructions on preparing the 12 training datasets and the 7 zero-shot evaluation datasets. We have provided the download link for the 7 evaluation datasets in DATASETS.md.
Symlink the prepared datasets to the datasets directory:
ln -s YOUR_DATASET_PATH datasets
Point clouds are never stored on disk: the training loader reads (RGB, depth, intrinsics) and back-projects on the fly.
Evaluation
Evaluation scripts are also provided in scripts/ for reproducing the results in our paper:
bash scripts/eval_l_512.sh
Swap l for b or h, and 512 for 256. Each script's header comment lists the numbers it should reproduce.
Training
PointDiT is trained in two stages: (1) 256×256 pre-training on SceneNet-RGBD and (2) 512×512 fine-tuning on the 11-dataset mixture.
bash scripts/train_stage1_256_b.sh
bash scripts/train_stage2_512_b.sh
The training scripts in scripts/ contain the exact commands and hyperparameters used for the experiments in our paper. Please refer to them for detailed configurations. Before training, you need to download the DINOv3 weights (see Installation) and prepare the datasets per DATASETS.md.
License
This is not an officially supported Google product. This project is not
eligible for the Google Open Source Software Vulnerability Rewards
Program.
This codebase is released under the Apache License 2.0 (see the LICENSE file at the root of
the repository). It also contains code derived from other open-source projects, which
remains under its original licenses; see THIRD_PARTY_NOTICES for the
full list and notices.
Citation
If you find this work useful, please consider citing:
@inproceedings{xu2026pointdit,
title={PointDiT: Pixel-Space Diffusion for Monocular Geometry Estimation},
author={Xu, Haofei and Wu, Rundi and Henzler, Philipp and Kalischek, Nikolai and Oechsle, Michael and Manhardt, Fabian and Pollefeys, Marc and Geiger, Andreas and Tombari, Federico and Niemeyer, Michael},
booktitle={ICML},
year={2026}
}
Acknowledgements
Our codebase builds upon several excellent open-source projects: JiT, DINOv3, MoGe, Depth Pro and utils3d. We thank all the authors for their great work.
PointDiT 是一个研究代码库，实现了一个极简的像素空间扩散 Transformer，可从单张单目图像估计 3D 几何（点图）。