What is this?
PointDiT is a research codebase for a minimalist pixel-space Diffusion Transformer that estimates 3D geometry (point maps) from a single monocular image.
PointDiT is a research codebase for a minimalist pixel-space Diffusion Transformer that estimates 3D geometry (point maps) from a single monocular image.Is this a special architecture or a hybrid system?It is a plain Vision Transformer that denoises raw 3D point map patches directly in pixel space, conditioned on image tokens from a frozen pre-trained DINOv3 encoder — no point map tokenizer, no latent diffusion, no hybrid architecture.
It is a plain Vision Transformer that denoises raw 3D point map patches directly in pixel space, conditioned on image tokens from a frozen pre-trained DINOv3 encoder — no point map tokenizer, no latent diffusion, no hybrid architecture.How do I run it on my own images?Set IMAGEDIR at the top of scripts/demo_l512.sh and run it; the loader globs .jpg/.png files recursively (nested folders are fine), and results land in generation/. No ground truth or camera intrinsics are needed.
Set IMAGEDIR at the top of scripts/demo_l512.sh and run it; the loader globs .jpg/.png files recursively (nested folders are fine), and results land in generation/. No ground truth or camera intrinsics are needed.What do I need installed before trying it?Python 3.12, PyTorch 2.7.0, and CUDA 12.8, plus the gated DINOv3 weights requested separately from the official DINOv3 repo and placed under pretrained/dinov3/.
Python 3.12, PyTorch 2.7.0, and CUDA 12.8, plus the gated DINOv3 weights requested separately from the official DINOv3 repo and placed under pretrained/dinov3/.Do the released checkpoints include everything needed to run the model?No — the DINOv3 encoder weights are gated and can't be redistributed, so they're excluded from the released checkpoints and must be requested and placed manually before evaluation, demo, or training.
No — the DINOv3 encoder weights are gated and can't be redistributed, so they're excluded from the released checkpoints and must be requested and placed manually before evaluation, demo, or training.Do I need a CUDA GPU to run this?Yes, the codebase is built against CUDA 12.8 and PyTorch 2.7.0, so a CUDA-capable GPU is required.
Yes, the codebase is built against CUDA 12.8 and PyTorch 2.7.0, so a CUDA-capable GPU is required.What model sizes are available?PointDiT-B, L, and H, each released at 256x256 and 512x512 resolutions, with all weights hosted on Hugging Face.
PointDiT-B, L, and H, each released at 256x256 and 512x512 resolutions, with all weights hosted on Hugging Face.How is the model trained?Training happens in two stages: 256x256 pre-training on SceneNet-RGBD, followed by 512x512 fine-tuning on an 11-dataset mixture.
Training happens in two stages: 256x256 pre-training on SceneNet-RGBD, followed by 512x512 fine-tuning on an 11-dataset mixture.Does it store point clouds on disk during training?No — the training loader reads RGB, depth, and camera intrinsics and back-projects to 3D on the fly, so point clouds are never materialized on disk.
No — the training loader reads RGB, depth, and camera intrinsics and back-projects to 3D on the fly, so point clouds are never materialized on disk.What are similar projects?marigold-v2, SolarWM, WorldSculpt, vggt, Mira-Scene, video-prediction-policy-2, tapnet, wuji-ego-mint
marigold-v2, SolarWM, WorldSculpt, vggt, Mira-Scene, video-prediction-policy-2, tapnet, wuji-ego-mintWhich topics is it in?vision-transformer, 3d-reconstruction, model-training, pretrained-models, diffusion-models, paper, dataset, evaluation, fine-tuning, 3d, transformer, academic-research
vision-transformer, 3d-reconstruction, model-training, pretrained-models, diffusion-models, paper, dataset, evaluation, fine-tuning, 3d, transformer, academic-research