vggt-omega README

The project's README from github.com/facebookresearch/vggt-omega. As one markdown file: README.md.

VGGT-Ω

Project Page arXiv

Jianyuan Wang1,2 Minghao Chen1 Shangzhan Zhang1 Nikita Karaev1
Johannes Schönberger2 Patrick Labatut2 Piotr Bojanowski2 David Novotny
Andrea Vedaldi1,2 Christian Rupprecht1

1Visual Geometry Group, University of Oxford; 2Meta AI

Updates

Pretrained models

Before using the models, please request access to the checkpoints here. Once your request is approved, you can download the checkpoints. Please note that access requests are reviewed by an automated process based on the information provided in the request. The authors are not involved in the review process and cannot approve or reject individual applications. However, the 🤗 Hugging Face demo is available to everyone.

Model Resolution Text-aligned Checkpoint Date Note
VGGT-Omega-1B-512 512 No Download May 2026 Recommended for in-the-wild applications, no benchmarking.
VGGT-Omega-1B-416-Reproduction 416 No Download Sep 2026 Retrained for benchmarking, as detailed in reproduction.md.
VGGT-Omega-1B-256-Text-Alignment 256 Yes Download May 2026

Quick Start

First, clone this repository and install the dependencies:

git clone [email protected]:facebookresearch/vggt-omega.git
cd vggt-omega
pip install -r requirements.txt
pip install -e .

Now, try the model with a few lines of code:

import torch

from vggt_omega.models import VGGTOmega
from vggt_omega.utils.load_fn import load_and_preprocess_images
from vggt_omega.utils.pose_enc import encoding_to_camera

checkpoint_path = "path/to/vggt_omega_1b_512.pt"
image_names = ["path/to/imageA.png", "path/to/imageB.png", "path/to/imageC.png"]

model = VGGTOmega().to("cuda").eval()
model.load_state_dict(torch.load(checkpoint_path, map_location="cpu"))

images = load_and_preprocess_images(image_names, image_resolution=512).to("cuda")

with torch.inference_mode():
    predictions = model(images)

extrinsics, intrinsics = encoding_to_camera(
    predictions["pose_enc"],
    predictions["images"].shape[-2:],
)

depth = predictions["depth"]
depth_conf = predictions["depth_conf"]
camera_and_register_tokens = predictions["camera_and_register_tokens"]
camera_tokens = camera_and_register_tokens[:, :, :1]
registers = camera_and_register_tokens[:, :, 1:]

For the text-aligned checkpoint, use VGGTOmega(enable_alignment=True) with image_resolution=256 and read predictions["text_alignment_embedding"].

Interactive Demo

Install the demo dependencies:

pip install -r requirements_demo.txt

Launch the Gradio demo with a local checkpoint path:

python demo_gradio.py \
  --checkpoint checkpoints/VGGT-Omega-1B-512/model.pt \
  --image-resolution 512

The demo accepts uploaded images or a video, runs camera and depth inference, and visualizes the depth-unprojected point cloud and predicted cameras as a GLB scene.

Runtime and GPU Memory

We benchmark the end-to-end peak GPU memory usage of VGGT-Omega-1B-512 on a single NVIDIA A100 GPU with 624x416 input images. The measurement covers the full inference program, from loading the model weights onto the GPU through the forward pass, so it includes both the memory needed to store the model itself and the memory used by inference activations and buffers. In other words, a GPU with at least the listed available memory is able to run the corresponding number of input frames under this setup.

Input Frames 1 10 25 50 100 200 300 400 500
Peak Memory (GB) 6.02 6.67 7.80 9.66 13.37 20.82 28.26 35.71 43.15

The benchmark uses load_and_preprocess_images with the default mode="balanced" and image_resolution=512. For these roughly 3:2 landscape images, this produces 624x416 inputs. You can set mode="max_size" to resize the longest side to 512 instead; for the same aspect ratio, this gives about 512x336 inputs and uses less GPU memory.

License

See the LICENSE file for details about the license under which this code is made available.

[^release]: This Release is intended to support the open source research community.

@misc{wang2026vggtomega,
      title={VGGT-$\Omega$},
      author={Jianyuan Wang and Minghao Chen and Shangzhan Zhang and Nikita Karaev and Johannes Schönberger and Patrick Labatut and Piotr Bojanowski and David Novotny and Andrea Vedaldi and Christian Rupprecht},
      year={2026},
      eprint={2605.15195},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2605.15195},
}

vggt-omega (facebookresearch/vggt-omega)

Searching Database…

4.7K ·GitHub
Vggt-omega
这是什么？VGGT-Omega 是来自牛津大学视觉几何组（Visual Geometry Group）和 Meta AI 的研究模型，可以从一组图像或一段视频中重建三维几何、相机位姿和深度，输出点云和相机预测。
这是我要导入的库，还是独立应用？它是一个在代码中调用的 Python 模型/库（导入模型类并运行推理），另有一个可选的 Gradio 网页演示用于交互式使用。
怎么让它跑起来？克隆仓库并安装依赖，然后加载模型并对你的图像或视频运行；要交互式使用，安装演示依赖，并启动指向本地检查点的 Gradio 演示。
使用预训练模型之前需要先申请什么吗？是的，你需要申请检查点的访问权限，获得批准后才能下载；访问审核是自动进行的，不由作者手动处理。
不自己下载检查点，我能直接试用吗？可以，Hugging Face 上的 Gradio 演示对所有人开放，即使不走检查点访问申请也能使用。
需要多少 GPU 显存？在单张 A100 上，1B-512 模型处理 1 帧约需 6GB，500 帧输入时峰值显存约升至 43GB；用 mode='maxsize' 代替默认的 'balanced' 缩放方式可以减少内存占用。
有哪些预训练检查点可用？三个变体：VGGT-Omega-1B-512（推荐用于真实场景使用）、用于基准测试的 416 分辨率复现检查点，以及 256 分辨率、还会额外输出文本对齐嵌入的文本对齐检查点。
Gradio 演示实际展示了什么？它接受上传的图像或视频，运行相机和深度推理，并把深度反投影得到的点云连同预测的相机可视化为 GLB 三维场景。
除了推理，还包含训练代码吗？是的，作者发布了训练代码，以及用于准备训练数据的工具（数据集收集、转换、清洗和智能体辅助的视觉审查）和一条监督式几何过滤流水线。
为什么检查点会有两个不同的版本？发布由额外一次训练得到的第二个检查点，是为了验证可复现性，并回应针对原始检查点的一个潜在疑虑；现在应以它作为基准比较的参考。
What are similar projects?vggt, lingbot-map, ml-sharp, nerfstudio, flex-pi, pointdit, Depth-Anything-3, wuji-ego-mint
Which topics is it in?point-cloud, 3d-reconstruction, gradio, multimodal, 3d, academic-research, inference, nvidia, huggingface, computer-vision, cuda, pytorch
Creator
facebookresearch
Size
42 MB
Stars
4.7K
Updated
CommitSep 22
Age
5mo
Commits · 90 Days