idol-sdk README

项目的 README,来自 github.com/speridlabs/idol-sdk。合并为一个 markdown 文件:README.md。

Idol

A single image to 3DGS object model

Idol turns one picture of an object into a 3D Gaussian splat. Give it a render, a generated image or a photo: it removes the background, frames the object and generates up to 1,048,576 Gaussians, including the sides the picture never showed. It runs on consumer GPUs, using 7.2 GB of VRAM and can run in under 2 seconds.

Hugging Face Hugging Face Spaces Python 3.10+

Each pair: the input image and the generated object turning. The demo has 93 more examples ready to run and takes your own images as well.

Model

Output 3D Gaussians: 32,768 to 1,048,576 (default 262,144), standard 3DGS .ply
DiT 370M parameters
Image conditioning DINOv3 ViT-H+ and FLUX.2 VAE
Decoder 288M parameters
Background removal BiRefNet
Sampler Euler, 50 steps, classifier-free guidance 3.0, time shift 3.0

Install

pip install -e .              # Python >= 3.10, PyTorch >= 2.5
pip install -e ".[demo]"      # + Gradio, to run the demo locally

The weights are downloaded from Hugging Face on first use, you need to request access to facebook/dinov3-vith16plus-pretrain-lvd1689m and black-forest-labs/FLUX.2-dev first, then log in with hf auth login.

flash-attn, if installed, is used for attention; otherwise PyTorch's scaled_dot_product_attention is (same result, slower). Set ATTN_BACKEND=sdpa or ATTN_BACKEND=flash_attn to choose.

Inference

python -m idol.sample --image photo.jpg --mode real
python -m idol.sample --image renders/ --mode synthetic --out outputs

For each image this writes <name>.ply (open it in any Gaussian splat viewer), <name>_input.png (the image the model saw, after background removal and cropping) and <name>_camera.json (the predicted camera of the input).

Pick the mode for your input:

--mode for
synthetic (default) computer generated images
real real world photographs

Other flags:

flag default
--seed 42 same seed, same object
--steps 50 sampling steps
--cfg_strength 3.0 classifier-free guidance
--rescale_t 3.0 time shift: higher spends more steps at high noise
--num_gaussians 262144 32,768 to 1,048,576, a multiple of 32
--dinov3_dtype bf16 fp32 runs the DINOv3 encoder in full precision (1.7 GB more memory)
--weights speridlabs/idol a Hugging Face repo or a local folder
--device best available cuda, mps or cpu (default: CUDA, then MPS, then CPU; or IDOL_DEVICE)
--no_optimize optimized turn off the faster sampling (see below)

Faster sampling is on by default on CUDA The first image in a process is slower while it pytorch compiles; the rest run at full speed.

Python API

from idol import IdolPipeline

pipeline = IdolPipeline.from_pretrained("speridlabs/idol")
output = pipeline("photo.jpg", mode="real", seed=42, num_gaussians=1_048_576)
output.gaussian.save_ply("photo.ply")
output.preprocessed_image.save("photo_input.png")

The image encoders come from the Hugging Face repos named in the weights' config.json. image_encoders takes them from somewhere else, by name: another repo id, a local folder in the same layout (FLUX.2's with its autoencoder under vae/), or a model you have already loaded. Only repo ids are checked for gated access, so local copies load offline without a token:

pipeline = IdolPipeline.from_pretrained("speridlabs/idol", image_encoders={
    "dinov3": "/models/dinov3-vith16plus-pretrain-lvd1689m",  # or a loaded transformers DINOv3 model
    "flux2_vae": "/models/FLUX.2-klein-4B",                   # any FLUX.2 repo with the same autoencoder in vae/
})

Demo

The Gradio app behind the Hugging Face Space is in demo/, with the example images. It runs on ZeroGPU there and on any local GPU:

pip install -e ".[demo]"
cd demo && python app.py

More details about the model

Component Parameters Precision on GPU Weights in VRAM
DiT (denoiser) 370.6M FP16 transformer blocks; FP32 embeddings and norms ~0.8–1.0 GB
Octree decoder 50.8M FP16 transformer blocks; FP32 embeddings and norms ~0.1–0.2 GB
Gaussian decoder 237.3M FP16 transformer blocks; FP32 embeddings and norms ~0.5–0.9 GB
BiRefNet (background removal) 220.7M FP16 ~0.44 GB
DINOv3 ViT-H+/16 840.6M bf16 ~1.68 GB
FLUX.2 VAE 84.1M bf16 (FP32 checkpoint, cast on load) ~0.17 GB
Full request ~1.80B mostly FP16 / bf16, some FP32 ~3.7–4.4 GB of weights, 7.2 GB peak

Benchmark

Idol, TripoSplat, TRELLIS and LGM, scored at the input view (PSNR, SSIM, LPIPS, IoU) and at seven new views (DINOv2, CLIP, sharpness distance). Mean scores, 1 × H100 per model. Bold is the best in each column.

Model Setting PSNR ↑ SSIM ↑ LPIPS ↓ IoU ↑ DINOv2 ↑ CLIP ↑ Sharpness distance ↓ Request time ↓
Idol 50 steps, 262,144 Gaussians 12.25 0.726 0.213 0.862 0.781 0.885 0.81 6.8 s
TripoSplat 50 steps, 262,144 Gaussians 11.76 0.712 0.225 0.860 0.743 0.868 1.03 9.9 s
TRELLIS Paper setting, 50 + 50 steps 11.89 0.716 0.235 0.845 0.754 0.868 0.97 7.1 s
LGM Default, 65,536 Gaussians 12.46* 0.722 0.237 0.834 0.575 0.784 1.22 1.1 s

* LGM has the highest PSNR despite visibly blurry results, because PSNR averages squared pixel error.

Across GPUs

One image to 262,144 Gaussians in 3.4 s. Request time at 50 steps: median over 8 images, with the same code and settings on every GPU, one GPU per machine.

GPU VRAM Request time
B300 269 GB 3.4 s
B200 179 GB 3.6 s
H200 140 GB 6.6 s
H100 80 GB 6.7 s
RTX 5090 32 GB 14.1 s
A100 80 GB 16.3 s
L40S 45 GB 16.6 s
RTX 4090 24 GB 19.1 s
RTX 3080 10 GB 46.2 s
L4 22 GB 54.6 s

Apple Silicon (MPS)

Picked automatically on a Mac, or with --device mps. Sampling is eager (the fast path is CUDA-only). On an M4 MacBook Air: ~4 min for 10 steps and 65,536 Gaussians, ~18 min for the default 50 steps and 262,144 (needs 24 GB).

Acknowledgements

Idol builds on TripoSplat. It uses DINOv3, the FLUX.2 VAE and BiRefNet.

Made with ❤️ by speridlabs.com

idol-sdk (speridlabs/idol-sdk)

正在搜索数据库…

这是什么?Idol 是一个推理 SDK,它能把一张物体的单张图片转换成 3D 高斯溅射(.ply 文件),同时去除背景、调整物体取景,并生成最多 1,048,576 个高斯点,包括照片中没有拍到的侧面。
这是命令行工具还是我可以导入的库?两者都是:它是一个 Python SDK,可以通过 Python API 调用,也附带一个可以直接运行推理的 CLI,并且包含一个可选的 Gradio 演示应用。
我该怎么让它运行起来?首次使用时权重会从 Hugging Face 下载,但你必须先申请访问受限的 DINOv3 和 FLUX.2 模型仓库,并用 'hf auth login' 登录;然后运行 CLI,并为你的输入图片选择 synthetic 或 real 模式。
它处理真实照片和合成渲染图的方式一样吗?不一样,你需要选择正确的 --mode 参数:synthetic(默认值)针对计算机生成的图像进行了调优,real 则适用于真实世界拍摄的照片。
它和 TRELLIS 或 LGM 这类图生 3D 模型相比如何?在论文的基准测试中,Idol 在大多数感知和多视角一致性指标(DINOv2、CLIP、清晰度)上优于 TripoSplat、TRELLIS 和 LGM,并且在 H100 上以 50 步运行时速度更快;不过 LGM 的原始 PSNR 更高,尽管其输出明显更模糊。
究竟是哪种模型架构在生成高斯点?由一个 370M 参数的 DiT 去噪器生成,它以 DINOv3 ViT-H+ 图像特征和 FLUX.2 VAE 作为条件输入,并配合一个 288M 参数的高斯解码器以及用于去除背景的 BiRefNet,采样时使用 50 步 Euler 步进。
不同 GPU 上的推理速度如何?处理一张图片、生成 262,144 个高斯点的请求耗时,从 B300 上的 3.4 秒到 L4 上的约 54.6 秒不等,具体取决于可用显存和计算能力。
我能在细节和速度之间做取舍吗?可以。--num-gaussians 可以设置在 32,768 到 1,048,576 之间(必须是 32 的倍数,默认值为 262,144),而 --steps、--cfg-strength 和 --rescale-t 则可以进一步调整采样质量与速度的平衡。
大小
6.7 MB
星标
1.2K
更新
提交Nov 09
时长
21天
提交 · 90 天