WorldSculpt README

The project's README from github.com/AlayaLab/WorldSculpt. As one markdown file: README.md.

WorldSculpt: Generating Compositional Worlds from Grounded Videos

Muyao Niu1,2   Jixuan He1   Ruihan Yu1   Lian Fu1   Yonghao Yu1   Zheng-Hui Huang1   Yifan Zhan1   Fengbo Lan1  
Yongtao Ge1   Yinqiang Zheng2   Kaipeng Zhang1,✉   Zhixiang Wang1,✉  
1 Alaya Lab   2 The University of Tokyo   ✉ Corresponding Authors  
Teaser image of WorldSculpt
Given RGB images with instance masks and 3D boxes, WorldSculpt produces a compositional mesh representation for very complex scenes consisting of hundreds of individual objects.

TL;DR

🧭 Pipeline

WorldSculpt pipeline

✨ News

🚀 Getting Started

Installation

Step 1: Following TRELLIS.2 Installation

Please first follow the installation guide of TRELLIS.2 to set up the base environment.

Step 2: Installing Additional Dependencies

NATTEN_CUDA_ARCH is the compute capability of your GPU. Print yours with

python -c "import torch; print('%d.%d' % torch.cuda.get_device_capability())"
# H100; replace 9.0 with your own compute capability
NATTEN_CUDA_ARCH="9.0" NATTEN_N_WORKERS=8 pip install natten==0.21.0 --no-build-isolation
pip install https://github.com/LDYang694/Storages/releases/download/20260430/utils3d-0.0.2-py3-none-any.whl
pip install peft pillow imageio imageio-ffmpeg tqdm easydict opencv-python-headless trimesh transformers==4.57.1 zstandard kornia timm diffusers accelerate gradio plyfile matplotlib scikit-image scikit-learn fpsample iopath pycocotools ftfy
pip install "setuptools<81"

Step 3: Downloading Pretrained Checkpoints

hf download AlayaLab/WorldSculpt --local-dir ./pretrained
hf download TencentARC/Pixal3D --local-dir ./pretrained/Pixal3D

Step 4: Downloading Data

# UE-MeshyScene
hf download AlayaLab/WorldSculpt_data --include "UE-MeshyScene/*" --local-dir ./input --repo-type=dataset
cd ./input/UE-MeshyScene
tar xvf scene*.tar
cd ../..

# Marble DEMO data
hf download AlayaLab/WorldSculpt_data --include "Marble/*" --local-dir ./input --repo-type=dataset
cd ./input/Marble
tar xvf Marble.tar.gz
cd ../..

Step 5 🚀🚀: Running Inference

Every option is passed explicitly — the script has no defaults, so a run is fully described by its own command line. Step 4 extracts each dataset into its own subdirectory, so INPUT_ROOT points at that subdirectory, not at ./input (the script reads $INPUT_ROOT/<scene>/transforms.json).

# UE-MeshyScene, scene_00001
INPUT_ROOT=./input/UE-MeshyScene OUTPUT_ROOT=./output CKPT_ROOT=./pretrained \
SAMPLER=official SS_STEP=15000 SHAPE_STEP=15000 \
RENDER=1 FACE_BUDGET=1000000 GPU=0 \
./inference.sh scene_00001

# Marble demo data, marble_serene_living_room_countryside_view
INPUT_ROOT=./input/Marble OUTPUT_ROOT=./output CKPT_ROOT=./pretrained \
SAMPLER=official SS_STEP=15000 SHAPE_STEP=15000 \
RENDER=1 FACE_BUDGET=1000000 GPU=0 \
./inference.sh marble_serene_living_room_countryside_view

🙏 Acknowledgements

We thank Pixal3D, TRELLIS.2, and DINOv3 for their wonderful work and open-source repositories.

📖 Citation

If you find this work useful, please cite:


@misc{niu2026worldsculptgeneratingcompositionalworlds,
      title={WorldSculpt: Generating Compositional Worlds from Grounded Videos}, 
      author={Muyao Niu and Jixuan He and Ruihan Yu and Lian Fu and Yonghao Yu and Zheng-Hui Huang and Yifan Zhan and Fengbo Lan and Yongtao Ge and Yinqiang Zheng and Kaipeng Zhang and Zhixiang Wang},
      year={2026},
      eprint={2609.05416},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2609.05416}, 
}

WorldSculpt (AlayaLab/WorldSculpt)

Searching Database…

这是什么东西?WorldSculpt 是一种研究方法的官方实现,该方法能把真实拍摄的 Grounded 视频转成 3D 场景:从带实例掩码和 3D 包围盒的 RGB 图像出发,为包含数百个独立物体的非常复杂的场景构建组合式网格(mesh)。
这是个拿来就能跑的应用,还是研究代码?是研究代码,不是应用:它是一篇论文的官方实现。2026 年 9 月发布的版本包含 Python 推理脚本、HuggingFace 上的预训练权重、arXiv 论文和项目主页——所有操作都在命令行里完成。
我该怎么把它跑起来?五步:先安装 TRELLIS.2 的基础环境,然后在把 NATTENCUDAARCH 设为你显卡的计算能力后安装额外依赖,接着下载预训练权重和数据,最后运行推理脚本。
安装之前我需要准备什么?一块支持 CUDA 的 NVIDIA 显卡——你需要把 NATTENCUDAARCH 设成你显卡的计算能力——此外还要先装好 TRELLIS.2 环境作为基础,再装上 NATTEN 之类的额外依赖。
跑推理时有什么要注意的坑吗?推理脚本没有任何默认值:每个选项都必须在命令行里显式传入。另外,INPUTROOT 必须指向解压后包含 transforms.json 的数据集子目录,而不是 ./input。
它是怎么处理有几百个物体的场景的?核心思路是把为单个物体训练的生成先验复用到整个场景上:作者对 Pixal3D 做了微调,让它能接受被遮挡的多视角输入,然后用它对非常复杂的场景进行组合式网格化。
我可以把 Marble 世界喂给它吗?可以——论文展示的一个应用就是在 Marble 3DGS 世界上运行同一套流水线,把它转换成物体级别的组合式网格。
UE-MeshyScene 基准测试是什么?这是随论文一起发布的数据集,提供了逐帧、逐实例的标注,用于在复杂场景上评估组合式网格重建的效果。
我能自己训练或微调这个模型吗?README 只涉及推理:2026 年 9 月发布的版本只带了推理代码和预训练权重,没有介绍任何训练流程。
Creator
AlayaLab
Size
8.8 MB
Stars
345
Updated
CommitSep 16
Age
1mo
Commits · 90 Days