183 ·GitHub
What is this?
InsertAny3D is a pipeline that combines a Unity scene, a generative 3D model, and multi-view geometry alignment to insert a new object into a Unity scene as a Gaussian Splatting asset, estimating its position, rotation and scale in Unity world coordinates.
InsertAny3D is a pipeline that combines a Unity scene, a generative 3D model, and multi-view geometry alignment to insert a new object into a Unity scene as a Gaussian Splatting asset, estimating its position, rotation and scale in Unity world coordinates.Is this a Unity plugin, or just the server-side pipeline?It's only the server side: Python code that takes Unity's multi-view renders and an edited image, then produces a Gaussian Splatting object and pose. The Unity scripts, editor setup and Unity project itself are not included and must be supplied by you.
It's only the server side: Python code that takes Unity's multi-view renders and an edited image, then produces a Gaussian Splatting object and pose. The Unity scripts, editor setup and Unity project itself are not included and must be supplied by you.How do I get it running?Clone the repo from GitHub, or grab a ModelScope snapshot if you don't need Git, then follow the full install order in install.md. Once set up, run a single task with tools/run_insert_pipeline.py or a batch of jobs with tools/run_insert_batch.py.
Clone the repo from GitHub, or grab a ModelScope snapshot if you don't need Git, then follow the full install order in install.md. Once set up, run a single task with tools/run_insert_pipeline.py or a batch of jobs with tools/run_insert_batch.py.What do I need besides the server code - a Unity project, a GPU?You need your own Unity project to supply the task manifest (left/center/right views, depth, camera params and prompts), plus a CUDA GPU. The default 3D provider is TRELLIS, with SAM3D Objects and Hunyuan3D-2 as alternatives; Hunyuan3D-2 runs in its own separate CUDA/PyTorch environment.
You need your own Unity project to supply the task manifest (left/center/right views, depth, camera params and prompts), plus a CUDA GPU. The default 3D provider is TRELLIS, with SAM3D Objects and Hunyuan3D-2 as alternatives; Hunyuan3D-2 runs in its own separate CUDA/PyTorch environment.What's not included here, and what can go wrong?The Unity project and Unity-side scripts are not in this repo and must be supplied by the caller; model weights, Python venvs and scene/run data are also kept out of Git. End-to-end quality still depends on GPU, CUDA, the upstream model services and input data quality.
The Unity project and Unity-side scripts are not in this repo and must be supplied by the caller; model weights, Python venvs and scene/run data are also kept out of Git. End-to-end quality still depends on GPU, CUDA, the upstream model services and input data quality.Does this require a GPU to run?Yes, it is GPU/CUDA dependent end to end: Gaussian generation, GIM matching and pose estimation all rely on GPU-backed models, and Hunyuan3D-2 even needs its own dedicated CUDA/PyTorch environment.
Yes, it is GPU/CUDA dependent end to end: Gaussian generation, GIM matching and pose estimation all rely on GPU-backed models, and Hunyuan3D-2 even needs its own dedicated CUDA/PyTorch environment.Which output file should Unity actually import?Unity should import 06_sags/inserted_object.ply, the object extracted from the combined anchor+object Gaussian. The earlier 02_trellis/sample.ply still contains the anchor and would duplicate the original scene object if imported directly.
Unity should import 06_sags/inserted_object.ply, the object extracted from the combined anchor+object Gaussian. The earlier 02_trellis/sample.ply still contains the anchor and would duplicate the original scene object if imported directly.How does the pipeline figure out where to place the generated object?It renders novel views of the generated object, matches them to the original Unity views with GIM, back-projects depth, and computes a similarity transform (position, rotation, scale) from generated world to Unity world coordinates, writing the result to pose.json for Unity to apply.
It renders novel views of the generated object, matches them to the original Unity views with GIM, back-projects depth, and computes a similarity transform (position, rotation, scale) from generated world to Unity world coordinates, writing the result to pose.json for Unity to apply.Why doesn't an unrecognized 3D provider just fall back to TRELLIS?An unknown provider fails at the preflight stage instead of silently falling back, so a misconfigured job stops early with a clear error rather than producing unexpected results.
An unknown provider fails at the preflight stage instead of silently falling back, so a misconfigured job stops early with a clear error rather than producing unexpected results.Why are model weights and Unity projects kept out of Git?To keep the repo small and reproducible: weights are fetched by install scripts, and Unity projects, scene data and run outputs are treated as per-task data rather than code, with third-party sources rebuilt from pinned commits and patches.
To keep the repo small and reproducible: weights are fetched by install scripts, and Unity projects, scene data and run outputs are treated as per-task data rather than code, with third-party sources rebuilt from pinned commits and patches.What are similar projects?ml-sharp, WorldSculpt, Mira-Scene, vggt, nerfstudio, wuji-ego-mint, gaussian-splatting, pointdit
ml-sharp, WorldSculpt, Mira-Scene, vggt, nerfstudio, wuji-ego-mint, gaussian-splatting, pointditWhich topics is it in?3d-gaussian-splatting, gaussian-splatting, pose-estimation, unity, 3d-reconstruction, segmentation, game-development, 3d, academic-research, generative-ai, computer-vision, cuda
3d-gaussian-splatting, gaussian-splatting, pose-estimation, unity, 3d-reconstruction, segmentation, game-development, 3d, academic-research, generative-ai, computer-vision, cuda