kimodo.cpp README

The project's README from github.com/localai-org/kimodo.cpp. As one markdown file: README.md.

kimodo.cpp

GGML/C++ implementation of NVIDIA's Kimodo text-to-motion model.

Status

The five released Kimodo motion checkpoints accept either a UTF-8 prompt or a precomputed LLM2Vec embedding and generate local rotations plus root translations on CPU or Vulkan:

NVIDIA's Python API expands SOMA's predicted 30 joints to a relaxed-hand 77-joint presentation skeleton. The native API currently returns the 30 joints the model actually predicts. The text encoder uses eight-layer Vulkan chunks by default; set KIMODO_TEXT_LAYER_CHUNK=1..32 to tune VRAM use.

Included: checked GGUF loading, safetensors conversion, DDIM sampling, C/C++ APIs, conditioned multi-prompt transitions, CPU/Vulkan parity tests, skeleton-only GLB export, and a local text-to-motion demo. General constraint input, 77-joint SOMA expansion, skinned-mesh GLB export, and quantised models are not implemented yet.

Build and test on Linux

Install a C++23 compiler, CMake 3.25+, Ninja, Python 3 with the Hugging Face CLI (pip install huggingface_hub), and the Vulkan loader/headers for Vulkan support. GGML is a pinned Git submodule:

git submodule update --init --recursive
scripts/download_gguf_weights.sh --output "$PWD" --model soma-rp-v1.1
cmake --preset debug
cmake --build --preset debug
ctest --preset debug

The standard test suite requires the local motion GGUF, text bundle, and fixtures. It never downloads weights by itself. release, asan-ubsan, and fuzz presets are also available.

Nix is optional and provides these dependencies reproducibly:

nix develop path:. --command cmake --preset debug
nix develop path:. --command cmake --build --preset debug

For sanitizer work:

nix develop path:. --command cmake --preset asan-ubsan
nix develop path:. --command cmake --build --preset asan-ubsan
nix develop path:. --command env \
  LD_LIBRARY_PATH="$PWD/build/asan-ubsan/ggml/src:$PWD/build/asan-ubsan/ggml/src/ggml-vulkan:$LD_LIBRARY_PATH" \
  ASAN_OPTIONS=detect_leaks=0:abort_on_error=1 UBSAN_OPTIONS=print_stacktrace=1 \
  ctest --preset asan-ubsan --output-on-failure

Leak detection is disabled because Vulkan loader/driver allocations are global to the process. The GGUF parser fuzzer requires Clang.

API

include/kimodo/kimodo_capi.h is the C API. Model loading checks the motion GGUF and text bundle before inference. Use kimodo_generate_embedding for 4096 F32 values or kimodo_generate for text. Both return the selected model's root translations and local XYZW rotations; query the joint count from the result rather than assuming a fixed skeleton.

Demo

After building the debug preset and downloading the native GGUF bundle:

go run ./demo -addr 0.0.0.0:8094

Open http://localhost:8094. The left sidebar contains the prompt and a persistent history; choosing a previous animation restores its prompt for a new generation. Every successful animation also writes a standalone animation.glb beside its raw streams, for example demo-output/<animation-id>/animation.glb. It contains the selected animated node hierarchy (no mesh), ready to copy into a Three.js project. It is also available from /api/animations/<animation-id>/animation.glb while the demo is running.

Weights

Ready-to-run native GGML weights are published under the Hugging Face LocalAI-io organisation (not GitHub's localai-org). The reusable Llama-3-Kimodo-GGML text encoder is separate from the four redistributable motion repositories, each of which preserves a one-to-one relationship to its NVIDIA upstream:

Download one or repeat --model to install several:

scripts/download_gguf_weights.sh --output "$PWD" \
  --model soma-rp-v1.1 --model g1-rp-v1

The installer verifies each published manifest and SHA-256 hashes. Use --motion-only when supplying a precomputed 4096-float LLM2Vec embedding. SMPL-X RP is deliberately absent from the published-weight installer: its internal-R&D licence prohibits distributing derivative models, so it must be converted locally after the user obtains the upstream checkpoint under its gated terms.

The text bundle includes converted Meta Llama 3 material and retains its separate terms. Review every selected model card before downloading or redistributing.

License

The C++ port and its original tooling are licensed under Apache-2.0; see LICENSE. GGML and the model weights retain their respective licences.

Motion checkpoint Upstream terms Commercial use
Kimodo-SMPLX-RP-v1 NVIDIA Internal Scientific Research and Development Model License No; internal, non-production R&D only; derivative model redistribution is prohibited
SOMA RP/SEED v1.1 NVIDIA Open Model License Permitted by the model licence
G1 RP/SEED v1 NVIDIA Open Model License Permitted by the model licence

The SMPL-X warning is about NVIDIA's trained Kimodo checkpoint, not the mere fact that its output uses an SMPL-X-shaped 22-joint hierarchy. Converting that checkpoint to GGUF is a new runtime representation of the same weights and does not replace its licence. Skeleton names, parent links, and the Apache-2.0 port source do not by themselves make the SOMA or G1 checkpoints non-commercial. The SMPL-X Hugging Face metadata, model card, and access terms identify the internal-R&D licence; treat those restrictive terms as controlling even though an apparently inconsistent LICENSE file has also appeared in that upstream repository.

Regenerating the bundle

This is only needed to reproduce a conversion. The SMPL-X checkpoint and Llama base model are gated. After accepting their Hugging Face licences and authenticating, download the exact revisions and hash manifests with:

nix develop path:. --command hf auth login
scripts/download_weights.sh --output "$PWD/models" --with-text \
  --model smplx-rp-v1 --model soma-rp-v1.1 --model soma-seed-v1.1 \
  --model g1-rp-v1 --model g1-seed-v1

Convert the local LLM2Vec model to the native component bundle with:

nix develop path:. --command scripts/convert_llm2vec_bundle.sh \
  "$PWD/models/llama3-8b-instruct-base" "$PWD/generated/llm2vec-text-bundle"

Validate a prospective release without network access, then explicitly upload it from an account allowed to publish to LocalAI-io:

nix develop path:. --command python scripts/publish_gguf.py --component motion \
  --motion-model soma-rp-v1.1
nix develop path:. --command python scripts/publish_gguf.py --component motion \
  --motion-model soma-rp-v1.1 --upload --confirm-upstream-licences
nix develop path:. --command python scripts/publish_gguf.py --component text \
  --upload --confirm-upstream-licences

kimodo.cpp (localai-org/kimodo.cpp)

Searching Database…

913 ·GitHub
Kimodo.cpp
What is this?This is a C++/GGML port of NVIDIA's Kimodo text-to-motion model: you give it a natural-language prompt and it generates skeleton animation (joint rotations and root translations).
Is this a library or a ready-to-run app?It's a native C/C++ library with a C API, plus a model installer CLI and a local web demo that exports animations as GLB files.
How do I get it running?Pull the pinned GGML submodule, build with CMake/Ninja, then download one of the published GGML checkpoints with the model installer (repeat --model to grab several) before running the demo or API.
What do I need installed before building it?A C++23 compiler, CMake 3.25+, Ninja, Python 3 with the Hugging Face CLI, and the Vulkan loader/headers if you want Vulkan acceleration; Nix can supply all of this reproducibly.
Any catch before I use the SMPL-X model?SMPL-X RP is deliberately left out of the automatic installer because NVIDIA's internal R&D licence forbids redistributing derivative models, so you must convert it yourself after accepting the gated upstream terms. General constraint input, full 77-joint SOMA expansion, skinned-mesh GLB export, and quantised models also aren't implemented yet.
Does this run on Windows and macOS too?Build and test instructions in the README target Linux only.
What skeletons can it animate?Five checkpoints cover three skeleton families: SMPL-X RP (22 joints), SOMA RP/SEED (a 30-joint compact control skeleton, expandable to 77 joints in NVIDIA's Python API), and G1 RP/SEED (34 Unitree G1 robot joints).
What does the demo produce?Every successful generation writes a standalone animation.glb with the animated node hierarchy (no mesh) next to its raw output streams, ready to drop into a Three.js scene, and the file is also served over the demo's HTTP API.
Can I skip text and feed it a precomputed embedding?Yes — pass a precomputed 4096-float LLM2Vec embedding via the kimodo_generate_embedding C function or the installer's --motion-only flag instead of a raw text prompt.
Where do the model weights come from?Ready-to-run GGML weights for the four redistributable checkpoints are hosted on Hugging Face under the LocalAI-io organisation, separate from the reusable Llama-3-Kimodo-GGML text encoder.
What are similar projects?skin-tokens.cpp, framefields, UniMate, image-to-3dlab, TRELLIS, NoGraphicsAPI, llama.cpp, motion-bricks.cpp
Which topics is it in?text-to-3d, character-animation, motion, computer-graphics, cmake, vulkan, 3d, animation, nvidia, huggingface, cpp, c++
Creator
localai-org
Size
0.4 MB
Stars
913
Updated
CommitSep 17
Age
1mo
Commits · 90 Days