mixamo-llm-mocap README

The project's README from github.com/squall01337/mixamo-llm-mocap. As one markdown file: README.md.

Mixamo LLM Mocap

Turn any locked-camera video — filmed or AI-generated — into a clean FK animation on any Mixamo character. One performer, or two fighting each other. No mocap suit, no manual keyframing, and every stage scriptable enough that an AI agent can run the whole loop.

license blender gpu agent

showcase

Left: AI-generated source video. Right: the automatic retarget on a Mixamo character in Blender — 10 seconds, nine punches, a slip under and a high side kick, straight through the pipeline.

showcase — two fighters

Two performers, one plate, one pass. The left fighter throws four punches and a roundhouse; the right one blocks, folds over the body shot and ducks under the kick. Both tracks are split out of the same video by screen side, retargeted onto two Mixamo characters with different proportions — Y Bot and Ninja — and placed at the distance the performers actually stood, measured from the footage.

How it works

video plate (locked camera, T-pose bookends)
   │
   ├─ 1. estimate_pose_gvhmr.py    GVHMR (SMPL-X mesh recovery) → 33 landmarks + pelvis height
   ├─ 2. analyze_landmarks.py     numeric beat detection → you write a beat sheet from NUMBERS
   ├─ 3. action_specs/<name>.json  the motion as data: support schedule, rest blends, fists
   ├─ 4. lift_to_mixamo.py         direction-preserving retarget onto YOUR rig's proportions
   ├─ 5. apply_mixamo_fk.py        FK aim + foot planting, inside live Blender (via Blender MCP)
   ├─ 6. qa_clip.py                automated gate: no explosions, no pops, no foot skate
   ├─ 7. compare_reference.py      frame-by-frame vs the video → which windows still differ
   ├─ 8. compare_pair.py           two-character plates: separation, reach, intrusion
   ├─ 9. run_in_blender.py contact real mesh-vs-mesh collision between two characters
   └─ 10. render_preview.py        preview + side-by-side showcase video

With two performers in the plate, stages 1–7 run once per fighter (--person left|right splits the tracks), setup_duo.py builds one scene holding both characters, and compare_pair.py checks what only exists when there are two of them: whether they stand, reach and miss each other the way the performers did.

The estimator provides mesh-quality joints; the lift keeps its segment directions but rebuilds every position from your character's measured bone lengths; the apply plants feet by solving hip height (never IK — Mixamo rigs are FK-only); the spec contributes only what a video cannot know: which foot is the support in each phase (including "none" for airborne beats), when fists close, where the clip locks back to rest.

Why it's different

Quickstart

  1. Install — docs/INSTALL.md walks through every dependency (list below).

  2. Build your rig scene:

    blender --background --python pipeline\setup_rig.py -- --fbx ybot.fbx --out ybot_rest.blend

  3. Run a plate (Blender open on the scene; plate rules in docs/PROMPTING.md):

    tools\GVHMR\.venv\Scripts\python.exe pipeline\estimate_pose_gvhmr.py --video plates\<name>\<name>.mp4 --out plates\<name>\landmarks.json
tools\GVHMR\.venv\Scripts\python.exe pipeline\analyze_landmarks.py --landmarks plates\<name>\landmarks.json
# beat sheet → action_specs\<name>.json  (schema: docs/PIPELINE.md)
tools\GVHMR\.venv\Scripts\python.exe pipeline\lift_to_mixamo.py --spec action_specs\<name>.json
python pipeline\run_in_blender.py all action_specs\<name>.json
tools\GVHMR\.venv\Scripts\python.exe pipeline\qa_clip.py --spec action_specs\<name>.json
tools\GVHMR\.venv\Scripts\python.exe pipeline\compare_reference.py --spec action_specs\<name>.json
tools\GVHMR\.venv\Scripts\python.exe pipeline\render_preview.py action_specs\<name>.json --showcase

    compare_reference.py tells you which frame windows still differ from the video; the last command produces preview.mp4 and the side-by-side showcase.mp4 — the same format as the demo GIF above.

    Two-performer plates add --person left|right to the estimate, one spec per fighter, and a compare_pair.py run — see docs/PIPELINE.md section 9.

  4. Iterate with docs/PIPELINE.md and docs/PITFALLS.md.

What you need to bring (and where to get it)

What Where Notes
A Mixamo character — any model mixamo.com → Characters → download FBX Binary, T-pose Adobe's terms don't allow redistributing them; setup_rig.py builds and validates the scene from your download
Blender 5.1+ blender.org
Blender MCP add-on (official, Blender Lab) blender.org/lab/mcp-server enable Allow Online Access; the apply talks to its socket
GVHMR (the pose estimator — not in this repo) github.com/zju3dv/GVHMR clone into tools/GVHMR; install per docs/INSTALL.md — including a working Windows recipe (docs/requirements_gvhmr_windows.txt + prebuilt pytorch3d wheel)
GVHMR checkpoints (~5 GB) HuggingFace mirror exact curl commands in docs/INSTALL.md
SMPL-X body model smpl-x.is.tue.mpg.de free research registration → download SMPL-X v1.1, place SMPLX_NEUTRAL.npz as shown in docs/INSTALL.md
GPU ~8 GB VRAM developed on an RTX 4080

Docs

Doc What it covers
docs/INSTALL.md Every dependency, step by step, Windows-proven
docs/PIPELINE.md The operational loop + the action_spec schema, field by field
docs/RIG.md Mixamo rig conventions: spaces, units, the rules that must never break
docs/PITFALLS.md Every mistake this pipeline's development paid for, so you don't pay twice
docs/PROMPTING.md Writing gen-video plate prompts that survive retargeting

License

MIT — see LICENSE, including third-party notes (Mixamo, GVHMR, SMPL-X, Blender MCP).

mixamo-llm-mocap (squall01337/mixamo-llm-mocap)

blender
Searching Database…

378 ·GitHub
Mixamo-llm-mocap
What is this?Mixamo LLM Mocap turns any locked-camera video - filmed or AI-generated - into a clean FK animation on any Mixamo character, using a GVHMR pose estimator, a spec-driven retarget step, and a Blender MCP apply stage. It's built so an AI agent can run the entire pipeline end to end.
Is this a Blender add-on, or a set of command-line scripts?It's a chain of Python command-line scripts for pose estimation, retargeting and QA, with a socket connection into Blender via the official MCP add-on for the final apply step - there's nothing to install as a click-and-use Blender plugin.
How do I run a full plate through the pipeline?Install the dependencies per docs/INSTALL.md, run setuprig.py to build a rig scene from your own Mixamo character download, then run the pipeline stages with Blender open on that scene; comparereference.py checks the retarget against the source video and the final command produces preview.mp4 and a side-by-side showcase.mp4.
What do I need to install besides this repo?Blender 5.1+ with the official MCP add-on, the GVHMR pose estimator (cloned separately and installed per docs/INSTALL.md), its roughly 5GB of checkpoints, and the SMPL-X body model obtained through a free research registration.
Any catches before I start?Mixamo characters aren't bundled - Adobe's terms forbid redistributing them, so you have to download your own FBX from mixamo.com, and GVHMR plus its checkpoints have to be fetched and set up separately.
Does the install work on Windows, or only Linux/macOS?Windows is explicitly supported - docs/INSTALL.md includes a working Windows recipe for GVHMR with a dedicated requirements file and a prebuilt pytorch3d wheel.
Do I need a beefy GPU for the pose estimation?Yes - the GVHMR estimator needs roughly 8GB of VRAM; the project was developed and tested on an RTX 4080.
Does it only work with one performer per video?No - with two performers in frame, setupduo.py and comparepair.py split the tracks by screen side, retarget each onto a separate Mixamo character, and verify strike reach, separation and limb collisions frame by frame against the footage.
What defines a new motion for the pipeline?A small JSON 'actionspec' that declares which foot supports each phase (including airborne beats), when fists close, and where the clip locks back to rest - no animation code is needed.
How does it catch bad retargets before they ship?A numeric QA gate flags exploded bones, hip pops, foot skate and broken rest poses, and comparereference.py measures things like hand height and limb distance against the source video, reporting exactly which frame windows diverge.
Does it work on any Mixamo character, or only specific ones?Any Mixamo character - setuprig.py builds and measures a scene from your own Mixamo FBX download (rest pose, bone lengths, hip and ground heights), and every later stage reads that measured profile.
What are similar projects?Motion2MixamoRig, freemocap, image-to-3dlab, openpose, framefields, InsertAny3D, kimodo, storyboard-ai
Which topics is it in?motion-capture, pose-estimation, blender, video-processing, game-development, 3d, animation, nvidia, computer-vision, cuda, ai-agent, gpu
Creator
squall01337
Size
11.4 MB
Stars
378
Updated
CommitSep 26
Age
2mo
Commits · 90 Days