CodeVideoModel README

The project's README from github.com/code-video-model/CodeVideoModel. As one markdown file: README.md.

Code Video Model

Code Video Model Team

Tech Report Project Page Hugging Face GitHub Code Code License: Apache 2.0 Examples License: CC BY 4.0

Demo Video

https://github.com/user-attachments/assets/d1a8594d-14fa-49aa-a101-03da77ed585f

Installation

Use Ubuntu 22.04 and a CUDA 12.6-compatible NVIDIA driver for MiniMax-H3.

git clone https://github.com/code-video-model/CodeVideoModel.git
cd CodeVideoModel

conda create -n cvm python=3.10 -y
conda activate cvm
conda install -c conda-forge "ffmpeg[version='>=6',build='gpl_*']" -y
python -m pip install --upgrade pip
python -m pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 \
  --index-url https://download.pytorch.org/whl/cu126
python -m pip install -r requirements.txt
python -m playwright install chromium

The Three.js stage uses GPT-6. For GPT-Image-2, set OPENROUTER_API_KEY securely in your environment.

Checkpoint

Download the MiniMax-H3 checkpoint:

python download_h3.py --output models/MiniMax-H3-ref2va

Quick Start

Each demo bundles a prompt, a reference image, and a Three.js video, ready for direct generation with MiniMax-H3.

Example Directory under examples/
Sports Car sports_car
Tank Boarding tank_boarding
Loft Tour loft_tour
Robot Navigation robot_navigation

Set example to any directory name above:

example=tank_boarding
python -m inference.generate_video \
  --code-video "examples/${example}/code_video.mp4" \
  --reference-image "examples/${example}/reference.png" \
  --prompt-file "examples/${example}/prompt.txt" \
  --model-root models/MiniMax-H3-ref2va \
  --keep-steps 5 \
  --output "outputs/${example}.mp4"

Inference

1. Text → Three.js Codebase

Generate an editable scene from a text prompt. GPT-6 builds the codebase and refines it through direct visual review:

python -m inference.generate_threejs \
  --prompt "A red car follows a curved road through a sunlit town, with a smooth tracking camera." \
  --output outputs/threejs

The generated scene files are saved in the output workspace's case directory. Prompts can be entered directly or read from a text file.

Export the video:

python -m inference.render video --scene outputs/threejs

View the scene in your browser:

python -m inference.viewer --scene outputs/threejs --port 8000

Open the local URL printed in the terminal. Play or scrub the timeline, step through frames, and select a source file to read its code. In the World Inspector, drag to orbit, Shift-drag to pan, and scroll to zoom; toggle the camera frustum, trajectory, and grid.

2. Three.js Video → Reference Image

Strict alignment

Extract the video's first frame and use GPT-Image-2 to create a 1536×864 reference, with instructions to preserve the scene's composition and content:

python -m inference.generate_reference \
  --video outputs/threejs/case/reconstruction.mp4 \
  --output outputs/threejs/reference.png

You can also provide an already extracted frame instead of a video.

Text-guided reference

Use the first frame as loose visual context and a text prompt to guide the result, without requiring the same camera, composition, or object placement:

python -m inference.generate_reference_guided \
  --video examples/loft_tour/code_video.mp4 \
  --prompt "A sunlit modern loft with a leather sofa and floating stairs, photographed at eye level." \
  --output outputs/loft_reference_guided.png

This mode also accepts an extracted frame and a scene description loaded from a text file. It produces a 1536×864 reference image.

3. Three.js Video + Reference Image → MiniMax-H3 Video

Generate a video using the proxy, reference image, and original scene prompt:

python -m inference.generate_video \
  --code-video outputs/threejs/case/reconstruction.mp4 \
  --reference-image outputs/threejs/reference.png \
  --prompt-file outputs/threejs/prompt.txt \
  --model-root models/MiniMax-H3-ref2va \
  --keep-steps 5 \
  --output outputs/threejs/video.mp4

Create a pair

Both original and edited workspaces include scene code, a proxy video, a reference image, and a prompt. Generate a new edited result from the provided original:

python -m inference.generate_pair \
  --prompt-file examples/chair_placement/edited/prompt.txt \
  --anchor examples/chair_placement/original \
  --edit-file examples/chair_placement/edit.txt \
  --pair-type scene \
  --model-root models/MiniMax-H3-ref2va \
  --output outputs/chair_placement

Benchmark

Download on Hugging Face. CodeVideoBench provides 22 cases across eight applications, with editable Three.js programs, proxy videos, reference images, prompts, and a browser-based viewer.

python -m pip install huggingface_hub
hf download code-video-model/CodeVideoBench \
  --repo-type dataset \
  --local-dir data/CodeVideoBench

To browse the gallery and inspect the prepared scenes:

python -m http.server 8001 --bind 127.0.0.1 --directory data/CodeVideoBench

To run a prepared case with H3, use Cases/<case_id>/code_video.mp4, reference.png, and prompt.txt as the three inputs in Inference 3.

Evaluation

Evaluate Structure, Style, and Quality with Qwen3.8-27B through OpenRouter. Set OPENROUTER_API_KEY in your environment, then run:

python evaluate_video.py \
  --video outputs/sports_car.mp4 \
  --code-video examples/sports_car/code_video.mp4 \
  --reference-image examples/sports_car/reference.png \
  --prompt-file examples/sports_car/prompt.txt \
  --output outputs/sports_car_evaluation

The script makes one request per dimension and saves scores.

Citation

If you find this repository useful, please cite:

@misc{liang2026codevideomodel,
  title        = {Code Video Model},
  author       = {Liang, Zhiyang and Zhang, Jiawei and Qian, Guocheng and Chen, Dongdong and Wan, Ziyu},
  year         = {2026},
  month        = oct,
  note         = {Preprint},
  doi          = {10.13140/RG.2.2.34014.80961},
  url          = {https://code-video-model.github.io/tech_report.pdf}
}

CodeVideoModel (code-video-model/CodeVideoModel)

threejs
Searching Database…

573 ·GitHub
CodeVideoModel
What is this?CodeVideoModel is a research pipeline that turns a text prompt into an editable Three.js scene codebase, converts that scene into a reference image, and then renders a final video with the MiniMax-H3 model. It also ships CodeVideoBench, a benchmark of 22 evaluation cases for this code-to-video workflow.
Is this a ready-made app or a chain of research scripts?It's a multi-stage inference pipeline of Python scripts (text-to-code, video-to-reference-image, scene-to-video) plus a local browser viewer for inspecting the generated Three.js scenes frame by frame.
How do I run the pipeline?Download the MiniMax-H3 checkpoint, then run the bundled examples under examples/ (sportscar, tankboarding, lofttour, robotnavigation) or your own prompt through the text-to-code, video-to-reference, and scene-to-video inference steps in order.
What do I need installed before running it?An Ubuntu 22.04 machine with a CUDA 12.6-compatible NVIDIA GPU driver for MiniMax-H3, plus your own OPENROUTER_API_KEY to call GPT-6 for scene code generation, GPT-Image-2 for reference images, and Qwen3-VL for evaluation.
Does running the full pipeline cost money beyond the GPU?Yes - the reference-image step (GPT-Image-2) and the automated Structure/Style/Quality evaluation (Qwen3-VL-27B) both run through OpenRouter using your own API key, so every run bills your OpenRouter account on top of the local GPU cost.
Who is this built for?Researchers and engineers working on text-to-video or video-editing systems who want fine-grained, code-level control over 3D scenes before they are rendered into video with MiniMax-H3.
What is CodeVideoBench?It is the project's benchmark: 22 cases across eight applications, each with an editable Three.js program, a proxy video, a reference image, and a prompt, plus a browser-based viewer to inspect the prepared scenes.
Why go through a Three.js codebase instead of generating video directly?Representing the scene as editable Three.js code lets GPT-6 refine composition, camera, and trajectory precisely before rendering, and makes it possible to create matched original/edited scene pairs for controlled video editing experiments.
What can I do in the browser viewer?You can scrub the timeline, step through frames, inspect the source file behind each frame, and in the World Inspector drag to orbit, Shift-drag to pan, scroll to zoom, and toggle the camera frustum, trajectory, and grid.
Can I supply my own reference image instead of extracting one from video?Yes, the video-to-reference-image step accepts an already extracted frame in place of a source video for both the strict-alignment and text-guided generation modes.
What are similar projects?generative-models, FreeVideo, ComfyUI-ALLinONE-MinimaxH3, LTX-Video, hyperflow, tapnet, WanGP, CogVideo
Which topics is it in?gpt-image-2, minimax-h3, minimax, 3d-generation, image-to-video, openrouter, text-to-video, text-to-image, threejs, webgl, code-generation, video-processing
Creator
code-video-model
Size
23.8 MB
Stars
573
Updated
CommitOct 13
Age
1mo
Commits · 90 Days