CodeVideoModel README

The project's README from github.com/code-video-model/CodeVideoModel. As one markdown file: README.md.

Code Video Model

Code Video Model Team

Tech Report Project Page Hugging Face GitHub Code Code License: Apache 2.0 Examples License: CC BY 4.0

Demo Video

https://github.com/user-attachments/assets/d1a8594d-14fa-49aa-a101-03da77ed585f

Installation

Use Ubuntu 22.04 and a CUDA 12.6-compatible NVIDIA driver for MiniMax-H3.

git clone https://github.com/code-video-model/CodeVideoModel.git
cd CodeVideoModel

conda create -n cvm python=3.10 -y
conda activate cvm
conda install -c conda-forge "ffmpeg[version='>=6',build='gpl_*']" -y
python -m pip install --upgrade pip
python -m pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 \
  --index-url https://download.pytorch.org/whl/cu126
python -m pip install -r requirements.txt
python -m playwright install chromium

The Three.js stage uses GPT-6. For GPT-Image-2, set OPENROUTER_API_KEY securely in your environment.

Checkpoint

Download the MiniMax-H3 checkpoint:

python download_h3.py --output models/MiniMax-H3-ref2va

Quick Start

Each demo bundles a prompt, a reference image, and a Three.js video, ready for direct generation with MiniMax-H3.

Example Directory under examples/
Sports Car sports_car
Tank Boarding tank_boarding
Loft Tour loft_tour
Robot Navigation robot_navigation

Set example to any directory name above:

example=tank_boarding
python -m inference.generate_video \
  --code-video "examples/${example}/code_video.mp4" \
  --reference-image "examples/${example}/reference.png" \
  --prompt-file "examples/${example}/prompt.txt" \
  --model-root models/MiniMax-H3-ref2va \
  --keep-steps 5 \
  --output "outputs/${example}.mp4"

Inference

1. Text → Three.js Codebase

Generate an editable scene from a text prompt. GPT-6 builds the codebase and refines it through direct visual review:

python -m inference.generate_threejs \
  --prompt "A red car follows a curved road through a sunlit town, with a smooth tracking camera." \
  --output outputs/threejs

The generated scene files are saved in the output workspace's case directory. Prompts can be entered directly or read from a text file.

Export the video:

python -m inference.render video --scene outputs/threejs

View the scene in your browser:

python -m inference.viewer --scene outputs/threejs --port 8000

Open the local URL printed in the terminal. Play or scrub the timeline, step through frames, and select a source file to read its code. In the World Inspector, drag to orbit, Shift-drag to pan, and scroll to zoom; toggle the camera frustum, trajectory, and grid.

2. Three.js Video → Reference Image

Strict alignment

Extract the video's first frame and use GPT-Image-2 to create a 1536×864 reference, with instructions to preserve the scene's composition and content:

python -m inference.generate_reference \
  --video outputs/threejs/case/reconstruction.mp4 \
  --output outputs/threejs/reference.png

You can also provide an already extracted frame instead of a video.

Text-guided reference

Use the first frame as loose visual context and a text prompt to guide the result, without requiring the same camera, composition, or object placement:

python -m inference.generate_reference_guided \
  --video examples/loft_tour/code_video.mp4 \
  --prompt "A sunlit modern loft with a leather sofa and floating stairs, photographed at eye level." \
  --output outputs/loft_reference_guided.png

This mode also accepts an extracted frame and a scene description loaded from a text file. It produces a 1536×864 reference image.

3. Three.js Video + Reference Image → MiniMax-H3 Video

Generate a video using the proxy, reference image, and original scene prompt:

python -m inference.generate_video \
  --code-video outputs/threejs/case/reconstruction.mp4 \
  --reference-image outputs/threejs/reference.png \
  --prompt-file outputs/threejs/prompt.txt \
  --model-root models/MiniMax-H3-ref2va \
  --keep-steps 5 \
  --output outputs/threejs/video.mp4

Create a pair

Both original and edited workspaces include scene code, a proxy video, a reference image, and a prompt. Generate a new edited result from the provided original:

python -m inference.generate_pair \
  --prompt-file examples/chair_placement/edited/prompt.txt \
  --anchor examples/chair_placement/original \
  --edit-file examples/chair_placement/edit.txt \
  --pair-type scene \
  --model-root models/MiniMax-H3-ref2va \
  --output outputs/chair_placement

Benchmark

Download on Hugging Face. CodeVideoBench provides 22 cases across eight applications, with editable Three.js programs, proxy videos, reference images, prompts, and a browser-based viewer.

python -m pip install huggingface_hub
hf download code-video-model/CodeVideoBench \
  --repo-type dataset \
  --local-dir data/CodeVideoBench

To browse the gallery and inspect the prepared scenes:

python -m http.server 8001 --bind 127.0.0.1 --directory data/CodeVideoBench

To run a prepared case with H3, use Cases/<case_id>/code_video.mp4, reference.png, and prompt.txt as the three inputs in Inference 3.

Evaluation

Evaluate Structure, Style, and Quality with Qwen3.8-27B through OpenRouter. Set OPENROUTER_API_KEY in your environment, then run:

python evaluate_video.py \
  --video outputs/sports_car.mp4 \
  --code-video examples/sports_car/code_video.mp4 \
  --reference-image examples/sports_car/reference.png \
  --prompt-file examples/sports_car/prompt.txt \
  --output outputs/sports_car_evaluation

The script makes one request per dimension and saves scores.

Citation

If you find this repository useful, please cite:

@misc{liang2026codevideomodel,
  title        = {Code Video Model},
  author       = {Liang, Zhiyang and Zhang, Jiawei and Qian, Guocheng and Chen, Dongdong and Wan, Ziyu},
  year         = {2026},
  month        = oct,
  note         = {Preprint},
  doi          = {10.13140/RG.2.2.34014.80961},
  url          = {https://code-video-model.github.io/tech_report.pdf}
}

CodeVideoModel (code-video-model/CodeVideoModel)

Searching Database…

556 ·GitHub
CodeVideoModel
这是什么？CodeVideoModel 是一个研究型流水线：它把文本提示词转换成可编辑的 Three.js 场景代码，再将该场景转成一张参考图，最后用 MiniMax-H3 模型渲染出最终视频。它还附带 CodeVideoBench，一个针对这种代码生成视频流程的基准测试，包含 22 个评估用例。
这是一个现成的应用，还是一套研究脚本？它是一条由多个 Python 脚本组成的多阶段推理流水线（文本生成代码、视频转参考图、场景生成视频），外加一个本地浏览器查看器，可以逐帧检查生成的 Three.js 场景。
怎么运行这个流水线？先下载 MiniMax-H3 检查点，然后按顺序执行文本生成代码、视频转参考图、场景生成视频这几个推理步骤，跑 examples/ 下自带的示例（sportscar、tankboarding、lofttour、robotnavigation），也可以换成你自己的提示词。
运行之前需要装好什么？需要一台 Ubuntu 22.04 机器，配备与 CUDA 12.6 兼容的 NVIDIA GPU 驱动来运行 MiniMax-H3；另外还要有自己的 OPENROUTER_API_KEY，用于调用 GPT-6 生成场景代码、GPT-Image-2 生成参考图，以及 Qwen3-VL 做评估。
除了 GPU 之外，跑完整条流水线还要花钱吗？要——参考图这一步（GPT-Image-2）和自动化的 Structure/Style/Quality 评估（Qwen3-VL-27B）都是通过 OpenRouter 调用的，用的是你自己的 API 密钥，所以每次运行除了本地 GPU 的成本，还会计入你的 OpenRouter 账户费用。
这是给谁用的？面向研究文本生成视频或视频编辑系统的研发人员和工程师，他们希望在用 MiniMax-H3 把 3D 场景渲染成视频之前，能对场景进行细粒度、代码级别的控制。
CodeVideoBench 是什么？它是这个项目的基准测试：覆盖八个应用方向共 22 个用例，每个用例包含一个可编辑的 Three.js 程序、一个代理视频、一张参考图和一个提示词，还带一个基于浏览器的查看器，用来检查准备好的场景。
为什么要绕道 Three.js 代码，而不是直接生成视频？把场景表示成可编辑的 Three.js 代码，可以让 GPT-6 在渲染之前精确调整构图、镜头和运动轨迹，而且能构造出相互匹配的原始/编辑场景对，用于受控的视频编辑实验。
在浏览器查看器里我能做什么？你可以拖动时间轴、逐帧查看、查看每一帧背后的源文件；在 World Inspector 里，拖动可以环绕旋转，按住 Shift 拖动可以平移，滚轮可以缩放，还可以开关相机视锥体、轨迹线和网格。
我可以自己提供参考图，不用从视频里截取吗？可以，视频转参考图这一步接受已经截取好的帧来代替源视频，strict-alignment 和 text-guided 两种生成模式都支持。
What are similar projects?generative-models, FreeVideo, ComfyUI-ALLinONE-MinimaxH3, LTX-Video, hyperflow, tapnet, WanGP, CogVideo
Which topics is it in?gpt-image-2, minimax-h3, minimax, 3d-generation, image-to-video, text-to-video, openrouter, text-to-image, threejs, webgl, code-generation, video-processing
Creator
code-video-model
Size
23.8 MB
Stars
556
Updated
CommitOct 13
Age
1mo
Commits · 90 Days