EdenLive README

项目的 README,来自 github.com/VAST-AI-Research/EdenLive。合并为一个 markdown 文件:README.md。

EdenLive

EdenLive: Real-Time Generative Rendering from Any Visual Reference

Project Page Model License

Project Page · Code · Models · Paper

Zehuan Huang1,*, Tianxing Xu1,*, Chunli Peng1,*, Cheng Zhang1,*, Zeyi Zheng1,†, Yifei Song1,†, Zekai Gu1,†, Zhipeng Yu1, Peng Wang1,‡, Yanpei Cao1,✉

* Equal contribution · † Intern at VAST · ‡ Project lead · ✉ Corresponding author

1VAST · Project Eden

TL;DR:

https://github.com/user-attachments/assets/243f1392-b41e-4ee6-a9c0-f5be8263bbec

Overview

EdenLive is a real-time generative renderer that treats the reference video as the interface between an interactive program and neural visual synthesis: the upstream program executes the rules and maintains the world state, whereas EdenLive renders the resulting reference video, guided by a target first frame and a text prompt, into detailed frames.

The project page demonstrates over 24 FPS, minute-scale generation, and 480P/720P output. The 5B model supports single-GPU deployment at 720P.

We have open-sourced:

Installation

One Python environment runs both the offline inference in edenlive/ and the real-time service in realtime/.

Verified version
OS / GPU Linux x86-64, NVIDIA GPUs with 80 GB (CUDA 13 driver)
Python 3.10
PyTorch 2.14.0.dev20260717+cu130 nightly (realtime/requirements.txt pins 2.13.0, cu130), torchvision 0.28.0
diffusers / transformers / accelerate 0.39.0 / 5.12.1 / 1.0.1
decord / pandas / pyarrow 0.6.0 / 2.3.3 / 25.0.1
imageio / imageio-ffmpeg 2.37.4 / 0.6.0
Node.js 22 (only to fetch three.js for the game renders)
git clone https://github.com/VAST-AI-Research/EdenLive.git EdenLive
cd EdenLive

conda create -n edenlive python=3.10 -y
conda activate edenlive

# 1. PyTorch for a CUDA 13 driver (use .../whl/cu128 for CUDA 12.8).
pip install torch==2.13.0 torchvision==0.28.0 --index-url https://download.pytorch.org/whl/cu130

# 2. The real-time service and its pinned dependencies.
cd realtime
pip install --no-deps -e ./flashdreams -e ./apps/cam2v -e ./integrations_v2/eden
pip install -r requirements.txt
python -m playwright install chromium
npm install --prefix games
cd ..

# 3. Offline inference and video I/O dependencies.
pip install av==17.1.0 decord==0.6.0 pandas==2.3.3 pyarrow==25.0.1 imageio==2.37.4 imageio-ffmpeg==0.6.0
pip install --no-deps -e .

The default 5B bidirectional profile uses FA4 temporal text attention and requires SM90, BF16, flash-attn-4==4.0.0b31, and nvidia-cutlass-dsl==4.7.1. Use --no-deps when installing kernels into an existing environment to preserve PyTorch. See docs/zh/wan22-fa4.md.

Model weights

hf download VAST-AI/EdenLive --local-dir ./models/EdenLive

Everything loads from one directory, models/EdenLive under the repository root by default (about 210 GB in total).

models/EdenLive/
  dit-14b/            diffusion_pytorch_model.safetensors, config.json  14B few-step (chunk-AR) DiT
  dit-14b-bidir/      diffusion_pytorch_model.safetensors, config.json  14B bidirectional DiT
  dit-5b/             diffusion_pytorch_model.safetensors, config.json  5B few-step (chunk-AR) DiT
  dit-5b-bidir/       diffusion_pytorch_model.safetensors, config.json  5B bidirectional DiT
  vae-14b/            Wan2.1_VAE.pth, config.json
  vae-5b/             Wan2.2_VAE.pth, config.json
  decoder-14b/        lighttaew2_1.pth                    TAEHV decoder (real-time only)
  decoder-5b/         lighttaew2_2.pth                    TAEHV decoder (real-time only)
  scheduler-14b/      scheduler_config.json               offline only
  scheduler-5b/       scheduler_config.json               offline only
  umt5/               text_encoder/, tokenizer/           UMT5 text encoder
  clip/               image_encoder/, image_processor/    CLIP, 14B I2V only
  vision-encoder/     backbone/, projection.safetensors   proxy-DINO encoder
  flux-c2f/           Diffusers FLUX tree + c2f_lora.safetensors   workbench first frame (real-time only)

The default weight directory is models/EdenLive, relative to the working directory. To specify another directory:

python -m edenlive.scripts.infer --config ... model.pretrained_dir=/data/EdenLive

The real-time service reads the same tree through MODEL_ROOT in realtime/deploy.conf.

Quick start

Two proxy recordings ship with the repo under examples/samples/, with a manifest that points at them:

examples/samples/
  manifest.csv        id, prompt, proxy_video, first_frame
  cryo-crossing/       cover.png, first-frame.png, prompts.json, prompts.txt, recording.mp4
  pulse-skirmish/      same five files

recording.mp4 is the greybox proxy (1280x704), and first-frame.png is its first frame. cover.png is a photoreal render of that frame; the manifest's first_frame points to this file. prompts.json contains staticPrompt and timestamped dynamicPrompts; prompts.txt is the corresponding text version.

Mode Wan2.1-I2V-14B @ 480p Wan2.2-TI2V-5B @ 720p
Bidirectional wan21_14b_480p_bidir.yaml wan22_5b_720p_bidir.yaml
Few-step (chunk-AR) wan21_14b_480p_few_step.yaml wan22_5b_720p_few_step.yaml

Bidirectional inference: all frames are denoised jointly.

torchrun --nproc_per_node=2 -m edenlive.scripts.infer \
  --config edenlive/configs/wan21_14b_480p_bidir.yaml \
  runtime.sequence_parallel.size=2
python -m edenlive.scripts.infer \
  --config edenlive/configs/wan22_5b_720p_bidir.yaml

Few-step inference: chunk-autoregressive sampling, 4 steps per chunk.

torchrun --nproc_per_node=2 -m edenlive.scripts.infer \
  --config edenlive/configs/wan21_14b_480p_few_step.yaml \
  runtime.sequence_parallel.size=2
python -m edenlive.scripts.infer \
  --config edenlive/configs/wan22_5b_720p_few_step.yaml

Real-time service: uses game frames as input and streams video over WebRTC. Configure the weight directory, service addresses, ports, and GPU count in realtime/deploy.conf. 5B uses one GPU by default. Set NPROC for multi-GPU inference.

Set TURN_HOST (the relay address reachable by the browser), TURN_IFACE (the relay interface), and TURN_SECRET (your own password) in realtime/deploy.conf. To run existing games, execute on the GPU host:

cd realtime
apt-get install coturn  # First-time installation
bash turn_server.sh    # Start TURN first
bash start.sh 14b       # Model service; or use 5b

The workbench — a game library, a chat that builds new games from a text description, and a play window — needs all four services. Its first-frame renderer (the FLUX worker) runs FLUX.2 [klein] 9B under the FLUX Non-Commercial License, so the workbench is for non-commercial use only, and that license requires content filtering on the images it generates (see License). Playing registered games with start.sh does not load FLUX.

The bridge host requires the claude CLI, claude auth login, and first-run trust confirmation. Game generation uses --dangerously-skip-permissions. For separate hosts, copy the bridge's .workbench/token to the GPU host's realtime/ directory and set BRIDGE_HOST; leave it empty for a single-host deployment.

cd realtime

# 1. TURN relay, GPU host; skip if already running
bash turn_server.sh

# 2. Workbench bridge, Claude CLI host
bash workbench_bridge.sh

# 3. FLUX first-frame service, GPU host
bash workbench_flux.sh

# 4. Model service, GPU host
bash launch.sh

# Stop services on this host
bash stop.sh

Open http://<host>:8089 (APP_PORT in deploy.conf).

Model Application slug
5B game2v-eden-world-v2-5b-taehv
14B game2v-eden-world-v2-14b-taehv

The real-time service uses the TAEHV decoder. See realtime/README.md for the service topology and developer documentation.

Inference with Reactor Runtime

The Reactor integration serves EdenLive's four-step 5B pipeline with synchronized draft and generated video, keyboard controls, live prompt updates, and a browser demo. It runs on one B200 and keeps the native inference and training entrypoints unchanged.

Install the Reactor CLI and prepare EDEN_BUNDLE following the source and weights instructions. The bundle uses a pinned seven-game source snapshot; the current repository's game registry has different entries.

cd reactor
reactor build
reactor run --gpus device=0 --shm-size 2g --weights "$EDEN_BUNDLE" --port 8080

Wait for curl http://localhost:8080/health to report "state":"available". Then, in a second terminal from the repository root:

cd reactor/demo
npm ci
VITE_REACTOR_URL=http://localhost:8080 npm run dev

Open the URL printed by Vite, select Local, click Connect, choose a game and reference, then press Start. Update prompt changes upcoming appearance while keeping the video history. See the Reactor guide for playback controls and SSH/TURN networking.

Your own data

The default profiles read a recording manifest (CSV/JSONL/JSON/parquet), one recording per request. Each directory contains recording.mp4, cover.png, and prompts.json; text comes from staticPrompt and dynamicPrompts:

id,proxy_video,first_frame
shot_001,shot_001/recording.mp4,shot_001/cover.png
python -m edenlive.scripts.infer --config CONFIG.yaml \
  data.dataset.params.path=/data/my_manifest.csv \
  data.dataset.params.root=/data

Real-Time Inference with LightX2V on RTX 5090

LightX2V provides accelerated inference for EdenLive's 14B and 5B models in both bidirectional and four-step autoregressive modes. Optimizations include low-bit quantization, efficient kernels and operator fusion, KV cache optimizations, torch.compile, CUDA Graphs, and streaming LightTAE decoding.

Real-time inference on a single RTX 5090: EdenLive 5B achieves approximately 24.7 FPS inference throughput at 1280×704, meeting the throughput requirement for real-time 24 FPS playback.

See the EdenLive deployment guide for environment setup, model conversion, quantization, and launch commands.

Citation

@misc{edenlive2026,
  title  = {EdenLive: Real-Time Generative Rendering from Any Visual Reference},
  author = {Zehuan Huang and Tianxing Xu and Chunli Peng and Cheng Zhang and Zeyi Zheng and Yifei Song and Zekai Gu and Zhipeng Yu and Peng Wang and Yanpei Cao},
  year   = {2026},
  url    = {https://vast-eden.ai/?p=edenlive}
}

Acknowledgements

Built on Wan2.1 / Wan2.2, 🤗 Diffusers, DINOv3, FLUX.2 [klein] by Black Forest Labs, and the LightX2V LightTAE decoders. The game renders use Three.js, and the workbench's game-generation skill builds on GameBlocks.

The real-time streaming service under realtime/ builds on NVIDIA FlashDreams.

License

First-party code is licensed under Apache 2.0 — see LICENSE and NOTICE. realtime/ is also Apache 2.0 and keeps its own LICENSE, NOTICE and THIRD-PARTY-NOTICES. The vendored html-to-image bundle and GameBlocks-derived skill modules retain their MIT licenses and original copyright notices; per-file scope is recorded in REUSE.toml.

The model weights are distributed separately and are not all Apache 2.0:

Component License
dit-*, vae-*, umt5/, scheduler-*, decoder-* Apache 2.0
clip/ (LAION CLIP ViT-H/14 / OpenCLIP) MIT
vision-encoder/ (DINOv3 and derivatives) DINOv3 License
flux-c2f/ (FLUX.2 [klein] 9B plus our LoRA; workbench first frame only) FLUX Non-Commercial License — non-commercial, non-production use only

Each component ships its own license and notice files alongside the weights; see the model repository's README for details.

EdenLive (VAST-AI-Research/EdenLive)

正在搜索数据库…

这是什么?EdenLive 是一个实时生成式渲染器。它以目标首帧和文本提示为引导,把参考视频(由游戏引擎或编程智能体生成)转换为细节丰富、接近照片真实感的视频流,帧率达到 24 FPS 以上。
这是成品应用,还是带模型权重的研究代码?这是一个开源的研究代码库:包含 Python 推理与训练代码,以及单独分发的 5B 和 14B 模型权重(约 210GB)。此外还有可选的实时流媒体服务,以及一个基于浏览器的“工作台”,用于体验从提示词到游戏的演示。
我该怎么把它跑起来?在同一个 Python 3.10 环境中同时配置离线推理(edenlive/)和实时服务(realtime/),将约 210GB 的权重目录下载到 models/EdenLive,然后运行 YAML 配置文件(例如 wan225b720pbidir.yaml)进行离线渲染;或者配置 realtime/deploy.conf 并启动服务,以便通过 WebRTC 实时播放。
我需要特定的 GPU 和 CUDA 环境吗?需要。经过验证的环境是:Linux x86-64 系统,配备 80GB 显存的 NVIDIA GPU(CUDA 13 驱动)、Python 3.10、PyTorch 2.14 夜间构建版(cu130),以及用于游戏渲染器的 Node.js 22。默认的 5B 双向配置还需要 SM90 架构的 GPU,并安装 flash-attn-4 和 nvidia-cutlass-dsl 内核。
在试用完整的提示词转游戏工作台之前,有什么需要注意的吗?工作台的首帧渲染器运行 FLUX.2 klein 9B 模型,并受 FLUX 非商业许可证约束,因此整个工作台仅限非商业用途,并且需要对生成的图像进行内容过滤。使用 start.sh 游玩已注册的游戏则不会加载 FLUX。
这个能在消费级显卡上跑,还是必须用数据中心显卡?5B 模型可以在单张消费级 RTX 5090 上进行实时推理,在 1280x704 分辨率下约为 24.7 FPS,达到 24 FPS 的实时目标。而经过验证的参考环境在默认配置下使用的是 80GB 数据中心级 GPU。
EdenGolem 是什么?EdenGolem(v0.1)是一个内置技能,可以把一段纯文本提示词转换为粗略的、可直接游玩的游戏。随后,EdenLive 会将其作为仓库中提示词转游戏流程的一部分,进行细节渲染。
双向推理和少步推理有什么区别?双向推理会联合去噪一个片段中的所有帧,以获得最佳画质;而少步(分块自回归)推理每个片段只使用 4 步去噪,从而实现实时流媒体播放。
Reactor Runtime 集成带来了什么?Reactor 为 EdenLive 的四步 5B 流程提供服务,支持同步的草稿视频与生成视频、键盘控制、实时提示词更新和浏览器演示。它无需修改原生推理代码,即可在单张 B200 GPU 上运行。
LightX2V 带来了什么?LightX2V 是面向 14B 和 5B 模型的加速推理后端,增加了低比特量化、融合内核、KV 缓存优化、torch.compile/CUDA Graphs,以及流式 LightTAE 解码器,从而可以在单张 RTX 5090 上实现实时播放。
大小
124.1 MB
星标
448
更新
提交Nov 13
时长
1个月
提交 · 90 天