PersonaLive README

The project's README from github.com/GVCLab/PersonaLive. As one markdown file: README.md.

PersonaLive

Expressive Portrait Image Animation for Live Streaming

Zhiyuan Li1,2,3 · Chi-Man Pun1,📪 · Chen Fang2 · Jue Wang2 · Xiaodong Cun3,📪

1 University of Macau    2 Dzine.ai    3 GVC Lab, Great Bay University

GitHub

highlight

  

📋 TODO

⚖️ Disclaimer

⚙️ Framework

Image 1

We present PersonaLive, a real-time and streamable diffusion framework capable of generating infinite-length portrait animations.

🚀 Getting Started

🛠 Installation

# clone this repo
git clone https://github.com/GVCLab/PersonaLive
cd PersonaLive

# Create conda environment
conda create -n personalive python=3.10
conda activate personalive

# Install packages with pip
pip install -r requirements_base.txt

⏬ Download weights

Option 1: Download pre-trained weights of base models and other components (sd-image-variations-diffusers and sd-vae-ft-mse). You can run the following command to download weights automatically:

python tools/download_weights.py

Option 2: Download pre-trained weights into the ./pretrained_weights folder from one of the below URLs:

Finally, these weights should be organized as follows:

pretrained_weights
├── onnx
│   ├── unet_opt
│   │   ├── unet_opt.onnx
│   │   └── unet_opt.onnx.data
│   └── unet
├── personalive
│   ├── denoising_unet.pth
│   ├── motion_encoder.pth
│   ├── motion_extractor.pth
│   ├── pose_guider.pth
│   ├── reference_unet.pth
│   └── temporal_module.pth
├── sd-vae-ft-mse
│   ├── diffusion_pytorch_model.bin
│   └── config.json
├── sd-image-variations-diffusers
│   ├── image_encoder
│   │   ├── pytorch_model.bin
│   │   └── config.json
│   ├── unet
│   │   ├── diffusion_pytorch_model.bin
│   │   └── config.json
│   └── model_index.json
└── tensorrt
    └── unet_work.engine

🎞️ Offline Inference

Run offline inference with the default configuration:

python inference_offline.py

⚠️ Note for RTX 50-Series (Blackwell) Users: xformers is not yet fully compatible with the new architecture. To avoid crashes, please disable it by running:

python inference_offline.py --use_xformers False

📸 Online Inference

📦 Setup Web UI

# install Node.js 18+
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash
nvm install 18

source web_start.sh

🏎️ Acceleration (Optional)

Converting the model to TensorRT can significantly speed up inference (~ 2x ⚡️). Building the engine may take about 20 minutes depending on your device. Note that TensorRT optimizations may lead to slight variations or a small drop in output quality.

# Install packages with pip
pip install -r requirements_trt.txt

# src/models/motion_encoder/FAN_temporal_feature_extractor.py
self.pos_embed.pos_embed.requires_grad = False

# Converting the model to TensorRT
python torch2trt.py

💡 PyCUDA Installation Issues: If you encounter a "Failed to build wheel for pycuda" error during the installation above, please follow these steps:

# Install PyCUDA manually using Conda (avoids compilation issues):
conda install -c conda-forge pycuda "numpy<2.0"

# Open requirements_trt.txt and comment out or remove the line "pycuda==2024.1.2"

# Install other packages with pip
pip install -r requirements_trt.txt

# Converting the model to TensorRT
python torch2trt.py

⚠️ The provided TensorRT model is from an H100. We recommend ALL users (including H100 users) re-run python torch2trt.py locally to ensure best compatibility.

▶️ Start Streaming

python inference_online.py --acceleration none (for RTX 50-Series) or xformers or tensorrt

Then open http://0.0.0.0:7860 in your browser. (*If http://0.0.0.0:7860 does not work well, try http://localhost:7860)

How to use: Upload Image ➡️ Fuse Reference ➡️ Start Animation ➡️ Enjoy! 🎉

PersonaLive

Regarding Latency: Latency varies depending on your device's computing power. You can try the following methods to optimize it:

  1. Lower the "Driving FPS" setting in the WebUI to reduce the computational workload.
  2. You can increase the multiplier (e.g., set to num_frames_needed * 4 or higher) to better match your device's inference speed. https://github.com/GVCLab/PersonaLive/blob/6953d1a8b409f360a3ee1d7325093622b29f1e22/webcam/util.py#L73

🚄 Model Training

PersonaLive training is organized into three stages. Approximate training time on 8x H100 with default configs: Stage 1 ~13h, Stage 2 ~15h, Stage 3 ~20h.

1️⃣ Environment setup

Install base dependencies first (see installation section), then install training-only packages:

pip install -r requirements_train.txt

If you use multi-GPU or multi-node training, configure Accelerate once before launching training:

accelerate config

2️⃣ Data preparation

Your dataset should contain a videos directory and a matching boxes directory:

Datasets
├── VFHQ
│   ├── videos
│   │   ├── example1.mp4
│   │   ├── example2.mp4
│   │   └── ...
│   └── boxes
│       ├── example1.pt
│       ├── example2.pt
│       └── ...
└── ...

Preprocessing example:

# 1) Extract face / eye / mouth boxes from each frame
python tools/get_boxes.py --video_dir ./Datasets/VFHQ/videos --save_dir ./Datasets/VFHQ/boxes --workers 8

# 2) Generate meta json: [{"video_path": ".../videos/xxx.mp4"}, ...]
python tools/extract_meta_info.py --root_path ./Datasets/VFHQ --dataset_name VFHQ

Then set data.meta_paths in each training config:

data:
  meta_paths:
    - "./data/VFHQ_meta.json"
    - "./data/OtherDataset_meta.json"

3️⃣ Download weights

Download the training initialization weights: X-NeMo, pose_guider, and stylegan2_discriminator.

pretrained_weights
├── xnemo
│   ├── xnemo_motion_encoder.pth
│   ├── xnemo_denoising_unet.pth
│   ├── xnemo_reference_unet.pth
│   └── xnemo_temporal_module.pth
├── sd-vae-ft-mse
│   ├── diffusion_pytorch_model.bin
│   └── config.json
├── sd-image-variations-diffusers
│   ├── image_encoder
│   │   ├── pytorch_model.bin
│   │   └── config.json
│   ├── unet
│   │   ├── diffusion_pytorch_model.bin
│   │   └── config.json
│   └── model_index.json
├── pose_guider.pth
└── stylegan2_discriminator_ffhq512.pth

4️⃣ Training workflow

Stage 1: Image-level warm-up

Run:

accelerate launch train_stage1.py --config ./configs/train/personalive_stage1.yaml

Default output folder: ./exp_output/personalive_stage1/

Stage 2: Image-level adversarial refinement

Update configs/train/personalive_stage2.yaml to point to Stage 1 outputs:

motion_encoder_path: './exp_output/personalive_stage1/motion_encoder-xxxxx.pth'
denoising_unet_path: './exp_output/personalive_stage1/denoising_unet-xxxxx.pth'
reference_unet_path: './exp_output/personalive_stage1/reference_unet-xxxxx.pth'
pose_guider_path: './exp_output/personalive_stage1/pose_guider-xxxxx.pth'

Run:

accelerate launch train_stage2.py --config ./configs/train/personalive_stage2.yaml

Default output folder: ./exp_output/personalive_stage2/

Stage 3: Temporal module fine-tuning for streaming

Update configs/train/personalive_stage3.yaml to point to Stage 2 outputs:

motion_encoder_path: './exp_output/personalive_stage2/motion_encoder-xxxxx.pth'
denoising_unet_path: './exp_output/personalive_stage2/denoising_unet-xxxxx.pth'
reference_unet_path: './exp_output/personalive_stage2/reference_unet-xxxxx.pth'
pose_guider_path: './exp_output/personalive_stage2/pose_guider-xxxxx.pth'
discriminator_path: './exp_output/personalive_stage2/discriminator-xxxxx.pth'

Run:

accelerate launch train_stage3.py --config ./configs/train/personalive_stage3.yaml

Default output folder: ./exp_output/personalive_stage3/

📚 Community Contribution

Special thanks to the community for providing helpful setups! 🥂

🎬 More Results

👀 Visualization results

🤺 Comparisons

⭐ Citation

If you find PersonaLive useful for your research, welcome to cite our work using the following BibTeX:

@article{li2025personalive,
  title={PersonaLive! Expressive Portrait Image Animation for Live Streaming},
  author={Li, Zhiyuan and Pun, Chi-Man and Fang, Chen and Wang, Jue and Cun, Xiaodong},
  journal={arXiv preprint arXiv:2512.11253},
  year={2025}
}

❤️ Acknowledgement

This code is mainly built upon Moore-AnimateAnyone, X-NeMo, StreamDiffusion, RAIN and LivePortrait, thanks to their invaluable contributions.

PersonaLive (GVCLab/PersonaLive)

Searching Database…

4K ·GitHub
PersonaLive
这是什么？PersonaLive 是一个支持实时流式推理的扩散模型框架，能把单张肖像图变成无限时长的说话视频，专为直播场景设计。它已被 CVPR 2026 接收。
这是现成的应用，还是要自己运行的研究代码？它是学术研究代码（Python、扩散模型），附带一个用于推理的 WebUI，社区还另行构建了 ComfyUI 集成，并不是打包好的消费级应用。
怎么把它跑起来？安装基础依赖，把预训练的基础模型权重（sd-image-variations-diffusers 和 sd-vae-ft-mse）下载到 ./pretrainedweights，然后启动 WebUI，对你的肖像图运行推理。
运行它需要很强的 GPU 吗？需要，这是一个扩散模型，必须有 CUDA GPU；流式推理模式只需 12GB 显存就能生成长视频。
生成的结果可以随便用吗？不可以。README 声明本项目仅面向学术研究发布，用户不得用它生成有害、诽谤或违法内容，作者对滥用行为不承担任何责任。
只能推理，还是也能训练自己的模型？两者都可以：仓库提供了用于推理的预训练权重，以及完整训练代码，分为三个阶段（图像热身、对抗精炼、时序微调），在作者的配置下于 8 张 H100 GPU 上大约耗时 13/15/20 小时。
在 WebUI 里可以调什么来控制延迟？你可以调低 Driving FPS 设置来减少计算负载，或者调高帧数倍率（numframesneeded），以更好地匹配你 GPU 的推理速度。
有 ComfyUI 版本吗？有，ComfyUI-PersonaLive 是由 @okdalto 贡献的社区独立集成，把这个模型带进了 ComfyUI 的节点式工作流。
PersonaLive 建立在哪些先前工作之上？它建立在 Moore-AnimateAnyone、X-NeMo、StreamDiffusion、RAIN 和 LivePortrait 之上，把这些项目的思路融合成一条实时流式处理管线。
有人在 Windows 上跑通过吗？社区分享了一份在配备 RTX 50 系（Blackwell）GPU 的 Windows 上运行的指南，包括 TensorRT 转换的注意事项，不过官方并没有提供 Windows 打包版本。
What are similar projects?SolarWM, Wan2.2, generative-models, PhotoMaker, LTX-Video, FramePack, InstantID, HunyuanVideo
Which topics is it in?cvpr, talking-head, livestream, diffusion, comfyui, diffusion-models, webui, video-generation, academic-research, image-generation, real-time, streaming
Creator
GVCLab
Size
150.3 MB
Stars
4K
Updated
CommitAug 28
Age
10mo
Commits · 90 Days