PersonaLive README

The project's README from github.com/GVCLab/PersonaLive. As one markdown file: README.md.

PersonaLive

Expressive Portrait Image Animation for Live Streaming

Zhiyuan Li1,2,3 · Chi-Man Pun1,📪 · Chen Fang2 · Jue Wang2 · Xiaodong Cun3,📪

1 University of Macau    2 Dzine.ai    3 GVC Lab, Great Bay University

GitHub

highlight

  

📋 TODO

⚖️ Disclaimer

⚙️ Framework

Image 1

We present PersonaLive, a real-time and streamable diffusion framework capable of generating infinite-length portrait animations.

🚀 Getting Started

🛠 Installation

# clone this repo
git clone https://github.com/GVCLab/PersonaLive
cd PersonaLive

# Create conda environment
conda create -n personalive python=3.10
conda activate personalive

# Install packages with pip
pip install -r requirements_base.txt

⏬ Download weights

Option 1: Download pre-trained weights of base models and other components (sd-image-variations-diffusers and sd-vae-ft-mse). You can run the following command to download weights automatically:

python tools/download_weights.py

Option 2: Download pre-trained weights into the ./pretrained_weights folder from one of the below URLs:

Finally, these weights should be organized as follows:

pretrained_weights
├── onnx
│   ├── unet_opt
│   │   ├── unet_opt.onnx
│   │   └── unet_opt.onnx.data
│   └── unet
├── personalive
│   ├── denoising_unet.pth
│   ├── motion_encoder.pth
│   ├── motion_extractor.pth
│   ├── pose_guider.pth
│   ├── reference_unet.pth
│   └── temporal_module.pth
├── sd-vae-ft-mse
│   ├── diffusion_pytorch_model.bin
│   └── config.json
├── sd-image-variations-diffusers
│   ├── image_encoder
│   │   ├── pytorch_model.bin
│   │   └── config.json
│   ├── unet
│   │   ├── diffusion_pytorch_model.bin
│   │   └── config.json
│   └── model_index.json
└── tensorrt
    └── unet_work.engine

🎞️ Offline Inference

Run offline inference with the default configuration:

python inference_offline.py

⚠️ Note for RTX 50-Series (Blackwell) Users: xformers is not yet fully compatible with the new architecture. To avoid crashes, please disable it by running:

python inference_offline.py --use_xformers False

📸 Online Inference

📦 Setup Web UI

# install Node.js 18+
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash
nvm install 18

source web_start.sh

🏎️ Acceleration (Optional)

Converting the model to TensorRT can significantly speed up inference (~ 2x ⚡️). Building the engine may take about 20 minutes depending on your device. Note that TensorRT optimizations may lead to slight variations or a small drop in output quality.

# Install packages with pip
pip install -r requirements_trt.txt

# src/models/motion_encoder/FAN_temporal_feature_extractor.py
self.pos_embed.pos_embed.requires_grad = False

# Converting the model to TensorRT
python torch2trt.py

💡 PyCUDA Installation Issues: If you encounter a "Failed to build wheel for pycuda" error during the installation above, please follow these steps:

# Install PyCUDA manually using Conda (avoids compilation issues):
conda install -c conda-forge pycuda "numpy<2.0"

# Open requirements_trt.txt and comment out or remove the line "pycuda==2024.1.2"

# Install other packages with pip
pip install -r requirements_trt.txt

# Converting the model to TensorRT
python torch2trt.py

⚠️ The provided TensorRT model is from an H100. We recommend ALL users (including H100 users) re-run python torch2trt.py locally to ensure best compatibility.

▶️ Start Streaming

python inference_online.py --acceleration none (for RTX 50-Series) or xformers or tensorrt

Then open http://0.0.0.0:7860 in your browser. (*If http://0.0.0.0:7860 does not work well, try http://localhost:7860)

How to use: Upload Image ➡️ Fuse Reference ➡️ Start Animation ➡️ Enjoy! 🎉

PersonaLive

Regarding Latency: Latency varies depending on your device's computing power. You can try the following methods to optimize it:

  1. Lower the "Driving FPS" setting in the WebUI to reduce the computational workload.
  2. You can increase the multiplier (e.g., set to num_frames_needed * 4 or higher) to better match your device's inference speed. https://github.com/GVCLab/PersonaLive/blob/6953d1a8b409f360a3ee1d7325093622b29f1e22/webcam/util.py#L73

🚄 Model Training

PersonaLive training is organized into three stages. Approximate training time on 8x H100 with default configs: Stage 1 ~13h, Stage 2 ~15h, Stage 3 ~20h.

1️⃣ Environment setup

Install base dependencies first (see installation section), then install training-only packages:

pip install -r requirements_train.txt

If you use multi-GPU or multi-node training, configure Accelerate once before launching training:

accelerate config

2️⃣ Data preparation

Your dataset should contain a videos directory and a matching boxes directory:

Datasets
├── VFHQ
│   ├── videos
│   │   ├── example1.mp4
│   │   ├── example2.mp4
│   │   └── ...
│   └── boxes
│       ├── example1.pt
│       ├── example2.pt
│       └── ...
└── ...

Preprocessing example:

# 1) Extract face / eye / mouth boxes from each frame
python tools/get_boxes.py --video_dir ./Datasets/VFHQ/videos --save_dir ./Datasets/VFHQ/boxes --workers 8

# 2) Generate meta json: [{"video_path": ".../videos/xxx.mp4"}, ...]
python tools/extract_meta_info.py --root_path ./Datasets/VFHQ --dataset_name VFHQ

Then set data.meta_paths in each training config:

data:
  meta_paths:
    - "./data/VFHQ_meta.json"
    - "./data/OtherDataset_meta.json"

3️⃣ Download weights

Download the training initialization weights: X-NeMo, pose_guider, and stylegan2_discriminator.

pretrained_weights
├── xnemo
│   ├── xnemo_motion_encoder.pth
│   ├── xnemo_denoising_unet.pth
│   ├── xnemo_reference_unet.pth
│   └── xnemo_temporal_module.pth
├── sd-vae-ft-mse
│   ├── diffusion_pytorch_model.bin
│   └── config.json
├── sd-image-variations-diffusers
│   ├── image_encoder
│   │   ├── pytorch_model.bin
│   │   └── config.json
│   ├── unet
│   │   ├── diffusion_pytorch_model.bin
│   │   └── config.json
│   └── model_index.json
├── pose_guider.pth
└── stylegan2_discriminator_ffhq512.pth

4️⃣ Training workflow

Stage 1: Image-level warm-up

Run:

accelerate launch train_stage1.py --config ./configs/train/personalive_stage1.yaml

Default output folder: ./exp_output/personalive_stage1/

Stage 2: Image-level adversarial refinement

Update configs/train/personalive_stage2.yaml to point to Stage 1 outputs:

motion_encoder_path: './exp_output/personalive_stage1/motion_encoder-xxxxx.pth'
denoising_unet_path: './exp_output/personalive_stage1/denoising_unet-xxxxx.pth'
reference_unet_path: './exp_output/personalive_stage1/reference_unet-xxxxx.pth'
pose_guider_path: './exp_output/personalive_stage1/pose_guider-xxxxx.pth'

Run:

accelerate launch train_stage2.py --config ./configs/train/personalive_stage2.yaml

Default output folder: ./exp_output/personalive_stage2/

Stage 3: Temporal module fine-tuning for streaming

Update configs/train/personalive_stage3.yaml to point to Stage 2 outputs:

motion_encoder_path: './exp_output/personalive_stage2/motion_encoder-xxxxx.pth'
denoising_unet_path: './exp_output/personalive_stage2/denoising_unet-xxxxx.pth'
reference_unet_path: './exp_output/personalive_stage2/reference_unet-xxxxx.pth'
pose_guider_path: './exp_output/personalive_stage2/pose_guider-xxxxx.pth'
discriminator_path: './exp_output/personalive_stage2/discriminator-xxxxx.pth'

Run:

accelerate launch train_stage3.py --config ./configs/train/personalive_stage3.yaml

Default output folder: ./exp_output/personalive_stage3/

📚 Community Contribution

Special thanks to the community for providing helpful setups! 🥂

🎬 More Results

👀 Visualization results

🤺 Comparisons

⭐ Citation

If you find PersonaLive useful for your research, welcome to cite our work using the following BibTeX:

@article{li2025personalive,
  title={PersonaLive! Expressive Portrait Image Animation for Live Streaming},
  author={Li, Zhiyuan and Pun, Chi-Man and Fang, Chen and Wang, Jue and Cun, Xiaodong},
  journal={arXiv preprint arXiv:2512.11253},
  year={2025}
}

❤️ Acknowledgement

This code is mainly built upon Moore-AnimateAnyone, X-NeMo, StreamDiffusion, RAIN and LivePortrait, thanks to their invaluable contributions.

PersonaLive (GVCLab/PersonaLive)

livestream
Searching Database…

4K ·GitHub
PersonaLive
What is this?PersonaLive is a real-time, streamable diffusion framework that animates a single portrait image into an infinite-length talking video, designed for live-streaming use cases. It was accepted to CVPR 2026.
Is this a ready-made app or research code I run myself?It is academic research code (Python, diffusion models) that ships with a WebUI for inference and a separate community-built ComfyUI integration, not a packaged consumer app.
How do I get it running?Install the base dependencies, download the pretrained base-model weights (sd-image-variations-diffusers and sd-vae-ft-mse) into ./pretrainedweights, then launch the WebUI to run inference on your own portrait image.
Do I need a serious GPU to run this?Yes, this is a diffusion model that needs a CUDA GPU; the streaming inference mode can generate long videos with as little as 12GB of VRAM.
Can I use the output for anything I want?No. The README states the project is released for academic research only and users must not use it to generate harmful, defamatory, or illegal content, with the authors disclaiming any responsibility for misuse.
Is this just inference, or can I also train my own model?Both: the repo ships pretrained weights for inference plus full training code organized into three stages (image warm-up, adversarial refinement, temporal fine-tuning), which took roughly 13h/15h/20h on 8x H100 GPUs in the authors' setup.
What can I tweak in the WebUI to manage latency?You can lower the Driving FPS setting to reduce the computational load, or increase the frame multiplier (numframesneeded) to better match your GPU's inference speed.
Is there a ComfyUI version?Yes, ComfyUI-PersonaLive is a separate community-built integration contributed by @okdalto that brings the model into the ComfyUI node-based workflow.
What prior work does PersonaLive build on?It's built on top of Moore-AnimateAnyone, X-NeMo, StreamDiffusion, RAIN, and LivePortrait, combining ideas from these projects into one real-time streaming pipeline.
Has anyone gotten it working on Windows?The community has shared a guide for running it on Windows with RTX 50-series (Blackwell) GPUs, including notes on TensorRT conversion, though it isn't officially packaged for Windows.
What are similar projects?SolarWM, Wan2.2, generative-models, PhotoMaker, LTX-Video, FramePack, InstantID, HunyuanVideo
Which topics is it in?cvpr, talking-head, livestream, diffusion, comfyui, diffusion-models, webui, video-generation, academic-research, image-generation, real-time, streaming
Creator
GVCLab
Size
150.3 MB
Stars
4K
Updated
CommitAug 28
Age
10mo
Commits · 90 Days