ComfyUI-Ref2VA-VSA README

The project's README from github.com/Kablex/ComfyUI-Ref2VA-VSA. As one markdown file: README.md.

ComfyUI-Ref2VA-VSA: Ultra-Fast Character Video Generation (72s on RTX 4090)

ComfyUI MiniMax-H3 Speed Hardware

Breakthrough inference speed for MiniMax H3 Reference-to-Video (Ref2VA / R2VA): Generate a full 5-second, 24 fps video conditioned on reference character images in ~72s (warm) / 95s (first run) on a single consumer NVIDIA GeForce RTX 4090 (24GB).

🎬 Empirical Benchmark & Results

Both videos were generated on the exact same NVIDIA GeForce RTX 4090 (24GB), using the exact same prompt, character reference image, seed (981445682258077), and resolution (1344×768, 5.16s @ 24 fps / 124 frames).

Input Reference Character
Input Character Reference Image

🚀 Ref2VA VSA (Ours - 4 Steps)

⏱️ 95s (1st run) / ~72s (warm)

🔬 Video Delta Net / VDN-H3 (8 Steps)

⏱️ 213s (1st run) / ~135s (warm)
Ref2VA VSA Preview
▶ Download Full 1344x768 Video (MP4) 		VDN-H3 Preview
▶ Download Full 1344x768 Video (MP4)
  • Sampling: 4 steps (euler / simple)
  • Attention: 75% video sparsity top-k (sparse DiT)
  • Speedup: 2.24x faster than VDN; 9x faster than native H3
  • Peak VRAM: ~13.5 GB
  • Sampling: 8 steps (er_sde / beta)
  • Attention: Hybrid (windowed dense + linear delta state)
  • Speedup: ~3.5x faster than native H3
  • Peak VRAM: ~18.5 GB

📊 Comprehensive Performance Comparison (RTX 4090)

Pipeline Attention Mechanism Steps / Sampler 1st Run Latency Warm Wall Time Peak VRAM
MiniMax H3 Native Dense Dense Global Softmax 50 steps / Euler ~650s ~650s (10.8 min) ~22.5 GB
H3 Turbo LoRA (Dense) Dense Global Softmax 4 steps / Euler ~210s ~190s (3.1 min) ~21.0 GB
Video Delta Net (VDN-H3) Windowed + Linear Delta 8 steps / er_sde 213.0s ~135s (2.2 min) ~18.5 GB
Ref2VA VSA (This Repo) 🚀 75% Video Sparsity + Dense Prefix 4 steps / Euler 95.2s ~72 seconds ~13.5 GB

🧠 Technical Architecture: Why Ref2VA was Hard for VSA

FastVideo pioneered Visual Sparse Attention (VSA) for text-to-video (T2V) by clustering video tokens into 3D spatial-temporal tiles (4×4×4) and dynamically pruning 75–90% of tiles via a learned gating projection (to_gate_compress).

However, Ref2VA (Reference-to-Video) was widely considered incompatible with VSA because:

  1. Ref2VA prepends dynamic multimodal condition segments (reference image latents, reference audio latents, and text prompt tokens) ahead of the generated video sequence.
  2. Naive 3D tiling causes tokens from different modalities to straddle the same tile, corrupting the conditioning masks and breaking identity fidelity.

The Solution: Ref2VAVSAGatePatch

Our patch resolves this with an engineered two-tier attention layout:

  1. Multimodal Dense-Exempt Prefix: The geometry mapper isolates text, reference image, and reference audio tokens into segment-pure tiles that are completely exempt from top-k pruning. Reference tokens always remain 100% dense, guaranteeing strict character identity adherence.
  2. Video-Only Sparse Attention: VSA top-k pruning is applied strictly to generated-video key tiles.
  3. Gate Transplant: The 50 learned to_gate_compress projection matrices are transplanted directly onto the Ref2VA base model (minimax_h3_ref2va_pruned_int8_convrot.safetensors).

📦 Requirements

Place the following files in your ComfyUI/models/ directories:

Model Type File Name Destination Directory Download Link
Diffusion Model minimax_h3_ref2va_pruned_int8_convrot.safetensors models/diffusion_models/ Download from Comfy-Org
VSA Gate fasth3_vsa_gate.safetensors models/loras/ Download from Hugging Face
Turbo LoRA (4-Step) minimax_h3_ref2v_turbo_4step_v0.1_comfyui_bf16.safetensors models/loras/ Download from LightX2V
Text Encoder qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors models/text_encoders/ Download from Comfy-Org
Video VAE minimax_h3_video_vae_fp16.safetensors models/vae/ Download from Comfy-Org
Audio VAE minimax_h3_audio_vae_fp32.safetensors models/vae/ Download from Comfy-Org

💡 Tip: The VSA Gate can also be extracted locally from any official FastH3 base checkpoint using the included tools/extract_vsa_gate.py script.

🚀 Quick Start

  1. Clone or copy this repository into your ComfyUI custom_nodes directory:

    cd ComfyUI/custom_nodes
git clone https://github.com/Kablex/ComfyUI-Ref2VA-VSA

  2. Copy the example character image into your ComfyUI input folder:

    cp ComfyUI-Ref2VA-VSA/assets/example_character.jpg ComfyUI/input/

  3. Restart ComfyUI. The node appears under: FastH3/VSA -> Ref2VA VSA Gate Transplant (EXPERIMENTAL)

  4. Load the ready-to-use workflow from workflows/ref2va_vsa_4step_rtx4090.json and click Queue Prompt.

🛠️ Node Wiring

[UNETLoader (Ref2VA INT8)]
        │
        ▼
[LoraLoaderModelOnly (Turbo 4-Step LoRA, strength=1.0)]
        │
        ▼
[Ref2VAVSAGatePatch (fasth3_vsa_gate.safetensors, sparsity=0.75)]
        │
        ▼
[MiniMaxH3SigmaShift (shift_video=12, shift_audio=3)]
        │
        ├──► [BasicScheduler (steps=4, scheduler=simple)]
        │
        └──► [BasicGuider] ──► [SamplerCustomAdvanced (sampler=euler)]

🧰 Extracting the Gate File (tools/extract_vsa_gate.py)

If you have a FastH3 VSA base checkpoint (minimax_h3_fastvideo_vsa_datafree_1300step_4step_int8_convrot.safetensors), you can extract fasth3_vsa_gate.safetensors yourself:

python tools/extract_vsa_gate.py \
    --input /path/to/minimax_h3_fastvideo_vsa_datafree_1300step_4step_int8_convrot.safetensors \
    --output /path/to/ComfyUI/models/loras/fasth3_vsa_gate.safetensors

📄 License & Acknowledgements

ComfyUI-Ref2VA-VSA (Kablex/ComfyUI-Ref2VA-VSA)

Searching Database…

743 ·GitHub
ComfyUI-Ref2VA-VSA
What is this?It's a ComfyUI custom node that dramatically speeds up Reference-to-Video (Ref2VA) generation for the MiniMax H3 model, producing a full 5-second, 24fps character video in about 72 seconds on a single RTX 4090 instead of over 10 minutes.
How do I get this running in ComfyUI?Clone the repo into your ComfyUI custom_nodes directory, download the required checkpoint files (diffusion model, VSA gate, turbo LoRA, text encoder, video/audio VAE) into their matching ComfyUI/models subfolders, restart ComfyUI, then load the ready-made workflow from workflows/ref2va_vsa_4step_rtx4090.json.
What hardware and software does this need?You need ComfyUI with native MiniMax-H3 support, comfy-kitchen with CUDA Sol-Attention support, PyTorch 2.4+/CUDA 12.1+, and an NVIDIA GPU with 24GB VRAM such as an RTX 3090, RTX 4090, A5000, or L40S.
Is this ready for everyday use?The node is explicitly flagged as experimental in the UI and requires manually downloading several large checkpoint files into specific ComfyUI model folders before it works.
Can I run this on a smaller consumer GPU?No, it needs at least a 24GB-VRAM NVIDIA card; peak usage during generation is about 13.5GB, but the checkpoints and pipeline are built around 24GB-class GPUs.
Why was Ref2VA considered incompatible with sparse attention before this?Ref2VA prepends reference image, audio, and text tokens ahead of the video sequence, and naive 3D tiling mixed tokens from different modalities into the same tile, corrupting conditioning masks and breaking character identity.
What is VSA (Visual Sparse Attention)?VSA is a FastVideo technique that clusters video tokens into 3D spatio-temporal tiles and prunes 75-90% of them via a learned gating projection, cutting attention compute for video diffusion models.
How much faster is this than the standard MiniMax H3 pipeline?It's about 9x faster than the native 50-step dense H3 pipeline and 2.24x faster than the alternative Video Delta Net (VDN-H3) speedup approach.
Can I extract the VSA gate file myself instead of downloading it?Yes, the included tools/extract_vsa_gate.py script can pull the gate weights from any official FastH3 VSA base checkpoint.
What are similar projects?FreeVideo, ComfyUI-YCNodes-MiniMax-H3, LTX-Video, CogVideo, SparkDiffusion, Evoke, vdn-minimax-h3, hyperflow
Which topics is it in?comfyui-nodes, image-to-video, attention-mechanism, diffusion-models, high-performance, gpu-acceleration, quantization, multimodal, video-generation, inference, nvidia, generative-ai
Creator
Kablex
Size
4.1 MB
Stars
743
Updated
CommitOct 17
Age
1mo
Commits · 90 Days