MatAnyone-E README

项目的 README,来自 github.com/Shuyun-Wang/MatAnyone-E。合并为一个 markdown 文件:README.md。

MatAnyone-E Logo

Sparse Memory Propagation for Efficient High-Resolution Video Matting

Shuyun Wang1,2  Shangchen Zhou2†  Peiqing Yang2  Yuekun Dai2  Xin Yu3*  Chen Change Loy2*
1The University of Queensland  2S-Lab, Nanyang Technological University  3University of Adelaide
†Project lead *Corresponding authors

4K speed race: MatAnyone 2 at 2.3 FPS vs MatAnyone-E at 28.6 FPS on one H200

MatAnyone-E mattes native 4K video at 28.6 FPS on a single H200 (with torch.compile), 12× faster than MatAnyone 2, using 3.1 GB of GPU memory, while keeping fine details such as hair strands.

:movie_camera: More results on the project page.


📮 Update

🏄🏻‍♀️ TODO

🔎 Overview

overview

MatAnyone-E keeps the architecture of MatAnyone 2 and replaces the two parts whose cost grows fastest with resolution:

⚡ Efficiency

FPS and peak GPU memory on one NVIDIA A800-SXM4-80GB (fp16); (C.) = with torch.compile.

Resolution MatAnyone 2 Ours Ours (C.) MatAnyone 2 memory Ours memory Ours (C.) memory
1024×1024 29.5 41.3 47.3 1.60 GB 0.92 GB 0.61 GB
1920×1080 13.6 26.6 37.6 5.47 GB 1.66 GB 1.07 GB
2560×1440 5.6 16.5 25.1 16.20 GB 3.04 GB 2.02 GB
3840×2160 ~1.1 8.1 13.6 ~79.69 GB 6.08 GB 3.89 GB

~ extrapolated (a full memory bank does not fit in 80 GB). On one H200 with torch.compile: 73.07 FPS at 1080p, 28.62 FPS at 4K.

📑 Citation

If you find our repo useful for your research, please consider citing our paper:

@article{wang2026matanyonee,
  title   = {{MatAnyone-E}: Sparse Memory Propagation for Efficient High-Resolution Video Matting},
  author  = {Shuyun Wang and Shangchen Zhou and Peiqing Yang and Yuekun Dai and Xin Yu and Chen Change Loy},
  journal = {arXiv preprint arXiv:2610.35724},
  year    = {2026}
}

📝 License

This project is licensed under NTU S-Lab License 1.0. Redistribution and use should follow this license.

👏 Acknowledgement

This project is built upon MatAnyone 2 and MatAnyone. Thanks for their awesome works!

📧 Contact

If you have any questions, please feel free to reach us at [email protected].

MatAnyone-E (Shuyun-Wang/MatAnyone-E)

正在搜索数据库…

这是什么?MatAnyone-E 是一个用于高分辨率视频抠像的研究模型,即把视频分离为前景和背景并生成 alpha 遮罩。与原版 MatAnyone 2 相比,它的运行速度快得多,所需 GPU 显存也少得多,同时仍能保留发丝等细节。
这是一个全新的模型,还是某个已有模型的改进版本?它保留了 MatAnyone 2 的架构,但替换了两个随分辨率提升而成本增长最快的部分:一个是稀疏记忆传播模块,它只对经过修剪的查询与记忆配对进行高分辨率匹配;另一个是通过蒸馏原版解码器训练得到的轻量级解码器。
我现在就能自己运行它吗?暂时不行。目前只发布了论文和项目页面;推理代码和模型权重、Hugging Face 演示以及 ComfyUI 自定义节点仍标记为 TODO。
代码发布后,我需要什么硬件?需要一块 CUDA GPU,显存大小取决于目标分辨率。论文的基准测试使用的是 NVIDIA A800-80GB 和 H200。即使使用 torch.compile 优化,原生 4K 抠像至少仍需约 3.9 GB 显存。
无论分辨率多大,它都能保持快速和轻量吗?显存占用仍会随分辨率增长。在 4K 分辨率下,作者甚至无法直接测量 MatAnyone 2 的显存占用,只能进行推算,因为完整的记忆库无法装进 80GB 的 GPU 显存。
为什么叫“稀疏记忆传播”?先通过一次低成本的低分辨率计算,找出哪些查询位置不确定,以及哪些已存储的记忆帧对它们真正有用;然后,高成本的高分辨率匹配只针对这一小部分经过修剪的配对进行,而不是对所有内容都执行。
它比原版 MatAnyone 2 快多少?在 H200 上使用 torch.compile 处理原生 4K 视频时,它可达到 28.6 FPS,而 MatAnyone 2 约为 1.1 FPS,快约 12 倍。同时,它的 GPU 显存占用为 3.1 GB,而 MatAnyone 2 推算约需 80 GB。
大小
3.2 MB
星标
196
更新
提交Nov 05
时长
27天
提交 · 90 天