SMD README

项目的 README,来自 github.com/zjwfufu/SMD。合并为一个 markdown 文件:README.md。

Simplifying and Improving On-Policy Distillation
for Few-Step Diffusion

Project Page arXiv:2610.18472 Hugging Face Model Collection Datasets 中文文档

SMD examples: images on the left, 3D assets in the center, and video sequences on the right.

SMD is a simple and scalable method for few-step diffusion distillation across image, video, audio-video and 3D generation.

Models

See the model card for Hugging Face weights, standalone inference and example outputs.

The configurations below use four sampling steps unless noted otherwise. Model names link to their training configs.

Task Models
Text to image FLUX.1-dev, Qwen-Image, SD3.5-Medium, FLUX.2 Klein 4B, Lens
Image editing Qwen-Image-Edit-2511
Text to video Wan2.1 1.3B and 14B, causal Wan2.1 1.3B
Audio-video MiniMax H3 T2VA (4-step, 8-step) and Ref2VA (4-step, 8-step)
Image to 3D Hunyuan3D-2.1 Shape, TRELLIS.2 Shape and Texture

Additional configs cover SD3.5 reward specialists and multiple teachers, plus FLUX.2 9B distillation into Klein 4B and Lens.

Installation

Requirements:

The setup script creates an environment with Python 3.12, PyTorch 2.7.1, CUDA 12.6 wheels and FlashAttention.

Run from the repository root:

bash environments/setup_conda.sh
conda activate smd

For Hunyuan3D mesh previews and TRELLIS.2, install the additional environment:

bash environments/setup_3d_environment.sh
conda activate smd_3d_extend

Inference

For released weights, follow the standalone examples in the Hugging Face model collection.

For your own training checkpoints, export the student weights first, then run the inference script.

For example, with Qwen-Image:

python tools/download_resources.py --model qwen_image
python infer.py qwen_image --weights /path/to/exported_student \
  --output /path/to/samples --prompt "A red ceramic mug on a wooden table."

Pass your exported model directory to --weights and choose where to save samples with --output. The model directory must contain both student weights and sampling metadata.

See the inference guide for other models and options.

[!NOTE] H3 uses a student LoRA and cached conditions; no fused base checkpoint is needed. Export the adapter with the sampling metadata from the same checkpoint.

Launching with python defaults to one GPU; use --n-gpus to choose the GPU count. See H3 export and inference.

Training

Choose a configuration from the training experiments and prepare your data using the data guide.

For example, train Qwen-Image on two GPUs with a Parquet file containing a prompt column:

export SMD_OUTPUT_ROOT=/path/to/training_runs
torchrun --standalone --nproc_per_node=2 train.py \
  --config smd/configs/experiments/qwen_image.py --download-resources \
  - dataloader_train.dataset_path=/path/to/prompts.parquet \
  log_config.name=qwen-image-my-data

Replace the example data file and output directory with your own paths.

The --download-resources option downloads required resources automatically. Parameters after the standalone - override the selected configuration.

The SMD_OUTPUT_ROOT environment variable sets the root directory for checkpoints and previews. W&B logging defaults to offline mode.

See the training guide for run directories, resume, logging and preview settings.

Agent skills

To make SMD easier to use and extend, we provide agent skills for environment setup, training and inference, and model integration in the skills directory.

Each skill includes instructions and references that you or your coding assistant can follow.

Acknowledgments and license

SMD builds on NVIDIA FastGen's training infrastructure and uses model implementations from Hugging Face Diffusers.

We are grateful to the authors and maintainers of these projects for giving us a foundation to build on.

We thank the authors of FLUX.1, FLUX.2, Qwen-Image and Qwen-Image-Edit, Stable Diffusion 3.5, Wan2.1, MiniMax H3, Lens, Hunyuan3D-2.1 and TRELLIS.2 for sharing their models and code. These releases made it possible to explore SMD across image, video, audio-video and 3D generation.

We also thank the authors of Self Forcing and Causal Forcing for sharing their code and checkpoints. Our causal Wan implementation builds on Self Forcing and uses the chunkwise initialization and teacher checkpoints from Causal Forcing.

See LICENSE, NOTICE and third-party licenses for code licensing and attribution.

Model weights, teacher adapters and datasets retain their upstream licenses.

Citation

@article{smd,
  title   = {Simplifying and Improving On-Policy Distillation for Few-Step Diffusion},
  author  = {Jiawei Zhang and Ziyu Wan and Nanye Ma and Long Zhao and Dong Chen and Hongyu Liu and Qifeng Chen and Dongdong Chen},
  year    = {2026},
  journal = {arXiv preprint arXiv:2610.18472},
}

SMD (zjwfufu/SMD)

正在搜索数据库…

277 ·GitHub
这是什么?SMD 是一种研究方法及其代码库,用于将扩散模型蒸馏为快速的少步生成器,涵盖图像、视频、音视频和 3D 生成任务。
这是现成的应用还是研究代码库?这是一个研究代码库,包含训练和推理脚本以及配置文件,并不是打包好的应用程序。你需要用配置文件运行 Python 脚本,并使用导出的检查点。
怎样才能让它运行起来?在装有 Conda 的 Linux x86_64 机器上,于仓库根目录运行所提供的安装脚本,即可构建 Python 3.12 / PyTorch 2.7.1 / CUDA 12.6 环境,并从源码编译 FlashAttention;Hunyuan3D 和 TRELLIS.2 还需要另外一个额外环境。
我需要 NVIDIA 显卡和 CUDA 工具包吗?需要。必须有一块 NVIDIA 显卡,且驱动与 CUDA 12.6 兼容;另外还需要 CUDA 工具包和 C++ 编译器,以便从源码编译 FlashAttention。
能在 Windows 或 Mac 上运行吗?不能。README 要求使用装有 Conda 的 Linux x86_64 系统,且没有提及对 Windows 或 macOS 的支持。
这里的所有内容都是 Apache-2.0 协议吗?SMD 代码本身采用 Apache-2.0 协议,但模型权重、教师适配器和数据集则保留其各自上游的许可证。
哪些模型可以直接使用?Hugging Face 上发布的学生模型权重涵盖:文生图(FLUX.1-dev、Qwen-Image、SD3.5-Medium、FLUX.2 Klein 4B、Lens)、图像编辑(Qwen-Image-Edit-2511)、文生视频(Wan2.1)、音视频(MiniMax H3)以及图生 3D(Hunyuan3D-2.1、TRELLIS.2)。
我如何用自己的数据训练 SMD?选择一个训练配置,并按照数据指南准备一个包含 prompt 列的 Parquet 文件,然后在你的 GPU 上启动训练(例如在两块 GPU 上训练 Qwen-Image)。SMD_OUTPUT_ROOT 用于设置检查点和预览图的保存位置,W&B 日志默认为离线模式。
每个模型的推理方式都一样吗?大体一样:导出学生模型权重和采样元数据,然后使用 --weights 和 --output 参数运行推理脚本。MiniMax H3 是例外,它使用学生 LoRA 加缓存的条件,而不是融合后的检查点。
智能体技能(agent skills)有什么用?skills 目录包含用于环境搭建、训练、推理和模型集成的智能体技能。这些内容编写得便于你或编程助手按步骤操作,从而更轻松地使用和扩展 SMD。
作者
zjwfufu
大小
27 MB
星标
277
更新
提交Oct 21
时长
1个月
提交 · 90 天