DreamX-Creator README

The project's README from github.com/AMAP-ML/DreamX-Creator. As one markdown file: README.md.

DreamX-Creator teaser

DreamX-Creator 1.0: Democratizing Native Audio-Video Generation at 2K Resolution

DreamX Team

arXiv HuggingFace ModelScope License

DreamX-Creator 1.0 is a research framework for native joint audio-video generation. Given a first frame and a text prompt, its implemented base generator jointly models modality-specialized video and audio streams, using Gated Cross-Modal Attention and Progressive Joint Training to enable bidirectional audio-video interaction.

The broader system combines Audio-Video Reinforcement Learning with Modality-Aware Multimodal Feedback to improve visual and audio quality, semantic consistency, and fine-grained audio-video synchronization. Autoregressive 1-Step 2K Refinement then upgrades the generated video to high-quality 2K output while preserving content, motion, and audio-aligned timing.

:clapper: Demo

:fire: News

:calendar: Plan

:open_file_folder: Repository Structure

:package: Model Weights

Model weights are distributed on HuggingFace and ModelScope and should be placed under checkpoints/ (details in checkpoints/README.md):

checkpoints/
├── creator/                     # DreamX-Creator 1.0 joint generator (7B, LoRA merged)
│   ├── video_model/             # video DiT shards + config
│   ├── audio_model/             # audio DiT + config
│   └── cross_attn_weights.safetensors  # gated A2V/V2A cross-modal attention
├── audio_vae/                   # CreatorDACVAE audio VAE
├── refiner/                     # 2K refiner
│   ├── sr_dit_5b.pt             # SR-DiT 5B refiner
│   ├── latent_upsampler_flash.pt       # FlashLatentUpsampler (default)
│   ├── latent_upsampler_2d_causal.pt   # causal 2D latent upsampler (optional)
│   └── lightvae_nu_scheme3.pt          # distilled fast decoder (optional, off by default)
└── wan2.2_ti2v_5b/              # shared Wan2.2-TI2V-5B dependencies
    ├── Wan2.2_VAE.pth           # video VAE
    ├── models_t5_umt5-xxl-enc-bf16.pth  # UMT5-xxl text encoder
    └── google/umt5-xxl/         # tokenizer

The wan2.2_ti2v_5b/ directory can also be downloaded directly from Wan-AI/Wan2.2-TI2V-5B; only the three entries above are needed.

:rocket: Quickstart

Each subdirectory is self-contained with its own requirements.txt and README.

1. Joint audio-video generation (first frame + prompt to synchronized video with audio):

cd audio_video_generation
pip install -r requirements.txt
./inference.sh                        # runs the default Verse-Bench case (case1)

See audio_video_generation/README.md for input overrides, CPU-offload options, output details, and multi-GPU sequence-parallel inference.

2. 2K refinement (super-resolve a generated or external video, audio unchanged):

cd ../video_refiner
pip install -r requirements.txt
INPUT=/path/to/video.mp4 bash run_inference.sh

See video_refiner/README.md for the full list of knobs (KV cache, window attention, speed/quality trade-offs).

:books: Citation

If you find DreamX-Creator useful in your research, please consider citing our technical report:

@misc{zhu2026dreamxcreatordemocratizingnativeaudiovideo,
  title={DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution},
  author={Jiashu Zhu and Yanhao Zheng and Ruitian Tian and Rujing Dang and Shen Zhang and Bingze Song and Jiachen Lei and Ruimin Lin and Jiahong Wu and Xiangxiang Chu},
  year={2026},
  eprint={2608.31106},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2608.31106},
}

:scroll: License

This project is licensed under the Apache License 2.0. See LICENSE for details.

:sparkles: Acknowledgement

We would like to thank the Wan Team, the OpenMOSS Team, and the VideoX-Fun Team for their outstanding open-source work on Wan, MOVA, and VideoX-Fun, respectively.

DreamX-Creator (AMAP-ML/DreamX-Creator)

Searching Database…

312 ·GitHub
DreamX-Creator
What is this?DreamX-Creator is an open-source research framework for native joint audio-video generation: it produces video and a synchronized audio track together, refined up to 2K resolution.
What do you feed it, and what do you get back?You give it a first frame plus a text prompt; the 7B generator returns video with a matching audio track, and the SR-DiT 5B refiner then upgrades the result to high-quality 2K.
How do I get it running?Each component (audiovideogeneration/, videorefiner/) is a self-contained folder with its own requirements.txt and README: install that environment, download the weights into checkpoints/, then run its inference scripts.
Which model weights do I need, and from where?From HuggingFace or ModelScope: the DreamX 7B audio-video generator, the SR-DiT 5B refiner, and Wan-AI/Wan2.2-TI2V-5B — all placed under checkpoints/ as its README describes.
Is everything described in the technical report actually available?Only the inference side: released weights and code cover the 7B generator and the 1-step 2K refiner, while distilled, faster models with fewer sampling steps are still an unchecked roadmap item.
Can I use it freely, including commercially?Yes — Apache 2.0 lets you use, modify and redistribute it, including in commercial work, as long as you keep the license.
Will it run on an ordinary machine, or do I need a serious GPU?The 7B generator is built to run on a single GPU, with CPU-offload and memory options for tighter setups and optional multi-GPU sequence-parallel inference for faster runs.
How does it keep audio and video in sync?The generator uses Gated Cross-Modal Attention with Progressive Joint Training for bidirectional audio-video interaction, and the broader pipeline adds audio-video reinforcement learning with modality-aware multimodal feedback to improve synchronization, consistency and quality.
Why does it need a first frame instead of just a prompt?The released base generator is image-conditioned: your first frame anchors the scene while the prompt drives the motion and sound. The README does not mention a prompt-only mode.
Is this built on other open-source work?Yes — the team credits the Wan, OpenMOSS and VideoX-Fun teams and reuses Wan2.2-TI2V-5B weights; there is also a technical report you can cite if the framework helps your research.
What are similar projects?LTX-Video, hyperflow, DiffSynth-Studio, LTX-2, diffusers, Open-Sora, MiniMax-H3, CogVideo
Which topics is it in?wan, super-resolution, diffusion, image-to-video, audio-generation, attention-mechanism, text-to-video, reinforcement-learning, multimodal, video-generation, audio, video
Creator
AMAP-ML
Size
3.6 MB
Stars
312
Updated
CommitSep 03
Age
1mo
Commits · 90 Days