12.2K ·GitHub
What is this?
Open-Sora Plan is an open-source community project that aims to reproduce OpenAI's Sora text-to-video model, led by a PKU-affiliated AIGC lab with contributions from Huawei and Pengcheng Lab.
Open-Sora Plan is an open-source community project that aims to reproduce OpenAI's Sora text-to-video model, led by a PKU-affiliated AIGC lab with contributions from Huawei and Pengcheng Lab.Is this a ready-to-use video generator, or a research codebase for training models?It is a research codebase: training/inference code, architecture reports, and downloadable model checkpoints for building and running video diffusion models yourself, not a polished consumer app.
It is a research codebase: training/inference code, architecture reports, and downloadable model checkpoints for building and running video diffusion models yourself, not a polished consumer app.How do I get it running?For NPU hardware, check out the mindspeedmmdit branch and follow its README for configuration; a GPU version is still being prepared and marked as not yet available.
For NPU hardware, check out the mindspeedmmdit branch and follow its README for configuration; a GPU version is still being prepared and marked as not yet available.Do I need an NVIDIA GPU, or does it run on something else?The current v1.5.0 release was fully trained and is run on Huawei Ascend 910-series NPUs using the MindSpeed-MM framework, not standard GPUs.
The current v1.5.0 release was fully trained and is run on Huawei Ascend 910-series NPUs using the MindSpeed-MM framework, not standard GPUs.Is the GPU version ready to use right now?No, the GPU setup path is explicitly listed as not yet implemented; only the NPU path is documented and usable today.
No, the GPU setup path is explicitly listed as not yet implemented; only the NPU path is documented and usable today.What is WFVAE?WFVAE is the project's custom video autoencoder that achieves higher PSNR than the VAE used in Wan2.1 at an 8x8x8 downsampling rate, which lowers training cost for the diffusion model built on top of it.
WFVAE is the project's custom video autoencoder that achieves higher PSNR than the VAE used in Wan2.1 at an 8x8x8 downsampling rate, which lowers training cost for the diffusion model built on top of it.What is the SUV sparse DiT architecture?SUV (Skiparse 3D) is a sparse attention diffusion transformer design used in v1.3.0 and v1.5.0 that nearly matches dense DiT performance while running over 35% faster.
SUV (Skiparse 3D) is a sparse attention diffusion transformer design used in v1.3.0 and v1.5.0 that nearly matches dense DiT performance while running over 35% faster.Can it generate image-to-video, not just text-to-video?Yes, starting with v1.2.0 the project added an image-to-video model that also supports transition generation using starting and ending frame conditions.
Yes, starting with v1.2.0 the project added an image-to-video model that also supports transition generation using starting and ending frame conditions.How does v1.5.0 compare to other open video models?By combining the higher-compression WFVAE with the SUV sparse DiT, v1.5.0 reaches performance comparable to HunyuanVideo (Open-Source) using an 8B-parameter model trained on 40 million video samples.
By combining the higher-compression WFVAE with the SUV sparse DiT, v1.5.0 reaches performance comparable to HunyuanVideo (Open-Source) using an 8B-parameter model trained on 40 million video samples.What are similar projects?SolarWM, Wan2.2, LTX-Video, Evoke, TRELLIS.2, LongCat-Video, Wan2.1, SparkDiffusion
SolarWM, Wan2.2, LTX-Video, Evoke, TRELLIS.2, LongCat-Video, Wan2.1, SparkDiffusionWhich topics is it in?generative-model, ai-research, aigc, attention, image-to-video, text-to-video, diffusion-models, video-generation, academic-research, transformer, artificial-intelligence, generative-ai
generative-model, ai-research, aigc, attention, image-to-video, text-to-video, diffusion-models, video-generation, academic-research, transformer, artificial-intelligence, generative-ai