Office Hours
DeepSpeed hosts regular office hours on the last Tuesday of each month at 12:00 America/New_York to discuss development plans, features, etc. This meeting is public for anyone to join and ask questions.
The meeting is hosted on Zoom and can be joined here.
Latest News
More news
Extreme Speed and Scale for DL Training
DeepSpeed enabled the world's most powerful language models (at the time of this writing) such as MT-530B and BLOOM. DeepSpeed offers a confluence of system innovations, that has made large scale DL training effective, and efficient, greatly improved ease of use, and redefined the DL training landscape in terms of scale that is possible. These innovations include ZeRO, ZeRO-Infinity, 3D-Parallelism, Ulysses Sequence Parallelism, DeepSpeed-MoE, etc.
DeepSpeed Adoption
DeepSpeed was an important part of Microsoft’s
AI at Scale
initiative to enable next-generation AI capabilities at scale, where you can find more
information here.
DeepSpeed has been used to train many different large-scale models, below is a list of several examples that we are aware of (if you'd like to include your model please submit a PR):
DeepSpeed has been integrated with several different popular open-source DL frameworks such as:
Build Pipeline Status
| Description |
Status |
| NVIDIA |
  |
| AMD |
 |
| CPU |
 |
| Intel Gaudi |
 |
| Intel XPU |
 |
| Integrations |
 |
| Misc |
    |
| Huawei Ascend NPU |
 |
Installation
The quickest way to get started with DeepSpeed is via pip, this will install
the latest release of DeepSpeed which is not tied to specific PyTorch or CUDA
versions. DeepSpeed includes several C++/CUDA extensions that we commonly refer
to as our 'ops'. By default, all of these extensions/ops will be built
just-in-time (JIT) using torch's JIT C++ extension loader that relies on
ninja to build and
dynamically link them at runtime.
Requirements
- PyTorch must be installed before installing DeepSpeed.
- For full feature support we recommend a version of PyTorch that is >= 2.0 and ideally the latest PyTorch stable release.
- A CUDA or ROCm compiler such as nvcc or hipcc used to compile C++/CUDA/HIP extensions.
- Specific GPUs we develop and test against are listed below, this doesn't mean your GPU will not work if it doesn't fall into this category it's just DeepSpeed is most well tested on the following:
- NVIDIA: Pascal, Volta, Ampere, and Hopper architectures
- AMD: MI100 and MI200
Contributed HW support
- DeepSpeed now support various HW accelerators.
| Contributor |
Hardware |
Accelerator Name |
Contributor validated |
Upstream validated |
| Huawei |
Huawei Ascend NPU |
npu |
Yes |
No |
| Intel |
Intel(R) Gaudi(R) 2 AI accelerator |
hpu |
Yes |
Yes |
| Intel |
Intel(R) Xeon(R) Processors |
cpu |
Yes |
Yes |
| Intel |
Intel(R) Data Center GPU Max series |
xpu |
Yes |
Yes |
| Tecorigin |
Scalable Data Analytics Accelerator |
sdaa |
Yes |
No |
PyPI
We regularly push releases to PyPI and encourage users to install from there in most cases.
pip install deepspeed
After installation, you can validate your install and see which extensions/ops
your machine is compatible with via the DeepSpeed environment report.
ds_report
If you would like to pre-install any of the DeepSpeed extensions/ops (instead
of JIT compiling) or install pre-compiled ops via PyPI please see our advanced
installation instructions.
Windows
Many DeepSpeed features are supported on Windows for both training and inference. You can read more about this in the original blog post here. Among features that are currently not supported are async io (AIO) and GDS (which does not support Windows).
- Install PyTorch, such as pytorch 2.3+cu121.
- Install Visual C++ build tools, such as VS2022 C++ x64/x86 build tools.
- Launch Cmd console with Administrator permissions for creating required symlink folders and ensure MSVC tools are added to your PATH or launch the Developer Command Prompt for Visual Studio 2022 with administrator permissions.
- Run
build_win.bat to build wheel in dist folder.
Further Reading
All DeepSpeed documentation, tutorials, and blogs can be found on our website: deepspeed.ai
CI funding
This being an open source project we rely on others to provide us resources for CI hardware. At this moment Modal is kindly supporting our GPU CI runs by funding the hardware for us. Modal is an AI infrastructure platform for inference, fine-tuning, batch jobs and more. Get started with $30/mo in free credits today at https://modal.com. We have been getting an amazing support from Modal's team and will surely recommend them to your business.
Contributing
DeepSpeed welcomes your contributions! Please see our
contributing guide for more details on formatting, testing,
etc.
Thanks so much to all of our amazing contributors!
Developer Certificate of Origin
This project welcomes contributions and suggestions. Most contributions require you to
agree to a Developer Certificate of Origin DCO
stating that they agree to the terms published at https://developercertificate.org for
that particular contribution.
DCOs are per-commit, so each commit needs to be signed off. These can be signed in
the commit by adding the -s flag. DCO enforcement can also be signed off in the PR
itself by clicking on the DCO enforcement check.
Code of Conduct
This project has adopted the Microsoft Open Source Code of
Conduct. For more information see the
Code of Conduct FAQ or contact
[email protected] with any additional questions or comments.
Publications
- Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He. (2019) ZeRO: memory optimizations toward training trillion parameter models. arXiv:1910.02054 and In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '20).
- Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. (2020) DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '20, Tutorial).
- Minjia Zhang, Yuxiong He. (2020) Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping. arXiv:2010.13369 and NeurIPS 2020.
- Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, Yuxiong He. (2021) ZeRO-Offload: Democratizing Billion-Scale Model Training. arXiv:2101.06840 and USENIX ATC 2021. [paper] [slides] [blog]
- Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan, Samyam Rajbhandari, Conglong Li, Xiangru Lian, Ji Liu, Ce Zhang, Yuxiong He. (2021) 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed. arXiv:2102.02888 and ICML 2021.
- Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He. (2021) ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning. [arXiv:2104.07857](https:/
DeepSpeed 是一个深度学习优化库，让超大规模模型的分布式训练和推理变得简单、高效、有效——MT-530B 和 BLOOM 等模型就是靠它训练出来的。