What is this?
DeepSpeed is a deep learning optimization library that makes distributed training and inference of very large models easy, efficient and effective — it powered models like MT-530B and BLOOM.
DeepSpeed is a deep learning optimization library that makes distributed training and inference of very large models easy, efficient and effective — it powered models like MT-530B and BLOOM.Is it a library I plug into my PyTorch code, or a separate platform I deploy?It's a Python library you install on top of PyTorch and call from your own training code — not a service or a standalone app.
It's a Python library you install on top of PyTorch and call from your own training code — not a service or a standalone app.What does my machine need before installing?PyTorch must already be installed (2.0+ recommended), plus a CUDA or ROCm compiler such as nvcc or hipcc to compile the C++/CUDA/HIP extensions.
PyTorch must already be installed (2.0+ recommended), plus a CUDA or ROCm compiler such as nvcc or hipcc to compile the C++/CUDA/HIP extensions.Does everything work on Windows, or are there gaps?Many training and inference features work on Windows, but async io (AIO) and GDS are unsupported, and building there needs Visual C++ build tools plus an admin console to run buildwin.bat.
Many training and inference features work on Windows, but async io (AIO) and GDS are unsupported, and building there needs Visual C++ build tools plus an admin console to run buildwin.bat.Do I need a big GPU cluster, or does it help on a single machine too?It's aimed at extreme scale — ZeRO-Infinity is literally about breaking the GPU memory wall — though the ZeRO-Offload line of work focuses on democratizing billion-scale training for more modest hardware.
It's aimed at extreme scale — ZeRO-Infinity is literally about breaking the GPU memory wall — though the ZeRO-Offload line of work focuses on democratizing billion-scale training for more modest hardware.Will it work with my GPU — AMD, Intel, or only NVIDIA?Most tested on NVIDIA Pascal, Volta, Ampere and Hopper and AMD MI100/MI200; Huawei Ascend NPU, Intel Gaudi 2, Xeon CPUs, Intel GPU Max and Tecorigin SDAA are contributor-supported.
Most tested on NVIDIA Pascal, Volta, Ampere and Hopper and AMD MI100/MI200; Huawei Ascend NPU, Intel Gaudi 2, Xeon CPUs, Intel GPU Max and Tecorigin SDAA are contributor-supported.What is ZeRO?It's DeepSpeed's headline memory optimization; together with ZeRO-Infinity, 3D-Parallelism, Ulysses Sequence Parallelism and DeepSpeed-MoE it forms the core innovation set behind extreme-scale training.
It's DeepSpeed's headline memory optimization; together with ZeRO-Infinity, 3D-Parallelism, Ulysses Sequence Parallelism and DeepSpeed-MoE it forms the core innovation set behind extreme-scale training.How large are the models people train with it?Hundreds of billions of parameters: Megatron-Turing NLG 530B, Jurassic-1 178B, BLOOM 176B, GLM 130B, YaLM 100B — and DeepSpeed targets the trillion-parameter range.
Hundreds of billions of parameters: Megatron-Turing NLG 530B, Jurassic-1 178B, BLOOM 176B, GLM 130B, YaLM 100B — and DeepSpeed targets the trillion-parameter range.Does it work with Hugging Face Transformers?Yes — it integrates with Transformers and Accelerate, plus Lightning, MosaicML, Determined and MMEngine, so you can often add it to an existing stack.
Yes — it integrates with Transformers and Accelerate, plus Lightning, MosaicML, Determined and MMEngine, so you can often add it to an existing stack.Can it handle very long input sequences?Yes — the project's Arctic Long Sequence Training (ALST) work with DeepSpeed targets scalable, efficient training on multi-million-token sequences.
Yes — the project's Arctic Long Sequence Training (ALST) work with DeepSpeed targets scalable, efficient training on multi-million-token sequences.What are similar projects?ColossalAI, verl, mamba, ktransformers, LMCache, petals, DeepGEMM, vllm
ColossalAI, verl, mamba, ktransformers, LMCache, petals, DeepGEMM, vllmWhich topics is it in?intel, rocm, mixture-of-experts, compression, amd, training, microsoft, gpu-acceleration, large-language-models, distributed-systems, inference, nvidia
intel, rocm, mixture-of-experts, compression, amd, training, microsoft, gpu-acceleration, large-language-models, distributed-systems, inference, nvidia