18.2K ·GitHub
What is this?
NVIDIA's open-source toolkit for training large transformer (LLM) models at scale. It comes as Megatron-LM, a reference setup with ready training scripts, and Megatron Core, a composable library of GPU-optimized building blocks for training models across thousands of GPUs.
NVIDIA's open-source toolkit for training large transformer (LLM) models at scale. It comes as Megatron-LM, a reference setup with ready training scripts, and Megatron Core, a composable library of GPU-optimized building blocks for training models across thousands of GPUs.Is it one library or several components?Two components in one repo: Megatron-LM is a reference example with pre-configured training scripts, while Megatron Core is a composable library meant to be built into custom training frameworks. A separate tool, Megatron Bridge, converts checkpoints between Hugging Face and Megatron formats.
Two components in one repo: Megatron-LM is a reference example with pre-configured training scripts, while Megatron Core is a composable library meant to be built into custom training frameworks. A separate tool, Megatron Bridge, converts checkpoints between Hugging Face and Megatron formats.How do I get it installed?Install from PyPI, or clone the repo and install from source. Source builds are memory-hungry, so if compilation runs out of memory cap the parallel jobs, e.g. MAXJOBS=4 uv pip install -e . NVIDIA's NGC containers and the Installation Guide cover the remaining options, and a Your First Training Run guide picks up from there.
Install from PyPI, or clone the repo and install from source. Source builds are memory-hungry, so if compilation runs out of memory cap the parallel jobs, e.g. MAXJOBS=4 uv pip install -e . NVIDIA's NGC containers and the Installation Guide cover the remaining options, and a Your First Training Run guide picks up from there.Any version or compatibility gotchas I should know about?Python 3.10 support is going away: the upcoming 0.17.0 release requires Python 3.12 or newer, so downstream projects must raise their floor. Building from source can also exhaust memory on modest machines.
Python 3.10 support is going away: the upcoming 0.17.0 release requires Python 3.12 or newer, so downstream projects must raise their floor. Building from source can also exhaust memory on modest machines.Who is this for — researchers or framework developers?Both, split by component: research teams learning distributed training and running quick experiments use the ready-made Megatron-LM scripts, while framework developers and ML engineers building custom training pipelines use Megatron Core's building blocks.
Both, split by component: research teams learning distributed training and running quick experiments use the ready-made Megatron-LM scripts, while framework developers and ML engineers building custom training pipelines use Megatron Core's building blocks.Do I need NVIDIA hardware to run this, or will any GPU do?It is built and benchmarked around NVIDIA GPUs only — the published results are on H100 clusters, up to 6,144 GPUs for a 462B-parameter model. The largest runs assume a multi-node cluster, not a single machine.
It is built and benchmarked around NVIDIA GPUs only — the published results are on H100 clusters, up to 6,144 GPUs for a 462B-parameter model. The largest runs assume a multi-node cluster, not a single machine.How large are the models it has been benchmarked with?The published benchmarks train 2B to 462B parameter models on clusters of up to 6,144 H100 GPUs, reaching 47-48% Model FLOP Utilization at the top end, with production features like checkpointing and fault tolerance included in the measured throughput.
The published benchmarks train 2B to 462B parameter models on clusters of up to 6,144 H100 GPUs, reaching 47-48% Model FLOP Utilization at the top end, with production features like checkpointing and fault tolerance included in the measured throughput.How does it split training work across thousands of GPUs?Through five parallelism strategies: tensor, pipeline, data, expert and context parallelism (TP, PP, DP, EP, CP), combined with FP16, BF16, FP8 and FP4 mixed precision. Dynamic Context Parallelism adds up to 1.48x speedup for variable-length sequence training.
Through five parallelism strategies: tensor, pipeline, data, expert and context parallelism (TP, PP, DP, EP, CP), combined with FP16, BF16, FP8 and FP4 mixed precision. Dynamic Context Parallelism adds up to 1.48x speedup for variable-length sequence training.Can I bring in Hugging Face checkpoints?Yes. Megatron Bridge converts checkpoints bidirectionally between Hugging Face and Megatron formats and ships production-ready recipes for popular models, including conversion, inference and pretraining recipes for DeepSeek-V4.
Yes. Megatron Bridge converts checkpoints bidirectionally between Hugging Face and Megatron formats and ships production-ready recipes for popular models, including conversion, inference and pretraining recipes for DeepSeek-V4.Which recent models does it support?Recent releases add initial DeepSeek-V4 support, optimized MoE recipes for DeepSeek-V3, Mixtral and Qwen3, the Falcon-H1 hybrid transformer-Mamba architecture with BitNet ternary quantization, and GPT-OSS features like YaRN RoPE scaling and attention sinks.
Recent releases add initial DeepSeek-V4 support, optimized MoE recipes for DeepSeek-V3, Mixtral and Qwen3, the Falcon-H1 hybrid transformer-Mamba architecture with BitNet ternary quantization, and GPT-OSS features like YaRN RoPE scaling and attention sinks.What are similar projects?TensorRT-LLM, halo, DMI, LittleBit, verl, Meshy, Intern-Decision, airllm
TensorRT-LLM, halo, DMI, LittleBit, verl, Meshy, Intern-Decision, airllmWhich topics is it in?parallel, model-training, moe, transformers, high-performance, quantization, large-language-models, distributed-systems, nvidia, huggingface, cuda, deep-learning
parallel, model-training, moe, transformers, high-performance, quantization, large-language-models, distributed-systems, nvidia, huggingface, cuda, deep-learning