Topics / #distributed-training

Distributed-training projects on GitHub

Training machine learning models across many GPUs and machines.

As of , the most-starred Distributed-training projects on GitHub are nanochat (58.7K stars), Made-With-ML (49.8K stars) and ray (44.1K stars). GitHub.Locker tracks 39 Distributed-training repositories with 679.1K stars combined. The fastest-growing on October 16 was nanochat, with +31 stars in a day.

39Projects
679.1KStars in total
+166Stars in a day
Browse in the catalog →github.com/topics/distributed-training ↗

Hot right now

Most stars gained in a day · Oct 16

+31

nanochat

karpathy/nanochat · ★ 58.7K

A minimal, hackable harness for training your own LLM end to end (tokenization, pretraining, finetuning, evaluation, inference) on a single GPU node, ending in a chatbot you talk to over a CLI.

+20

train-llm-from-scratch

FareedKhan-dev/train-llm-from-scratch · ★ 12.2K

A from-scratch tutorial and codebase for training your own LLM end to end: tokenizing raw text, pretraining a GPT-style Transformer, and then post-training it with SFT, a reward model, DPO/ORPO/KTO, PPO and GRPO into an aligned, reasoning-style assistant.

+18

higgsfield

higgsfield-ai/higgsfield · ★ 6.1K

Higgsfield is an open-source, fault-tolerant GPU orchestration system and machine learning framework for training huge models, from billions to trillions of parameters, such as LLMs.

+12

ray

ray-project/ray · ★ 44.1K

Ray is an open-source framework for scaling AI and Python applications: a core distributed runtime plus a set of AI libraries that simplify ML compute.

The all-time top

39 projects · by stars

  1. #1 · karpathy/nanochat

    nanochat

    A minimal, hackable harness for training your own LLM end to end (tokenization, pretraining, finetuning, evaluation, inference) on a single GPU node, ending in a chatbot you talk to over a CLI.

    58.7K
  2. #2 · GokuMohandas/Made-With-ML

    Made-With-ML

    Made-With-ML is a free hands-on course that teaches how to design, develop, deploy, and iterate on production-grade machine learning applications, combining ML with software engineering best practices.

    49.8K
  3. #3 · ray-project/ray

    ray

    Ray is an open-source framework for scaling AI and Python applications: a core distributed runtime plus a set of AI libraries that simplify ML compute.

    44.1K
  1. 4

    ColossalAI

    41.4Khpcaitech/ColossalAI · Python

    Colossal-AI is an open-source system for training and running large AI models more cheaply, faster and with less hardware, by providing a toolbox of parallelism and memory-optimization techniques.

  2. 5

    LLM101n

    37.5Kkarpathy/LLM101n

    LLM101n is a planned course by Andrej Karpathy that walks through building a 'Storyteller' AI language model end-to-end, from basics all the way to a ChatGPT-like web app, implemented from scratch in Python, C and CUDA.

  3. 6

    pytorch-image-models

    37.2Khuggingface/pytorch-image-models · Python

    PyTorch Image Models (timm) — the largest collection of PyTorch image encoders and backbones, shipped with pretrained weights plus train, eval, inference and export scripts.

  4. 7

    self-llm

    32.5Kdatawhalechina/self-llm · Jupyter Notebook

    Self-llm is a Chinese-language tutorial collection that teaches beginners how to configure environments, deploy, and fine-tune (full-parameter or LoRA) 50+ open-source large language models and multimodal models on Linux.

  5. 8

    pytorch-lightning

    31.4KLightning-AI/pytorch-lightning · Python

    PyTorch Lightning is a deep learning framework for pretraining and finetuning AI models of any size, scaling from a single CPU to thousands of GPUs or TPUs without changing your model code.

  6. 9

    open-r1

    26.5Khuggingface/open-r1 · Python

    Open R1 is a fully open effort to reproduce the training pipeline behind DeepSeek-R1, providing scripts to run SFT and GRPO training, generate synthetic reasoning data, and evaluate the resulting reasoning models.

  7. 10

    vit-pytorch

    25.6Klucidrains/vit-pytorch · Python

    A PyTorch implementation of the Vision Transformer (ViT), an image classification model that reaches SOTA results using just a single transformer encoder instead of convolutions.

  8. 11

    Paddle

    24.1KPaddlePaddle/Paddle · C++

    PaddlePaddle is an open-source deep learning platform — China's first independently developed one, open-sourced since 2016 — spanning a core framework, model libraries, end-to-end dev kits and service platforms.

  9. 12

    examples

    24.1Kpytorch/examples · Python

    A curated set of short, standalone PyTorch example projects covering computer vision, text, and reinforcement learning, each meant to be emulated in your own work.

  10. 13

    verl

    23.9Kverl-project/verl · Python

    verl is a flexible, efficient and production-ready reinforcement learning training library for LLMs, initiated by ByteDance's Seed team and maintained by the community. It is the open-source version of the HybridFlow paper (EuroSys 2025).

  11. 14

    ml-engineering

    19.5Kstas00/ml-engineering · Python

    An open book on machine learning engineering: methodologies, tools and step-by-step instructions for training, fine-tuning and running inference on large language models and multi-modal (VLM) models, written by the engineer who led the BLOOM-176B training.

  12. 15

    CNTK

    17.6Kmicrosoft/CNTK · C++

    The Microsoft Cognitive Toolkit (CNTK) is an open-source deep-learning toolkit that describes neural networks as a series of computational steps in a directed graph.

  13. 16

    ms-swift

    15.9Kmodelscope/ms-swift · Python

    ms-swift is the ModelScope community's fine-tuning and deployment framework for LLMs and multimodal models, covering training, inference, evaluation, quantization and deployment in one pipeline.

  14. 17

    llm_interview_note

    15.3Kwdndev/llm_interview_note · HTML

    This is a curated Chinese-language knowledge base of large language model (LLM) concepts and interview questions, aimed at algorithm/application engineers preparing for LLM-related job interviews.

  15. 18

    horovod

    14.7Khorovod/horovod · Python

    Horovod is a distributed deep learning training framework that adds distributed training support to TensorFlow, Keras, PyTorch, and Apache MXNet, aiming to make scaling training across many GPUs and machines fast and easy.

  16. 19

    nni

    14.4Kmicrosoft/nni · Python

    NNI (Neural Network Intelligence) is Microsoft's open-source AutoML toolkit that automates the machine learning lifecycle: feature engineering, neural architecture search, model compression, and hyperparameter tuning.

  17. 20

    dgl

    14.3Kdmlc/dgl · Python

    DGL is an open-source Python package for deep learning on graphs: it makes it easy to build and train graph neural networks on top of deep learning frameworks you may already use.

  18. 21

    deeplearning4j

    14.3Kdeeplearning4j/deeplearning4j · Java

    Eclipse Deeplearning4J is a suite of JVM libraries for building, training and deploying deep learning applications, usable from Java, Scala, Kotlin and Clojure.

  19. 22

    PaddleNLP

    13KPaddlePaddle/PaddleNLP · Python

    PaddleNLP is a large language model (LLM) and small language model (SLM) development toolkit built on the PaddlePaddle deep learning framework, offering efficient training, lossless compression, and high-performance inference for large models.

  20. 23

    PaddleFormers

    13KPaddlePaddle/PaddleFormers · Python

    PaddleFormers is a Transformers-style library built on Baidu's PaddlePaddle deep learning framework, giving the PaddlePaddle ecosystem the same kind of model interface and training workflow as Hugging Face Transformers, with full training support for 100+ LLMs and vision-language models.

  21. 24

    axolotl

    12.6Kaxolotl-ai-cloud/axolotl · Python

    Axolotl is a free, open-source framework that streamlines post-training and fine-tuning of large language models — you describe your run in one YAML config and it drives the whole pipeline.

  22. 25

    train-llm-from-scratch

    12.2KFareedKhan-dev/train-llm-from-scratch · Python

    A from-scratch tutorial and codebase for training your own LLM end to end: tokenizing raw text, pretraining a GPT-style Transformer, and then post-training it with SFT, a reward model, DPO/ORPO/KTO, PPO and GRPO into an aligned, reasoning-style assistant.

  23. 26

    skypilot

    10.7Kskypilot-org/skypilot · Python

    SkyPilot is an open-source system for running, managing, and scaling AI workloads — training, serving, batch inference — on any infrastructure: your own Kubernetes or Slurm clusters, reserved GPUs, or 20+ clouds.

  24. 27

    yolov3

    10.6Kultralytics/yolov3 · Python

    Ultralytics' PyTorch implementation of the YOLOv3 real-time object detector, packaging three model variants - YOLOv3, YOLOv3-SPP and YOLOv3-tiny - with training, validation, inference and multi-format export tooling.

  25. 28

    metaflow

    10.3KNetflix/metaflow · Python

    Metaflow is an open-source Python framework for building, managing and deploying real-life AI and ML systems — from quick prototyping in notebooks to reliable production deployments.

  26. 29

    higgsfield

    6.1Khiggsfield-ai/higgsfield · Jupyter Notebook

    Higgsfield is an open-source, fault-tolerant GPU orchestration system and machine learning framework for training huge models, from billions to trillions of parameters, such as LLMs.

  27. 30

    InfraTech

    4.2KCalvinXKY/InfraTech · Jupyter Notebook

    InfraTech is a collection of Jupyter notebooks and articles sharing AI Infra knowledge: training/inference frameworks (PyTorch, vLLM/SGLang, slime/vime), performance acceleration, deep learning fundamentals and AI hardware/software basics.

  28. 31

    halo

    825whitecircle/halo · Python

    Halo is an open-source framework from White Circle for training large language and multimodal models: pre-training, SFT, preference optimization, and fully asynchronous multi-turn RL in one codebase, from a single GPU up to multi-node clusters.

  29. 32

    SolarWM

    777Junchao-cs/SolarWM · Python

    SolarWM is a fully open foundation for building interactive video world models, covering data preparation, scalable multi-backbone training, and long-horizon inference.

  30. 33

    verl

    716XiaomiMiMo/verl · Python

    verl is an open-source RL post-training framework for large language models, started by ByteDance Seed as the open-source version of the HybridFlow paper. This XiaomiMiMo fork adds agentic RL training code for MiMo, with reproduction code for five RL environments on top of verl 0.9.0.dev.

  31. 34

    vdn-minimax-h3

    617OpenVDN/vdn-minimax-h3 · Python

    VDN-Minimax-H3 is a hybrid-attention video generation model built on MiniMax H3 that renders video faster than it plays, supporting text-to-video-audio, image-to-video-audio, first/last-frame-to-video and more from one checkpoint.

  32. 35

    Meshy

    387OpenBMB/Meshy · Python

    Meshy is an open-source framework for running reinforcement learning (RL) training jobs where every role (inference, training, rollout) runs as an independent service, communicating through a shared TransferQueue data plane. It is built on SGLang and torchtitan.

  33. 36

    DeepEP-Ascend

    259deepseek-ai/DeepEP-Ascend · C++

    DeepEP-Ascend is a high-performance communication library for machine learning training and inference on Huawei Ascend NPUs, providing expert-parallel all-to-all operations for MoE dispatch and combine.

  34. 37

    ApexAgents-SkyRL-Recipe

    217Mercor-Intelligence/ApexAgents-SkyRL-Recipe · Python

    It's a complete, public recipe for RL-training knowledge-work agents on Mercor's APEX-Agents data, built on the open-source SkyRL training library and the open-source harbor sandbox/grading toolkit. It's the companion code for Mercor's 'Training Frontier Knowledge Work Agents' blogpost.

  35. 38

    object-permanence

    97hokindeng/object-permanence · Python

    This is the official codebase for the paper 'Training Object Permanence in World Models' (arXiv:2609.28654). It bundles a Blender-based data factory that renders videos of objects persisting through occlusion, plus a training stack that teaches this behavior to a world model on AWS Trainium2.

  36. 39

    DMI

    65ProjectDMX/DMI · Python

    DMI is a decoupled, asynchronous observability stack that gives real-time access to a running LLM's internal state — residual streams, attention patterns, MLP outputs, KV-cache slices, logits — during inference or training, with minimal overhead.

Often tagged together