ml-engineering README

The project's README from github.com/stas00/ml-engineering. As one markdown file: README.md.

Machine Learning Engineering Open Book

This is an open collection of methodologies, tools and step by step instructions to help with successful training and fine-tuning of large language models and multi-modal models and their inference.

This is a technical material suitable for LLM/VLM training engineers and operators. That is the content here contains lots of scripts and copy-n-paste commands to enable you to quickly address your needs.

This repo is an ongoing brain dump of my experiences training Large Language Models (LLM) (and VLMs); a lot of the know-how I acquired while training the open-source BLOOM-176B model in 2022 and IDEFICS-80B multi-modal model in 2023, and RAG models at Contextual.AI in 2024.

I've been compiling this information mostly for myself so that I could quickly find solutions I have already researched in the past and which have worked, but as usual I'm happy to share these notes with the wider ML community.

Table of Contents

Part 1. Insights

  1. The AI Battlefield Engineering - what you need to know in order to succeed.

  2. How to Choose a Cloud Provider - these questions will empower you to have a successful compute cloud experience.

  3. When Is It Worth Upgrading GPUs? - a practical framework for deciding whether a GPU generation upgrade is worth its cost, worked through on a real H200 → B200 benchmark.

Part 2. Hardware

  1. Compute - accelerators, CPUs, CPU memory.

  2. Storage - local, distributed and shared file systems.

  3. Network - intra- and inter-node networking.

Part 3. Orchestration

  1. Orchestration Systems - managing containers and resources
  2. SLURM - Simple Linux Utility for Resource Management

Part 4. Training

  1. Training - model training-related guides

Part 5. Inference

  1. Inference - model inference insights

Part 6. Development

  1. Debugging and Troubleshooting - how to debug easy and difficult issues

  2. And more debugging

  3. Testing - numerous tips and tools to make test writing enjoyable

Part 7. Miscellaneous

  1. Resources - LLM/VLM chronicles

Updates

I announce any significant updates on my twitter channel https://twitter.com/StasBekman.

Ebook versions of the book

You can download various ebook formats of this book:

I will try to rebuild these once in a few weeks or so, but if you want the latest ebook versions, the instructions for building are here.

Thanks to HuggingFace for giving me permission to host my book's ebook formats at the HF hub.

SKILL.md for AI agents

I maintain a SKILL.md file that you can use to teach your AI agent to train and operate large-scale ML models better.

See also the companion skills: The Art of Debugging and Stas' Python Cookbook.

Courses

Lectures/Talks

Discussions

If you want to discuss something related to ML engineering this repo has the community discussions available - so please don't hesitate to share your experience or start a new discussion about something you're passionate about.

Key comparison tables

High end accelerators:

Networks:

Shortcuts

Things that you are likely to need to find quickly and often.

Tools:

Guides:

Gratitude

None of this would have been possible without me being entrusted with doing the specific LLM/VLM trainings I have learned the initial know-how from. This is a privilege that only a few enjoy due to the prohibitively expensive cost of renting huge ML compute clusters. So hopefully the rest of the ML community will vicariously learn from these notes.

Special thanks go to Thom Wolf who proposed that I lead the BLOOM-176B training back when I didn't know anything about large scale training. This was the project that catapulted me into the intense learning process. And, of course, HuggingFace for giving me the opportunity to work full time on BLOOM-176B and later on IDEFICS-80B trainings.

Recently, I continued expanding my knowledge and experience while training models and building scalable training/inference systems at Contextual.AI and I'm grateful for that opportunity to Aman and Douwe.

I'd also like to thank the numerous contributors who have been making this text awesome and error-free.

Contributing

If you found a bug, typo or would like to propose an improvement please don't hesitate to open an Issue or contribute a PR.

License

The content of this site is distributed under Attribution-ShareAlike 4.0 International.

Citation

@misc{bekman2024mlengineering,
  author = {Bekman, Stas},
  title = {Machine Learning Engineering Open Book},
  year = {2023-2026},
  publisher = {Stasosphere Online Inc.},
  journal = {GitHub repository},
  url = {https://github.com/stas00/ml-engineering}
}

My repositories map

✔ Books: Machine Learning Engineering | The Art of Debugging | Stas' Python Cookbook

✔ Applications: ipyexperiments

✔ Tools and Cheatsheets: bash | conda | git | jupyter-notebook | make | python | tensorboard | unix

✔ Other Machine Learning: ML ways | Porting

ml-engineering (stas00/ml-engineering)

Searching Database…

19.5K ·GitHub
Ml-engineering
这是什么东西?一本关于机器学习工程的开源书籍:涵盖训练、微调和运行大语言模型及多模态(VLM)模型的方法论、工具和分步操作指南,作者是主导 BLOOM-176B 训练的工程师。
它是用来读的书,还是要安装的软件?它是一本书而不是软件包:你可以在 GitHub 上免费阅读,或下载 PDF 和 EPUB 版本,书里满是脚本和可以复制粘贴到你自己的机器上运行的命令。
我该怎么开始使用它?直接在 GitHub 上阅读就行,或者从 HuggingFace hub 下载 PDF/EPUB。还有一个配套课程 "Lessons Learned from Training LLMs",先讲精炼的要点,需要深入的地方再展开。
我需要有属于自己的 GPU 集群才能从中受益吗?阅读是免费的,但动手实践的部分——比如 torch-distributed-gpu-test.py 这类工具和网络基准测试——是设计在训练用的 GPU 机器上运行的,所以只有一台笔记本电脑的话,大部分内容都体验不到。
这是一本完成度很高、打磨成熟的教科书吗?不是——作者称它是一部持续更新的脑洞笔记,还在不断扩充,电子书每隔几周才重新构建一次,更新会在他的 Twitter 频道上公布,所以有些章节比精心编写的教科书粗糙。
这本书是给新手看的,还是给已经在跑大规模训练的人看的?面向的是实际在一线工作的 LLM/VLM 训练工程师和运维人员,而不是机器学习新手——它默认你在处理真实的 GPU 作业,分享的是训练 BLOOM-176B 和 IDEFICS-80B 积累下来的实战经验,不是入门理论。
我可以复用或重新分发这些内容吗?可以,采用的是 CC BY-SA 4.0 协议:只要你注明作者出处并以相同方式共享,就可以自由分享和改编——这对开源书籍来说很常见,但和你熟悉的宽松代码许可证不太一样。
这本书到底讲了哪些内容?共七个部分:经验心得(如何选择云服务商、何时升级 GPU)、硬件(算力、存储、网络)、编排调度(包括 SLURM)、训练、推理、调试与测试,外加一个资源章节。
它附带开箱即用的工具吗?有:allreducebench.py 用来做网络吞吐量基准测试,比 nccl-tests 更简单;torch-distributed-gpu-test.py 用来检查节点间连通性;mamf-finder.py 用来测量实际能跑出的 TFLOPS,另外还有 PyTorch 调试和 SLURM 速查表。
我能用它来教我的 AI 编程助手吗?可以——这个仓库附带一个 SKILL.md,专门用来教会 AI 智能体更好地训练和运维大规模机器学习模型,还有配套技能:The Art of Debugging 和 Stas' Python Cookbook。
它能帮我决定该租还是买什么样的硬件吗?可以。书中维护着几份关键的对比表,涵盖加速器的 TFLOPS、显存容量与速度,以及节点间和节点内的网络速度,并通过一个真实的 H200 到 B200 的基准测试,帮你判断 GPU 升级值不值得花这笔钱。
What are similar projects?verl, ColossalAI, transformers, halo, skypilot, airllm, EvoVLM, ms-swift
Which topics is it in?hpc, courses, llm-training, vlm, distributed-training, skill-md, mlops, network, scalability, cheatsheet, storage, training
Creator
stas00
Size
27.9 MB
Stars
19.5K
Updated
CommitOct 16
Age
6y
Commits · 90 Days