llm-course README

The project's README from github.com/mlabonne/llm-course. As one markdown file: README.md.

LLM Course

𝕏 Follow me on X • 🤗 Hugging Face • 💻 Blog • 📙 LLM Engineer's Handbook


LLM Engineer's Handbook CoverThe LLM course is divided into three parts:

  1. 🧩 LLM Fundamentals is optional and covers fundamental knowledge about mathematics, Python, and neural networks.
  2. 🧑‍🔬 The LLM Scientist focuses on building the best possible LLMs using the latest techniques.
  3. 👷 The LLM Engineer focuses on creating LLM-based applications and deploying them.

[!NOTE] Based on this course, I co-wrote the LLM Engineer's Handbook, a hands-on book that covers an end-to-end LLM application from design to deployment. The LLM course will always stay free, but you can support my work by purchasing this book.

For a more comprehensive version of this course, check out the DeepWiki.

📝 Notebooks

A list of notebooks and articles I wrote about LLMs.

Toggle section (optional)

Tools

Notebook Description Notebook
🧐 LLM AutoEval Automatically evaluate your LLMs using RunPod Open In Colab
🥱 LazyMergekit Easily merge models using MergeKit in one click. Open In Colab
🦎 LazyAxolotl Fine-tune models in the cloud using Axolotl in one click. Open In Colab
⚡ AutoQuant Quantize LLMs in GGUF, GPTQ, EXL2, AWQ, and HQQ formats in one click. Open In Colab
🌳 Model Family Tree Visualize the family tree of merged models. Open In Colab
🚀 ZeroSpace Automatically create a Gradio chat interface using a free ZeroGPU. Open In Colab
✂️ AutoAbliteration Automatically abliteration models with custom datasets. Open In Colab
🧼 AutoDedup Automatically deduplicate datasets using the Rensa library. Open In Colab

Fine-tuning

Notebook Description Article Notebook
Fine-tune Llama 3.1 with Unsloth Ultra-efficient supervised fine-tuning in Google Colab. Article Open In Colab
Fine-tune Llama 3 with ORPO Cheaper and faster fine-tuning in a single stage with ORPO. Article Open In Colab
Fine-tune Mistral-7b with DPO Boost the performance of supervised fine-tuned models with DPO. Article Open In Colab
Fine-tune Mistral-7b with QLoRA Supervised fine-tune Mistral-7b in a free-tier Google Colab with TRL. Open In Colab
Fine-tune CodeLlama using Axolotl End-to-end guide to the state-of-the-art tool for fine-tuning. Article Open In Colab
Fine-tune Llama 2 with QLoRA Step-by-step guide to supervised fine-tune Llama 2 in Google Colab. Article Open In Colab

Quantization

Notebook Description Article Notebook
Introduction to Quantization Large language model optimization using 8-bit quantization. Article Open In Colab
4-bit Quantization using GPTQ Quantize your own open-source LLMs to run them on consumer hardware. Article Open In Colab
Quantization with GGUF and llama.cpp Quantize Llama 2 models with llama.cpp and upload GGUF versions to the HF Hub. Article Open In Colab
ExLlamaV2: The Fastest Library to Run LLMs Quantize and run EXL2 models and upload them to the HF Hub. Article Open In Colab

Other

Notebook Description Article Notebook
Merge LLMs with MergeKit Create your own models easily, no GPU required! Article Open In Colab
Create MoEs with MergeKit Combine multiple experts into a single frankenMoE Article Open In Colab
Uncensor any LLM with abliteration Fine-tuning without retraining Article Open In Colab
Improve ChatGPT with Knowledge Graphs Augment ChatGPT's answers with knowledge graphs. Article Open In Colab
Decoding Strategies in Large Language Models A guide to text generation from beam search to nucleus sampling Article Open In Colab

🧩 LLM Fundamentals

This section introduces essential knowledge about mathematics, Python, and neural networks. You might not want to start here but refer to it as needed.

Toggle section (optional)

1. Mathematics for Machine Learning

Before mastering machine learning, it is important to understand the fundamental mathematical concepts that power these algorithms.

  • Linear Algebra: This is crucial for understanding many algorithms, especially those used in deep learning. Key concepts include vectors, matrices, determinants, eigenvalues and eigenvectors, vector spaces, and linear transformations.
  • Calculus: Many machine learning algorithms involve the optimization of continuous functions, which requires an understanding of derivatives, integrals, limits, and series. Multivariable calculus and the concept of gradients are also important.
  • Probability and Statistics: These are crucial for understanding how models learn from data and make predictions. Key concepts include probability theory, random variables, probability distributions, expectations, variance, covariance, correlation, hypothesis testing, confidence intervals, maximum likelihood estimation, and Bayesian inference.

📚 Resources:

2. Python for Machine Learning

Python is a powerful and flexible programming language that's particularly good for machine learning, thanks to its readability, consistency, and robust ecosystem of data science libraries.

  • Python Basics: Python programming requires a good understanding of the basic syntax, data types, error handling, and object-oriented programming.
  • Data Science Libraries: It includes familiarity with NumPy for numerical operations, Pandas for data manipulation and analysis, Matplotlib and Seaborn for data visualization.
  • Data Preprocessing: This involves feature scaling and normalization, handling missing data, outlier detection, categorical data encoding, and splitting data into training, validation, and test sets.
  • Machine Learning Libraries: Proficiency with Scikit-learn, a library providing a wide selection of supervised and unsupervised learning algorithms, is vital. Understanding how to implement algorithms like linear regression, logistic regression, decision trees, random forests, k-nearest neighbors (K-NN), and K-means clustering is important. Dimensionality reduction techniques like PCA and t-SNE are also helpful for visualizing high-dimensional data.

📚 Resources:

3. Neural Networks

Neural networks are a fundamental part of many machine learning models, particularly in the realm of deep learning. To utilize them effectively, a comprehensive understanding of their design and mechanics is essential.

  • Fundamentals: This includes understanding the structure of a neural network, such as layers, weights, biases, and activation functions (sigmoid, tanh, ReLU, etc.)
  • Training and Optimization: Familiarize yourself with backpropagation and different types of loss functions, like Mean Squared Error (MSE) and Cross-Entropy. Understand various optimization algorithms like Gradient Descent, Stochastic Gradient Descent, RMSprop, and Adam.
  • Overfitting: Understand the concept of overfitting (where a model performs well on training data but poorly on unseen data) and learn various regularization techniques (dropout, L1/L2 regularization, early stopping, data augmentation) to prevent it.
  • Implement a Multilayer Perceptron (MLP): Build an MLP, also known as a fully connected network, using PyTorch.

📚 Resources:

4. Natural Language Processing (NLP)

NLP is a fascinating branch of artificial intelligence that bridges the gap between human language and machine understanding. From simple text processing to understanding linguistic nuances, NLP plays a crucial role in many applications like translation, sentiment analysis, chatbots, and much more.

  • Text Preprocessing: Learn various text preprocessing steps like tokenization (splitting text into words or sentences), stemming (reducing words to their root form), lemmatization (similar to stemming but considers the context), stop word removal, etc.
  • Feature Extraction Techniques: Become familiar with techniques to convert text data into a format that can be understood by machine learning algorithms. Key methods include Bag-of-words (BoW), Term Frequency-Inverse Document Frequency (TF-IDF), and n-grams.
  • Word Embeddings: Word embeddings are a type of word representation that allows words with similar meanings to have similar representations. Key methods include Word2Vec, GloVe, and FastText.
  • Recurrent Neural Networks (RNNs): Understand the working of RNNs, a type of neural network designed to work with sequence data. Explore LSTMs and GRUs, two RNN variants that are capable of learning long-term dependencies.

📚 Resources:

🧑‍🔬 The LLM Scientist

This section of the course focuses on learning how to build the best possible LLMs using the latest techniques.

1. The LLM Architecture

An in-depth knowledge of the Transformer architecture is not required, but it's important to understand the main steps of modern LLMs: converting text into numbers through tokenization, processing these tokens through layers including attention mechanisms, and finally generating new text through various sampling strategies.

📚 References:

llm-course (mlabonne/llm-course)

Searching Database…

83.7K ·GitHub
Llm-course
What is this?A free, open-source course for getting into Large Language Models (LLMs), combining theory roadmaps with hands-on Colab notebooks covering everything from math fundamentals to fine-tuning and deployment.
Is this a reading list or something I can actually run?Both: it's a structured roadmap split into three tracks (LLM Fundamentals, LLM Scientist, LLM Engineer) and it links out to dozens of runnable Colab notebooks and one-click tools for fine-tuning, merging, and quantizing models.
Where should a complete beginner start?Start with the optional LLM Fundamentals section, which builds up math, Python, and neural network basics before you move on to the LLM Scientist (model building) or LLM Engineer (application deployment) tracks depending on your goal.
Do I need my own GPU to follow along?No, several exercises are designed to run in a free-tier Google Colab (and one tool even spins up a chat UI on free ZeroGPU), though more advanced fine-tuning notebooks recommend better hardware for speed.
Will I actually learn to build and ship LLM apps, or is this just theory?Both: the LLM Scientist track covers building the best possible models (architecture, fine-tuning, quantization) while the separate LLM Engineer track covers creating and deploying LLM-based applications.
Do I need prior machine learning experience to follow this?No, the optional LLM Fundamentals module explicitly starts from scratch with linear algebra, Python, and neural network basics for people without a ML background, and advises skipping ahead only if you already know this material.
Why would I buy the author's book if the course is free?The LLM Engineer's Handbook, co-written by the author, gives a more comprehensive end-to-end guide to designing and deploying an LLM application; the course itself stays free regardless.
What one-click tools does the course link to for working with models?It links to Colab tools like LazyMergekit (merge models with MergeKit), LazyAxolotl (fine-tune in the cloud), and AutoQuant (quantize into GGUF, GPTQ, EXL2, AWQ, or HQQ formats in one click).
What is model abliteration, which comes up in the notebooks?It's a technique covered in one of the linked notebooks for removing an LLM's censorship/refusal behavior without retraining the model from scratch.
Is there a deeper version of this content somewhere else?Yes, the README points to a DeepWiki version of the course for a more comprehensive treatment of the same material.
What are similar projects?nlp_course, Hands-On-Large-Language-Models, llm_interview_note, CV, Foundations-of-LLMs, ai-engineering-hub, AI-Engineering-Lab, autumn-recruitment
Which topics is it in?roadmap, llms, course, jupyter notebook, quantization, fine-tuning, transformer, large-language-models, nlp, prompt-engineering, book, rag
Creator
mlabonne
Size
7.5 MB
Stars
83.7K
Updated
CommitFeb 05
Age
3y
Commits · 90 Days