nn-zero-to-hero README

The project's README from github.com/karpathy/nn-zero-to-hero. As one markdown file: README.md.

Neural Networks: Zero to Hero

A course on neural networks that starts all the way at the basics. The course is a series of YouTube videos where we code and train neural networks together. The Jupyter notebooks we build in the videos are then captured here inside the lectures directory. Every lecture also has a set of exercises included in the video description. (This may grow into something more respectable).

Lecture 1: The spelled-out intro to neural networks and backpropagation: building micrograd

Backpropagation and training of neural networks. Assumes basic knowledge of Python and a vague recollection of calculus from high school.

Lecture 2: The spelled-out intro to language modeling: building makemore

We implement a bigram character-level language model, which we will further complexify in followup videos into a modern Transformer language model, like GPT. In this video, the focus is on (1) introducing torch.Tensor and its subtleties and use in efficiently evaluating neural networks and (2) the overall framework of language modeling that includes model training, sampling, and the evaluation of a loss (e.g. the negative log likelihood for classification).

Lecture 3: Building makemore Part 2: MLP

We implement a multilayer perceptron (MLP) character-level language model. In this video we also introduce many basics of machine learning (e.g. model training, learning rate tuning, hyperparameters, evaluation, train/dev/test splits, under/overfitting, etc.).

Lecture 4: Building makemore Part 3: Activations & Gradients, BatchNorm

We dive into some of the internals of MLPs with multiple layers and scrutinize the statistics of the forward pass activations, backward pass gradients, and some of the pitfalls when they are improperly scaled. We also look at the typical diagnostic tools and visualizations you'd want to use to understand the health of your deep network. We learn why training deep neural nets can be fragile and introduce the first modern innovation that made doing so much easier: Batch Normalization. Residual connections and the Adam optimizer remain notable todos for later video.

Lecture 5: Building makemore Part 4: Becoming a Backprop Ninja

We take the 2-layer MLP (with BatchNorm) from the previous video and backpropagate through it manually without using PyTorch autograd's loss.backward(). That is, we backprop through the cross entropy loss, 2nd linear layer, tanh, batchnorm, 1st linear layer, and the embedding table. Along the way, we get an intuitive understanding about how gradients flow backwards through the compute graph and on the level of efficient Tensors, not just individual scalars like in micrograd. This helps build competence and intuition around how neural nets are optimized and sets you up to more confidently innovate on and debug modern neural networks.

I recommend you work through the exercise yourself but work with it in tandem and whenever you are stuck unpause the video and see me give away the answer. This video is not super intended to be simply watched. The exercise is here as a Google Colab. Good luck :)

Lecture 6: Building makemore Part 5: Building WaveNet

We take the 2-layer MLP from previous video and make it deeper with a tree-like structure, arriving at a convolutional neural network architecture similar to the WaveNet (2016) from DeepMind. In the WaveNet paper, the same hierarchical architecture is implemented more efficiently using causal dilated convolutions (not yet covered). Along the way we get a better sense of torch.nn and what it is and how it works under the hood, and what a typical deep learning development process looks like (a lot of reading of documentation, keeping track of multidimensional tensor shapes, moving between jupyter notebooks and repository code, ...).

Lecture 7: Let's build GPT: from scratch, in code, spelled out.

We build a Generatively Pretrained Transformer (GPT), following the paper "Attention is All You Need" and OpenAI's GPT-2 / GPT-3. We talk about connections to ChatGPT, which has taken the world by storm. We watch GitHub Copilot, itself a GPT, help us write a GPT (meta :D!) . I recommend people watch the earlier makemore videos to get comfortable with the autoregressive language modeling framework and basics of tensors and PyTorch nn, which we take for granted in this video.

Lecture 8: Let's build the GPT Tokenizer

The Tokenizer is a necessary and pervasive component of Large Language Models (LLMs), where it translates between strings and tokens (text chunks). Tokenizers are a completely separate stage of the LLM pipeline: they have their own training sets, training algorithms (Byte Pair Encoding), and after training implement two fundamental functions: encode() from strings to tokens, and decode() back from tokens to strings. In this lecture we build from scratch the Tokenizer used in the GPT series from OpenAI. In the process, we will see that a lot of weird behaviors and problems of LLMs actually trace back to tokenization. We'll go through a number of these issues, discuss why tokenization is at fault, and why someone out there ideally finds a way to delete this stage entirely.

Ongoing...

License

MIT

nn-zero-to-hero (karpathy/nn-zero-to-hero)

language-model
Searching Database…

24.8K ·GitHub
Nn-zero-to-hero
What is this?A course by Andrej Karpathy that teaches neural networks from the absolute basics, built around a series of YouTube lectures with accompanying Jupyter notebooks coded live in the videos.
Is this a coding library or a video course?It is primarily a video course; the repo just hosts the Jupyter notebooks built during each lecture inside the 'lectures' directory, plus exercises from the video descriptions.
How do I get started with the lectures?Start with Lecture 1 (building 'micrograd' for backprop basics), watch the matching YouTube video, and follow along with the Jupyter notebook in the lectures folder.
Do I need prior knowledge before starting?Lecture 1 assumes only basic Python knowledge and a vague recollection of high-school calculus; later lectures build on PyTorch (torch.Tensor, torch.nn) introduced progressively.
Is the course finished?No, it's still a work in progress with more lectures expected ('Ongoing...'), and some topics like residual connections and the Adam optimizer are explicitly left as later todos.
Who is this course for?Aspiring deep learning practitioners who want to understand neural networks and LLMs like GPT from first principles rather than just using high-level libraries.
How is the course structured?It progresses step by step: micrograd for backprop basics, then 'makemore' built up across four parts into an MLP and WaveNet-style character-level language model, then building a GPT from scratch, and finally building the GPT tokenizer used by OpenAI's models.
What is makemore?A character-level language model project used throughout several lectures, evolving from a simple bigram model to an MLP and then a WaveNet-style convolutional architecture for generating new text sequences.
Does the course cover building GPT itself?Yes, Lecture 7 builds a Generatively Pretrained Transformer from scratch following the 'Attention is All You Need' paper and OpenAI's GPT-2/GPT-3, and Lecture 8 builds the GPT Tokenizer using Byte Pair Encoding.
Are there exercises to practice with?Yes, every lecture has a set of exercises included in its video description, and Lecture 5 even provides a dedicated Google Colab exercise notebook.
What are similar projects?minGPT, llama3-from-scratch, LLMs-from-scratch, LLM101n, autumn-recruitment, nlp_course, train-llm-from-scratch, lectures
Which topics is it in?karpathy, attention-mechanism, language-model, neural-networks, gpt, course, jupyter notebook, large-language-models, transformer, nlp, data-science, beginner-friendly
Creator
karpathy
Size
<0.1 MB
Stars
24.8K
Updated
CommitFeb 20
Age
4y
Commits · 90 Days