micrograd README

The project's README from github.com/karpathy/micrograd. As one markdown file: README.md.

micrograd

awww

A tiny Autograd engine (with a bite! :)). Implements backpropagation (reverse-mode autodiff) over a dynamically built DAG and a small neural networks library on top of it with a PyTorch-like API. Both are tiny, with about 100 and 50 lines of code respectively. The DAG only operates over scalar values, so e.g. we chop up each neuron into all of its individual tiny adds and multiplies. However, this is enough to build up entire deep neural nets doing binary classification, as the demo notebook shows. Potentially useful for educational purposes.

Installation

pip install micrograd

Example usage

Below is a slightly contrived example showing a number of possible supported operations:

from micrograd.engine import Value

a = Value(-4.0)
b = Value(2.0)
c = a + b
d = a * b + b**3
c += c + 1
c += 1 + c + (-a)
d += d * 2 + (b + a).relu()
d += 3 * d + (b - a).relu()
e = c - d
f = e**2
g = f / 2.0
g += 10.0 / f
print(f'{g.data:.4f}') # prints 24.7041, the outcome of this forward pass
g.backward()
print(f'{a.grad:.4f}') # prints 138.8338, i.e. the numerical value of dg/da
print(f'{b.grad:.4f}') # prints 645.5773, i.e. the numerical value of dg/db

Training a neural net

The notebook demo.ipynb provides a full demo of training an 2-layer neural network (MLP) binary classifier. This is achieved by initializing a neural net from micrograd.nn module, implementing a simple svm "max-margin" binary classification loss and using SGD for optimization. As shown in the notebook, using a 2-layer neural net with two 16-node hidden layers we achieve the following decision boundary on the moon dataset:

2d neuron

Training a GPT

For a more advanced example, see microgpt, which trains and samples from a full GPT-2-like transformer in pure, dependency-free Python. It builds on a more efficient and better version of the autograd engine here (storing local gradients at forward time instead of per-op backward closures), and is the complete algorithm in a single file — everything else is just efficiency. See also the accompanying explainer post for a detailed walkthrough.

Tracing / visualization

For added convenience, the notebook trace_graph.ipynb produces graphviz visualizations. E.g. this one below is of a simple 2D neuron, arrived at by calling draw_dot on the code below, and it shows both the data (left number in each node) and the gradient (right number in each node).

from micrograd import nn
n = nn.Neuron(2)
x = [Value(1.0), Value(-2.0)]
y = n(x)
dot = draw_dot(y)

2d neuron

Running tests

To run the unit tests you will have to install PyTorch, which the tests use as a reference for verifying the correctness of the calculated gradients. Then simply:

python -m pytest

License

MIT

micrograd (karpathy/micrograd)

Searching Database…

17.9K ·GitHub
Micrograd
What is this?Micrograd is a tiny scalar-valued autograd engine that implements backpropagation (reverse-mode autodiff) over a dynamically built DAG, plus a small neural network library on top with a PyTorch-like API.
Is this a full deep learning framework like PyTorch, or something smaller?It's a minimal educational library, not a production framework: the core autograd engine is about 100 lines of code and the neural net library on top is about 50 lines.
How do I try it out?Install the package, then use the demo.ipynb notebook to train a 2-layer MLP binary classifier on the moons dataset, or write your own scalar operations directly using the micrograd.nn module.
Do I need anything beyond Python to use it?No extra dependencies are needed to use the engine itself, but if you want to run the unit tests you need PyTorch installed, since the tests compare gradients against PyTorch as a reference.
How does it handle neural net layers under the hood?It only operates over scalar values, so every neuron is chopped up into individual tiny adds and multiplies rather than using vectorized tensor operations — this makes it simple but not performant for real training.
Can I visualize the computation graph?Yes, the tracegraph.ipynb notebook produces graphviz visualizations of the DAG, showing both the data value and gradient at each node by calling drawdot on your expression.
Is there a bigger project built on this?Yes, microgpt extends the ideas here into a full GPT-2-like transformer trained in pure, dependency-free Python, using a more efficient version of the autograd engine.
What can the included neural net actually do?The demo trains a 2-layer MLP with two 16-node hidden layers as a binary classifier using an SVM max-margin loss and SGD, achieving a decision boundary on the moons dataset.
What are similar projects?spinningup, fastbook, MLAlgorithms, DeepLearning, llama3-from-scratch, homemade-machine-learning, ai-edu, pytorch-tutorial
Which topics is it in?graph, algorithms, neural-network, jupyter notebook, beginner-friendly, pytorch, learning, tutorial, deep-learning, research, machine-learning, education
Creator
karpathy
Size
0.2 MB
Stars
17.9K
Updated
CommitAug 03
Age
6y
Commits · 90 Days