micrograd README

The project's README from github.com/karpathy/micrograd. As one markdown file: README.md.

micrograd

awww

A tiny Autograd engine (with a bite! :)). Implements backpropagation (reverse-mode autodiff) over a dynamically built DAG and a small neural networks library on top of it with a PyTorch-like API. Both are tiny, with about 100 and 50 lines of code respectively. The DAG only operates over scalar values, so e.g. we chop up each neuron into all of its individual tiny adds and multiplies. However, this is enough to build up entire deep neural nets doing binary classification, as the demo notebook shows. Potentially useful for educational purposes.

Installation

pip install micrograd

Example usage

Below is a slightly contrived example showing a number of possible supported operations:

from micrograd.engine import Value

a = Value(-4.0)
b = Value(2.0)
c = a + b
d = a * b + b**3
c += c + 1
c += 1 + c + (-a)
d += d * 2 + (b + a).relu()
d += 3 * d + (b - a).relu()
e = c - d
f = e**2
g = f / 2.0
g += 10.0 / f
print(f'{g.data:.4f}') # prints 24.7041, the outcome of this forward pass
g.backward()
print(f'{a.grad:.4f}') # prints 138.8338, i.e. the numerical value of dg/da
print(f'{b.grad:.4f}') # prints 645.5773, i.e. the numerical value of dg/db

Training a neural net

The notebook demo.ipynb provides a full demo of training an 2-layer neural network (MLP) binary classifier. This is achieved by initializing a neural net from micrograd.nn module, implementing a simple svm "max-margin" binary classification loss and using SGD for optimization. As shown in the notebook, using a 2-layer neural net with two 16-node hidden layers we achieve the following decision boundary on the moon dataset:

2d neuron

Training a GPT

For a more advanced example, see microgpt, which trains and samples from a full GPT-2-like transformer in pure, dependency-free Python. It builds on a more efficient and better version of the autograd engine here (storing local gradients at forward time instead of per-op backward closures), and is the complete algorithm in a single file — everything else is just efficiency. See also the accompanying explainer post for a detailed walkthrough.

Tracing / visualization

For added convenience, the notebook trace_graph.ipynb produces graphviz visualizations. E.g. this one below is of a simple 2D neuron, arrived at by calling draw_dot on the code below, and it shows both the data (left number in each node) and the gradient (right number in each node).

from micrograd import nn
n = nn.Neuron(2)
x = [Value(1.0), Value(-2.0)]
y = n(x)
dot = draw_dot(y)

2d neuron

Running tests

To run the unit tests you will have to install PyTorch, which the tests use as a reference for verifying the correctness of the calculated gradients. Then simply:

python -m pytest

License

MIT

micrograd (karpathy/micrograd)

Searching Database…

17.9K ·GitHub
Micrograd
这是什么？Micrograd 是一个微型的标量自动求导引擎，它在动态构建的有向无环图（DAG）上实现反向传播（反向模式自动微分），并在其之上提供一个类似 PyTorch API 的小型神经网络库。
这是像 PyTorch 那样完整的深度学习框架，还是更小的东西？它是一个极简的教学用库，而不是生产级框架：核心自动求导引擎约 100 行代码，其上的神经网络库约 50 行。
我该怎么试用它？先安装该包，然后使用 demo.ipynb 笔记本在 moons 数据集上训练一个两层 MLP 二分类器；或者直接使用 micrograd.nn 模块编写你自己的标量运算。
除了 Python，使用它还需要别的东西吗？使用引擎本身不需要额外依赖；但如果你想运行单元测试，就需要安装 PyTorch，因为测试会以 PyTorch 的梯度结果作为参考进行比对。
它在底层是如何处理神经网络层的？它只对标量值进行运算，因此每个神经元都会被拆分成一个个微小的加法和乘法，而不是使用向量化的张量运算。这让它简单明了，但不适合真实场景下的训练，性能较差。
我能把计算图可视化吗？可以。tracegraph.ipynb 笔记本会生成 DAG 的 graphviz 可视化图，通过对你的表达式调用 drawdot，显示每个节点上的数据值和梯度。
有没有基于它构建的更大项目？有。microgpt 在此基础上扩展出一个类似 GPT-2 的完整 Transformer，它使用纯 Python 实现、无任何依赖，并采用了更高效版本的自动求导引擎。
内置的神经网络实际能做什么？演示中训练了一个两层 MLP，它包含两个 16 节点的隐藏层，作为二分类器，使用 SVM 最大间隔损失和 SGD 进行训练，并在 moons 数据集上得到了决策边界。
What are similar projects?spinningup, fastbook, MLAlgorithms, DeepLearning, llama3-from-scratch, homemade-machine-learning, ai-edu, pytorch-tutorial
Which topics is it in?graph, algorithms, neural-network, jupyter notebook, beginner-friendly, pytorch, learning, tutorial, deep-learning, research, machine-learning, education
Creator
karpathy
Size
0.2 MB
Stars
17.9K
Updated
CommitAug 03
Age
6y
Commits · 90 Days