Build a tiny autograd
Build a tiny autograd
This note puts the whole journey into one program of about 100 lines. You build a scalar autograd engine, wrap it in a small network, and train the network to learn XOR. The design follows Andrej Karpathy's micrograd. It uses plain Python with no libraries beyond math and random.
Copy the code blocks in order into one file, such as tiny_autograd.py, and run it.
The plan
| Part | What it does | Earlier note |
|---|---|---|
Value |
Holds a number, its gradient, and how it was made | Computation graphs |
backward |
Fills every gradient in topological order | Computation graphs, Backpropagation |
Neuron, Layer, MLP |
Build a network from Value objects |
A single neuron, Layers and the forward pass |
| Training loop | Squared error loss and gradient descent | Loss functions, Gradient descent |
Step 1: the Value class
A Value wraps one number. It stores the number in data, its gradient in grad, the values it was made from in parents, and a function backward_step. That function pushes the node's gradient to its parents with the local derivative.
Each operation creates a new Value and attaches the matching backward rule. The rules come straight from the table of local derivatives:
- Add passes the gradient through to both parents.
- Multiply sends each parent the gradient times the other parent's value.
- tanh sends the gradient times \(1 - t^2\), where \(t\) is its output.
Every rule uses +=, because a value used twice must add up both contributions.
import math
import random
class Value:
def __init__(self, data, parents=()):
self.data = data
self.grad = 0.0
self.parents = parents
self.backward_step = lambda: None
def __repr__(self):
return f"Value(data={self.data:.4f}, grad={self.grad:.4f})"
def __add__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data + other.data, (self, other))
def backward_step():
self.grad += out.grad
other.grad += out.grad
out.backward_step = backward_step
return out
def __mul__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data * other.data, (self, other))
def backward_step():
self.grad += other.data * out.grad
other.grad += self.data * out.grad
out.backward_step = backward_step
return out
def tanh(self):
t = math.tanh(self.data)
out = Value(t, (self,))
def backward_step():
self.grad += (1 - t * t) * out.grad
out.backward_step = backward_step
return out
def __neg__(self):
return self * -1
def __sub__(self, other):
return self + (-other)
def __radd__(self, other):
return self + other
def __rmul__(self, other):
return self * other
def backward(self):
order = []
visited = set()
def visit(node):
if node not in visited:
visited.add(node)
for parent in node.parents:
visit(parent)
order.append(node)
visit(self)
self.grad = 1.0
for node in reversed(order):
node.backward_step()
Step 2: subtraction and the backward pass
The second half of the class needs no new derivative rules. Subtraction is an add and a multiply by \(-1\), so it reuses both rules. The __radd__ and __rmul__ methods let a plain number sit on the left, as in 2 * x or sum(...), which starts from 0.
The backward method builds the topological order with a depth-first search. It sets the output gradient to \(1\) and runs each backward_step in reverse order.
Step 3: check the engine
Rebuild the graph from the computation graphs note, \(d = ab + a\) with \(a = 2\) and \(b = 3\). The engine should match the hand result.
a = Value(2.0)
b = Value(3.0)
d = a * b + a
d.backward()
print("check:", a.grad, b.grad)
It prints check: 4.0 2.0, the same as the backward pass done by hand.
Step 4: the network
A Neuron holds one weight per input and a bias, all as Value objects. Its forward pass is the score from the single neuron note, followed by tanh. Random weights between \(-1\) and \(1\) give each neuron a different starting point.
A Layer is a list of neurons that read the same input. An MLP chains layers. MLP([2, 4, 1]) has 2 inputs, one hidden layer with 4 neurons, and 1 output. Every class has parameters(), which returns the Value objects that training may change.
class Neuron:
def __init__(self, n_inputs):
self.w = [Value(random.uniform(-1, 1)) for _ in range(n_inputs)]
self.b = Value(0.0)
def __call__(self, x):
total = self.b
for wi, xi in zip(self.w, x):
total = total + wi * xi
return total.tanh()
def parameters(self):
return self.w + [self.b]
class Layer:
def __init__(self, n_inputs, n_outputs):
self.neurons = [Neuron(n_inputs) for _ in range(n_outputs)]
def __call__(self, x):
return [neuron(x) for neuron in self.neurons]
def parameters(self):
return [p for neuron in self.neurons for p in neuron.parameters()]
class MLP:
def __init__(self, sizes):
self.layers = [Layer(a, b) for a, b in zip(sizes, sizes[1:])]
def __call__(self, x):
for layer in self.layers:
x = layer(x)
return x[0] if len(x) == 1 else x
def parameters(self):
return [p for layer in self.layers for p in layer.parameters()]
The output neuron also uses tanh, so every prediction lies between \(-1\) and \(1\).
Step 5: train on XOR
The dataset is XOR, the task a single neuron could not learn. The targets are \(-1\) and \(1\) to match the tanh output range.
Each training step does four things:
- Run the forward pass on all four inputs.
- Add up the squared errors into one loss.
- Reset every gradient to zero, then call
loss.backward(). - Move every parameter against its gradient.
Step 3 resets the gradients first because the backward rules add to grad. Without the reset, each step would also carry the gradients of every earlier step.
xs = [[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]]
ys = [-1.0, 1.0, 1.0, -1.0]
random.seed(1)
model = MLP([2, 4, 1])
learning_rate = 0.1
for step in range(201):
predictions = [model(x) for x in xs]
loss = sum((p - y) * (p - y) for p, y in zip(predictions, ys))
for p in model.parameters():
p.grad = 0.0
loss.backward()
for p in model.parameters():
p.data -= learning_rate * p.grad
if step % 20 == 0:
print(f"step {step:3d} loss {loss.data:.4f}")
for x in xs:
print(x, round(model(x).data, 3))
The result
With seed 1 the program prints:
check: 4.0 2.0
step 0 loss 4.1022
step 20 loss 2.2341
step 40 loss 0.1625
step 60 loss 0.0761
step 80 loss 0.0476
step 100 loss 0.0341
step 120 loss 0.0263
step 140 loss 0.0213
step 160 loss 0.0178
step 180 loss 0.0152
step 200 loss 0.0133
[0.0, 0.0] -0.955
[0.0, 1.0] 0.939
[1.0, 0.0] 0.94
[1.0, 1.0] -0.938
The loss falls from about 4.1 to about 0.01. Each prediction ends close to its target of \(-1\) or \(1\). The network found XOR weights by itself, with gradients from your own engine.
Try next
- Change the seed. Each seed starts from different weights, so the loss curve changes. Seeds 0 to 19 all end below 0.07.
- Remove the hidden layer with
MLP([2, 1]). The loss gets stuck at 4.0, as the single neuron note predicts. - Raise
learning_rateto1.0. The first steps overshoot, the tanh units saturate near \(-1\) or \(1\), and the loss gets stuck near 8. - Add a
relumethod with the local derivative \(1\) for positive inputs and \(0\) otherwise. Use it in the hidden layer.
Sources
- Andrej Karpathy, micrograd, the design this engine follows
- Andrej Karpathy, The spelled-out intro to neural networks and backpropagation: building micrograd
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, Chapter 6: Deep Feedforward Networks, section 6.5
Cards
What four things does a scalar autograd Value store?
Reveal
Its number in data, its gradient in grad, the values it was made from in parents, and a backward_step function that pushes its gradient to those parents.
In a scalar autograd engine, what does backward_step do for out = self * other?
Reveal
self.grad += other.data * out.grad
other.grad += self.data * out.grad
In a scalar autograd engine, what does backward_step do for out = self.tanh() with output \(t\)?
Reveal
self.grad += (1 - t * t) * out.grad
Why do the backward rules in a scalar autograd engine use += on grad?
Reveal
A value used in several operations must add up the gradient from each use. Assigning with = would keep only the last one.
How can a scalar autograd engine support subtraction without a new backward rule?
Reveal
Write a - b as a + (b * -1). It reuses the add and multiply rules.
What does Value.backward() do in a scalar autograd engine?
Reveal
It builds a topological order with a depth-first search, sets its own gradient to \(1\), and calls backward_step on every node in reverse order.
Why does a training loop reset every gradient to zero before calling loss.backward()?
Reveal
The backward rules add to grad. Without the reset, each step would also carry the gradients of every earlier step.
What are the four parts of one training step for a small network?
Reveal
Run the forward pass, compute one loss, reset the gradients and backpropagate, then move every parameter against its gradient.
Why does a tanh network trained on XOR use targets \(-1\) and \(1\)?
Reveal
The tanh output lies between \(-1\) and \(1\), so those targets sit inside the range the network can produce.