Build a tiny autograd

Build a tiny autograd

This note puts the whole journey into one program of about 100 lines. You build a scalar autograd engine, wrap it in a small network, and train the network to learn XOR. The design follows Andrej Karpathy's micrograd. It uses plain Python with no libraries beyond math and random.

Copy the code blocks in order into one file, such as tiny_autograd.py, and run it.

The plan

Part What it does Earlier note
Value Holds a number, its gradient, and how it was made Computation graphs
backward Fills every gradient in topological order Computation graphs, Backpropagation
Neuron, Layer, MLP Build a network from Value objects A single neuron, Layers and the forward pass
Training loop Squared error loss and gradient descent Loss functions, Gradient descent

Step 1: the Value class

A Value wraps one number. It stores the number in data, its gradient in grad, the values it was made from in parents, and a function backward_step. That function pushes the node's gradient to its parents with the local derivative.

Each operation creates a new Value and attaches the matching backward rule. The rules come straight from the table of local derivatives:

  • Add passes the gradient through to both parents.
  • Multiply sends each parent the gradient times the other parent's value.
  • tanh sends the gradient times \(1 - t^2\), where \(t\) is its output.

Every rule uses +=, because a value used twice must add up both contributions.

import math
import random


class Value:
    def __init__(self, data, parents=()):
        self.data = data
        self.grad = 0.0
        self.parents = parents
        self.backward_step = lambda: None

    def __repr__(self):
        return f"Value(data={self.data:.4f}, grad={self.grad:.4f})"

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other))

        def backward_step():
            self.grad += out.grad
            other.grad += out.grad

        out.backward_step = backward_step
        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other))

        def backward_step():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad

        out.backward_step = backward_step
        return out

    def tanh(self):
        t = math.tanh(self.data)
        out = Value(t, (self,))

        def backward_step():
            self.grad += (1 - t * t) * out.grad

        out.backward_step = backward_step
        return out

    def __neg__(self):
        return self * -1

    def __sub__(self, other):
        return self + (-other)

    def __radd__(self, other):
        return self + other

    def __rmul__(self, other):
        return self * other

    def backward(self):
        order = []
        visited = set()

        def visit(node):
            if node not in visited:
                visited.add(node)
                for parent in node.parents:
                    visit(parent)
                order.append(node)

        visit(self)
        self.grad = 1.0
        for node in reversed(order):
            node.backward_step()

Step 2: subtraction and the backward pass

The second half of the class needs no new derivative rules. Subtraction is an add and a multiply by \(-1\), so it reuses both rules. The __radd__ and __rmul__ methods let a plain number sit on the left, as in 2 * x or sum(...), which starts from 0.

The backward method builds the topological order with a depth-first search. It sets the output gradient to \(1\) and runs each backward_step in reverse order.

Step 3: check the engine

Rebuild the graph from the computation graphs note, \(d = ab + a\) with \(a = 2\) and \(b = 3\). The engine should match the hand result.

a = Value(2.0)
b = Value(3.0)
d = a * b + a
d.backward()
print("check:", a.grad, b.grad)

It prints check: 4.0 2.0, the same as the backward pass done by hand.

Step 4: the network

A Neuron holds one weight per input and a bias, all as Value objects. Its forward pass is the score from the single neuron note, followed by tanh. Random weights between \(-1\) and \(1\) give each neuron a different starting point.

A Layer is a list of neurons that read the same input. An MLP chains layers. MLP([2, 4, 1]) has 2 inputs, one hidden layer with 4 neurons, and 1 output. Every class has parameters(), which returns the Value objects that training may change.

class Neuron:
    def __init__(self, n_inputs):
        self.w = [Value(random.uniform(-1, 1)) for _ in range(n_inputs)]
        self.b = Value(0.0)

    def __call__(self, x):
        total = self.b
        for wi, xi in zip(self.w, x):
            total = total + wi * xi
        return total.tanh()

    def parameters(self):
        return self.w + [self.b]


class Layer:
    def __init__(self, n_inputs, n_outputs):
        self.neurons = [Neuron(n_inputs) for _ in range(n_outputs)]

    def __call__(self, x):
        return [neuron(x) for neuron in self.neurons]

    def parameters(self):
        return [p for neuron in self.neurons for p in neuron.parameters()]


class MLP:
    def __init__(self, sizes):
        self.layers = [Layer(a, b) for a, b in zip(sizes, sizes[1:])]

    def __call__(self, x):
        for layer in self.layers:
            x = layer(x)
        return x[0] if len(x) == 1 else x

    def parameters(self):
        return [p for layer in self.layers for p in layer.parameters()]

The output neuron also uses tanh, so every prediction lies between \(-1\) and \(1\).

Step 5: train on XOR

The dataset is XOR, the task a single neuron could not learn. The targets are \(-1\) and \(1\) to match the tanh output range.

Each training step does four things:

  1. Run the forward pass on all four inputs.
  2. Add up the squared errors into one loss.
  3. Reset every gradient to zero, then call loss.backward().
  4. Move every parameter against its gradient.

Step 3 resets the gradients first because the backward rules add to grad. Without the reset, each step would also carry the gradients of every earlier step.

xs = [[0.0, 0.0], [0.0, 1.0], [1.0, 0.0], [1.0, 1.0]]
ys = [-1.0, 1.0, 1.0, -1.0]

random.seed(1)
model = MLP([2, 4, 1])
learning_rate = 0.1

for step in range(201):
    predictions = [model(x) for x in xs]
    loss = sum((p - y) * (p - y) for p, y in zip(predictions, ys))

    for p in model.parameters():
        p.grad = 0.0
    loss.backward()

    for p in model.parameters():
        p.data -= learning_rate * p.grad

    if step % 20 == 0:
        print(f"step {step:3d}  loss {loss.data:.4f}")

for x in xs:
    print(x, round(model(x).data, 3))

The result

With seed 1 the program prints:

check: 4.0 2.0
step   0  loss 4.1022
step  20  loss 2.2341
step  40  loss 0.1625
step  60  loss 0.0761
step  80  loss 0.0476
step 100  loss 0.0341
step 120  loss 0.0263
step 140  loss 0.0213
step 160  loss 0.0178
step 180  loss 0.0152
step 200  loss 0.0133
[0.0, 0.0] -0.955
[0.0, 1.0] 0.939
[1.0, 0.0] 0.94
[1.0, 1.0] -0.938

The loss falls from about 4.1 to about 0.01. Each prediction ends close to its target of \(-1\) or \(1\). The network found XOR weights by itself, with gradients from your own engine.

Try next

  • Change the seed. Each seed starts from different weights, so the loss curve changes. Seeds 0 to 19 all end below 0.07.
  • Remove the hidden layer with MLP([2, 1]). The loss gets stuck at 4.0, as the single neuron note predicts.
  • Raise learning_rate to 1.0. The first steps overshoot, the tanh units saturate near \(-1\) or \(1\), and the loss gets stuck near 8.
  • Add a relu method with the local derivative \(1\) for positive inputs and \(0\) otherwise. Use it in the hidden layer.

Sources


Cards

What four things does a scalar autograd Value store?

Reveal

Its number in data, its gradient in grad, the values it was made from in parents, and a backward_step function that pushes its gradient to those parents.

In a scalar autograd engine, what does backward_step do for out = self * other?

Reveal
self.grad += other.data * out.grad
other.grad += self.data * out.grad

In a scalar autograd engine, what does backward_step do for out = self.tanh() with output \(t\)?

Reveal
self.grad += (1 - t * t) * out.grad

Why do the backward rules in a scalar autograd engine use += on grad?

Reveal

A value used in several operations must add up the gradient from each use. Assigning with = would keep only the last one.

How can a scalar autograd engine support subtraction without a new backward rule?

Reveal

Write a - b as a + (b * -1). It reuses the add and multiply rules.

What does Value.backward() do in a scalar autograd engine?

Reveal

It builds a topological order with a depth-first search, sets its own gradient to \(1\), and calls backward_step on every node in reverse order.

Why does a training loop reset every gradient to zero before calling loss.backward()?

Reveal

The backward rules add to grad. Without the reset, each step would also carry the gradients of every earlier step.

What are the four parts of one training step for a small network?

Reveal

Run the forward pass, compute one loss, reset the gradients and backpropagate, then move every parameter against its gradient.

Why does a tanh network trained on XOR use targets \(-1\) and \(1\)?

Reveal

The tanh output lies between \(-1\) and \(1\), so those targets sit inside the range the network can produce.