Backpropagation
Backpropagation
Backpropagation runs the backward pass of a computation graph through a whole network. It finds the gradient of one final number with respect to every weight and bias in a single sweep. A 1986 paper by Rumelhart, Hinton, and Williams showed that it trains hidden layers well and made it widely known.
The idea
A network's forward pass is a long chain of adds, multiplies, and activations. Backpropagation starts at the final output and moves backward through that chain. At each operation it applies the chain rule with the local derivative.
One backward pass costs about as much as a few forward passes, however many parameters the network has. That efficiency is why large networks can train at all.
Parameter gradients
A parameter gradient is the derivative of the final output with respect to one trainable parameter. It says how the output moves when you nudge that parameter.
Worked example
Take a linear neuron \(y = wx + b\), written as two steps: \(a = wx\) and \(y = a + b\). Let \(x = 3\), \(w = 2\), and \(b = 1\). The forward pass gives \(a = 6\) and \(y = 7\).
Walk backward:
- \(\frac{\partial y}{\partial a} = 1\) and \(\frac{\partial y}{\partial b} = 1\), from the add.
- \(\frac{\partial a}{\partial w} = x = 3\), from the multiply.
- The chain rule gives \(\frac{\partial y}{\partial w} = 1 \cdot 3 = 3\).
So nudging \(w\) moves \(y\) three times as much. Nudging \(b\) moves \(y\) by the same amount.
Through an activation
An activation adds one more step to the chain. The tanh function has a simple derivative in terms of its own output:
For a tanh neuron \(y = \tanh(wx + b)\) with \(z = wx + b\):
For ReLU the local derivative is \(1\) when \(z > 0\) and \(0\) when \(z < 0\). A ReLU unit that outputs zero passes no gradient back.
Gradient checking
A new backward rule can hide a bug. A gradient check compares it with the forward difference from the derivatives note. Nudge one parameter by a tiny \(h\), rerun the forward pass, and divide the change by \(h\). The two numbers should agree to several decimal places.
import math
def neuron(w, b, x):
return math.tanh(w * x + b)
x, w, b = 2.0, -0.5, 0.5
z = w * x + b
y = math.tanh(z)
dy_dz = 1 - y**2
dy_dw = dy_dz * x
dy_db = dy_dz * 1.0
h = 1e-6
check_w = (neuron(w + h, b, x) - neuron(w, b, x)) / h
check_b = (neuron(w, b + h, x) - neuron(w, b, x)) / h
print(f"z = {z}, y = {y:.4f}")
print(f"dy/dw: chain rule {dy_dw:.4f}, estimate {check_w:.4f}")
print(f"dy/db: chain rule {dy_db:.4f}, estimate {check_b:.4f}")
Both rows agree: \(\frac{\partial y}{\partial w} \approx 1.5729\) and \(\frac{\partial y}{\partial b} \approx 0.7864\).
The forward difference needs one extra forward pass per parameter. That is too slow for training a network with millions of parameters. It is fast enough to test a backward rule on a small example.
What the gradient is taken of
So far the final number was the network output. For training, the final number is the loss, which measures how wrong the output is. The next note defines the loss.
Sources
- David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams, Learning representations by back-propagating errors, Nature 323, 1986
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, Chapter 6: Deep Feedforward Networks, section 6.5
- 3Blue1Brown, Neural Networks, Chapter 4: Backpropagation calculus
- Andrej Karpathy, micrograd
Cards
What does backpropagation compute in a neural network?
Reveal
The gradient of one final number, usually the loss, with respect to every weight and bias. It applies the chain rule backward through the network's computation graph.
What is a parameter gradient?
Reveal
The derivative of the final output or loss with respect to one trainable parameter. It says how the result moves when that parameter is nudged.
For \(a = wx\) and \(y = a + b\) with \(x = -2\), \(w = 4\), and \(b = 3\), what is \(\frac{\partial y}{\partial w}\)?
Reveal
\(\frac{\partial y}{\partial a} = 1\) and \(\frac{\partial a}{\partial w} = x = -2\), so \(\frac{\partial y}{\partial w} = -2\).
For a linear neuron \(y = wx + b\), what is \(\frac{\partial y}{\partial b}\)?
Reveal
\(1\). A change in the bias moves the output by the same amount.
What is the derivative of \(\tanh(z)\), written in terms of its output?
Reveal
What is the local derivative of ReLU, \(\max(0, z)\)?
Reveal
\(1\) when \(z > 0\) and \(0\) when \(z < 0\). A ReLU unit that outputs zero passes no gradient back.
How does a gradient check test a backward rule?
Reveal
Nudge one parameter by a tiny \(h\), rerun the forward pass, and divide the output change by \(h\). The estimate should match the backpropagated gradient to several decimal places.
Why is the forward difference used to check gradients and rarely used to train?
Reveal
It needs one extra forward pass per parameter, which is far too slow for a network with millions of parameters. Backpropagation gets every gradient in about the cost of a few forward passes.