Backpropagation

Backpropagation

Backpropagation runs the backward pass of a computation graph through a whole network. It finds the gradient of one final number with respect to every weight and bias in a single sweep. A 1986 paper by Rumelhart, Hinton, and Williams showed that it trains hidden layers well and made it widely known.

The idea

A network's forward pass is a long chain of adds, multiplies, and activations. Backpropagation starts at the final output and moves backward through that chain. At each operation it applies the chain rule with the local derivative.

One backward pass costs about as much as a few forward passes, however many parameters the network has. That efficiency is why large networks can train at all.

Parameter gradients

A parameter gradient is the derivative of the final output with respect to one trainable parameter. It says how the output moves when you nudge that parameter.

Worked example

Take a linear neuron \(y = wx + b\), written as two steps: \(a = wx\) and \(y = a + b\). Let \(x = 3\), \(w = 2\), and \(b = 1\). The forward pass gives \(a = 6\) and \(y = 7\).

Walk backward:

  • \(\frac{\partial y}{\partial a} = 1\) and \(\frac{\partial y}{\partial b} = 1\), from the add.
  • \(\frac{\partial a}{\partial w} = x = 3\), from the multiply.
  • The chain rule gives \(\frac{\partial y}{\partial w} = 1 \cdot 3 = 3\).

So nudging \(w\) moves \(y\) three times as much. Nudging \(b\) moves \(y\) by the same amount.

Through an activation

An activation adds one more step to the chain. The tanh function has a simple derivative in terms of its own output:

\[ \frac{d}{dz} \tanh(z) = 1 - \tanh(z)^2 \]

For a tanh neuron \(y = \tanh(wx + b)\) with \(z = wx + b\):

\[ \frac{\partial y}{\partial w} = (1 - y^2)\, x, \qquad \frac{\partial y}{\partial b} = 1 - y^2 \]

For ReLU the local derivative is \(1\) when \(z > 0\) and \(0\) when \(z < 0\). A ReLU unit that outputs zero passes no gradient back.

Gradient checking

A new backward rule can hide a bug. A gradient check compares it with the forward difference from the derivatives note. Nudge one parameter by a tiny \(h\), rerun the forward pass, and divide the change by \(h\). The two numbers should agree to several decimal places.

import math


def neuron(w, b, x):
    return math.tanh(w * x + b)


x, w, b = 2.0, -0.5, 0.5
z = w * x + b
y = math.tanh(z)

dy_dz = 1 - y**2
dy_dw = dy_dz * x
dy_db = dy_dz * 1.0

h = 1e-6
check_w = (neuron(w + h, b, x) - neuron(w, b, x)) / h
check_b = (neuron(w, b + h, x) - neuron(w, b, x)) / h
print(f"z = {z}, y = {y:.4f}")
print(f"dy/dw: chain rule {dy_dw:.4f}, estimate {check_w:.4f}")
print(f"dy/db: chain rule {dy_db:.4f}, estimate {check_b:.4f}")

Both rows agree: \(\frac{\partial y}{\partial w} \approx 1.5729\) and \(\frac{\partial y}{\partial b} \approx 0.7864\).

The forward difference needs one extra forward pass per parameter. That is too slow for training a network with millions of parameters. It is fast enough to test a backward rule on a small example.

What the gradient is taken of

So far the final number was the network output. For training, the final number is the loss, which measures how wrong the output is. The next note defines the loss.

Sources


Cards

What does backpropagation compute in a neural network?

Reveal

The gradient of one final number, usually the loss, with respect to every weight and bias. It applies the chain rule backward through the network's computation graph.

What is a parameter gradient?

Reveal

The derivative of the final output or loss with respect to one trainable parameter. It says how the result moves when that parameter is nudged.

For \(a = wx\) and \(y = a + b\) with \(x = -2\), \(w = 4\), and \(b = 3\), what is \(\frac{\partial y}{\partial w}\)?

Reveal

\(\frac{\partial y}{\partial a} = 1\) and \(\frac{\partial a}{\partial w} = x = -2\), so \(\frac{\partial y}{\partial w} = -2\).

For a linear neuron \(y = wx + b\), what is \(\frac{\partial y}{\partial b}\)?

Reveal

\(1\). A change in the bias moves the output by the same amount.

What is the derivative of \(\tanh(z)\), written in terms of its output?

Reveal
\[ \frac{d}{dz} \tanh(z) = 1 - \tanh(z)^2 \]

What is the local derivative of ReLU, \(\max(0, z)\)?

Reveal

\(1\) when \(z > 0\) and \(0\) when \(z < 0\). A ReLU unit that outputs zero passes no gradient back.

How does a gradient check test a backward rule?

Reveal

Nudge one parameter by a tiny \(h\), rerun the forward pass, and divide the output change by \(h\). The estimate should match the backpropagated gradient to several decimal places.

Why is the forward difference used to check gradients and rarely used to train?

Reveal

It needs one extra forward pass per parameter, which is far too slow for a network with millions of parameters. Backpropagation gets every gradient in about the cost of a few forward passes.