Derivatives and the chain rule
Derivatives and the chain rule
Training needs to know how each parameter affects the error. Derivatives answer that question. This note covers the three ideas training uses: the derivative, the gradient, and the chain rule.
Derivative
For a function \(y = f(x)\), the derivative \(f'(x)\) is the local slope at \(x\). It says how fast the output changes when the input changes a little. For a small step \(\epsilon\):
Worked example
Let \(f(x) = x^2\), so \(f'(x) = 2x\). At \(x = 3\) the slope is \(6\). The approximation predicts
The exact value is \(9.0601\), very close to the prediction.
Estimating a derivative with numbers
You can estimate a slope with no calculus. Nudge the input by a tiny \(h\) and measure how much the output moves:
This is the forward difference.
def f(x):
return x**2
h = 0.0001
estimate = (f(3 + h) - f(3)) / h
print(round(estimate, 6))
It prints 6.0001. The exact slope is \(6\). The estimate gets closer as \(h\) shrinks, until floating point rounding takes over. A later note uses this estimate to check gradients.
Gradient
A function of several inputs has one partial derivative per input. The partial derivative \(\frac{\partial f}{\partial x_1}\) is the slope along \(x_1\) while every other input stays fixed. The gradient \(\nabla f\) collects all of them in one vector.
Worked example
Let \(f(x_1, x_2) = x_1 x_2\). Holding \(x_2\) fixed, the slope along \(x_1\) is \(x_2\). Holding \(x_1\) fixed, the slope along \(x_2\) is \(x_1\). At \((2, 3)\):
The gradient points in the direction where \(f\) grows fastest. Training steps the opposite way to shrink the error.
Chain rule
When \(x\) affects \(z\) through a middle value \(y\), the chain rule multiplies the two local slopes. If \(y = g(x)\) and \(z = f(y)\), then
Worked example
Let \(y = x^2\) and \(z = 3y\). At \(x = 2\):
- \(\frac{dy}{dx} = 2x = 4\)
- \(\frac{dz}{dy} = 3\)
- \(\frac{dz}{dx} = 3 \cdot 4 = 12\)
Check it directly: \(z = 3x^2\), so \(\frac{dz}{dx} = 6x = 12\) at \(x = 2\).
A network is a long chain of small steps. The chain rule lets you find the slope of the whole chain by multiplying the slopes of its steps. The next note organizes that work.
Sources
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, Chapter 4: Numerical Computation
- 3Blue1Brown, Essence of Calculus
Cards
What does the derivative \(f'(x)\) tell you about a function \(f\)?
Reveal
The local slope at \(x\). It says how fast the output changes when the input changes a little.
How does the derivative approximate \(f(x + \epsilon)\) for a small step \(\epsilon\)?
Reveal
What is the forward difference estimate of \(f'(x)\)?
Reveal
for a small \(h\).
Use a forward difference with \(h = 0.001\) to estimate \(f'(2)\) for \(f(x) = x^2\).
Reveal
The exact slope is \(4\).
What is the gradient \(\nabla f\) of a function with several inputs?
Reveal
The vector of its partial derivatives, one per input. It points in the direction where \(f\) grows fastest.
For \(f(x_1, x_2) = x_1 x_2\), what is \(\nabla f\) at \(x_1 = -4\) and \(x_2 = 2\)?
Reveal
\(\frac{\partial f}{\partial x_1} = x_2 = 2\) and \(\frac{\partial f}{\partial x_2} = x_1 = -4\), so
State the chain rule when \(y = g(x)\) and \(z = f(y)\).
Reveal
Let \(y = x^2\) and \(z = 5y\). What is \(\frac{dz}{dx}\) at \(x = -2\)?
Reveal
\(\frac{dy}{dx} = 2x = -4\) and \(\frac{dz}{dy} = 5\), so \(\frac{dz}{dx} = 5(-4) = -20\).