Derivatives and the chain rule

Derivatives and the chain rule

Training needs to know how each parameter affects the error. Derivatives answer that question. This note covers the three ideas training uses: the derivative, the gradient, and the chain rule.

Derivative

For a function \(y = f(x)\), the derivative \(f'(x)\) is the local slope at \(x\). It says how fast the output changes when the input changes a little. For a small step \(\epsilon\):

\[ f(x + \epsilon) \approx f(x) + \epsilon f'(x) \]

Worked example

Let \(f(x) = x^2\), so \(f'(x) = 2x\). At \(x = 3\) the slope is \(6\). The approximation predicts

\[ f(3.01) \approx 9 + 0.01(6) = 9.06. \]

The exact value is \(9.0601\), very close to the prediction.

Estimating a derivative with numbers

You can estimate a slope with no calculus. Nudge the input by a tiny \(h\) and measure how much the output moves:

\[ f'(x) \approx \frac{f(x + h) - f(x)}{h} \]

This is the forward difference.

def f(x):
    return x**2


h = 0.0001
estimate = (f(3 + h) - f(3)) / h
print(round(estimate, 6))

It prints 6.0001. The exact slope is \(6\). The estimate gets closer as \(h\) shrinks, until floating point rounding takes over. A later note uses this estimate to check gradients.

Gradient

A function of several inputs has one partial derivative per input. The partial derivative \(\frac{\partial f}{\partial x_1}\) is the slope along \(x_1\) while every other input stays fixed. The gradient \(\nabla f\) collects all of them in one vector.

Worked example

Let \(f(x_1, x_2) = x_1 x_2\). Holding \(x_2\) fixed, the slope along \(x_1\) is \(x_2\). Holding \(x_1\) fixed, the slope along \(x_2\) is \(x_1\). At \((2, 3)\):

\[ \nabla f = \begin{bmatrix} \frac{\partial f}{\partial x_1} \\ \frac{\partial f}{\partial x_2} \end{bmatrix} = \begin{bmatrix} 3 \\ 2 \end{bmatrix} \]

The gradient points in the direction where \(f\) grows fastest. Training steps the opposite way to shrink the error.

Chain rule

When \(x\) affects \(z\) through a middle value \(y\), the chain rule multiplies the two local slopes. If \(y = g(x)\) and \(z = f(y)\), then

\[ \frac{dz}{dx} = \frac{dz}{dy} \cdot \frac{dy}{dx}. \]

Worked example

Let \(y = x^2\) and \(z = 3y\). At \(x = 2\):

  • \(\frac{dy}{dx} = 2x = 4\)
  • \(\frac{dz}{dy} = 3\)
  • \(\frac{dz}{dx} = 3 \cdot 4 = 12\)

Check it directly: \(z = 3x^2\), so \(\frac{dz}{dx} = 6x = 12\) at \(x = 2\).

A network is a long chain of small steps. The chain rule lets you find the slope of the whole chain by multiplying the slopes of its steps. The next note organizes that work.

Sources


Cards

What does the derivative \(f'(x)\) tell you about a function \(f\)?

Reveal

The local slope at \(x\). It says how fast the output changes when the input changes a little.

How does the derivative approximate \(f(x + \epsilon)\) for a small step \(\epsilon\)?

Reveal
\[ f(x + \epsilon) \approx f(x) + \epsilon f'(x) \]

What is the forward difference estimate of \(f'(x)\)?

Reveal
\[ f'(x) \approx \frac{f(x + h) - f(x)}{h} \]

for a small \(h\).

Use a forward difference with \(h = 0.001\) to estimate \(f'(2)\) for \(f(x) = x^2\).

Reveal
\[ \frac{4.004001 - 4}{0.001} = 4.001 \]

The exact slope is \(4\).

What is the gradient \(\nabla f\) of a function with several inputs?

Reveal

The vector of its partial derivatives, one per input. It points in the direction where \(f\) grows fastest.

For \(f(x_1, x_2) = x_1 x_2\), what is \(\nabla f\) at \(x_1 = -4\) and \(x_2 = 2\)?

Reveal

\(\frac{\partial f}{\partial x_1} = x_2 = 2\) and \(\frac{\partial f}{\partial x_2} = x_1 = -4\), so

\[ \nabla f = \begin{bmatrix} 2 \\ -4 \end{bmatrix}. \]

State the chain rule when \(y = g(x)\) and \(z = f(y)\).

Reveal
\[ \frac{dz}{dx} = \frac{dz}{dy} \cdot \frac{dy}{dx} \]

Let \(y = x^2\) and \(z = 5y\). What is \(\frac{dz}{dx}\) at \(x = -2\)?

Reveal

\(\frac{dy}{dx} = 2x = -4\) and \(\frac{dz}{dy} = 5\), so \(\frac{dz}{dx} = 5(-4) = -20\).