Activation functions
Activation functions
A stack of linear layers is still one linear function. An activation function bends each hidden value, and that bend lets a network learn shapes a straight line cannot. This note uses it to solve XOR.
Why linear layers collapse
Take two layers with no activation: \(f^{(1)}(x) = 2x\) and \(f^{(2)}(h) = 3h\). Their composition is
which is one linear layer. The same holds for matrices. A layer that computes \(xW_1\), followed by a layer that multiplies by \(W_2\), gives \((xW_1)W_2 = x(W_1 W_2)\). The product \(W_1 W_2\) is a single matrix. Biases fold in the same way. Adding linear layers never adds power.
Activation function
A hidden unit applies a linear step and then a fixed nonlinear function \(g\):
The function \(g\) is the activation function. It acts on each entry separately.
Two common choices:
| Name | Formula | Output range |
|---|---|---|
| ReLU | \(g(z) = \max(0, z)\) | \([0, \infty)\) |
| tanh | \(g(z) = \tanh(z)\) | \((-1, 1)\) |
ReLU sets negative values to zero and keeps positive values unchanged. It is the default choice in modern networks. The tanh function squashes any value into the range from \(-1\) to \(1\). The build note at the end of this journey uses tanh.
Worked example
Apply ReLU to \(z = \begin{bmatrix} -2 & 0.5 & 3 \end{bmatrix}\). The result is \(\begin{bmatrix} 0 & 0.5 & 3 \end{bmatrix}\).
Solving XOR
The Deep Learning book gives one exact XOR solution with two ReLU hidden units. The hidden layer uses
and the output layer uses weights \(w = (1, -2)\) and bias \(0\).
For each XOR input, compute \(h = \max(0, xW + c)\) and then \(y = h \cdot w\):
| \(x\) | \(xW + c\) | \(h\) | \(y\) |
|---|---|---|---|
| \((0, 0)\) | \((0, -1)\) | \((0, 0)\) | \(0\) |
| \((0, 1)\) | \((1, 0)\) | \((1, 0)\) | \(1\) |
| \((1, 0)\) | \((1, 0)\) | \((1, 0)\) | \(1\) |
| \((1, 1)\) | \((2, 1)\) | \((2, 1)\) | \(0\) |
The outputs match XOR exactly. ReLU moved the four points into a new hidden space. There, \((0, 1)\) and \((1, 0)\) land on the same point, and one straight line separates the two classes.
def relu(z):
return max(0, z)
W = [[1, 1], [1, 1]]
c = [0, -1]
w = [1, -2]
for x in [(0, 0), (0, 1), (1, 0), (1, 1)]:
h = [relu(x[0] * W[0][j] + x[1] * W[1][j] + c[j]) for j in range(2)]
y = h[0] * w[0] + h[1] * w[1]
print(x, "hidden", h, "output", y)
The outputs are 0, 1, 1, and 0. Delete the call to relu and the outputs become 2, 1, 1, and 0, which no longer match XOR.
This network was set by hand. The rest of the journey builds the tools that let a network find such weights by itself.
Sources
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, Chapter 6: Deep Feedforward Networks, sections 6.1 and 6.3
Cards
Why does a network need a nonlinear activation between its layers?
Reveal
Linear layers composed together are still one linear function. The activation breaks that collapse, so extra layers add power.
Two layers with no activation compute \(f^{(1)}(x) = 2x\) and \(f^{(2)}(h) = 3h\). What single function is their composition?
Reveal
\(f^{(2)}\big(f^{(1)}(x)\big) = 3(2x) = 6x\), which is one linear layer.
What is the formula for the ReLU activation?
Reveal
It sets negative values to zero and keeps positive values unchanged.
What range of outputs does the tanh activation produce?
Reveal
Every output lies strictly between \(-1\) and \(1\).
Apply ReLU to \(z = \begin{bmatrix} -4 & 2 & 0 & 1.5 \end{bmatrix}\).
Reveal
\(\begin{bmatrix} 0 & 2 & 0 & 1.5 \end{bmatrix}\)
How does a hidden layer with an activation function \(g\) compute its output \(h\) from input \(x\)?
Reveal
A linear step comes first, then \(g\) acts on each entry separately.
How does a one-hidden-layer ReLU network solve XOR?
Reveal
The ReLU hidden layer moves the four inputs into a new space. There, \((0, 1)\) and \((1, 0)\) land on the same point, and one straight line separates the two classes.