Activation functions

Activation functions

A stack of linear layers is still one linear function. An activation function bends each hidden value, and that bend lets a network learn shapes a straight line cannot. This note uses it to solve XOR.

Why linear layers collapse

Take two layers with no activation: \(f^{(1)}(x) = 2x\) and \(f^{(2)}(h) = 3h\). Their composition is

\[ f^{(2)}\big(f^{(1)}(x)\big) = 3(2x) = 6x, \]

which is one linear layer. The same holds for matrices. A layer that computes \(xW_1\), followed by a layer that multiplies by \(W_2\), gives \((xW_1)W_2 = x(W_1 W_2)\). The product \(W_1 W_2\) is a single matrix. Biases fold in the same way. Adding linear layers never adds power.

Activation function

A hidden unit applies a linear step and then a fixed nonlinear function \(g\):

\[ h = g(xW + c) \]

The function \(g\) is the activation function. It acts on each entry separately.

Two common choices:

Name Formula Output range
ReLU \(g(z) = \max(0, z)\) \([0, \infty)\)
tanh \(g(z) = \tanh(z)\) \((-1, 1)\)

ReLU sets negative values to zero and keeps positive values unchanged. It is the default choice in modern networks. The tanh function squashes any value into the range from \(-1\) to \(1\). The build note at the end of this journey uses tanh.

Worked example

Apply ReLU to \(z = \begin{bmatrix} -2 & 0.5 & 3 \end{bmatrix}\). The result is \(\begin{bmatrix} 0 & 0.5 & 3 \end{bmatrix}\).

Solving XOR

The Deep Learning book gives one exact XOR solution with two ReLU hidden units. The hidden layer uses

\[ W = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}, \qquad c = \begin{bmatrix} 0 & -1 \end{bmatrix}, \]

and the output layer uses weights \(w = (1, -2)\) and bias \(0\).

For each XOR input, compute \(h = \max(0, xW + c)\) and then \(y = h \cdot w\):

\(x\) \(xW + c\) \(h\) \(y\)
\((0, 0)\) \((0, -1)\) \((0, 0)\) \(0\)
\((0, 1)\) \((1, 0)\) \((1, 0)\) \(1\)
\((1, 0)\) \((1, 0)\) \((1, 0)\) \(1\)
\((1, 1)\) \((2, 1)\) \((2, 1)\) \(0\)

The outputs match XOR exactly. ReLU moved the four points into a new hidden space. There, \((0, 1)\) and \((1, 0)\) land on the same point, and one straight line separates the two classes.

def relu(z):
    return max(0, z)


W = [[1, 1], [1, 1]]
c = [0, -1]
w = [1, -2]

for x in [(0, 0), (0, 1), (1, 0), (1, 1)]:
    h = [relu(x[0] * W[0][j] + x[1] * W[1][j] + c[j]) for j in range(2)]
    y = h[0] * w[0] + h[1] * w[1]
    print(x, "hidden", h, "output", y)

The outputs are 0, 1, 1, and 0. Delete the call to relu and the outputs become 2, 1, 1, and 0, which no longer match XOR.

This network was set by hand. The rest of the journey builds the tools that let a network find such weights by itself.

Sources


Cards

Why does a network need a nonlinear activation between its layers?

Reveal

Linear layers composed together are still one linear function. The activation breaks that collapse, so extra layers add power.

Two layers with no activation compute \(f^{(1)}(x) = 2x\) and \(f^{(2)}(h) = 3h\). What single function is their composition?

Reveal

\(f^{(2)}\big(f^{(1)}(x)\big) = 3(2x) = 6x\), which is one linear layer.

What is the formula for the ReLU activation?

Reveal
\[ g(z) = \max(0, z) \]

It sets negative values to zero and keeps positive values unchanged.

What range of outputs does the tanh activation produce?

Reveal

Every output lies strictly between \(-1\) and \(1\).

Apply ReLU to \(z = \begin{bmatrix} -4 & 2 & 0 & 1.5 \end{bmatrix}\).

Reveal

\(\begin{bmatrix} 0 & 2 & 0 & 1.5 \end{bmatrix}\)

How does a hidden layer with an activation function \(g\) compute its output \(h\) from input \(x\)?

Reveal
\[ h = g(xW + c) \]

A linear step comes first, then \(g\) acts on each entry separately.

How does a one-hidden-layer ReLU network solve XOR?

Reveal

The ReLU hidden layer moves the four inputs into a new space. There, \((0, 1)\) and \((1, 0)\) land on the same point, and one straight line separates the two classes.