Layers and the forward pass
Layers and the forward pass
A network is many neurons arranged in layers. This note shows how values flow from the input to the output. Nothing is trained yet.
Layer
A layer is a row of neurons that all read the same input. Each neuron has its own weights and bias. With matrices, one layer is one product plus a bias row:
Here \(x\) is the input row, column \(j\) of \(W\) holds the weights of neuron \(j\), and \(b\) holds one bias per neuron. A layer with 2 inputs and 3 neurons has a \(2 \times 3\) weight matrix and 3 biases.
Worked example
Let
First \(xW = \begin{bmatrix} 2 + 2 & 6 - 1 \end{bmatrix} = \begin{bmatrix} 4 & 5 \end{bmatrix}\). Adding the bias gives \(z = \begin{bmatrix} 5 & 5 \end{bmatrix}\).
A network is a composition
A feedforward network applies layers one after another. With three layers:
Each layer's output is the next layer's input. Values only move forward. No layer feeds back into an earlier one.
Hidden layers
The last layer is the output layer. Every layer between the input and the output is a hidden layer. Its values are a hidden representation.
The training data gives a target for the final output only. It never says what a hidden layer should produce. Training has to discover useful hidden values by itself.
Worked example
Feed the hidden values \(h = \begin{bmatrix} 5 & 5 \end{bmatrix}\) from above into an output layer with one neuron, weights \((2, -1)\), and bias \(0\):
Forward pass
The forward pass evaluates the network from input to output with the current parameters. For the example network:
- Input: \(x = (2, 1)\)
- Hidden layer: \(h = (5, 5)\)
- Output layer: \(y = 5\)
The forward pass computes no gradients and changes no parameters. It only evaluates the current function.
In Python
def layer(x, W, b):
return [sum(x[k] * W[k][j] for k in range(len(x))) + b[j] for j in range(len(b))]
x = [2, 1]
h = layer(x, [[1, 3], [2, -1]], [1, 0])
y = layer(h, [[2], [-1]], [0])
print("hidden:", h)
print("output:", y)
It prints hidden: [5, 5] and output: [5].
This network is still linear. Two linear layers in a row can always be rewritten as one linear layer, so stacking them adds no power. The next note adds the step that changes this.
Sources
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, Chapter 6: Deep Feedforward Networks
- 3Blue1Brown, Neural Networks, Chapter 1: But what is a neural network?
Cards
How does one layer of a network transform an input row \(x\)?
Reveal
Column \(j\) of \(W\) holds the weights of neuron \(j\), and \(b\) holds one bias per neuron.
A layer has 2 inputs and 3 neurons. What shapes do its weights and biases have?
Reveal
The weight matrix is \(2 \times 3\), and there are 3 biases, one per neuron.
How is a three-layer feedforward network written as a composition of functions?
Reveal
Each layer's output is the next layer's input.
What is a hidden layer in a neural network?
Reveal
Any layer between the input and the output layer. The training data gives targets for the output only, so training must discover the hidden values by itself.
What does the forward pass of a neural network do?
Reveal
It evaluates the network from input to output with the current parameters. It computes no gradients and changes no parameters.
A layer computes \(z = xW + b\). Find \(z\) for
Reveal
First \(xW = \begin{bmatrix} 14 & 5 \end{bmatrix}\).
Then \(z = \begin{bmatrix} 14 - 2 & 5 + 1 \end{bmatrix} = \begin{bmatrix} 12 & 6 \end{bmatrix}\).
A hidden layer outputs \(h = (2, -3)\). The output neuron has weights \((4, 1)\) and bias \(2\). What is its output before any activation?
Reveal
\(y = 2(4) + (-3)(1) + 2 = 7\).