Layers and the forward pass

Layers and the forward pass

A network is many neurons arranged in layers. This note shows how values flow from the input to the output. Nothing is trained yet.

Layer

A layer is a row of neurons that all read the same input. Each neuron has its own weights and bias. With matrices, one layer is one product plus a bias row:

\[ z = xW + b \]

Here \(x\) is the input row, column \(j\) of \(W\) holds the weights of neuron \(j\), and \(b\) holds one bias per neuron. A layer with 2 inputs and 3 neurons has a \(2 \times 3\) weight matrix and 3 biases.

Worked example

Let

\[ x = \begin{bmatrix} 2 & 1 \end{bmatrix}, \qquad W = \begin{bmatrix} 1 & 3 \\ 2 & -1 \end{bmatrix}, \qquad b = \begin{bmatrix} 1 & 0 \end{bmatrix}. \]

First \(xW = \begin{bmatrix} 2 + 2 & 6 - 1 \end{bmatrix} = \begin{bmatrix} 4 & 5 \end{bmatrix}\). Adding the bias gives \(z = \begin{bmatrix} 5 & 5 \end{bmatrix}\).

A network is a composition

A feedforward network applies layers one after another. With three layers:

\[ f(x) = f^{(3)}\big(f^{(2)}\big(f^{(1)}(x)\big)\big) \]

Each layer's output is the next layer's input. Values only move forward. No layer feeds back into an earlier one.

Hidden layers

The last layer is the output layer. Every layer between the input and the output is a hidden layer. Its values are a hidden representation.

The training data gives a target for the final output only. It never says what a hidden layer should produce. Training has to discover useful hidden values by itself.

Worked example

Feed the hidden values \(h = \begin{bmatrix} 5 & 5 \end{bmatrix}\) from above into an output layer with one neuron, weights \((2, -1)\), and bias \(0\):

\[ y = 5(2) + 5(-1) + 0 = 5 \]

Forward pass

The forward pass evaluates the network from input to output with the current parameters. For the example network:

  1. Input: \(x = (2, 1)\)
  2. Hidden layer: \(h = (5, 5)\)
  3. Output layer: \(y = 5\)

The forward pass computes no gradients and changes no parameters. It only evaluates the current function.

In Python

def layer(x, W, b):
    return [sum(x[k] * W[k][j] for k in range(len(x))) + b[j] for j in range(len(b))]


x = [2, 1]
h = layer(x, [[1, 3], [2, -1]], [1, 0])
y = layer(h, [[2], [-1]], [0])
print("hidden:", h)
print("output:", y)

It prints hidden: [5, 5] and output: [5].

This network is still linear. Two linear layers in a row can always be rewritten as one linear layer, so stacking them adds no power. The next note adds the step that changes this.

Sources


Cards

How does one layer of a network transform an input row \(x\)?

Reveal
\[ z = xW + b \]

Column \(j\) of \(W\) holds the weights of neuron \(j\), and \(b\) holds one bias per neuron.

A layer has 2 inputs and 3 neurons. What shapes do its weights and biases have?

Reveal

The weight matrix is \(2 \times 3\), and there are 3 biases, one per neuron.

How is a three-layer feedforward network written as a composition of functions?

Reveal
\[ f(x) = f^{(3)}\big(f^{(2)}\big(f^{(1)}(x)\big)\big) \]

Each layer's output is the next layer's input.

What is a hidden layer in a neural network?

Reveal

Any layer between the input and the output layer. The training data gives targets for the output only, so training must discover the hidden values by itself.

What does the forward pass of a neural network do?

Reveal

It evaluates the network from input to output with the current parameters. It computes no gradients and changes no parameters.

A layer computes \(z = xW + b\). Find \(z\) for

\[ x = \begin{bmatrix} 1 & 3 \end{bmatrix}, \quad W = \begin{bmatrix} 2 & -1 \\ 4 & 2 \end{bmatrix}, \quad b = \begin{bmatrix} -2 & 1 \end{bmatrix}. \]
Reveal

First \(xW = \begin{bmatrix} 14 & 5 \end{bmatrix}\).

Then \(z = \begin{bmatrix} 14 - 2 & 5 + 1 \end{bmatrix} = \begin{bmatrix} 12 & 6 \end{bmatrix}\).

A hidden layer outputs \(h = (2, -3)\). The output neuron has weights \((4, 1)\) and bias \(2\). What is its output before any activation?

Reveal

\(y = 2(4) + (-3)(1) + 2 = 7\).