Matrix multiplication
Matrix multiplication
A neural network does most of its work with one operation: matrix multiplication. This note covers what a matrix stores and how a product combines inputs with weights.
Matrix
A matrix is a rectangular grid of numbers. Its shape is its number of rows, then its number of columns.
This matrix has shape \(2 \times 2\). A single row of numbers is also a matrix, with shape \(1 \times n\).
Networks use matrices because one product handles many inputs and many weights at once.
The product rule
If \(A\) has shape \(m \times n\) and \(B\) has shape \(n \times p\), then \(C = AB\) has shape \(m \times p\). Each entry of \(C\) is a sum over the shared index \(k\):
In words, entry \((i, j)\) pairs row \(i\) of \(A\) with column \(j\) of \(B\). It multiplies matching entries and adds the results.
The inner sizes must match. A \(2 \times 3\) matrix times a \(3 \times 4\) matrix works and gives \(2 \times 4\). A \(2 \times 3\) matrix times a \(2 \times 3\) matrix has no product.
Worked example
Take one input with three values and a weight matrix for two outputs:
The input has 3 entries, so \(W\) has 3 rows. \(W\) has 2 columns because we want 2 outputs.
The first output pairs \(x\) with the first column of \(W\):
The second output pairs \(x\) with the second column:
So \(xW = \begin{bmatrix} 14 & 8 \end{bmatrix}\).
In Python
A matrix can be a list of rows. This function checks the shapes, then applies the product rule.
def matmul(A, B):
if len(A[0]) != len(B):
raise ValueError(f"inner sizes differ: {len(A[0])} and {len(B)}")
rows, inner, cols = len(A), len(B), len(B[0])
return [
[sum(A[i][k] * B[k][j] for k in range(inner)) for j in range(cols)]
for i in range(rows)
]
x = [[2, -1, 3]]
W = [[1, 2], [0, -1], [4, 1]]
print(matmul(x, W))
It prints [[14, 8]], the same result as the hand calculation. Calling matmul(W, W) raises a ValueError, because \(W\) has 2 columns and 3 rows.
Sources
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, Chapter 2: Linear Algebra
- 3Blue1Brown, Essence of Linear Algebra
Cards
What is the shape of a matrix?
Reveal
Its number of rows, then its number of columns. A matrix with 2 rows and 3 columns has shape \(2 \times 3\).
If \(A\) has shape \(4 \times 3\) and \(B\) has shape \(3 \times 5\), what is the shape of \(AB\)?
Reveal
\(4 \times 5\). The inner sizes, \(3\) and \(3\), match, and the outer sizes give the result shape.
When does the matrix product \(AB\) exist?
Reveal
When the number of columns of \(A\) equals the number of rows of \(B\).
What formula gives entry \((i, j)\) of the matrix product \(C = AB\)?
Reveal
Pair row \(i\) of \(A\) with column \(j\) of \(B\), multiply matching entries, and add.
Why do neural networks use matrix multiplication?
Reveal
One product combines many inputs with many weights at once. A single product computes every neuron in a layer.
Compute \(xW\) for
Reveal
The first entry is \(1(2) + 4(1) + (-2)(-1) = 8\).
The second entry is \(1(0) + 4(3) + (-2)(2) = 8\).
So \(xW = \begin{bmatrix} 8 & 8 \end{bmatrix}\).