Loss functions
Loss functions
A network needs one number that says how wrong it is. A loss function turns a prediction and its target into that number. Training then lowers it, and backpropagation starts from it.
Loss
A loss function compares the prediction \(\hat{y}\) with the target \(y\) and returns a single number. A perfect prediction gets the smallest loss. Worse predictions get larger losses.
For a whole dataset, training adds up or averages the per-example losses. The result is called the cost or the objective, written \(J\).
Squared error
Squared error is the simplest loss:
The difference \(e = y - \hat{y}\) is the prediction error. Squaring it makes every error positive and punishes large errors much more than small ones.
Worked example
With target \(y = 1\):
- A prediction of \(\hat{y} = 0.7\) gives \(L = 0.3^2 = 0.09\).
- A prediction of \(\hat{y} = 0.9\) gives \(L = 0.1^2 = 0.01\).
The closer prediction has the smaller loss.
The gradient of the loss
Backpropagation needs the derivative of the loss with respect to the prediction. For squared error:
For \(y = 1\) and \(\hat{y} = 0.7\), this is \(2(0.7 - 1) = -0.6\). The negative sign says that raising \(\hat{y}\) lowers the loss. The backward pass starts here and carries this slope back to every parameter.
Cross-entropy
When the network outputs a probability \(p\) that the class is \(1\), the usual loss is binary cross-entropy:
Only one term survives for each target. For \(y = 1\) the loss is \(-\log p\). For \(y = 0\) it is \(-\log(1 - p)\).
The Deep Learning book derives this loss from maximum likelihood. Minimizing it makes the model assign high probability to the labels in the training data. It punishes a confident wrong answer very hard, since \(-\log p\) grows without limit as \(p\) approaches \(0\).
Worked example
With \(y = 1\) and \(p = 0.8\), the loss is \(-\log 0.8 \approx 0.223\). With \(p = 0.95\) it falls to \(-\log 0.95 \approx 0.051\).
In Python
import math
def squared_error(y, y_hat):
return (y - y_hat) ** 2
def cross_entropy(y, p):
return -(y * math.log(p) + (1 - y) * math.log(1 - p))
print(round(squared_error(1, 0.7), 4), round(squared_error(1, 0.9), 4))
print(round(cross_entropy(1, 0.8), 3), round(cross_entropy(1, 0.95), 3))
print(round(cross_entropy(1, 0.01), 3))
It prints 0.09 0.01, then 0.223 0.051, then 4.605. The last line shows the large loss for a confident wrong answer.
Use squared error when you want every number easy to trace by hand. Use cross-entropy when the output is a probability.
Sources
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, Chapter 5: Machine Learning Basics, section 5.5 on maximum likelihood
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, Chapter 6: Deep Feedforward Networks, section 6.2
Cards
What does a loss function do?
Reveal
It compares a prediction with its target and returns a single number that measures the error. Training lowers that number.
What is the squared error loss for target \(y\) and prediction \(\hat{y}\)?
Reveal
Find the squared error loss for \(y = 0\) and \(\hat{y} = 0.4\).
Reveal
\(L = (0 - 0.4)^2 = 0.16\).
What is the derivative of the squared error loss \((y - \hat{y})^2\) with respect to the prediction \(\hat{y}\)?
Reveal
What is the binary cross-entropy loss for target \(y\) and predicted probability \(p\)?
Reveal
What is the binary cross-entropy loss when \(y = 0\) and \(p = 0.25\)?
Reveal
Only the second term survives, so \(L = -\log(0.75) \approx 0.288\).
Why does binary cross-entropy punish a confident wrong answer so hard?
Reveal
For target \(1\) the loss is \(-\log p\), which grows without limit as \(p\) approaches \(0\).
When should a network use cross-entropy loss over squared error?
Reveal
When its output is a probability. Cross-entropy comes from maximum likelihood, and it trains probability outputs well.