Loss functions

Loss functions

A network needs one number that says how wrong it is. A loss function turns a prediction and its target into that number. Training then lowers it, and backpropagation starts from it.

Loss

A loss function compares the prediction \(\hat{y}\) with the target \(y\) and returns a single number. A perfect prediction gets the smallest loss. Worse predictions get larger losses.

For a whole dataset, training adds up or averages the per-example losses. The result is called the cost or the objective, written \(J\).

Squared error

Squared error is the simplest loss:

\[ L = (y - \hat{y})^2 \]

The difference \(e = y - \hat{y}\) is the prediction error. Squaring it makes every error positive and punishes large errors much more than small ones.

Worked example

With target \(y = 1\):

  • A prediction of \(\hat{y} = 0.7\) gives \(L = 0.3^2 = 0.09\).
  • A prediction of \(\hat{y} = 0.9\) gives \(L = 0.1^2 = 0.01\).

The closer prediction has the smaller loss.

The gradient of the loss

Backpropagation needs the derivative of the loss with respect to the prediction. For squared error:

\[ \frac{\partial L}{\partial \hat{y}} = -2(y - \hat{y}) = 2(\hat{y} - y) \]

For \(y = 1\) and \(\hat{y} = 0.7\), this is \(2(0.7 - 1) = -0.6\). The negative sign says that raising \(\hat{y}\) lowers the loss. The backward pass starts here and carries this slope back to every parameter.

Cross-entropy

When the network outputs a probability \(p\) that the class is \(1\), the usual loss is binary cross-entropy:

\[ L = -\big[\, y \log p + (1 - y) \log(1 - p) \,\big] \]

Only one term survives for each target. For \(y = 1\) the loss is \(-\log p\). For \(y = 0\) it is \(-\log(1 - p)\).

The Deep Learning book derives this loss from maximum likelihood. Minimizing it makes the model assign high probability to the labels in the training data. It punishes a confident wrong answer very hard, since \(-\log p\) grows without limit as \(p\) approaches \(0\).

Worked example

With \(y = 1\) and \(p = 0.8\), the loss is \(-\log 0.8 \approx 0.223\). With \(p = 0.95\) it falls to \(-\log 0.95 \approx 0.051\).

In Python

import math


def squared_error(y, y_hat):
    return (y - y_hat) ** 2


def cross_entropy(y, p):
    return -(y * math.log(p) + (1 - y) * math.log(1 - p))


print(round(squared_error(1, 0.7), 4), round(squared_error(1, 0.9), 4))
print(round(cross_entropy(1, 0.8), 3), round(cross_entropy(1, 0.95), 3))
print(round(cross_entropy(1, 0.01), 3))

It prints 0.09 0.01, then 0.223 0.051, then 4.605. The last line shows the large loss for a confident wrong answer.

Use squared error when you want every number easy to trace by hand. Use cross-entropy when the output is a probability.

Sources


Cards

What does a loss function do?

Reveal

It compares a prediction with its target and returns a single number that measures the error. Training lowers that number.

What is the squared error loss for target \(y\) and prediction \(\hat{y}\)?

Reveal
\[ L = (y - \hat{y})^2 \]

Find the squared error loss for \(y = 0\) and \(\hat{y} = 0.4\).

Reveal

\(L = (0 - 0.4)^2 = 0.16\).

What is the derivative of the squared error loss \((y - \hat{y})^2\) with respect to the prediction \(\hat{y}\)?

Reveal
\[ \frac{\partial L}{\partial \hat{y}} = 2(\hat{y} - y) \]

What is the binary cross-entropy loss for target \(y\) and predicted probability \(p\)?

Reveal
\[ L = -\big[\, y \log p + (1 - y) \log(1 - p) \,\big] \]

What is the binary cross-entropy loss when \(y = 0\) and \(p = 0.25\)?

Reveal

Only the second term survives, so \(L = -\log(0.75) \approx 0.288\).

Why does binary cross-entropy punish a confident wrong answer so hard?

Reveal

For target \(1\) the loss is \(-\log p\), which grows without limit as \(p\) approaches \(0\).

When should a network use cross-entropy loss over squared error?

Reveal

When its output is a probability. Cross-entropy comes from maximum likelihood, and it trains probability outputs well.