A single neuron

A single neuron

A neuron is the smallest unit of a network. This note builds one, teaches it AND and OR, and then watches it fail on XOR. That failure is the reason networks have hidden layers.

The score

A neuron multiplies each input by a weight, adds the results, and adds a bias. The result is its score:

\[ s = w_1 x_1 + w_2 x_2 + \cdots + w_n x_n + b \]

The perceptron, an early model from the 1950s, turns the score into a class with a threshold:

\[ y = \begin{cases} 1 & \text{if } s > 0 \\ 0 & \text{otherwise} \end{cases} \]

Worked example

Let \(x = (1, 1)\), \(w = (1, 1)\), and \(b = -1.5\). Then \(s = 1 + 1 - 1.5 = 0.5\). The score is positive, so the output is \(1\).

Weights and bias

A weight sets how strongly one input pushes the score. With \(x_1 = 2\), a weight of \(1\) adds \(2\) to the score. A weight of \(3\) adds \(6\).

The bias shifts the point where the score crosses zero. For \(s = 2x + b\):

  • With \(b = 0\), the score is zero at \(x = 0\).
  • With \(b = -4\), the score is zero where \(2x - 4 = 0\), so at \(x = 2\).

Training changes both the weights and the bias. Together they are the parameters of the neuron.

Decision boundary

The decision boundary is the set of inputs where the score is exactly zero. For two inputs it is the line

\[ w_1 x_1 + w_2 x_2 + b = 0. \]

For \(s = x_1 + x_2 - 1.5\), the boundary is the line \(x_1 + x_2 = 1.5\). The point \((1, 1)\) scores \(0.5\) and sits on the positive side. The point \((0, 0)\) scores \(-1.5\) and sits on the negative side.

A single neuron can only split its inputs with one straight line. That limit decides what it can learn.

Learning AND, OR, and XOR

The perceptron learning rule adjusts the parameters after each mistake. With target \(t\), output \(y\), and learning rate \(\eta\):

\[ w_i \leftarrow w_i + \eta (t - y) x_i, \qquad b \leftarrow b + \eta (t - y) \]

When the output is correct, \(t - y = 0\) and nothing changes.

The XOR targets are:

\(x_1\) \(x_2\) AND OR XOR
0 0 0 0 0
0 1 0 1 1
1 0 0 1 1
1 1 1 1 0

This code trains the same neuron on all three tasks and counts its mistakes after training.

def predict(x, w, b):
    s = w[0] * x[0] + w[1] * x[1] + b
    return 1 if s > 0 else 0


def train(targets, epochs=25, rate=0.1):
    w, b = [0.0, 0.0], 0.0
    inputs = [(0, 0), (0, 1), (1, 0), (1, 1)]
    for _ in range(epochs):
        for x, t in zip(inputs, targets):
            error = t - predict(x, w, b)
            w = [w[0] + rate * error * x[0], w[1] + rate * error * x[1]]
            b += rate * error
    return sum(predict(x, w, b) != t for x, t in zip(inputs, targets))


print("AND mistakes:", train([0, 0, 0, 1]))
print("OR mistakes:", train([0, 1, 1, 1]))
print("XOR mistakes:", train([0, 1, 1, 0]))

AND and OR finish with 0 mistakes. XOR always keeps at least one mistake, however long it trains.

Why XOR fails

Plot the four XOR inputs. The two points with target \(1\) sit on one diagonal. The two points with target \(0\) sit on the other. No single straight line puts both \(1\) points on one side and both \(0\) points on the other side. A single neuron draws exactly one line, so it cannot represent XOR.

The fix is to stack neurons into layers and add a nonlinear step between them. The next notes build both.

Sources


Cards

What score does a single neuron compute from inputs \(x_1, \dots, x_n\)?

Reveal
\[ s = w_1 x_1 + w_2 x_2 + \cdots + w_n x_n + b \]

Each input is multiplied by its weight, and the bias \(b\) is added.

What does a weight control in a neuron?

Reveal

How strongly one input pushes the neuron's score up or down.

What does the bias do in a neuron?

Reveal

It shifts the point where the score crosses zero. Without a bias, the decision boundary must pass through the origin.

For a two-input neuron with score \(s = w_1 x_1 + w_2 x_2 + b\), where is the decision boundary?

Reveal

On the line where the score is zero:

\[ w_1 x_1 + w_2 x_2 + b = 0 \]

A perceptron outputs \(1\) when \(s > 0\) and \(0\) otherwise. What does it output for \(x = (2, -1)\), \(w = (3, 4)\), and \(b = -2\)?

Reveal

The score is \(2(3) + (-1)(4) - 2 = 0\). The score is not greater than zero, so the output is \(0\).

State the perceptron learning rule for weight \(w_i\), with target \(t\), output \(y\), and learning rate \(\eta\).

Reveal
\[ w_i \leftarrow w_i + \eta (t - y) x_i \]

The bias updates the same way with \(x_i\) replaced by \(1\). A correct output gives \(t - y = 0\) and no change.

Why can a single neuron never learn XOR?

Reveal

A single neuron splits its inputs with one straight line. No straight line puts \((0, 1)\) and \((1, 0)\) on one side and \((0, 0)\) and \((1, 1)\) on the other.

For a neuron with score \(s = 2x_1 - x_2 - 4\), which side of the decision boundary holds the point \((3, 1)\)?

Reveal

The score is \(2(3) - 1 - 4 = 1\). It is positive, so \((3, 1)\) is on the positive side.