A single neuron
A single neuron
A neuron is the smallest unit of a network. This note builds one, teaches it AND and OR, and then watches it fail on XOR. That failure is the reason networks have hidden layers.
The score
A neuron multiplies each input by a weight, adds the results, and adds a bias. The result is its score:
The perceptron, an early model from the 1950s, turns the score into a class with a threshold:
Worked example
Let \(x = (1, 1)\), \(w = (1, 1)\), and \(b = -1.5\). Then \(s = 1 + 1 - 1.5 = 0.5\). The score is positive, so the output is \(1\).
Weights and bias
A weight sets how strongly one input pushes the score. With \(x_1 = 2\), a weight of \(1\) adds \(2\) to the score. A weight of \(3\) adds \(6\).
The bias shifts the point where the score crosses zero. For \(s = 2x + b\):
- With \(b = 0\), the score is zero at \(x = 0\).
- With \(b = -4\), the score is zero where \(2x - 4 = 0\), so at \(x = 2\).
Training changes both the weights and the bias. Together they are the parameters of the neuron.
Decision boundary
The decision boundary is the set of inputs where the score is exactly zero. For two inputs it is the line
For \(s = x_1 + x_2 - 1.5\), the boundary is the line \(x_1 + x_2 = 1.5\). The point \((1, 1)\) scores \(0.5\) and sits on the positive side. The point \((0, 0)\) scores \(-1.5\) and sits on the negative side.
A single neuron can only split its inputs with one straight line. That limit decides what it can learn.
Learning AND, OR, and XOR
The perceptron learning rule adjusts the parameters after each mistake. With target \(t\), output \(y\), and learning rate \(\eta\):
When the output is correct, \(t - y = 0\) and nothing changes.
The XOR targets are:
| \(x_1\) | \(x_2\) | AND | OR | XOR |
|---|---|---|---|---|
| 0 | 0 | 0 | 0 | 0 |
| 0 | 1 | 0 | 1 | 1 |
| 1 | 0 | 0 | 1 | 1 |
| 1 | 1 | 1 | 1 | 0 |
This code trains the same neuron on all three tasks and counts its mistakes after training.
def predict(x, w, b):
s = w[0] * x[0] + w[1] * x[1] + b
return 1 if s > 0 else 0
def train(targets, epochs=25, rate=0.1):
w, b = [0.0, 0.0], 0.0
inputs = [(0, 0), (0, 1), (1, 0), (1, 1)]
for _ in range(epochs):
for x, t in zip(inputs, targets):
error = t - predict(x, w, b)
w = [w[0] + rate * error * x[0], w[1] + rate * error * x[1]]
b += rate * error
return sum(predict(x, w, b) != t for x, t in zip(inputs, targets))
print("AND mistakes:", train([0, 0, 0, 1]))
print("OR mistakes:", train([0, 1, 1, 1]))
print("XOR mistakes:", train([0, 1, 1, 0]))
AND and OR finish with 0 mistakes. XOR always keeps at least one mistake, however long it trains.
Why XOR fails
Plot the four XOR inputs. The two points with target \(1\) sit on one diagonal. The two points with target \(0\) sit on the other. No single straight line puts both \(1\) points on one side and both \(0\) points on the other side. A single neuron draws exactly one line, so it cannot represent XOR.
The fix is to stack neurons into layers and add a nonlinear step between them. The next notes build both.
Sources
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, Chapter 1: Introduction
- Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, Chapter 6: Deep Feedforward Networks, section 6.1 on XOR
Cards
What score does a single neuron compute from inputs \(x_1, \dots, x_n\)?
Reveal
Each input is multiplied by its weight, and the bias \(b\) is added.
What does a weight control in a neuron?
Reveal
How strongly one input pushes the neuron's score up or down.
What does the bias do in a neuron?
Reveal
It shifts the point where the score crosses zero. Without a bias, the decision boundary must pass through the origin.
For a two-input neuron with score \(s = w_1 x_1 + w_2 x_2 + b\), where is the decision boundary?
Reveal
On the line where the score is zero:
A perceptron outputs \(1\) when \(s > 0\) and \(0\) otherwise. What does it output for \(x = (2, -1)\), \(w = (3, 4)\), and \(b = -2\)?
Reveal
The score is \(2(3) + (-1)(4) - 2 = 0\). The score is not greater than zero, so the output is \(0\).
State the perceptron learning rule for weight \(w_i\), with target \(t\), output \(y\), and learning rate \(\eta\).
Reveal
The bias updates the same way with \(x_i\) replaced by \(1\). A correct output gives \(t - y = 0\) and no change.
Why can a single neuron never learn XOR?
Reveal
A single neuron splits its inputs with one straight line. No straight line puts \((0, 1)\) and \((1, 0)\) on one side and \((0, 0)\) and \((1, 1)\) on the other.
For a neuron with score \(s = 2x_1 - x_2 - 4\), which side of the decision boundary holds the point \((3, 1)\)?
Reveal
The score is \(2(3) - 1 - 4 = 1\). It is positive, so \((3, 1)\) is on the positive side.