The perceptron

The perceptron (Rosenblatt, 1958) is the first trainable neuron: a weighted sum followed by a step function that outputs 1 if the sum is positive and 0 (or −1) otherwise.

It learns with a simple rule: for each training example, predict; if wrong, nudge the weights towards the correct answer:

If the prediction is right, $y-\hat{y}=0$ and nothing changes. A missed positive adds $\eta \mathbf{x}$ to the weights; a false positive subtracts it.

The perceptron convergence theorem guarantees that if the data is linearly separable, this rule finds a separating line in a finite number of updates. If it isn't separable, the perceptron never settles.