Perceptrons to Multi-Layer Networks

Classical machine learning closed with the bias-variance tradeoff — a lens for judging any model's mistakes. This module opens with the model type behind nearly everything from here forward: the neural network. It starts from something you've technically already met.

The perceptron: one neuron

A perceptron computes a weighted sum of its inputs, adds a bias, and passes the result through a function that decides whether to "fire."

Look closely and that first line is exactly logistic regression's w·x + b from two lessons ago. A perceptron with a sigmoid activation is logistic regression — one "neuron" is not a new idea, it's a renaming of one you already understand.

The wall it hits — and it's the same wall

Because a single perceptron is just w·x + b through an activation, its decision boundary has the exact same limitation logistic regression had: one straight cut through the data. A single perceptron, on its own, cannot learn a pattern like XOR — where the correct answer flips in a way no single straight line can separate. This was, historically, a real dead end for the field until one fix was found.

Stacking layers — with one crucial catch

The fix is to stack perceptrons into layers, feeding one layer's output into the next as input — a multi-layer perceptron (MLP). Layers in between the input and final output are called hidden layers. But stacking only works if the activation function genuinely bends the boundary — otherwise, as the next lesson proves directly, it buys you nothing at all, no matter how many layers you stack.

With that one catch handled correctly, an MLP with enough hidden units can approximate essentially any function mapping inputs to outputs — curved boundaries, XOR, anything. That result has a name, the universal approximation theorem, and it's the theoretical reason neural networks are worth building at all.

One honest caveat

"Neuron" is a metaphor borrowed loosely from neuroscience, not a claim that this is how brains work. A perceptron is a weighted sum and a fixed function — biological neurons are vastly more complex, and the resemblance mostly ends at the name. It's worth not reading more into the word than the math actually says.