Backpropagation, Normalization, and Dropout

The last two lessons covered what a network is made of — stacked layers with non-linear activations between them. This lesson covers what actually makes a network of meaningful depth trainable at all, beyond the basic chain rule from the math module.

Backpropagation, at network scale

Backpropagation is the two-layer chain rule example from the calculus lesson, run across every layer of a real network. Concretely: run the network forward, get a prediction, compute the loss. Then walk backward — layer by layer, from output to input — multiplying gradients together exactly as the chain rule demands, so every weight in every layer ends up with a precise answer to "if I nudge you slightly, how does the final loss change?" It's the same operation whether the network has 2 layers or 200 — depth changes how long the chain is, not what the algorithm does at each link.

Batch norm and layer norm: keeping numbers well-behaved

As a signal passes through many layers, its scale can drift — activations can grow very large or shrink toward zero, which makes gradients unstable and training slow or unreliable. Normalization layers fix this by explicitly re-centering and re-scaling activations partway through the network.

Batch normalization computes μ and σ² across every example in the current training batch, per feature. Layer normalization instead computes them across a single example's own features, independent of everything else in the batch — which matters because it works identically whether you're processing one sequence or a thousand at once. That batch-independence is exactly why transformers, which process sequences of widely varying length, use layer norm rather than batch norm almost universally.

Dropout: an ensemble hiding inside one network

Dropout randomly zeroes out a fraction of a layer's units on every training step — commonly 10-50% of them, chosen freshly each time. Forced to work with a different random subset of itself on every step, the network can't lean too heavily on any one unit memorizing something specific to the training set. At inference time, dropout is turned off entirely and every unit contributes.

This is worth connecting directly back to the random forest lesson: training with dropout is, informally, like training an enormous number of smaller overlapping networks at once — every step uses a different random subset of units — and then averaging their combined behavior at inference. Same underlying idea as an ensemble of trees voting, applied inside a single network instead of across many separate ones.