Overfitting, Regularization, and the Bias-Variance Tradeoff

Every model in this module — regression, trees, forests, SVMs, K-Means — can fail in the same specific way, and so can every neural network in the modules that follow. This lesson names that failure mode precisely, because everything from here forward assumes you recognize it.

Overfitting and underfitting

Overfitting is what happens when a model is flexible enough to memorize its training data — including the random noise in it — rather than learning the actual underlying pattern. A decision tree with no depth limit, or a model with far more parameters than training examples, can drive its training error to zero while getting worse and worse at new data. Underfitting is the opposite failure: a model too simple to capture the real pattern, performing badly everywhere, training data included.

The bias-variance tradeoff

These two failure modes are opposite ends of one underlying tradeoff. A model's total expected error splits cleanly into three parts:

Bias is error from a model being too rigid — a straight line trying to fit a curve will always miss it in the same systematic way, no matter how much data you give it. That's underfitting. Variance is error from a model being too sensitive to exactly which training examples it happened to see — retrain it on a slightly different sample and its predictions swing wildly. That's overfitting. Irreducible noise is randomness genuinely present in the data that no model can predict away. Nearly every choice in model design — how deep a tree, how many parameters, how much regularization — is a knob trading bias against variance.

Regularization: penalizing complexity directly

Regularization fights overfitting by directly penalizing complexity inside the loss function itself, not just hoping the model behaves. The two standard forms add a penalty proportional to the size of the weights:

λ (lambda) controls how strongly complexity is penalized — larger λ means simpler, more constrained models. L2 shrinks every weight smoothly toward zero without usually reaching it exactly; L1 tends to push many weights to exactly zero, effectively deciding some features don't matter at all. If "weight decay" from the AdamW optimizer, two lessons ago, sounded familiar — it's this same L2 idea, applied as a separate step rather than folded into the loss.

How to actually spot it

In practice, overfitting is diagnosed by holding out a validation set the model never trains on, and watching both losses across training. Underfitting looks like both training and validation loss staying high together. Overfitting has a distinctive signature: training loss keeps falling while validation loss stops improving and starts climbing — the exact point where the model has stopped learning the pattern and started memorizing the noise.

That closes out classical machine learning. The next module moves to neural networks specifically — starting from the single perceptron, which turns out to be a name you've already learned under a different one.