Gradient Descent: How Models Actually Learn

This closes the loop on the last three lessons. You know what a model's numbers are (vectors and matrices), how to measure how wrong it is (cross-entropy loss), and how to compute exactly how each parameter affects that error (the gradient, via the chain rule). This lesson covers the last step: what to actually do with the gradient once you have it.

The entire update rule

Gradient descent is one line. For every parameter θ in the model:

Read right to left: ∇L is the gradient of the loss (from the calculus lesson) — the direction that increases error the fastest. Subtracting it moves the parameter in the opposite direction — the one that decreases error. η (eta) is the learning rate, a small number controlling how big that step is. That single subtraction, applied to every one of a model's billions of parameters, repeated millions of times over a training run, is the entire learning process.

The learning rate tradeoff

The learning rate is one of the first numbers anyone training a model has to choose, because both directions fail differently. Set it too high, and each update overshoots — the loss can bounce around or diverge entirely instead of settling down. Set it too low, and training technically works but takes an impractically long time to get anywhere, since each step barely moves the parameters.

Watch it descend, by hand

What Adam and AdamW add on top

Plain gradient descent uses the same learning rate for every parameter, every step. In practice, nearly every modern model is trained with Adam instead, which improves on it in two ways: it keeps a running average of recent gradients (momentum, so it doesn't overreact to one noisy step), and it tracks how large each parameter's gradients have typically been, so parameters with consistently small gradients get proportionally larger steps and vice versa. AdamW is the version almost every large model actually uses — the same idea, with one fix: it applies weight decay (a small pull toward zero, to discourage runaway weights) separately from the gradient-based update, which turns out to train more reliably than earlier optimizers that mixed the two together.

The full training loop, named end to end

Put the last four lessons together and here is a complete, named description of how a language model is actually trained:

1. Take a batch of text; represent it as vectors (linear algebra).
2. Run it forward through the model's matrices and attention layers to get a predicted probability distribution over the next word (linear algebra + softmax, from probability).
3. Compare that distribution to the actual next word using cross-entropy loss (probability).
4. Compute the gradient of that loss with respect to every parameter, via the chain rule, working backward through the network (calculus).
5. Update every parameter with AdamW using that gradient (this lesson).
6. Repeat, billions of times, over enormous amounts of text.

That loop, run at a scale of trillions of words and hundreds of billions of parameters, is the entire recipe behind every model this site has discussed. Before returning to how transformers specifically use it, the next module puts it to work on something smaller first — the classical models that came before deep learning, and in many cases still quietly run more of the software you use than any neural network does.