RNNs, LSTMs, and GRUs: How Models Used to Handle Sequences

The Deep Learning module covered networks that take one fixed-size input and produce one output. Text doesn't fit that shape — a sentence can be 3 words or 300. This lesson covers the architecture built specifically to handle that, and the exact way it broke down on long sequences.

The recurrent neural network

A recurrent neural network (RNN) processes a sequence one word at a time, carrying forward a hidden state that's meant to summarize everything read so far:

At every time step t, the new hidden state h_t is built from the previous hidden state and the current word's embedding — the same weights W_h and W_x are reused at every step. This is exactly the sequential process the attention lesson described as the "before" picture: word by word, in order, each step depending on the last.

Why its memory kept failing

Training an RNN means backpropagating through every one of those time steps — an idea with its own name, backpropagation through time, but mechanically the same chain rule from the calculus lesson. The problem is what that chain rule actually computes: the gradient gets multiplied by W_h once per time step. Multiply a number smaller than 1 by itself fifty times and it collapses toward zero; multiply a number larger than 1 fifty times and it explodes.

In practice this meant a vanilla RNN could learn short-range patterns fine, but by 50 or 100 words back, the gradient signal connecting an early word to the current prediction had usually either vanished to nothing or blown up — the network genuinely could not learn dependencies across long stretches of text, no matter how much data it saw.

LSTM: a memory that can skip multiplication

The LSTM (Long Short-Term Memory) fixes this with a specific structural change: alongside the hidden state, it keeps a separate cell state that gets updated by addition, not multiplication by a repeated weight matrix.

Both gates are learned, small neural networks in their own right (a sigmoid over a linear combination of the input and previous hidden state), so the model can choose per-step how much to remember versus forget. Critically, because the cell state update is addition rather than repeated matrix multiplication, a gradient can flow backward through many time steps largely unchanged, sidestepping the vanishing problem above.

GRU: the simplified version

The GRU (Gated Recurrent Unit) keeps the same core idea — learned gates controlling what gets kept versus overwritten — with fewer moving parts: two gates instead of three, and no separate cell state, just one hidden state that both roles are folded into. It's cheaper to compute and, on many tasks, performs comparably to a full LSTM — a genuinely useful simplification, not a downgrade.

LSTMs and GRUs pushed how far back a model could reliably reach considerably further than a vanilla RNN — but not infinitely far, and never in parallel, since every step still depends on the one before it. The next lesson covers the specific failure this caused in translation, and the fix that was invented for it two years before the transformer existed.