Encoder-Decoder Models and the Bottleneck That Started It All

This is the last lesson before Core AI, and it exists to correct a small but genuine oversimplification in how attention usually gets introduced. Attention wasn't invented for transformers. It was invented two years earlier, inside an LSTM, to fix one specific, well-defined problem in machine translation.

The encoder-decoder architecture

Translating a sentence — English in, French out — doesn't fit a single RNN cleanly, because the input and output are two different sequences of two different lengths. The standard solution was an encoder-decoder (or "sequence-to-sequence") model: one RNN — the encoder — reads the entire English sentence, one word at a time, using exactly the recurrence from the last lesson. Its final hidden state, after reading the whole sentence, becomes a single summary vector. A second RNN — the decoder — takes that one vector and generates the French sentence from it, one word at a time.

The bottleneck

Look closely at what that architecture demands: an entire sentence — however long — has to be squeezed into one fixed-size vector before the decoder ever sees any of it. For a five-word sentence, that's workable. For a five-paragraph one, it's not — details from early in the input reliably got lost by the time that single vector was formed, well before the vanishing-gradient problem from the last lesson even entered the picture. Translation quality measurably fell off as sentences got longer, and this was the specific, named reason: everything the decoder could ever use had to survive being compressed into one vector first.

The fix — attention's actual origin

The 2014 fix, from a paper by Bahdanau, Cho, and Bengio, was to stop forcing everything through one vector. Instead of handing the decoder only the encoder's final hidden state, give it access to every hidden state the encoder ever produced — one per input word — and let the decoder learn, at each output step, a weighted combination of which of those input words matter most right now.

That's attention, in its original form: a learned relevance score between the thing being generated and every piece of the input, turned into weights, used to blend the input representations. The 2017 "Attention Is All You Need" paper's actual innovation, covered next, was more radical than inventing attention — it was noticing that if attention can look at every position anyway, the RNN carrying information step-by-step is no longer doing any useful work, and can be removed entirely. What's left, running on attention alone, is the transformer.