Embeddings: Turning Anything Into a Vector

Every lesson in Core AI, still ahead, casually referred to a word's "embedding" as if it were obvious where that vector comes from. It's the last real gap before that module — closing it makes attention's Q, K, and V vectors feel inevitable rather than assumed.

The problem: words are not numbers

Everything in this site's math — matrix multiplication, dot products, gradients — operates on vectors of numbers. A word like "cat" is not a number. Before any of that math can run, every word first has to become a vector, and the obvious naive approach — assign each word a unique ID like cat = 4,217 — fails badly: it implies "cat" and "4,218" ("caterpillar," say) are numerically close for no meaningful reason, and gives the model nothing to generalize with.

The embedding table

The actual solution is an embedding table: a matrix with one row per word in the model's vocabulary, where each row is a vector of learned numbers — commonly hundreds or thousands of them.

Looking up a word's embedding is just reading one row out of this matrix. And here's the part that closes the loop with the parameters lesson: this entire table is a set of learned parameters, no different from any weight matrix elsewhere in the model. It starts training as random numbers and is updated by the exact same gradient descent loop as everything else — every time the model sees a word used in context, backpropagation nudges that word's row a little.

What ends up encoded, and how we know

After training on enough text, these vectors end up capturing real structure in language — not because anyone designed them to, but because words with similar meanings need similar vectors to make the model's next-word predictions work well. This is empirically well-documented and not a marketing claim: relationships between word vectors trained this way have been shown to capture analogies — famously, something close to vector("king") − vector("man") + vector("woman") landing near vector("queen") in classic embedding studies. Nobody hand-coded that relationship; it fell out of predicting text well enough, at scale.

Beyond words

Nothing about this idea is specific to text. An image model embeds patches of pixels; a recommendation system embeds users and products into the same vector space so that "similar" can be measured as a distance; the agentic AI module, later on this site, touches on embedding entire documents for search. Whenever you hear "we embedded X," it means exactly this: X was turned into a vector, placed in a space where distance and direction mean something, using a table (or a small network) trained the same way as everything else on this site — one gradient step at a time.

With this, Core AI's opening move — every word turns into a vector before attention ever runs — is no longer a black box. One module stands between here and Core AI: the sequence models attention actually replaced, and the precise problem that made replacing them necessary.