Vectors, Matrices, and the Dot Product

The last module compared ChatGPT, Claude, and Gemini as products — what to use them for, not how they work. This module is the bridge to actually understanding the second question. It's four short lessons — linear algebra, calculus, probability, optimization — covering only the math that shows up in how these models work, nothing more, so the next module (Core AI) reads as mechanism instead of magic.

Why this module exists

You don't need a math degree to use AI, and the earlier lessons were written so you wouldn't need one. But if you want to actually follow what's happening inside the mechanism — not just the intuition, the real thing — there are exactly four ideas underneath almost everything: vectors, matrices, derivatives, and probability. Once you have these, the rest of this site (and most AI research) stops feeling like a wall of unfamiliar symbols.

A vector is just a list of numbers

A vector is an ordered list of numbers, written in brackets: [3, 1, 4]. That's it — no more mystery than that. What makes vectors useful for AI is that a list of numbers can represent almost anything: a word's meaning, a pixel's color, a user's preferences.

When the attention lesson talked about a word being turned into a "query," "key," and "value" vector, this is precisely what that meant: three different lists of numbers, each capturing a different aspect of that word, produced by multiplying the word's embedding through a learned matrix.

Real models use vectors with hundreds or thousands of numbers, not three — but the operations you're about to learn work exactly the same way regardless of length.

A matrix is a grid of numbers

A matrix is a grid of numbers — rows and columns instead of a single line. Where a vector represents a single piece of data, a matrix is almost always something that transforms vectors: it's the thing that turns a word's raw embedding into its query vector, or turns 12,288 numbers into the next layer's 12,288 numbers.

Every parameter you read about in the earlier module — 175 billion of them for GPT-3 — is a single entry inside one of a model's weight matrices. Training is the process of nudging every one of those grid entries a tiny amount, repeatedly, until the matrices transform inputs into better outputs.

Matrix multiplication — the actual mechanic

This is the one operation worth being able to do by hand, because a neural network's forward pass is this operation, repeated thousands of times. To multiply a matrix by a vector: for each row of the matrix, multiply it element-by-element against the vector, then add up the results. That sum becomes one number in the output.

The dot product — a special case you already know

The dot product of two vectors is exactly one row of the matrix multiplication above: multiply matching positions together, add the results. It produces a single number, and that number is larger when the two vectors point in a similar direction.

This is precisely the operation the attention lesson's Q Kᵀ step performs — a query vector dotted against a key vector, over and over, to produce a relevance score for every pair of words. There isn't a separate "attention math" — it's this one operation, applied at scale.

It's this, over and over

Here's the payoff for sitting through this lesson: a transformer, end to end, is vectors (embeddings) being repeatedly multiplied by matrices (learned weights), with a handful of simple non-linear steps mixed in between layers. The GPU lesson earlier put it in hardware terms — "multiply two numbers, add the result to a running total, repeated an enormous number of times." This is the exact same fact, now in the notation you'll actually see in a paper or a codebase.