Parameters: What "a 70B Model" Actually Means

The previous lesson covered attention — the mechanism a model uses to decide which words matter to which others. That mechanism runs inside a network built from an enormous number of adjustable numbers. When you see a model described as "70B" or read that GPT-3 had 175 billion parameters, this is what that's counting — and it's worth understanding what it actually means, because "bigger number" gets treated as a headline stat without much explanation of why it matters.

What a parameter actually is

A parameter is a single adjustable number inside the network — a weight, one of the values attention itself combines when it blends information between words. A parameter isn't a fact, a rule, or a memorized sentence. It's a knob. Training is the process of turning millions or billions of these knobs, a tiny amount at a time, until the network's predictions get consistently better across huge amounts of text.

"175 billion parameters" means the network has 175 billion of these knobs. No single knob means anything on its own — the knowledge and capability live in the pattern across all of them together, the same way a photograph isn't stored in any one pixel.

Why more usually helps

More parameters means more capacity to represent complicated patterns — finer distinctions, longer chains of reasoning, a wider range of topics held at once. This is why scaling up has been such a reliable lever for improving model quality: it's not the only lever, but it's consistently a real one.

The catch is that capacity alone doesn't automatically translate into capability. A network with a huge number of parameters that's only shown a small amount of training text can't use most of that capacity productively — there isn't enough signal to meaningfully tune that many knobs. Scale in parameters and scale in training data tend to move together for a reason.

See the scale

These categories are illustrative, not exact boundaries — but the two real reference points (GPT-2 and GPT-3) are from OpenAI's own published research and give a genuine sense of how fast this scaled in just one year.

Where it stops mattering to you

If you're using AI tools rather than training them, the exact parameter count of the model behind ChatGPT, Claude, or Gemini isn't something you can find out — none of the current frontier labs publish that number, and it changes across model versions anyway. What you can actually observe is the real thing that matters: does this model handle your task well, at a speed and cost you're happy with? That's worth judging directly, by trying the task, rather than by chasing a spec number nobody will confirm.

Where the parameter count does matter directly is the next two lessons: it's the reason scaling costs real compute (which is why Mixture of Experts exists — more on that next), and it's the reason AI needs the specific kind of hardware covered at the end of this module.