Mixture of Experts: How Models Get Bigger Without Getting Slower

The last lesson established that more parameters generally means more capable — and that scaling parameters, training data, and compute together is how models got so much better, so fast. It also left a problem unsolved: a bigger network is a slower, more expensive network, because in the design covered so far, every single parameter runs for every single word. Mixture of Experts is the architectural trick that breaks that link.

The scaling tradeoff

In a standard ("dense") network, generating one word means running the calculation through every parameter in the model. Double the parameter count, and you roughly double the compute needed for every single word the model produces — capacity and cost climb together, with no way to get one without the other.

That's a real ceiling. At some point, a dense model with more total knowledge becomes too slow and too expensive to actually serve to anyone, no matter how capable it is on paper.

The idea: don't use it all, every time

A Mixture of Experts (MoE) model is built differently. Instead of one giant network, it's split into many smaller sub-networks called experts. A small "router" looks at each word as it comes in and decides which handful of experts — not all of them — should process it. The rest of the experts stay completely idle for that word.

The result: the model's total parameter count can be enormous, because you're adding up many experts. But the compute cost for any single word only depends on the small number of experts actually activated for it. You get access to a much larger pool of learned capacity without paying the full compute bill for all of it on every step.

See the routing

Pick a few example tokens below and watch which experts the router activates for each one.

Why this matters, even if you never train a model

If you use AI products, this is part of why some very capable models feel fast and reasonably priced despite clearly having a huge amount of underlying capability — sparse activation is one of the main reasons that combination is possible at all. It's a good example of how a lot of recent AI progress isn't just "bigger," it's smarter about which parts of "bigger" actually need to run for any given piece of work — which leads directly into the final lesson in this module: the hardware all of this actually runs on.