GPUs: Why AI Needs Different Hardware Than Your Laptop

This module has covered how a model decides what matters (attention), what a parameter actually is, and how Mixture of Experts keeps a huge model affordable to run. All of it — attention, dense layers, routing included — reduces at the level of raw arithmetic to one operation performed an enormous number of times: multiply two numbers, add the result to a running total. That single fact explains why AI needed a different kind of chip.

Why not just use a CPU?

The processor in your laptop is a CPU, and it's built to be extremely good at a particular kind of work: complex, branching instructions that mostly have to happen in a specific order — opening this file depends on the result of checking that permission, which depends on an earlier lookup. A CPU has a handful of very powerful cores, each capable of juggling that kind of sequential, decision-heavy logic well.

Training or running a neural network is a completely different shape of problem. It's not a long chain of dependent decisions — it's the same simple operation, repeated millions or billions of times, almost all of which can happen at once because most of them don't depend on each other. Using a CPU for that is like assigning your five best surgeons to paint a football stadium one seat at a time — technically capable, wildly mismatched to the job.

The parallel workload

A GPU takes the opposite bet: instead of a few powerful cores, it packs in thousands of small, simple ones, all capable of doing the same basic operation at the same time. GPUs were originally built for rendering graphics — coloring in millions of pixels, mostly independently of each other — which turns out to be almost exactly the same shape of problem as running a neural network. That coincidence is a large part of why GPUs became the default hardware for AI rather than a chip designed from scratch for the purpose.

See it side by side

Below, both a simulated CPU and GPU process the same 24 tiles — same total work, different number of things happening at once.

What this means for you, practically

If you're building AI-powered products through an API — which is how most people build with AI — you almost certainly never touch a GPU directly. The provider runs the model on their own hardware; you send a request and get a response. So why does any of this matter to you?

Because it explains costs and limits that would otherwise look arbitrary. A bigger model, or a longer piece of text, means more of those multiply-add operations, which means more GPU time, which is why larger models generally cost more per request and respond more slowly. It's not a pricing decision made in a vacuum — it's downstream of real compute being consumed on real hardware, one parallel batch at a time.

That's the last piece of how these models work under the hood. The next module puts that aside and turns practical: how to actually get good results out of the AI you already have access to, starting today.