Activation Functions: Why Networks Need Non-Linearity

The last lesson claimed that stacking layers only helps if the activation function is genuinely non-linear — otherwise you gain nothing. That claim is worth proving directly, because once you've seen the proof, every activation function in this lesson stops being an arbitrary design choice.

The proof: stacked linear layers collapse into one

A linear layer computes f(x) = Ax + b. Stack two of them, with no activation function between:

No matter how many linear layers you stack — 2, 10, 1,000 — the result always simplifies to one linear function, exactly as powerful as a single layer. All that extra depth buys precisely nothing without something non-linear breaking the chain. That's not an approximation or a rule of thumb; it's an algebraic fact.

ReLU — the default for years

ReLU (Rectified Linear Unit) is almost absurdly simple:

Positive inputs pass through unchanged; negative inputs become exactly zero. That sharp bend at zero is all the non-linearity the proof above requires — and it's cheap to compute and has well-behaved gradients for positive inputs, which is a large part of why it became the default for so long.

GELU and SiLU — smoother alternatives

ReLU's sharp corner at zero turns out to have a downside: any input that lands slightly negative gets a gradient of exactly zero, contributing nothing to learning. Modern transformers mostly use smoother functions that don't have this dead zone.

Both share the same shape of idea: instead of a hard cutoff, multiply the input by a smooth value between 0 and 1 that grows with the input — a soft, differentiable version of "let this through or don't," rather than ReLU's all-or-nothing gate. GLU-based variants (gated linear units built from these) are what most current large language models actually use inside their feed-forward layers.

Which one, and where

The choice isn't arbitrary or purely a matter of taste, but it also isn't the make-or-break decision it might sound like — many of these functions perform similarly in practice, and the difference shows up mainly at the margins, in training stability and final performance on very large models. What does matter, and matters absolutely, is that some non-linear function sits between every pair of weight matrices in a network. Remove it, and the proof at the top of this lesson applies immediately — no matter how many billions of parameters you have.