Linear & Logistic Regression: The Simplest Models That Actually Work

The last module covered the four pieces of math underneath every model on this site. This module is where they first get used to build something you'd actually call a model — starting with the two oldest, simplest ones, which are still exactly what get reached for whenever a full neural network would be overkill.

Linear regression: fitting a line

Linear regression predicts a number by fitting a straight line (or, with more than one input, a flat plane) through your data. With one input feature, it's the equation you learned in school:

w (the slope) and b (the intercept) are exactly the kind of parameters covered in the Core AI module — here there are just two of them instead of billions, which makes them a good place to actually see what "fitting" means.

The loss function

To fit a line, you need a number that measures how wrong it currently is. Linear regression almost always uses mean squared error (MSE):

Squaring the error does two things: it makes every error positive (so overshooting and undershooting don't cancel out), and it penalizes large errors disproportionately more than small ones — a prediction that's off by 10 contributes 100 times more loss than one off by 1, not 10 times more.

Fitting it with gradient descent

This is the same gradient descent from the previous module, applied to exactly two parameters instead of billions. Compute the gradient of the loss with respect to w and b, then step against it.

From predicting a number to predicting a probability

Linear regression predicts an unbounded number — useful for price, temperature, age. But "spam or not spam" needs a probability between 0 and 1. Logistic regression is the fix: take the exact same w·x + b, then squash it through the sigmoid function.

Whatever number w·x + b produces, sigmoid maps it into a valid probability: a very negative z gets squashed toward 0, a very positive one toward 1, and z = 0 lands at exactly 0.5. The loss changes too — instead of MSE, logistic regression is trained with the same cross-entropy loss from the probability lesson two lessons ago, comparing the predicted probability against the true 0-or-1 label.

The decision boundary

Because σ(z) = 0.5 exactly when z = 0, the model's prediction flips from "class 0" to "class 1" exactly where w·x + b = 0. With two input features, that equation describes a straight line splitting the plane in two — the decision boundary. Everything logistic regression can ever learn is some version of that one straight (or flat) cut through the data — which is exactly why it struggles on data that isn't linearly separable, and why the next lesson introduces models that can bend that boundary into curves.