K-Means and PCA: Finding Structure Without Labels

Every model so far in this module needed labeled examples — a known correct answer to train against. This lesson covers the other half of classical machine learning: finding structure in data that has no labels at all.

Learning without labels

Unsupervised learning doesn't predict anything against a known answer. Instead, it asks a different kind of question: "what groups naturally exist in this data?" or "how can I describe this data with fewer numbers, losing as little as possible?" The two classic answers are K-Means and PCA.

K-Means: finding groups by iterating toward them

K-Means finds k groups (you choose k) with a genuinely simple loop:

1. Place k centroids somewhere (often at random points from the data).
2. Assign every point to whichever centroid is nearest.
3. Move each centroid to the average position of the points now assigned to it.
4. Repeat steps 2–3 until nothing changes.

PCA: compressing data without losing the point

PCA (Principal Component Analysis) solves a different problem: you have data with hundreds of numbers per example, and you want a much smaller number of numbers that still capture most of what matters.

The linear algebra lesson mentioned eigenvalues only in passing — this is where they actually earn their keep. PCA computes the covariance matrix of the data (how much every pair of features varies together), then finds its eigenvectors. Each eigenvector points along one direction of spread in the data, and its paired eigenvalue tells you how much of the data's total variance lies along that direction.

Sort the eigenvectors by their eigenvalues, keep the top few, and project every data point onto just those directions. You've gone from hundreds of numbers to a handful, keeping whichever combinations of the original features actually varied the most — and discarding the directions that barely varied at all, on the reasoning that a dimension the data barely moves along wasn't carrying much information to begin with.