Decision Trees, Random Forests, and SVMs

The last lesson covered two models that draw one straight line through the data. This lesson covers three models that don't share that limitation — each in a genuinely different way.

Decision trees: nested yes/no questions

A decision tree predicts by asking a sequence of questions: "is feature 1 above 5?" then, depending on the answer, another question, and another, until it reaches a leaf holding a final prediction. Unlike linear or logistic regression, there's no single equation — the model is the tree of questions itself, and it can carve up the data into arbitrarily irregular regions, not just one straight cut.

How a split is actually chosen

Training a tree means deciding, at every branch, which question actually separates the classes best. The standard measure for a classification tree is Gini impurity:

where pᵢ is the fraction of samples in the node belonging to class i. A pure node (every sample the same class) scores 0; a node split evenly between classes scores close to its maximum. The tree tries every possible question at every step and picks whichever one produces the biggest drop in impurity across the resulting branches.

Random forests: many weak trees, one strong vote

A single deep tree memorizes its training data easily — it can keep splitting until every leaf holds one exact sample, which generalizes badly (the next lesson names this precisely: high variance). A random forest fixes this without changing what a tree is at all — it trains many trees, each on a random subset of the training data and a random subset of features, then lets them vote. Individual trees can each be overconfident and wrong in different, uncorrelated ways; averaging over enough of them cancels most of that noise out. This general trick — train many imperfect models and combine them — ensembling, shows up throughout machine learning wherever a single model's errors are inconsistent enough to average away.

Support vector machines

A support vector machine (SVM) goes back to drawing a straight boundary like logistic regression, but with a different goal: instead of any line that separates the classes, it finds the one line with the maximum possible margin — the widest possible gap — to the nearest point on either side. Those nearest points are the "support vectors" the model is named after; every other point could move around freely without changing the boundary at all.

For data that genuinely isn't separable by any straight line, SVMs use what's called the kernel trick — mapping the data into a higher-dimensional space where a straight cut does exist, then translating that cut back into a curved boundary in the original space. The elegant part, which is well beyond what this lesson needs to prove, is that the math never actually has to construct that higher-dimensional space explicitly to do this.