Supervised learning

Decision Trees & Ensemble Learning

Rule-based trees, and combining many of them into forests and boosted ensembles.

6 lessons~53 min4 formulas6 questions

Not started yet

Quiz mastery0%

What you’ll take away

Trees split greedily on the feature that most reduces impurity (entropy or Gini).
Single deep trees overfit; ensembles fix that.
Bagging and random forests average decorrelated trees to cut variance.
Boosting adds trees sequentially to cut bias — the workhorse for tabular data.

Major concepts (syllabus)

Decision trees, entropy, information gain, random forests, bagging, boosting

Lessons

H (S) = - c \sum p_{c} lo g_{2} p_{c}

0 for a pure node, 1 bit for a 50/50 split.

I G (S, A) = H (S) - v \sum \frac{∣ S _{v} ∣}{∣ S ∣} H (S_{v})

Entropy reduction from splitting on A.

G (S) = 1 - c \sum p_{c}^{2}

Alternative impurity used by CART.

F_{m} (x) = F_{m - 1} (x) + ν h_{m} (x)

Each stage adds a small model with learning rate ν.

Bagging: Training models on bootstrap samples and averaging them to reduce variance.
Boosting: Building an ensemble sequentially, each model correcting the errors of the previous ones.
Bootstrap sample: A sample of n examples drawn with replacement from a dataset of size n.
Decision tree: Model that predicts through a sequence of feature-threshold questions ending in leaves.
Entropy: Measure of class mixing in a node: −Σ p log₂ p. Zero when pure.
Gini impurity: 1 − Σ p²: alternative to entropy for measuring node impurity.
Information gain: Reduction in entropy achieved by a split.
Random forest: Bagged decision trees that each consider random feature subsets at splits.