Information gain

Choosing the split that reduces entropy the most.

Information gain is the drop in entropy from splitting node $S$ on attribute $A$ :

I G (S, A) = H (S) - v \sum \frac{∣ S _{v} ∣}{∣ S ∣} H (S_{v})

The second term is the weighted average entropy of the children. The tree chooses the split with the largest information gain — the one that makes the children purest.

Information gain is biased towards features with many distinct values (like an ID column, which gives perfectly pure leaves). The gain ratio (used by C4.5) corrects for this by dividing by the split's own entropy.

Formula to remember

Gini impurity

G (S) = 1 - c \sum p_{c}^{2}

Alternative impurity used by CART.

Quick check · +5 XP

Boosting trains its models…

Your notes

Saved in this browser as you type.