Boosting

Boosting builds an ensemble sequentially. Each new model focuses on the examples the current ensemble gets wrong, and the final prediction is a weighted sum: $F(\mathbf{x})={\sum }_m{\alpha }_m{h}_m(\mathbf{x})$.

AdaBoost reweights the training examples after each round — misclassified ones get heavier — and gives each weak learner a vote ${\alpha }_m$ based on its accuracy. Gradient boosting fits each new tree to the *residuals* (more generally, the negative gradient of the loss) of the current ensemble.

Boosting mainly reduces bias, turning many weak learners (shallow trees, even depth-1 'stumps') into a strong one. It can overfit if run too long, so it uses a small learning rate (shrinkage) and early stopping.