Bagging

Bagging (bootstrap aggregating) reduces variance by averaging many models trained on different versions of the data:

• Draw $B$ bootstrap samples: each is $n$ examples drawn *with replacement* from the training set (about 63% unique examples each).
• Train one model — usually a deep, low-bias tree — on each sample.
• Predict by majority vote (classification) or averaging (regression).

Averaging $B$ models with variance ${\sigma }^2$ and pairwise correlation $\rho$ gives variance $\rho {\sigma }^2+\frac{1-\rho }{B}{\sigma }^2$. Bagging lowers the second term; it works best for unstable, high-variance models like deep trees.