Feature selection

Feature selection keeps a subset of the original features. Fewer features means faster models, less overfitting, easier interpretation and cheaper data collection.

Filter methods score each feature independently of any model — correlation with the target, mutual information, chi-squared tests — and keep the top ones. Fast, but they ignore interactions.

Wrapper methods search over subsets by actually training models: forward selection (add the best feature one at a time), backward elimination (remove the worst), recursive feature elimination. Accurate but expensive.

Embedded methods select features as part of training: lasso zeroes out weights, and tree-based models report feature importances.