Parametric vs. nonparametric learning

A parametric model has a fixed number of parameters, no matter how much data you collect. Linear regression with $d$ features always has $d+1$ parameters. Parametric models make strong assumptions about the shape of $f$, are fast to train and predict, and need less data — but they underfit if the assumption is wrong.

A nonparametric model's complexity grows with the data. k-nearest neighbours stores every training example; a decision tree can keep splitting as data grows; kernel methods keep one coefficient per training point. They make fewer assumptions and can fit complicated shapes, but need more data and more memory and are slower at prediction time.

Neural networks are technically parametric (a fixed number of weights), but they are so flexible that in practice they behave much like nonparametric models.