Model complexity

Model complexity has many knobs depending on the model family:

• Polynomial regression: the degree.
• Decision trees: maximum depth, minimum samples per leaf.
• k-NN: smaller k = more complex.
• SVMs: the C parameter and kernel width.
• Neural networks: number of layers, units, and training epochs.
• Any model: number of features.

A rough rule: the number of effective parameters should be well below the number of training examples, unless strong regularisation or architecture constraints are in place.

Regularisation lets you keep a flexible model while controlling its effective complexity — a ridge model with many features but large λ behaves like a much simpler one.