Soft margins and hinge loss

Real data is rarely perfectly separable. The soft-margin SVM adds slack variables ${\xi }_i\ge 0$ that let points violate the margin, at a cost:

Equivalently, it minimises the hinge loss $\max (0,1-y_{i}f({\mathbf{x}}_i))$ plus an L2 penalty. Hinge loss is zero for points correctly classified beyond the margin and grows linearly otherwise.

$C$ controls the trade-off. Large C: violations are expensive → narrow margin, fits training data tightly, risk of overfitting. Small C: violations are cheap → wide margin, more training errors, smoother boundary. Choose C by cross-validation.