Model comparison

To claim model A is better than model B, the comparison must be fair:

• Use the same data splits — ideally the same cross-validation folds — for every model.
• Tune every model with the same effort; a tuned model vs. a default one isn't a fair fight.
• Always include baselines: majority class or mean prediction, and a simple linear model.
• Report variation, not just the mean — e.g. accuracy 0.86 ± 0.02 over 10 folds.

If the difference is smaller than the fold-to-fold variation, it may be noise. A paired test across folds (paired t-test, Wilcoxon signed-rank) or McNemar's test on a single test set checks significance.

Consider more than the metric: training and prediction time, memory, interpretability, robustness, and maintenance cost. A slightly less accurate model that is 100× faster and explainable may be the right choice.