Glossary · Machine Learning

A

Accuracy: Fraction of predictions that are correct. Misleading when classes are imbalanced.
Activation function: Non-linear function applied to a neuron's weighted sum, such as ReLU, sigmoid or tanh.
Adam: Optimiser that adapts each parameter's step size using running averages of gradients and squared gradients.
AUC: Area under the ROC curve: probability a random positive is ranked above a random negative. 0.5 is random.

B

Backpropagation: Computing gradients of the loss for every weight by applying the chain rule backwards through the network.
Bagging: Training models on bootstrap samples and averaging them to reduce variance.
Batch size: Number of examples used to compute one gradient update.
Bayes' theorem: P(H|E) = P(E|H)·P(H) / P(E): updating a prior belief with evidence.
Bias (statistical): Systematic error from wrong model assumptions; high bias causes underfitting.
Bias–variance trade-off: Simpler models have more bias and less variance; complex ones the reverse. Test error is minimised in between.
Boltzmann machine: Energy-based network of stochastic binary units whose configurations follow the Boltzmann distribution.
Boosting: Building an ensemble sequentially, each model correcting the errors of the previous ones.
Bootstrap sample: A sample of n examples drawn with replacement from a dataset of size n.

C

Classification: Predicting a discrete category.
Clustering: Grouping unlabelled examples so that similar ones end up together.
Confusion matrix: Table of predicted vs. actual classes: TP, FP, TN, FN for binary problems.
Convolution: Sliding a small learnable filter over an input and summing element-wise products to build a feature map.
Covariance: How two variables vary together; positive when they rise together.
Cross-entropy: Loss for probabilistic classification: −log of the probability assigned to the true class.
Cross-validation: Rotating which fold is held out for validation and averaging the scores.
Curse of dimensionality: Data sparsity and distance concentration that make high-dimensional learning hard.

D

DBSCAN: Density-based clustering that finds clusters of any shape and marks low-density points as noise.
Decision boundary: The surface in feature space where a classifier's prediction changes class.
Decision tree: Model that predicts through a sequence of feature-threshold questions ending in leaves.
Deep learning: Machine learning with neural networks of many layers.
Dimensionality reduction: Representing data with fewer features while keeping most of its information.
Discriminator: GAN network that tries to tell real samples from generated ones.
Divergence: Non-negative measure of difference between distributions, zero only if they are equal; need not be symmetric.
Dropout: Regularisation that randomly switches off neurons during training.

E

Early stopping: Stopping training when validation loss stops improving.
Eigenvector: A vector whose direction a matrix leaves unchanged: Av = λv.
Empirical risk: Average loss over the training sample.
Entropy: Measure of class mixing in a node: −Σ p log₂ p. Zero when pure.
Epoch: One full pass over the training data.
Euclidean distance: Straight-line (L2) distance: square root of the sum of squared coordinate differences.
Expected risk: Average loss over the true data distribution — what we really want to minimise.

F

F-divergence: Family of divergences built from a convex f with f(1) = 0 applied to the density ratio.
F1-score: Harmonic mean of precision and recall.
Feature: An input variable describing an example (a column in the dataset).
Feature selection: Keeping only a useful subset of the original features.
Federated learning: Training a shared model on decentralised data; only model updates leave the clients.

G

GAN: Generative adversarial network: a generator and a discriminator trained against each other.
Generalization: Performing well on new data from the same distribution as the training data.
Generator: GAN network that turns random noise into synthetic samples.
Gini impurity: 1 − Σ p²: alternative to entropy for measuring node impurity.
Gradient: Vector of partial derivatives; points in the direction of steepest increase.
Gradient descent: Iteratively moving parameters against the gradient to reduce the loss.

H

Hellinger distance: Bounded, symmetric distance between distributions based on square roots of densities.
Hinge loss: max(0, 1 − y·f(x)): the SVM loss, zero beyond the margin.
Hyperparameter: A setting chosen before training (learning rate, k, depth, λ), tuned on validation data.

I

Information gain: Reduction in entropy achieved by a split.
Inner product: Sum of element-wise products of two vectors; measures alignment.
IPM: Integral probability metric: largest difference in expectations over a class of test functions.

J

Jacobian: Matrix of all first partial derivatives of a vector-valued function.
JS divergence: Symmetric, bounded divergence: average KL of each distribution to their mixture.

K

k-means: Clustering that alternates assigning points to the nearest centroid and recomputing centroids.
k-NN: Predicting from the labels of the k nearest training points.
Kernel trick: Computing inner products in a high-dimensional feature space directly via a kernel function.
KL divergence: Σ P log(P/Q): information lost when Q approximates P. Asymmetric.

L

Label: The target value the model should predict.
Lasso: Least squares with an L1 penalty; drives some weights exactly to zero.
Learning rate: Step size of gradient-based updates.
Linear regression: Predicting a continuous target as a weighted sum of features plus a bias.
Logistic regression: Linear score passed through a sigmoid to model a class probability.
Loss function: Cost of a single prediction compared with the truth.

M

Manhattan distance: L1 distance: sum of absolute coordinate differences.
Margin: Distance between the decision boundary and the nearest training points.
Mini-batch: Small random subset of the data used for one gradient step.
MMD: Maximum mean discrepancy: distance between kernel mean embeddings of two distributions.
Mode collapse: GAN failure where the generator produces only a few kinds of output.

N

Naïve Bayes: Classifier applying Bayes' theorem with conditionally independent features.
Normal equation: Closed-form least squares solution w = (XᵀX)⁻¹Xᵀy.

O

Overfitting: Fitting noise in the training data: low training error, high test error.

P

Padding: Adding border pixels around an input so convolution keeps size and edge information.
Parameter: A value learned during training, such as a weight or bias.
PCA: Principal component analysis: projecting data onto directions of greatest variance.
Perceptron: Single neuron with a step activation, trained by updating weights on mistakes.
Pooling: Downsampling a feature map by taking the max or average over small regions.
Precision: TP / (TP + FP): share of predicted positives that are correct.

R

R²: Fraction of the target's variance explained by the model.
Random forest: Bagged decision trees that each consider random feature subsets at splits.
RBF kernel: Gaussian kernel exp(−γ‖x − z‖²): similarity that decays with distance.
RBM: Restricted Boltzmann machine: visible and hidden layers with no within-layer connections.
Recall: TP / (TP + FN): share of actual positives found.
Regression: Predicting a continuous number.
Regularization: Penalising model complexity to improve generalisation.
Reinforcement learning: Learning a policy from rewards by acting in an environment.
ReLU: max(0, z): the default hidden-layer activation.
Residual: Difference between the true and predicted value.
Ridge: Least squares with an L2 penalty; shrinks weights smoothly.
RMSE: Root mean squared error, in the target's units.
ROC curve: True positive rate vs. false positive rate across all thresholds.

S

SGD: Stochastic gradient descent: updating from one example or a small batch at a time.
Sigmoid: 1 / (1 + e⁻ᶻ): squashes any number into (0, 1).
Silhouette score: Clustering quality per point: (b − a) / max(a, b), from −1 to 1.
Softmax: Turns a vector of scores into probabilities that sum to 1.
Supervised learning: Learning to predict labels from labelled examples.
Support vectors: Training points on or inside the SVM margin that define the boundary.
SVD: Singular value decomposition X = USVᵀ; the stable way to compute PCA.

T

Test set: Held-out data used once to estimate final performance.
Total variation: Largest difference in probability two distributions assign to the same event.

U

Underfitting: A model too simple to capture the pattern: high training and test error.
Unsupervised learning: Finding structure in data without labels.

V

Validation set: Data used to compare models and tune hyperparameters.
Variance (model): How much a model changes with different training samples; high variance causes overfitting.

W

Wasserstein distance: Earth mover's distance: minimum cost of transporting one distribution onto another.