Formula notebook

All 84 formulas in the course, grouped by topic. Copy any of them as LaTeX.

84 formulas

01 Introduction to Machine Learning

SIR model

Slides

\frac{d S}{d t} \frac{d I}{d t} \frac{d R}{d t} = - β \frac{S I}{N} = β \frac{S I}{N} - γ I = γ I

Flow between Susceptible, Infected and Recovered compartments.

$β$: transmission rate
$γ$: recovery rate
$N$: population size

Dataset

D = {(x_{i}, y_{i})}_{i = 1}^{n}

A supervised dataset: n examples, each a feature vector with its label.

02 Mathematical Foundations

Inner product

a^{⊤} b = j \sum a_{j} b_{j} = ∥ a ∥ ∥ b ∥ cos θ

Element-wise products summed; relates to the angle between vectors.

Manhattan distance

Slides

d (A, B) = i = 1 \sum n ∣ a_{i} - b_{i} ∣

Sum of absolute differences (L1).

Euclidean distance

Slides

d (A, B) = i = 1 \sum n (a_{i} - b_{i})^{2}

Straight-line distance (L2).

Lp norm

∥ x ∥_{p} = (j \sum ∣ x_{j} ∣^{p})^{1/ p}

p = 1 Manhattan, p = 2 Euclidean, p → ∞ max-norm.

Jacobian matrix

Slides

J_{ij} = \frac{\partial f _{i}}{\partial x _{j}}

Derivative of a vector-valued function; how it transforms local space.

Expectation

E [X] = x \sum x p (x)

Probability-weighted average.

Variance

Var (X) = E [(X - μ)^{2}]

Expected squared distance from the mean.

Covariance

Cov (X, Y) = E [(X - μ_{X}) (Y - μ_{Y})]

Positive when X and Y tend to rise together.

Bayes' theorem

P (A ∣ B) = \frac{P ( B ∣ A ) P ( A )}{P ( B )}

Posterior = likelihood × prior / evidence.

03 Learning & Generalization

Bias–variance decomposition

E [(y - f^(x))^{2}] = Bias [f^(x)]^{2} + Var [f^(x)] + σ^{2}

Expected error = squared bias + variance + irreducible noise.

Ridge objective

J (w) = ∥ y - X w ∥_{2}^{2} + λ ∥ w ∥_{2}^{2}

L2 penalty shrinks all weights.

Lasso objective

J (w) = ∥ y - X w ∥_{2}^{2} + λ ∥ w ∥_{1}

L1 penalty sets some weights exactly to zero.

04 Learning as an Optimization Problem

Expected risk

R (h) = E_{(x, y) \sim P} [L (h (x), y)]

Average loss over the true (unknown) data distribution.

Empirical risk

R^(h) = \frac{1}{n} i = 1 \sum n L (h (x_{i}), y_{i})

Average loss on the training sample.

ERM

h^= ar g h \in H min R^(h)

Pick the hypothesis with the lowest training loss.

Gradient descent

θ \leftarrow θ - η \nabla_{θ} J (θ)

Step against the gradient with learning rate η.

Mini-batch gradient

\nabla_{θ} J \approx \frac{1}{∣ B ∣} i \in B \sum \nabla_{θ} L (h_{θ} (x_{i}), y_{i})

Gradient estimated on a batch B; |B| = 1 is SGD.

05 Similarity / Divergence Measures

F-divergence

Slides

D_{f} (P ∥ Q) = E_{Q} [f (\frac{d P}{d Q})]

Convex f with f(1) = 0; a ratio-based divergence.

KL divergence (discrete)

Slides

D_{K L} (P ∥ Q) = i \sum P (i) lo g \frac{P ( i )}{Q ( i )}

Asymmetric; in [0, ∞); infinite for non-overlapping supports.

KL divergence (continuous)

Slides

D_{K L} (P ∥ Q) = \int P (x) lo g \frac{P ( x )}{Q ( x )} d x

Continuous form of KL.

JS divergence

Slides

D_{J S} (P ∥ Q) M = \frac{1}{2} D_{K L} (P ∥ M) + \frac{1}{2} D_{K L} (Q ∥ M) = \frac{1}{2} (P + Q)

Symmetric; equals log 2 for non-overlapping distributions.

Pearson χ²

χ_{P}^{2} (P ∥ Q) = i \sum \frac{( P ( i ) - Q ( i ) ) ^{2}}{Q ( i )}

Squared differences weighted by Q.

Neyman χ²

χ_{N}^{2} (P ∥ Q) = i \sum \frac{( P ( i ) - Q ( i ) ) ^{2}}{P ( i )}

Squared differences weighted by P.

Total variation

T V (P, Q) = \frac{1}{2} i \sum ∣ P (i) - Q (i) ∣

Largest difference in probability assigned to any event.

Hellinger distance

H^{2} (P, Q) = \frac{1}{2} i \sum (P (i) - Q (i))^{2}

Bounded in [0, 1] and symmetric.

Integral Probability Metric

Slides

E (P, Q) = f \in F sup E_{P} f (x) - E_{Q} f (x)

The function class 𝓕 decides which metric you get.

Wasserstein-1

W_{1} (P, Q) = ∥ f ∥_{L} \leq 1 sup E_{P} f (x) - E_{Q} f (x)

Kantorovich–Rubinstein dual form; used by WGAN.

MMD

MMD^{2} = E [k (x, x^{'})] - 2 E [k (x, y)] + E [k (y, y^{'})]

x, x' ~ P and y, y' ~ Q, with kernel k.

Kolmogorov distance

K (P, Q) = x sup ∣ F_{P} (x) - F_{Q} (x) ∣

Largest gap between CDFs.

Cramér distance

ℓ_{2}^{2} (P, Q) = \int (F_{P} (x) - F_{Q} (x))^{2} d x

Integrated squared gap between CDFs.

06 Linear Regression

Linear model

y^= w^{⊤} x + b

Weighted sum of features plus intercept.

Least squares loss

J (w) = i = 1 \sum n (y_{i} - w^{⊤} x_{i})^{2}

Sum of squared residuals.

Normal equation

w^{*} = (X^{⊤} X)^{- 1} X^{⊤} y

Closed-form least-squares solution.

MSE gradient

\nabla_{w} MSE = \frac{2}{n} X^{⊤} (X w - y)

Used for gradient-descent training.

RMSE

RMSE = \frac{1}{n} i \sum (y_{i} - y^_{i})^{2}

Error in the target's units; punishes big errors.

MAE

MAE = \frac{1}{n} i \sum ∣ y_{i} - y^_{i} ∣

Average absolute error; robust to outliers.

R²

R^{2} = 1 - \frac{\sum _{i} ( y _{i} - y ^ _{i} ) ^{2}}{\sum _{i} ( y _{i} - y ˉ ) ^{2}}

Fraction of variance explained; 1 is perfect.

07 Classification

Sigmoid

σ (z) = \frac{1}{1 + e ^{- z}}

Maps any real number into (0, 1).

Logistic model

P (y = 1 ∣ x) = σ (w^{⊤} x + b)

Probability of the positive class.

Binary cross-entropy

L = - \frac{1}{n} i \sum [y_{i} lo g p^_{i} + (1 - y_{i}) lo g (1 - p^_{i})]

Negative log-likelihood of Bernoulli labels.

Softmax

softmax (z)_{k} = \frac{e ^{z_{k}}}{\sum _{j} e ^{z_{j}}}

K scores to K probabilities that sum to 1.

08 Model Evaluation

Accuracy

\frac{T P + T N}{T P + T N + F P + F N}

Share of correct predictions.

Precision

\frac{T P}{T P + F P}

Of predicted positives, how many are right.

Recall

\frac{T P}{T P + F N}

Of actual positives, how many were found.

F1-score

F_{1} = \frac{2 P R}{P + R}

Harmonic mean of precision P and recall R.

True / false positive rate

T P R = \frac{T P}{T P + F N}, F P R = \frac{F P}{F P + T N}

The two axes of the ROC curve.

09 Overfitting and Regularization

Regularized objective

J (w) = R^(w) + λ Ω (w)

Data fit plus a complexity penalty Ω.

Ridge closed form

w^{*} = (X^{⊤} X + λ I)^{- 1} X^{⊤} y

λI keeps the matrix invertible.

10 Decision Trees & Ensemble Learning

Entropy

H (S) = - c \sum p_{c} lo g_{2} p_{c}

0 for a pure node, 1 bit for a 50/50 split.

Information gain

I G (S, A) = H (S) - v \sum \frac{∣ S _{v} ∣}{∣ S ∣} H (S_{v})

Entropy reduction from splitting on A.

Gini impurity

G (S) = 1 - c \sum p_{c}^{2}

Alternative impurity used by CART.

Boosting

F_{m} (x) = F_{m - 1} (x) + ν h_{m} (x)

Each stage adds a small model with learning rate ν.

11 Support Vector Machines

Margin

margin = \frac{2}{∥ w ∥}

Maximise the margin by minimising ‖w‖.

Soft-margin SVM

w, b min \frac{1}{2} ∥ w ∥^{2} + C i \sum max (0, 1 - y_{i} (w^{⊤} x_{i} + b))

Margin plus hinge penalties.

RBF kernel

K (x, z) = exp (- γ ∥ x - z ∥^{2})

Similarity decaying with distance.

12 Nearest-Neighbor & Probabilistic Methods

k-NN prediction

y^(x) = mode {y_{i} : i \in N_{k} (x)}

Majority vote among the k nearest neighbours.

Naïve Bayes

y^= ar g c max P (c) j \prod P (x_{j} ∣ c)

Prior × product of per-feature likelihoods.

13 Unsupervised Learning

k-means objective

J = k = 1 \sum K x_{i} \in C_{k} \sum ∥ x_{i} - μ_{k} ∥^{2}

Within-cluster sum of squares.

Silhouette

s (i) = \frac{b ( i ) - a ( i )}{max { a ( i ) , b ( i )}}

Near 1 means well clustered.

14 Dimensionality Reduction

Covariance matrix

Σ = \frac{1}{n - 1} (X - X ˉ)^{⊤} (X - X ˉ)

d × d feature covariances.

Principal directions

Σ v_{k} = λ_{k} v_{k}

λ_k is the variance along v_k.

Variance maximization

v_{1} = ar g ∥ v ∥ = 1 max v^{⊤} Σ v

First component keeps the most variance.

Reconstruction error

V min i \sum ∥ x_{i} - V V^{⊤} x_{i} ∥^{2}

Equivalent view: best low-rank reconstruction.

SVD

X = U S V^{⊤}

Columns of V are the principal directions of centred X.

Explained variance ratio

EVR_{k} = \frac{λ _{k}}{\sum _{j} λ _{j}}

Share of variance captured by component k.

15 Introductory Neural Networks

Neuron

a = ϕ (w^{⊤} x + b)

Weighted sum through an activation φ.

Perceptron update

w \leftarrow w + η (y - y^) x

Adjust weights only on mistakes.

Sigmoid / tanh / ReLU

σ (z) = \frac{1}{1 + e ^{- z}}, tanh (z), ReLU (z) = max (0, z)

Ranges (0,1), (−1,1) and [0,∞).

Layer forward pass

h^{(l)} = ϕ (W^{(l)} h^{(l - 1)} + b^{(l)})

Each layer transforms the previous one.

Backpropagation

δ^{(l)} = (W^{(l + 1) ⊤} δ^{(l + 1)}) ⊙ ϕ^{'} (z^{(l)}), \frac{\partial L}{\partial W ^{(l)}} = δ^{(l)} h^{(l - 1) ⊤}

Errors flow backwards via the chain rule.

2D convolution

Slides

(I * K) (i, j) = m \sum n \sum I (i + m, j + n) K (m, n)

Slide kernel K over image I and sum products.

Convolution output size

Slides

o = ⌊ \frac{n + 2 p - k}{s} ⌋ + 1

Input n, padding p, kernel k, stride s.

2D Gaussian kernel

Slides

G (x, y) = \frac{1}{2 π σ ^{2}} e^{- \frac{x ^{2} + y ^{2}}{2 σ ^{2}}}

Smoothing filter; largest weight at the centre.

L2 pooling

Slides

y = i \in R \sum x_{i}^{2}

L2 norm of each pooling region R.

16 Advanced Topics

GAN game (general)

Slides

G min D max E (X, G_{θ} (K))

D maximises the discrepancy; G minimises it.

Generative model goal

Slides

θ^{*} = ar g θ min E (X, G_{θ} (K))

Make the generated distribution match the data.

Vanilla GAN objective

G min D max E_{x} [lo g D (x)] + E_{z} [lo g (1 - D (G (z)))]

The original JS-based formulation.

Two-sample test

Slides

H_{0} : X = G_{θ} (K), H_{1} : X \neq = G_{θ} (K), reject if E^> r_{α}

Distribution-level view of the discriminator.

Boltzmann distribution

P (v, h) = \frac{1}{Z} e^{- E (v, h)}

Lower energy means higher probability; Z normalises.

RBM energy

E (v, h) = - a^{⊤} v - b^{⊤} h - v^{⊤} W h

Visible–hidden interactions only.

17 Distributed ML

Federated averaging

w_{t + 1} = k = 1 \sum K \frac{n _{k}}{n} w_{t + 1}^{(k)}

Global model = data-weighted average of client models.

Data-parallel gradient

\nabla J = \frac{1}{K} k = 1 \sum K \nabla J_{k}

Each worker computes a gradient on its shard; results are averaged.