Logistic regression

Logistic regression models the probability of the positive class as a linear score passed through the sigmoid:

The sigmoid maps $(-\infty ,\infty )$ to $(0,1)$, with $\sigma (0)=0.5$, $\sigma (z)\to 1$ for large $z$ and $\to 0$ for very negative $z$. Its derivative is neat: ${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))$.

Inverting gives the log-odds: $\log \frac{p}{1-p}={\mathbf{w}}^{\top }\mathbf{x}+b$. So each weight is the change in log-odds per unit of its feature; $e^{w_j}$ is the factor by which the odds multiply.

It is trained by minimising binary cross-entropy (equivalently, maximising likelihood). There is no closed form, but the loss is convex, so gradient descent finds the global optimum. The gradient is simply $\frac{1}{n}{X}^{\top }(\hat{\mathbf{p}}-\mathbf{y})$.