Parameter estimation asks how to choose model parameters from observed data. Suppose data $D$ was generated by a probabilistic model $p(D\mid \theta)$ with unknown parameters $\theta$. Maximum likelihood estimation chooses the value of $\theta$ that makes the observed data most probable.

The likelihood is

$$ L(\theta;D)=p(D\mid \theta). $$

If the data points are independent and identically distributed, then

$$ L(\theta;D)=\prod_{i=1}^{n} p(x_i\mid \theta). $$

Products of many probabilities can become very small and hard to differentiate. Therefore we usually maximise the log-likelihood:

$$ \ell(\theta)=\log L(\theta;D)=\sum_{i=1}^{n}\log p(x_i\mid \theta). $$

Because the logarithm is monotonic, the value of $\theta$ that maximises $L$ also maximises $\ell$.

Bernoulli example

For a biased coin, let $x_i \in {0,1}$ where $1$ means heads and $0$ means tails. If the probability of heads is $\phi$, then

$$ p(x_i\mid \phi)=\phi^{x_i}(1-\phi)^{1-x_i}. $$

The log-likelihood for $n$ flips is

$$ \ell(\phi)=\sum_{i=1}^{n}\left[x_i\log\phi+(1-x_i)\log(1-\phi)\right]. $$

Maximising this gives

$$ \hat\phi=\frac{1}{n}\sum_{i=1}^{n}x_i. $$

So the MLE for the probability of heads is the observed fraction of heads.

Gaussian example

For a Gaussian distribution with mean $\mu$ and variance $\sigma^2$,

$$ p(x\mid \mu,\sigma^2)=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right). $$

The MLE for the mean is the sample mean:

$$ \hat\mu=\frac{1}{n}\sum_{i=1}^{n}x_i. $$

This result matters because linear regression can be derived by assuming Gaussian noise around a linear prediction.

Linear regression model

Linear regression predicts a continuous target using a weighted sum of features:

$$ \hat y_i = w^T x_i + b. $$

The probabilistic version assumes

$$ y_i = w^T x_i + b + \epsilon_i, $$

where $\epsilon_i \sim \mathcal{N}(0,\sigma^2)$ is Gaussian noise. This gives

$$ y_i \mid x_i,w,b \sim \mathcal{N}(w^T x_i+b,\sigma^2). $$

Maximising the Gaussian likelihood is equivalent to minimising the sum of squared errors:

$$ \operatorname{SSE}(w,b)=\sum_{i=1}^{n}(y_i-w^T x_i-b)^2. $$

With the bias included inside $X$, the mean squared error objective is

$$ J(w)=\frac{1}{2n}\lVert Xw-y\rVert_2^2. $$

The gradient is

$$ \nabla_w J(w)=\frac{1}{n}X^T(Xw-y). $$

Setting this gradient to zero gives the normal equation:

$$ X^TXw=X^Ty. $$

If $X^TX$ is invertible, the closed-form solution is

$$ \hat w=(X^TX)^{-1}X^Ty. $$

In practice, gradient-based optimisation may be preferred when $d$ is large or when the matrix inverse is unstable.

Regularisation

Regularisation reduces overfitting by discouraging unnecessarily large or complex weights. Ridge regression adds an $L_2$ penalty:

$$ J_{ridge}(w)=\frac{1}{2n}\lVert Xw-y\rVert_2^2+\lambda\lVert w\rVert_2^2. $$

Ridge shrinks weights smoothly toward zero. It is useful when features are correlated or when $X^TX$ is close to singular.

Lasso regression adds an $L_1$ penalty:

$$ J_{lasso}(w)=\frac{1}{2n}\lVert Xw-y\rVert_2^2+\lambda\lVert w\rVert_1. $$

The $L_1$ norm is

$$ \lVert w\rVert_1=\sum_{j=1}^{d}|w_j|. $$

Lasso can set some weights exactly to zero, which makes it useful for feature selection. The reason is geometric: the $L_1$ penalty has corners, and the optimum often occurs at a corner where one or more coefficients are zero.

Bayesian interpretation

Regularisation can also be understood as maximum a posteriori estimation. Instead of maximising only $p(D\mid \theta)$, MAP maximises

$$ p(\theta\mid D)\propto p(D\mid \theta)p(\theta). $$

A Gaussian prior on weights leads to an $L_2$ penalty. A Laplace prior leads to an $L_1$ penalty. This connects regularisation to prior beliefs about parameter size.

Model selection and cross-validation

A hyperparameter is not learned directly by the training objective. Examples include $\lambda$, polynomial degree, number of neighbours, and tree depth. Cross-validation estimates how well a hyperparameter setting generalises.

In $k$-fold cross-validation, the data is split into $k$ parts. The model is trained on $k-1$ parts and validated on the remaining part. This is repeated so every part is used for validation once. The final validation score is the average over folds.

Regularisation can be read in two ways: as a numerical penalty that discourages overly complex parameters, and as a Bayesian prior that encodes which parameter values are expected before seeing the data.

Flashcards