Maximum Likelihood Estimation

Maximum likelihood estimation (MLE) is a method for estimating the parameters of a statistical model by finding the values that maximize the likelihood function, which measures the probability (or, for continuous data, the probability density) of the collected data given specific parameter values. It is one of the most widely used estimation methods in statistics, applicable to a broad range of models from simple distributions to complex regression and structural equation models.

MLE provides a principled, general purpose approach to parameter estimation that works across many types of statistical models. It underlies the fitting of logistic regression, Poisson regression, mixed effects models, survival models, structural equation models, and many other methods that researchers use daily. Understanding MLE helps researchers interpret model output, diagnose estimation problems, and appreciate the assumptions underlying their analyses. Its statistical properties, including consistency (converging to the true parameter value as sample size increases), asymptotic normality, and asymptotic efficiency (in large samples, its variance approaches the lowest achievable, the Cramér Rao lower bound), make it the default estimation method for many models in statistical software.

The core idea of MLE is to ask: given the observed data, which parameter values make these data most probable? The likelihood function L(theta | data) is defined as the probability (or probability density) of the observed data as a function of the parameter(s) theta. For independent observations, the likelihood is the product of the probability of each individual observation. In practice, the log likelihood (the natural logarithm of the likelihood) is used because it converts products into sums, simplifying the mathematics and computation.

The maximum likelihood estimate is the value of theta that maximizes the log likelihood function. For simple models, this can be found analytically by taking the derivative of the log likelihood with respect to theta, setting it to zero, and solving. For more complex models, numerical optimization algorithms (such as Newton Raphson, Fisher scoring, or the EM algorithm) iteratively search for the maximum.

Once the MLE is obtained, its properties allow the construction of confidence intervals and hypothesis tests. The standard error of the MLE is derived from the inverse of the Fisher information matrix, which measures the curvature of the log likelihood at the maximum. Steeper curvature (more information) produces smaller standard errors and more precise estimates. Likelihood ratio tests compare the fit of nested models through twice the difference in their log likelihoods, which approximately follows a chi squared distribution under the null hypothesis, with degrees of freedom equal to the number of parameters the simpler model drops.

For small samples, MLE can be biased, and alternative methods such as restricted maximum likelihood (REML), used in mixed effects models, adjust for this bias in variance component estimation. Bayesian approaches extend the MLE framework by combining the likelihood with a prior distribution to produce posterior estimates, which can be more stable when data are sparse.

MLE is appropriate whenever you are fitting a parametric statistical model and want estimates that are consistent, efficient, and asymptotically normal. It is the default method for logistic regression, Poisson regression, and structural equation modeling; Cox proportional hazards models are fitted by maximizing a partial likelihood, and maximum likelihood is one of the common extraction methods in factor analysis. In mixed effects models, REML is preferred for variance components, while full ML is used when comparing models with different fixed effects. When sample sizes are very small, penalized likelihood methods (such as Firth's correction for logistic regression) or Bayesian methods may be more appropriate.

A common error is interpreting a converged MLE as necessarily being the global maximum of the likelihood, when the optimization algorithm may have found a local maximum, particularly in models with complex likelihoods such as mixture models. Checking convergence diagnostics and trying multiple starting values helps address this issue. Another mistake is applying MLE to models where the regularity conditions (for example, a smooth, identifiable likelihood, a true parameter value that is not on the boundary of the parameter space, and a range of possible data values that does not depend on the parameter) are not met, which can produce unreliable estimates and invalid standard errors. Ignoring warnings about non positive definite Hessian matrices (which indicate that the information matrix cannot be properly inverted) and proceeding with the analysis is another common error. Finally, some researchers interpret the maximum likelihood estimate as the most probable parameter value, which is a Bayesian interpretation; the MLE is the parameter value that makes the data most probable, which is a subtle but important distinction.

A biostatistician fits a logistic regression model to predict 30 day mortality in 5,000 intensive care unit patients using age, APACHE II score, mechanical ventilation status, and vasopressor use as predictors. The model is estimated using MLE via the Newton Raphson algorithm, which converges in six iterations. The log likelihood at convergence is negative 1,245.3. The estimated odds ratio for vasopressor use is 2.8 (95% CI: 2.1 to 3.7), with the confidence interval derived from the inverse Fisher information matrix. A likelihood ratio test comparing the full model to a reduced model without vasopressor use yields chi squared = 48.2, df = 1, p < 0.001.