Part IV: Optimization for Likelihood

Maximum likelihood estimation reduces, in almost every case, to an optimization problem: find the parameter values that make the observed data most probable. Parts I–III of this book developed the what — what the likelihood function is, what properties its maximizer enjoys, and what we can infer from it. Part IV addresses the how: the numerical algorithms that actually locate that maximizer.

We begin with gradient methods (Chapter 12: Gradient Methods), the workhorses of modern optimization. Starting from nothing more than a Taylor expansion, we derive gradient descent, explore step-size strategies, and then follow the historical arc through stochastic gradient descent, momentum, and the adaptive methods (Adam and friends) that dominate machine-learning practice.

Newton and scoring methods (Chapter 13: Newton and Scoring Methods) exploit second-order information — the curvature of the log-likelihood — to achieve dramatically faster local convergence. We derive Newton–Raphson, Fisher scoring, and the iteratively reweighted least-squares (IRLS) algorithm used in every generalized linear model package.

Computing and inverting a full Hessian is expensive. Quasi-Newton methods (Chapter 14: Quasi-Newton Methods) approximate the Hessian on the fly using only gradient information. We derive BFGS and its limited-memory cousin L-BFGS, discuss trust-region strategies, and compare the practical trade-offs.

The EM algorithm (Chapter 15: The EM Algorithm) is the method of choice when the likelihood involves latent variables or missing data. We give a self-contained derivation via Jensen’s inequality, prove monotone ascent, and work through Gaussian mixture models in full detail. Extensions — ECM, MCEM, and variational EM — round out the chapter.

Finally, constrained optimization (Chapter 16: Constrained Optimization) handles the reality that parameters often live on restricted sets (probabilities must sum to one, variances must be positive). Lagrange multipliers, KKT conditions, barrier methods, and the practical trick of reparameterization are all developed here.

Together, these five chapters give a self-contained toolkit for turning any likelihood function into a concrete parameter estimate.

Optimization

• Chapter 12: Gradient Methods
  • 12.1 Gradient Descent from the Taylor Expansion
  • 12.2 Step-Size Selection
  • 12.3 Convergence Analysis
  • 12.4 Stochastic Gradient Descent
  • 12.5 Momentum
  • 12.6 Adaptive Methods
  • 12.7 Application to Log-Likelihood Maximization
  • 12.8 Summary
• Chapter 13: Newton and Scoring Methods
  • 13.1 Newton–Raphson from the Second-Order Taylor Expansion
  • 13.2 Application to Maximum Likelihood Estimation
  • 13.3 Fisher Scoring
  • 13.4 Connection to IRLS for Generalized Linear Models
  • 13.5 Modified Newton Methods
  • 13.6 Convergence of Newton’s Method
  • 13.7 Practical Issues
  • 13.8 Worked Example: Logistic Regression
  • 13.9 Summary
• Chapter 14: Quasi-Newton Methods
  • 14.1 Motivation
  • 14.2 The Secant Condition
  • 14.3 The BFGS Update
  • 14.4 Limited-Memory BFGS (L-BFGS)
  • 14.5 The SR1 Update
  • 14.6 Trust-Region Methods
  • 14.7 Comparison of Methods
  • 14.8 Summary
• Chapter 15: The EM Algorithm
  • 15.1 The Incomplete-Data Problem
  • 15.2 Derivation via Jensen’s Inequality
  • 15.3 Monotone Ascent: EM Never Decreases the Likelihood
  • 15.4 Example: Gaussian Mixture Model
  • 15.5 Example: Simple Missing Data
  • 15.6 ECM: Expectation Conditional Maximization
  • 15.7 MCEM: Monte Carlo EM
  • 15.8 Variational EM
  • 15.9 Convergence Rate of EM
  • 15.10 Generalizations and Connections
  • 15.11 Summary
• Chapter 16: Constrained Optimization
  • 16.1 Equality Constraints and Lagrange Multipliers
  • 16.2 Inequality Constraints and KKT Conditions
  • 16.3 Examples in Maximum Likelihood
  • 16.4 Augmented Lagrangian Method
  • 16.5 Barrier (Interior-Point) Methods
  • 16.6 Penalty Methods
  • 16.7 Reparameterization as an Alternative to Constraints
  • 16.8 Putting It All Together: Constrained MLE
  • 16.9 Summary