Part III: Maximum Likelihood Estimation

Part III develops the theory, practice, and inferential machinery of maximum likelihood estimation (MLE) — the single most important estimation principle in modern statistics. Building on the likelihood functions catalogued in Part II: The Likelihood Catalogue, we now ask: given a likelihood surface, how do we find its peak, what can we say about that peak, and how do we use it for inference?

The treatment is organised into three chapters:

• Chapter 9 (Chapter 9 — MLE Theory) establishes the theoretical backbone of MLE. We give a formal definition of the maximum likelihood estimator, then develop its large-sample properties—consistency, asymptotic normality, and efficiency—with full proof sketches. We state and explain the regularity conditions that underpin these results, prove the invariance (equivariance) property, and discuss finite-sample phenomena such as bias.

• Chapter 10 (Chapter 10 — Analytical MLE Solutions) works through closed-form MLE derivations for the most important parametric families: Normal, Exponential, Poisson, Binomial, Gamma, Beta, Uniform, and Multinomial. Every derivation proceeds step by step—write the log-likelihood, differentiate, solve, and verify via the second derivative—so that readers can reproduce each result with pencil and paper.

• Chapter 11 (Chapter 11 — Confidence Intervals and Hypothesis Testing) shows how to convert a fitted likelihood into confidence intervals and hypothesis tests. We derive the three classical test statistics—likelihood ratio, Wald, and score (Rao)—prove Wilks’ theorem, construct profile-likelihood confidence regions, and discuss multiple-testing corrections.

Maximum Likelihood Estimation

• Chapter 9 — MLE Theory
  • 9.1 Formal Definition of the Maximum Likelihood Estimator
  • 9.2 Existence and Uniqueness
  • 9.3 Consistency
  • 9.4 Asymptotic Normality
  • 9.5 Efficiency and the Cramer–Rao Lower Bound
  • 9.6 Invariance Property
  • 9.7 Regularity Conditions
  • 9.8 Finite-Sample Properties: Bias of the MLE
  • 9.9 Summary
• Chapter 10 — Analytical MLE Solutions
  • 10.1 Normal Distribution — \(\mathcal{N}(\mu, \sigma^2)\)
  • 10.2 Exponential Distribution — \(\text{Exp}(\lambda)\)
  • 10.3 Poisson Distribution — \(\text{Pois}(\lambda)\)
  • 10.4 Binomial Distribution — \(\text{Bin}(m, p)\)
  • 10.5 Gamma Distribution — \(\text{Gamma}(\alpha, \beta)\)
  • 10.6 Beta Distribution — \(\text{Beta}(\alpha, \beta)\)
  • 10.7 Uniform Distribution — \(\text{Unif}(0, \theta)\)
  • 10.8 Multinomial Distribution
  • 10.9 Summary of Analytical MLEs
• Chapter 11 — Confidence Intervals and Hypothesis Testing
  • 11.1 The Likelihood Ratio Test
  • 11.2 The Wald Test
  • 11.3 The Score (Rao) Test
  • 11.4 Comparison of the Three Tests
  • 11.5 Confidence Intervals from Fisher Information (Wald-Type CIs)
  • 11.6 Profile Likelihood Confidence Intervals
  • 11.7 Likelihood Ratio Confidence Regions
  • 11.8 Complete A/B Test Analysis: All Methods Together
  • 11.9 Multiple Testing Corrections
  • 11.10 Practical Guidance: Choosing a Method
  • 11.11 Summary