Likelihood-Based Inference

From Foundations to Research

---

Kevin Korfmann

A comprehensive guide from basic probability through research-grade algorithms

Photo by Joel Abraham on Unsplash

Likelihood-Based Inference: From Foundations to Research

This is a comprehensive guide to likelihood-based statistical inference, designed to take the reader from basic probability through to research-grade methods. Every result is derived from first principles with plain-English explanations alongside the mathematics.

Contents

• Part I: Foundations
  • Chapter 1: Probability Basics
    • 1.1 Sample Spaces and Events
    • 1.2 The Kolmogorov Axioms
    • 1.3 Conditional Probability
    • 1.4 The Law of Total Probability
    • 1.5 Bayes’ Theorem
    • 1.6 Independence
    • 1.7 Summary
  • Chapter 2: Random Variables
    • 2.1 What Is a Random Variable?
    • 2.2 Probability Mass Function (PMF)
    • 2.3 Probability Density Function (PDF)
    • 2.4 Cumulative Distribution Function (CDF)
    • 2.5 Expectation (Expected Value)
    • 2.6 Variance
    • 2.7 Moment Generating Functions
    • 2.8 Joint Distributions
    • 2.9 Covariance and Correlation
    • 2.10 Conditional Distributions and Conditional Expectation
    • 2.11 The Law of Large Numbers
    • 2.12 The Central Limit Theorem
    • 2.13 Summary
  • Chapter 3: Common Distributions
    • 3.1 Discrete Distributions
    • 3.2 Continuous Distributions
    • 3.3 Relationships Between Distributions
    • 3.4 Connecting the Hospital Story
    • 3.5 Summary
  • Chapter 4: The Likelihood Function
    • 4.1 Likelihood vs. Probability: The Key Distinction
    • 4.2 Formal Definition of the Likelihood Function
    • 4.3 The Log-Likelihood
    • 4.4 Finding the MLE Analytically
    • 4.5 The Score Function
    • 4.6 Fisher Information
    • 4.7 Sufficient Statistics
    • 4.8 The Likelihood Principle
    • 4.9 Bringing It All Together: Complete Light Bulb Analysis
    • 4.10 Summary
• Part II: The Likelihood Catalogue
  • Chapter 5 — Discrete Likelihoods
    • 5.1 Bernoulli Distribution
    • 5.2 Binomial Distribution
    • 5.3 Poisson Distribution
    • 5.4 Negative Binomial Distribution
    • 5.5 Geometric Distribution
    • 5.6 Hypergeometric Distribution
    • 5.7 Multinomial Distribution
    • 5.8 Zero-Inflated Poisson (ZIP) Distribution
    • 5.9 Summary and Comparison
  • Chapter 6 — Continuous Likelihoods
    • 6.1 Normal (Gaussian) Distribution
    • 6.2 Exponential Distribution
    • 6.3 Gamma Distribution
    • 6.4 Beta Distribution
    • 6.5 Log-Normal Distribution
    • 6.6 Weibull Distribution
    • 6.7 Pareto Distribution
    • 6.8 Student’s t-Distribution
    • 6.9 Chi-Squared Distribution
    • 6.10 F-Distribution
    • 6.11 Uniform Distribution
    • 6.12 Cauchy Distribution
    • 6.13 Summary Table
  • Chapter 7 — Multivariate Likelihoods
    • 7.1 Matrix Calculus Essentials
    • 7.2 Multivariate Normal Distribution
    • 7.3 Wishart Distribution
    • 7.4 Inverse-Wishart Distribution
    • 7.5 Dirichlet Distribution
    • 7.6 Multinomial Distribution (Multivariate Treatment)
    • 7.7 Summary
  • Chapter 8 — Specialized Likelihoods
    • 8.1 Profile Likelihood
    • 8.2 Partial Likelihood
    • 8.3 Marginal Likelihood
    • 8.4 Conditional Likelihood
    • 8.5 Composite Likelihood
    • 8.6 Quasi-Likelihood
    • 8.7 Pseudo-Likelihood
    • 8.8 Empirical Likelihood
    • 8.9 Censored and Truncated Likelihoods
    • 8.10 Penalized Likelihood
    • 8.11 Summary
• Part III: Maximum Likelihood Estimation
  • Chapter 9 — MLE Theory
    • 9.1 Formal Definition of the Maximum Likelihood Estimator
    • 9.2 Existence and Uniqueness
    • 9.3 Consistency
    • 9.4 Asymptotic Normality
    • 9.5 Efficiency and the Cramer–Rao Lower Bound
    • 9.6 Invariance Property
    • 9.7 Regularity Conditions
    • 9.8 Finite-Sample Properties: Bias of the MLE
    • 9.9 Summary
  • Chapter 10 — Analytical MLE Solutions
    • 10.1 Normal Distribution — \(\mathcal{N}(\mu, \sigma^2)\)
    • 10.2 Exponential Distribution — \(\text{Exp}(\lambda)\)
    • 10.3 Poisson Distribution — \(\text{Pois}(\lambda)\)
    • 10.4 Binomial Distribution — \(\text{Bin}(m, p)\)
    • 10.5 Gamma Distribution — \(\text{Gamma}(\alpha, \beta)\)
    • 10.6 Beta Distribution — \(\text{Beta}(\alpha, \beta)\)
    • 10.7 Uniform Distribution — \(\text{Unif}(0, \theta)\)
    • 10.8 Multinomial Distribution
    • 10.9 Summary of Analytical MLEs
  • Chapter 11 — Confidence Intervals and Hypothesis Testing
    • 11.1 The Likelihood Ratio Test
    • 11.2 The Wald Test
    • 11.3 The Score (Rao) Test
    • 11.4 Comparison of the Three Tests
    • 11.5 Confidence Intervals from Fisher Information (Wald-Type CIs)
    • 11.6 Profile Likelihood Confidence Intervals
    • 11.7 Likelihood Ratio Confidence Regions
    • 11.8 Complete A/B Test Analysis: All Methods Together
    • 11.9 Multiple Testing Corrections
    • 11.10 Practical Guidance: Choosing a Method
    • 11.11 Summary
• Part IV: Optimization for Likelihood
  • Chapter 12: Gradient Methods
    • 12.1 Gradient Descent from the Taylor Expansion
    • 12.2 Step-Size Selection
    • 12.3 Convergence Analysis
    • 12.4 Stochastic Gradient Descent
    • 12.5 Momentum
    • 12.6 Adaptive Methods
    • 12.7 Application to Log-Likelihood Maximization
    • 12.8 Summary
  • Chapter 13: Newton and Scoring Methods
    • 13.1 Newton–Raphson from the Second-Order Taylor Expansion
    • 13.2 Application to Maximum Likelihood Estimation
    • 13.3 Fisher Scoring
    • 13.4 Connection to IRLS for Generalized Linear Models
    • 13.5 Modified Newton Methods
    • 13.6 Convergence of Newton’s Method
    • 13.7 Practical Issues
    • 13.8 Worked Example: Logistic Regression
    • 13.9 Summary
  • Chapter 14: Quasi-Newton Methods
    • 14.1 Motivation
    • 14.2 The Secant Condition
    • 14.3 The BFGS Update
    • 14.4 Limited-Memory BFGS (L-BFGS)
    • 14.5 The SR1 Update
    • 14.6 Trust-Region Methods
    • 14.7 Comparison of Methods
    • 14.8 Summary
  • Chapter 15: The EM Algorithm
    • 15.1 The Incomplete-Data Problem
    • 15.2 Derivation via Jensen’s Inequality
    • 15.3 Monotone Ascent: EM Never Decreases the Likelihood
    • 15.4 Example: Gaussian Mixture Model
    • 15.5 Example: Simple Missing Data
    • 15.6 ECM: Expectation Conditional Maximization
    • 15.7 MCEM: Monte Carlo EM
    • 15.8 Variational EM
    • 15.9 Convergence Rate of EM
    • 15.10 Generalizations and Connections
    • 15.11 Summary
  • Chapter 16: Constrained Optimization
    • 16.1 Equality Constraints and Lagrange Multipliers
    • 16.2 Inequality Constraints and KKT Conditions
    • 16.3 Examples in Maximum Likelihood
    • 16.4 Augmented Lagrangian Method
    • 16.5 Barrier (Interior-Point) Methods
    • 16.6 Penalty Methods
    • 16.7 Reparameterization as an Alternative to Constraints
    • 16.8 Putting It All Together: Constrained MLE
    • 16.9 Summary
• Part V: Advanced and Research Topics
  • Chapter 17 – Bayesian Connections
    • 17.1 Bayes’ Theorem for Parameters
    • 17.2 Prior Specification
    • 17.3 Conjugate Priors
    • 17.4 MAP Estimation
    • 17.5 The Laplace Approximation
    • 17.6 Credible Intervals vs Confidence Intervals
    • 17.7 Bayesian Model Comparison
    • 17.8 Summary
  • Chapter 18 – Computational Methods
    • 18.1 Monte Carlo Integration
    • 18.2 The Metropolis–Hastings Algorithm
    • 18.3 Gibbs Sampling
    • 18.4 Hamiltonian Monte Carlo
    • 18.5 Convergence Diagnostics
    • 18.6 Variational Inference
    • 18.7 Integrated Nested Laplace Approximations (INLA)
    • 18.8 Comparison of Methods
    • 18.9 Summary
  • Chapter 19 – Model Selection
    • 19.1 The Bias–Variance Trade-off
    • 19.2 Akaike Information Criterion (AIC)
    • 19.3 Bayesian Information Criterion (BIC)
    • 19.4 Deviance Information Criterion (DIC)
    • 19.5 Widely Applicable Information Criterion (WAIC)
    • 19.6 Cross-Validation
    • 19.7 Likelihood Ratio Test for Nested Models
    • 19.8 Model Averaging
    • 19.9 A Practical Decision Guide
    • 19.10 Summary
  • Chapter 20 – Modern Research Frontiers
    • 20.1 Automatic Differentiation
    • 20.2 Neural Density Estimation
    • 20.3 Simulation-Based Inference (Likelihood-Free)
    • 20.4 Amortized Inference
    • 20.5 Normalizing Flows
    • 20.6 Variational Autoencoders (VAEs)
    • 20.7 Score Matching
    • 20.8 Connections Back to Classical Likelihood
    • 20.9 Summary
  • Chapter 21 – Numerical Considerations
    • 21.1 Floating-Point Arithmetic
    • 21.2 Working in Log-Space
    • 21.3 The Log-Sum-Exp Trick
    • 21.4 Numerical Differentiation
    • 21.5 Numerical Hessians
    • 21.6 Condition Numbers
    • 21.7 Cholesky vs Inverse for Solving Linear Systems
    • 21.8 Parameterization Matters
    • 21.9 Sparse and Structured Hessians
    • 21.10 Parallel and GPU Computation
    • 21.11 Software Libraries
    • 21.12 Summary
• Appendices
  • Appendix A: Linear Algebra Review
    • Vectors and Matrices
    • Matrix Multiplication, Transpose, Trace, and Determinant
    • Inverse Matrices and Positive Definiteness
    • Eigenvalues and Eigenvectors
    • Matrix Decompositions
    • Matrix Calculus Essentials
    • Key Matrix Calculus Identities
    • Quadratic Forms and Their Optimization
  • Appendix B: Calculus Review
    • Differentiation Rules
    • Partial Derivatives and Gradients
    • Taylor Series
    • Optimization
    • Integration Techniques
    • The Gamma Function
    • Stirling’s Approximation
    • Useful Integrals
  • Appendix C: Notation and Glossary
    • Mathematical Notation
    • Greek Letters and Their Typical Uses
    • Glossary of Key Terms
    • Common Abbreviations

Indices and tables

• Index

• Search Page