Bayesian Linear Regression

Machine learning has made extraordinary progress over the past decade, largely by optimizing models to minimize prediction error on large datasets. But as these models move into production and into decisions that affect real people, a new set of questions has emerged. It's no longer enough to ask "what does the model predict?" The harder question is "how much should we trust that prediction?" Bayesian methods were built to answer exactly that question, and their relevance to modern machine learning has never been greater.

Consider a self-driving vehicle system that uses regression models to estimate the distance to obstacles. A point estimate of 4.2 meters is useful. But knowing that the model's uncertainty spans anywhere from 3.1 to 6.8 meters under current lighting conditions is critical for deciding whether to brake. This kind of calibrated uncertainty is what Bayesian models provide naturally, as a built-in feature of how they represent knowledge rather than as an afterthought bolted on later.

The rise of probabilistic machine learning as a subfield reflects exactly this need. Researchers and engineers at organizations like DeepMind, Google Brain, and academic institutions have invested heavily in Bayesian deep learning, Gaussian processes, and probabilistic programming frameworks precisely because uncertainty quantification matters at scale.^[1] Bayesian linear regression is the cleanest, most interpretable entry point into this broader world of probabilistic modeling.

Beyond uncertainty, Bayesian methods offer a principled way to incorporate prior knowledge into a model. In many real-world problems, you don't start from scratch. A pharmaceutical company modeling drug efficacy has decades of clinical trial data informing reasonable parameter ranges. A financial analyst modeling asset returns has theoretical constraints from economic theory. Bayesian regression lets you encode that knowledge formally, so the model doesn't have to rediscover it from data alone. This is especially powerful when data is scarce or expensive to collect.

Bayesian methods also provide a coherent framework for model comparison. Rather than relying on heuristics like adjusted R-squared or cross-validation alone, Bayesian model comparison uses the marginal likelihood, which automatically penalizes model complexity in a principled way. This connects directly to Occam's Razor: simpler models that explain the data equally well are preferred, and the math enforces that preference without requiring arbitrary tuning.

There's also a practical argument rooted in the current state of the field. Libraries like PyMC, Stan, and TensorFlow Probability have dramatically lowered the barrier to implementing Bayesian models. What once required custom mathematical derivations and hand-coded samplers can now be expressed in a few dozen lines of Python. The computational cost that historically made Bayesian methods impractical has dropped substantially with modern hardware and smarter algorithms. The tools are ready. The question is whether you are.

Finally, Bayesian thinking cultivates a healthier relationship with data and models in general. It forces explicit acknowledgment that models are approximations, that parameters are uncertain, and that predictions carry error. This epistemic humility is not a weakness. It's a feature that makes Bayesian practitioners more careful, more honest about model limitations, and ultimately more effective at communicating results to stakeholders who need to make decisions based on those results.

Ordinary least squares (OLS) regression is one of the most widely used statistical tools in existence, and for good reason. It's computationally simple, mathematically elegant, and produces interpretable results under well-understood conditions. But it carries a set of limitations that become increasingly problematic as you move from textbook problems to messy real-world applications. Understanding these limitations is not about dismissing frequentist methods. It's about knowing where they strain and where a Bayesian approach picks up the slack.

The most fundamental limitation is that OLS produces point estimates. The output is a single coefficient vector, a single predicted value, a single residual standard error. There's no native representation of uncertainty in the parameters themselves. Frequentist confidence intervals exist, but they're often misunderstood. A 95% confidence interval does not mean there's a 95% probability that the true parameter lies within that range. It means that if you repeated the experiment many times, 95% of the intervals constructed that way would contain the true value. That's a subtle but important distinction, and it makes frequentist intervals harder to communicate and easier to misinterpret.^[2]

A second limitation involves small sample sizes. OLS performs well when data is abundant relative to the number of parameters being estimated. When data is sparse, OLS estimates become highly variable and unreliable. The model has no mechanism to pull estimates toward plausible values based on prior knowledge. It treats every dataset as if it arrived in a vacuum, with no context from previous studies, domain expertise, or theoretical constraints. In practice, this leads to overfitting and poor generalization, especially in high-dimensional settings.

Regularization methods like Ridge and Lasso were developed specifically to address overfitting in frequentist regression. They work by adding a penalty term to the loss function that shrinks coefficients toward zero. This is effective, but the choice of regularization strength (the hyperparameter lambda) is typically selected through cross-validation, which is computationally expensive and doesn't have a natural probabilistic interpretation. As you'll see in Chapter 6, Bayesian regression provides a unified framework where regularization emerges naturally from prior specification, with a clear probabilistic meaning attached to every choice.

Frequentist regression also struggles with model uncertainty. When you select a single model through stepwise selection, AIC minimization, or cross-validation, you're committing to one structure and ignoring the uncertainty about whether that structure is correct. This leads to overconfident predictions because the selected model is treated as if it were the true model. Bayesian model averaging, by contrast, maintains a distribution over models and produces predictions that account for model uncertainty explicitly.

A concrete example: suppose you're modeling housing prices with 20 candidate predictors. Frequentist stepwise selection might settle on 8 predictors and report coefficients as if those 8 were definitively correct. A Bayesian approach would maintain uncertainty over which predictors matter, producing predictions that reflect the genuine ambiguity in model structure. In a market with limited transaction data, that difference in honesty about uncertainty can be the difference between a useful model and a dangerously overconfident one.

Another practical limitation is the handling of hierarchical or grouped data. Frequentist approaches to multilevel data, such as students nested within schools or patients nested within hospitals, require mixed-effects models that can be tricky to specify and interpret. Bayesian hierarchical models handle this structure naturally, with priors at each level of the hierarchy providing automatic regularization and enabling partial pooling of information across groups. Chapter 7 covers this in depth.

It's worth being fair here: frequentist methods are not broken. They're well-calibrated, computationally efficient, and backed by a century of theoretical development. For large datasets with well-understood structure, OLS and its regularized variants often perform just as well as Bayesian alternatives. The limitations described above become most acute in specific conditions: small data, high dimensions, strong prior knowledge, hierarchical structure, or applications where uncertainty communication is critical. Knowing those conditions is what allows you to choose the right tool for the job.

The deeper issue is philosophical. Frequentist methods define probability as the long-run frequency of events in repeated experiments. This works well for coin flips and clinical trials with large sample sizes. But many real-world questions don't fit that mold. What's the probability that this specific patient will respond to this treatment? What's the probability that this particular economic policy will reduce unemployment? These are one-time events, not repeatable experiments. Bayesian probability, defined as a degree of belief updated by evidence, handles these questions more naturally and more honestly.

This book is structured to take you from foundational concepts to practical implementation in a logical, cumulative sequence. Each chapter introduces new ideas that build on what came before, so the order matters. That said, if you already have a strong background in a particular area, the chapters are written to be reasonably self-contained, with cross-references to earlier material when needed. Here's what to expect at each stage of the journey.

Chapters 1 and 2 establish the two pillars you'll need before combining them. Chapter 1 covers Bayesian thinking from the ground up: Bayes' theorem, the roles of prior, likelihood, and posterior, and the philosophical contrast between Bayesian and frequentist probability. Chapter 2 reviews classical linear regression, including least squares estimation, model assumptions, diagnostics, and the extension to multiple predictors. If you're already comfortable with both of these areas, you can move through them quickly. If either feels shaky, spending time here will pay dividends in every chapter that follows.

Chapter 3 is where the two pillars merge. It introduces Bayesian linear regression proper: how to write down the likelihood function for a regression model, how to specify priors over the regression coefficients and noise variance, and how to derive the posterior distribution by combining them. This chapter establishes the core mathematical machinery that everything else in the book depends on.

Chapter 4 goes deeper into a special class of priors called conjugate priors. When the prior and likelihood belong to compatible mathematical families, the posterior has a closed-form solution, meaning you can compute it analytically without numerical approximation. The Normal-Inverse-Gamma prior is the key conjugate family for Bayesian linear regression, and this chapter walks through the derivation and interpretation of the resulting posterior parameters in detail.

Chapter 5 focuses on what you actually do with the posterior once you have it. The central topics are credible intervals (the Bayesian analog of confidence intervals, with a more intuitive interpretation), the predictive distribution for new data points, and how to use uncertainty estimates to make better decisions. This chapter includes worked examples from domains where uncertainty quantification has direct practical consequences.

Chapter 6 connects Bayesian regression to the regularization techniques you may already use. It shows that Ridge regression is mathematically equivalent to Bayesian regression with a Gaussian prior, and that Lasso corresponds to a Laplace prior. The spike-and-slab prior is introduced as a more flexible tool for feature selection. This chapter reframes regularization as a principled Bayesian choice rather than an ad hoc penalty, which changes how you think about hyperparameter selection.

The progression through the remaining chapters looks like this:

• Chapter 7 extends the framework to hierarchical models, where parameters at one level of the model are themselves drawn from distributions governed by higher-level parameters. This is the natural Bayesian treatment of grouped or multilevel data.
• Chapter 8 introduces computational methods for cases where closed-form solutions don't exist. Markov Chain Monte Carlo (MCMC) and variational inference are the two main approaches, and this chapter covers both their mechanics and their practical trade-offs.
• Chapter 9 addresses model comparison and selection using Bayesian tools: marginal likelihood, Bayes factors, and information criteria like WAIC and leave-one-out cross-validation.
• Chapter 10 brings everything together with practical implementation guidance, including a survey of software tools (PyMC, Stan, scikit-learn's Bayesian modules) and case studies from finance, healthcare, and engineering.

Throughout the book, mathematical notation is introduced carefully and used consistently. You'll encounter probability distributions, matrix algebra, and calculus at various points, but the focus is always on building intuition alongside the formalism. Every major equation is accompanied by a plain-language explanation of what it means and why it matters. Where derivations are long or technical, the key steps are highlighted and the rest is summarized so you can follow the logic without getting lost in algebra.

Concrete examples appear in every chapter. Some are simple and stylized, designed to make a single concept clear. Others are drawn from real applications: predicting energy consumption from weather data, estimating treatment effects in clinical trials, modeling financial returns with uncertainty bounds. These examples are not decoration. They're the primary vehicle for showing how abstract concepts translate into practical decisions.

By the time you reach the conclusion, you should be able to do the following with confidence:

1. Specify a Bayesian linear regression model appropriate for a given problem, including a justified choice of prior.
2. Derive or compute the posterior distribution, either analytically or using computational tools.
3. Construct and interpret credible intervals and predictive distributions.
4. Recognize when regularization methods like Ridge and Lasso correspond to specific Bayesian prior choices.
5. Build hierarchical models for grouped or multilevel data.
6. Apply MCMC or variational inference when closed-form solutions are unavailable.
7. Compare models using Bayesian criteria and communicate the results clearly.
8. Implement Bayesian regression in Python using modern probabilistic programming libraries.

One final note on what this book is not. It is not a comprehensive treatment of Bayesian statistics in general. Topics like Bayesian nonparametrics, Gaussian processes, and Bayesian neural networks are mentioned where relevant but not covered in depth. The focus is deliberately narrow: Bayesian linear regression, understood deeply and applied practically. That narrowness is a feature. Mastering one framework thoroughly is more valuable than skimming a dozen. Once you have a solid grasp of Bayesian linear regression, the path to those more advanced topics becomes much clearer and much shorter.