Beyond the Numbers

There is a common assumption in quantitative fields that philosophy is a luxury, something interesting to think about but not essential for getting work done. This assumption is understandable but mistaken. Philosophy is not decoration added on top of statistics. It is embedded in the foundations of every statistical method, whether you acknowledge it or not. The question is not whether you are making philosophical commitments when you do regression analysis. The question is whether you are making them consciously or by accident.

Consider a straightforward example. A public health researcher runs a regression to study the relationship between income and life expectancy. The model shows a strong positive association: higher income predicts longer life. The researcher reports this finding and a policy team uses it to argue for income redistribution as a health intervention. But wait. Does the regression actually show that income causes longer life? Or does it show that some third factor, like education or neighborhood quality, drives both income and health outcomes? The regression itself cannot answer that question. Only philosophical reasoning about causation, confounding, and the structure of the data-generating process can.

This is not a rare edge case. It is the central challenge of applied statistics. Every regression model embeds assumptions about causal structure, measurement validity, and the scope of generalization. When those assumptions go unexamined, results get misinterpreted. Policies get built on shaky foundations. Scientific claims get overstated. The history of science is full of examples where technically correct statistical analyses led to deeply wrong conclusions, precisely because the philosophical scaffolding was never inspected.^[1]

Philosophy of science offers tools for exactly this kind of inspection. Concepts like epistemology (the study of knowledge and its limits), ontology (the study of what exists and how), and causal inference (the logic of cause-and-effect reasoning) give you a vocabulary for asking the right questions about your models. They help you distinguish between what your data can support and what you are projecting onto it. That distinction is not just academically interesting. It is practically essential.

There is also a more personal reason why philosophy matters for statistics. When you understand the deeper logic of what you are doing, the work becomes more engaging. Regression stops feeling like a checklist of procedures and starts feeling like a genuine inquiry. You are not just running software. You are constructing an argument about how the world works, testing it against evidence, and revising your beliefs accordingly. That is one of the most intellectually satisfying things a person can do, and it is exactly what good science looks like.

Some of the most important advances in statistical thinking have come from people who took philosophical questions seriously. The debate between frequentist and Bayesian approaches to probability, for instance, is fundamentally a philosophical disagreement about what probability means. Is it a long-run frequency of events, or a degree of belief held by a rational agent? That question has enormous practical consequences for how you interpret p-values, confidence intervals, and model uncertainty. You cannot resolve it by running more regressions. You have to think carefully about what you believe probability actually is.^[2]

None of this requires a philosophy degree. The concepts introduced in this book are accessible to anyone with curiosity and a willingness to slow down and think carefully. What it does require is a certain intellectual humility: a recognition that the tools you use carry assumptions, that those assumptions can be wrong, and that being a rigorous analyst means examining them rather than taking them for granted. That kind of humility is not a weakness. It is the foundation of trustworthy science.

A useful habit to develop: Before finalizing any regression analysis, ask yourself three questions. What am I assuming about the causal structure here? What would have to be true for my interpretation to be valid? And what alternative explanations have I not ruled out? These are philosophical questions, and they are just as important as checking your residual plots.

Most introductions to regression start with the formula Y = β₀ + β₁X + ε and spend the next several chapters explaining what each symbol means and how to estimate the parameters. That is a perfectly reasonable place to start. But it creates a subtle problem: it frames regression as a mathematical object first and a scientific tool second. The formula becomes the thing, rather than a compact representation of a much richer set of ideas.

Think about what that formula is actually saying. It claims that some outcome Y can be understood as a linear function of some predictor X, plus some irreducible noise ε. Every part of that claim is philosophically loaded. The linearity assumption says something about the shape of the relationship in the world. The inclusion of X and the exclusion of other variables says something about what you believe matters. The error term ε represents everything you have decided not to model, which is itself a philosophical choice about the limits of your theory. The formula is not neutral. It is a compressed argument.

Understanding regression beyond formulas means learning to read that argument. It means asking: why this functional form and not another? Why these variables and not others? What does it mean for the error term to be normally distributed, and why should we expect that to be true? These questions do not have purely mathematical answers. They require you to think about the phenomenon you are studying, the theory you are testing, and the kind of knowledge you are trying to produce.

Here is a concrete illustration. Suppose you are modeling the relationship between study hours and exam scores. A simple linear regression might fit the data reasonably well. But is the relationship truly linear? At some point, additional study hours probably yield diminishing returns. A student who studies 20 hours might not score twice as well as one who studies 10 hours. The linear model is a simplification, and knowing that it is a simplification changes how you interpret the results. The coefficient on study hours is not a universal law. It is an approximation that works within a certain range, under certain conditions, for a certain population. That is a very different thing.

This perspective also changes how you think about model fit. A high R-squared value is often treated as a sign of success, but it is really just a measure of how well your model describes the variance in your sample. It says nothing about whether your model is theoretically correct, whether it will generalize to new data, or whether the relationships it captures are causal. A model can fit the data beautifully and still be deeply misleading if the underlying assumptions are wrong. Conversely, a model with modest fit can be enormously valuable if it captures a genuine causal mechanism.^[3]

Going beyond formulas also means taking seriously the role of theory in regression analysis. In the best scientific practice, you do not start with data and search for patterns. You start with a theory about how the world works, derive predictions from that theory, and then use regression to test whether the data are consistent with those predictions. The theory tells you which variables to include, what functional form to expect, and what direction of effect to anticipate. Without theory, regression becomes a fishing expedition, and fishing expeditions are notoriously prone to finding things that are not really there.

The distinction between exploratory and confirmatory analysis is crucial here. Exploratory analysis uses regression to generate hypotheses. Confirmatory analysis uses regression to test them. Both are legitimate, but they require different standards of evidence and different levels of caution in interpretation. Mixing them up, running an exploratory analysis and then reporting the results as if they were confirmatory, is one of the most common sources of false positives in published research.^[4]

What does it look like to practice regression beyond formulas in real life? It looks like spending as much time thinking about your model before you run it as you do analyzing the output after. It looks like writing down your theoretical expectations before you see the results. It looks like being genuinely willing to revise your model when the assumptions do not hold, rather than just reporting the output and hoping no one checks. And it looks like communicating your results with appropriate humility, acknowledging what your model can and cannot tell you.

• Before running a model: Write down your theoretical expectations. Which direction do you expect each coefficient to go? Why?
• During model specification: Justify every variable you include and every variable you exclude. What does each inclusion or exclusion assume about the world?
• After getting results: Ask whether the results make theoretical sense, not just statistical sense. A significant coefficient that contradicts your theory is a signal to investigate, not celebrate.
• When communicating findings: Be explicit about what your model assumes and where those assumptions might break down.

This approach takes more time and more intellectual effort than simply running the numbers. But it produces better science, more honest communication, and more durable knowledge. That is the payoff for thinking about regression as a philosophical tool rather than a computational procedure.

This book is organized to build understanding progressively, moving from foundational philosophical concepts to increasingly specific applications in regression analysis. Each chapter introduces ideas that are then carried forward and deepened in later chapters. That means the book rewards reading in order, at least for the first pass. But it is also designed so that individual chapters can stand alone as references once you have the overall framework in mind.

The first two chapters establish the philosophical groundwork. Chapter 1 covers the basics of epistemology and the philosophy of science, giving you the conceptual vocabulary you will need throughout the rest of the book. Chapter 2 tackles the causation-versus-correlation distinction in depth, which is arguably the most important philosophical issue in applied statistics. If you already have a strong background in philosophy of science, you might move through these chapters quickly. If this territory is new to you, take your time. The ideas introduced here will pay dividends in every chapter that follows.

Chapters 3 through 6 form the philosophical core of the book. They examine regression itself as a philosophical tool, explore what the standard assumptions really mean, investigate the theory of measurement and variable selection, and analyze the epistemological foundations of estimation and inference. These chapters are where the book's central argument is developed most fully. They are also where you are most likely to encounter ideas that challenge assumptions you have been carrying around without realizing it.

Chapters 7 through 9 move into more applied territory, covering model selection, interpretation of results, and advanced topics including multiple regression, nonlinear models, and the Bayesian perspective. These chapters assume you are comfortable with the philosophical framework established earlier and focus on applying that framework to specific analytical challenges. Chapter 10 brings everything together through case studies drawn from social science and natural science, including an honest examination of cases where regression analysis went wrong and what those failures teach us.

Throughout the book, you will find several recurring features designed to make the material more concrete and actionable:

• Worked examples: Real-world scenarios that illustrate how philosophical concepts play out in actual analyses.
• Common pitfalls: Specific mistakes that analysts make when they ignore the philosophical dimensions of regression, with guidance on how to avoid them.
• Reflection questions: Prompts to help you apply the ideas to your own work and think through the implications for your specific field.
• Key terms: Important concepts are bolded when first introduced and defined in the glossary at the back of the book.
• Footnotes and citations: References to primary sources and further reading for those who want to go deeper on specific topics.

A word about the level of mathematical content. This book assumes you are familiar with the basics of regression: what a coefficient is, how ordinary least squares estimation works, what a p-value represents, and how to interpret R-squared. It does not require advanced mathematics, and it does not derive formulas from first principles. When mathematical notation appears, it is used to clarify ideas, not to demonstrate technical sophistication. If you find yourself wanting more mathematical depth on a particular topic, the further reading section at the back of the book points you toward resources that provide it.

It is also worth saying something about what this book is not. It is not a software tutorial. It does not walk you through regression in R, Python, Stata, or any other package. There are excellent resources for that purpose, and this book is designed to complement them, not replace them. The goal here is to give you a framework for thinking about what you are doing when you use those tools, not to teach you how to use the tools themselves.

Similarly, this book is not a comprehensive treatment of the philosophy of science. The philosophical concepts introduced here are selected because they are directly relevant to regression analysis and statistical reasoning. Readers who want a deeper grounding in philosophy of science will find suggestions in the further reading section. The goal is not to turn you into a philosopher but to give you enough philosophical literacy to be a more thoughtful and rigorous analyst.

One practical suggestion: as you read, keep your own work in mind. Think about analyses you have done or are currently doing. Ask yourself how the ideas in each chapter apply to your specific context. The most valuable learning happens when abstract concepts connect to concrete problems you actually care about. If you are a researcher in public health, think about how the discussion of causal inference applies to your study designs. If you are a data analyst in business, think about how the discussion of model selection applies to the forecasting models you build. The philosophical framework in this book is general, but its value is realized in the specific.

A note on intellectual honesty: Some of the ideas in this book will challenge things you have been taught or things you currently believe about statistics. That discomfort is a good sign. It means you are engaging with the material seriously. The goal is not to undermine your confidence in regression analysis but to give you a more accurate and nuanced understanding of what it can and cannot do. A tool you understand deeply is far more powerful than one you use on autopilot.

Finally, this book is written in the belief that the best statistical practice is inseparable from careful thinking about evidence, knowledge, and the nature of scientific claims. That belief is not just philosophical idealism. It is grounded in the practical reality that analyses built on unexamined assumptions tend to fail in predictable ways, while analyses built on clear thinking tend to hold up. The chapters ahead are an invitation to think more clearly about one of the most powerful tools in the modern scientific toolkit. The journey starts with a question that turns out to be surprisingly deep: what does it mean to know something from data?