Beyond the P-Value

The replication crisis changed everything. Beginning around 2011, a series of high-profile failures to reproduce landmark findings sent shockwaves through psychology, medicine, and the social sciences. The Open Science Collaboration's 2015 replication project attempted to reproduce 100 published psychology studies and found that only about 36 to 39 percent replicated with statistically significant results at the original effect size.^[1] That number was alarming not because psychology is uniquely flawed, but because it revealed how widespread the underlying problems were across empirical disciplines.

The causes of the replication crisis are statistical at their core. Underpowered studies, p-hacking, selective outcome reporting, and the misuse of significance thresholds all contributed to a literature filled with inflated effect sizes and false positives. These are not exotic problems confined to bad actors. They emerge naturally from the incentive structures of academic publishing, where positive results get published and null results get filed away. Understanding the statistical mechanisms behind the crisis is the first step toward not contributing to it.

Beyond replication, the stakes of statistical literacy extend into public life. During the COVID-19 pandemic, statistical claims about vaccine efficacy, relative versus absolute risk reduction, and the interpretation of confidence intervals appeared daily in mainstream media. Many of those claims were misrepresented, sometimes by researchers themselves. A 2021 analysis found that several prominent papers reporting vaccine efficacy figures used relative risk reduction without clearly communicating the corresponding absolute risk reduction, creating a misleading impression of benefit magnitude.^[2] The difference between those two numbers is not a technicality. It is the difference between an informed decision and a manipulated one.

Statistical errors also have direct financial consequences. In drug development, a Phase III trial that fails to replicate a Phase II finding costs hundreds of millions of dollars and years of research effort. In finance, models built on spurious correlations discovered through data mining have contributed to catastrophic losses. In education policy, interventions scaled nationally based on small, underpowered pilot studies have consumed enormous resources without producing the promised outcomes. The cost of poor statistical practice is not just academic. It is measured in money, time, and human welfare.

For statisticians specifically, this moment carries a professional responsibility. You are among the people best positioned to push back against sloppy practice, to educate collaborators, to review manuscripts with genuine rigor, and to design studies that produce trustworthy evidence. That responsibility requires more than knowing the formulas. It requires understanding the social and cognitive pressures that lead smart, well-intentioned researchers to make systematic errors, and knowing how to counteract those pressures in your own work and in the work you evaluate.

The good news is that the field is actively reforming. Pre-registration of study hypotheses and analysis plans has become standard in many journals. Effect size reporting requirements are increasingly common. The American Statistical Association issued a landmark statement in 2016 clarifying what p-values can and cannot tell us, followed by a 2019 special issue calling for a fundamental shift away from binary significance thinking.^[3] These reforms need statisticians who understand them deeply enough to implement and advocate for them. That is exactly what this book prepares you to do.

Statistical literacy is not a fixed destination. It is a practice of continual critical engagement with methods, assumptions, and evidence. The chapters ahead are designed to build that practice systematically, starting from the most fundamental concepts and moving toward the most nuanced challenges in modern research design and appraisal.

This book is organized around the most consequential problems in frequentist statistical practice. Each chapter targets a specific area where misunderstanding or misuse is common, explains the underlying mechanism clearly, and provides concrete tools for doing better. The progression is deliberate: foundational concepts come first, followed by increasingly complex methodological challenges, and finally the broader ecosystem of research practice and evidence synthesis.

Here is a brief map of what each section covers:

• Chapters 1 through 3 address the core mechanics of hypothesis testing: what p-values actually measure, how Type I and Type II errors work and trade off against each other, and how multiple comparisons silently inflate false discovery rates.
• Chapters 4 and 5 tackle the distinction between statistical and practical significance, covering effect sizes and the proper interpretation of confidence intervals, two areas where even experienced researchers routinely go wrong.
• Chapters 6 and 7 examine the research practices that corrupt the statistical record: data dredging, HARKing (Hypothesizing After Results are Known), p-hacking, publication bias, and selective outcome reporting.
• Chapters 8 and 9 zoom out to the level of the scientific literature, covering the replication crisis and the methods used to synthesize evidence across studies through meta-analysis.
• Chapters 10 and 11 provide complementary perspectives: Bayesian approaches that address some of frequentism's inherent limitations, and a systematic framework for critically appraising published research.

You do not need to read this book cover to cover in sequence, though doing so will give you the most coherent understanding. If you are primarily interested in improving your manuscript review skills, Chapter 11 and its accompanying checklist can be used as a standalone reference. If you are designing a new study and want to avoid common pitfalls, Chapters 2, 3, and 6 are the most immediately relevant. If you are trying to understand why a colleague's analysis produced suspicious results, Chapters 1 and 4 will give you the diagnostic vocabulary you need.

Each chapter includes concrete examples drawn from real research contexts. These examples are chosen not to embarrass specific researchers but to illustrate how common and understandable these errors are. The goal is recognition, not judgment. When you see how a particular error arises naturally from the pressures of research practice, you are better equipped to avoid it yourself and to explain it clearly to others.

Throughout the book, you will find practical frameworks and checklists designed for immediate use. These include a pre-analysis planning template, a guide to selecting appropriate effect size measures by study design, a funnel plot interpretation guide, and a comprehensive statistical appraisal checklist for evaluating published papers. These tools are meant to be used, not just read. Consider bookmarking the sections most relevant to your current work and returning to them when you need a structured reference.

One important note about scope: this book focuses on frequentist methods because they remain the dominant framework in most empirical disciplines. Bayesian methods receive dedicated coverage in Chapter 10, but the treatment there is comparative and practical rather than exhaustive. If you want a deep dive into Bayesian computation or prior specification, excellent dedicated resources exist for that purpose. The goal here is to help you understand when and why Bayesian thinking offers something that frequentist methods cannot, and how to incorporate that perspective into your critical toolkit.

Finally, this book assumes you are comfortable with the mathematical foundations of statistics. Derivations are included where they illuminate a conceptual point, but the emphasis throughout is on understanding and application rather than proof. The aim is to build the kind of deep, flexible understanding that lets you reason clearly about statistical problems you have never seen before, not just recognize the ones you have.

Before diving into the pitfalls, it is worth being precise about what the frequentist framework actually is and what it was designed to accomplish. Frequentist statistics is built on the idea that probability represents the long-run frequency of outcomes in repeated experiments. A probability of 0.05 means that if you repeated the same experiment under identical conditions an infinite number of times, the event in question would occur in 5 percent of those repetitions. This is a fundamentally different conception of probability from the Bayesian view, which treats probability as a degree of belief about uncertain quantities.

The modern frequentist framework for hypothesis testing emerged from the work of two distinct traditions that were later merged, sometimes uncomfortably. Ronald Fisher developed the concept of the p-value as a measure of evidence against a null hypothesis, intended as a continuous index of how surprising the data were under that hypothesis.^[4] Jerzy Neyman and Egon Pearson developed a different framework focused on decision-making, in which you specify both a null and an alternative hypothesis, set acceptable error rates in advance, and use the test to make a binary accept-or-reject decision.^[5]

These two frameworks have different goals and different interpretations of what a test result means. Fisher's p-value is a measure of evidence. Neyman and Pearson's rejection region is a decision rule with controlled long-run error rates. The hybrid that most researchers use today, in which a p-value below 0.05 is treated as a binary decision to reject the null hypothesis, combines elements of both frameworks in a way that neither Fisher nor Neyman and Pearson would have fully endorsed. Understanding this historical tension helps explain why so many conceptual confusions about p-values persist.

The key components of the frequentist hypothesis testing framework are worth stating clearly, because each one is a potential source of error when misunderstood:

• The null hypothesis (H₀): A specific, testable claim about a population parameter, typically representing no effect or no difference. The null hypothesis must be precisely specified before data collection.
• The alternative hypothesis (H₁): The claim that the null hypothesis is false. In Neyman-Pearson testing, this must also be specified in advance, including its direction (one-tailed vs. two-tailed) and the minimum effect size of interest.
• The significance level (α): The maximum acceptable probability of rejecting a true null hypothesis (Type I error rate), set before data collection. The conventional value of 0.05 is a historical convention, not a mathematical law.
• Statistical power (1 - β): The probability of correctly rejecting a false null hypothesis, which depends on sample size, effect size, and α. Adequate power requires explicit calculation before data collection.
• The p-value: The probability of observing a test statistic as extreme as or more extreme than the one obtained, assuming the null hypothesis is true. It is not the probability that the null hypothesis is true.

Each of these components carries assumptions that are frequently violated in practice. The null hypothesis is often vague or implicitly defined. The significance level is chosen after seeing the data. Power calculations are skipped or performed post-hoc. And the p-value is routinely interpreted as something it is not. The chapters that follow address each of these failure modes in detail, but it is useful to have the full framework in view from the start so you can see how the pieces connect.

One of the most important things to understand about the frequentist framework is that its guarantees are long-run guarantees. When you set α = 0.05, you are not saying that there is a 5 percent chance your specific result is a false positive. You are saying that if you applied this decision rule repeatedly across many studies, you would make a Type I error in at most 5 percent of cases where the null hypothesis is true. This distinction matters enormously. It means that a single p-value below 0.05 tells you very little about whether your specific result is a true effect. The validity of the inference depends on the entire procedure being followed correctly, including pre-specification of hypotheses, random sampling, and no peeking at the data before the planned stopping point.

Frequentist methods are powerful and well-suited to many research questions, particularly those involving large samples, well-defined populations, and clearly specified hypotheses. They are less well-suited to situations where prior information is relevant, where sample sizes are small, or where the research question involves estimating the probability that a specific hypothesis is true. Recognizing these boundaries is part of using the framework responsibly. The goal of this book is not to undermine frequentist statistics but to help you use it with the precision and awareness it demands.

The frequentist framework is not broken. But it is routinely used in ways that violate its own assumptions, producing results that look rigorous but are not. The first step toward better practice is understanding exactly what the framework promises and what it does not.

With that foundation in place, the chapters ahead can focus on the specific ways things go wrong and the specific steps you can take to prevent them. The journey starts with the most misunderstood number in all of science: the p-value.