If you have ever read a scientific paper, you have almost certainly come across a p-value. And if you found it hard to understand, you are far from alone. Many working scientists misunderstand it too.
The p-value is a number that measures the "statistical significance" of an experiment, and it is often treated as the verdict on whether the experiment really found something or didn't. It determines whether a drug is judged to have worked, or whether one group is judged to differ from another.
Researchers' careers, the publication of papers, and even our understanding of the world all hinge on p-values. That is why it is worth understanding what a p-value actually means, because it is not what many people think it is.
What does a p-value mean?
Suppose you want to know whether adults in one country are taller on average than adults in another. You cannot measure everyone, so you take a sample of 50 adults from each country.
The first sample averages 1.80 meters, and the second averages 1.77 meters. That 3-centimeter gap tells you very little on its own, because any two samples will differ to some degree depending on who happened to be picked.
So is the difference enough to show a real difference in average height across the whole population? Or is it the kind of gap that sampling luck alone could produce?
To find out, scientists run a statistical test and calculate a p-value. Say the result is p = 0.06.
Here is what that means: if there were actually no difference at all between the two countries, the chance of seeing a gap this large through sampling luck alone would be about 6%.
Note what it does not mean. It does not mean there is a 6% chance the result was a fluke, or a 94% chance that the two countries really differ.
The p-value tells you how likely a difference this large would be if nothing were going on, not the probability that nothing is going on. People mix these two up constantly, and even those who know better often slip.
Why 0.05?
By convention, a p-value below 0.05 is called "statistically significant." Because the height study produced a p-value of 0.06, it would be reported as showing no significant difference.
So where does the 0.05 threshold come from? Nowhere in particular.
The British statistician Ronald Fisher offered it in the 1920s as a rule of thumb, and the convenient number simply stuck.
Using 0.05 as the bar for "significance" does have one useful property. Even if there is truly no effect, setting the threshold at 0.05 means chance alone will produce a "significant" result about 1 time in 20.
This is the false-positive rate we have agreed to tolerate. It is not a claim about what share of published findings are wrong.
What is arbitrary is the choice of 1 in 20 rather than 1 in 100 or 1 in 10, and it produces some odd outcomes. A result of p = 0.049 gets reported as a discovery, while p = 0.051 is treated as though nothing happened.
"Not significant" does not mean "no difference"
Returning to the height experiment, it is tempting to read p = 0.06 as proof that people in the two countries are the same height. That would be completely wrong.
A nonsignificant result only means the difference was not unusual enough to cross the threshold. The true difference could be large. The study may simply have been too small, or too noisy, to detect it. Here, noise means the variation between individuals: the more people's heights vary within each country, the larger the p-value for the same 3-centimeter gap.
The opposite confusion is just as common. In everyday English, "significant" means large or important, but in statistics it only means a threshold was passed.
Imagine a study of 40,000 people finds that a drug raises IQ by 0.5 points, with p < 0.001. That easily clears the bar for "statistical significance," yet a 0.5-point IQ difference is not important at all.
Now imagine a smaller study in which IQ rises by 5 points on average, with p = 0.06. It would be labeled "not significant," even though the effect, if real, would be far more meaningful.
This is why p-values need to be read together with effect sizes. The effect size answers the question that usually matters most: how big was the difference?
The temptation of p-hacking
Because so much rides on getting below 0.05, the threshold creates temptation. Imagine a researcher testing a supplement who recruits 50 people and gets p = 0.08. Frustratingly close.
So they add 10 more people and test again. p = 0.06. Closer still. They add another 10, finally dip below 0.05, stop there, and write up the study as though the sample size had been fixed from the start.
Peeking at the data and stopping the moment it crosses the threshold raises the chance of a "significant" result even when nothing is going on. The same happens when researchers try many outcomes or subgroups and report only the ones that work. Run enough tests, and some will fall below 0.05 by chance alone.
This is called p-hacking, and there is evidence that researchers do it fairly often.
Should p-values be abandoned?
Some people think so. In 2019, more than 800 scientists signed a call to retire the term "statistical significance." At least one journal has banned p-values outright.
Others argue for putting more weight on effect sizes and confidence intervals, which show how large an effect might be and how precisely it has been estimated.
Emphasizing effect sizes and confidence intervals seems wise. But scrapping p-values altogether may go too far.
The p-value tells us something useful: how often a result as extreme as ours, or more so, would appear if there were truly no effect. The problem is treating it as a ruling on whether a hypothesis is true or whether a finding matters.