Statistically Significant and Other Lies Numbers Tell

Audio article

Checking narration… Playback speed 0.75×1×1.25×1.5×1.75×2×

Checking article narration…

There is a phrase, and the phrase is statistically significant, and I keep getting stuck on the second word. Significant. They actually called it significant. Then they left the rest of us to remember that it does not necessarily mean important, which is a fairly substantial misunderstanding to leave lying around in the name of the thing.

Significant. As if the universe, in its infinite and apparently quite bored wisdom, had finally decided that something deserved our attention. Something had happened. Something mattered. Perhaps even my puny, gasping, Calcutta-soaked existence might eventually amount to more than the maintenance of an organism that keeps needing things I cannot afford.

And what might this significant result be? Suppose a large, well-run study finds that a pill makes people fall asleep an average of twelve seconds faster than a placebo.

Twelve. Seconds.

I can lose more time than that trying to remember why I walked into a room. I can spend longer staring at my own reflection, wondering whose dumb idea it was to provide me with one. For someone who has trouble sleeping, twelve seconds is particularly insulting. I want to sleep. I want the night to stop being something I have to get through while waiting to become tired enough to get through the next day.

But sure. Significant. Break out the champagne and the tiny hats.

The difference could be real. That is what makes this more irritating, because now I cannot even dismiss the fucking thing. Measure carefully enough, with enough people, and a tiny effect can produce a very small p-value. You can be quite confident that something has changed and still have to ask why anybody should care. The calculation has not failed. We have simply expected it to do something it was never meant to do.

The ritual, in a great many papers, revolves around $0.05$. There are other cutoffs, but this is the one that has settled into people’s heads. A number we chose has become a border, a barbed wire fence stretched across human knowledge, separating the blessed from the damned with all the subtlety of a colonial partition.

Below $0.05$, we announce discovery. Above it, the embarrassed shuffling begins. There is a sudden, suspicious interest in the floor tiles.

Point oh four nine and the trumpets sound. Point oh five one and the result slouches home in defeat. Two nearly identical results, received as though one had brought us fire and the other had shat in the drinking water.

Nothing in nature has undergone that transformation. We have changed the label.

And I want to be clear about something, because clarity is one of the few virtues I can still attempt without first finding the money.

I want the line.

I know what is wrong with treating it this way, and I still want it. I need there to be a point at which the uncertainty stops, because uncertainty is the one thing I have in surplus. Whether there will be work. Whether the money will stretch. Whether I will sleep. Whether the thing I am worrying about is a problem I should act on or another evening spent exhausting myself over something I cannot resolve.

I have uncertainty at home. I came to you people because you had equipment.

You had beakers and centrifuges and machines that go beep in the night. I wanted a verdict. I wanted someone in a white coat to point at a chart and say, this matters, this is worth doing, you can stop going around and around this particular worry.

Instead I get, well, the result is statistically significant, but the benefit may be too small to matter clinically.

That is a perfectly reasonable statement. It is also a miserable thing to hear when you have been waiting for help. I am not angry that somebody measured the twelve seconds. I am angry at how easily those twelve seconds can be presented to somebody who has been awake for hours as though an answer has arrived.

I have worked as a statistician. I know what a p-value is supposed to tell us. Knowing does not make me immune to wanting more from it.

The p-value asks something fairly narrow: assuming the null hypothesis, often that there is no effect, and the other assumptions of the test are correct, how likely would the chosen test statistic be to take a value at least as extreme as the one we observed?

A useful question. It just does not tell me the probability that my theory is true. It does not tell me how much a treatment will help. It certainly does not establish that I am brilliant, although I can see how that last interpretation might become attractive after spending several years on something.

You want to know whether you are right. Whether your explanation holds up. Whether you have found something worth knowing. And here is a decimal answering a narrower question while everybody leans over it, trying to read their preferred answer into the space around it.

I understand the attraction of waving a small p-value in someone’s face like a crucifix at a vampire. There. A number. A calculation. Something that appears to have no interest in either of us and has nevertheless taken my side.

Except it has not taken my side. I have recruited it.

And once I have used it to win an argument, admitting what it cannot establish becomes considerably less enjoyable. Now I have to go back to somebody I dislike and explain that my confidence exceeded the information available to me.

I would much rather they discovered that independently, after I had left.

Because here is the ugly, squirming part. Science is supposed to let us find out what holds up. But researchers also have to get work published, and journals have often preferred a result that crosses the line. An honest, careful disappointment can have a harder time finding a home than something exciting with a little asterisk beside it.

So you run your experiment and do not get the result you hoped for. That happens. It should be something we can bear.

Instead you stare at the data the way a man stares at a toilet that has overflowed for the third time, trying to work out whether there is any interpretation under which this is going well.

You start looking.

You look at younger people. Older people. People who exercise. People who claim to exercise, which gives you a considerably larger group. You change the cutoff for age. You leave out a few awkward observations. You try another outcome. Somewhere in there, perhaps, are the left-handed men over forty who own dogs named Kevin.

There is nothing inherently wrong with exploring data. Looking can give you a good question for the next study. The trouble begins when you find one promising result among all those attempts and present it as though that had been the question all along.

You have asked and asked until something agreeable has come out, then concealed how much asking it took.

Torture the data long enough and it may confess to something. Waterboard it, deprive it of sleep, play motivational speeches at maximum volume. Eventually you get a whisper of significance, and you carry that whisper to a journal without bringing along the rest of the interrogation.

I understand this impulse too. That is what makes it uncomfortable. I would like to stand somewhere clean and condemn it, but I know the feeling of needing an answer badly enough that my need starts interfering with how I look for it.

With money as tight as it has been for me, I understand the urge to check a bank balance again when there is no particular reason to expect it to have changed.

Same account. Same money. Same me.

Just checking whether reality has reconsidered.

Refreshing a bank account is not p-hacking. You can keep pressing the button and the arithmetic will remain impressively unmoved by your distress. But I recognize the wish underneath it, the inability to leave an unwelcome answer alone.

There has to be something. Perhaps I have looked at it wrong. Perhaps there is another way.

Sometimes there is another way. That possibility is part of what keeps you there.

And the small study that misses the cutoff, what of that? It does not prove that nothing happened. A study may be too small or too noisy to distinguish an important effect from no effect.

It also does not prove that an important effect is hiding there, waiting for a bigger study to rescue it. The treatment may do very little. It may do nothing useful. We have to look at the estimated effect and the uncertainty around it, at what the study can reasonably rule out, rather than treating either side of the line as an excuse to stop thinking.

We are still uncertain.

I know that is an honest answer. I also know how badly it can land.

What have you been doing?

Reducing the uncertainty.

Well, reduce it faster. Some fellow on Facebook has already posted a conclusion, and people are sharing it, and he appears to have managed without the inconvenience of finding anything out.

It is difficult to compete with that. The careful person keeps having to explain what remains unknown. The confident one gets to finish the sentence.

And when you are frightened, when the question concerns somebody you love or a treatment you cannot easily afford to get wrong, a finished sentence can feel like help. You can know better and still want to hear it. I do not think that makes people stupid. I think it makes the responsibility of the person offering the answer rather more serious than we sometimes allow.

What I actually want to know is simple enough to ask and difficult enough to answer. How much did it help? How sure are you? What went wrong? What did it cost, including the unpleasant things that happened to the people taking it? Did somebody else, in another lab, with other people, find something similar?

If I am going to put something into my body, I would like to know what changed for the bodies you tested it on. I would like to know whether they noticed. Whether they felt better. Whether the benefit was worth what they had to put up with.

I want to know whether the people who read the reassuring headline have been given a fair account of what happened.

And that requires reading. Actual reading, with attention, and some grasp of what was measured, and enough patience to get past the sentence that appeared to settle everything.

I like reading. I still get tired. Being able to understand something does not mean I always have the energy to examine it as closely as I should. Sometimes I want somebody else to have done the careful work and to tell me, plainly, what it amounts to. That is not an unreasonable thing to want from a scientific paper.

But there is a little asterisk next to the number.

The asterisk is so small. The explanation is much longer. And I am tired enough for the difference to matter.

I want that star to mean somebody got better. I want it to mean the thing is worth trying. I want, for once, to get to the end of a result without finding another uncertainty I will have to carry around.

The number has made none of those promises. That part has come from me, helped along by anybody who found it useful to let me think so.

Just give me the star. I know there is more to read. I am sitting here knowing it.

P.S. Ronald Fisher helped popularize the five-percent cutoff in his 1925 book Statistical Methods for Research Workers, where he described taking that point as a limit as convenient. Convenient. Which is a bit like calling the guillotine a convenient haircut, once you have started using the line to decide which findings deserve to keep their heads.