A note from Teo
12 questions to ask before you believe a health claim
Was it tested in humans? Compared to what? Who paid? Twelve questions that take most health claims apart, in the order to ask them.
Most health claims that reach you have travelled a long way from the paper they started in. A scientist measures something small. A press office writes it up. A journalist shortens it. Someone selling something shortens it again. By the time it's in your feed, it's wearing a lab coat it never earned.
These are the twelve questions I ask of every claim, in the order I ask them. Most claims fall over by question four.
1. Was this tested in humans?
A lot of what gets called a discovery happened in a dish or in a mouse. That's where research starts, not where it ends. I wrote a note on why mice are not small people.
2. Who exactly was studied?
Even when it's people, it's some people. A famous 1989 trial of aspirin for preventing first heart attacks enrolled about 22,000 male doctors and no women. A trial in women reported in 2005 found a different picture. Same pill, different room.
3. Compared to what?
Better than nothing, better than a sugar pill and better than the usual treatment are three different claims. Find the comparison group.
4. How big was the effect, in absolute numbers?
"Half the risk" can mean one person in a thousand. Find the risk without it and the risk with it. This note shows how.
5. For how long?
A change that lasts six weeks is not a change that lasts six years. And a number without a time span is half a number.
6. One study or many?
One study is a lead, not a verdict. Look for what the other papers on the same question found.
7. Correlation, or a controlled trial?
"Linked to" means people were watched, not assigned. People who do one thing tend to differ in other ways too. A randomised trial is how you separate the thing from the people who choose it.
8. Who paid?
Funding doesn't make a result wrong. It's a reason to read the methods twice. The funding and conflicts of interest are listed at the end of the paper.
9. Has anyone failed to replicate it?
A result that only one team can find is a result waiting for a second opinion.
10. Does the headline match the paper's own conclusion?
Read the last paragraph of the paper. It's often more careful than everything written about it.
11. Is the person telling me this selling something?
A seller isn't automatically wrong. But the most famous health number on earth, 10,000 steps a day, traces back to a 1960s Japanese pedometer whose name meant "10,000 steps meter". When a team followed about 17,000 older women, the drop in deaths levelled off around 7,500 steps. A product name had gone unquestioned for fifty years.
12. What would change my mind?
Ask it of the claim, then of yourself. A real claim comes with a way to be wrong. When two Australian doctors reported in 1984 that a bacterium, not stress, was behind stomach ulcers, almost nobody believed them. The evidence won anyway, slowly, and they later shared a Nobel Prize. Good ideas can survive being tested. That's how you tell them apart.
The receipts
- Steering Committee of the Physicians' Health Study Research Group, New England Journal of Medicine, 1989.
- Ridker et al., New England Journal of Medicine, 2005.
- Lee et al., JAMA Internal Medicine, 2019.
- Marshall and Warren, The Lancet, 1984.
Get the whole method
The Receipts Checklist has all twelve questions with a real case for each, the paper to look up, the red flag that gives each one away, a glossary of fifteen sales phrases, and a one-page card for the fridge. If you'd rather have them on the wall, the first of The Evidence Posters is these twelve questions.
This is education about reading research, not medical advice. Take your questions to someone who can examine you.
