Concept
Publication Bias
methodology behavioural-insights evidence
Academic journals strongly prefer to publish studies that find a clear, positive effect over studies that find no effect ("null results") — because a researcher's career, funding, and reputation depend on being seen as the driving force behind exciting new findings, not on reporting that an idea didn't work. Nina Mazar argues this creates a systematically distorted public record: search for published papers testing any given nudge (a default, a social-norm message) and nearly every result you find will show a positive effect, because the null results were never published — or the researchers who found them never bothered submitting them in the first place.
A satirical paper makes the underlying problem vivid. In 2018, Harvard Medical School's Robert Yeh published "Parachute use to prevent death and major trauma when jumping from an aircraft: randomized controlled trial" in the British Medical Journal, reporting that wearing a parachute made no significant difference to survival for 23 skydivers. Buried in the methods section: the plane was stationary on the ground at zero kilometres an hour, and jumpers fell half a metre. The paper was a deliberate satire of how sensationalized, out-of-context findings get shared without readers checking the fine print — the same dynamic, Nina Mazar argues, that lets an unreplicated positive result circulate as settled fact (see Yeh Parachute Satire Paper).
The Behavioural Insights Team's practice is the antidote: they publish every trial's results, whether positive, negative, or null, removing the incentive structure that produces publication bias in the first place. See BIT-Berkeley 165-Trial Nudge Meta-Analysis for the resulting comparison between BIT's own field data and the inflated effect sizes reported in the published academic literature on the same interventions — the Voltage Effect.
Discussed in
- Episode 144 — 144-i-didn-t-want-to-publish-this-episode