Ioannidis, "Why Most Published Research Findings Are False"

By the early 2000s, many clinicians and methodologists doubted the reliability of the published biomedical literature. Underpowered trials, outcome switching, and the near-universal preference for positive results had accumulated into a literature that many suspected was systematically skewed. John Ioannidis, then at the University of Ioannina in Greece and Tufts-New England Medical Center in Boston, set out to model how skewed, extending a calculation that Wacholder and colleagues had called the false positive report probability in 2004.

In a theoretical paper he wrote alone for PLoS Medicine in 2005, Ioannidis used a Bayesian framework to calculate the positive predictive value of a research finding under varying assumptions. His model incorporated prior probability of a true effect, study power, significance threshold, and what he called the bias factor. He showed that smaller studies, smaller effect sizes, greater analytic flexibility, lower pre-study odds, greater financial interests and prejudice, and crowded fields with many teams chasing the same hypothesis each reduced the probability that a positive finding reflected a true effect. His simulations suggested that for most study designs and settings a claimed finding was more likely false than true.

The paper presented no new clinical data. That July, Ioannidis had reported in JAMA that of 45 highly cited clinical studies claiming an effective intervention, 7 had been contradicted by later research and 7 more had found stronger effects than later studies did. Steven Goodman and Sander Greenland argued in 2007 that the model's assumptions, such as counting only whether p fell below 0.05, overstated the problem, and in 2014 Leah Jager and Jeffrey Leek put the false discovery rate in five leading medical journals at 14%. Europe PMC lists more than 5,000 articles citing the 2005 paper.

Reforms aimed at the same problems were already under way. In September 2004 the International Committee of Medical Journal Editors announced that its 11 journals would not consider trials that had not been registered in a public register, and the CONSORT statement on reporting randomized trials dated from 1996. In psychology, the Open Science Collaboration reported in 2015 that only 36% of 100 replication attempts produced statistically significant results, against 97% of the original studies. Preclinical biomedical research showed similar patterns when replication studies were organized at scale.

Ioannidis continued producing methodological critiques of biomedical research practices throughout the 2010s, examining issues from surrogate endpoints to bias in nutrition research. In April 2014 Stanford launched the Meta-Research Innovation Center (METRICS), led by Ioannidis and Steven Goodman, to study and improve research practices.