A four-year study published in Science Magazine yesterday, August 27, 2015, found that fewer than half of 100 published findings of three prominent psychology journals were reproducible. The 270 researchers who collaborated on the Reproducibility Project: Psychology coordinated by the Center for Open Science (COS)—one of ARL’s three partners in SHARE—were able to replicate the findings of only 39 of the 100 original studies. Reproducibility is necessary for scientific evidence to be credible.
A failure to reproduce scientific results does not necessarily mean the original report was incorrect or fraudulent. The Reproducibility Project team noted that, although the most of the studies were repeated in collaboration with the original authors, small differences in when, where, or how the replication was carried out could influence the results. A researcher might conduct a study multiple times but only publish the successful result. “False positives” can also be caused by using too few subjects or by researcher bias.
Many organizations, funders, journals, and publishers are already working on improving scientific reproducibility by encouraging open sharing of research materials and data. One such effort was the creation of the Transparency and Openness Promotion (TOP) Guidelines published in Science in June of this year and supported by ARL. Since the Reproducibility Project began in 2011, similar projects have emerged in other fields such as the Reproducibility Project: Cancer Biology. Also, a discipline of metascience—scientific research about scientific research— is emerging. These initiatives should strengthen science in the long run.
For more information about the Reproducibility Project: Psychology, see:
Tom Bartlett, “The Results of the Reproducibility Project Are In. They’re Not Good.” Chronicle of Higher Education, August 28, 2015, http://chronicle.com/article/The-Results-of-the/232695/.
Benedict Carey, “Many Psychology Findings Not as Strong as Claimed, Study Says,” New York Times, August 27, 2015, http://www.nytimes.com/2015/08/28/science/many-social-science-findings-not-as-strong-as-claimed-study-says.html?_r=0.
Center for Open Science, “Massive Collaboration Testing Reproducibility of Psychology Studies Publishes Findings,” press release, August 27, 2015, http://centerforopenscience.org/pr/2015-08-27/.
Science Friday, “Putting Scientific Research to the Test,” to air on public radio stations in the US on August 28, 2015, 2:00–4:00 p.m. EDT, http://www.sciencefriday.com/segment/08/28/2015/putting-scientific-research-to-the-test.html.
