What does research reproducibility mean?
Science Translational MedicinePublished 1 June 2016
Steven N. Goodman, Daniele Fanelli, John P. A. Ioannidis
Citations1,317
SJR quartileQ1
SJR score6.72
SNIP2.66
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
An array of explicit and implicit definitions of reproducibility and related terminology are reviewed, and how to avoid potential misunderstandings when these terms are used as a surrogate for “truth” is discussed.
Abstract
The language and conceptual framework of "research reproducibility" are nonstandard and unsettled across the sciences. In this Perspective, we review an array of explicit and implicit definitions of reproducibility and related terminology, and discuss how to avoid potential misunderstandings when these terms are used as a surrogate for "truth."
Keywords
Computer ScienceDecision Sciences
ScienceEstimating the reproducibility of psychological science
8,673 Citations2015Alexander A. Aarts
A large-scale assessment suggests that experimental reproducibility in psychology leaves a lot to be desired, and correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
Psychological BulletinThe file drawer problem and tolerance for null results.
8,595 Citations1979Robert Rosenthal
Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed.
Ecological MonographsPseudoreplication and the Design of Ecological Field Experiments
8,292 Citations1984Stuart H. Hurlbert
Suggestions are offered to statisticians and editors of ecological journals as to how ecologists' under- standing of experimental design and statistics might be improved.
Nucleic Acids ResearchThe Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update
2,289 Citations2016Enis Afgan, Dannon Baker +18 more
Personality and Social Psychology ReviewHARKing: Hypothesizing After the Results are Known
2,058 Citations1998Norbert L. Kerr
It is conceded that the question of whether HARKing's costs exceed its benefits is a complex one that ought to be addressed through research, open discussion, and debate.
JAMAEmpirical Evidence for Selective Reporting of Outcomes in Randomized Trials
1,806 Citations2004An‐Wen Chan, Asbjørn Hróbjartsson +3 more
The reporting of trial outcomes is not only frequently incomplete but also biased and inconsistent with protocols and Published articles, as well as reviews that incorporate them, may therefore be unreliable and overestimate the benefits of an intervention.
EconometricaA Reality Check for Data Snooping
1,794 Citations2000Halbert White
The purpose here is to provide a straightforward procedure for testing the null hypothesis that the best model encountered in a specification search has no predictive superiority over a given benchmark model.
Journal of Experimental Psychology GeneralP-curve: A key to the file-drawer.
1,398 Citations2013Uri Simonsohn, Leif D. Nelson +1 more
By telling us whether the authors can rule out selective reporting as the sole explanation for a set of findings, p-curve offers a solution to the age-old inferential problems caused by file-drawers of failed studies and analyses.
NaturePolicy: NIH plans to enhance reproducibility
1,285 Citations2014Francis S. Collins, Lawrence A. Tabak
Francis S. Collins and Lawrence A. Tabak discuss initiatives that the US National Institutes of Health is exploring to restore the self-correcting nature of preclinical research.
Annals of Internal MedicineToward Evidence-Based Medical Statistics. 1: The <i>P</i> Value Fallacy
1,189 Citations1999Steven N. Goodman
The historical and logical foundations of the dominant school of medical statistics, sometimes referred to as frequentist statistics, are explored and the logical fallacy at the heart of this system is explicated, which maintains such a tenacious hold on the minds of investigators, policymakers, and journal editors.
The American StatisticianThe Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
1,182 Citations2006Andrew Gelman, Hal S. Stern
EconomicaSpecification Searches: Ad Hoc Inference with Non-Experimental Data.
1,136 Citations1981Jean‐François Richard, Edward E. Leamer
Circulation ResearchReproducibility in Science
1,051 Citations2014C. Glenn Begley, John P. A. Ioannidis
The reproducibility problems in basic and preclinical biomedical research are reviewed, some of the complexities are highlighted, and potential solutions that may help improve research quality and reproducible are discussed.
New England Journal of MedicineEffects of Adjuvant Tamoxifen and of Cytotoxic Therapy on Mortality in Early Breast Cancer
934 Citations1988Early Breast Cancer Trialists' Collaborative Group
This overview was able to demonstrate particularly clearly that both tamoxifen and cytotoxic therapy can reduce five-year mortality, and showed that combination chemotherapy was significantly more effective than single-agent therapy.
Annals of Internal MedicineToward Evidence-Based Medical Statistics. 2: The Bayes Factor
934 Citations1999Steven N. Goodman
Statistics in MedicineDetecting and describing heterogeneity in meta-analysis
722 Citations1998Rebecca Hardy, Simon G. Thompson
It is concluded that the test of heterogeneity should not be the sole determinant of model choice in meta-analysis, and inspection of relevant normal plots, as well as clinical insight, may be more relevant to both the investigation and modelling of heterogeneity.
ScienceComment on “Estimating the reproducibility of psychological science”
500 Citations2016Daniel T. Gilbert, Gary King +2 more
It is shown that this article contains three statistical errors and provides no support for the conclusion that the reproducibility of psychological science is surprisingly low, and that the data are consistent with the opposite conclusion.
Statistics in MedicineA comment on replication,<i>P</i>‐values and evidence
307 Citations1992Steven N. Goodman
It is shown that if the observed difference is the true one, the probability of repeating a statistically significant result, the 'replication probability', is substantially lower than expected.
American Journal of EpidemiologyReproducible Epidemiologic Research
300 Citations2006Roger D. Peng, Francesca Dominici +1 more
A standard for reproducibility is outlined and methods for reproducible research are proposed and implemented by use of a case study in air pollution and health.
Electronic documents give reproducible research a new meaning
224 Citations1992Jon F. Claerbout, Martin Karrenbach
A revolution in education and technology transfer follows from the marriage of word processing and software command scripts, and anuthor attaches to every figure caption a pushbutton or a name tag usable t,o recalculate the figure from all its data, parameters, and programs.
Journal of Economic SurveysTHE MEANING OF FAILED REPLICATIONS: A REVIEW AND PROPOSAL
209 Citations2015Michael A. Clemens
Psychonomic Bulletin & ReviewA Bayes factor meta-analysis of Bem’s ESP claim
201 Citations2011Jeffrey N. Rouder, Richard D. Morey
A meta-analytic Bayes factor is developed that describes how researchers should update their prior beliefs about the odds of hypotheses in light of data across several experiments, and finds that the evidence that people can feel the future with neutral and erotic stimuli to be slight.
American Journal of Clinical NutritionIs everything we eat associated with cancer? A systematic cookbook review
200 Citations2012Jonathan D. Schoenfeld, John P. A. Ioannidis
PLoS ONEBibliometric Evidence for a Hierarchy of the Sciences
193 Citations2013Daniele Fanelli, Wolfgang Glänzel
A “gradualist” view of scientific knowledge is suggested, suggesting that the Hierarchy of the Sciences provides the best rational framework to understand disciplines' diversity, by sampling nearly 29,000 papers published contemporaneously in 12 disciplines and measuring a set of parameters hypothesised to reflect theoretical and methodological consensus.
The American Statistician<i>P</i>-Value Precision and Reproducibility
169 Citations2011Dennis D. Boos, Leonard A. Stefanski
It is shown that p-values exhibit surprisingly large variability in typical data situations, and the use of *, **, and *** to denote levels 0.05, 0.01, and 0.001 of statistical significance in subject-matter journals is about the right level of precision for reporting p- values when judged by widely accepted rules for rounding statistical estimates.
Journal of Economic SurveysWHAT'S TO KNOW ABOUT THE CREDIBILITY OF EMPIRICAL ECONOMICS?
121 Citations2013John P. A. Ioannidis, Hristos Doucouliagos
JNCI Journal of the National Cancer InstituteMultiplicities in Cancer Research: Ubiquitous and Necessary Evils
52 Citations2012Donald A. Berry
The goal of this commentary is to elucidate the issues and provide recommendations for conducting and reporting results of empirical studies, with emphasis on the problems of multiple comparisons and other types of multiplicities, including what I call "silentmultiplicities."
Nature MethodsSolutions for quantifying P-value uncertainty and replication power
34 Citations2016Laura C. Lazzeroni, Ying Lü +1 more
This third type of interval allows an empirical assessment of power based on a P value alone, and can be used for a separate analysis of the follow-up or replication sample, and also for a ‘meta-analysis’ combining both samples.
Journal of VirologyCoagulation Factor Binding Orientation and Dimerization May Influence Infectivity of Adenovirus-Coagulation Factor Complexes
32 Citations2013Eric E. Irons, Justin W. Flatt +5 more
Dimmerization of FVII bound to Ad may be a contributing mechanistic factor for the differential infectivity of Ad-FX and Ad-FVII complexes, despite high-affinity binding of both these coagulation factors to the virus.
American Society of Clinical Oncology Educational BookAn Unappreciated Challenge to Oncology Drug Discovery: Pitfalls in Preclinical Research
19 Citations2013C. Glenn Begley
American Society of Clinical Oncology Educational BookAn Unappreciated Challenge to Oncology Drug Discovery: Pitfalls in Preclinical Research
13 Citations2013C. Glenn Begley
Physicians and scientists should view preclinical findings that lack rigor and robustness with skepticism and should proceed very cautiously in applying such findings to the clinic.
Journal of Radioanalytical and Nuclear ChemistryA cold fusion chronology
11 Citations1991Bruce V. Lewenstein, Walter Baur
PLoS ONEReplication Attempt: “Effect of BMAP-28 Antimicrobial Peptides on Leishmania Major Promastigote and Amastigote Growth: Role of Leishmanolysin in Parasite Survival”
6 Citations2014Elizabeth Iorns, William Gunn +5 more
It is observed that L-BMAP-28, as well as its D and RI isomers, demonstrate anti-leishmanial activity against L. major promastigotes in vitro, and the inhibitory effect was lower than what was seen in the original study.