Maximizing the Reproducibility of Your Research
Published 15 August 2016Open access
Frank A. Bosco, Joshua Carp, James G. Field, Hans IJzerman, Melissa Lewis, Marcus R. Munafò
Citations17
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
Open Science Collaboration (in press). Maximizing the reproducibility of your research. In S. O. Lilienfeld & I. D. Waldman (Eds.), Psychological Science Under Scrutiny: Recent Challenges and Proposed Solutions. New York, NY: Wiley.
Keywords
Decision Sciences
ScienceEstimating the reproducibility of psychological science
8,673 Citations2015Alexander A. Aarts
A large-scale assessment suggests that experimental reproducibility in psychology leaves a lot to be desired, and correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
Psychological BulletinThe file drawer problem and tolerance for null results.
8,595 Citations1979Robert Rosenthal
Quantitative procedures for computing the tolerance for filed and future null results are reported and illustrated, and the implications are discussed.
Psychological BulletinThe case for motivated reasoning.
8,180 Citations1990Ziva Kunda
It is proposed that motivation may affect reasoning through reliance on a biased set of cognitive processes--that is, strategies for accessing, constructing, and evaluating beliefs--that are considered most likely to yield the desired conclusion.
Nature reviews. NeurosciencePower failure: why small sample size undermines the reliability of neuroscience
7,878 Citations2013Katherine S. Button, John P. A. Ioannidis +5 more
It is shown that the average statistical power of studies in the neurosciences is very low, and the consequences include overestimates of effect size and low reproducibility of results.
Psychological ScienceFalse-Positive Psychology
6,712 Citations2011Joseph P. Simmons, Leif D. Nelson +1 more
It is shown that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings, flexibility in data collection, analysis, and reporting dramatically increases actual false- positive rates, and a simple, low-cost, and straightforwardly effective disclosure-based solution is suggested.
Journal of Pharmacology and PharmacotherapeuticsImproving bioscience research reporting: The ARRIVE guidelines for reporting animal research
6,348 Citations2010Carol Kilkenny, William J. Browne +3 more
1.2 Provide an accurate summary of the background, research objectives including details of the species or strain of animal used, key methods, principal findings, and conclusions of the study.
NatureRaise standards for preclinical cancer research
2,930 Citations2012C. Glenn Begley, Lee M. Ellis
C. Glenn Begley and Lee M. Ellis propose how methods, publications and incentives must change if patients are to benefit.
SciencePromoting an open research culture
2,761 Citations2015Brian A. Nosek, George Alter +37 more
Author guidelines for journals could help to promote transparency, openness, and reproducibility in science.
Journal of Personality and Social PsychologyAutomaticity of social behavior: Direct effects of trait construct and stereotype activation on action.
2,659 Citations1996John A. Bargh, Mark Chen +1 more
Psychological ScienceMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
2,333 Citations2012Leslie K. John, George Loewenstein +1 more
It is found that the percentage of respondents who have engaged in questionable practices was surprisingly high, which suggests that some questionable practices may constitute the prevailing research norm.
Journal of Clinical EpidemiologyCONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials
1,802 Citations2010David Moher, Sally Hopewell +7 more
Exploratory Research in the Social Sciences
1,671 Citations2001Robert A. Stebbins
Perspectives on Psychological ScienceScientific Utopia
1,336 Citations2012Brian A. Nosek, Jeffrey R. Spies +1 more
Strategies for improving scientific practices and knowledge accumulation are developed that account for ordinary human motivations and biases and can reduce the persistence of false findings.
Social PsychologyInvestigating Variation in Replicability
1,253 Citations2014Richard Klein, Kate A. Ratliff +49 more
Perspectives on Psychological ScienceAn Agenda for Purely Confirmatory Research
1,136 Citations2012Eric‐Jan Wagenmakers, Ruud Wetzels +3 more
This article proposes that researchers preregister their studies and indicate in advance the analyses they intend to conduct, and proposes that only these analyses deserve the label “confirmatory,” and only for these analyses are the common statistical tests valid.
Psychological BulletinConsequences of prejudice against the null hypothesis.
1,054 Citations1975Anthony G. Greenwald
Psychological ReportsWhy Summaries of Research on Psychological Theories are Often Uninterpretable
864 Citations1990Paul E. Meehl
Perspectives on Psychological ScienceThe Rules of the Game Called Psychological Science
853 Citations2012Marjan Bakker, Annette van Dijk +1 more
This paper considers 13 meta-analyses covering 281 primary studies in various fields of psychology and finds indications of biases and/or an excess of significant results in seven, highlighting the need for sufficiently powerful replications and changes in journal policies.
Perspectives on Psychological ScienceReplications in Psychology Research
783 Citations2012Matthew C. Makel, Jonathan A. Plucker +1 more
It was found that the majority of replications in psychology journals reported similar findings to their original studies (i.e., they were successful replications), however, replications were significantly less likely to be successful when there was no overlap in authorship between the original and replicating articles.
Psychological BulletinDo studies of statistical power have an effect on the power of studies?
750 Citations1989Peter Sedlmeier, Gerd Gigerenzer
PLoS BiologyBadges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency
749 Citations2016Mallory C. Kidwell, Ljiljana B. Lazarević +11 more
When badges were earned, reportedly available data were more likely to be actually available, correct, usable, and complete than when badges were not earned, and there was no change over time in the low rates of data sharing among comparison journals.
AnimalsImproving Bioscience Research Reporting: The ARRIVE Guidelines for Reporting Animal Research
713 Citations2014Carol Kilkenny, William J. Browne +3 more
A wealth of evidence shows that across many areas, the reporting of biomedical research is often inadequate, leading to the view that even if the science is sound, in many cases the publications themselves are not “fit for purpose”.
Journal of Experimental Social PsychologyThe Replication Recipe: What makes for a convincing replication?
704 Citations2013Mark J. Brandt, Hans IJzerman +8 more
Perspectives on Psychological ScienceAn Open, Large-Scale, Collaborative Effort to Estimate the Reproducibility of Psychological Science
630 Citations2012Open Science Collaboration
The Reproducibility Project is an open, large-scale, collaborative effort to systematically examine the rate and predictors of reproducibility in psychological science.
PeerJData reuse and the open data citation advantage
598 Citations2013Heather Piwowar, Todd Vision
There is a direct effect of third-party data reuse that persists for years beyond the time when researchers have published most of the papers reusing their own data, and a robust citation benefit from open data is found, although a smaller one than previously reported.
SciencePromoting Transparency in Social Science Research
566 Citations2014Edward Miguel, Colin F. Camerer +17 more
There is growing appreciation for the advantages of experimentation in the social sciences, and changes have been particularly pronounced in development economics, where hundreds of randomized trials have been carried out over the last decade.
NeuroImageGuidelines for reporting an fMRI study
563 Citations2008Russell A. Poldrack, Paul C. Fletcher +4 more
A set of guidelines for the reporting of methods and results in functional magnetic resonance imaging studies are outlined and a checklist to assist authors in preparing manuscripts that meet these guidelines are provided.
American PsychologistThe poor availability of psychological research data for reanalysis.
559 Citations2006Jelte M. Wicherts, Denny Borsboom +2 more
A failed attempt to obtain data reported in 141 empirical articles recently published by the APA, to assess the robustness of the research findings to outliers.
Advances in experimental social psychologySocially Situated Cognition: Cognition in its Social Context
511 Citations2004Eliot R. Smith, Gün R. Semin
The overall goals are to make social psychology part of the interdisciplinary integration emerging around the concept of situated cognition, and to advance these four themes as high-level conceptual principles that can organize seemingly disparate areas of research and theory within social psychology itself.
Current BiologyThe Availability of Research Data Declines Rapidly with Article Age
504 Citations2013Tim Vines, Arianne Albert +8 more
The results reinforce the notion that, in the long term, research data cannot be reliably preserved by individual researchers, and further demonstrate the urgent need for policies mandating data sharing via public archives.
NeuroImageThe secret lives of experiments: Methods reporting in the fMRI literature
495 Citations2012Joshua Carp
Significant gaps in methods reporting among fMRI studies are documented, and improved methodological descriptions in research reports would yield significant benefits for the field.
Frontiers in NeuroscienceOn the Plurality of (Methodological) Worlds: Estimating the Analytic Flexibility of fMRI Experiments
418 Citations2012Joshua Carp
Estimating the flexibility of neuroimaging analysis by submitting a single event-related fMRI experiment to a large number of unique analysis procedures reveals considerable flexibility, which suggests that false positive results may be more prevalent than expected in the literature.
The LancetClinical trial registration: a statement from the International Committee of Medical Journal Editors
411 Citations2004Catherine De Angelis, Jeffrey M. Drazen +11 more
Perspectives on Psychological ScienceTeaching Replication
179 Citations2012Michael C. Frank, Rebecca Saxe
It is argued that students in laboratory classes should replicate recent findings as part of their training in experimental methods, and it provides object lessons about the scientific process, the importance of reporting standards, and the value of openness.
Personnel PsychologyREVISITING THE FILE DRAWER PROBLEM IN META‐ANALYSIS: AN ASSESSMENT OF PUBLISHED AND NONPUBLISHED CORRELATION MATRICES
150 Citations2012Dan R. Dalton, Herman Aguinis +3 more
Perspectives on Psychological ScienceLow Hopes, High Expectations
142 Citations2012Olivier Klein, Stéphane Doyen +5 more
A content analysis of the method section of experiments reported in two psychology journals is reported, focusing on aspects of the procedure associated with experimenter bias and demand characteristics.
IEEE Transactions on Software EngineeringThe Impact of Design and Code Reviews on Software Quality: An Empirical Study Based on PSP Data
137 Citations2009Chris F. Kemerer, Mark C. Paulk
The results show that the PSP review rate is a significant factor affecting defect removal effectiveness, even after accounting for developer ability and other significant process variables.
Industrial and Organizational PsychologyHow Trustworthy Is the Scientific Literature in Industrial and Organizational Psychology?
126 Citations2013Sven Kepes, Michael A. McDaniel
Automated Defect Prevention: Best Practices in Software Management
121 Citations2007Dorota Huizinga, Adam Kolawa
Perspectives on Psychological SciencePsychDisclosure.org
108 Citations2013Etienne P. LeBel, Denny Borsboom +5 more
Personality and Social Psychology BulletinReporting and Interpreting Research in PSPB: Practices, Principles, and Pragmatics
94 Citations2009Deborah A. Kashy, M. Brent Donnellan +2 more
Perspectives on Psychological ScienceHarnessing the Undiscovered Resource of Student Research Projects
71 Citations2012Jon Grahe, Alan Reifman +5 more
It is suggested that undergraduate research can help advance the science of psychology and a hypothetical “question-list paradigm” is introduced as a mechanism to do this.
Perspectives on Psychological SciencePsychologists Are Open to Change, yet Wary of Rules
53 Citations2012Heather M. Fuchs, Mirjam A. Jenny +1 more
The proposal made by Simmons, Nelson & Simonsohn (2011) is a starting point for standards for data collection and reporting practices and it is found that psychologists are generally open to change.
Open MINDCollaborative Replications and Education Project (CREP)
40 Citations2013Grahe, Jon, Brandt, Mark +30 more
American Journal of EpidemiologyResearch Guidelines in the Era of Large-scale Collaborations: An Analysis of Genome-wide Association Study Consortia
32 Citations2012Melissa A. Austin, Marilyn S. Hair +1 more