The Value of Direct Replication
Perspectives on Psychological SciencePublished 1 January 2014
Daniel J. Simons
Citations644
SJR quartileQ1
SJR score5.39
SNIP4.96
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This commentary argues that direct replication by multiple laboratories is the only way to verify the reliability of an effect and questions the value of direct replicationby other laboratories are problematic.
Abstract
Reproducibility is the cornerstone of science. If an effect is reliable, any competent researcher should be able to obtain it when using the same procedures with adequate statistical power. Two of the articles in this special section question the value of direct replication by other laboratories. In this commentary, I discuss the problematic implications of some of their assumptions and argue that direct replication by multiple laboratories is the only way to verify the reliability of an effect.
Keywords
PsychologyDecision SciencesArts and Humanities
Behavioral and Brain SciencesThe weirdest people in the world?
12,056 Citations2010Joseph Henrich, Steven J. Heine +1 more
Psychological ScienceFalse-Positive Psychology
6,712 Citations2011Joseph P. Simmons, Leif D. Nelson +1 more
It is shown that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings, flexibility in data collection, analysis, and reporting dramatically increases actual false- positive rates, and a simple, low-cost, and straightforwardly effective disclosure-based solution is suggested.
Psychological BulletinBelief in the law of small numbers.
3,229 Citations1971Amos Tversky, Daniel Kahneman
Psychological ScienceMeasuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
2,333 Citations2012Leslie K. John, George Loewenstein +1 more
It is found that the percentage of respondents who have engaged in questionable practices was surprisingly high, which suggests that some questionable practices may constitute the prevailing research norm.
PLoS ONEEvaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research
1,672 Citations2013Matthew J. C. Crump, John V. McDonnell +1 more
This paper replicates a diverse body of tasks from experimental psychology including the Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks using participants recruited using AMT.
European Journal of PersonalityRecommendations for Increasing Replicability in Psychology
848 Citations2013Jens B. Asendorpf, Mark Conner +14 more
PLoS ONE“Positive” Results Increase Down the Hierarchy of the Sciences
750 Citations2010Daniele Fanelli
These results support the scientific status of the social sciences against claims that they are completely subjective, by showing that, when they adopt a scientific approach to discovery, they differ from the natural sciences only by a matter of degree.
Journal of Experimental Social PsychologyThe Replication Recipe: What makes for a convincing replication?
704 Citations2013Mark J. Brandt, Hans IJzerman +8 more
Psychonomic Bulletin & ReviewIs the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments
702 Citations2012Laura Germine, Ken Nakayama +4 more
It is demonstrated that collecting data from uncompensated, anonymous, unsupervised, self-selected participants need not reduce data quality, even for demanding cognitive and perceptual experiments.
Perspectives on Psychological ScienceIs the Replicability Crisis Overblown? Three Arguments Examined
695 Citations2012Harold Pashler, Christine R. Harris
It is argued that there are no plausible concrete scenarios to back up such forecasts and that what is needed is not patience, but rather systematic reforms in scientific practice.
Perspectives on Psychological ScienceThe Alleged Crisis and the Illusion of Exact Replication
609 Citations2014Wolfgang Stroebe, Fritz Strack
It is proposed that for meaningful replications, attempts at reinstating the original circumstances are not sufficient and replicators must ascertain that conditions are realized that reflect the theoretical variable(s) manipulated (and/or measured) in the original study.
Perspectives on Psychological SciencePriming, Replication, and the Hardest Science
331 Citations2014Joseph Cesario
It is argued that expectations of priming effects to be highly sensitive to variations in experimental features and subject populations are misguided and inconsistent with evolutionary understandings of the brain as a computational organ.
Perspectives on Psychological ScienceScience or Art? How Aesthetic Standards Grease the Way Through the Publication Bottleneck but Undermine Science
294 Citations2012Roger Giner‐Sorolla
To open the bottleneck, putting structures in place to reward broader forms of information sharing beyond the exquisite art of present-day journal publication is suggested, suggesting a more palatable solution to the crisis in psychological research.
SSRN Electronic JournalLife after P-Hacking
200 Citations2013Joseph P. Simmons, Leif D. Nelson +1 more
How researchers' commitment to avoid p-hacking will affect their research lives is discussed and one conclusion is that most experimental research cannot be successful without at least 50 observations per condition.
Perspectives on Psychological ScienceAn Intersensory Interaction Account of Priming Effects—and Their Absence
37 Citations2014Roberta L. Klatzky, J. David Creswell
A general model of intersensory interaction is described, in which two or more sources of information provide an estimate or “bid” on a property of the world, with the perceptual outcome being a weighted combination of the bids.