Replication

Replication is the process of repeating a study using similar or identical methods to determine whether the original findings can be reproduced. Direct replication uses the same methods, procedures, and conditions as the original study. Conceptual replication tests the same hypothesis or theory using different methods, populations, or settings to assess whether the findings generalize beyond the original context.

Replication is the cornerstone of scientific credibility. A finding that cannot be reproduced is of limited value, regardless of how statistically significant or theoretically appealing it appeared in the original study. The replication crisis in psychology, biomedical research, and other fields has revealed that a substantial proportion of published findings do not replicate, raising serious questions about the reliability of the evidence base. High profile replication projects have catalyzed reforms in research practice, including preregistration, open data, and larger sample sizes. In the Open Science Collaboration's replication of 100 psychology studies (published in 2015), 97% of the original studies had reported statistically significant results, but only 36% of the replications did, and the replication effects were on average about half the size of the original effects.

Replication is distinct from reproducibility in the narrow sense used by the US National Academies of Sciences, Engineering, and Medicine (2019): reproducibility means obtaining consistent results from the original data with the same code and analysis, whereas replicability means obtaining consistent results in a new study that collects its own data to answer the same question. Some fields use the two words the other way around, so it helps to say which is meant.

Direct replication involves following the original study's procedures as closely as possible, ideally using the same instruments, materials, populations, and analytical methods. The goal is to determine whether the original effect can be observed again when the study is repeated. A successful direct replication strengthens confidence in the original finding; a failure to replicate raises questions about the finding's reliability, the original study's methods, or both.

Conceptual replication tests the same underlying hypothesis using different operationalizations. For example, if the original study tested the effect of social exclusion on aggression using a specific laboratory paradigm, a conceptual replication might test the same relationship using a different measure of aggression or a different manipulation of social exclusion. Successful conceptual replication provides stronger evidence for the generality of the phenomenon than direct replication alone.

Several factors contribute to replication failures. Some original findings may be false positives, produced by small samples, flexible analyses, or publication bias. Others may be genuine but context dependent, failing to replicate when tested in different populations or settings. Methodological differences between the original and replication studies, even when unintentional, can also explain failures.

Registered replication reports, a form of Registered Report in which the replication protocol is peer reviewed and accepted for publication before data collection, help ensure that replication studies are conducted rigorously and published regardless of the outcome. Multisite replication studies, where the same protocol is executed at multiple independent laboratories, provide one of the strongest tests of a finding's reliability.

Prioritize replication when the original finding is influential (informing practice, policy, or theory), based on a small sample, produced using a flexible analytical approach, or derived from a single study without independent confirmation. Replication is particularly important when the original finding seems surprising, contradicts prior evidence, or has not been tested outside the original context. When designing a new study, consider incorporating replication of key prior findings as part of the study protocol.

A common error is dismissing a replication failure as solely attributable to methodological differences without seriously considering that the original finding may be unreliable. Conversely, interpreting a single replication failure as definitive evidence that the original finding is false overstates the evidential value of a single study. Researchers sometimes conduct underpowered replication studies that cannot detect the original effect even if it is real, producing inconclusive results. Finally, the scientific culture of undervaluing replication studies (viewing them as less innovative than original research) discourages researchers from investing in this essential scientific activity.

In a hypothetical case modeled on the elderly priming research, a highly cited psychology study reports that brief exposure to words related to elderly stereotypes causes college students to walk more slowly (N = 30 per condition). The effect is striking and widely reported in textbooks. A preregistered direct replication with 300 participants per condition, conducted across three university laboratories, finds no significant effect (Cohen's d = 0.03, 95% CI: negative 0.13 to 0.19). The replication team concludes that the original effect was likely a false positive, though they note that differences in participant demographics and cultural context cannot be entirely ruled out.