Reporting Bias

Reporting bias refers to systematic differences between reported and unreported research findings, encompassing both selective reporting of outcomes within individual studies and selective publication of entire studies based on the nature or direction of their results. It includes publication bias (the tendency to publish positive results), outcome reporting bias (the selective reporting of favorable outcomes within a study), and time lag bias (the faster publication of positive results).

Reporting bias distorts the available evidence by making the published literature an unrepresentative sample of all research conducted. When systematic reviews synthesize this biased literature, the resulting pooled estimates overestimate treatment effects and underrepresent adverse outcomes. Reporting bias has been documented across medical research: studies have shown that trials with positive results are published faster, in higher impact journals, and with greater frequency than trials with null or negative results. The selective reporting of outcomes within published studies further compounds the problem. Addressing reporting bias is essential for producing reliable evidence syntheses and making sound clinical decisions.

Publication bias operates at the study level. Studies with statistically significant results are more likely to be submitted for publication, accepted by journals, and cited by other researchers. Studies with null findings may languish as "file drawer" studies, never reaching the published record. This phenomenon has been confirmed empirically by following cohorts of registered clinical trials and comparing registration records against published results.

Outcome reporting bias operates within studies. Researchers may measure multiple outcomes but report only those that reached statistical significance. Comparing published results against registered trial protocols reveals that outcome switching (changing the primary outcome between registration and publication) and selective omission of negative outcomes are common. This bias is harder to detect than publication bias because it operates within the reported study rather than between published and unpublished studies.

Other forms of reporting bias include citation bias (studies with positive results being cited more frequently, amplifying their influence), language bias (findings being published in a particular language depending on their nature and direction, for example positive trials being more likely to appear in English language journals), and location bias (findings being published in journals that are easier or harder to access, or more or less well indexed in standard databases, depending on their nature and direction).

Strategies for detecting and mitigating reporting bias in systematic reviews include comparing published reports against trial registry entries (such as ClinicalTrials.gov), searching for grey literature and unpublished studies, using funnel plot analysis and statistical tests for asymmetry, and contacting study authors for unreported data. Reporting guidelines such as CONSORT ask authors to report all prespecified outcomes transparently, and trial registration mandates have improved but not eliminated the problem.

Consider reporting bias as a potential source of distortion in every systematic review and evidence synthesis. When rating the certainty of the evidence with GRADE, consider publication bias, one of its five domains for downgrading; selective reporting of results within studies is assessed as part of risk of bias, for example in the RoB 2 domain on selection of the reported result. Cochrane's ROB-ME tool assesses the risk of bias in a meta analysis when whole studies or particular results are missing because of their results. Search for unreported outcomes by comparing publications against trial registries. Discuss the potential impact of reporting bias on the review's conclusions in the limitations section. When conducting primary research, mitigate reporting bias by preregistering the study protocol, reporting all prespecified outcomes, and submitting manuscripts regardless of whether the results are positive or negative.

A common error is equating reporting bias with publication bias alone, ignoring the equally important problem of outcome reporting bias within studies. Another mistake is assuming that trial registration eliminates reporting bias; while registration enables detection, it does not prevent selective reporting. Researchers sometimes fail to search for unpublished studies or grey literature when conducting systematic reviews, accepting the limitations of the published evidence without attempting to supplement it. Finally, focusing on reporting bias detection without discussing its potential impact on the review's conclusions leaves readers without guidance on how to interpret the findings.

A team conducting a systematic review of a new antidepressant identifies 12 published trials, all reporting favorable outcomes. Checking ClinicalTrials.gov, they discover that 18 trials were registered, of which 6 remain unpublished three years after completion. They contact the investigators and obtain data from 4 of the 6 unpublished trials, all showing null results. Because the four unpublished trials carry about twice the weight of an average published trial, including them in the meta analysis reduces the pooled effect size from a standardized mean difference of 0.42 to 0.24, changing the interpretation from a small to moderate effect to a small one. Two unpublished trials remain unavailable, so some risk of bias due to missing evidence remains.