Grey Literature

Grey literature refers to research and information produced by organizations outside of traditional commercial and academic publishing channels. It includes government reports, policy documents, working papers, conference proceedings and abstracts, dissertations and theses, technical reports, white papers, preprints, and institutional publications. These materials are usually not subject to journal peer review, although some undergo other forms of review, and they often contain valuable data and findings that do not appear elsewhere. The widely cited Luxembourg definition (agreed at the Third International Conference on Grey Literature in 1997 and clarified in 2004) describes grey literature as material produced on all levels of government, academics, business and industry in print and electronic formats but not controlled by commercial publishers, that is, where publishing is not the primary activity of the producing body; the 2010 Prague definition adds that it is protected by intellectual property rights and of sufficient quality to be collected and preserved by libraries or institutional repositories.

Relying solely on peer reviewed journal articles for evidence synthesis introduces a systematic bias. Studies with statistically significant or positive results are more likely to be published in journals, while studies with null or negative results often remain in the grey literature (or go unreported entirely). Including grey literature in systematic reviews and meta analyses helps reduce this publication bias and provides a more complete picture of the evidence. Grey literature is also important because it often contains the most current data (government surveillance reports, pre prints), the most policy relevant analyses (health technology assessments, economic evaluations), and data from populations or settings underrepresented in the published literature. For researchers in global health, public policy, and social sciences, grey literature is an important source of evidence.

Identifying grey literature requires searching beyond standard bibliographic databases (such as PubMed and Scopus). Sources include government websites and repositories (such as the WHO Institutional Repository, the US Government Accountability Office), conference proceedings databases (such as Conference Proceedings Citation Index), dissertation databases (such as ProQuest Dissertations and Theses), trial registries (such as ClinicalTrials.gov), preprint servers (such as medRxiv, bioRxiv, SSRN), institutional repositories, and direct contact with experts and organizations in the field.

In a systematic review, the search strategy should explicitly state which grey literature sources were searched and how. Grey literature documents undergo the same screening and eligibility assessment as journal articles. However, quality assessment can be more challenging because grey literature may lack the structured reporting (methods sections, statistical details) that peer reviewed articles provide. Some grey literature may have been reviewed through institutional or governmental review processes that, while different from journal peer review, provide quality assurance.

For Cochrane reviews of interventions, the Handbook's conduct standards (MECIR) rate searching relevant grey literature sources, such as reports, dissertations, theses and conference abstracts, as highly desirable, and make searching trial registers mandatory where relevant to the topic. PRISMA 2020, a reporting guideline, asks authors to specify all databases, registers, websites, organizations, reference lists and other sources searched or consulted to identify studies. Systematic reviews that exclude grey literature should acknowledge this limitation and discuss how it might affect the results.

Grey literature should be considered in systematic reviews, and searching it is particularly important when publication bias is a concern, when the topic involves policy relevant evidence (which is often published by governments and NGOs rather than in journals), when the field is rapidly evolving (preprints provide the most current evidence), or when the review covers a topic with a limited journal literature (such as rare diseases or emerging health threats). Researchers conducting scoping reviews, evidence maps, and health technology assessments also routinely include grey literature.

A common error is excluding grey literature from systematic reviews without justification, which can introduce publication bias. Another mistake is failing to apply the same rigorous screening and quality assessment criteria to grey literature that are applied to published studies. Some researchers search only one or two grey literature sources, when a comprehensive search requires consulting multiple sources tailored to the topic. Treating all grey literature as low quality is incorrect; government statistical reports and health technology assessments, for example, may undergo rigorous internal review processes. Conversely, accepting grey literature uncritically without assessing its quality is also problematic. Finally, some researchers include grey literature in their search but do not document the search strategy, making the review difficult to reproduce.

A systematic review of interventions to reduce homelessness identifies 45 eligible studies: 28 from peer reviewed journals and 17 from grey literature sources (6 government evaluation reports, 4 working papers from research institutes, 3 conference presentations, 2 dissertations, and 2 organizational reports). A sensitivity analysis comparing the pooled effect with and without grey literature studies reveals that the journal only estimate is 0.35 (standardized mean difference), while including grey literature produces a smaller estimate of 0.27, consistent with publication bias inflating the journal only estimate. The review authors note that excluding grey literature would have overstated the pooled estimate, although publication bias is only one possible explanation for the difference.