Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography
International Journal of EpidemiologyPublished 30 April 2007Open access
Saskia C. Sanderson, ID Tatt, Julian P. T. Higgins
Citations1,521
SJR quartileQ1
SJR score2.69
SNIP2.48
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A number of useful assessment tools have been identified and a need to agree on critical elements for assessing susceptibility to bias in observational epidemiology and to develop appropriate evaluation tools.
Abstract
A number of useful assessment tools have been identified by this report. Tools should be rigorously developed, evidence-based, valid, reliable and easy to use. There is a need to agree on critical elements for assessing susceptibility to bias in observational epidemiology and to develop appropriate evaluation tools.
Keywords
Decision SciencesHealth ProfessionsEconomics, Econometrics and Finance
JAMAMeta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
20,682 Citations2000Donna F. Stroup
A checklist contains specifications for reporting of meta-analyses of observational studies in epidemiology, including background, search strategy, methods, results, discussion, and conclusion should improve the usefulness ofMeta-an analyses for authors, reviewers, editors, readers, and decision makers.
Journal of Epidemiology & Community HealthThe feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions.
8,257 Citations1998Sara H. Downs, Nicholas Black
It is shown that it is feasible to develop a checklist that can be used to assess the methodological quality not only of randomised controlled trials but also non-randomised studies and it is possible to produce a Checklist that provides a profile of the paper, alerting reviewers to its particular methodological strengths and weaknesses.
The LancetImproving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement
4,808 Citations1999David Moher, Fadi Hammal +4 more
This report hopes this report will generate further thought about ways to improve the quality of reports of meta-analyses of RCTs and that interested readers, reviewers, researchers, and editors will use the QUOROM statement and generate ideas for its improvement.
BMC Medical Research MethodologyThe development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews
3,624 Citations2003Penny Whiting, Anne WS Rutjes +3 more
BMJAssessing the quality of controlled clinical trials
3,334 Citations2001Peter Jüni, Douglas G Altman +1 more
The concept of study quality and the methods used to assess quality are discussed and the methodology for both the assessment of quality and its incorporation into systematic reviews and meta-analysis is discussed.
JAMAUsers' guides to the medical literature
2,981 Citations1993Gordon Guyatt
A 55-year-old man who had his serum cholesterol level measured at a shopping mall 2 months ago has an elevated cholesterol level, but before deciding on a treatment recommendation, you elect to find out just how big a reduction in the risk of CHD this patient could expect from a cholesterollowering diet or drug therapy.
Health Technology AssessmentEvaluating non-randomised intervention studies
2,926 Citations2003Jonathan J Deeks, Jacqueline Dinnes +6 more
The inability of case-mix adjustment methods to compensate for selection bias and the inability to identify non- randomised studies that are free of selection bias indicate that non-randomised studies should only be undertaken when RCTs are infeasible or unethical.
JAMACognitive and Behavioral Outcomes of School-Aged Children Who Were Born Preterm
2,394 Citations2002Adnan Bhutta, Mario A. Cleves +3 more
Children who were born preterm are at risk for reduced cognitive test scores and their immaturity at birth is directly proportional to the mean cognitive scores at school age.
JAMAThe Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
1,983 Citations1999Peter Jüni
The data indicate that the use of summary scores to identify trials of high quality is problematic, and relevant methodological aspects should be assessed individually and their influence on effect sizes explored.
BMJLow grade inflammation and coronary heart disease: prospective study and updated meta-analyses
1,688 Citations2000John Danesh
Findings suggest that some inflammatory processes, unrelated to the chronic infections studied here, are likely to be involved in coronary heart disease.
American Journal of EpidemiologyA META-ANALYSIS OF PHYSICAL ACTIVITY IN THE PREVENTION OF CORONARY HEART DISEASE
1,402 Citations1990Jesse A. Berlin, Graham A. Colditz
The authors apply the techniques of meta-analysis to data extracted from the published literature by Powell et al. to make formal quantitative statements and to explore features of study design that influence the observed relation between physical activity and coronary heart disease risk.
Controlled Clinical TrialsAssessing the quality of randomized controlled trials: An annotated bibliography of scales and checklists
1,304 Citations1995David Moher, Alejandro R. Jadad +4 more
An annotated bibliography of scales and checklists developed to assess quality is presented, giving readers a quantitative index of the likelihood that the reported methodology and results are free of bias.
Occupational and Environmental MedicineOccupational risk factors for shoulder pain: a systematic review
575 Citations2000Daniëlle van der Windt, Elaine Thomas +5 more
It seems likely that shoulder pain is the result of many factors, including physical load and the psychosocial work environment, and future longitudinal research should evaluate the relative importance of each individual risk factor and the role of potential confounding variables to set priorities for the prevention of shoulder pain in occupational settings.
PubMedCritical appraisal of the health research literature: prevalence or incidence of a health problem.
550 Citations1998Patricia Loney, L W Chambers +3 more
Criteria that can be used by health professionals to critically appraise research articles that estimate the prevalence or incidence of a disease or health problem are identified and discussed.
PainThe clinical course and prognostic factors of non-specific neck pain: a systematic review
530 Citations1998J.A.J. Borghouts, Bart W. Koes +1 more
Bearing in mind the limited number of studies and the low methodological quality, there are some indications that the localization (radiation to the arms/neurologic signs) and radiologic findings (degenerative changes in the discs and joints) are not associated with a worse prognosis.
Scandinavian Journal of Work Environment & HealthPhysical risk factors for neck pain
520 Citations2000Geertje AM Ariëns, Willem van Mechelen +3 more
It was concluded that there is some evidence for a positive relationship between neck pain and the following work-related risk factors: neck flexion, arm force, arm posture, duration of sitting, twisting or bending of the trunk, hand-arm vibration, and workplace design.
American Journal of Preventive MedicineData collection instrument and procedure for systematic reviews in the guide to community preventive services11The names and affiliations of the Task Force members are listed on page v of this supplement and at http://www.thecommunityguide.org
464 Citations2000Stephanie Zaza, Linda K. Wright-De Agüero +9 more
The form provides a structured format that researchers and others can use to review the content and quality of papers, conduct systematic reviews, or develop manuscripts and balances flexibility for the evaluation of papers with different study designs and intervention types.
JAMAQuality of Reporting of Randomized Trials as a Measure of Methodologic Quality
400 Citations2002Karin Huwiler-Müntener
Similar quality of reporting may hide important differences in methodologic quality, and well-conducted trials may be reported badly, so a clear distinction should be made between these 2 dimensions of the quality of RCTs.
BMC Medical Research MethodologyA systematic review of the content of critical appraisal tools
391 Citations2004Persis Katrak, Andrea Bialocerkowski +3 more
AddictionExploring the dose‐response relationship between alcohol consumption and the risk of several alcohol‐related conditions: a meta‐analysis
389 Citations1999Giovanni Corrao, Vincenzo Bagnardi +2 more
There is a great need for well-conducted epidemiological studies performed in several countries, to examine the dose-response relationship between alcohol intake and the risk of several alcohol-related conditions, as well as the role of drinking pattern in determining the risk.
JAMAA Meta-analysis of Alcohol Consumption in Relation to Risk of Breast Cancer
370 Citations1988Matthew P. Longnecker
There was strong evidence to support a dose-response relation in both the case-control and follow-up epidemiologic data, and this is interpreted as strongly supportive of an association between alcohol consumption and risk of breast cancer.
BMJCritical appraisal of published research: introductory guidelines.
367 Citations1991F.G.R. Fowkes, Patrick Fulton
Getting to grips with the details of papers and conducting a critical appraisal of the research may be a task many find difficult because there are not research experts and junior doctors have not had training in the appraisal of papers during medical education.
BiostatisticsOn the bias produced by quality scores in meta-analysis, and a hierarchical view of proposed solutions
357 Citations2001Sander Greenland
It is argued that response-surface estimation based on quality items is preferable to quality weighting, because it is unlikely that a low-dimensional appraisal will ever be adequate (especially over different applications), and because quality scores are often used to contrast, model, or modify meta-analysis results.
Journal of Clinical EpidemiologyRating the quality of evidence for clinical practice guidelines
334 Citations1996David C. Hadorn, David W. Baker +2 more
The system for rating the quality of medical evidence developed and used during creation of the Agency for Health Care Policy and Research-sponsored heart failure guideline is described.
American Journal of Industrial MedicineWorkplace ergonomic factors and the development of musculoskeletal disorders of the neck and upper limbs: A meta‐analysis
322 Citations1991Susan Stock
When the results of these studies are compared and aggregated where appropriate, they provide strong evidence of a causal relationship between repetitive, forceful work and the development of musculoskeletal disorders of the tendons and tendon sheaths in the hands and wrists and nerve entrapment of the median nerve at the carpal tunnel.
Annals of Internal MedicineManuscript Quality before and after Peer Review and Editing at <i>Annals of Internal Medicine</i>
320 Citations1994Steven N. Goodman, Jesse Berlin +2 more
Whether the quality of accepted manuscripts was improved by peer-review and editorial processes and, if it was, which aspects were most improved, is studied.
JAMAUsers' Guides to the Medical Literature
310 Citations1994Mitchell Levine
A search for a key article on the safety of β-adrenergic agonists in the treatment of asthma and how to respond to patients asking him about media reports of an increased risk of death associated with these medications.
European Journal of OrthodonticsA systematic review of the relationship between overjet size and traumatic dental injuries
221 Citations1999Quang Ngọc Nguyễn
The aim of this study was to aggregate the risk of traumatic dental injury due to overjet using several published papers and performing a meta-analysis on the results, and it can be concluded that children with an overjet larger than 3 mm are approximately twice as much at risk of injury to anterior teeth than children with a overjet smaller than 3mm.
Journal of DentistrySystematic review of population-based epidemiological studies of oro-facial pain
203 Citations2001Tatiana V. Macfarlane, Anne‐Marie Glenny +1 more
Age, gender and psychological factors were found to be associated with OFP, however there was not enough information on other factors such as local mechanical and co-morbidities to draw any reliable conclusions.
JAMAInstruments for Assessing the Quality of Drug Studies Published in the Medical Literature
179 Citations1994Mildred K. Cho
The two instruments developed, one measuring methodologic quality and one measuring clinical relevance of articles reporting clinical research, are reliable, valid, and applicable to a variety of research designs.
Critical Care MedicineAdult respiratory distress syndrome
156 Citations1996Bryan G. Garber, Paul C. Hébert +3 more
The lack of reproducible definitions for ARDS or its potential risk factors in 49% of studies raises concerns about the validity of the conclusions of these studies regarding the association between ARDS and the supposed risk factors.
The American Journal of MedicineMethodologic standards and contradictory results in case-control research
153 Citations1979Ralph I. Horwitz, Alvan R. Feinstein
Cited standards can help readers to evaluate the results of casecontrol research and can help designers of future studies to improve their scientific structure and credibility.
BMJUse of check lists in assessing the statistical content of medical studies.
141 Citations1986Martin J. Gardner, David Machin +1 more
Examination of the use of the check lists showed that most papers contained statistical failings, many of which could easily be remedied.
Journal of Bone and Joint SurgeryClosed Fractures of the Tibial Shaft
139 Citations1998Benjamin Littenberg, Loryn P. Weinstein +5 more
The results of the present review suggest that the data from the published literature are inadequate for decision-making with regard to the treatment of closed fractures of the tibia.
Health Technology AssessmentGeriatric rehabilitation following fractures in older people: a systematic review.
137 Citations2000Cameron, Maria Crotty +10 more
Sources included the bibliographies of all studies considered for the review, personal communication with experts in the field, searching of reference collections of group members including that of the Cochrane Musculoskeletal Injuries Group specialist database.
BMJThe scandal of poor epidemiological research: ***
133 Citations2004Erik von Elm, Matthias Egger
Reporting guidelines are needed for observational epidemiology to improve the quality of data and reduce the likelihood of errors in future studies.
PubMedLinks between passive smoking and disease: a best-evidence synthesis. A report of the Working Group on Passive Smoking.
132 Citations1990Spitzer Wo, Lawrence +8 more
There is strong evidence of an association between residential exposure to environmental tobacco smoke and both respiratory illness and reduction of lung function, and also between maternal smoking and reduced birth weight.
Academic MedicineThe effects of medical school curricula, faculty role models, and biomedical research support on choice of generalist physician careers
130 Citations1995Doug Campos‐Outcalt, Janet H. Senf +2 more
The authors evaluated and reviewed the literature on the effects of medical school curricula, faculty role models, and federal biomedical research support on the specialty choices of U.S. medical students and found there was a consistent inverse correlation between the amount of federal biomedicalResearch support received and the percentage of a school's graduates choosing generalist careers.
Archives of Internal MedicineThe Impact of Research Quality and Study Design on Epidemiologic Estimates of the Effect of Nonsteroidal Anti-inflammatory Drugs on Upper Gastrointestinal Tract Disease
120 Citations1992Paola Bollini
A meta-analysis of epidemiologic studies of upper gastrointestinal tract disease related to NSAIDs found the design and quality of the studies appear to be strong independent predictors of the risk estimate.
Pain PhysicianMedial Branch Neurotomy in Management of Chronic Spinal Pain:Systematic Review of the Evidence
118 Citations2002Laxmaiah Manchikanti
A systematic review of the evidence of effectiveness of radiofrequency Denervation in the management of chronic spinal pain provided strong evidence that radiofrequency denervation offers short-term relief and moderate evidence of long-term pain relief of facet joint origin.
Annals of Internal MedicineCosts, Outcomes, and Patient Satisfaction by Provider Type for Patients with Rheumatic and Musculoskeletal Conditions: A Critical Review of the Literature and Proposed Methodologic Standards
117 Citations1997Daniel H. Solomon, David W. Bates +2 more
Critically reviewed studies that compare specialist with generalist care of patients with musculoskeletal conditions with regard to appropriateness, clinical outcomes, costs, and patient satisfaction and offered methodologic standards to guide the interpretation of these studies.
PubMedAid to the evaluation of therapeutic studies.
111 Citations1989Joan Reisch, Jon E. Tyson +1 more
Journal of Chronic DiseasesGuidelines for reading case-control studies
105 Citations1987Michael J. Lichtenstein, Cynthia D. Mulrow +1 more
The proposed guidelines serve as a framework for readers to effectively assess the validity of a case-control report.
The Pediatric Infectious Disease JournalSecondary failure rates of measles vaccines: a metaanalysis of published studies
98 Citations1996Jennifer Anders, Robert M. Jacobson +3 more
Although reports of measles related to secondary failure exist, studies that permit the calculation of the rate of secondary failure demonstrate that the rate appears to be < 0.2%.
Annals of Emergency MedicineLife Support Courses: Are They Effective?
95 Citations1996Mona Jabbour, Martin H. Osmond +1 more
Among providers, retention of knowledge and skills acquired by participation in support courses is poor, however, refresher activities increase knowledge retention and there is evidence that use of the Advanced Trauma Life Support course has decreased mortality and morbidity.
Journal of Pediatric SurgeryDevelopment of a quality assessment scale for retrospective clinical studies in pediatric surgery
91 Citations2003Shawn J. Rangel, Jennifer L. Kelsey +3 more
A standardized and reliable quality assessment scale designed for retrospective studies in pediatric surgery is described, providing the practicing surgeon with a knowledge base to critically evaluate published retrospective data and potential applications include developing standardized reporting guidelines for use in peer-reviewed journals.
PEDIATRICSAid to the Evaluation of Therapeutic Studies
91 Citations1989Joan Reisch, Jon E. Tyson +1 more
The evaluation form provides a comprehensive set of well-accepted standards of research in a format that encourages detailed, consistent, and thoughtful evaluation of therapeutic studies.
Journal of Clinical EpidemiologyAn addition to the controversy on sunlight exposure and melanoma risk: A meta-analytical approach
81 Citations1995Patricia J. Nelemans, F.H.J. Rampen +2 more
It is concluded that evidence to support the intermittent sunlight theory is still far from complete and the lack of standardized measures for sunlight exposure warrants cautious interpretation of these results.
American Journal of Kidney DiseasesMethods used to evaluate the quality of evidence underlying the National Kidney Foundation-dialysis outcomes quality initiative clinical practice guidelines: Description, findings, and implications
74 Citations2000Earl P. Steinberg, Garabed Eknoyan +5 more
The approach the National Kidney Foundation-Dialysis Outcomes Quality Initiative used to assess the strength of published evidence pertinent to individual NKF-DOQI Clinical Practice Guidelines is described, as well as the relationship between that approach and methods used by the US Preventive Services Task Force, the Cochrane Collaboration, and the Agency for Health Care Policy and Research to rate the quality and/or strength of evidence.
American Journal of EpidemiologyControversy of Oral Contraceptives and Risk of Rheumatoid Arthirtis: Meta-analysis of Conflicting Studies and Review of Conflicting Meta-analyses with Special Emphasis on Analysis of Heterogeneity
74 Citations1996Manel Pladevall, George L. Delclos +4 more
There is no conclusive evidence of a protective effect of oral contraceptives on the risk of developing rheumatoid arthritis and consensus is needed on how meta-analyses of observational studies should be conducted.
Journal of DentistryMeta-analysis of anterior veneer restorations in clinical studies
73 Citations1998C.M. Kreulen, N.H.J. Creugers +1 more
The evaluative and statistical basis of clinical VR studies may be improved to obtain an effective inference of the results, and porcelain VRs show acceptable longevity after 3 years, which appears to be better than that of preformed acrylic VRs.
PubMedAn approach to the development of practice guidelines for community health interventions.
71 Citations1995T W Gyorkos, T N Tannenbaum +8 more
The Community Health Practice Guidelines project was initiated to develop a systematic approach to the critical evaluation of evidence on the effectiveness and efficiency of community health interventions and to the formulation of evidence-based practice recommendations.
Clinical RehabilitationPrognostic factors in the subacute phase after stroke for the future residence after six months to one year. A systematic review of the literature
60 Citations2003Ronald Meijer, Daniela Ihnenfeldt +3 more
At present there is insufficient evidence concerning possible predictors in the subacute stage of stroke to make an evidence-based prediction of the future residence.
Journal of Adolescent HealthChecklist for the evaluation of research articles
58 Citations1994Robert H. DuRant
A checklist was developed to assist physicians and other health care professionals in posing appropriate questions when reading or reviewing research articles to cast doubts on the stated conclusions.
International Journal of Technology Assessment in Health CareProstheses for Primary Total Hip Replacement:<i>A Critical Appraisal of the Literature</i>
57 Citations1995Delma E. Cowley
The published literature does not provide solid evidence for the superiority of cement-free or hybrid prostheses over modern cemented types.
PubMedA systematic overview of the effectiveness of home visiting as a delivery strategy for public health nursing interventions.
50 Citations1996Donna Ciliska, Sarah Hayward +6 more
Positive outcomes included improvement in children's mental development, mental health and physical growth; reduction in mother's anxiety, depression, and tobacco use; improvement in maternal employment, nutrition and other health habits; government cost saving; and reduced level of care required for the elderly.
Nutrition and CancerDevelopment of a scoring system to judge the scientific quality of information from case‐control and cohort studies of nutrition and disease
43 Citations1995Barrie Margetts, Rachel Thompson +5 more
Applying the scoring system to a review of meat and cancer risk suggested that, taking the score into account, there were relatively few studies that scored well, but these studies tended to provide more consistent information.
Medical EducationA checklist system for critical review of medical literature
38 Citations1985Cecilie Krogh
An approach was devised in a family practice residency setting for teaching students and doctors to review medical literature quickly while remaining critical of what is read through the use of a simple checklist which guides the user in reading an article.
Journal of Clinical NursingReading research critically. II. An introduction to appraisal: assessing the evidence
28 Citations1994Mark Avis
In this paper guidelines for evaluating the quality of evidence and the validity of conclusions are suggested, and a check-list of evaluation points is presented.
Journal of Chronic DiseasesObservational studies of cause-effect relationships: An analysis of methodologic problems as illustrated by the conflicting data for the role of oral contraceptives in the etiology of rheumatoid arthritis
27 Citations1986John M. Esdaile, Ralph I. Horwitz
The analysis leads to the conclusion that the case-control study suggesting no association of oral contraceptives with the subsequent development of rheumatoid arthritis provides the best estimate of the effect of oral contraceptive currently available.
Current Treatment Options in OncologyEarly cervical cancer
24 Citations2000Karen H. Lu, Thomas W. Burke
Recent data convincingly demonstrate that the addition of cisplatin-based chemotherapy significantly improves overall survival rates in cervical cancer patients who undergo radiation therapy.
International Journal of Radiation Oncology*Biology*PhysicsIs the published literature a reliable guide for deciding between alternative treatments for patients with early cervical cancer?
24 Citations1989Paolo Zola, T. Volpe +5 more
The quality of the methodology and reporting of studies on the treatment of early cervical cancer published in English and French language over the period 1975-1985 were examined using an explicit, pre-defined protocol aimed at assessing their internal validity and generalizability.
South African Journal of Clinical NutritionEvidence-based nutrition - review of nutritional epidemiological studies
19 Citations2002B.M. Margetts, HH Vorster +1 more
Journal of General Internal MedicineQuality of published reports of the prognosis of community-acquired pneumonia
15 Citations1994Catherine A. Carson, Michael J. Fine +4 more
Despite improvement in overall quality of published articles, systematic errors exist in the design and reporting of studies related to the prognosis of community-acquired pneumonia.
International Journal of EpidemiologyForward projection—using critical appraisal in the design of studies
13 Citations2002Mark Elwood
It is helpful in discussing study design with students or with health professionals to use the principles of critical appraisal in study design, and it is useful early in the work, when a preliminary approach has been outlined.
PubMedCritical appraisal of prognostic evidence: practical rules.
8 Citations2002António Vaz Carneiro
The guidelines for critical appraisal of a prognostic study are discussed, in terms of the validity, importance and applicability of its results.
Journal of General Internal MedicineBlood glucose and diabetic retinopathy
8 Citations1986Cynthia D. Mulrow, Michael J. Lichtenstein
Although initial diagnostic blood glucose concentration was associated with subsequent retinopathy development, the reviewed studies did not provide evidence that tight glucose control prevented or retarded its development and progression.