Assessing the quality of research
BMJPublished 1 January 2004Open access
Paul Glasziou, Jan P. Vandenbroucke, Iain Chalmers
Citations380
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
Inflexible use of evidence hierarchies confuses practitioners and irritates researchers. So how can we improve the way we assess research?
Keywords
Decision SciencesMedicineHealth Professions
Medical Entomology and ZoologyExperimental and Quasi-Experimental Designs for Research
15,238 Citations1963Donald T. Campbell, Julian C. Stanley +1 more
Journal of the Royal Society of MedicineThe environment and disease: association or causation?
7,311 Citations2015Austin Bradford Hill
The objects of this newly founded Section of Occupational Medicine are to provide a means, not readily afforded elsewhere, whereby physicians and surgeons with a special knowledge of the relationship between sickness and injury and conditions of work may discuss their problems, not only with each other, but also with colleagues in other fields, by holding joint meetings with other Sections of the Society.
Proceedings of the Royal Society of MedicineThe Environment and Disease: Association or Causation?
6,888 Citations1965Austin Bradford Hill
This paper contrasts Bradford Hill’s approach with a currently fashionable framework for reasoning about statistical associations – the Common Task Framework – and suggests why following Bradford Hill, 50+ years on, is still extraordinarily reasonable.
JAMAThe Hazards of Scoring the Quality of Clinical Trials for Meta-analysis
1,983 Citations1999Peter Jüni
The data indicate that the use of summary scores to identify trials of high quality is problematic, and relevant methodological aspects should be assessed individually and their influence on effect sizes explored.
JAMAEmpirical Evidence of Design-Related Bias in Studies of Diagnostic Tests
1,808 Citations1999Jeroen G. Lijmer
These data provide empirical evidence that diagnostic studies with methodological shortcomings may overestimate the accuracy of a diagnostic test, particularly those including nonrepresentative patients or applying different reference standards.
New England Journal of MedicineProblems of Spectrum and Bias in Evaluating the Efficacy of Diagnostic Tests
1,732 Citations1978David F. Ransohoff, Alvan R. Feinstein
To determine why many diagnostic tests have proved to be valueless after optimistic introduction into medical practice, a series of investigations and identified two major problems that can cause erroneous statistical results for the "sensitivity" and "specificity" indexes of diagnostic efficacy.
JAMAUsers' Guides to the Medical Literature
742 Citations2000Gordon H. Guyatt, Robert Haynes +7 more
This series provides clinicians with strategies and tools to interpret and integrate evidence from published research in their care of patients to relate to the value-laden nature of clinical decisions and to the hierarchy of evidence postulated by evidence-based medicine.
BiometricsAssessment of Diagnostic Tests When Disease Verification is Subject to Selection Bias
673 Citations1983Colin B. Begg, Robert A. Greenes
In the assessment of the statistical properties of a diagnostic test, for example the sensitivity and specificity of the test, it is common to derive estimates from a sample limited to those cases for whom subsequent definitive disease verification is obtained.
New England Journal of MedicineVariability in Radiologists' Interpretations of Mammograms
598 Citations1994Joann G. Elmore, Carolyn K. Wells +3 more
Efforts to improve accuracy and reduce variability in interpretation may increase the effectiveness of mammography in detecting early breast cancers.
Annals of Internal MedicineIn Defense of Case Reports and Case Series
537 Citations2001Jan P. Vandenbroucke
Case reporting remains important for detection of side effects of drugs, whether adverse or beneficial, even in the modern age of designer drugs and pharmacogenomics.
CirculationDeterminants of sensitivity and specificity of electrocardiographic criteria for left ventricular hypertrophy.
533 Citations1990Daniel Levy, Sherif B. Labib +4 more
It is suggested that electrocardiographic detection of LVH can be improved by incorporating information about noncardiac factors that impact on electrocardsiographic sensitivity for LVH, presumably by attenuating QRS voltage.
Cochrane Database of Systematic ReviewsRandomisation to protect against selection bias in healthcare trials
440 Citations2011Jan Odgaard‐Jensen, Gunn Elisabeth Vist +8 more
The results of randomised and non-randomised studies sometimes differed and it is not generally possible to predict the magnitude, or even the direction, of possible selection biases and consequent distortions of treatment effects from studies with non- random allocation or controlled trials with inadequate or unclear allocation concealment.
Statistics in MedicineStatistical methods for assessing the influence of study characteristics on treatment effects in ‘meta‐epidemiological’ research
390 Citations2002Jonathan A C Sterne, Peter Jüni +4 more
It is shown that both within- and between meta-analysis heterogeneity may be of importance in the analysis of meta-epidemiological studies, and that confounding exists between the effects of publication status and trial quality.
BMJAn evidence based approach to individualising treatment
386 Citations1995Paul Glasziou, Les Irwig
Before making firm conclusions, the assumptions of fixed adverse effects and constant reduction in relative risk need to be checked.
BMJChoosing the best research design for each question
374 Citations1997David L. Sackett, John E. Wennberg
The American Journal of MedicineFactors affecting sensitivity and specificity of exercise electrocardiography
372 Citations1984Mark A. Hlatky, David B. Pryor +4 more
The sensitivity and specificity of exercise electrocardiography vary with clinical history, extent of disease, and treadmill performance; the sensitivity and Specificity of other diagnostic tests may also vary.
New England Journal of MedicineThe Declining Specificity of Exercise Radionuclide Ventriculography
319 Citations1983Alan Rozanski, George Diamond +4 more
Two factors are responsible for the temporal decline in specificity: a change in the population being tested (pretest referral bias) and a preferential selection of patients with a positive test response for coronary angiography (post-test referral bias).
PubMedLetters, numbers, symbols and words: how to communicate grades of evidence and recommendations.
286 Citations2003Holger J. Schünemann, Dana Best +2 more
The advantages and disadvantages of using letters, numbers, symbols or words to represent grades of evidence and recommendations are discussed.
Annals of Internal MedicineSpectrum Bias in the Evaluation of Diagnostic Tests: Lessons from the Rapid Dipstick Test for Urinary Tract Infection
262 Citations1992Mark S. Lachs, Irving Nachamkin +4 more
The leukocyte esterase and bacterial nitrite rapid dipstick test for UTI is susceptible to spectrum bias, which may be responsible for differences in the test's sensitivity reported in previous studies.
Ultrasound in Medicine & BiologyROC analysis of noninvasive tests for peripheral arterial disease
234 Citations1996Jeroen G. Lijmer, M. G. Myriam Hunink +3 more
This study suggests that, in assessing whether a patient has significant peripheral arterial disease, determining an ABI is justified (ROC area 0.95 +/- 0.02), and a single test, the femoral PI, is sufficient.
Annals of Internal MedicineThe Electrocardiographic Exercise Test in a Population with Reduced Workup Bias: Diagnostic Performance, Computerized Interpretation, and Multivariable Prediction
216 Citations1998Victor F. Froelicher, Kenneth G. Lehmann +11 more
This study reduced workup bias prospectively by following a protocol that required patients to agree to undergo both exercise testing and coronary angiography, and allowed no more than 25% of the patients at any one site to have had a recent treadmill test.
Progress in Cardiovascular DiseasesThe diagnostic accuracy of the exercise electrocardiogram: A meta-analysis of 22 years of research
199 Citations1989Robert Detrano, R Gianrossi +1 more
CirculationSex and Test Verification Bias
194 Citations1997Véronique L. Roger, Patricia A. Pellikka +4 more
In clinical practice, test verification bias results in a lower observed specificity and a higher sensitivity of exercise echocardiography in women, while in men the positive predictive value and adjusted sensitivity are lower compared with that in men.
American Heart JournalComparison of the sensitivity and specificity of exercise electrocardiography in biased and unbiased populations of men and women
191 Citations1995Anthony P. Morise, George Diamond
The accuracy of exercise electrocardiography is lower in women than men irrespective of whether a biased or an unbiased group is used, however, these differences cannot be explained on the basis of sex-related differences in posttest referral bias.
JAMAContext Bias
190 Citations1996T K Egglin
Radiologists' diagnoses are significantly influenced by the context of interpretation, even when spectrum and verification bias are avoided, and this "context bias" effect is unique to the evaluation of subjectively interpreted tests, and illustrates the difficulty of obtaining unbiased estimates of diagnostic accuracy.
EpidemiologyLimitations of Sensitivity, Specificity, Likelihood Ratio, and Bayesʼ Theorem in Assessing Diagnostic Probabilities
187 Citations1997Karel G.M. Moons, Gerrit‐Anne van Es +3 more
The extent to which the sensitivity, specificity, and likelihood ratio of the exercise test to diagnose coronary artery disease vary across subgroups of a certain patient population is evaluated.
Cochrane Database of Systematic ReviewsRandomisation to protect against selection bias in healthcare trials
183 Citations2007Regina Kunz, Gunn Elisabeth Vist +1 more
American Journal of RoentgenologyInterpretation of radiographs: effect of clinical history
171 Citations1981Peter M. Doubilet, PG Herman
An effect of clinical information on interpretation of radiographs has been assumed, and test films were included in the dialy work load of readers who were unaware that a study was being carried out to investigate this effect in a real-life setting.
Journal of the American College of CardiologyExercise-induced st segment depression in the diagnosis of multivessel coronary disease: A meta analysis
154 Citations1989Robert Detrano, R Gianrossi +5 more
Exclusion of patients with right bundle branch block and comparison with a "better" exercise test were both significantly associated with sensitivity for the prediction of triple vessel or left main coronary artery disease.
Osteoporosis InternationalAcceptance and Compliance with External Hip Protectors: A Systematic Review of the Literature
142 Citations2002Natasja M. van Schoor, W.L.J.M. Devillé +3 more
Compliance is a very complex, but important issue in hip protector research and implementation, and adjustments should be made to the protector and the underwear, while maintaining the force attenuation capacity.
Annals of Internal MedicineEthnic and Sex Bias in Primary Care Screening Tests for Alcohol Use Disorders
137 Citations1998Jeffrey R. Steinbauer, Scott B. Cantor +2 more
This study tested for bias in the accuracy of three common self-report screening tests across sex and ethnic subgroups of primary care patients and found no evidence of ethnic and sex bias in screening accuracy.
Journal of Neurology Neurosurgery & PsychiatryFallacies in the pathological confirmation of the diagnosis of Alzheimer's disease
136 Citations1998John V. Bowler, David G. Muñoz +2 more
Correcting for dual pathology and verification bias halves the positive predictive value of the clinical diagnosis of Alzheimer’s disease.
Medical Entomology and ZoologyClinical Case Reporting in Evidence-based Medicine
127 Citations1999Milos Jenicek
The importance of modern case reporting on medical casuistics and how to prepare a single case report is discussed.
The American Journal of MedicineFactors affecting sensitivity and specificity of a diagnostic test: the exercise thallium scintigram
121 Citations1988Robert Detrano, András Jánosi +4 more
The reported sensitivity of thallium scintigraphy is higher and the specificity lower than that expected in clinical practice because of the presence of workup bias and the inappropriate inclusion of post-infarct patients.
BiometricsEvaluating Diagnostic Tests
119 Citations1981L. A. Thibodeau
This paper considers misclassification as nonsampling error, evaluating its effect on the observed sensitivity and specificity in diagnostic tests.
CHEST JournalReproducibility of the Histologic Diagnosis of Pneumonia Among a Panel of Four Pathologists
111 Citations1997David E. Corley, Steven H. Kirtland +5 more
The results established the necessity for standardization of histologic criteria for studies using biopsy as the gold standard for bacterial pneumonia.
PubMedDiagnostic accuracy of technetium-99m-MIBI myocardial SPECT in women and men.
102 Citations1998C Santana-Boado, Jaume Candell‐Riera +8 more
The sensitivity of 99mTc-MIBI myocardial SPECT in women was lower than in men when only the select minority was considered, and when the silent majority was considered (correction of selection bias) sensitivity and specificity results did not differ significantly between the sexes.
BMJAnecdotes as evidence
94 Citations2003Jeffrey K Aronson
There need guidelines for reporting anecdotes of suspected adverse drug reactions and there is a need for guidance on how to report these anecdotes to the proper authorities.
RadiologyImpact of clinical history on fracture detection with radiography.
85 Citations1988Kevin S. Berbaum, G Y el-Khoury +4 more
Analysis of receiver operator characteristic parameters indicates that clues regarding location of trauma facilitate detection of fractures, and improvement is based largely on an increased true- positive rate without an increased false-positive rate, regardless of the decision criteria of the radiologist.
JAMAThe Impact of Clinical History on Mammographic Interpretations
85 Citations1997Joann G. Elmore
Overall diagnostic accuracy was not altered, but recommendations were affected for appropriate further diagnostic workup: an alerting history increased the number of workups recommended in patients without cancer and a nonalerting history led to fewer recommended workups in the cancer patients.
Journal of Clinical EpidemiologyThe importance of work-up (verification) bias correction in assessing the accuracy of SPECT thallium-201 testing for the diagnosis of coronary artery disease
82 Citations1996Michael P. Cecil, Andrzej S. Kosinski +5 more
The observed sensitivity and specificity of stress (exercise and dipyridamole) single photon emission computed tomographic (SPECT) thallium testing for the detection of coronary artery disease by angiography was determined, and previously published equations were applied to correct for work-up bias.
BMJValidity of anecdotal reports of suspected adverse drug reactions: the problem of false alarms
81 Citations1982G R Venning
Criteria for assessing the validity of first reports of suspected adverse reactions should include: reactions on rechallenge, a pharmacological basis for the adverse reaction, immediate acute reactions, local reactions at the site of administration, reactions with a new route of administration of a drug known to provoke such reactions by another route, and the repeated occurrence of very rare events.
Annals of Internal MedicineAlternative Medicine: A “Mirror Image” for Scientific Reasoning in Conventional Medicine
78 Citations2001Jan P. Vandenbroucke, Anton J.M. de Craen
Medical Decision MakingEstimating Diagnostic Test Accuracy Using a "Fuzzy Gold Standard"
74 Citations1995Charles E. Phelps, Alan D. Hutson
This study uses Monte Carlo methods to analyze the consequences of having a criterion standard that contains some error when analyzing the accuracy of a diagnostic test using ROC curves, and proposes a two-truth method that selectively eliminates those cases where the gold standard is most ambiguous.
Journal of Clinical EpidemiologyBody Mass Index Compared to Dual-Energy X-Ray Absorptiometry: Evidence for a Spectrum Bias
71 Citations1997François Curtin, Alfredo Morabia +2 more
The clinical implication of this spectrum bias is that, in men or women weighing less than 80 kg, measures of obesity other than BMI, such as bioelectrical impedance, should be preferred.
Evidence-Based MedicineHow many Cochrane reviews are needed to cover existing evidence on the effects of healthcare interventions?
70 Citations2003Susan Mallett
The number of studies in a typical Cochrane review is used to provide an estimate of the number of Cochrane reviews that are needed to cover all trials currently in the Cochrane Central Register of Controlled Trials (CENTRAL).
Statistics in MedicineEffect of verification bias on positive and negative predictive values
66 Citations1994Xiao‐Hua Zhou
This paper shows that the properties of the estimators of positive and negative predictive values using only patients with verified disease statuses are unbiased and provide consistent estimators for the variances of these estimators under the assumption that the probability of selecting a patient for a disease verification procedure does not depend directly on the true disease status of the patient.
PubMedInfluence of training and experience in fine-needle aspiration biopsy of breast. Receiver operating characteristics curve analysis.
59 Citations1987Michael B. Cohen, Ryan P. Rodgers +6 more
It is shown that training and experience are important in the interpretation of breast FNAB specimens and the utility of receiver operating characteristics analysis in anatomic pathology is demonstrated.
American Journal of RoentgenologyImpact of clinical history on radiographic detection of fractures: a comparison of radiologists and orthopedists
56 Citations1989Kevin S. Berbaum, EA Franken +1 more
The findings show that localizing clues are important to both orthopedists and radiologists when searching for fractures, and localization clues improve ability of orthopedic surgeons to detect fractures in the trauma patient even more than they improve the ability of radiologists.
American Journal of RoentgenologyDoes knowledge of the clinical history affect the accuracy of chest radiograph interpretation?
55 Citations1990Barbara C. Good, Lawrence A. Cooperstein +5 more
It is suggested that knowledge of clinical history does not affect the accuracy of chest radiograph interpretations for the detection of interstitial disease, nodules, and pneumothoraces.
Medical Decision MakingWorkup Bias in Prediction Research
55 Citations1987Robert J. Panzer, Anthony L. Suchman +1 more
The authors modeled the effects of workup bias on the ability of early clinical findings to predict intracerebral hemorrhage in patients with stroke and founditivity was increased, but both specificity and likelihood ratios were decreased in the biased sample.
European Journal of CancerInter-observer and intra-observer variability of mammogram interpretation: a field study
53 Citations1992Giovannino Ciccone, Paolo Vineis +2 more
To evaluate the performance of radiologists in mammographic mass screening, seven radiologists read blindly the mammograms of 45 women (two views for each breast) and the readings were repeated after 2 years.
Obstetrics and GynecologyEffect of study design on the association between nuchal translucency measurement and Down syndrome
49 Citations1999Ben W. Mol
Studies with verification bias reported higher sensitivities, but also slightly higher specificities of nuchal translucency measurement than studies without verification bias, and it is postulate that the experience of the sonographist might be an explanation for the differences.
RadiologyThe effect of clinical bias on the interpretation of myelography and spinal computed tomography.
49 Citations1982O. P. Eldevik, G. Dugstad +2 more
This study suggests a tendency of observers to interpret questionable myelographic or computed tomographic findings as positive when they correlate with clinical findings.
CHEST JournalStratification of Patients According to Prior Cardiopulmonary Disease and Probability Assessment Based on the Number of Mismatched Segmental Equivalent Perfusion Defects
48 Citations1993Paul D. Stein, Alexander Gottschalk +2 more
Stratification on the basis of the presence or absence of prior cardiopulmonary disease enhanced the ability of ventilation/perfusion scan readers to assign an accurate positive predictive value and specificity to individual patients based on the observed number of mismatched segmental equivalent perfusion defects.
Caries ResearchFactors Involved in Validity Measurements of Diagnostic Tests for Approximal Caries – A Meta-Analysis
43 Citations2009H.M. van Rijkom, E.H. Verdonschot
Dz values obtained from in vivo studies were significantly different from those obtaining from in vitro studies (p < 0.05), indicating that study design had a significant impact on the measurement of the validity of the evaluated test for approximal caries diagnosis.
Medical Decision MakingAffirmative Actions
41 Citations1991George Diamond
Clinical estimates of test efficacy can be distorted by the differential referral of positive and negative test responders for outcome verification, and selection bias significantly distorts the determination of diagnostic and prognosis test accuracy in directionally opposite ways.
Journal of Clinical EpidemiologyProblems in selecting the adequate patient population from existing data files for assessment studies of new diagnostic tests
39 Citations1995Yvonne T. van der Schouw
Assessment of new diagnostic tests in populations with disease suspicion, as defined by signs, symptoms and complaints found large variations in the test characteristics of ultrasonography, suggesting selection bias depending on the true disease status.
ACP Journal ClubHow many Cochrane reviews are needed to cover existing evidence on the effects of health care interventions?
37 Citations2003Susan Mallett, DPhil, Mike Clarke, DPhil
American Journal of Clinical PathologyEffect of Clinical History on Diagnostic Accuracy in the Cytologic Interpretation of Bronchial Brush Specimens
36 Citations2000Stephen S. Raab, Thaira Oweity +6 more
Overall, the cytopathologists were more reluctant to make a definitive malignant diagnosis without history compared with history, and the average positive predictive value of amalignant diagnosis with and without history was almost identical.
NeurologyThe effect of spectrum bias on the utility of magnetic resonance imaging and evoked potentials in the diagnosis of suspected multiple sclerosis
32 Citations1996Paul O’Connor, Catherine M. Tansey +3 more
Both EP and MRI results demonstrated an association between disease frequency, disease severity, and test sensitivity with greater disease frequency and intensity suggesting more impressive diagnostic test performance, underscores that diagnostic tests perform differently in different groups of patients.
American Journal of Clinical PathologyPathology and Probability:<i>Likelihood Ratios and Receiver Operating Characteristic Curves in the Interpretation of Bronchial Brush Specimens</i>
31 Citations1995Stephen S. Raab, P. Thomas +8 more
It is concluded that qualitative diagnoses contain important information and can be interpreted effectively with LR and ROC.
CytopathologyEstimating the sensitivity of cervical cytology: errors of interpretation and test limitations
26 Citations1996Guglielmo Ronco, G Montanari +5 more
This study estimates the sensitivity of cervical cytology as a method of detecting abnormal smears among those obtained in the presence of cervical intraepithelial neoplasia (CIN), and creates a set of smears with a high a priori probability of being positive.
Scandinavian Journal of Primary Health CareThe spectrum of patients strongly influences the usefulness of diagnostic tests for pneumonia
24 Citations1993Hasse Melbye, Bjørn Straume
BMJOral contraception and health
21 Citations1999D C Skegg
While the British media have often produced more heat than light, scientists in Britain have contributed more than their share of evidence about the safety of oral contraceptives, and this week sees another publication from the Royal College of General Practitioners' oral contraception study.
PubMedInterobserver variability in the interpretation of myocardial images with Tc-99m-labeled diphosphonate and pyrophosphate.
18 Citations1980A Cuarón, A Acero +4 more
The results justify the setting of special programs to evaluateperiodically the performance of every physician who interprets studies, to establish his optimal individual criterion level instead of using a fixed criterion level to decide whether an image is positive.
Medical Decision MakingEvaluating Rapid Tests for Streptococcal Pharyngitis
17 Citations1987P DeNeef
An analysis of how errors in the comparison cultures affect the apparent accuracy of new tests for streptococcal pharyngitis shows that when evaluating a new test stricter conditions must be placed on the accuracy of the cultures used for comparison.
PubMedMethodologic problems in exercise testing research. Are we solving them?
16 Citations1988Robert Detrano, Kenneth P. Lyons +4 more
Biological PsychiatryThe effect of diagnostic methodology on the sensitivity of the TRH stimulation test for depression: A literature review
13 Citations1990George W. Arana, Michael N. Zarzar +1 more
The sensitivity for depression with the TRH test in the DSM-III reports was 34.8% compared with six RDC reports in which the sensitivity was 51%, which will help in future modifications and revisions of the DSM.
TrialsBridging case-control studies and randomized trials
13 Citations2001Frits R. Rosendaal
The author discusses the purpose of randomization and observation, and under which conditions one design may be preferred to another, and how results obtained by different designs may complement each other.
Acta OncologicaOver- and Underestimation of the Sensitivity of a Diagnostic Malignancy test due to Various Selections of the Study Population
9 Citations1990Adam Taube, Bengt Tholander
A simple model is given, explaining why some sensitivity estimates do not give a realistic picture of an indicator, and an estimate of a ‘future relevant sensitivity’ can be obtained.
Medical Decision MakingDiagnostic Workup Bias in the Evaluation of a Test
7 Citations1982David F. Ransohoff, Walter Muir
An examination of two studies report markedly divergent results about the usefulness of serum ferritin in diagnosing iron overload in relatives of patients with hereditary hemochromatosis suggests that diagnostic workup bias also may explain the difference.
…