Reporting and Methods in Clinical Prediction Research: A Systematic Review
PLoS MedicinePublished 22 May 2012Open access
Walter Bouwmeester, Nicolaas P. A. Zuithoff, Susan Mallett, Mirjam I. Geerlings, Yvonne Vergouwe, Ewout W. Steyerberg
Citations582
SJR quartileQ1
SJR score4.28
SNIP3.21
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
It is found that the majority of prediction studies do not follow current methodological recommendations and need to be re-examined.
Abstract
The majority of prediction studies in high impact journals do not follow current methodological recommendations, limiting their reliability and applicability.
Keywords
Decision SciencesMathematicsEconomics, Econometrics and Finance
PLoS MedicineThe Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: Guidelines for Reporting Observational Studies
21,612 Citations2007Erik von Elm, Douglas G. Altman +5 more
The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Initiative developed recommendations on what should be included in an accurate and complete report of an observational study, resulting in a checklist of 22 items that relate to the title, abstract, introduction, methods, results, and discussion sections of articles.
The LancetThe Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies
18,697 Citations2007Erik von Elm, Douglas G. Altman +4 more
The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) initiative developed recommendations on what should be included in an accurate and complete report of an observational study, resulting in a checklist of 22 items that relate to the title, abstract, introduction, methods, results, and discussion sections of articles.
Statistics in MedicineMULTIVARIABLE PROGNOSTIC MODELS: ISSUES IN DEVELOPING MODELS, EVALUATING ASSUMPTIONS AND ADEQUACY, AND MEASURING AND REDUCING ERRORS
9,798 Citations1996Frank E. Harrell, Kerry L. Lee +1 more
BMJCONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials
9,055 Citations2010David Moher, Sally Hopewell +7 more
Journal of Clinical EpidemiologyA simulation study of the number of events per variable in logistic regression analysis
9,033 Citations1996Peter Peduzzi, John Concato +3 more
Findings indicate that low EPV can lead to major problems, and the regression coefficients were biased in both positive and negative directions, and paradoxical associations (significance in the wrong direction) were increased.
PLoS MedicineStrengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration
8,408 Citations2007Jan P. Vandenbroucke, Erik von Elm +8 more
A checklist of items that should be addressed in Reports of Observational Studies in Epidemiology (STROBE) Statement, a general reporting recommendations for descriptive observational studies and studies that investigate associations between exposures and health outcomes is developed.
American Journal of EpidemiologyRelaxing the Rule of Ten Events per Variable in Logistic and Cox Regression
4,137 Citations2006Eric Vittinghoff, Charles E. McCulloch
A large simulation study of other influences on confidence interval coverage, type I error, relative bias, and other model performance measures found a range of circumstances in which coverage and bias were within acceptable levels despite less than 10 EPV.
BMC Medical Research MethodologyThe development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews
3,624 Citations2003Penny Whiting, Anne WS Rutjes +3 more
Annals of Internal MedicineStrengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and Elaboration
3,523 Citations2007Jan P. Vandenbroucke, Erik von Elm +8 more
Journal of Clinical EpidemiologyInternal validation of predictive models
2,629 Citations2001Ewout W. Steyerberg, Frank E. Harrell +4 more
It is concluded that split-sample validation is inefficient, and bootstrapping is recommended for estimation of internal validity of a predictive logistic regression model.
Journal of Clinical EpidemiologyReview: A gentle introduction to imputation of missing values
2,581 Citations2006A. Rogier T. Donders, Geert J. M. G. van der Heijden +2 more
Statistics in MedicineDichotomizing continuous predictors in multiple regression: a bad idea
2,228 Citations2005Patrick Royston, Douglas G. Altman +1 more
It is argued that the simplicity achieved is gained at a cost; dichotomization may create rather than avoid problems, notably a considerable loss of power and residual confounding.
Journal of Clinical EpidemiologyImportance of events per independent variable in proportional hazards regression analysis II. Accuracy and precision of regression estimates
2,141 Citations1995Peter Peduzzi, John Concato +2 more
The analytical effect of the number of events per variable (EPV) in a proportional hazards regression analysis was evaluated using Monte Carlo simulation techniques for data from a randomized trial containing 673 patients and 252 deaths, in which seven predictor variables had an original significance level of p < 0.10.
JNCI Journal of the National Cancer InstituteReporting Recommendations for Tumor Marker Prognostic Studies (REMARK)
2,052 Citations2005Lisa M. McShane, Douglas G. Altman +4 more
Journal of the American Statistical AssociationRegression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis
1,975 Citations2003Sunil J Rao
The basic Bayesian framework must be constrained, use of the step function in computing the probability that a team would rank best or worst in a league, and implementation of a Dirichlet process prior are presented.
JAMAEmpirical Evidence of Design-Related Bias in Studies of Diagnostic Tests
1,808 Citations1999Jeroen G. Lijmer
These data provide empirical evidence that diagnostic studies with methodological shortcomings may overestimate the accuracy of a diagnostic test, particularly those including nonrepresentative patients or applying different reference standards.
Statistics in the health sciencesClinical Prediction Models
1,513 Citations2008Ewout W. Steyerberg
This paper presents a case study on survival analysis: Prediction of secondary cardiovascular events and lessons from case studies on overfitting and optimism in prediction models.
BMJPrognosis and prognostic research: validating a prognostic model
1,458 Citations2009Doug Altman, Yvonne Vergouwe +2 more
Douglas Altman and colleagues describe how to validate models and discuss some of the problems that need to be solved to validate prognostic models.
Annals of Internal MedicineEvaluation of the Quality of Prognosis Studies in Systematic Reviews
1,452 Citations2006Jill A. Hayden, Pierre Côté +1 more
A review of reviews to describe methods used to assess the quality of prognosis studies and to describe how well current practices assess potential biases is to develop recommendations to guide future quality appraisal.
Statistics in MedicineWhat do we mean by validating a prognostic model?
1,425 Citations2000Douglas G. Altman, Patrick Royston
How to validate a model is considered and it is suggested that it is desirable to consider two rather different aspects - statistical and clinical validity - and some general approaches to validation are examined.
New England Journal of MedicineClinical Prediction Rules
1,374 Citations1985John H. Wasson, Harold C. Sox +2 more
Qualitative standards that can be used to decide whether a prediction rule is suitable for adoption in a clinician's practice are described and applied to 33 reports of prediction rules.
BMJPrognosis and prognostic research: what, why, and how?
1,371 Citations2009Karel G.M. Moons, Patrick Royston +3 more
Why research into prognosis is important and how to design such research is explained are explained.
Annals of Internal MedicineAssessing the Generalizability of Prognostic Information
1,170 Citations1999Amy C. Justice, Kenneth E. Covinsky +1 more
BMJPrognosis and prognostic research: Developing a prognostic model
1,148 Citations2009Patrick Royston, Karel G.M. Moons +2 more
In the second article in their series, Patrick Royston and colleagues describe different approaches to building clinical prognostic models.
BMJThe impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews
1,114 Citations2010Jamie J Kirkham, Kerry Dwan +5 more
Outcome reporting bias is an under-recognised problem that affects the conclusions in a substantial proportion of Cochrane reviews and individuals conducting systematic reviews need to address explicitly the issue of missing outcome data for their review to be considered a reliable source of evidence.
Annals of Internal MedicineThe Risk of Determining Risk with Multivariable Models
1,112 Citations1993John Concato, Alvan R. Feinstein +1 more
The purpose in the current research was to note the frequency with which multivariable analyses now appear in general medical journals, to identify some common problems and desirable precautions in the analyses, and to determine how well the challenges are being met.
JAMAUsers' Guides to the Medical Literature
1,003 Citations2000Thomas McGinn, Gordon Guyatt +5 more
Level 1 CDRs have the potential to inform clinical judgment, to change clinical behavior, and to reduce unnecessary costs, while maintaining quality of care and patient satisfaction.
BMJPrognosis and prognostic research: application and impact of prognostic models in clinical practice
948 Citations2009Karel G.M. Moons, Douglas G. Altman +2 more
Journal of Clinical EpidemiologyInappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis
903 Citations1996Guo‐Wen Sun, Thomas Shook +1 more
This article uses both hypothetical and actual data to show how a nonsignificant risk factors in bivariable analysis may actually be a significant risk factor in multivariableAnalysis if confounding is properly controlled.
American Journal of EpidemiologyA Critical Look at Methods for Handling Missing Covariates in Epidemiologic Regression Analyses
886 Citations1995Sander Greenland, William D. Finkle
The authors recommend that epidemiologists avoid using the missing-indicator method and use more sophisticated methods whenever a large proportion of data are missing, and contrast the results of multiple imputation to simple methods in the analysis of a case-control study of endometrial cancer.
Annals of Internal MedicineTranslating Clinical Research into Clinical Practice: Impact of Using Prediction Rules To Make Decisions
806 Citations2006Brendan M. Reilly, Arthur T. Evans
A fifth level of evidence is proposed because it is believed that broad verification of a prediction rule's clinical impact is no less important than that of the prediction rule on which it is based, and progressive evidentiary standards emphasize that a prediction rules rises to the level of a decision rule only if clinicians use its predictions to help make decisions for patients.
JAMAClinical Prediction Rules<subtitle>A Review and Suggested Modifications of Methodological Standards</subtitle>
770 Citations1997Andreas Laupacis
Although clinical prediction rules comply with some methodological criteria, for other criteria, better compliance is needed.
Journal of Clinical EpidemiologyExternal validation is necessary in prediction research:
697 Citations2003Sacha E. Bleeker, Henriëtte A. Moll +5 more
For relatively small data sets, internal validation of prediction models by bootstrap techniques may not be sufficient and indicative for the model's performance in future patients.
Data Archiving and Networked Services (DANS)Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative
697 Citations2003
Journal of Clinical EpidemiologySubstantial effective sample sizes were required for external validation studies of predictive logistic regression models
644 Citations2005Yvonne Vergouwe, Ewout W. Steyerberg +2 more
A minimum of 100 events and 100 nonevents is suggested for external validation samples to detect that a model predicted too high probabilities, when predictions were on average 1.5 times too high on the odds scale.
Journal of Clinical EpidemiologyInternal and external validation of predictive models: A simulation study of bias and precision in small samples
578 Citations2003Ewout W. Steyerberg, Sacha E. Bleeker +3 more
Bootstrapping for internal validation gives reasonably valid estimates of the expected optimism in predictive performance provided that any selection of predictors is taken into account, and should be used for external validation.
Journal of Clinical EpidemiologyValidation, updating and impact of clinical prediction rules: A review
574 Citations2008Diane B Toll, K.J.M. Janssen +2 more
An overview of the research steps that need to follow the development of diagnostic or prognostic prediction rules are provided, including validity assessment, updating, and impact assessment of clinical prediction rules.
Annals of Emergency MedicineMethodologic Standards for the Development of Clinical Decision Rules in Emergency Medicine
573 Citations1999Ian G. Stiell, George A. Wells
The 6 major stages in the development and testing of a new clinical decision rule are considered and a number of standards within each stage are discussed.
Canadian Medical Association JournalEvidence of bias and variation in diagnostic accuracy studies
570 Citations2006Anne WS Rutjes, Johannes B. Reitsma +4 more
Meta-analyses of the diagnostic accuracy of tests through an electronic search of the databases MEDLINE, EMBASE, DARE and MEDION found that design features and clinical characteristics of patient groups should be carefully considered when designing new studies and by readers when appraising the results of such studies.
Statistics in MedicineValidation and updating of predictive logistic regression models: a study on sample size and shrinkage
549 Citations2004Ewout W. Steyerberg, Gerard Borsboom +3 more
A logistic regression model may be used to provide predictions of outcome for individual patients at another centre than where the model was developed to improve predictions for future patients.
Journal of Clinical EpidemiologyStepwise Selection in Small Data Sets A Simulation Study of Bias in Logistic Regression Analysis
503 Citations1999Ewout W. Steyerberg
It is concluded that stepwise selection may result in a substantial bias of estimated regression coefficients of selected covariables, similar to that found in the GUSTO-I trial.
Clinical ChemistryCase–Control and Two-Gate Designs in Diagnostic Accuracy Studies
500 Citations2005Anne WS Rutjes, Johannes B. Reitsma +3 more
Researchers and readers should be aware of spectrum effects in diagnostic case-control studies as a result of the restricted sampling of cases and/or controls, which can lead to changes in estimates of diagnostic accuracy.
Theoretical Epidemiology: Principles of Occurrence Research in Medicine
483 Citations1985Olli S. Miettinen
JAMATheoretical Epidemiology: Principles of Occurrence Research in Medicine
470 Citations1986Sander Greenland
For me and many of my colleagues, Miettinen's unpublished course text of the early 1970s was the first systematic introduction to theoretical epidemiology the authors saw; indeed, it was the only such text available until the 1980s; the final finished product bears little resemblance to the 1970s text in either form or substance.
PubMedClinical prediction rules. A review and suggested modifications of methodological standards.
430 Citations1997A Laupacis, Nitin Sekar +1 more
Statistics in MedicineAdjusting for partially missing baseline measurements in randomized trials
381 Citations2004Ian R. White, Simon G. Thompson
Joint modelling of baseline and outcome is the most efficient method, subject to three conditions, which are illustrated in a randomized trial in community psychiatry.
Journal of Clinical EpidemiologyUpdating methods improved the performance of a clinical prediction model in new patients
380 Citations2007K.J.M. Janssen, Karel G.M. Moons +3 more
When the performance is poor in new patients, updating methods can be applied to adjust the model, rather than to develop a new model.
BMC Medical Informatics and Decision MakingSystematic review of prognostic models in traumatic brain injury
343 Citations2006Pablo Perel, Phil Edwards +2 more
Journal of the Royal Statistical Society Series C (Applied Statistics)The Use of Resampling Methods to Simplify Regression Models in Medical Statistics
319 Citations1999Willi Sauerbrei
The problems of replication stability, model complexity, selection bias and an over‐optimistic estimate of the predictive value of a model are discussed together with several proposals based on resampling methods, which favour greater simplicity of the final regression model.
Clinical Epidemiology: Principles, Methods, And Applications For Clinical Research
294 Citations2008Diederick E. Grobbee, Arno W. Hoes
Clinical epidemiology: principles, methods, and applications for clinical research, Clinical epidemiological principles and applications, کتابجیتال - آذرسا.
Breast Cancer Research and TreatmentMethodological challenges in the evaluation of prognostic factors in breast cancer
272 Citations1998Douglas G. Altman, Gary H. Lyman
New England Journal of MedicineFive-Year Risk of Colorectal Neoplasia after Negative Screening Colonoscopy
254 Citations2008Thomas F. Imperiale, Elizabeth A. Glowinski +4 more
Journal of Internal MedicineThe use and reporting of multiple imputation in medical research – a review
233 Citations2010Andrew Mackinnon
The use and reporting of multiple imputation in medical research – a review and the role of data quality in this review are reviewed.
British Journal of CancerMissing covariate data within cancer prognostic studies: a review of current reporting and proposed guidelines
222 Citations2004Andrea Burton, Douglas G. Altman
BMC MedicineReporting methods in studies developing prognostic models in cancer: a review
212 Citations2010Susan Mallett, Patrick Royston +3 more
It is found that published prognostic models are often characterised by both use of inappropriate methods for development of multivariable models and poor reporting, and models are limited by the lack of studies based on prospective data of sufficient sample size to avoid overfitting.
BMC MedicineReporting performance of prognostic models in cancer: a review
193 Citations2010Susan Mallett, Patrick Royston +3 more
Many published prognostic models have been developed using poor methods and many with poor reporting, both of which compromise the reliability and clinical relevance of models, prognostic indices and risk groups derived from them.
JAMAPrehospital Termination of Resuscitation in Cases of Refractory Out-of-Hospital Cardiac Arrest
189 Citations2008Comilla Sasson
Journal of Clinical EpidemiologyUnpredictable bias when using the missing indicator method or complete case analysis for missing confounder values: an empirical example
181 Citations2010Mirjam J. Knol, Kristel J.M. Janssen +6 more
MIM should not be used in handling missing confounder data because it gives unpredictable bias of the odds ratio even with small percentages of missing values, and CC can be used when missing values are completely random, but it gives loss of statistical power.
BMJSample sizes of studies on diagnostic accuracy: literature survey
180 Citations2006Lucas M. Bachmann, Milo A. Puhan +2 more
The number of participants in most studies on diagnostic accuracy is probably too small to analyse variability of measures of accuracy across patient subgroups, and no studies reported that sample size was calculated on the basis of preplanned analyses of subgroups.
Human Reproduction UpdatePrediction models in reproductive medicine: a critical appraisal†
177 Citations2009Esther Leushuis, Jan Willem van der Steeg +6 more
There are three models with good predictive performance that can be used reliably as a guide for making decisions about fertility treatment, in patients similar to the development population.
Journal of Clinical EpidemiologyA systematic review finds methodological improvements necessary for prognostic models in determining traumatic brain injury outcomes
176 Citations2008Nino A. Mushkudiani, Chantal W.P.M. Hukkelhoven +5 more
Although most models agree on the three most important predictors, many were developed on small sample sizes within single centers and hence lack generalizability.
BMJManchester triage system in paediatric emergency care: prospective observational study
158 Citations2008Mirjam van Veen, Ewout W. Steyerberg +5 more
The Manchester triage system has moderate validity in paediatric emergency care, but errs on the safe side, with much more over-triage than under-Triage compared with an independent reference standard for urgency.
Annals of Internal MedicinePrehypertension during Young Adulthood and Coronary Calcium Later in Life
153 Citations2008Mark J. Pletcher, Kirsten Bibbins‐Domingo +7 more
The CARDIA Study found that young adults with systolic prehypertension were more likely than those without preHypertension to have coronary artery calcium, and these investigators observed associations with coronary calcium levels measured later in life.
BMC Medical Research MethodologyComparison of imputation methods for handling missing covariate data when fitting a Cox proportional hazards model: a resampling study
148 Citations2010Andrea Marshall, Douglas G. Altman +1 more
Of the MI approaches investigated, MI using MICE-PMM produced the least biased estimates and better model performance measures, however, this MI approach still produced biased regression coefficient estimates with 75% missingness.
BMC Medical Research MethodologyAdvantages of the nested case-control design in diagnostic research
136 Citations2008Cornelis Biesheuvel, Yvonne Vergouwe +4 more
Journal of Clinical EpidemiologyIdentifying phases of investigation helps planning, appraising, and applying the results of explanatory prognosis studies
133 Citations2008Jill A. Hayden, Pierre Côté +2 more
Researchers should consider the approach to prognosis (explanatory or outcome prediction) and phase of investigation, use best methods to limit biases, report completely, and cautiously interpret results.
Cancer InvestigationPrognostic Models: A Methodological Framework and Review of Models for Breast Cancer
125 Citations2009Douglas G. Altman
This chapter describes a review of published prognostic models that have been developed to predict the outcome of future breast cancer patients and identifies models for which there has been an evaluation of how successful the models have been when used in a different setting.
Seminars in Urologic OncologyValidity of prognostic models: When is a model clinically useful?
123 Citations2002Yvonne Vergouwe, Ewout W. Steyerberg +2 more
It is concluded that good calibration and discriminative ability are not sufficient for a model to be clinically useful, if the model is able to provide useful additional information for clinical decision making.
British Journal of CancerReporting of prognostic studies of tumour markers: a review of published articles in relation to REMARK guidelines
121 Citations2009Susan Mallett, Antje Timmer +2 more
A reporting aid, the REMARK profile, motivated by the CONSORT flowchart is proposed, which would greatly improve reporting and enhance prognostic research.
Journal of Clinical EpidemiologyA review of two journals found that articles using multivariable logistic regression frequently did not report commonly recommended assumptions
118 Citations2004Kenneth J. Ottenbacher, Heather R. Ottenbacher +2 more
Journal of Clinical EpidemiologyBias arising from missing data in predictive models
117 Citations2006Marc H. Gorelick
All three methods of handling large amounts of missing data can lead to biased estimates of the OR and of model performance in predictive models.
Annals of Internal MedicineComposite Outcomes in Cardiovascular Research: A Survey of Randomized Trials
115 Citations2008Eric Lim, Adam J. Brown +3 more
How composite outcomes are currently used, constructed, and reported in cardiovascular medicine and surgery is reviewed to assess and to quantitatively assess the contribution of common, clinically important individual outcomes to the composite outcome.
PLoS MedicineStrengthening the Reporting of Genetic Risk Prediction Studies: The GRIPS Statement
88 Citations2011A. Cecile J.W. Janssens, John P. A. Ioannidis +4 more
Nature Clinical Practice OncologyPrimer: an evidence-based approach to prognostic markers
78 Citations2005Douglas G. Altman, Richard D Riley
Prognostic markers can help to identify patients at different degrees of risk for specific outcomes, facilitate treatment choice, and aid patient counseling, but an evidence-based approach is required, as it is usually difficult to ascertain the benefit of a marker from single studies and a clear view is only likely to emerge from looking across multiple studies.
Journal of Clinical EpidemiologyStrengthening the reporting of Genetic RIsk Prediction Studies: the GRIPS Statement
36 Citations2011A. Cecile J.W. Janssens, John P. A. Ioannidis +3 more
The GRIPS Statement is published and will also be published by PLoS Medicine, BMJ, Circulation: Cardiovascular Genetics, European Journal of Clinical Investigation, European Journal of Epidemiology, European Journal of Human Genetics, Genetics in Medicine, Genome Medicine, and Journal of Clinical Epidemiology.
Statistics in MedicineResidual mass histology in testicular cancer: development and validation of a clinical prediction rule
22 Citations2001Ewout W. Steyerberg, Yvonne Vergouwe +3 more
Practical NeurologyPrognostic models
13 Citations2008Peter M. Rothwell
How prognostic models are best derived, and how their reliability should be assessed, are considered, drawing on examples from various different neurological disorders.
…