Prognosis and prognostic research: validating a prognostic model
BMJPublished 28 May 2009
Doug Altman, Yvonne Vergouwe, Patrick Royston, Karel G.M. Moons
Citations1,458
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Douglas Altman and colleagues describe how to validate models and discuss some of the problems that need to be solved to validate prognostic models.
Abstract
Prognostic models are of little clinical value unless they are shown to work in other samples. Douglas Altman and colleagues describe how to validate models and discuss some of the problems
Keywords
Medicine
Statistics in MedicineMULTIVARIABLE PROGNOSTIC MODELS: ISSUES IN DEVELOPING MODELS, EVALUATING ASSUMPTIONS AND ADEQUACY, AND MEASURING AND REDUCING ERRORS
9,798 Citations1996Frank E. Harrell, Kerry L. Lee +1 more
European Journal of Cardio-Thoracic SurgeryEuropean system for cardiac operative risk evaluation (EuroSCORE)
3,376 Citations1999Samer A.M. Nashef, F. Roques +4 more
EuroSCORE is a simple, objective and up-to-date system for assessing heart surgery, soundly based on one of the largest, most complete and accurate databases in European cardiac surgical history and is recommend its widespread use.
Statistics in MedicineWhat do we mean by validating a prognostic model?
1,425 Citations2000Douglas G. Altman, Patrick Royston
How to validate a model is considered and it is suggested that it is desirable to consider two rather different aspects - statistical and clinical validity - and some general approaches to validation are examined.
BMJPrognosis and prognostic research: what, why, and how?
1,371 Citations2009Karel G.M. Moons, Patrick Royston +3 more
Why research into prognosis is important and how to design such research is explained are explained.
Annals of Internal MedicineAssessing the Generalizability of Prognostic Information
1,170 Citations1999Amy C. Justice, Kenneth E. Covinsky +1 more
BMJPrognosis and prognostic research: Developing a prognostic model
1,148 Citations2009Patrick Royston, Karel G.M. Moons +2 more
In the second article in their series, Patrick Royston and colleagues describe different approaches to building clinical prognostic models.
BMJPrognosis and prognostic research: application and impact of prognostic models in clinical practice
948 Citations2009Karel G.M. Moons, Douglas G. Altman +2 more
Annals of Internal MedicineTranslating Clinical Research into Clinical Practice: Impact of Using Prediction Rules To Make Decisions
806 Citations2006Brendan M. Reilly, Arthur T. Evans
A fifth level of evidence is proposed because it is believed that broad verification of a prediction rule's clinical impact is no less important than that of the prediction rule on which it is based, and progressive evidentiary standards emphasize that a prediction rules rises to the level of a decision rule only if clinicians use its predictions to help make decisions for patients.
Journal of Clinical EpidemiologyExternal validation is necessary in prediction research:
697 Citations2003Sacha E. Bleeker, Henriëtte A. Moll +5 more
For relatively small data sets, internal validation of prediction models by bootstrap techniques may not be sufficient and indicative for the model's performance in future patients.
Journal of Clinical EpidemiologySubstantial effective sample sizes were required for external validation studies of predictive logistic regression models
644 Citations2005Yvonne Vergouwe, Ewout W. Steyerberg +2 more
A minimum of 100 events and 100 nonevents is suggested for external validation samples to detect that a model predicted too high probabilities, when predictions were on average 1.5 times too high on the odds scale.
Statistics in MedicineValidation and updating of predictive logistic regression models: a study on sample size and shrinkage
549 Citations2004Ewout W. Steyerberg, Gerard Borsboom +3 more
A logistic regression model may be used to provide predictions of outcome for individual patients at another centre than where the model was developed to improve predictions for future patients.
Archives of SurgeryPostoperative Mortality and Morbidity in French Patients Undergoing Colorectal Surgery
522 Citations2005Arnaud Alvès
Colorectal resection in France is associated with a 3.4% mortality rate and a 35% morbidity rate, and knowledge of the risk factors could help surgeons manage cases.
European Respiratory JournalA chart of failure risk for noninvasive ventilation in patients with COPD exacerbation
381 Citations2005Marco Confalonieri, Giancarlo Garuti +9 more
The prediction chart can function as a simple tool to predict the risk of failure of noninvasive positive pressure ventilation and thus improve clinical management of patients tailoring medical intervention.
BMJCommentary: Prognostic models: clinically useful or quickly forgotten?
372 Citations1995Jeremy C Wyatt, D. G Altman
The main reasons why doctors reject published prognostic models are lack of clinical credibility and lack of evidence that a prognostic model can support decisions about patient care (that is, evidence of accuracy, generality, and effectiveness).
Statistics in MedicineValidation techniques for logistic regression models
319 Citations1991Michael E. Miller, Siu L. Hui +1 more
A model-based approach developed by Cox is adapted for use in model validation, which allows identification of problematic predictor variables in the prediction model as well as influential observations in the validation data that adversely affect the fit of the model.
Journal of the Royal Statistical Society Series C (Applied Statistics)The Use of Resampling Methods to Simplify Regression Models in Medical Statistics
319 Citations1999Willi Sauerbrei
The problems of replication stability, model complexity, selection bias and an over‐optimistic estimate of the predictive value of a model are discussed together with several proposals based on resampling methods, which favour greater simplicity of the final regression model.
Journal of Clinical EpidemiologyMeasuring the accuracy of prognostic judgments in oncology
178 Citations1997William J. Mackillop, Carol F. Quirt
It was concluded that oncologists' estimates of probability of cure are surprisingly accurate, but estimates of the duration of survival of incurable patients are much less accurate, and subject to a "horizon effect" similar to that recognized in weather forecasting.
European Journal of Cardio-Thoracic SurgeryValidation of the EuroSCORE model in Australia☆
157 Citations2006Cheng‐Hon Yap, Christopher M. Reid +6 more
The additive and logistic EuroSCORE model does not accurately predict outcomes in this group of cardiac surgery patients from six Australian institutions, Hence, the use of the EuroSCore models for risk prediction may not be appropriate in Australia.
Annals of SurgeryThe AFC Score: Validation of a 4-Item Predicting Score of Postoperative Mortality After Colorectal Resection for Cancer or Diverticulitis
118 Citations2007Arnaud Alvès, Yves Panís +4 more
The validity of the AFC score in this population was found very high based both on the Hosmer-Lemeshow goodness of fit test and on the area under the ROC curve, and that discriminatory capacity was higher than other currently used risk scoring systems such as the Glasgow or ASA score.
AnaesthesiaAssessing the applicability of scoring systems for predicting postoperative nausea and vomiting
111 Citations2005J.E. van den Bosch, Cor J. Kalkman +5 more
Neither of the scoring systems provided a risk threshold for administering anti‐emetic prophylaxis that yielded satisfying results in terms of predictive values, sensitivity and specificity, so they do not guarantee accurate prediction of the risk of postoperative nausea and vomiting in other patient populations.
Statistics in MedicineConstruction, validation and updating of a prognostic model for kidney graft survival
74 Citations1995Hans C. van Houwelingen, J. Thorogood
The construction, validation and updating of a prognostic model for kidney graft survival is reported using data from the Eurotransplant database, and recommendations are made to place less emphasis on the selection of variables and cut-off points, and more focus on the fine-tuning of the prognostic models by means of low-dimensional parametric models in independent data sets.
PubMedPredicting complications from acute cough in pre-school children in primary care: a prospective cohort study.
33 Citations2004Alastair D Hay, Tom Fahey +2 more
A clinical prediction rule for complications in pre-school children presenting to primary care with acute cough could be used to individualise the management of acute cough in pre -school children.
The Annals of Thoracic SurgeryPredictive accuracy study: comparing a statistical model to clinicians’ estimates of outcomes after coronary bypass surgery
32 Citations2000Joan Ivanov, Michael A. Borger +4 more
Clinicians trusted their own empiric estimates rather than a predictive rule and overestimated the probability of operative mortality and ICU stay greater than 48 hours.
Journal of General Internal MedicineStatistically derived predictive models caveat emptor
15 Citations1999Lisa I. Iezzoni
Justice, Covinsky, and Berlin offer a systematic approach for evaluating the performance of prognostic models, but as they note, the relative importance of different aspects of external validity depends on one’s purpose.
American Heart JournalExternal validation of a percutaneous coronary intervention mortality prediction model in patients with acute coronary syndromes
10 Citations2006Jaroslav A. Hubáček, P. Diane Galbraith +5 more
The enhanced MM developed specifically for the subset of patients with ACS demonstrated better prediction and cross-validated performance, and can be useful for risk-adjustment analyses and for prognostication for individual patients.
PubMedValidation of a clinical rule to predict complications of acute cough in preschool children: a prospective study in primary care.
10 Citations2007Alastair D Hay, Catharine Gorst +3 more
No evidence was found to validate the clinical rule for predicting complications of acute cough, possibly as a result of spectrum bias, confounding by indication, and/or chance.
Journal of Clinical EpidemiologyA Comparison of Model-Building Strategies for Lower Respiratory Tract Infection in Long-Term Care
6 Citations1999Mark Loeb, Stephen D. Walter +4 more
Five strategies for creating predictive models of lower respiratory tract infection in residents of long-term care facilities were compared and the data-derived and consensus models showed the highest discriminative ability while the linear judgment model showed inferior performance.