External validation is necessary in prediction research:
Journal of Clinical EpidemiologyPublished 1 September 2003
Sacha E. Bleeker, Henriëtte A. Moll, Ewout W. Steyerberg, A. Rogier T. Donders, G Derksen‐Lubsen, Diederick E. Grobbee
Citations697
SJR quartileQ1
SJR score3.15
SNIP2.66
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
For relatively small data sets, internal validation of prediction models by bootstrap techniques may not be sufficient and indicative for the model's performance in future patients.
Abstract
For relatively small data sets, internal validation of prediction models by bootstrap techniques may not be sufficient and indicative for the model's performance in future patients. External validation is essential before implementing prediction models in clinical practice.
Keywords
Medicine
Statistics in MedicineMULTIVARIABLE PROGNOSTIC MODELS: ISSUES IN DEVELOPING MODELS, EVALUATING ASSUMPTIONS AND ADEQUACY, AND MEASURING AND REDUCING ERRORS
9,798 Citations1996Frank E. Harrell, Kerry L. Lee +1 more
BiometrikaA note on a general definition of the coefficient of determination
5,718 Citations1991N. J. D. Nagelkerke
Journal of Clinical EpidemiologyInternal validation of predictive models
2,629 Citations2001Ewout W. Steyerberg, Frank E. Harrell +4 more
It is concluded that split-sample validation is inefficient, and bootstrapping is recommended for estimation of internal validity of a predictive logistic regression model.
Journal of the American Statistical AssociationEstimating the Error Rate of a Prediction Rule: Improvement on Cross-Validation
2,161 Citations1983Bradley Efron
New England Journal of MedicineClinical Prediction Rules
1,374 Citations1985John H. Wasson, Harold C. Sox +2 more
Qualitative standards that can be used to decide whether a prediction rule is suitable for adoption in a clinician's practice are described and applied to 33 reports of prediction rules.
Journal of the American Statistical AssociationImprovements on Cross-Validation: The 632+ Bootstrap Method
1,369 Citations1997Bradley Efron, Robert Tibshirani
It is shown that a particular bootstrap method, the .632+ rule, substantially outperforms cross-validation in a catalog of 24 simulation experiments and also considers estimating the variability of an error rate estimate.
Annals of Internal MedicineAssessing the Generalizability of Prognostic Information
1,170 Citations1999Amy C. Justice, Kenneth E. Covinsky +1 more
JAMAUsers' Guides to the Medical Literature
1,003 Citations2000Thomas McGinn, Gordon Guyatt +5 more
Level 1 CDRs have the potential to inform clinical judgment, to change clinical behavior, and to reduce unnecessary costs, while maintaining quality of care and patient satisfaction.
Journal of the American Statistical AssociationRegression with Missing<i>X</i>'s: A Review
974 Citations1992Roderick J. A. Little
Regression With Missing X's: A Review Author(s): Roderick J. A.
American Journal of EpidemiologyA Critical Look at Methods for Handling Missing Covariates in Epidemiologic Regression Analyses
886 Citations1995Sander Greenland, William D. Finkle
The authors recommend that epidemiologists avoid using the missing-indicator method and use more sophisticated methods whenever a large proportion of data are missing, and contrast the results of multiple imputation to simple methods in the analysis of a case-control study of endometrial cancer.
JAMAClinical Prediction Rules<subtitle>A Review and Suggested Modifications of Methodological Standards</subtitle>
770 Citations1997Andreas Laupacis
Although clinical prediction rules comply with some methodological criteria, for other criteria, better compliance is needed.
Statistics in MedicinePrognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets
756 Citations2000Ewout W. Steyerberg, Marinus J.C. Eijkemans +2 more
It is found that stepwise selection with a low alpha led to a relatively poor model performance, when evaluated on independent data, and shrinkage methods in full models including prespecified predictors and incorporation of external information are recommended, when prognostic models are constructed in small data sets.
Journal of the American Statistical AssociationRegression With Missing X's: A Review
622 Citations1992Roderick J. A. Little
PubMedRegression models for prognostic prediction: advantages, problems, and suggested solutions.
566 Citations1985Harrell Fe, Lee Kl +2 more
Regression models can make more accurate predictions than other methods such as stratification and recursive partitioning, when model assumptions are thoroughly examined; steps are taken when assumptions are violated; and the method of model formulation does not result in overfitting the data.
Journal of Clinical EpidemiologyStepwise Selection in Small Data Sets A Simulation Study of Bias in Logistic Regression Analysis
503 Citations1999Ewout W. Steyerberg
It is concluded that stepwise selection may result in a substantial bias of estimated regression coefficients of selected covariables, similar to that found in the GUSTO-I trial.
New England Journal of MedicineOutpatient Management without Antibiotics of Fever in Selected Infants
478 Citations1993M. Douglas Baker, Louis M. Bell +1 more
With the use of strict screening criteria, a substantial number of febrile one-to-two-month-old infants can be cared for safely as outpatients and without antibiotics.
The Journal of PediatricsIdentification of infants unlikely to have serious bacterial infection although hospitalized for suspected sepsis
348 Citations1985Ron Dagan, Keith R. Powell +2 more
It is concluded that previously healthy infants younger than 3 months with an acute illness are unlikely to have serious bacterial infection if they have no findings consistent with ear, soft tissue, or skeletal infections and have normal white blood cell and band form counts and normal urine findings.
Journal of the Royal Statistical Society Series C (Applied Statistics)The Use of Resampling Methods to Simplify Regression Models in Medical Statistics
319 Citations1999Willi Sauerbrei
The problems of replication stability, model complexity, selection bias and an over‐optimistic estimate of the predictive value of a model are discussed together with several proposals based on resampling methods, which favour greater simplicity of the final regression model.
PEDIATRICSPractice Guideline for the Management of Infants and Children 0 to 36 Months of Age With Fever Without Source
282 Citations1993Larry J. Baraff, David L. Schriger +5 more
These guidelines do not eliminate all risk or strictly confine antibiotic treatment to children likely to have occult bacteremia and may individualize therapy based on clinical circumstances or adopt a variation of these guidelines based on a different interpretation of the evidence.
BMJWhen should clinical guidelines be updated?
253 Citations2001Paul Shekelle, M P Eccles +2 more
A set of principles and a pragmatic model for assessing whether guidelines need to be updated are proposed and the need for an efficient mechanism for identifying when guidelines require updating is urgent.
Annals of Emergency MedicinePredictors of Occult Pneumococcal Bacteremia in Young Febrile Children
159 Citations1998Nathan Kuppermann, Gary Fleisher +1 more
Patients with OPB were younger, more frequently ill-appearing, and had higher temperatures, WBC, ANC, and ABC than patients without bacteremia, and independent predictors of OPB in children 3 to 36 months of age with temperatures of 39 degrees C or higher treated as outpatients include ANC, temperature, and age younger than 2 years.
Statistica NeerlandicaApplication of Shrinkage Techniques in Logistic Regression Analysis: A Case Study
126 Citations2001Ewout W. Steyerberg, Marinus J.C. Eijkemans +1 more
This work compared the performance of three variants of shrinkage techniques: a linear shrinkage factor, which shrinks all coefficients with the same factor; penalized maximum likelihood (or ridge regression), where a penalty factor is added to the likelihood function such that coefficients are shrunk individually according to the variance of each covariable.
The Pediatric Infectious Disease JournalProbability of bacterial infections in febrile infants less than three months of age
113 Citations1992Larry J. Baraff, SCOTT A. OSLUND +2 more
Medical Decision MakingAssessing Rater Performance without a "Gold Standard" Using Consensus Theory
111 Citations1997Susan C. Weller, N. Clay Mann
With high rater competencies, the model retrieves accurate estimates of competency and case diagnoses even when raters' responses are biased, and a Monte Carlo simulation shows that model appear to be robust even with bias.
Methods of Information in MedicineThe Measurement of Performance in Probabilistic Diagnosis
107 Citations1978J. Hilden, J. Dik F. Habbema +1 more
The measurement of performance in probabilistic diagnosis and the trustworthiness of the exact values of the diagnostic probabilities are compared.
PubMedThe measurement of performance in probabilistic diagnosis. II. Trustworthiness of the exact values of the diagnostic probabilities.
101 Citations1978Jørgen Hilden, J. Dik F. Habbema +1 more
EpidemiologyRedundancy of Single Diagnostic Test Evaluation
91 Citations1999Karel G.M. Moons, Gerri-Anne van Es +4 more
Benefits of single-test evaluation are illustrated and principles of diagnostic research are discussed, whereby diagnostic research should always apply an approach of constructing, extending, and validating diagnostic models in agreement with routine clinical work-up using logistic regression analyses.
Statistica NeerlandicaShrinkage and Penalized Likelihood as Methods to Improve Predictive Accuracy
88 Citations2001Hans C. van Houwelingen
The Journal of PediatricsObservation, history, and physical examination in diagnosis of serious illnesses in febrile children ≤24 months
79 Citations1987Paul L. McCarthy, Robert M. Lembo +3 more
Assessment of appearance in a child with fever, when performed in a careful, integrated, stepwise fashion, improves the sensitivity of the history and physical examination in detecting serious illnesses in febrile children.
PEDIATRICSPredictive Value of Abnormal Physical Examination Findings in Ill-Appearing and Well-Appearing Febrile Children
77 Citations1985Paul L. McCarthy, Robert M. Lembo +3 more
The results, indicating an important interaction between a febrile child's appearance and physical examination findings, are discussed in terms of probability reasoning in clinical decision making.
PEDIATRICSPredictors of Bacteremia in Febrile Children 3 to 36 Months of Age
73 Citations2000Daniel J. Isaacman, Justine Shults +3 more
A logistic regression formula was developed that could be used to develop a unique risk value for each patient based on temperature, gender, and ANC, and it is shown that this approach offers a useful alternative to predictions based on fever and WBC alone.
Acta PaediatricaPrediction of bacterial meningitis in children with meningeal signs: reduction of lumbar punctures
71 Citations2001Rianne Oostenbrink, KGM Moons +3 more
The aim of this study was to derive and validate a clinical rule to predict bacterial meningitis in children with meningeal signs, to guide decisions on the performance of lumbar punctures.
Pediatric Emergency CareSigns of meningeal irritation at the emergency department: How often bacterial meningitis?
67 Citations2001Rianne Oostenbrink, Karel G.M. Moons +4 more
Bacterial meningitis is present in 30% of children with signs of meningeal irritation, and presence of one of the signs ofMeningeal irritated assessed by the pediatrician was related to bacterial meneditis in 39%.
Acta PaediatricaPredicting serious bacterial infection in young children with fever without apparent source
66 Citations2001Sacha E. Bleeker
European Journal of PediatricsA predictive model to estimate the risk of serious bacterial infections in febrile infants
61 Citations1996Rudolphus Berger, Marjolein Y. Berger +3 more
CRP, duration of fever, the “standardized clinical impression score”, a history of diarrhoea and focal signs of infection were the independent, most powerful predictors of SBI in febrile infants, identified by logistic regression analysis.
Acta PaediatricaPredicting serious bacterial infection in young children with fever without apparent source
58 Citations2001SE Bleeker, KGM Moons +3 more
Using multivariate logistic regression and the area under the receiver operating characteristic curve (ROC area), the diagnostic value of predictors for serious bacterial infection was judged, resulting in a risk stratification.
The Pediatric Infectious Disease JournalFever without source
38 Citations1992Larry J. Baraff, SIDNEY I. LEE
Pediatric Emergency CareDuration of fever and its relationship to bacteremia in febrile outpatients three to 36 months old
36 Citations1997Stephen J. Teach, Gary Fleisher
Children with occult bacteremia have significantly shorter durations of fever than patients without bacterenmia, but this difference is small and not clinically useful.
PEDIATRICSShould Blood Cultures Be Obtained in the Evaluation of Young Febrile Children Without Evident Focus of Bacterial Infection? A Decision Analysis of Diagnostic Management Strategies
31 Citations1989Michael S. Kramer, David A. Lane +1 more
The risk of unnecessary hospitalization and intravenous antibiotic treatment of the relatively large number of children whose bacteremia spontaneously resolves appears to outweigh the benefit of preventing serious infectious sequelae in the few children in whom positive blood culture results permit timely intervention.
Archives of Disease in ChildhoodMarkers of serious illness in infants under 6 months old presenting to a children's hospital.
28 Citations1990Peter Hewson, S M Humphries +3 more
Six hundred and eighty two assessments were performed on 641 babies under 6 months of age who presented to the emergency department of the Royal Children's Hospital, Melbourne to try and determine the best markers of serious illness in young infants.