Reflections on experimental research in medical education
Advances in Health Sciences EducationPublished 21 April 2008
David A. Cook, Thomas J. Beckman
Citations195
SJR quartileQ1
SJR score1.36
SNIP2.29
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Several important yet oft neglected issues in designing experimental research in education are discussed, including randomization controls for only a subset of possible confounders and single-group pretest–posttest studies are susceptible to numerous validity threats.
Abstract
As medical education research advances, it is important that education researchers employ rigorous methods for conducting and reporting their investigations. In this article we discuss several important yet oft neglected issues in designing experimental research in education. First, randomization controls for only a subset of possible confounders. Second, the posttest-only design is inherently stronger than the pretest-posttest design, provided the study is randomized and the sample is sufficiently large. Third, demonstrating the superiority of an educational intervention in comparison to no intervention does little to advance the art and science of education. Fourth, comparisons involving multifactorial interventions are hopelessly confounded, have limited application to new settings, and do little to advance our understanding of education. Fifth, single-group pretest-posttest studies are susceptible to numerous validity threats. Finally, educational interventions (including the comparison group) must be described in detail sufficient to allow replication.
Keywords
Medicine
The Discovery of Grounded Theory
23,416 Citations2017Barney G. Glaser, Anselm L. Strauss
Medical Entomology and ZoologyExperimental and Quasi-Experimental Designs for Research
15,238 Citations1963Donald T. Campbell, Julian C. Stanley +1 more
How to Design and Evaluate Research in Education
13,426 Citations1990Jack R. Fraenkel, Norman E. Wallen
This chapter discusses the research methodologies used in Qualitative and Qualitative Research, as well as some of the techniques used in quantitative and qualitative research.
New England Journal of MedicineRandomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs
3,474 Citations2000John Concato, Nirav R. Shah +1 more
The results of well-designed observational studies (with either a cohort or a case-control design) do not systematically overestimate the magnitude of the effects of treatment as compared with those in randomized, controlled trials on the same topic.
American Journal of Public HealthImproving the Reporting Quality of Nonrandomized Evaluations of Behavioral and Public Health Interventions: The TREND Statement
2,575 Citations2004Don C. Des Jarlais, Cynthia M. Lyles +2 more
The initial version of the Transparent Reporting of Evaluations with Nonrandomized Designs (TREND) statement is presented, which emphasize the reporting of theories used and descriptions of intervention and comparison conditions, research design, and methods of adjusting for possible biases in evaluation studies that use nonrandomized designs.
Psychological BulletinHow we should measure "change": Or should we?
2,496 Citations1970Lee J. Cronbach, Lita Furby
New England Journal of MedicineA Comparison of Observational Studies and Randomized, Controlled Trials
2,352 Citations2000Kjell Benson, Arthur J. Hartz
Little evidence is found that estimates of treatment effects in observational studies reported after 1984 are either consistently larger than or qualitatively different from those obtained in randomized, controlled trials.
Biostatistics : The Bare Essentials
1,502 Citations1993Geoffrey R. Norman, David L. Streiner
New England Journal of MedicineDisciplinary Action by Medical Boards and Prior Behavior in Medical School
839 Citations2005Maxine A. Papadakis, Arianne Teherani +6 more
In this case-control study, disciplinary action among practicing physicians by medical boards was strongly associated with unprofessional behavior in medical school and students with the strongest association were those who were described as irresponsible or as having diminished ability to improve their behavior.
American Journal of Health-System PharmacyBiostatistics: The Bare Essentials
476 Citations1997Michael G. Kendrach
JAMAPhysician Scores on a National Clinical Skills Examination as Predictors of Complaints to Medical Regulatory Authorities
460 Citations2007Robyn Tamblyn, Michał Abrahamowicz +12 more
After adjusting for the predictive ability of the clinical decision-making score in the traditional written examination, the patient-physician communication scores in the clinical skills examination remained significantly predictive of retained complaints.
American Journal of OphthalmologyA comparison of observational studies and randomized, controlled trials
459 Citations2000Kjell Benson, Arthur J. Hartz
Little evidence is found that estimates of treatment effects in observational studies reported after 1984 are either consistently larger than or qualitatively different from those obtained in randomized, controlled trials.
Medical EducationDescription, justification and clarification: a framework for classifying the purposes of research in medical education
457 Citations2008David A. Cook, Georges Bordage +1 more
This review concludes that the quality of medical education research in the United States is under threat due to a combination of factors, including the high level of uncertainty about the sources of abuse, and the low level of awareness of abuse in the medical profession.
Medical TeacherBEME Guide No. 1: Best Evidence Medical Education
335 Citations1999R. M. Harden
Best evidence medical education (BEME) is the implementation, by teachers in their practice, of methods and approaches to education based on the best evidence available, taking into account a number of factors-the QUESTS dimensions.
Academic MedicineThe Research We Still Are Not Doing: An Agenda for the Study of Computer-Based Learning
288 Citations2005David A. Cook
The author updates the agenda for research in CBL (including Web-based learning), saying that CBL is not a panacea, but holds great promise, and that media-comparative studies be replaced by rigorous, theory-guided comparisons of CBL interventions.
BMJEvaluating and researching the effectiveness of educational interventions
265 Citations1999L. Hutchinson
Defining true effectiveness, separating out the part played by the various components of an educational intervention, and clarifying the real cost:benefit ratio are as difficult in educational research as they are in the evaluation of a complex treatment performed on a sample group of people who each have different needs, circumstances, and personalities.
Psychological MethodsThe role of method in treatment effectiveness research: Evidence from meta-analysis.
244 Citations2001David B. Wilson, Mark W. Lipsey
A synthesis of 319 meta-analyses of psychological, behavioral, and educational treatment research was conducted to assess the influence of study method on observed effect sizes relative to that of substantive features of the interventions, highlighting the difficulty of detecting treatment outcomes.
Medical EducationQuality of reporting of experimental studies in medical education: a systematic review
239 Citations2007David A. Cook, Thomas J. Beckman +1 more
Determine the prevalence of essential elements of reporting in experimental studies in medical education and investigate the impact of teacher training on the quality of reports in these studies.
Medical EducationThe value of basic science in clinical diagnosis: creating coherence among signs and symptoms
218 Citations2004Nicole N. Woods, Lee R. Brooks +1 more
This work investigated whether learning basic science mechanisms may have mnemonic value in helping students remember signs and symptoms, in comparison with learning the relation between symptoms and diagnoses directly.
Academic MedicineA Systematic Review of the Methodological Rigor of Studies Evaluating Cultural Competence Training of Health Professionals
209 Citations2005Eboni G. Price‐Haywood, Mary Catherine Beach +10 more
Lack of methodological rigor limits the evidence for the impact of cultural competence training on minority health care quality and more attention should be paid to the proper design, evaluation, and reporting of these training programs.
Advances in Health Sciences EducationBest Evidence Medical Education
189 Citations2000Ronald M. Harden, Janet Grant +2 more
Best Evidence Medical Education is the implementation, by teachers in their practice, of methods and approaches to education based on the best evidence available, which involves a professional judgement by the teacher taking into account a number of factors — the QUESTS dimensions.
JAMAEducational Epidemiology
132 Citations2004Patricia A. Carney, David W. Nierenberg +4 more
The creation of a comprehensive national network of educational epidemiologists could enhance collaboration and the development of a strong educational research foundation.
Academic MedicineImpact of Self-Assessment Questions and Learning Styles in Web-Based Learning: A Randomized, Controlled, Crossover Trial
105 Citations2006David A. Cook, Warren G. Thompson +3 more
Instructional methods that actively engage learners improve learning outcomes and hold implications for both Web-based learning and “traditional” educational activities.
Academic MedicineThe Need for Evidence in Medical Education: The Development of Best Evidence Medical Education as an Opportunity to Inform, Guide, and Sustain Medical Education Research
98 Citations2004W. Dale Dauphinée, Sharon Wood-Dauphinée
The authors conclude that the very nature of being professional in today's social and fiscal context demands that medical educators provide evidence of effectiveness and efficiency of their programs while at the same time BEME and medical education research continue to grow and mature.
Medical EducationVolunteer bias in medical education research: an empirical study of over three decades of longitudinal data
82 Citations2007Clara A. Callahan, Mohammadreza Hojat +1 more
The issue of whether medical education research outcomes can be biased by students' refusal to allow their data to be used in outcomes research should be empirically addressed to assure the validity of research findings.
JAMATrends in Study Methods Used in Undergraduate Medical Education Research, 1969-2007
77 Citations2007Amy Baernstein, Hillary K. Liss +2 more
The annual number of published studies of undergraduate medical education interventions demonstrating methodological rigor has been increasing, however, considerable opportunities for improvement remain.
Academic MedicineA RIME Perspective on the Quality and Relevance of Current and Future Medical Education Research
69 Citations2004Judy A. Shea, Louise Arnold +1 more
The authors argue that the quality and relevance of current work can be enhanced when research is situated within a general framework and questions are asked that are based on literature and theory and push the field toward new knowledge.
BMJGuidelines for evaluating papers on educational interventions
68 Citations1999Education Group for Guidelines on Evaluation
Academic MedicineMoving the Field Forward: Going Beyond Quantitative–Qualitative*
55 Citations2007Georges Bordage
It is proposed that a more productive approach to moving the field forward is to devote greater effort to theory building, theory testing, and programmatic research.
Academic MedicineQuestioning Competence: A Discourse Analysis of Attending Physicians’ Use of Questions to Assess Trainee Competence
38 Citations2007Tara J T Kennedy, Lorelei Lingard
The identification of specific linguistic features of the process of competence assessment by APs provides a framework for faculty development and future study of the function and effects of such discourse patterns.