The Role of Formative Evaluation in Implementation Research and the QUERI Experience
Journal of General Internal MedicinePublished 1 February 2006Open access
Cheryl B. Stetler, Marcia W. Legro, Carolyn Wallace, Candice Bowman, Marylou Guihan, Hildi Hagedorn
Citations512
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
This article describes the importance and role of 4 stages of formative evaluation in our growing understanding of how to implement research findings into practice in order to improve the quality of clinical care. It reviews limitations of traditional approaches to implementation research and presents a rationale for new thinking and use of new methods. Developmental, implementation-focused, progress-focused, and interpretive evaluations are then defined and illustrated with examples from Veterans Health Administration Quality Enhancement Research Initiative projects. This article also provides methodologic details and highlights challenges encountered in actualizing formative evaluation within implementation research.
Keywords
Decision SciencesHealth Professions
Case Study Research: Design and Methods
74,912 Citations1984Robert K. Yin
SAGE Publications eBooksQualitative research & evaluation methods
15,249 Citations2002Michael Quinn Patton
Medical Entomology and ZoologyExperimental and Quasi-Experimental Designs for Research
15,238 Citations1963Donald T. Campbell, Julian C. Stanley +1 more
Evaluation: A Systematic Approach
2,952 Citations1979Peter H. Rossi, Mark W. Lipsey +1 more
American Journal of Public HealthWhy Don’t We See More Translation of Health Promotion Research to Practice? Rethinking the Efficacy-to-Effectiveness Transition
1,855 Citations2003Russell E. Glasgow, Edward Lichtenstein +1 more
This work addresses the assumption that effectiveness research naturally and logically follows from successful efficacy research, and offers recommendations to help close the gap between efficacy and effectiveness research and to guide evaluation and possible adoption of new programs.
The Clearing House A Journal of Educational Strategies Issues and IdeasDiffusion of Innovations
1,848 Citations1971Michael T. Hinkemeyer
Utilization-focused evaluation : the new century text
1,474 Citations1997Michael Quinn Patton
BMJComplex interventions: how “out of control” can a randomised controlled trial be?
1,255 Citations2004Penelope Hawe, Alan Shiell +1 more
A radical departure from the way large scale interventions are typically conceptualised is proposed, which could liberate interventions to be responsive to local context and potentially more effective while still allowing meaningful evaluation in controlled designs.
Handbook of Practical Program Evaluation
967 Citations2010Joseph S. Wholey, Harry P. Hatry +1 more
Journal of Advanced NursingGetting evidence into practice: the meaning of `context'
785 Citations2002Brendan McCormack, Alison Kitson +4 more
BMJ Quality & SafetyIngredients for change: revisiting a conceptual framework
716 Citations2002Jo Rycroft‐Malone
A concept analysis has shown that the relationship between the elements and sub-elements and their relative importance need to be better understood when implementing evidence based practice and it is planned to develop it into a practical tool to aid those involved in planning, implementing, and evaluating the impact of changes in health care.
BMJ Quality & SafetyProcess evaluation on quality improvement interventions
341 Citations2003Marlies Hulscher
This paper presents a framework containing features of QI interventions that might influence success, and can be used as a starting point to address all three aspects of process evaluation mentioned above.
JAMAA Qualitative Study of Increasing β-Blocker Use After Myocardial Infarction
299 Citations2001Elizabeth H. Bradley, Eric S. Holmboe +4 more
This study provides a context for understanding efforts to improve care in the hospital setting by describing a taxonomy for classifying and evaluating such efforts and suggests possible elements of successful efforts to increase β-blocker use for patients with AMI.
Annals of Internal MedicineCreating the Evidence Base for Quality Improvement Collaboratives
273 Citations2004Brian S. Mittman
The quality improvement collaborative (QIC) examined by Landon and colleagues in this issue is arguably the health care delivery industry's most important response to quality and safety gaps; it represents substantial investments of time, effort, and funding.
BMJ Quality & SafetyDesigning a quality improvement intervention: a systematic approach
212 Citations2003Marloes Amantia van Bokhoven
Decisions that have to be made during the design process of a quality improvement intervention are visualised, allowing them to serve as a starting point for a systematic evaluation of the intervention.
BMJ Quality & SafetyEffectiveness of quality improvement: learning from evaluations
209 Citations2002Kieran Walshe
The effectiveness of many quality improvement interventions has been studied, and research suggests that most have highly variable effects which depend heavily on the context in which they are used and the way they are implemented.
Journal of Biomedical InformaticsGrounding a new information technology implementation framework in behavioral science: a systematic analysis of the literature on IT use
204 Citations2003Rita Kukafka, Stephen B. Johnson +2 more
A novel framework that is intended to guide synthesis of more than one theoretical perspective for the purpose of planning multi-level interventions to enhance IT use is presented, adapted from PRECEDE/PROCEDE, a conceptual framework used by health planners to direct interventions that account for the multiple determinants of behavior.
Nursing ResearchIntervention Fidelity
184 Citations2004Sheila Judge Santacroce, Lisa M. Maccarelli +1 more
The technology model of intervention fidelity as it applies to psychosocial interventions is described and its application in nursing intervention research is proposed as a way to support accurate conclusions about validity.
BMC Health Services ResearchPRIME – PRocess modelling in ImpleMEntation research: selecting a theoretical basis for interventions to change clinical practice
139 Citations2003Anne Walker, Jeremy Grimshaw +4 more
The Joint Commission Journal on Quality ImprovementGuideline Implementation: What the Literature Doesn’t Tell Us
134 Citations2000Leif I. Solberg
Reviews of guideline implementation trials have focused on how to change the behavior of individual clinicians, with little attention to the impact of practice systems or organizational support of clinician behavior or the role of the practice environmental context within which change is being attempted.
PubMedRole of the organization in translating research into evidence-based practice.
134 Citations2003Cheryl B. Stetler
A conceptual framework is presented to assist those attempting to bring about widespread evidence-based practice and emphasizes the role of culture, capacity, and infrastructure as integral to the long-term, sustained success of such efforts.
Medical CareQuality Enhancement Research Initiative (QUERI)
132 Citations2000John G. Demakis, Lynn McQueen +2 more
The 6-step QUERI process is described and an overview of relevant programmatic details, including QUerI's rigorous review process, and VHA's unique qualifications for establishing a national model for quality improvement are presented.
Evaluation ReviewFormative and Process Evaluation of Health Promotion and Disease Prevention Programs
128 Citations1993Mary-Anne Dehar, Sally Casswell +1 more
Greater application of formative and process evaluation to such programs in the future has the potential to lead to better designed and more effective programs, and improved understanding of the factors influencing program outcomes.
Journal of the American Medical Informatics AssociationHuman Factors Barriers to the Effective Use of Ten HIV Clinical Reminders
118 Citations2003Emily S. Patterson, Anh Duy Nguyen +2 more
Reducing human factors barriers to the use of HIV CRs has the potential to increase use of the CRs and thereby improve the quality of HIV care.
Journal of the American Medical Informatics AssociationOverview of the Veterans Health Administration (VHA) Quality Enhancement Research Initiative (QUERI): Table 1
109 Citations2004Lynn McQueen, Brian S. Mittman +1 more
The mission, history, structure, and activities of QUERI are described, and a brief summary of key findings and impacts are provided.
Personnel PsychologyFORMATIVE EVALUATION: AN INTEGRATIVE PRACTICE MODEL AND CASE STUDY
87 Citations2002Kenneth G. Brown, Megan W. Gerhardt
Social Science & MedicineA framework for evaluating community-based heart disease prevention programs
67 Citations1986David Altman
A framework encompassing the evaluation of process, impact and social relevance guides this discussion, and the goal is to divide the question, "do community-based health programs work?", into more easily answered evaluation questions.
Health Education ResearchThe challenge of evaluating complex interventions: a framework for evaluating media advocacy
61 Citations2002Martine Stead
A comprehensive framework for the evaluation of media advocacy is presented and a series of indicators and research methods for evaluating media advocacy at the levels of formative, process and outcome evaluation are proposed.
Worldviews on Evidence-Based NursingA Qualitative Study to Identify Barriers and Facilitators to Implementation of Pilot Interventions in the Veterans Health Administration (VHA) Northwest Network
54 Citations2004NANCY SHARP, Sandra Pineros +3 more
A top priority for facilitating interventions should be the development of educational materials that teach intervention teams how to anticipate barriers and make plans to address them, as well as identifying and fostering local experts in planning and implementing interventions.
Journal of Health Services Research & PolicyEvaluation of a complex intervention for changing professional behaviour: the Evidence Based Out Reach (EBOR) Trial
51 Citations2002Irwin Nazareth, Nick Freemantle +3 more
Although this study is limited by a post hoc rather than a pre hoc design, it provides a pragmatic approach to understanding the factors influencing the pathway of change in prescribing behaviour in response to academic outreach visits.
Health Promotion PracticeTranslating Efficacy Research to Effectiveness Studies in Practice: Lessons From Research to Promote Smoking Cessation in Community Health Centers
36 Citations2004Jane G. Zapka, Karin Valentine Goins +2 more
A case study of the process evaluation undertaken as part of Quit Together, a randomized trial of the effectiveness of an intervention to improve smoking cessation and relapse prevention among low-income pregnant and postpartum women who receive care at community health centers (CHCs).
Drug and Alcohol DependenceVariations in evidence-based clinical practices in nine United States Veterans Administration opioid agonist therapy clinics
34 Citations2004Mark L Willenbring, Hildi Hagedorn +2 more
Wide variability in clinical practices and outcomes across clinics supports the importance of individualizing quality improvement strategies to address specific performance gaps.
The Joint Commission Journal on Quality ImprovementCan Practice Guidelines Be Transported Effectively to Different Settings? Results from a Multicenter Interventional Study
31 Citations2001Judith H. Lichtman, Sarah A. Roumanis +4 more
Even with a standardized implementation protocol, consistent results across institutions were not obtained when a clinical guideline for chest pain was implemented beyond its original setting, demonstrating the importance of understanding the local factors that influence guideline implementation.
Maternal and Child Health JournalA Formative Evaluation in Maternal and Child Health Practice: The Partners for Life Nutrition Education Program for Pregnant Women
29 Citations2003Neal R. Boyd, Richard A. Windsor
It is important for program developers to use results from formative evaluations to make changes in problem areas prior to implementation of a full-scale impact evaluation.
Evaluation ReviewMany A Slip Between Cup And Lip
25 Citations2003Louise Forsetlund, Kristine Olaisen Talseth +3 more
Whether an intervention for promoting evidence-based public health practice had been delivered as intended and to explore the reasons for its lack of impact are documented.
The American Journal of MedicineQuality improvement studies: the need is there but so are the challenges
23 Citations2000Harlan M. Krumholz, Jeph Herrin
PubMedIntegration of evidence into practice and the change process: fall prevention program as a model.
20 Citations2000Cheryl B. Stetler, Barbara Corrigan +2 more
Details are provided on how multiple sources of evidence were collected and integrated to engage clinical managers and staff in an evolving process designed to both reduce falls and enhance evidence-based thinking.
Health Education & BehaviorUses of Process Evaluation in the Maryland WIC 5-a-Day Promotion Program
20 Citations2000Stephen Havas, Jean Anliker +3 more
Using a crossover design, the authors conducted a 6-month intervention program aimed at increasing fruit and vegetable consumption among women served by the Women, Infants, and Children program in Baltimore City and six Maryland counties.
Applied Nursing ResearchIssues in intervention research: Maintaining integrity
14 Citations1994Karin T. Kirchhoff, Colleen A. Dille
Issues in implementing an intervention study are discussed, using an example of one in which the entire nursing staff of a unit administered the intervention and whether the intervention was indeed administered.
Preventive MedicineFormative evaluation in school-based health promotion investigations
13 Citations1989Richard I. Evans, Bettye E. Raines +1 more
This article presents a case history of the application of formative evaluation drawn from a health promotion project conducted by the authors and includes as background for this case history, definitions offormative evaluation and an analysis of its costs.
PubMedStaff influenza vaccination: lessons learned.
13 Citations2004Sherri L. LaVela, Marcia W. Legro +2 more
The extent to which HCWs promoted safe, quality care for patients with SCI/D as related to their beliefs about, and receipt/non-receipt, of influenza vaccination was examined.
JONA The Journal of Nursing AdministrationImpact of Organizational Redesign
11 Citations1997Judith A. Effken, Cheryl B. Stetler
The authors describe an alternative program evaluation approach that proved successful for assessing and enhancing large-scale organizational change in an acute-care medical center.
PubMedUnderstanding a VA preventive care clinical reminder: lessons learned.
7 Citations2004Carolyn Wallace, Michael Hatzakis +2 more
"lessons learned" about using a computerized clinical reminder (CCR) for influenza in spinal cord injury (SCI) centers in the Veterans Health Administration are presented.
PubMedProvider attitudes and beliefs about clinical practice guidelines.
6 Citations2004Stephen L. Luther, Audrey E. Nelson +1 more
Overall, SCI care providers expressed positive attitudes towards CPG, including the two SCI guidelines included in this study, and nurses represented the largest provider group participating in this survey and consistently expressed the most positive responses.