Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda
Administration and Policy in Mental Health and Mental Health Services ResearchPublished 18 October 2010Open access
Enola K. Proctor, Hiie Silmere, Ramesh Raghavan, Peter S. Hovmand, Greg Aarons, Alicia C. Bunger
Citations8,795
SJR quartileQ1
SJR score1.26
SNIP1.40
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A heuristic, working “taxonomy” of eight conceptually distinct implementation outcomes—acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, and sustainability—along with their nominal definitions is proposed.
Abstract
An unresolved issue in the field of implementation research is how to conceptualize and evaluate successful implementation. This paper advances the concept of "implementation outcomes" distinct from service system and clinical treatment outcomes. This paper proposes a heuristic, working "taxonomy" of eight conceptually distinct implementation outcomes-acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, and sustainability-along with their nominal definitions. We propose a two-pronged agenda for research on implementation outcomes. Conceptualizing and measuring implementation outcomes will advance understanding of implementation processes, enhance efficiency in implementation research, and pave the way for studies of the comparative effectiveness of implementation strategies.
Keywords
Health Professions
BMJCrossing the Quality Chasm: A New Health System for the 21st Century
10,744 Citations2001Alexi Baker
Analyzing health care organizations as complex systems, Crossing the Quality Chasm also documents the causes of the quality gap, identifies current practices that impede quality care, and explores how systems approaches can be used to implement change.
Journal for Healthcare QualityCrossing the Quality Chasm: A New Health System for the 21st Century
8,568 Citations2002Committee on Quality of Health Care in America
American Journal of Public HealthEvaluating the public health impact of health promotion interventions: the RE-AIM framework.
7,277 Citations1999Russell E. Glasgow, Thomas Vogt +1 more
A model for evaluating public health interventions that assesses 5 dimensions: reach, efficacy, adoption, implementation, implementation and maintenance is proposed (termed the RE-AIM model).
International Journal of Man-Machine StudiesUser acceptance of information technology: system characteristics, user perceptions and behavioral impacts
4,721 Citations1993Fred D. Davis
Overall, TAM provides an informative representation of the mechanisms by which design choices influence user acceptance, and should therefore be helpful in applied contexts for forecasting and evaluating user acceptance of information technology.
Implementation research: a synthesis of the literature.
3,784 Citations2005Fixsen Dl, Naoom Sf +2 more
BMJ Quality & SafetyMaking psychological theory useful for implementing evidence based practice: a consensus approach
3,607 Citations2005Susan Michie
A set of behaviour change domains agreed by a consensus of experts is available for use in implementation research and applications of this domain list will enhance understanding of the behaviour change processes inherent in implementation of evidence-based practice.
Administration and Policy in Mental Health and Mental Health Services ResearchImplementation Research in Mental Health Services: an Emerging Science with Conceptual, Methodological, and Training challenges
1,937 Citations2008Enola K. Proctor, John Landsverk +4 more
This paper seeks to advance implementation science in mental health services by over viewing the emergence of implementation as an issue for research, by addressing key issues of language and conceptualization, and by presenting a heuristic skeleton model for the study of implementation processes.
JAMADisseminating Innovations in Health Care
1,911 Citations2003Donald M. Berwick
The theory and research on the dissemination of innovations and applications of that theory to health care are examined and at least 7 recommendations for health care executives who want to accelerate the rate of diffusion of innovations within their organizations are suggested.
JAMAThe Meaning of Translational Research and Why It Matters
1,867 Citations2008Steven H. Woolf
Besides academic centers, foundations, industry, disease-related organizations, and individual hospitals and health systems have also established translational research programs and at least 2 journals are devoted to the topic.
Academy of Management ReviewThe Challenge of Innovation Implementation
1,747 Citations1996Katherine J. Klein, Joann Sorra
Health Education ResearchA review of research on fidelity of implementation: implications for drug abuse prevention in school settings
1,679 Citations2003Linda Dusenbury
A review of the fidelity of implementation research literature spanning a 25-year period found definitions and measures of fidelity were found not to be consistent across studies, and new definitions are proposed.
Clinical Psychology ReviewPROGRAM INTEGRITY IN PRIMARY AND EARLY SECONDARY PREVENTION: ARE IMPLEMENTATION EFFECTS OUT OF CONTROL?
1,521 Citations1998Andrew V. Dane, Barry H. Schneider
The omission of integrity data, particularly measures of adherence, may compromise the internal validity of outcome studies in the prevention literature and be inconsistent with the adaptation of interventions to the needs of receiving communities.
Mental Health Services ResearchMental Health Provider Attitudes Toward Adoption of Evidence-Based Practice: The Evidence-Based Practice Attitude Scale (EBPAS)
1,304 Citations2004Gregory A. Aarons
Attitudes toward adoption of EBPs can be reliably measured and vary in relation to individual differences and service context and EBP implementation plans should include consideration of mental health service provider attitudes as a potential aid to improve the process and effectiveness of dissemination efforts.
Journal of Business ResearchOrganizational innovation adoption: a multi-level framework of determinants and opportunities for future research
1,245 Citations2002R.T. Frambach, Niels Schillewaert
Implementation ScienceSpecifying and reporting complex behaviour change interventions: the need for a scientific method
961 Citations2009Susan Michie, Dean L. Fixsen +2 more
Implementation Science is introducing a policy of initially encouraging and subsequently requiring the scientific reporting of complex behaviour change interventions.
Journal of Public Health Management and PracticeA Glossary for Dissemination and Implementation Research in Health
714 Citations2008Borsika A. Rabin, Ross C. Brownson +3 more
This glossary provides definitions for the key concepts and terms of D&I Research in health (in both public health and clinical settings) to aid in the development of more standardized and established terminology for D &I Research, facilitate the communication across different stakeholders, and ultimately contribute to higher-quality D&i Research.
Journal of General Internal MedicineToward Evidence-Based Quality Improvement.
503 Citations2006Jeremy Grimshaw, Martin Eccles +5 more
There is an imperfect evidence base to support decisions about which guideline dissemination and implementation strategies are likely to be efficient under different circumstances, and decision makers need to use considerable judgment about how best to use the limited resources they have for quality improvement activities.
Administration and Policy in Mental Health and Mental Health Services ResearchImplementation of Evidence-based Practice in Child Welfare: Service Provider Perspectives
455 Citations2007Gregory A. Aarons, Lawrence A. Palinkas
To better understand the implementation process in the child-welfare system, comprehensive home-based services case managers who were actively engaged in implementing an EBP to reduce child neglect in a state youth services system were interviewed.
Evaluation and Program PlanningBuilding capacity and sustainable prevention innovations: a sustainability planning model
409 Citations2004Knowlton Johnson, Carol E. Hays +1 more
Current Directions in Psychological ScienceInnovation Implementation
408 Citations2005Katherine J. Klein, Andrew P. Knight
BMJ Quality & SafetyBeyond usability: designing effective technology implementation systems to promote patient safety
384 Citations2004B-T Karsh
The literature on diffusion of innovations, technology acceptance, organisational justice, participative decision making, and organisational change is reviewed and strategies for promoting successful implementation are provided.
Organization ScienceA Simulation-Based Approach to Understanding the Dynamics of Innovation Implementation
348 Citations2002Nelson P. Repenning
A model is synthesized that describes the process through which participants in an organization develop commitment to using a newly adopted innovation and suggests three new constructs--reversion, regeneration, and the motivation threshold--characterizing the dynamics of implementation, providing an internally consistent theory of how seemingly rational decision rules can create the apparent paradox of innovations that generate early results but fail to produce sustained benefit.
Implementation ScienceHealth researchers in Alberta: an exploratory comparison of defining characteristics and knowledge translation activities
269 Citations2007Mandi Newton, Carole A. Estabrooks +4 more
These findings illustrate important differences between health researchers and provide beginning insights into their professional characteristics and engagement in Mode I and Mode II activities.
American Journal of Health PromotionMeasuring the Diffusion of Innovative Health Promotion Programs
244 Citations1992Allan Steckler, Robert M. Goodman +3 more
Development of six questionnaires to measure the extent to which health promotion programs are successfully disseminated and the instruments, which measure the four steps of the diffusion process, have construct validity since they were develóped within existing theories and are derived from the work of previous researchers.
Journal of Advanced NursingIntegrative nursing research — an important strategy to further the development of nursing science and nursing practice
243 Citations1997Marit Kirkevold
It is argued that integrative nursing research has a great potential for clarifying the theoretical perspective and substance of the nursing discipline as well as making research-based knowledge more accessible to clinical nurses.
American Journal of Preventive MedicineeHealth Evaluation and Dissemination Research
232 Citations2007Russell E. Glasgow
Key challenges in evaluating eHealth intervention and behavior change programs are reviewed, and recommendations for the types of designs, measures, and methods needed to accelerate the integration of proven eHealth programs into practice are made.
Implementation ScienceToward a policy ecology of implementation of evidence-based practices in public mental health settings
225 Citations2008Ramesh Raghavan, Charlotte Lyn Bright +1 more
Health Education QuarterlyDevelopment of Level of Institutionalization Scales for Health Promotion Programs
184 Citations1993Robert M. Goodman, Kenneth R. McLeroy +2 more
The instrument, which is available from the authors, may be used as both a research instrument and a diagnostic tool in assessing the institutionalization of health promotion programs.
Aggression and Violent BehaviorDissemination of evidence-based parenting and family support strategies: Learning from the Triple P—Positive Parenting Program system approach
177 Citations2005Karen Turner, Matthew R. Sanders
Implementation ScienceDifficulties implementing a mental health guideline: an exploratory investigation using psychological theory
170 Citations2007Susan Michie, Stephen Pilling +5 more
The Qualitative ReportMission Drift In Qualitative Research, Or Moving Toward A Systematic Review Of Qualitative Studies, Moving Back To A More Systematic Narrative Review
167 Citations2015Kip Jones
Medical Care Research and ReviewThe Adoption of Innovations by Provider Organizations in Health Care
160 Citations2007Colleen Beecken Rye, John R. Kimberly
A comprehensive census of studies examining the adoption of and disengagement from innovations in health care provider organizations is created to organize these studies into an inductively derived classification scheme and assess the studies' strengths and weaknesses.
PsychotherapyTreatment adherence process research in family therapy: A rationale and some practical guidelines.
143 Citations1996Aaron Hogue, Howard A. Liddle +1 more
Journal of Consulting and Clinical PsychologyStatewide adoption and initial implementation of contingency management for substance-abusing adolescents.
140 Citations2008Scott W. Henggeler, Jason E. Chapman +5 more
The findings support the amenability of public sector practitioners to adopt evidence-based practices and suggest that the predictors of adoption and initial implementation are complex and multifaceted.
Journal of Pain and Symptom ManagementUsability of a Computerized PAINReportIt in the General Public with Pain and People with Cancer Pain
140 Citations2003Diana J. Wilkie, Miriam Judge +4 more
Qualitative and quantitative data from the study support PAINReportIt as a feasible method for patients to self-report their pain and improve directions and practice screens would likely improve patients' independence in completing PAIN reportIt.
Administration and Policy in Mental Health and Mental Health Services ResearchEngaging and Recruiting Counties in an Experiment on Implementing Evidence-based Practice in California
102 Citations2008Patricia Chamberlain, C. Hendricks Brown +7 more
The initial phase of a randomized trial that tests two methods of implementing Multidimensional Treatment Foster Care in 40 non-early adopting California counties supports the feasibility of using a randomized design to rigorously test contrasting implementation models and engaging system leaders to participate in the trial.
Psychiatric Rehabilitation JournalFidelity of supported employment: Lessons learned from the National Evidence-Based Practice Project.
88 Citations2008Gary R. Bond, Gregory J. McHugo +3 more
Examination and interpreting fidelity changes over two years of program implementation suggested four factors that facilitated movement toward high fidelity: Discontinuing non-evidence-based vocational services, making rapid structural changes through administrative action, measuring key process indicators to move toward desired changes, and gradually improving integration of the employment specialists with clinical services.
Journal of Substance Abuse TreatmentCounselor assessments of training and adoption barriers
88 Citations2007Norma G. Bartholomew, George W. Joe +2 more
Training assessments were collected from substance abuse treatment counselors who attended a training on dual diagnosis and another on therapeutic alliance as part of a state-sponsored conference to assess counselor perceptions of training quality, relevance, and resources in relation to its use during the 6 months after the conference.
Journal of Substance Abuse TreatmentAssessing fidelity in individual and family therapy for adolescent substance abuse
79 Citations2007Aaron Hogue, Sarah Dauber +6 more
The TBRS-C demonstrated strong interrater reliability for goal-specific ratings of treatment adherence, and modest reliability forgoal-specific and global ratings of therapist competence, evidence of construct validity, and discriminant validity with an observational measure of therapeutic alliance.
Psychiatric ServicesPreferences for Schizophrenia Treatment Outcomes Among Public Policy Makers, Consumers, Families, and Providers
73 Citations2003Martha Shumway, Teri Saunders +5 more
Policy makers and primary stakeholders place similar value on some of the major goals of schizophrenia treatment, with both groups valuing functional outcomes most highly, but the difference between groups in the importance placed on medication side effects may lead to conflicts in the allocation of resources.
The Journal of Behavioral Health Services & ResearchImplementation of Evidence-Based Practice and Organizational Performance
68 Citations2008Peter S. Hovmand, David F. Gillespie
Results from the simulations show how gains in performance depended on organizations’ initial inertia and initial efficiency and that only the most efficient organizations may see benefits in organizational performance from implementing EBP.
The Journal of Clinical PsychiatryCost-Efficacy of Individual and Combined Treatments for Panic Disorder
68 Citations2007R. Kathryn McHugh, Michael W. Otto +4 more
CBT emerged as the most durable and cost-effective monotherapy and, hence, should be considered as a particularly valuable treatment from the perspective of cost accountability.
Journal of Drug EducationAdoption of Evidence-Based Practices among Substance Abuse Treatment Providers
57 Citations2008Nancy A. Haug, Michael S. Shopshire +3 more
While respondents reported agency support for EBPs, most expressed interest in further training, and the movement toward EBPs in addiction treatment and the need for effective dissemination and training in this area is highlighted.
Journal of Clinical Child & Adolescent PsychologyPredicting Therapist Adherence to a Transported Family-Based Treatment for Youth
53 Citations2005Sonja K. Schoenwald, Elizabeth J. Letourneau +1 more
Therapist demographic variables, professional training and experience, endorsement of the MST model, perceived difficulty and rewards of doing MST, and perceived similarity to treatments previously used did not predict adherence.
The Journal of Behavioral Health Services & ResearchService penetration by persons with severe mental illness: How should it be measured?
46 Citations2002Paul G. Stiles, Roger A. Boothroyd +2 more
The conceptualization and operationalization of service penetration are discussed and an exploratory study is presented using data from the same persons using very different data sources; these data were collected during an evaluation of a Medicaid managed care system in Florida.
Journal of General Internal MedicineUptake of Electronic Prescribing in Community-Based Practices
43 Citations2008Michael A. Fischer, Christine Vogeli +3 more
Multivariable models showed that younger clinicians, pediatricians, and prescribers in larger practices exhibited higher uptake rates as a proportion of total prescriptions, and clinician use of e-prescribing increased steadily in the first 12 months of an initiative sponsoring e-Prescribing systems.
Addictive BehaviorsAcceptability of a computerized alcohol screening and advice routine in an emergency department setting—a patient perspective
38 Citations2004Anna Karlsson, Preben Bendtsen
Whether the computerized advice is sufficient for a change in alcohol habits among excessive drinkers is still to be shown in controlled trials, but the concept could be one means of making preventive measures more feasible in a real-world ED setting.
Health Education QuarterlyDevelopment of Level of Institutionalization Scales for Health Promotion Programs [and] Commentary.
36 Citations1993Robert M. Goodman
Clinical Psychology Science and PracticeWhat happens after the innovation is identified?
35 Citations2002Scott W. Henggeler, Terry Lee
Drug and Alcohol ReviewFeasibility and acceptability of a mental health screening tool and training programme in the youth alcohol and other drug (AOD) sector
32 Citations2007Leanne Hides, Dan I. Lubman +3 more
Preliminary evidence is provided of the feasibility and acceptability of the mental health screening tool to young people and the effectiveness of the training package within the youth AOD sector.
BMC Public HealthCosts of the 'Hartslag Limburg' community heart health intervention
29 Citations2006Emma T Ronckers, Wim Groot +3 more
The current study calculated the costs of a community programme in a detailed and systematic way, allowing the costs to be easily adapted to other countries and regions, and showed that the difference between economic costs and the costs incurred by the coordinating agency can be very large.
Journal of Evaluation in Clinical PracticeOntario doctors’ attitudes toward and use of clinical practice guidelines in oncology
28 Citations2006Ian D. Graham, Melissa Brouwers +2 more
By understanding the relationship between doctors' perceptions of specific guidelines and their subsequent adherence to them, guideline developers will be better positioned to produce quality evidence-based guidelines that doctors will find acceptable, and therefore, be more predisposed to use.
Drug and Alcohol ReviewImplementation of the ‘Fresh Start’ smoking cessation programme to 23 antenatal clinics: a randomized controlled trial investigating two methods of dissemination
20 Citations2001Margaret Cooke, Richard P. Mattick +1 more
Investigation of the effect of two methods of dissemination on the implementation of a smoking cessation programme and use of smoking cessation interventions in antenatal clinics found smoking cessation intervention significantly increased after dissemination.
BMC Family PracticeDiffusion of an e-learning programme among Danish General Practitioners: A nation-wide prospective survey
19 Citations2008Frans Boch Waldorff, Annette Plesner Steenstrup +3 more
Fertility and SterilityOutcome and experience of implementing spinal muscular atrophy carrier screening on sperm donors
17 Citations2010Pamela Callum, Jennifer Iger +3 more
The risk for specific medical problems in donor offspring can be significantly reduced by incorporating new genetic tests, such as spinal muscular atrophy carrier screening, into donor screening practices; however, future efforts should focus on communicating the limitations of genetic screening to donor gamete recipients and on the development of guidelines for implementing new Genetic tests on donors.
Medical Informatics and the Internet in MedicineUse of on-line evidence databases by Australian public health practitioners
11 Citations2004Armita Adily, Johanna Westbrook +2 more
There has been only partial uptake of on-line databases among this population health workforce, particularly according to attainment of relevant postgraduate qualifications, and greater effort to increase its use is recommended to ensure population health does not fall behind hospital-based clinicians.
Oxford University Press eBooksSystematic Reviews in Public Health
6 Citations2006Melissa L McPheeters, Peter A. Briss +2 more