Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research
EvaluationPublished 1 April 2012
Bruno Marchal, Sara Van Belle, Josefien van Olmen, Tom Hoerée, Guy Kegels
Citations487
SJR quartileQ1
SJR score0.80
SNIP1.28
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This review maps how the concepts of realist evaluation are applied in health systems research and which methodological problems are encountered and found a great diversity in the depth of application of the philosophical concepts, use of terminology and scope of application in the research process.
Abstract
This overview aims to stimulate conceptual and practical discussions to help unlock the full potential of realist evaluation in health systems research. Based on a structured literature search, this review maps how the concepts of realist evaluation are applied in health systems research and which methodological problems are encountered. We found a great diversity in the depth of application of the philosophical concepts, use of terminology and scope of application in the research process. Terms of theory-driven evaluation, theories of change and realist evaluation are often used interchangeably. Diverging views exist regarding the nature of ‘mechanism’ and the difference between mechanism and essential context condition. A lack of methodological guidance was highlighted by many authors. Realist evaluation is slowly gaining traction in health systems research, but more clarity is needed concerning the definitions of mechanisms and context and how the configuration of context, mechanism and intervention can be described and assessed.
Keywords
Decision SciencesHealth Professions
Journal of Health Services Research & PolicyRealist review - a new method of systematic review designed for complex policy interventions
3,230 Citations2005Ray Pawson, Trisha Greenhalgh +2 more
A model of research synthesis designed to work with complex social interventions or programmes, and which is based on the emerging ‘realist’ approach to evaluation is offered, to enable decision-makers to reach a deeper understanding of the intervention and how it can be made to work most effectively.
Academy of Management ReviewA set-theoretic approach to organizational configurations
2,143 Citations2007Peer C. Fiss
It is argued that research on organizational configurations has been limited by a mismatch between theory and methods and introduced set-theoretic methods as a viable alternative for overcoming this mismatch.
BMJDesigning and evaluating complex interventions to improve health care
1,334 Citations2007Neil Campbell, Elizabeth Murray +8 more
The importance of ground work in getting usable results is explained and the effectiveness of complex interventions can be difficult and time consuming.
Medical Entomology and ZoologyTheory-driven evaluations
1,176 Citations1990Huey-Tsyh Chen
After UUO, renal DCs exhibit inflammatory morphological and functional characteristics and are more effective antigen-presenting cells, but they do not directly contribute to tubulointerstitial damage and fibrosis.
American Journal of Public HealthEvidence-Based Public Health: Moving Beyond Randomized Trials
1,075 Citations2004César G. Victora, Jean‐Pierre Habicht +1 more
There is an urgent need to develop evaluation standards and protocols for use in circumstances where RCTs are not appropriate, and for evaluating large-scale interventions, studies with plausibility designs are often the only feasible option and may provide valid evidence of impact.
Choice Reviews OnlineMintzberg on management: inside our strange world of organizations
1,047 Citations1990
American Journal of EvaluationUnpacking Black Boxes: Mechanisms and Theory Building in Evaluation
848 Citations2010Brad Astbury, Frans L. Leeuw
JAMAThe Science of Improvement
789 Citations2008Donald M. Berwick
Recent controversies about the evaluation of rapid response teams provide a case in point and show the importance of adjusting research methods to fit research questions.
New Directions for EvaluationTheory‐based evaluation: Past, present, and future
618 Citations1997Carol H. Weiss
Theory-based evaluation examines conditions of program implementation and mechanisms that mediate between processes and outcomes as a means to understand when and how programs work.
BMJ Quality & SafetyResearch designs for studies evaluating the effectiveness of change and improvement strategies
588 Citations2003M Eccles
A range of experimental designs are discussed including single and multiple arm randomised controlled trials and the use of more complex factorial and block designs and the impact of randomisation at both group and individual levels.
American Journal of EvaluationA Systematic Review of Theory-Driven Evaluation Practice From 1990 to 2009
526 Citations2010Chris L. S. Coryn, Lindsay A. Noakes +2 more
Evaluation and Program PlanningOutcome pattern matching and program theory
474 Citations1989William M. K. Trochim
Hypothetical examples ofpattern matching in program outcome evaluation contexts are presented along with consideration of the value of pattern matching for theory-based research.
Evaluation ReviewEvaluating With Sense
428 Citations1983Huey-Tsyh Chen, Peter H. Rossi
Although the randomized controlled experiment conceptualized as a "black box" approach has dominated the discussions of impact assessment since the classic statements of Campbell and Stanley (1966), the use of theoretical models in connection with impact assessment can both heighten the power of experimental designs and compensate for some of the major deficiencies of quasi-experimental designs.
EvaluationTheories of Change and Realistic Evaluation
375 Citations2007Avril Blamey, Mhairi Mackenzie
Two proponents of theory-based approaches to evaluation that have found favour in the UK in recent years are Theories of Change and Realistic Evaluation are shared and evolving views on the points of connection and digression between the approaches are shared.
Milbank QuarterlyHow Do You Modernize a Health Service? A Realist Evaluation of Whole‐Scale Transformation in London
374 Citations2009Trisha Greenhalgh, Charlotte Humphrey +4 more
This project evaluated a major change effort in inner London, funded by a charitable donation of approximately $21 million, which spanned four large health care organizations, and sought to "modernize" these services with a view to making health care more efficient, effective, and patient centered.
American Journal of Public HealthThe Importance of External Validity
294 Citations2007Allan Steckler, Kenneth R. McLeroy
A new emphasis of the Journal on external validity for appropriate manuscripts is announced and selected aspects of research results translated to public health practice are addressed.
International Journal for Quality in Health CareUnderstanding what works—and why—in quality improvement: the need for theory-driven evaluation
229 Citations2007Kieran Walshe
This point of view is founded on a fundamental misunderstanding of the place of experimental methods in investigating and understanding complex social interventions, which is commonplace particularly among clinicians and biomedical researchers and which can seriously hamper those both researching and implementing these interventions.
Journal of the Operational Research SocietyThe contribution of critical realism as an underpinning philosophy for OR/MS and systems
223 Citations2000John Mingers
This paper proposes that a particular account of the philosophy of science, known as ‘critical realism’, is especially suitable as an underpinning of OR/MS.
Social ForcesThe Multi-Goal, Theory-Driven Approach to Evaluation: A Model Linking Basic and Applied Social Science
217 Citations1980Huey-Tsyh Chen, Peter H. Rossi
In this paper, the reasons for no effect in current program evaluation are analyzed theoretically and an alternative evaluation method, the "multi-goal, theory-driven" approach, is proposed.
Evaluation and Program PlanningThe theory-driven approach to validity
204 Citations1987Huey-Tsyh Chen, Peter H. Rossi
Journal of Advanced NursingCritical realism: a way forward for evaluation research in nursing?
200 Citations2003Phil McEvoy, David Richards
It is argued that critical realism provides a coherent framework for evaluation research that is based on the understanding of causal mechanisms, and is particularly important for frontline services seeking to use evidence-based interventions.
American Journal of EvaluationReconstructing Program Theories: Methods Available and Problems to be Solved
200 Citations2003Frank‐Erik de Leeuw
EvaluationEvidence, Understanding and Complexity
199 Citations2003Marian Barnes, Elizabeth Matka +1 more
Studying the Organisation and Delivery of Health Services
195 Citations2004
EvaluationUsing Realistic Evaluation to Evaluate a Practice-level Intervention to Improve Primary Healthcare for Patients with Long-term Mental Illness
193 Citations2005Richard Byng, Ian Norman +1 more
The method developed for this ‘realistic evaluation’, the mechanisms behind the integration of linked specialist workers and the practical and theoretical issues arising from the use of the realistic evaluation framework are described.
American Journal of EvaluationReconstructing Program Theories: Methods Available and Problems to be Solved
173 Citations2003Frans L. Leeuw
Implementation ScienceA realistic evaluation: the case of protocol-based care
167 Citations2010Jo Rycroft‐Malone, Marina Fontenla +2 more
This approach provided a useful interpretive framework with which to make sense of the multiple factors that were simultaneously at play and being observed through various data sources, and for developing explanatory theory about using standardised care approaches in practice.
BMC Health Services ResearchA realist evaluation of the management of a well- performing regional hospital in Ghana
159 Citations2010Bruno Marchal, McDamien Dedzo +1 more
This case suggests that a well-balanced HRM bundle can stimulate organisational commitment of health workers even with narrow decision spaces, and realist evaluation provides an appropriate approach to increase the usefulness of case studies to managers and policymakers.
New Directions for EvaluationTheory‐based evaluation: Reflections ten years on: Theory‐based evaluation: Past, present, and future
146 Citations2007Patricia Rogers, Carol H. Weiss
Critical Public HealthStrong theory, flexible methods: Evaluating complex community-based initiatives
130 Citations2001Ken Judge, Linda Bauld
It is argued that theory-based evaluation can strengthen programme design and implementation, as well as promote policy and practice learning about the most effective interventions for health improvement, and concludes that sophisticated theory building social change mechanisms in community settings is essential if real learning is to be generated from concerted efforts to achieve social change.
Evaluation and Program PlanningIssues in the theory-driven perspective
128 Citations1989Huey-Tsyh Chen, Peter H. Rossi
BMC Health Services ResearchExposing the key functions of a complex intervention for shared care in mental health: case study of a process evaluation
114 Citations2008Richard Byng, Ian Norman +2 more
Case discussions and improved liaison at times of crisis, rather than for proactive recall, were the key functions of shared care contributing to the success of Mental Health Link.
BMC Public HealthEvaluation of health promotion in schools: a realistic evaluation approach using mixed methods
113 Citations2010Jeanine Pommier, Marie‐Renée Guevel +1 more
A research protocol of a program developed to evaluate HP is presented, taking into account the mechanisms, context and outcomes, as defined in realistic evaluation, adjusted to the authors' own French context using an MM approach.
Journal of Advanced NursingA realist study of the mechanisms of cardiac rehabilitation
104 Citations2005Alexander M. Clark, Heather K. Whelan +2 more
Patients' accounts indicated that the didactic content of cardiac rehabilitation was not strongly linked to longer-term health behaviour change, and the main positive effects of cardiac rehab were related to the effect of participation on mediating social and body-focused mechanisms that were triggered when the rehabilitation setting was perceived to be safe.
BMC Public HealthHow to develop a theory-driven evaluation design? Lessons learned from an adolescent sexual and reproductive health programme in West Africa
94 Citations2010Sara B Van Belle, Bruno Marchal +2 more
Family PracticeWanted--new methodologies for health service research. Is complexity theory the answer?
91 Citations2006David Kernick
This paper explores alternative perspectives and methodological opportunities that arise from viewing health service as a complex non-linear system and may help to explain the limitations of current analytical techniques.
EvaluationBuilding Collaborative Capacity through `Theories of Change'
84 Citations2002Helen Sullivan, Marian Barnes +1 more
Journal of Evaluation in Clinical PracticeComplexity and health – yesterday's traditions, tomorrow's future
83 Citations2009Joachim P. Sturmberg, Carmel M. Martin
‘If it is complex it means the authors don’t really understand it, and the way forward is to break the problem down into its part to make sense of it’.
Evaluation Journal of AustralasiaRealist Evaluation in Practice: Health and Social Work
81 Citations2007Eric Weir, Mansoor A.F. Kazi
This chapter discusses the development of Realism for Practice, and the use of Interpretivist Research Methods in Realist Evaluation.
Implementation ScienceImplementing accountability for reasonableness framework at district level in Tanzania: a realist evaluation
79 Citations2011Stephen Maluka, Peter Kamuzora +5 more
It is found that while the A4R approach to priority setting was helpful in strengthening transparency, accountability, stakeholder engagement, and fairness, the efforts at integrating it into the current district health system were challenging.
The LancetWhat do we mean by rigorous health-systems research?
78 Citations2008Anne Mills, Lucy Gilson +3 more
Health & Social Care in the CommunityThe evaluation of health and social care partnerships: an analysis of approaches and synthesis for the future
75 Citations2006Helen Dickinson
It is suggested that interpreting a wide range of approaches to evaluate health and social care partnerships through a framework of critical realism may overcome a number of these difficulties.
Journal of Nursing ScholarshipA Theory‐Driven Approach to Evaluating Quality of Nursing Care
74 Citations2004Souraya Sidani, Diane Doran +1 more
Determining how these variables influence each other provides a valid and comprehensive understanding of the contribution of nursing within the health care system.
The International Journal of Health Planning and ManagementA realistic approach to the evaluation of the quality management movement in health care systems: a comparison between European and African contexts based on Mintzberg's organizational models
67 Citations2004Pierre Blaise, Guy Kegels
It is suggested that instilling professionalism may be a way forward for the quality movement in Africa to improve patient focus and responsiveness of responsible professionals and to study the wide contextual diversity of the African experience with quality management.
Critical Public HealthTackling health inequalities through partnership working: Learning from a realistic evaluation
62 Citations2000David Evans, Amanda Killoran
The importance of understanding how project mechanisms worked in the context of national and local policy change is emphasized, and lessons are identified for UK Health Improvement Programmes, Health Action Zones and Primary Care Groups.
Journal of Nursing EducationRealist Evaluation as a Framework for the Assessment of Teaching About the Improvement of Care
61 Citations2009Greg Ogrinc, Paul B. Batalden
The unique qualities of the realist evaluation, the basic components and steps in a realists evaluation, and an example that uses this technique to evaluate teaching about improvement of care in a clinical setting are described.
International Journal of Nursing StudiesDeveloping a managed clinical network in palliative care: a realistic evaluation
60 Citations2006Debbie Tolson, Jean McIntosh +2 more
The collaborative approach nurtured by the realistic evaluation framework was found particularly helpful and there was consensus that the evaluation had become integral to the intervention itself.
Journal of Evaluation in Clinical PracticeEvaluating complex public health interventions: theory, methods and scope of realist enquiry
57 Citations2007James B. Connelly
By understanding the ontological and epistemological claims of critical realism, it is possible to transcend the methodological inadequacy of the standard model approach and take a coherent approach to evaluation of complex public health interventions.
Evaluation and Program PlanningThe conceptual framework of the theory-driven perspective
57 Citations1989Huey-Tsyh Chen
Nursing InquiryApplying a realist(ic) framework to the evaluation of a new model of emergency department based mental health nursing practice
54 Citations2010Timothy Wand, Kathryn White +1 more
An overview of critical realism, a modern philosophical perspective that seeks contextualised causal understandings of social phenomena, is provided as a means of obtaining a deeper appreciation of how complex programmes work, for whom they work, and under what circumstances.
BMC Public HealthTurning around an ailing district hospital: a realist evaluation of strategic changes at Ho Municipal Hospital (Ghana)
49 Citations2010Bruno Marchal, McDamien Dedzo +1 more
This study shows how a hospital management team in Ghana succeeded in resuscitating an ailing hospital and showed how a realist evaluation approach could be used in the research of the management of health care organisations to explain how management interventions may or may not work.
EvaluationReview: Promoting Realism in Evaluation
44 Citations1998George Julnes, Melvin M. Mark +1 more
Journal of Evaluation in Clinical PracticeEvaluating change in health care practice: lessons from three studies
40 Citations2003Sally Redfern, Sara Christian +1 more
Three recent evaluations of nursing practice are described as examples to illustrate challenges encountered when evaluating changes in practice developments in health care and the merits of realistic evaluation are suggested.
EvaluationA realistic evaluation of fines for hospital discharges: Incorporating the history of programme evaluations in the analysis
36 Citations2011Ana Manzano
A realist evaluation of financial incentives in English hospital discharge policy is used to illustrate how previous evaluations of the same programme theory were inspected for evidence to construct context, mechanisms and outcomes.
Policy & PoliticsAnalysing process in healthcare: the methodological and theoretical challenges
33 Citations2003Michael Calnan, Ewan Ferlı́e
EvaluationIs Realist Evaluation a Realistic Approach for Complex Reforms?
30 Citations2008Lene Holm Pedersen, Olaf Rieper
Evaluation ReviewAn Application of Theory-Driven Evaluation to a Drop-In Youth Center
30 Citations2000Céline Mercier, Myra Piat +2 more
Results are used to illustrate how a theory-driven approach was relevant for the context and objectives of this evaluation, as well as how it was used to develop knowledge useful for action, social intervention theory, and further research.
American Journal of EvaluationAdvancing the development and application of theory-based evaluation in the practice of public health
29 Citations1999Galen E. Cole
Evaluation and Program PlanningThe validity of conclusions in evaluation research: A further development of Chen and Rossi's theory-driven approach
28 Citations1989Herbert L. Costner
The general conclusion is that despite the more demanding requirements of the modelling strategy, it promises a greater yield of policy relevant conclusions than do evaluation designs in the “black box” tradition.
American Journal of EvaluationAdvancing the Development and Application of Theory-based Evaluation in the Practice of Public Health
27 Citations1999Galen E. Cole
A framework is introduced that illustrates different levels, and processes within each level, that should be considered when constructing program theories, and is valuable for theory-based discrepancy evaluation.
Journal of Management in MedicineA realistic theory of health sector management
23 Citations2000Jim Connelly
The main elements of critical realism are introduced and a conceptual space for the cumulation of critical realist case-studies and managerial craft knowledge is cleared.
BMC Health Services ResearchUsing a realist approach to evaluate smoking cessation interventions targeting pregnant women and young people
20 Citations2010Flora Douglas, Denise Gray +1 more
This study is intended to determine not only the components and processes, but also the possible effectiveness of this set of health interventions, and contribute to the evidence base about smoking cessation interventions aimed at priority groups in Scotland.
European Journal of Clinical NutritionGetting beyond outcomes: a realist approach to help understand the impact of a nutritional intervention during smoking cessation
16 Citations2009Mhairi Mackenzie, P. Koshy +3 more
The benefits that a ‘realist’ approach can bring to an outcome study using the example of a nutritional intervention offered as an adjunct to the existing smoking cessation programmes to limit post-cessation weight gain are discussed.
Health Education ResearchA methodological inquiry into the evaluation of smoking cessation programmes
14 Citations1999Mamoru Kaneko
This paper examines a methodological controversy and aims to show the advantages of introducing an alternative methodological approach, i.e. the 'scientific realist approach', into evaluation studies on health education programmes for smoking cessation.
PubMedDeveloping an evidence base for intermediate care delivered by GPs with a special interest.
12 Citations2005David Kernick, Russell Mannion
The randomised controlled trial by Baker et al 4 in this month's Journal showing no differences in clinical outcomes between orthopaedic hospital and practice-based clinics reflects an early and developing evidence base of the effectiveness of GPwSIs.
Journal of Health Organization and ManagementRealism in evidence based medicine: interpreting the randomised controlled trial
11 Citations2004Jim Connelly
It is demonstrated that by using two realist concepts "mechanism" and "context" a number of misinterpretations of randomised, controlled trials from within the dominant empiricist paradigm may be rectified.
Global Health PromotionPerception de quelques mécanismes favorables à la réduction des inégalités sociales de santé en France
10 Citations2011Valéry Ridde, Anne Guichard
EvaluationSolving Methodological Challenges Using a Theory-driven Evaluation in the Study of Complex Patient Care
7 Citations2002Patricia Grocott, Sarah Cowley +1 more
Two research designs were used in a study of the palliative management of malignant wounds to demonstrate the inherent challenges in producing generalizable knowledge from a complex subject, and the rationale for a form of theory-driven evaluation is explained.
Evaluation and Program PlanningEvaluating a community oriented primary care program: Lessons learned through a theory-oriented approach
4 Citations1997Eunice Rodriguez, June P. Mead
…