How to Evaluate Complex Interventions
16
How to evaluate complex interventions Astrid Brousselle, Ph.D. Canada research Chair in Evaluation and Health System Improvement (crc-easy.ca) Full professor, community health sciences departement Researcher, Centre de recherche de l’Hôpital Charles-LeMoyne Université de Sherbrooke Toronto, 2015, Program Science Symposium
Transcript of How to Evaluate Complex Interventions
How to evaluate complex interventions
Astrid Brousselle, Ph.D.
Canada research Chair in Evaluation and Health System Improvement (crc-easy.ca)
Full professor, community health sciences departement
Researcher, Centre de recherche de l’Hôpital Charles-LeMoyne
Université de Sherbrooke
Toronto, 2015, Program Science Symposium
What is evaluation?
- Determining the merit, worth and value (judgement, several criteria)
- Of an intervention, or of its components (resources, process, results)
- Systematic inquiry
- Final objective: improvement of the intervention, social betterment, accountability, social agenda, action
Merriam-Webster: “to judge the value or condition of (someone or something) in a careful and thoughtful way”
Normative evaluation Evaluative research
Assessment of the results
Assessment of the process
Strategic analysis
Effect analysis
Economic eval.
Production analysis
Implementation analysis
Logic analysis
SEVERAL QUESTIONS
Assessment of the structure
The problem
Objectives
Activities/services
Resources
Effects
Context
Lots of models
from Shadish Cook and Leviton (1991)
PracticePractice
Values
KnowledgeUtilization
Social programming Problem
Pawson & Tilley
Source: Alkin MC and Christie CA (2004) An evaluation theory tree. In: Alkin MC (ed.) Evaluation Roots. Tracing Theorists’ Views and Influences. Thousand Oaks, CA: SAGE, 12–65. ADAPTED from Shadish Cook and Leviton (1991)
The utilization paradise
The lobbying
zone
The knowledge-
driven swamp
Evaluation models and use
CONTANDRIOPOULOS, D., BROUSSELLE, A. (2012) “Evaluation models and evaluation use?” Evaluation: The International Journal of Theory Research and Practice, vol. 18: 61-77.
Complex Intervention
Complex interventions are characterized by :
• Uncertainty, unpredictability• Interdependence among a large number of actors• Adapting and evolving actors• Emergent outcomes created by the connections or
relationships in the system• Non-linearity between inputs and outputs
(Shiell, Hawe, & Gold, 2008, cited in Zimmerman et al. 2012 )
Analysing complex interventions
Reduction trap / complexity trap
Embracing complexity:
• Accepting paradoxes (not contradictions) and dealing with them
• Analyzing the relations, the “in between”• Accepting uncertainty• Using multiple perspectives
(Zimmerman et al. 2012 )
A statement that appears to be
self-contradictory or silly but may include a latent
truth.
A statement that appears to be
self-contradictory or silly but may include a latent
truth.
Evaluation answers to interventions’ complexity
Logic analysis
Realistic evaluation
Contribution analysis
Economic value of programs
Logic analysis
Source: Rey et al. 2012
Crucial characteristics of the intervention
Effects
Desired effectsDesired effects
Other effects
Critical conditions
Direct logic analysis
Alternatives (interventions)
Effects
Other effects
Critical conditions
Reverse logic analysis
Desired effectsDesired effects
Realistic evaluation
“What works, for whom, under which circumstances?”
Theory
Hypotheses
Observations
Program specification
MechanismsContextsOutcomes
Contribution analysis
Source: Mayne, 2012
Economic value
TCHOUAKET, E., BROUSSELLE, A., FANSI, A., DIONNE, P-A., BERTRAND, E., FORTIN, C. (2013) The economic value of Quebec's water fluoridation program, Journal of Public Health, 26 juin 2013, open access: springerlink.com
What these approaches share
Centrality of the “theory of change”
Importance of the context
Integration of various perspectives
Plausible interpretation (uncertainty)
Valid and rigorous research process
Creativity
http://www.jr-art.net
ReferencesAlkin MC and Christie CA (2004) An evaluation theory tree. In: Alkin MC (ed.) Evaluation Roots. Tracing Theorists’ Views and Influences. Thousand Oaks, CA: SAGE, 12–65.
Champagne F., Contandriopoulos A-P, Brousselle A., Hartz Z., Denis J-L. (2011-2e édition) « L'évaluation dans le domaine de la santé : concepts et méthodes », dans BROUSSELLE A., Champagne F., Contandriopoulos A‑P., Hartz Z. (Eds) L'Évaluation : Concepts et méthodes, Presses de l’Université de Montréal : 49-70.
Contandriopoulos, D., Brousselle, A. (2012) “Evaluation models and evaluation use?” Evaluation: The International Journal of Theory Research and Practice, vol. 18: 61-77.
Mayne J. Contribution analysis: Coming of age? Evaluation 2012; 18(3): 270-280.
Dubois N, Lloyd S, Houle J, Mercier C, Brousselle A, Rey L. Discussion: Practice-based evaluation as a response to address intervention complexity. Canadian Journal of Program Evaluation 2012; 26(3): 105-113.
Pawson, R., & Tilley, N. (2005). Realistic evaluation. In S. Mathison (Ed.), Encyclopedia of evaluation (pp. 362-367). Thousand Oaks, CA: Sage Publications.
Rey L., Brousselle , A., Dedobbeleer, N. (2012) «Logic analysis: testing program theory to better evaluate complex interventions», dans Houle, J., Dubois, N., Lloyd, S., Mercier C., Hartz, Z., Brousselle, A. (Ed.) (2012) « L'évaluation des interventions complexes », numéro spécial, Revue Canadienne d'Évaluation de Programme/ Canadian Journal of Program Evaluation, Volume 26(3): 61-89.
Shadish W.R., Cook T.D., Leviton L.C. (1991) «Good Theory for Social Program Evaluation», in Shadish W.R., Cook T.D., Leviton L.C., Foundations of program evaluation. Theories of practice, Sage Publications: 36-67.
Shiell, A., Hawe, P., & Gold, L. (2008). Complex interventions or complex systems? Implications for health economic evaluation. BMJ, 336(7656), 1281–1283. doi:10.1136/bmj.39569.510521.AD
Tchouaket, E., Brousselle , A., Fansi, A., Dionne, P-A., Bertrand, E., Fortin, C. (2013) The economic value of Quebec's water fluoridation program, Journal of Public Health, 26 juin 2013, open access: springerlink.com
Zimmerman, B., Dubois, N., Houle, N., Lloyd, S., Mercier, C., Brousselle, A. , Rey, L. (2012) «How does complexity impact evaluation?: An Introduction to the Special Issue», numéro spécial, Revue Canadienne d'Évaluation de Programme/ Canadian Journal of Program Evaluation, Volume 26(3):v-xx.
