Making  Causal  Claims  

CES Ottawa

22 October 2012

John Mayne, Ph D Advisor on Public Sector Performance

Adjunct Professor, University of Victoria john.mayne@rogers.com

Outline  

•  A ‘new’ perspective on causality for evaluation

•  Demonstrating contributory causality •  Current issue in contribution analysis

The  Context  

•  Interventions are aimed at bringing about outcomes

•  Interventions are not acting on their own. There may be: –  other events and conditions at play –  other interventions at work –  other relevant contextual factors

•  Many if not most interesting outcomes result from such a mix of actions and context

The  Issue  

•  In this context, what can we say about the relationship between the intervention and observed results? – The intervention ‘caused’ the results? – The intervention made a difference? – The intervention contributed to the results? – The intervention can be attributed to a

specific net result? •  What makes sense?

Improving    educa:on  outcomes  for  girls    

Intervention operates through raising the knowledge, skills and awareness of teachers.   Other relevant factors here might be: •  the willingness of teachers to support the

education of girls •  the support of parents for their daughters

to attend schools and study at home •  the ability of girls to get to the schools •  the adequacy of the schools to

accommodate girls

Conceptualizing  Causality  

What kind of causal relation exists then between an intervention (X) and an impact (Y)? •  Can we say X causes Y? No •  Is X necessary for Y? No •  Is X sufficient for Y? No But we clearly want to make some causal link between the intervention and the impact

Conceptualizing  Causality  

•  As might be imagined, these conundrums have been well addressed in the vast literature on causality. There is an answer!

•  An intervention works as part of a broader causal package. And if it works, then this causal package is indeed sufficient to bring about the impact.

•  Further, if the intervention is ‘working’, then it is an essential part of this causal package.

INUS  Causality  

•  This is the very useful concept of INUS causality and INUS conditions.

•  An Insufficient but Necessary part of a condition that is itself Unnecessary but Sufficient for the occurrence of the effect (Mackie 1974).

•  Mackie argued that much of the time, when we are talking about causality we are in fact talking about INUS causality.

Interven:on  Causality  

•  Thus, an intervention “made a difference” when: – The intervention causal package was

sufficient to bring about the impact, and – The intervention was a necessary

component of the causal package •  The intervention in this case is a contributory cause. On its own it is neither necessary nor sufficient.

The  Interven:on  as  Trigger  

So the intervention is one among several ‘causes’. But is that all? We probably expect more, that the intervention: •  acts as a trigger to start the causal chain

(the spark that lights the fire) •  and may act as sustaining support for

change along the way (gasoline to keep the fire going)

A principal contributory cause

Meaningful  Causal  Ques:ons  

1.  Has the intervention made a difference? o Is the intervention a contributory cause? o What role has the intervention played?

2.  Why has the outcome occurred? o How did the causal factors bring about the

result? o What was the context and the mechanisms? o Were there unintended outcomes?

11

Making  a  difference  

Contributory cause •  The intervention was a necessary

element of the causal package which was sufficient to produce an observed result

•  This is a useful and operational definition of 'making a difference’

Demonstra:ng  Contributory  Cause  

How then to show that the intervention made a difference? •  Connecting to theory-based approaches •  Sufficiency through generative

(process) causality approaches

One approach: contribution analysis

14

Theories  of  change  

•  A results chain with embedded assumptions, risks and other explanatory factors identified

•  An explanation of what has to happen for the results chain to work

Reduction in smoking

Anti-smoking campaign

Assumptions: target is reached, message is heard, message is convincing, no other major influences at work

Risks: target not reached, poor message, peer pressure to smoke very strong

Other Explanatory Factors: reduction due to trend pressure or price increases

15

A Generic Theory of Change

Activities and Outputs

Reach & Reaction

Unintended effects

Changes in knowledge, attitudes skills, opportunities

and incentives

Unintended effects

Behaviour changes

Unintended effects

End Results Theory of Change

Unintended effects

Assumptions: How are behavioural changes in the target population expected to influence the desired end result? What has to happen? What factors influence these processes? Risks: Risks to the link not occurring. Other Explanatory Factors: Socio-economic factors

Assumptions: How are changes in knowledge, attitudes, skills, opportunities and/or incentives expected to change behaviour? What has to happen? What factors influence these processes? Risks: Risks to the link not occurring. Other Explanatory Factors: Peer or trend pressure; other interventions

Assumptions: How does the intervention expect to enhance knowledge, attitudes skills, opportunities and/or incentives? What has to happen? What factors influence these processes? Risks: Risks to the link not occurring. Other Explanatory Factors: Other interventions; self-learning

Assumptions: How and to what extent does the intervention output expect to reach people? What has to happen? What contextual factors influence these processes? Risks: Risks to the link not occurring.

Assumptions: How do external factors influence the realization of the intervention’s ToC? Risks: Risks to the links in the ToC not occurring as expected. Other Explanatory Factors: Socio-economic factors; other interventions

External Influences

Theories  of  Change  and    Causal  Packages  

•  Theories of change are causal packages, models of causal sufficiency and more: – ToC identify supporting factors

(assumptions) and confounding factors (risks)

– ToC set out the relationship between the supporting factors and the intervention

•  ToC is a model of the intervention as a contributing cause

Approaches  to  demonstra:ng  causality  

•  Regularity frameworks that depend on the frequency of association between cause and effect - basis for statistical approaches

•  Counterfactual frameworks that depend on the difference between two otherwise identical cases –basis for experimental and quasi experimental approaches

•  Comparative frameworks that depend on combinations of causes that lead to an effect - basis for ‘configurational’ approaches, such as QCA

•  Generative frameworks that depend on identifying the causal links and ‘mechanisms’ that explain effects –basis for ‘theory based’ and ‘realist’ approaches.

Genera:ve  Causa:on  

•  Process or mechanistic causality •  Tracing the links in between causal

events •  Everyday causality: auto mechanic, air

crashes, forensic work, doctors •  The basis for theory-based evaluation

approaches

19

Contribu:on  analysis:    the  prac:ce  

1.  Set out the causal question 2.  Critically develop the expected theory of

change 3.  Gather the existing evidence verifying

(or not) the theory of change 4.  Assess the resulting contribution story 5.  Seek out additional evidence 6.  Revise & strengthen the contribution

story

Contribu:on  Analysis  

CA shows that an intervention is a contributory cause: •  The expected results occurred •  The causal package is sufficient

–  supporting factors (assumptions) occurred and any other supporting factors have been included in the ToC

–  plausible rival explanations have been accounted for

•  The intervention is necessary for the package to be sufficient

Issues  in  CA  

•  Telling the contribution story (in relation to the counterfactual approach) – CGIAR’s new gold standard for causality?

•  What is rigorous CA? – Needs attention – Patton’s rigorous thinking idea

•  Using CA in planning & designing interventions

A  strong  causal  claim  

•  The intervention causal package was sufficient to bring about a result and the intervention was a necessary part of the package. Further, the intervention was the trigger.

•  “The comprehensive intervention was sufficient to cause the collective results and the intervention actions were the trigger that made it all happen.”

Rigor  in  CA  

•  Needs some work and attention •  Patton suggest that:

“Rigorous thinking supersedes rigorous methods”

•  Discusses criteria for rigorous thinking

Ex-­‐ante  use  of  ToC/CA  

•  Identifying supporting factors as conditions in need of addressing

•  Assessing the likelihood the intervention will work

•  Identifying what needs to be monitored •  As a basis for adaptive management,

using evolving ToC

Main  Messages  •  We expect most interventions are principal

contributory causes – The intervention causal package is sufficient &

the intervention is essential to the package •  Want to also know why the result

occurred; to be able to explain •  ToC are models of the intervention as a

contributory cause—models of causal sufficiency

•  Contribution analysis and other T-B approaches can be used to demonstrate causality

AMribu:on-­‐Contribu:on  

•  Evaluations that do not address the causal issue in a systematic manner are generally weak

•  Need to go beyond gathering opinions on causality

•  There are lots of aspects of the causal issue that can be usefully and credibly explored, using a variety of tools.

Some  References  •  DFID (2012). Broadening the Range of Designs and Methods

for Impact Evaluation, London. Available at http://www.dfid.gov.uk/R4D/Output/189575/Default.aspx

•  Mayne, J. (ed) (2012). Contribution Analysis. Evaluation, Special Issue, 18(3).

•  Mayne, J. (2012). Making Causal Claims. ILAC Brief 26. Available at

•  Mayne, J. (2008). Contribution Analysis: An Approach to Exploring Cause and Effect, ILAC Brief 16. Available at http://www.cgiar-ilac.org/files/publications/briefs/ILAC_Brief16_Contribution_Analysis.pdf

•  Funnell, S. and P. Rogers (2011). Purposeful Program Theory. Jossey-Bass.

27