1

Process tracing for impact evaluationPotentials and Pitfalls

Melanie PuntonUK Evaluation Society Conference (12 May 2015)

2

Punton & Welle (2015) Straws-in-the-wind, Hoops and Smoking Guns: What can Process Tracing Offer to Impact Evaluation? http://www.ids.ac.uk/publication/straws-in-the-wind-hoops-and-smoking-guns-what-can-process-tracing-offer-to-impact-evaluation

Stedman-Bryce (2013) Health for All: Towards Free Universal Health Care in Ghanahttp://policy-practice.oxfam.org.uk/publications/effectiveness-review-health-for-all-towards-free-universal-health-care-in-ghana-306376

3

Process Tracing:

A qualitative method for assessing causal inference within a single case

Sits in the ‘evaluator’s toolbox’ alongside• Theory-based evaluation• Contribution analysis• Realist evaluation

4

Why process tracing?

0 Useful where pathways of change are uncertain 0Tells you WHY change happened0 Rigorous approach through ex-post design without a

control group

0Time intensive0 Requires the outcome to be known0 Uncharted territory

5

A brief history of process-tracing

0 A method stemming from the social and political sciences. Major contributors: Alexander George, Andrew Bennett, David Collier

0 Originally used to explain historical events, e.g. the cause of the Cuban Missile Crisis (Allison and Zelikow 1999)

0 A ‘contested method’0 Beach and Pederson’s book (2013): detailed theory and

step-by-step guidance0 Limited examples of its application in evaluation

NB: See Punton & Welle (2015) for references

6

Counterfactual framework

Generative framework

Experimental (RCTs)Quasi-experimental

Process tracingRealist evaluation

Intervention A

Outcome B????

Opening up the black box

7

Causal mechanisms

Intervention A

Outcome B????

Causal mechanism

Generative causal framework

8

Causal mechanisms

Part 3Part 2Part 1

Intervention A

Entity 1 Entity 2 Entity 3

Activity 1 Activity 2 Activity 3

Outcome B

9

The Universal Health Care Campaign in Ghana

0Collaborative advocacy effort by civil society to promote universal free access to health care

0Core funding from Oxfam until 20130Lobbying, mobilising, media and research0Effectiveness review in 2012-13, using elements of

process tracing

Stedman-Bryce (2013) Effectiveness Review: Health for All: Towards Free Universal Health Care in Ghanahttp://policy-practice.oxfam.org.uk/publications/effectiveness-review-health-for-all-towards-free-universal-health-care-in-ghana-306376

10

A (simple) example of a process tracing mechanism

Universal Health Care

campaign

Free universal

health care

Civil society

Conduct coordinated advocacy activities for free universal health care

Part 1

Public

Part 2

Become aware of the limitations of current health care financing

Public

Part 3

Demand free universal health care from government actors

Govern-ment

actors

Part 4

Increasingly support free universal health care, based on desire for public support

Govern-ment

Part 5

Amend policies and processes to move towards free universal health care

Activity

Entity

11

Differences between a mechanism and a theory of change…

0No leaps in logic between the parts0Every part should be necessary0Every part should be observable and empirically

measurable

12

Collecting evidence

• Examples of coordinated advocacy linked to campaign (e.g. events, platforms, meetings)

• Interview data suggesting that the campaign increased knowledge about and interest in healthcare reform among civil society actors

• Number of advocacy events held before the campaign started, compared to number held during campaign

Hypothesis: civil society conducts coordinated advocacy for free healthcare as a result of the Campaign

13

Collecting evidence

• Examples of coordinated advocacy taking place before the campaigns

• Interviews suggesting that the campaign made little difference to activities

Alternative hypothesis: The advocacy campaign made little difference to the coordination or activities of CSOs

14

“We balance probabilities and choose the most likely. It is the scientific use of the imagination.” – The Hound of the Baskervilles

15

Assessing evidence: four tests• Analogous to a criminal trial• The evidence for each hypothesis

is weighed….• …in order to increase the

researcher’s confidence in the hypothesis.

• Bayesian probability logic followed to assess the strength of each part of the causal chain.

• Evidence examined using concepts of necessity and sufficiency.

16

Assessing evidence: four tests

Straw in the wind test

Evidence: hotel receipts, suggestive text messages- Neither necessary

nor sufficient

Hypothesis: John shot Mary

Hoop test

Evidence: John lacks an alibi- Necessary, but not

sufficient

Smoking gun test

Evidence: John was found holding a gun over Mary’s body- Sufficient, but not

necessary

Double decisive test

Evidence: A tamper-proof CCTV camera recorded the murder- Necessary and

sufficient

17

Assessing evidence: four testsU

niqu

enes

s of

evi

denc

e

Low (evidence is insuffi cient for h )

High (evidence is suffi cient for h )

Hig

h(e

vide

nce

is

nece

ssar

y fo

r h

)

Low

(evi

denc

e is

not

ne

cess

ary

for

h)

Hoop tests Double-decisive

Straw in the wind tests

Smoking gun tests

18

Assessing evidence: four testsStraw in the wind test

Hypothesis: The Campaign significantly increased the capacity of member CSOs to plan and work together on the issue of free health care

Evidence:1. Campaign members were given the opportunity to

discuss and debate a report on the complex National Health Insurance Scheme, in order to increase their understanding on the topic

2. Before the report was published, there was little coordinated advocacy for free universal healthcare

Increase the plausibility of the hypothesis but do not prove it or disprove alternative hypotheses. BUT together they provide stronger evidence than they do alone

19

Assessing evidence: four tests

Hoop test

Hypothesis: The Campaign significantly increased the capacity of member CSOs to plan and work together on the issue of free health care

Evidence: Until the campaign started, there was little notable coordinated advocacy on this issue.

Not enough on its own to prove the hypothesis, BUT evidence must pass the test in order to keep the hypothesis under consideration

20

Smoking gun test

Hypothesis: The National Health Insurance Authority in Ghana revised methodology for calculating membership because of pressure created by the campaign

Evidence: Ghana delegation at international meeting stated that the campaign’s report ‘prompted us to revise our figures’ (despite initially publically dismissing the report)

Evidence is sufficient to give high confidence that the hypothesis is true. BUT this type of evidence is hard to come by.

Assessing evidence: four tests

21

The end goal…

0Researcher can express level of confidence in each part of the mechanism

0Mechanism is only as strong as its weakest part0 If the researcher is confident in the whole

mechanism…0…the researcher can express confidence that the

mechanism holds in that particular case: that A caused B, and why.

22

Why process tracing?0 Useful where pathways of change are uncertain 0 Rigorous approach through ex-post design without a control

group0Tells you WHY change happened0Useful in a mixed methods design, or in combination with e.g.

Contribution Analysis?

0 Time intensive0 Requires the outcome to be known0Uncharted territory0 Risk of one weak link causing the whole mechanism to be

rejected

23

Questions?

melanie.punton@itad.com

Punton & Welle (2015) Straws-in-the-wind, Hoops and Smoking Guns: What can Process Tracing Offer to Impact Evaluation? http://www.ids.ac.uk/publication/straws-in-the-wind-hoops-and-smoking-guns-what-can-process-tracing-offer-to-impact-evaluation