Impact Evaluation – A Science or an Art?

www.3ieimpact.org Jyotsna Puri

Assessing the impact of humanitarian relief assistance: a challenge and an opportunity

Asian Development Bank - International Initiative for Impact Evaluation

Video Lecture Series

Jyotsna Puri

Jyotsna Puri www.3ieimpact.org

What is an impact evaluation?

The holy grail statement: An xxxx programme caused

a 7% (xx%) increase (amount of change and direction of change) in the income (some measure) of the included population.

We need comparison groups

to know what would have happened in the absence of the programme

Photo: Rong Shoujun / Xinhua Press / Corbis (for the pile of junk)


So in fact

Before After

Project

Comparison

More than one observation required per

group to avoid bias


In humanitarian assistance

• There is a big gap between the requirement and availability of funds. – In 2011, shortage of funds

amounted to $3.4 billion

Critical that we know if programmes are working, how much, why and under what circumstances.


How can impact evaluations help?

• Magnitude of change? How large was the impact? For how long?

• Implementation science: What amount of assistance & with what frequency is it best delivered?

• Best delivery option e.g. cash or kind?

Photo: Claire Pismont

Photo: NRC/Olivia Grey Pritchard


How can impact evaluations help?

• What difference did my agency make? Why?

• Could it have been done in a more cost-effective manner?

• Was there a larger impact on some groups than others? Picture: UNHCR/L. Addario


LESSON 1: IMPACT EVALUATIONS CAN HELPFULLY SUPPLEMENT OTHER EVALUATION EFFORTS IN HUMANITARIAN ASSISTANCE


IE in humanitarian context are hard!

• Need for speed • Imperfect and absent

data • Multiplicity of actors • High co-variability • Ethics

Impact evaluations can be difficult!

• Need for speed • Imperfect and

absent data • Multiplicity of

actors • High co-variability • Ethics

Photo from Progress Report ‘Hunger Under The Cloudless Skies’

The special case of humanitarian work

Presenter

Presentation Notes

Need for speed: Speed and coverage of interventions is of singular importance in humanitarian assistance. Usually there is no time to train teams and plan an evaluation. Arguably this constraint is much more critical in the case of unanticipated emergencies that are rapid-onset rather than slow protracted crises. Multiplicity of actors: There is usually more than one actor (for example, the Indian Ocean Tsunami had 42 international agencies alone and this number does not include national and local agencies.) It is therefore hard to plan impact of a set of cohesive actions for an impact evaluation, ex ante. High co-variability: Large areas are often affected (high co-variability) and it is difficult to identify counterfactuals because it is not easy to find locations or beneficiaries that look similar to the affected population (but were not affected) or were affected but not targeted for reasons unrelated to their conditions. Evaluations of preventive action: For humanitarian assistance related activities that are directed at prevention rather than post-emergency cure, clearly it is difficult, for ethical and technical reasons, to construct an explicit counterfactual for an intervention that seeks to prevent a severe drought from developing into a famine or similarly, an escalation of tension from developing into a full-blown conflict.


RESULT 2: IMPACT EVALUATIONS REQUIRE CAPACITY AND EFFORT BUT HELP REMOVE BIASES IN MEASUREMENT

LESSON 2: IMPACT EVALUATIONS REQUIRE CAPACITY AND EFFORT BUT HELP REMOVE BIASES IN MEASUREMENT


3. Taking stock

• Only 39 impact evaluations - 29 HAD a theory of change.

• But - 23 did not have any balance tests - 29 did not have any power analysis (to

show confidence in results) - Only 5 discuss ethical concerns.

Presenter

Presentation Notes

Most work done in health, nutrition & peace building Importance of evidence in education, livelihoods, WASH, nutrition, health, early recovery


RESULT 3: THERE ARE FEW HIGH QUALITY EVALUATIONS OF HUMANITARIAN ASSISTANCE. THERE IS HIGH NEED.

LESSON 3: THERE ARE FEW HIGH QUALITY EVALUATIONS OF HUMANITARIAN ASSISTANCE. BUT THERE IS HIGH NEED.


ACTED

NGO

Hypothetical Case Study

Presenter

Presentation Notes

Data on outcome variable (haemoglobin level) are taken from health section of the Sri Lanka Income and Expenditure Survey 2006/2007. We also assume that the mean value of haemoglobin level among children aged 6-36 months is 100 g/l (cut-off value to diagnose anaemia is 110 g/l). Standard deviation is 15.


Treatment type

Agency on the ground

Sample size (hhs)

5% min. detectable effect

Comparison group: General food distribution

ACTED 100

Treatment group1: RUSF

Sri Lankan NGO

100

Treatment group 2: General food distribution + RUSF

ACTED and Sri Lankan NGO

100

Ethics: Factorial design – no one is excluded from food

distribution. Data: Not very large data

collection effort. Time: In Chad, it took 18

months.


Lesson 4 Impact evaluations are possible in humanitarian assistance.

– They do not make aid recipients worse off. – They can help in better planning. – They do not require long periods of time or large

datasets.

Photo: Claire Pismont (for the second, landfill looking picture)


Thank you.

[email protected] www.3ieimpact.org

Presenter

Presentation Notes

Most agencies measure the outcome but don’t make the connection with impacts. This is also the tyranny of log frames. Important to be cognizant of this. Ascribing the impacts to the project and not: To business as usual; To another donor/event; Establishing the connection between the outcome and impact.

Impact Evaluation – A Science or an Art?

Documents

Transcript of Impact Evaluation – A Science or an Art?