The Asia Pacific Disaster Report 2015 – Disasters without Borders
Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation
description
Transcript of Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation
![Page 1: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/1.jpg)
Are Disasters Any Different?
Challenges and Opportunitiesfor Post-Disaster Impact Evaluation
Alison Buttenheim, Princeton UniversityHoward White, 3ie
Rizwana Siddiqui, PIDEKatie Hsih, Princeton University
April 1, 2009Cairo
![Page 2: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/2.jpg)
2
3ie post-disaster impact evaluation (PDIE) study
Motivation:
• Frequency and severity of natural disasters
• Quantity of assistance provided in post-disaster settings
• Recent interest from humanitarian and development sectors in more and better impact evaluation
• Opportunity to use Pakistan ERRA experience as case study
![Page 3: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/3.jpg)
3
3ie post-disaster impact evaluation (PDIE) study
Goals:
• Review existing approaches to PDIE
• Develop a framework for rigorous PDIE
• Apply framework to the 2005 Pakistan earthquake case
• Identify a set of principles to guide PDIE
![Page 4: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/4.jpg)
4
Disasters
Natural events:414 reported in 2007
(CRED criteria)
![Page 5: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/5.jpg)
5
Disasters
Natural events:414 reported in 2007
(CRED criteria)
Human consequences:211 million affected
16,847 lives lostUSD 100+ billion damages
![Page 6: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/6.jpg)
6
Disasters
Natural events:414 reported in 2007
(CRED criteria)
Human consequences:211 million affected
16,847 lives lostUSD 100+ billion damages
Institutional responses
![Page 7: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/7.jpg)
7
Post-disaster relief and recovery efforts
• USD 5.9 billion (pledged) for 2005 Pakistan earthquake
• USD 13.5 billion (pledged) for 2004 Indian Ocean tsunami
• Actors: Diverse mix of governments, funders, IFIs, aid agencies, humanitarian agencies, int’l/local NGOs.
![Page 8: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/8.jpg)
8
How does PD assistance get evaluated?
• Extensive process evaluations
• Multiple levels of analysis (project, agency, sector, disaster)
• Some joint evaluations (e.g. TEC)
• Review of ALNAP database, etc. suggests few examples of “rigorous” impact evaluation
![Page 9: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/9.jpg)
9
Why so little focus on IE in PD settings?
![Page 10: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/10.jpg)
10
Why so little focus on IE in PD settings?
“Disasters are
different”
“Disasters are
different”
“Disasters are
different”“Disasters
are different”“Disasters
are different”
“Disasters are
different”
![Page 11: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/11.jpg)
11
Are disasters any different?
1. Unpredictable, rapid-onset event
2. Proven life-saving measures cannot be randomized or withheld
3. Mismatch between resources and need (sometimes)
4. Absence of baseline data (usually)
5. Which counterfactual is the right one?
![Page 12: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/12.jpg)
12
Are disasters any different? Maybe not…
1. Nonrandom exposure to disaster event and consequences
2. Nonrandom assignment of interventions
3. Fragile states/vulnerable populations
4. Multiple concurrent interventions
5. Which counterfactual is the right one?
![Page 13: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/13.jpg)
13
• Bangladesh floods, 1998
• Hurricane Mitch, 1998
• Indian Ocean tsunami, 2004
• Hurricane Katrina, 2005
Lessons learned from other PDIE experiences
![Page 14: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/14.jpg)
14
Disaster-related time periods
14
Pre-disaster Immediate post-disaster
Post-intervention (1)
Post-intervention (2)
Emergency
Relief
Recovery/Reconstruction
t-1 t0 t1 t2D
ISASTER
![Page 15: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/15.jpg)
15
Disaster-related populations
15
A Disaster-affected households* that receive assistance or interventions
B Disaster-affected households that do not receive assistance or interventions‡
C Non-affected households† that were similar to A before the disaster
* or communities (or other unit of analysis)‡ or receive them later, or receive different ones† or less-affected households/communities
![Page 16: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/16.jpg)
16
Time Description Disaster-affected households, treated
t0-t-1
t1-t-1
Disaster-related losses
Restoration to baseline
A0-A-1
A1-A-1
t1-t0
t2-t-1
t2-t0
t2-t1
Recovery from disaster
Sustained restoration to baseline
Sustained recovery from disaster
Persistence of recovery
A1-A0
A2-A-1
A2-A0
A2-A1
Within treatment group, single-difference over time
![Page 17: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/17.jpg)
17
Time Description Disaster-affected households, treated
t0-t-1
t1-t-1
Disaster-related losses
Restoration to baseline
A0-A-1
A1-A-1
t1-t0
t2-t-1
t2-t0
t2-t1
Recovery from disaster
Sustained restoration to baseline
Sustained recovery from disaster
Persistence of recovery
A1-A0
A2-A-1
A2-A0
A2-A1
Within treatment group, single-difference over time
ERRA: “Build Back Better”
![Page 18: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/18.jpg)
18
Time Description Disaster-affected households, treated
t0-t-1
t1-t-1
Disaster-related losses
Restoration to baseline
A0-A-1
A1-A-1
t1-t0
t2-t-1
t2-t0
t2-t1
Recovery from disaster
Sustained restoration to baseline
Sustained recovery from disaster
Persistence of recovery
A1-A0
A2-A-1
A2-A0
A2-A1
Within treatment group, single-difference over time
Problems: Recall bias if no baseline; attribution?
![Page 19: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/19.jpg)
19
Time Description Affected treated-Non-affected
t-1 Baseline (pre-disaster) A-1-C-1
t0 Emergency (immediate post-disaster) A0-C0
t1 Relief/Reconstruction (post-intervention #1) A1-C1
t2 Recovery (post-intervention #2) A2-C2
Cross-sectional, single-difference over treatment groups (A vs. C)
![Page 20: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/20.jpg)
20
Time Description Affected treated-Non-affected
t-1 Baseline (pre-disaster) A-1-C-1
t0 Emergency (immediate post-disaster) A0-C0
t1 Relief/Reconstruction (post-intervention #1) A1-C1
t2 Recovery (post-intervention #2) A2-C2
Cross-sectional, single-difference over treatment groups (A vs. C)
![Page 21: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/21.jpg)
21
Time Description Affected treated-Non-affected
t-1 Baseline (pre-disaster) A-1-C-1
t0 Emergency (immediate post-disaster) A0-C0
t1 Relief/Reconstruction (post-intervention #1) A1-C1
t2 Recovery (post-intervention #2) A2-C2
Cross-sectional, single-difference over treatment groups (A vs. C)
Implied counterfactual: What would “A” households look like if there had been no disaster?
![Page 22: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/22.jpg)
22
Time Description Affected treated-Non-affected
t-1 Baseline (pre-disaster) A-1-C-1
t0 Emergency (immediate post-disaster) A0-C0
t1 Relief/Reconstruction (post-intervention #1) A1-C1
t2 Recovery (post-intervention #2) A2-C2
Cross-sectional, single-difference over treatment groups (A vs. C)
Problems: Is there an appropriate “C” group? If so, were they observed? Attribution?
![Page 23: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/23.jpg)
23
Time Description Affected — Unaffected
t0-t-1
t1-t-1
Disaster-related losses
Restoration to baseline
(A0-A-1)
(A1-A-1)
— (C0-C-1)
— (C1-C-1)
t1-t0
t2-t-1
t2-t0
t2-t1
Recovery from disaster
Sustained restoration to baseline
Sustained recovery from disaster
Persistence of recovery
(A1-A0)
(A2-A-1)
(A2-A0)
(A2-A1)
— (C1-C0)
— (C2-C-1)
— (C2-C0)
— (C2-C1)
Difference-in-difference (A vs. C)
![Page 24: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/24.jpg)
24
Time Description Affected — Unaffected
t0-t-1
t1-t-1
Disaster-related losses
Restoration to baseline
(A0-A-1)
(A1-A-1)
— (C0-C-1)
— (C1-C-1)
t1-t0
t2-t-1
t2-t0
t2-t1
Recovery from disaster
Sustained restoration to baseline
Sustained recovery from disaster
Persistence of recovery
(A1-A0)
(A2-A-1)
(A2-A0)
(A2-A1)
— (C1-C0)
— (C2-C-1)
— (C2-C0)
— (C2-C1)
Difference-in-difference (A vs. C)
Controls time-variant factors that are the same between A & C
![Page 25: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/25.jpg)
25
Time Description Affected treated-Affected control
t-1 Baseline (pre-disaster) A-1-B-1
t0 Emergency (immediate post-disaster) A0-B0
t1 Relief/Reconstruction (post-intervention #1) A1-B1
t2 Recovery (post-intervention #2) A2-B2
Cross-sectional, single-difference over treatment groups (A vs. B)
![Page 26: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/26.jpg)
26
Time Description Affected treated-Affected control
t-1 Baseline (pre-disaster) A-1-B-1
t0 Emergency (immediate post-disaster) A0-B0
t1 Relief/Reconstruction (post-intervention #1) A1-B1
t2 Recovery (post-intervention #2) A2-B2
Cross-sectional, single-difference over treatment groups (A vs. B)
![Page 27: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/27.jpg)
27
Time Description Affected treated-Affected control
t-1 Baseline (pre-disaster) A-1-B-1
t0 Emergency (immediate post-disaster) A0-B0
t1 Relief/Reconstruction (post-intervention #1) A1-B1
t2 Recovery (post-intervention #2) A2-B2
Cross-sectional, single-difference over treatment groups (A vs. B)
Implied counterfactual: What would “A” households look like if there had been no intervention?
![Page 28: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/28.jpg)
28
Time Description Affected treated-Affected control
t-1 Baseline (pre-disaster) A-1-B-1
t0 Emergency (immediate post-disaster) A0-B0
t1 Relief/Reconstruction (post-intervention #1) A1-B1
t2 Recovery (post-intervention #2) A2-B2
Cross-sectional, single-difference over treatment groups (A vs. B)
Problems: How were interventions assigned to A but not to B?
![Page 29: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/29.jpg)
29
Time Description Disaster-affected households
“Treated” — “Control”
t0-t-1
t1-t-1
Disaster-related losses
Restoration to baseline
(A0-A-1)
(A1-A-1)
— (B0-B-1)
— (B1-B-1)
t1-t0
t2-t-1
t2-t0
t2-t1
Recovery from disaster
Sustained restoration to baseline
Sustained recovery from disaster
Persistence of recovery
(A1-A0)
(A2-A-1)
(A2-A0)
(A2-A1)
— (B1-B0)
— (B2-B-1)
— (B2-B0)
— (B2-B1)
Difference-in-difference (A vs. B)
![Page 30: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/30.jpg)
30
Time Description Disaster-affected households
“Treated” — “Control”
t0-t-1
t1-t-1
Disaster-related losses
Restoration to baseline
(A0-A-1)
(A1-A-1)
— (B0-B-1)
— (B1-B-1)
t1-t0
t2-t-1
t2-t0
t2-t1
Recovery from disaster
Sustained restoration to baseline
Sustained recovery from disaster
Persistence of recovery
(A1-A0)
(A2-A-1)
(A2-A0)
(A2-A1)
— (B1-B0)
— (B2-B-1)
— (B2-B0)
— (B2-B1)
Difference-in-difference (A vs. B)
Controls time-variant factors that are the same between A & B
![Page 31: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/31.jpg)
31
World Bank impact evaluation of housing and livelihood grants
![Page 32: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/32.jpg)
32
• Instrumental variable approach to disaster impact:
– Villages at same distance from epicenter, at same elevation and slope had comparable pre-disaster SES
– Villages at different distance from fault line experienced different earthquake severity.
World Bank impact evaluation of housing and livelihood grants
![Page 33: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/33.jpg)
33
• Instrumental variable approach to disaster impact:
– Villages at same distance from epicenter, at same elevation and slope had comparable pre-disaster SES
– Villages at different distance from fault line experienced different earthquake severity.
World Bank impact evaluation of housing and livelihood grants
A1-C1
![Page 34: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/34.jpg)
34
• Instrumental variable approach to disaster impact:
– Villages at same distance from epicenter, at same elevation and slope had comparable pre-disaster SES
– Villages at different distance from fault line experienced different earthquake severity.
• Variation in receipt of relief and recovery funds:– Between-district variation in implementing agency for
housing grant – Threshold eligibility for livelihoods grant of 5
dependents/households: regression continuity design.
World Bank impact evaluation of housing and livelihood grants
A1-C1
![Page 35: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/35.jpg)
35
• Instrumental variable approach to disaster impact:
– Villages at same distance from epicenter, at same elevation and slope had comparable pre-disaster SES
– Villages at different distance from fault line experienced different earthquake severity.
• Variation in receipt of relief and recovery funds:– Between-district variation in implementing agency for
housing grant – Threshold eligibility for livelihoods grant of 5
dependents/households: regression continuity design.
World Bank impact evaluation of housing and livelihood grants
A1-C1
A1-B1
![Page 36: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/36.jpg)
36
ERRA impact evaluation case study
1. Evaluation opportunities using existing data & HH sample
– Household data collection at t2
– Retrospective household reports of t0
– Use of ongoing government household surveys (e.g., HIES) as baseline
– Randomization of some interventions from 2009
![Page 37: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/37.jpg)
37
ERRA impact evaluation case study
2. Evaluation opportunities in a future disaster– Maintain surveillance sample in disaster-prone regions
– Household-level data collection at t0
– Randomized interventions, e.g,• Timing of interventions:
– Group 1: Housing grant first, followed by livelihood cash grant
– Group 2: Livelihood cash grant first, followed by housing grant
• Conditionality of grants
• Types of interventions, e.g, different formats or recipients of livelihoods cash grant
![Page 38: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/38.jpg)
38
PDIE Guiding Principles
1. PDIE is necessary to ensure that relief and recovery funds are appropriately targeted, effective, and efficient.
2. Each phase of a disaster (emergency, relief, recovery/reconstruction) presents distinct evaluation challenges and therefore may require a different evaluation approach or methodology.
3. “Evaluation preparedness” is an important part of disaster preparedness.
![Page 39: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/39.jpg)
39
PDIE Guiding Principles
5. PDIE should incorporate evaluation of (pre-disaster) investments in disaster mitigation, prevention, and resilience.
6. Rigorous PDIE requires the tools and perspectives of multiple disciplines and sectors.
7. Quantitative PDIE can benefit from the qualitative and mixed-methods approaches.
![Page 40: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/40.jpg)
40
PDIE Guiding Principles
7. Proportionate changes in outcomes over time and over groups can be as instructive as changes in levels.
8. Change-over-time impact evaluations should recognize two distinct baselines: pre-disaster, and immediately post-disaster.
![Page 41: Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation](https://reader036.fdocuments.in/reader036/viewer/2022062814/56816841550346895dde1328/html5/thumbnails/41.jpg)
41
PDIE Guiding Principles (ct’d)
9. PDIE will be most successful when the goals of the intervention are clearly defined through a logical framework or similar model; when the interventions are appropriately targeted, and when the purpose/use of the evaluation is clear.
10. Experimental and quasi-experimental approaches are feasible in PDIE if ethical, logistical and “fit” issues are adequately addressed.