Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation Alison...
-
Upload
katrina-roberts -
Category
Documents
-
view
218 -
download
0
Transcript of Are Disasters Any Different? Challenges and Opportunities for Post-Disaster Impact Evaluation Alison...
Are Disasters Any Different?
Challenges and Opportunities
for Post-Disaster Impact Evaluation
Alison Buttenheim, Princeton UniversityHoward White, 3ie
Rizwana Siddiqui, PIDEKatie Hsih, Princeton University
April 1, 2009Cairo
2
3ie post-disaster impact evaluation (PDIE) study
Motivation:
• Frequency and severity of natural disasters
• Quantity of assistance provided in post-disaster
settings
• Recent interest from humanitarian and development
sectors in more and better impact evaluation
• Opportunity to use Pakistan ERRA experience as
case study
3
3ie post-disaster impact evaluation (PDIE) study
Goals:
• Review existing approaches to PDIE
• Develop a framework for rigorous PDIE
• Apply framework to the 2005 Pakistan
earthquake case
• Identify a set of principles to guide PDIE
4
Disasters
Natural events:414 reported in 2007
(CRED criteria)
5
Disasters
Natural events:414 reported in 2007
(CRED criteria)
Human consequences:211 million affected
16,847 lives lostUSD 100+ billion damages
6
Disasters
Natural events:414 reported in 2007
(CRED criteria)
Human consequences:211 million affected
16,847 lives lostUSD 100+ billion damages
Institutional responses
7
Post-disaster relief and recovery efforts
• USD 5.9 billion (pledged) for 2005 Pakistan
earthquake
• USD 13.5 billion (pledged) for 2004 Indian
Ocean tsunami
• Actors: Diverse mix of governments,
funders, IFIs, aid agencies, humanitarian
agencies, int’l/local NGOs.
8
How does PD assistance get evaluated?
• Extensive process evaluations
• Multiple levels of analysis (project, agency,
sector, disaster)
• Some joint evaluations (e.g. TEC)
• Review of ALNAP database, etc. suggests
few examples of “rigorous” impact evaluation
9
Why so little focus on IE in PD settings?
10
Why so little focus on IE in PD settings?
“Disasters are
different”
“Disasters are
different”
“Disasters are
different”
“Disasters are
different”“Disasters
are different”
“Disasters are
different”
11
Are disasters any different?
1. Unpredictable, rapid-onset event
2. Proven life-saving measures cannot be
randomized or withheld
3. Mismatch between resources and need
(sometimes)
4. Absence of baseline data (usually)
5. Which counterfactual is the right one?
12
Are disasters any different? Maybe not…
1. Nonrandom exposure to disaster event and
consequences
2. Nonrandom assignment of interventions
3. Fragile states/vulnerable populations
4. Multiple concurrent interventions
5. Which counterfactual is the right one?
13
• Bangladesh floods, 1998
• Hurricane Mitch, 1998
• Indian Ocean tsunami, 2004
• Hurricane Katrina, 2005
Lessons learned from other PDIE experiences
14
Disaster-related time periods
14
Pre-disaster Immediate post-disaster
Post-intervention (1)
Post-intervention (2)
Emergency
Relief
Recovery/Reconstruction
t-1 t0 t1 t2D
ISA
ST
ER
15
Disaster-related populations
15
A Disaster-affected households* that
receive assistance or interventions
B Disaster-affected households that do
not receive assistance or
interventions‡
C Non-affected households† that were
similar to A before the disaster
* or communities (or other unit of analysis)‡ or receive them later, or receive different ones† or less-affected households/communities
16
Time Description Disaster-affected households, treated
t0-t-1
t1-t-1
Disaster-related losses
Restoration to baseline
A0-A-1
A1-A-1
t1-t0
t2-t-1
t2-t0
t2-t1
Recovery from disaster
Sustained restoration to baseline
Sustained recovery from disaster
Persistence of recovery
A1-A0
A2-A-1
A2-A0
A2-A1
Within treatment group, single-difference over time
17
Time Description Disaster-affected households, treated
t0-t-1
t1-t-1
Disaster-related losses
Restoration to baseline
A0-A-1
A1-A-1
t1-t0
t2-t-1
t2-t0
t2-t1
Recovery from disaster
Sustained restoration to baseline
Sustained recovery from disaster
Persistence of recovery
A1-A0
A2-A-1
A2-A0
A2-A1
Within treatment group, single-difference over time
ERRA: “Build Back Better”
18
Time Description Disaster-affected households, treated
t0-t-1
t1-t-1
Disaster-related losses
Restoration to baseline
A0-A-1
A1-A-1
t1-t0
t2-t-1
t2-t0
t2-t1
Recovery from disaster
Sustained restoration to baseline
Sustained recovery from disaster
Persistence of recovery
A1-A0
A2-A-1
A2-A0
A2-A1
Within treatment group, single-difference over time
Problems: Recall bias if no baseline; attribution?
19
Time Description Affected treated-Non-affected
t-1 Baseline (pre-disaster) A-1-C-1
t0 Emergency (immediate post-disaster) A0-C0
t1 Relief/Reconstruction (post-intervention #1) A1-C1
t2 Recovery (post-intervention #2) A2-C2
Cross-sectional, single-difference over treatment groups (A vs. C)
20
Time Description Affected treated-Non-affected
t-1 Baseline (pre-disaster) A-1-C-1
t0 Emergency (immediate post-disaster) A0-C0
t1 Relief/Reconstruction (post-intervention #1) A1-C1
t2 Recovery (post-intervention #2) A2-C2
Cross-sectional, single-difference over treatment groups (A vs. C)
21
Time Description Affected treated-Non-affected
t-1 Baseline (pre-disaster) A-1-C-1
t0 Emergency (immediate post-disaster) A0-C0
t1 Relief/Reconstruction (post-intervention #1) A1-C1
t2 Recovery (post-intervention #2) A2-C2
Cross-sectional, single-difference over treatment groups (A vs. C)
Implied counterfactual: What would “A” households look like if there had been no disaster?
22
Time Description Affected treated-Non-affected
t-1 Baseline (pre-disaster) A-1-C-1
t0 Emergency (immediate post-disaster) A0-C0
t1 Relief/Reconstruction (post-intervention #1) A1-C1
t2 Recovery (post-intervention #2) A2-C2
Cross-sectional, single-difference over treatment groups (A vs. C)
Problems: Is there an appropriate “C” group? If so, were they observed? Attribution?
23
Time Description Affected — Unaffected
t0-t-1
t1-t-1
Disaster-related losses
Restoration to baseline
(A0-A-1)
(A1-A-1)
— (C0-C-1)
— (C1-C-1)
t1-t0
t2-t-1
t2-t0
t2-t1
Recovery from disaster
Sustained restoration to baseline
Sustained recovery from disaster
Persistence of recovery
(A1-A0)
(A2-A-1)
(A2-A0)
(A2-A1)
— (C1-C0)
— (C2-C-1)
— (C2-C0)
— (C2-C1)
Difference-in-difference (A vs. C)
24
Time Description Affected — Unaffected
t0-t-1
t1-t-1
Disaster-related losses
Restoration to baseline
(A0-A-1)
(A1-A-1)
— (C0-C-1)
— (C1-C-1)
t1-t0
t2-t-1
t2-t0
t2-t1
Recovery from disaster
Sustained restoration to baseline
Sustained recovery from disaster
Persistence of recovery
(A1-A0)
(A2-A-1)
(A2-A0)
(A2-A1)
— (C1-C0)
— (C2-C-1)
— (C2-C0)
— (C2-C1)
Difference-in-difference (A vs. C)
Controls time-variant factors that are the same between A & C
25
Time Description Affected treated-Affected control
t-1 Baseline (pre-disaster) A-1-B-1
t0 Emergency (immediate post-disaster) A0-B0
t1 Relief/Reconstruction (post-intervention #1) A1-B1
t2 Recovery (post-intervention #2) A2-B2
Cross-sectional, single-difference over treatment groups (A vs. B)
26
Time Description Affected treated-Affected control
t-1 Baseline (pre-disaster) A-1-B-1
t0 Emergency (immediate post-disaster) A0-B0
t1 Relief/Reconstruction (post-intervention #1) A1-B1
t2 Recovery (post-intervention #2) A2-B2
Cross-sectional, single-difference over treatment groups (A vs. B)
27
Time Description Affected treated-Affected control
t-1 Baseline (pre-disaster) A-1-B-1
t0 Emergency (immediate post-disaster) A0-B0
t1 Relief/Reconstruction (post-intervention #1) A1-B1
t2 Recovery (post-intervention #2) A2-B2
Cross-sectional, single-difference over treatment groups (A vs. B)
Implied counterfactual: What would “A” households look like if there had been no intervention?
28
Time Description Affected treated-Affected control
t-1 Baseline (pre-disaster) A-1-B-1
t0 Emergency (immediate post-disaster) A0-B0
t1 Relief/Reconstruction (post-intervention #1) A1-B1
t2 Recovery (post-intervention #2) A2-B2
Cross-sectional, single-difference over treatment groups (A vs. B)
Problems: How were interventions assigned to A but not to B?
29
Time Description Disaster-affected households
“Treated” — “Control”
t0-t-1
t1-t-1
Disaster-related losses
Restoration to baseline
(A0-A-1)
(A1-A-1)
— (B0-B-1)
— (B1-B-1)
t1-t0
t2-t-1
t2-t0
t2-t1
Recovery from disaster
Sustained restoration to baseline
Sustained recovery from disaster
Persistence of recovery
(A1-A0)
(A2-A-1)
(A2-A0)
(A2-A1)
— (B1-B0)
— (B2-B-1)
— (B2-B0)
— (B2-B1)
Difference-in-difference (A vs. B)
30
Time Description Disaster-affected households
“Treated” — “Control”
t0-t-1
t1-t-1
Disaster-related losses
Restoration to baseline
(A0-A-1)
(A1-A-1)
— (B0-B-1)
— (B1-B-1)
t1-t0
t2-t-1
t2-t0
t2-t1
Recovery from disaster
Sustained restoration to baseline
Sustained recovery from disaster
Persistence of recovery
(A1-A0)
(A2-A-1)
(A2-A0)
(A2-A1)
— (B1-B0)
— (B2-B-1)
— (B2-B0)
— (B2-B1)
Difference-in-difference (A vs. B)
Controls time-variant factors that are the same between A & B
31
World Bank impact evaluation of housing and livelihood grants
32
• Instrumental variable approach to disaster impact:
– Villages at same distance from epicenter, at same elevation and slope had comparable pre-disaster SES
– Villages at different distance from fault line experienced different earthquake severity.
World Bank impact evaluation of housing and livelihood grants
33
• Instrumental variable approach to disaster impact:
– Villages at same distance from epicenter, at same elevation and slope had comparable pre-disaster SES
– Villages at different distance from fault line experienced different earthquake severity.
World Bank impact evaluation of housing and livelihood grants
A1-C1
34
• Instrumental variable approach to disaster impact:
– Villages at same distance from epicenter, at same elevation and slope had comparable pre-disaster SES
– Villages at different distance from fault line experienced different earthquake severity.
• Variation in receipt of relief and recovery funds:
– Between-district variation in implementing agency for housing grant
– Threshold eligibility for livelihoods grant of 5 dependents/households: regression continuity design.
World Bank impact evaluation of housing and livelihood grants
A1-C1
35
• Instrumental variable approach to disaster impact:
– Villages at same distance from epicenter, at same elevation and slope had comparable pre-disaster SES
– Villages at different distance from fault line experienced different earthquake severity.
• Variation in receipt of relief and recovery funds:
– Between-district variation in implementing agency for housing grant
– Threshold eligibility for livelihoods grant of 5 dependents/households: regression continuity design.
World Bank impact evaluation of housing and livelihood grants
A1-C1
A1-B1
36
ERRA impact evaluation case study
1. Evaluation opportunities using existing data & HH sample
– Household data collection at t2
– Retrospective household reports of t0
– Use of ongoing government household surveys (e.g., HIES)
as baseline
– Randomization of some interventions from 2009
37
ERRA impact evaluation case study
2. Evaluation opportunities in a future disaster– Maintain surveillance sample in disaster-prone regions
– Household-level data collection at t0
– Randomized interventions, e.g,
• Timing of interventions:
– Group 1: Housing grant first, followed by livelihood cash grant
– Group 2: Livelihood cash grant first, followed by housing grant
• Conditionality of grants
• Types of interventions, e.g, different formats or recipients of livelihoods cash
grant
38
PDIE Guiding Principles
1. PDIE is necessary to ensure that relief and recovery
funds are appropriately targeted, effective, and efficient.
2. Each phase of a disaster (emergency, relief,
recovery/reconstruction) presents distinct evaluation
challenges and therefore may require a different
evaluation approach or methodology.
3. “Evaluation preparedness” is an important part of
disaster preparedness.
39
PDIE Guiding Principles
5. PDIE should incorporate evaluation of (pre-disaster)
investments in disaster mitigation, prevention, and
resilience.
6. Rigorous PDIE requires the tools and perspectives of
multiple disciplines and sectors.
7. Quantitative PDIE can benefit from the qualitative and
mixed-methods approaches.
40
PDIE Guiding Principles
7. Proportionate changes in outcomes over time and
over groups can be as instructive as changes in
levels.
8. Change-over-time impact evaluations should
recognize two distinct baselines: pre-disaster, and
immediately post-disaster.
41
PDIE Guiding Principles (ct’d)
9. PDIE will be most successful when the goals of the
intervention are clearly defined through a logical
framework or similar model; when the interventions
are appropriately targeted, and when the purpose/use
of the evaluation is clear.
10. Experimental and quasi-experimental approaches are
feasible in PDIE if ethical, logistical and “fit” issues are
adequately addressed.