Felix Naschold University of Wyoming Christopher B. Barrett
Cornell University May 2012 seminar presentation University of
Sydney A stochastic dominance approach to program evaluation And an
application to child nutritional status in arid and semi-arid
Kenya
Slide 2
Motivation 1. Program Evaluation Methods By design they focus
on mean Ex: average treatment effect (ATE) In practice, often
interested in broader distributional impact Limited possibility for
doing this by splitting sample 2. Stochastic dominance By design,
look at entire distribution Now commonly used in snapshot welfare
comparisons But not for program evaluation. Ex:
differences-in-differences 3. This paper merges the two
Diff-in-Diff (DD) evaluation using stochastic dominance (SD) to
compare changes in distributions over time between intervention and
control populations 2
Slide 3
Main Contributions 1. Proposes DD-based SD method for program
evaluation 2. First application to evaluating welfare changes over
time 3. Specific application to new dataset on changes in child
nutrition in arid and semi-arid lands (ASAL) of Kenya Unique, large
dataset of 600,000+ observations collected by the Arid Lands
Resource Management Project (ALRMP II) in Kenya (One of) first to
use Z-scores of Mid-upper arm circumference (MUAC) 3
Slide 4
Main Results 4 1. Methodology (relatively) straight-forward
extension of SD to dynamic context: static SD results carry over
Interpretation differs (as based on cdfs) Only feasible up to
second order SD 2. Empirical results Child malnutrition in Kenyan
ASALs remains dire No average treatment effect of ALRMP
expenditures Differential impact with fewer negative changes in
treatment sublocations ALRMP a nutritional safety net?
Slide 5
Program evaluation (PE) methods 5 Fundamental problem of PE:
want to but cannot observe a persons outcomes in treatment and
control state Solution 1: make treatment and control look the same
(randomization) Gives average treatment effect as Solution 2:
compare changes across treatment and control
(Difference-in-Difference) Gives average treatment effect as:
Slide 6
New PE method based on SD 6 Objective: to look beyond the
average treatment effect Approach: SD compares entire distributions
not just their summary statistics Two advantages 1. Circumvents
(highly controversial) cut-off point Examples: poverty line, MUAC
Z-score cut-off 2. Unifies analysis for broad classes of welfare
indicators
Slide 7
Stochastic Dominance 7 First order: A FOD B up to iff S th
order: A s th order dominates B iff MUAC Z- score Cumulative % of
population F A (x) F B (x) 0x max
Slide 8
SD and single differences 8 These SD dominance criteria Apply
directly to single difference evaluation (across time OR across
treatment and control groups) Do not directly apply to DD
Literature to date: Single paper: Verme (2010) on single
differences SD entirely absent from the program evaluation
literature (e.g., Handbook of Development Economics)
Slide 9
Expanding SD to DD estimation - Method 9
Slide 10
Expanding SD to DD: interpretation differences 10 1. Cut-off
point in terms of changes not levels. Cdf orders change from most
negative to most positive initial poverty blind or initial
malnutrition blind. (Partial) remedy: run on subset of
ever-poor/always-poor 2. Interpretation of dominance orders FOD:
differences in distributions of changes between intervention and
control sublocations SOD: degree of concentration of these changes
at lower end of distributions TOD: additional weight to lower end
of distribution. Is there any value to doing this for welfare
changes irrespective of absolute welfare? Probably not.
Slide 11
Setting and data 11 Arid and Semi-arid districts in Kenya
Characterized by pastoralism Highest poverty incidences in Kenya,
high infant mortality and malnutrition levels above emergency
thresholds Data From Arid Lands Resource Management Project (ALRMP)
Phase II 28 districts, 128 sublocations, June 05- Aug 09, 602,000
child obs. Welfare Indicator: MUAC Z-scores Severe malnutrition in
2005/6: Median child MUAC z-score -1.22/-1.12
(Intervention/Control) 10 percent of children had Z-scores below
-2.31/-2.14 (I/C) 25 percent of children had Z-scores below
-1.80/-1.67 (I/C)
Slide 12
The pseudo panel 12 Sublocation-specific pseudo panel
2005/06-2008/09 Why pseudo-panel? 1. Inconsistent child identifiers
2. MUAC data not available for all children in all months 3.
Graduation out of and birth into the sample How? 14 summary
statistics for annual mean monthly sublocation - specific stats:
mean & percentiles and poverty measures Focus on malnourished
children Thus, present analysis median MUAC Z-score of children z 0
Control and intervention according to project investment
Slide 13
Results: DD Regression 13 Pseudo panel regression model where D
is the intervention dummy variable of interest NDVI is a control
for agrometeorological conditions L are District fixed effects to
control for unobservables within major jurisdictions No
statistically significant average program impact
Slide 14
DD regression panel results 14 (1)(2)(3)(4)(5) VARIABLES median
of MUAC Z
23 2. Poverty and Welfare orderings (Foster and Shorrocks 1988)
Let U(F) be the class of symmetric utilitarian welfare functions
Then A P B iff A U B Examples: U 1 represents the monotonic
utilitarian welfare functions such that u>0. Less malnutrition
is better, regardless for whom. U 2 represents equality preference
welfare functions such that u0. A transfer is valued more lower in
the distribution Bottom line: For welfare levels tests up to third
order make sense SD, poverty & social welfare orderings
(2)
Slide 24
The data (2) extent of malnutrition 24
Slide 25
DD Regression 2 25 Individual MUAC Z-score regression To test
program impact with much larger data set Still no statistically
significant average program impact
Slide 26
Results DD regression indiv data 26 Robust p-values in
parentheses *** p