Designing Rigorous Impact Evaluations of Agricultural ... · Designing Rigorous Impact Evaluations...
Transcript of Designing Rigorous Impact Evaluations of Agricultural ... · Designing Rigorous Impact Evaluations...
Abt Associates | pg 1
Designing Rigorous Impact Evaluations of Agricultural “Pull” Mechanisms Stephen Bell, Tulika Narayan, & Judy Geyer Abt Associates 4 September, 2014
Abt Associates | pg 2
“Pull” Mechanims: – Why? – What?
Missing markets for socially-essential activities/ products in the agricultural value chain, such as -- healthy strains of grain -- effective harvest management
Traditional “push” mechanisms have failed to provide sustainable solutions to agricultural market failures
Growing appreciation of the potential role of the private sector
Create “prizes”/economic rewards to private sector actors when they deliver desired results
With time, phase out the prizes
Abt Associates | pg 3
3
Outline of Talk
Introducing the AgResults initiative
Research questions and intervention model
Nigeria Aflasafe Pilot
Kenya On-Farm Storage Pilot
General lessons for designing agricultural IEs
Abt Associates | pg 5
AgResults: A Program for Trying
and Learning Many “pull” pilots:
Nigeria (Aflasafe)
Kenya (On-Farm Storage)
Zambia (Provitamin-A Maize)
Myanmar and/or India (Newcastle Disease Innoculation)
[others in conception stage]
-----------------------------------------------------------------
Many donors:
Australia (DFAT), Canada (DFA), U.K. (DFID), U.S. (USAID), Bill & Melinda Gates Foundation, World Bank
Abt Associates | pg 6
The AgResults Evaluation
Goal = answer two policy questions:
1. Do “pull” mechanisms increase farmers’ adoption of
improved agricultural technologies?
2. Are sustainable markets created that outlive “pulls”?
Basic tasks for Abt Associates:
Define evaluation framework (integrative across pilots)
Specify data and methodologies for 5 “pull” pilot studies
Carry out 3 of those studies
Provide input to donors’ learning agenda (reports, data)
Abt Associates | pg 8
Detailed Research Questions
ions What is the impact of the [name] pilot on… … private sector involvement in markets for improved technologies? … farmers’ uptake of improved technology? … smallholder incomes? … consumers’ use of derivative products?
Are effects sustainable once the “pull” incentives end?
What are the lessons for future “pull” initiatives?
Abt Associates | pg 9
Evaluation Framework: Economic
Model of the Agriculture Value Chain
Farmer Distributor Consumer Social Planner
Goal Maximize Profit
Maximize Profit
Maximize Utility
Maximize Social Welfare
Choices Technology, capital/labor
Technology, capital/labor
Consumption Interference Mechanisms
Constraints.. Capital/labor costs
Capital/labor costs
Wealth, information
Funding Knowledge
Price, Quantity
Onfarm Product
Price, Quantity
Market Product
Abt Associates | pg 10
10
The Social Planner’s Problem and
Tools
Market failures:
– Insufficient consumer / farmer (i) knowledge of risks, and
(ii) demand for solutions: aflatoxins, Newcastle disease
– Consumer reluctance to engage in behavioral change:
bio-fortified “orange” maize
– Initial technology too expensive until production and
demand increase: grain storage solutions
“Push” mechanisms lower costs
“Pull” mechanisms provide extra revenue conditional on
desired change in quantity/kind of products sold in market
Abt Associates | pg 11
Market without Intervention
Farmer Distributor Consumer
Maximize Profit Maximize Profit Maximize Utility
Social Planner
Maximize Social Welfare
costs costs
Price, Quantity
Price, Quantity
Onfarm Product
Market Product
costs
Abt Associates | pg 12
Add “Push” Mechanisms
Farmer Distributor Consumer
Maximize Profit Maximize Profit Maximize Utility
Social Planner
Maximize Social Welfare
costs costs
New Price Quantity
New Price Quantity
Onfarm Product
Market Product
costs
Abt Associates | pg 13
Add “Pull” Mechanism Instead
Farmer Distributor Consumer
Maximize Profit Maximize Profit Maximize Utility
Social Planner
Maximize Social Welfare
costs costs
New Price Quantity
New Price Quantity
Onfarm Product
Market Product
costs
Abt Associates | pg 14
Sustain the Change, Long-Run
Farmer Technology
Distributor Technology
Consumer Information
Maximize Profit Maximize Profit Maximize Utility
Social Planner
Maximize Social Welfare
costs costs
New Price Quantity
New Price Quantity
Onfarm Product
Market Product
costs
Abt Associates | pg 16
Theory of Change - Nigeria
Incentives for sale of aflatoxin-free maize
• Increase in treatment of crops with Aflasafe
• Increase in availability of aflatoxin-free maize
Increased Awareness
•Adoption of Aflasafe by farmers
• Increase in awareness of health benefits of aflatoxin-free maize
Smallholder benefits
• Increase in smallholder income (incentive for greater yields)
•Consumption of aflatoxin-free maize
Technical Assistance
Abt Associates | pg 17
Impact Inference Strategy
Random assignment of 72 villages
3 years of phased implementation (2014-16)
Baseline survey of farmers from cohorts B & C
(2014)
End-line survey of farmers from cohorts A, B, & C
(2015)
1-year impact (B vs. C)
2-year impact (A vs. C)
Abt Associates | pg 18
Challenges and Responses
Implementer buy-in
-- two Evaluation Design Workshops with
implementers (welcomed a “lottery”!)
Sampling farmers to maximize detection of impacts -- implementers provided lists of likely participating
farmers in each village before randomizing
can measure impacts on projected participants
-- lists proved uneven for cohort A
-- obtained post-randomization updates for cohorts
B & C use “ASPES” to measure impact on a
symmetrically-predicted endogenous subgroup
(Bell & Peck, 2013)
Abt Associates | pg 20
Theory of Change - Kenya
Incentives for sale of on-farm storage
•Sales and marketing by private sector to promote on-farm storage
Increased Awareness
•Adoption of on-farm storage by farmers
•Awareness of proper use of on-farm storage
•Awareness of benefits of storing maize
Smallholder benefits
• Increase in smallholder income (better timing of sales to open market)
• Increased food security
Abt Associates | pg 21
Impact Inference Strategy
Implementation plan rules out contemporaneous
comparison group
All-at-once rollout no comparable maize-growing
regions for comparison sites
Community-wide marketing no “untreated”
farmers in intervention sites
No choice but to use pre-post comparison
-- do it with strongest possible data
-- apply robustness check
Abt Associates | pg 22
Challenges and Responses
The future, absent the intervention, is not static -- use trend line in outcomes before the pilot to project “untreated” trajectory past that point -- prioritize addition of more pre-program time points (over post-program time points) when collecting data interrupted time-series
Trend may shift on its own -- track an outcome not affected by the intervention but affected by the same confounding factors (rainfall, pest burden) -- test for “false effects” on yield per acre -- include as a control variable
Abt Associates | pg 24
Broader Lessons for Agricultural IE
Be flexible and creative with impact inference
strategies
Build a constituency “on the ground” to support key
evaluation functions
Build theory into research design and data priorities
Worry about statistical precision to detect impacts –
and be cautious in interpreting null findings
Preserve resources to allow long-run (e.g., post-
”pull”) follow-up