Mixed Methods in Impact Evaluation May 2011
-
Upload
3ie -
Category
Technology
-
view
716 -
download
3
description
Transcript of Mixed Methods in Impact Evaluation May 2011
![Page 1: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/1.jpg)
www.3ieimpact.orgHoward White
Mixed Methods in Impact Evaluation
Howard White
International Initiative for Impact Evaluation
![Page 2: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/2.jpg)
www.3ieimpact.orgHoward White
Measurement is not evaluation
![Page 3: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/3.jpg)
www.3ieimpact.orgHoward White
Why did the Bangladesh Integrated Nutrition Program (BINP) fail?
Why did the Bangladesh Integrated Nutrition Project (BINP) fail?
![Page 4: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/4.jpg)
www.3ieimpact.orgHoward White
Comparison of impact estimates
![Page 5: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/5.jpg)
www.3ieimpact.orgHoward White
Summary of theory
Target group participate in program (mothers of young children)
Target group for nutritional counselling is the relevant one
Exposure to nutritional counselling results in knowledge acquisition and behaviour change
Behaviour change sufficient to change child nutrition
Improved nutritional outcomes
Children are
correctly identified to be enrolled in the program
Food is delivered to those enrolled
Supplementary feeding is supplemental, i.e. no leakage or substitution
Food is of sufficient
quantity and quality
![Page 6: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/6.jpg)
www.3ieimpact.orgHoward White
The theory of change
Target group participate in program (mothers of young children)
Target group for nutritional counselling is the relevant one
Exposure to nutritional counselling results in knowledge acquisition and behaviour change
Behaviour change sufficient to change child nutrition
Improved nutritional outcomes
Children are
correctly identified to be enrolled in the program
Food is delivered to those enrolled
Supplementary feeding is supplemental, i.e. no leakage or substitution
Food is of sufficient
quantity and quality
Right target group for nutritional counselling
![Page 7: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/7.jpg)
www.3ieimpact.orgHoward White
The theory of change
Target group participate in program (mothers of young children)
Target group for nutritional counselling is the relevant one
Exposure to nutritional counselling results in knowledge acquisition and behaviour change
Behaviour change sufficient to change child nutrition
Improved nutritional outcomes
Children are
correctly identified to be enrolled in the program
Food is delivered to those enrolled
Supplementary feeding is supplemental, i.e. no leakage or substitution
Food is of sufficient
quantity and quality
Knowledge acquired and used
![Page 8: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/8.jpg)
www.3ieimpact.orgHoward White
The theory of change
Target group participate in program (mothers of young children)
Target group for nutritional counselling is the relevant one
Exposure to nutritional counselling results in knowledge acquisition and behaviour change
Behaviour change sufficient to change child nutrition
Improved nutritional outcomes
Children are
correctly identified to be enrolled in the program
Food is delivered to those enrolled
Supplementary feeding is supplemental, i.e. no leakage or substitution
Food is of sufficient
quantity and quality
The right children are enrolled in the programme
![Page 9: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/9.jpg)
www.3ieimpact.orgHoward White
The theory of change
Target group participate in program (mothers of young children)
Target group for nutritional counselling is the relevant one
Exposure to nutritional counselling results in knowledge acquisition and behaviour change
Behaviour change sufficient to change child nutrition
Improved nutritional outcomes
Children are
correctly identified to be enrolled in the program
Food is delivered to those enrolled
Supplementary feeding is supplemental, i.e. no leakage or substitution
Food is of sufficient
quantity and quality
Supplementary feeding is supplementary
![Page 10: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/10.jpg)
www.3ieimpact.orgHoward White
Participation rates
0 .0
0 .1
0 .2
0 .3
0 .4
0 .5
0 .6
0 .7
0 .8
0 .9
1 .0
B a se va lu e L iv in g w ith m o th e r-in -la w inR a jn a ga r o r S h a h ra s ti
H igh e r e d u ca tio n N o w a te r o r sa n ita tio n(re m o te lo ca tio n )
Pro
ba
bil
ity
of
pa
rtic
ipa
tio
n i
n g
row
th m
on
ito
rin
ng L iv ing in
R a jnaga r o r S hah ras ti
L iv ing w ith m o the r-in -law
in R o r S
![Page 11: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/11.jpg)
www.3ieimpact.orgHoward White
Data used in BINP study
• Project evaluation data (three rounds)
• Save the Children evaluation
• Helen Keller Nutritional Surveillance Survey
• DHS (one round)
• Project reports
• Anthropological studies of village life
• Action research (focus groups, CNP survey)
![Page 12: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/12.jpg)
www.3ieimpact.orgHoward White
Examples of ‘atheoretical’ IEs
• School capitation grant studies that don’t ask how the money was used
• BCC intervention studies that don’t ask if behaviour has changed (indeed, almost any study that does not capture behavior change)
• Microfinance studies that don’t look at use of funds and cash flow
• Studies of capacity development that don’t ask if knowledge acquired and used
![Page 13: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/13.jpg)
www.3ieimpact.orgHoward White
Different parts of causal chainrequire different analysis
• Factual versus counterfactual
• Examples of factual– Use of funds– Targeting– Participatory processes
• Quantitative and qualitative and the combination of the two
![Page 14: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/14.jpg)
www.3ieimpact.orgHoward White
Examples from AP SHGs
Number of SHGs and % penetration
Drop outs & corrupt practices
The angry man
Returns to cows and goats: quantitative ethnography
![Page 15: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/15.jpg)
www.3ieimpact.orgHoward White
Why the angry man was angry
Loan allocation is to households not individuals
![Page 16: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/16.jpg)
www.3ieimpact.orgHoward White
More examples
• The disconnected in connected villages pretty much everywhere
• The role of the community in social funds in Malawi and Zambia
![Page 17: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/17.jpg)
www.3ieimpact.orgHoward White
Poorest don’t connect
![Page 18: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/18.jpg)
www.3ieimpact.orgHoward White
People participate in making bricks, not decisions
![Page 19: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/19.jpg)
www.3ieimpact.orgHoward White
I have to go now. I have a community in my office
![Page 20: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/20.jpg)
www.3ieimpact.orgHoward White
Also mix methods for identification strategy
![Page 21: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/21.jpg)
www.3ieimpact.orgHoward White
AP village fund allocation
Fixed funds per community: more households per SHG
Lower membership rates in larger villages
![Page 22: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/22.jpg)
www.3ieimpact.orgHoward White
The puzzle of the disconnected households
![Page 23: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/23.jpg)
www.3ieimpact.orgHoward White
You can’t carry
electricity on boats
![Page 24: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/24.jpg)
www.3ieimpact.orgHoward White
Mixing methods
• Understanding context to– Shape evaluation questions– Design data collection
• Mapping out theory(ies) of change
• Addressing factual questions, leading to…
• … interpretation of counterfactual findings
One example on data collection
![Page 25: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/25.jpg)
www.3ieimpact.orgHoward White
General principle: the quality of data deteriorates the more formal the process of data collection
![Page 26: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/26.jpg)
www.3ieimpact.orgHoward White
What do questionnaires miss?
• Consumption– Festivals– Labour exchange– Wildfoods
• Net income from household enterprises
• Abuse of nearly all kinds
• Who is a household member?
![Page 27: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/27.jpg)
www.3ieimpact.orgHoward White
Protein in Northern Zambia
![Page 28: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/28.jpg)
www.3ieimpact.orgHoward White
Who is in the household
roster?
This is my sister, Hana.
Her mummy is my mummy’s
sister
![Page 29: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/29.jpg)
www.3ieimpact.orgHoward White
What is to be done?
•Know what questions to ask and how
•Proxy measures
•Enumerator training
•Contrived informality
![Page 30: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/30.jpg)
www.3ieimpact.orgHoward White
The challenge of integration
• Parallel studies not integrated studies (multi-disciplinary not inter-disciplinary)
• Why?– At best silo mentality, at worst arrogance (“trust me, I’m an
economist”)– Academic incentives– People just don’t know how to do it
• What to do?– Start with theory of change– Team members who bridge studies– Detailed team discussions around causal chain– Quality of external peer review
![Page 31: Mixed Methods in Impact Evaluation May 2011](https://reader035.fdocuments.in/reader035/viewer/2022062418/554d7883b4c9053e0c8b4b33/html5/thumbnails/31.jpg)
www.3ieimpact.orgHoward White
Thank you
Visit www.3ieimpact.org