Impact evaluation in 7 or 8 steps

Step 1

Engage with the stakeholders

June mission in Mali

• Government interested in a field experiment (RCT) using project and control villages

• Government interested in three main issues:o Good governance: how to make sure that children

are fed?o Education: what is the impact on attendance

rates?o Local economy: how the project benefits small

farmers?

Step 2

Define relevant evaluation questions

Impact on education and nutrition

• Impact on enrolment and achievements, and what is the role played by school quality?

• Does the programme improve attention and cognition?

• Does the programme improve nutritional status? Is there catching-up growth?

• What is the extent of substitution effects within the household?

• What is the impact of the programme on the diet of the poor?

Impact on agriculture

• What is the impact on small farmers in the short term (incomes) and in the long term (farm investments)?

• What is the impact of the intervention on prices and therefore on consumers?

• What is the impact on the wider economy at the village and regional levels?

Step 3

Build a theory of change

Overall programme theory

Agriculture pathways

Education pathways

Nutrition pathways

Step 4

Define the indicators

Welfare outcomes

• Welfare outcomes are the MDGs metrics

Intermediate welfare outcomes

• Sometimes welfare outcomes cannot be observed because:• Occur in the very long term (example increase in

employment)• Are not observable (example maternal mortality)

• Intermediate outcomes are used in these cases: proxy indicators of the final outcomes along the causal chain

Education: outcome indicators

• Enrolment, attendance rates and drop-outs• Achievement tests (test scores on maths and

language)• Attention and cognition

Nutrition: outcome indicators

• Anthropometric measurement.• Measures of diet composition

Food security: outcome indicators

• Full income questionnaire for farmers will provide data on:

marketed surplus Farm profits Technology and capitalisation Input use

Step 5

The evaluation design

The Mali evaluation design

• The government is expanding the intervention to 60 of the most vulnerable communes

• A commune is an administrative unit comprising 5 to 15 villages and a similar number of schools

• The groups considered by the study are: Control group (no intervention) Standard school feeding Home grown school feeding

Level 1 comparison: school feeding-control group

• MOE needs to know the impact of the intervention on educational indicators

• First comparison is between any school feeding and a control group with no intervention

• Outcomes of interests are: Enrolment rates Learning achievements Attention and cognition Nutritional outcomes

Level 2 comparison: school feeding /home-grown school feeding

• The second comparison is between the conventional government programme and the ‘home grown’ programme

• Outcomes of interest: Small farmers’ incomes Overall programme performance

Selection of schools and communes

• In each of the 60 communes Mayors will select two school for the intervention of which one will be randomly assigned to the programme (pair-matching design or stratification by commune). A protocol is designed to avoid contamination.

• Of the 60 communes assigned to the programme, 30 will be randomly assigned to the home grown component

Step 6

Set the sample size

Sample size

• We collected data from 30 households in each village: 20 households with children aged 5 to12 10 farmer households (with or without children)

• Sample size is: 1,200 farmer households 3,600 farmer and non-farmer households 6,000 to 7,000 children of primary school age

Calculate the sample size

• The size needs to be sufficiently large to detect the expected effect of the programme

• Detecting sample size is guesswork and the goal is to produce lower and upper bounds rather than exact samples

• There is statistical software which is designed to do this

Power

• You need a powerful sample to detect impact• The power of your sample will be a function of• The expected programme impact (increasing)• The variance of the outcome of interest

(decreasing)• The homogeneity within clusters (decreasing)• The desired level of Type I error (decreasing)

Step 7

Run a household survey

main issues

• Choose the unit of observation

• Find existing datasets and surveys

• Establish the timing of data collection

• Establish whether collecting cross-section or panel data

• Establish the number of surveys

• Design the questionnaires• Administer the survey

Choose the unit of observation

• The preferred level of observation is the ‘individual’ or the ‘household’ • Note that individuals are difficult to interview (ex:

consumption data or panel)• Household is the most frequent unit of observation

• Observations can also be made at cluster level (village, school, clinic or other)• Note that sample size will be small• Data is difficult to collect (who is interviewed?)

Data scoping

• Before starting collecting any data you should first investigate what data and surveys are available:• Census data can be used to frame the sample or to extract

control variables• Existing household surveys (LSMS or DHS) can be used to

form control groups in matching techniques• Project monitoring data can be used to observe trends• Survey maybe underway in the same areas. This is rarely

the case, but piggybacking is theoretically possible

Timing of data collection

• 3 main issues to consider:• How many surveys will be run?• Baseline, midterm and end-line

• When is the survey starting and for how long?• Seasonality issues need to be considered

• What is the recall period adopted in the questionnaire?• Short recall is more reliable but loses information

Cross-section or panel data?

• A cross-section survey collects data from a population at a point in time

• A panel survey collects data from a population repeated times

• Panel data are preferable because they simplify the analysis• But panel data are not always feasible• But attrition can be large and there can be

differential attrition

Questionnaire design

• Identify the modules that are needed. For example: a household roster, an education module, a consumption module etc.

• Look at existing questionnaire designed by other researchers in similar context

• Examples can be taken from:• LSMS surveys• DHS surveys• Other resources

Running a survey in practice

• Hire a firm with the desired capacity• Ensure enumerators are properly trained and

manuals are available• Test the questionnaires many times • Ensure supervision of enumerators in the field• Ensure households collaborate • Obtain ethical approval

Step 8

Analyse the data