1

Impact evaluation in the absence of baseline surveys

By Fabrizio Felloni, Office of Evaluation, IFAD

International Workshop on Development Impact Evaluation, Paris, November 15, 2006

2

The context of IFAD

Relatively small projects: 2005 median of IFAD loans = US$ 15.5 m, project costs = US$ 26 m

Focus on rural poverty reduction

Traditionally: limited field presence of IFAD (15 countries on a pilot basis), IFAD not executing or supervising projects, limited self- evaluation

This scenario is evolving with new Action Plan

3

Field-based evaluation at IFAD - OE

Necessary to make up for distance of headquarters from the field and information gap

Several types: project, country programme and corporate evaluations

All include field visit and some form of primary data collection

Project evaluations conducted just before or soon after project closure

4

Methodological requirements

Standardised methodology for project and country programme evaluations requires assessing impact (standardised categories)

No standardised data collection methods: to be identified at approach paper phase

Impact is but one of the analytical domains (also relevance, effectiveness, efficiency, sustainability innovation, performance of partners)

So no “dedicated” instrument for impact assessment

5

Shoe string evaluation in action

Considerations from personal experience

6

A case of shoe string impact evaluation

See Bamberger et alii AJE, 25 (1), 2004

A number of constraints

1. Time and budget (impact is one of the evaluation domains)

2. Poor performance of M&E function at project level

3. Absence or limited usefulness of baseline data (now changing: baseline survey with anthropometric and hh asset indicators for all new projects)

7

Logical steps for impact assessment

Preliminary quantitative mini-survey

Multi-disciplinary field visit (mainly qualitative + direct observations)

Impact assessment

Formulate first impact hypotheses, collect evidence on selected “basic indicators”

Validate hypotheses, probe on a set of narrower questions

Triangulation of mini-survey, focus groups and individual interviews + key informants

1 2 3

8

A pragmatic approach

Within this context, impact assessment based on triangulation, still important qualitative component

Still place for theory-based approach Quantitative survey used to test and generate new

hypotheses, better focus questions during main mission

Small sample size: 200 – 350 respondents including project and control. Size determined by practical issues (represent project activities, time, transportation, budget)

9

“Ideal” scenario for the survey

Ti and Ci : measurable characteristic of the population, i = time of observation (0,1).

Unfortunately, this scenario is almost never found

T0: programme group T1: programme group

C0: control group C1: control group

Baseline Follow-up

1. Best case scenario: quasi-experimental design

10

Typical scenarios

1. Programme group only

T0: programme group T1: programme group

Baseline Follow-up

2. No baseline at all: most frequent case

???

Evaluation

11

Other common issues

Classical problems with “control samples” (selection bias, spill-over effects, non-compliance)

Ex ante: (i) visit “similar” communities or hh in administrative areas outside project, (ii) select “new entries”

Ex post: Mostly dealt with qualitatively at mission phase (triangulation)

Main constraint to use of econometric techniques: availability of trained specialists, time (impact is one of the evaluation domains)

12

Dealing with lack of baseline data Several options (not mutually exclusive)

1. Reconstructing baseline data ex post: recall method (more later)

2. Use key informants and triangulate (mostly qualitative)

3. Reconstruct a baseline “scenario” with secondary data (not always practical given absence and quality of baseline studies)

4. Single difference with econometric techniques: some practical obstacles (workload, time constraints, availability of trained specialists)

13

Recall methods

Ask about current situation (e.g. cropping practices) now and at programme start-up

T1: programme group

C1: control groupC0

T0

recall

recall

14

Typical problems with recall methods

Telescoping of major events / expenditures

Under-estimation of small and routine events / expenditures

Recall time line (events that are 3 -7 years old)

Unintended misidentification of project start-up

“Strategic” behaviour of respondents (to please the interviewer or express complaints)

Some indicators are more complex to identify and remember with precision (income)

15

Some techniques to control problems Concentrate on few impact variables that are easier to

“visualise” and recall. Some examples: household appliances, livestock size (depending on the

context), cropping patterns, agricultural and grazing practices,

community initiatives) Help identify baseline point by helping recollect key facts

and events Do not simply ask “what”, ask “why”, i.e. respondents to

state causal linkages. E.g. the number of goats increased: why? and how? Also useful for attribution.

Pre-test the instruments

16

Practical examples

17

Ex 1.The Gambia: Rural Finance Project (2004) Preliminary survey

- Project and control group - Recall: income and assets at hh

and kafo level

Data analysis

- Descriptive statistics and significance tests + principal component analysis

- Generated two hypotheses: (i) limited overall impact on hh income; (ii) biases against relatively poorer hh in

villages

18

Gambia, Rural Finance (cont’d) Field mission: focus groups,

individual interviews + key informants

- Confirmed limited effects on income generation opportunities

- Credit collateral: discouraged participation from poorer hh, ineffective in establishing credit discipline

Main observations:

- Some validity threats in recall data on income and monetary assets

- Consistency with qualitative findings

- Help focus the scope of field mission

19

Ex. 2 Ghana Upper East Region

Similar to the Gambia case (project & control, recall)

Multi-component agricultural project: main intervention, small dams

Recall on household productive and other durable assets

Main findings seemed to show larger effects for project group

Some methodological shortcomings

- difficult to find matched observation for control group (given multi-component nature)

- small sample size of control group may have affected significance tests

20

Example 3. Morocco, Southern Oasis Again, project and control,

with recall method

Many interventions, very heterogeneous, difficult to standardise questionnaires

Focus on perceptions of trends (e.g. income generating opportunities, irrigation / potable water availability, feed for livestock)

Hypothesis: the project was effective as a buffer measure during years of drought. Supported by qualitative analysis in field mission

21

Concluding remarks Preliminary survey and recall methods never a stand – alone

measure but rather propaedeutic to (mainly) qualitative mission

Triangulation to validate reliability of reconstructed baseline: survey data, with field observations, focus group, individual interviews and key informants

By and large, trends suggested by preliminary survey found to be consistent with qualitative data

Some legitimate concerns on accuracy of estimated means for certain indicators (income, monetary assets)

22

Concluding remarks (cont’d)

Evolution towards focus on perceived trends on a narrower set of key indicators

Cost effective to conduct preliminary work with local specialists and students as enumerators

Project teams consulted in planning and sampling phase. Results and database made available

Valuable experience for local students – enumerators

In principle, replicable model for public authorities in charge of programme implementation