Theory of Change and Measurement - Abdul Latif Jameel ... · Theory of Change: Top-Down 20 20 Needs...

Session 2

Theory of Change and Measurement

Impact Evaluation Workshop, Lilongwe, Dec. 14th, 2011

Martin Abel

1

• Session 1: Introduction to impact evaluations

• Session 2: Theory of change and measurement

• Session 3: Group work: Theory of Change

• Session 4: Randomized Evaluation Design

• Session 5: Group work: Evaluation Design

Workshop Schedule – Day 1

2

Recap: Measuring Impact (I)

Time

Pri

mar

y O

utc

om

e

Impact

Intervention

3

Outline

• Theory of Change (ToC)

– Definition

– Case Study 1: Village quotas in India

– Mapping from ToC to indicators

– Case Study 2: Primary education in Madagascar

• Data collection

– When to collect data?

– Data collection instruments

4

Theory of Change (ToC)

• Definition: – A theory of change is a description of how an

intervention is supposed to deliver the desired results.

• ToC: – sets out the causal logic of how and why a

particular project, program, or policy will reach its intended outcomes (hypothesis).

5

Back to Chocolate

• What is the chain of causality from the programme (chocolate) to the outcome (test score)?

• Do you think those who receive chocolates will perform better or worse? Why?

6

People like

chocolate

Participants eat chocolates

The ToC takes us beyond the BLACK BOX

Intervention

Chocolates given to randomly selected participants

Participants are more motivated

Less focused in class

Intervention

Inputs

Intermediary outcomes

Outcomes

Assumptions

Positive effect on learning outcomes

Outputs

Participants are distracted

More focused in class

Negative effect on learning outcomes

Black Box

If we don’t see an effect: where was the causal chain interrupted?

Black Box

8

Case Study 1 Quotas in Village Councils in India

Duflo et al. 2008. “Women as policy makers – evidence from a randomized policy experiment in India”.

• 1993 Constitutional Amendment in India.

• Main characteristics of quota policy – Reservation of leadership to women

– Implemented in randomly selected 1/3 of villages

• Main goals of Village Council – Decentralise decision making, especially public goods

“Every village has to become a self-sufficient republic” (Ghandi)

– Give villagers more say

• Main goals of quota

Why may these goals not be achieved? 9

Case Study Example Quotas in Village Councils in India

10

Needs assessment

Intervention

Output indicators


Outcomes

Village Quotas: A Possible Theory of Change

Quotas

More female council leaders

Different public goods

Different health and education outcomes?

Low investment in education and health at village level

Women are more empowered

Village Quotas: A Possible Theory of Change

11

Needs assessment

Intervention

Output indicators


Outcomes

Assumptions

Village leader has decision power

Women have different

preferences

Public goods reflect

women’s preferences

Quotas





Women are more empowered

ToC to Indicators

12

• Once the chain of Theory of Change has been drawn up:

what data should we collect at every step of the way to discriminate between different plausible chains of causality?

Village quotes in India

Theory of Change

Quotas




13

Needs assessment

Intervention

Output indicators


Outcomes


Women are empowered

14

Gender of leader

Female attendance in council meetings; Time

women speak

Budget allocation; Type of public goods available

Indicators

Village quotas in India

ToC → Indicators

Quotas



Different health & education outcomes?


Women are empowered

Literacy levels, BMI of children

15

Village quotas in India

ToC → Indicators → Results

More female leaders

Female attendance

Active participation

Allocations shift; More investment in

roads, drinking water

Results

Gender of leader

Indicators Quotas



Different health & education outcomes?


Women are empowered

Literacy levels, BMI of children

Not yet measured ?

Female attendance in council meetings; Time

women speak

Budget allocation; Type of public goods available

• The ToC can help us to identify potential interventions

• Example: Imagine, you are the mayor of Lilongwe – In the last 5 years, there was a large increase of both

cell phone usage and number of traffic accidents involving pedestrians

What policies could tackle this problem?

16

Example 2: Cell Phone Usage and Pedestrian Accidents

17

Case Study 2 Primary Education Management in Madagascar

Nguyen & Lassibile, 2008. “Improving management in education: evidence from a randomized experiment in Madagascar”.

• Improved access to education but quality remains poor – Pass rate of 63% among Grade 5s – Poor teacher performance identified as key problem

• Madagascar Ministry of Education implemented programme to see whether top-down and/or bottom-up approach is effective in improving teacher performance

18

Primary education management in Madagascar

Context

• Two approaches:

– Top-down approach: provide district and subdistrict administrators with information and tools for supervision

– Bottom-up approach: encourage parental involvement through provision of information and accountability meetings, as well as provide schools with teaching tools

Which method is most effective at increasing teacher effort and learning outcomes?

19


Context


Theory of Change: Top-Down

20 20

Needs assessment

Intervention

Output indicators


Primary outcome

Intensity and frequency of monitoring increases

Learning outcomes improve

Poor learning levels in primary school

Teacher performance improves

Top-down monitoring programme

Officials receive tools and information

Assumptions

Use of tools increases

monitoring

Increased monitoring causes increased

teacher effort



Needs assessment

Intervention

Output indicators


Primary outcome





Theory of Change: Top-Down

Officials receive tools and information


ToC → Indicators

22





Self-reported receipt and usage rates

No. of visits to schools, allocation of time & budget

Attendance, lesson plans, frequency & quality of

evaluations

Indicators

Student attendance, test scores

Officials receive tools & information




23





Self-reported receipt and usage rates

No. of visits to schools, allocation of time & budget

Attendance, lesson plans, frequency & quality of

evaluations

Indicators


Schools not visited more often, allocations

unchanged

Teacher behaviour entirely unaffected

Test scores unchanged

Results

Tools were received, tools were used Officials receive tools & information



Theory of Change: Bottom-Up

24

Community-based monitoring and accountability programme

Parents attend meetings


24

Needs assessment

Intervention

Output indicators


Primary outcome



Teaching tools delivered to schools

Parents hold schools accountable

Teachers use of tools


Theory of Change: Bottom-Up

25




25

Needs assessment

Intervention

Output indicators


Primary outcome



Teaching tools delivered to schools

Parents hold schools accountable

Teachers use of tools

Parents are interested in

Holding schools accountable influences

teacher effort

Assumptions

Tools make teaching

more effective


ToC → Indicators

26






Tools delivered to schools

Parents hold schools

accountable

Teachers make use of tools

Indicators

No. of participants at meetings, tools present

Participation of parents, self-reported usage of

tools

Attendance, lesson plans, frequency &

quality of evaluations


Attendance at meetings good, tools received



27






Tools delivered to schools

Parents hold schools

accountable

Schools make use of tools

No. of participants at meetings, tools present

Participation of parents, self-reported usage of

tools

Attendance, lesson plans, frequency &

quality of evaluations

Indicators


Communication with parents unchanged, schools used tools

Attendance unchanged, lesson plan & eval ↑

Attendance and test scores ↑

Results

Theory of Change - Summary

• ToC is useful to help define research questions / hypotheses – Both for experimental and non-experimental research

• ToC important for identifying all outputs, outcomes and assumptions we need to measure

• We learn lessons about underlying mechanism which we can generalize to other programs – “Projects can rarely be replicated, though the

mechanism underlying success or failure will often be replicable and transportable.” (Deaton)

• Different chains of causality can have different policy implications

28

Outline

• Theory of Change (ToC)

– Definition

– Case Study 1: Village quotas in India

– Mapping from ToC to indicators

– Case Study 2: Primary education in Madagascar

• Data collection

– When to collect data?

– Data collection instruments

29

When to collect data?

• Baseline Survey: before the programme is implemented

Target

Population

Not in

evaluation

Evaluation

Sample

Assignment

Treatment

group

Control group

Endline Survey

Measure Impact

Baseline Survey

30


• Measure inputs and outputs specified in ToC (Process Evaluation)

– Ex: Tutoring program for poor students

• How many tutoring sessions are held? How long is each session? How many students attend? Is the appropriate teaching material available?

• RCT compliance: Do people in treatment group receive the program, and those in control not? 31

Target

Population

Not in

evaluation

Evaluation

Sample

Assignment

Treatment

group

Control group

Endline Survey

Measure Impact

Baseline Survey


• Endline Survey: after treatment (timing depends on intervention)

– Multiple follow-up surveys possible (medium vs. long-run effects)

• Eg. Village quotas in India – does the perception of women change over time?

32

Target

Population

Not in

evaluation

Evaluation

Sample

Assignment

Treatment

group

Control group

Endline Survey

Measure Impact

Baseline Survey

1. Administrative data

2. Focus groups

3. Survey questionnaires

4. Non-standard instruments

Data collection instruments or methods

33

Instruments (I)

1. Administrative data

– Cheap if routinely collected

– May not be accurate, specific to population of interest, or collect all necessary characteristics

• Case Study 1 - Village quotas in India: 1991 census was used to draw the sample and check that the treatment and control groups were balanced

• Case Study 2 - Primary education management in Madagascar: Scores from the national Grade 5 exam used to measure learning outcomes

34

Instruments (II)

2. Focus group discussions

– Interview with more than one individual at a time

– Can be cheap way to test feasibility, exhaustibility etc

– Useful for understanding local context

3. Survey questionnaires

– Captures information reported by individual

– Can be cross sectional or panel

• Case Study 2 - Primary education management in Madagascar: school questionnaires gathered information on teacher behaviour, usage of tools etc

35

Instruments (III)

• Non-standard instruments:

– Some information can’t easily be collected through standard instruments

– Eg. Corruption, teacher attendance, political participation, service delivery, community trust, female empowerment

• Random spot checks can be useful

– Can make use of unscheduled data collection by:

• Visiting construction sites to measure progress of public goods delivery

• Measuring teacher and student absenteeism

• Visiting police stations incognito to measure discrimination or

corruption

36

Implicit Association Tests (IAT) – Ex: Measuring Discrimination

• Perceptions of marginalised groups is often interesting to researchers, but difficult to measure with explicit questions

“A doctor and his son have an accident and the doctor is killed. The boy is rushed to surgery, and the surgeon says, “this is my son.” What is going on?”

A) The doctor’s wife was unfaithful B) The surgeon is the boy’s mother

Crime

Prison

Education University

Instruments (IV)

Case study:

• Identify Theory of Change

• Identify instruments and outcomes to test the ToC

Your organization’s program:

• Discuss Theory of Change

• What are (intermediary) outcomes and how would you measure them?

Group Work

Group 1: Participation, Accountability and Transparency Group Leader: Cristobal Marshall Organizations: MEJN - Liu Lathu

CCJP PACENET

Group 3: Participation, Accountability and Transparency Group Leader: Martin Abel Organizations: Church & Society

CHRR Trocaire

Group 5: Agriculture / Livelihood Group Leader: Michael Roscitt Organizations: CISANET

ACODE Oxfam

Group 7: Health

Group Leader: Willa Brown Organizations: Norwegian Church Aid

Action Aid Malawi Danish Church Aid

Group Work Group 2: Participation, Accountability and Transparency Group Leader: Clare Hofmeyr Organizations: ACCU TRAC.fm

MISA

Group 4: Economic Governance / Livelihoods

Group Leader: Thomas Coen

Organizations: MEJN Local Development Fund

BMAU

Group 6: Women and Children

Group Leader: Niall Keleher Organizations: PLAN

CARE CISCBE

Annex

40

I. Measurability: a measurable indicator is observable, feasible, and detectable

– Observability: indicators must be observed in the real world.

– Feasibility: indicators must be politically, ethically and financially measurable

– Detectability: study must have instrument and statistical power to measure indicator

Characteristics of good indicators (I)

41

II. Precision: the more exhaustive and exclusive the indicator, the more precise it is

– Exhaustive: most outcomes can be measured by more than one indicator.

– Exclusive: indicators is affected by the outcome of interest and nothing else

42

Characteristics of good indicators (II)

III. Reliability: indicators are reliable when they are hard to counterfeit

– Make sure answer to question is not easily forgettable

– Deliberate misreporting: try to avoid providing any incentive for people to lie in their answer

Use proxy estimators that measures the direct consequence of the same outcome

Remove incentive of respondent to lie by not revealing which is the ‘desirable’ answer

43

Characteristics of good indicators (III)

Theory of Change and Measurement - Abdul Latif Jameel ... · Theory of Change: Top-Down 20 20 Needs...

Documents

Transcript of Theory of Change and Measurement - Abdul Latif Jameel ... · Theory of Change: Top-Down 20 20 Needs...