Remedying Education:�Evidence from two Randomized �

Evaluations�

Esther Duflo and David Donaldson�

14.73, Fall 2009, lecture 11�

Introduction

• The achievements of the educational system in urban India are unimpressive:

• Pratham conducts every year the ASER survey: a survey of reading and math skills in all the of the district of india VIDEO

• In standard 3, only 50% of the kids can read a very simple paragraph (standard 1 level)

Introduction

• The achievements of the educational system in urban India are unimpressive:

• This is a problem common to many developing countries, leading to drop out.

• Improving the quality of education has become the primary concern of policy makers. – Example: DPEP (in India) was the largest World Bank

funded program. Seeks to improve the quality of education through teacher training, teaching material, etc.

Improving the quality of education is difficult

• Evidence that interventions seeking to improve access by reducing the cost of attending do increase enrollment – Schultz (2001); Vermeersh (2002); Kremer et al (2002); Miguel Kremer

• Evidence that interventions giving incentive for children to perform well in school do impact tests scores – Kremer et al. (2002); Angrist and Lavy (2002).

• However, harder to improve the performance of children already in government schools by providing inputs: – Kremer et al (2001): textbooks – Banerjee, Kremer, Jacob (2002): second teacher – Kremer et al. (2002): teacher incentives

• Is it the case that only incentives and rates of returns are relevant, quality of inputs do not matter? Or have we not found the “right”package of intervention?

Remedying Education

• A possibility is that the interventions that have been tried until now increase the resources to help the teachers do the same thing as before.

• It may be more effective to change the way teachings is done: Changing the actors andfocusing on basic competencies.

• This study evaluate and compares two approaches to do this: –Remedial education (Balsakhi) –Computer assisted learning (CAL)

• Two interventions in the center of the policy debate in education in India.

This Study

• Large scale randomized evaluations • The programs are implemented by Pratham, a

large NGO (currently reaching 258,000 children) • Two cities and 3 years. • Vadodara:

–Year 1 (2001-2002): Balsakhi – Year 2: Balsakhi & CAL (stratified)– Year 3: CAL

• Mumbai –Year 1: Balsakhi –Year 2: Balsakhi

The “Balsakhi” Program• Pratham hires instructors with 10th – 12th grade

education from the local communities for 500 Rs ($10 US) per month.

• Provides a short session of basic training and monthlyfocus groups, in classroom management strategies, andthe use of Pratham developed educational aids.

• High turnover among Balsakhi (they stay for a year, on average), but no recruiting difficulties.

• Each school provided one Balsakhi, who works eitherwith 3rd or 4th grade children. She works with 20 lowperforming students for two hours each day.

• No formal control of which students are assigned.

Motivation for the BalsakhiProgram

• Automatic promotion through 4th grade • Teachers teach the official curriculum

– Mindset – Pupil/Teacher ratios

• Children who start lagging behind lose ground and loose interest, leading to drop out or to them being kicked out of the school system.

Program’s strengths

• Improve student/teacher ratio (both in remedialeducation groups and for other children)

• Provide targeted instruction at child’s current level of achievement

• Better communication between the teacher, parents, and students

• Balsakhis may care more about the students’wellbeing than teachers do

• Cheap (Teacher salary is about Rs2000-4000 per month).

• Work with the school sytem: can be scaled up.

Program’s potential weaknesses

• Compared to teachers, Balsakhis have limited education and little formal training

• Children are removed from regular classroom for 2 hours a day

• Work within the municipal school structures:– Teachers can refuse Balsakhis – Teachers can use Balsakhis as an assistant– School schedule

Study Design

• Vadodara: – Year 0 (November 2000-March 2001): 97 schools divided into

two groups; one group receives a Balsakhi for standard 3, the other remains a comparison group. School year interrupted by earthquake.

– Year 1 (July 2001-March 2002): 97 schools randomly divided into two groups:

• Group A receives Balsakhi in grade 3 • Group B receives Balsakhi in grade 4

– Year 2 (July 2002-March 2003): • Group A receives Balsakhi in grade 4 • Group B receives Balsakhi in grade 3 • Add another 25 schools, randomly allocated to groups A and B

Study Design

• Mumbai L-ward – Year 1 (2001-2002): 62 schools randomly divided into

two groups: • Group A receives Balsakhi in grade 3 • Group B receives Balsakhi in grade 2

–Year 2 (2002-2003): • Group A was supposed to receive Balsakhi in grade 4 • Group B was supposed to receive Balsakhi in grade 3 • In practice, about 1/3 of schools in the treatment group did

not receive a balsakhi (the balsakhi failed the competency test and no replacement was hired).

Sample DesignNumber of Schools Number of Children

Standard Study Group Number of Schools Number of Children Treated With Balsakhi Vadodara Year 1 Three Balsakhi 48 2596 - -

No Balsakhi 48 2527 - -Four Balsakhi 48 2414 - -Year 2 No Balsakhi 49 2619 - -Three Balsakhi 61 3146 61 952

No Balsakhi 61 2906 0 0 Four Balsakhi 61 3167 61 1011

No Balsakhi 61 3170 0 0 Mumbai Year 1 Three Balsakhi 32 2592 - -Year 2 No Balsakhi 35 2182 - -Three Balsakhi 39 2530 28 636

No Balsakhi 38 2943 . . Four Balsakhi 38 2812 27 688

No Balsakhi 39 2460 . .

Computer Assisted Learning

• Vadodara Municipal Corporation had distributed 4 computers per schools in 80% of the primary schools in the city.

• Computer remained unused, or used only by older students or teachers.

• Pratham had experience with CAL in Mumbai and had developed software in the local language.

• They started a program where they used the existing computers (plus some purchased with World Bank grant), loaded them witheducational games for math reinforcement.

• Grade 4 were provided 2 hours of shared computer time per week (2 kids per computer)—one hour during school time, one hour before or after.

• Supervised by a young woman from the community given 5 days of computer training; the woman ensures that the computers are usedand help children choose level. She gives no instruction.

• In year 2, more effort went into designing a better “curriculum”, combining worksheet work and games.

CAL: Strength and weaknesses

• Reinforce basic math competencies using simple games

• Child gets individual stimulus • More exciting than school in India. • Existing evaluations are not very encouraging:

–Angrist-Lavy in Israel –Krueger and Rouse: RE of “fast word”.

• However results may be very different in India where the quality of teaching is very low.

Study Design

• Year 2: Half of the schools were randomly selected: – Excluding from the study the schools were

computers could not be installed – Stratify by: Balsakhi treatment status, Math

test scores in previous year, gender mix,language of instruction.

– 55 schools in treatment, 56 in comparison,and about 3,000 children per group.

• Year 3: groups were switched.

Attendance and Attrition

• Attendance: – Collected weekly by independent research

assistants (not balsakhi and not teachers).– No impact of Balsakhi – Perhaps a small impact of CAL.

Effect of BAL on attendanceTable 11: Attendance

Standard 3 Standard 4 Avg Attendance Simple Avg Attendance Simple

Sept-Oct Differences Sept-Oct Differences Mumbai

Year 1 RA Attendance 0.86 -0.009 (0.014)

Roster Attendance 0.92 -0.010 (0.011)

Year 2 RA Attendance 0.85 0.007 0.86 -0.012 (0.022) (0.017)

Roster Attendance 0.89 -0.004 0.89 -0.023 (0.018) (0.015)

Vadodara

Year 1 RA Attendance 0.69 -0.009 0.70 0.045 (0.017) (0.023)

Year 2 RA Attendance 0.74 -0.005 0.75 0.011 (0.013) (0.011)

Effect of CAL on attendanceTable 11: Attendance

Standard 4 Avg Attendance Simple

Sept-Oct Differences

Year 1 RA Attendance 0.75 0.001 (0.011)

Year 2 RA Attendance 0.69 0.028 (0.015)

Attrition

• If the program affects drop out or propensity to appear forthe test, any difference in test scores among children whoappeared for the test may be biased: For example, if low-scoring children are more likely to appear for the test in thetreatment group, this will tend to make the program effect look too small.

• To avoid this, we attempt to follow up every child who tookthe pre-test, and stop looking for them only if they have leftfor the village. – Attrition below 10%, in Mumbai and in Vadodara year 2, higher

in Vadodara year 1 (communal tensions) – No difference between treatment and comparison groups. – Attritors have lower pre-test scores, but this is similar across

treatment and controls.

Table 6: Attrition Patterns

Vadodara, year 1 Vadodara, year 2 Balsakhi No Balsakhi Difference Balsakhi No Balsakhi Difference

Standard 3, All Percent attrition 0.193 0.181 0.012 0.040 0.040 0.000 Difference in score at pretest attriters-stayers (0.020) (0.008)

-0.130 -0.222 0.092 -0.060 -0.075 0.015 (0.093) (0.131)

Standard 4, All Percent attrition 0.188 0.168 0.020 0.052 0.051 0.000 Difference in score at pretest attriters-stayers (0.021) (0.020)

-0.178 -0.176 -0.002 -0.098 0.015 -0.113 (0.077) (0.155)

Table 6: Attrition Patterns

Bombay, year 1 Bombay, year 2 Bombay 2 years Balsakhi No Balsakhi Difference Balsakhi No Balsakhi Difference Balsakhi No Balsakhi Difference

Standard 3, All Percent attrition 0.070 0.075 -0.004 0.077 0.073 0.005 0.255 0.250 -0.006 Difference in score at pretest attriters-stayers (0.015) (0.010) (0.009)

-0.146 -0.274 0.128 -0.330 -0.193 -0.137 -0.445 -0.594 0.149 (0.169) (0.129) (0.151)

Standard 4, All Percent attrition 0.063 0.070 -0.006 Difference in score at pretest attriters-stayers (0.010)

-0.180 -0.427 0.247 (0.139)

CAL Attrition Table 6 : Attrition patterns

CAL No CAL Difference Vadodara, year 2

CAL No CAL Vadodara, year 3

Difference Standard 4, All Percent attrition 0.041 0.051 -0.010 0.0003 0.0003 -0.00002 Difference in score at pretest attriters-stayers

-0.194 0.188 (0.016) -0.381 -0.509 0.161

(0.0004) -0.670

(0.134) (0.074)

Test scores

• Test scores: Test developed by Pratham to test competencies from grade 1 through 3, in math and language. Piloted in year 0 inVadodara and in several other locations (now used by the Bombay local government to assess competencies in ALL the Bombay schools!). – Vadodara, grade 3 and 4

• Year 1: Pre and Mid test (post test in the middle of the riots). • Year 2: Pre, mid, and post test

– Mumbai, grade 3 • Year 1: Pre and Post test (Pre test conducted in September) • Year 2: Pre-test and Post test

Normalizing test scores

• We want to be able to compare different interventions which improve quality, but each test scores has its own scale.

• Therefore, in education research, we present most results using normalized test scores:for all scores we subtract the mean of the pre-test scores in the comparison group, and divide by the standard deviation in this group.

• The test score will by construction average to zero in comparison group at baseline

• Now we can compare different interventions across each other.

Results: BALSAKHI

• Random choice of treatment and control school should ensure that the treatment and comparison schools are no different, except forthe fact that the treatment schools were exposedto the program.

• There could be random pre-existing differences. To avoid this: – Stratification by language of instruction and pupil

teacher ratio in Vadodara –Stratification by pre-test scores in Bombay

• Tables 2 to 5: no pre-existing differences

Pre-test scores, Vadodara

Year 1 Year 2 BAL NO BAL Difference BAL NO BAL Difference

STANDARD 3 Math -0.008 0.000 -0.008 0.039 0.000 0.039

(0.084) (0.074) Verbal 0.059 0.000 0.059 0.025 0.000 0.025

(0.089) (0.082) STANDARD 4 Math -0.046 0.000 -0.046 0.441 0.452 -0.011

(0.080) (0.019) Verbal -0.059 0.000 -0.059 0.340 0.353 -0.013

(0.017)

Pre-test scores, MumbaiYear 1 Year 2

BAL NO BAL Difference BAL NO BAL Difference STANDARD 3 Math 0.002 0.000 0.002 -0.070 0.000 -0.070

Verbal 0.100 0.000 (0.108) 0.100 0.025 0.000

(0.087) 0.025

STANDARD 4 (0.108) (0.082)

Math 0.051 0.000 0.051

Verbal 0.083 0.000 (0.077) 0.083

(0.083)

Effect of the Balsakhi program • Simple differences: Post test

– Differences between Comparison and treatment children – ALL children, irrespective of whether or not they work with balsakhi

• In Mumbai, simple difference is normalized by whether or not the school was assigned a balsakhi (using the treatment dummy as instrument).

• Effect: – Math: 0.13 to 0.17 in year 1, 0.35 to 0.44 in year 2 – Language: 0.06 to 0.15 in year 1, 0.09 to 0.30 in year 2

• Post-test score distribution in treatment schools statistically dominate post-test score distribution in comparison schools.

Post-test: Vadodara, year 1

Year 1 Year 2 Treatment Control Difference Treatment Control Difference

STANDARD 3 Math 0.391 0.221 0.170 1.698 1.259 0.439

(0.100) (0.116) Verbal 0.840 0.688 0.152 1.245 0.998 0.247

(0.106) (0.103) STANDARD 4 Math 0.231 0.088 0.142 1.197 0.869 0.329

(0.095) (0.087) Verbal 0.677 0.617 0.060 0.916 0.621 0.295

(0.108) (0.089)

Differences in Differences

• Children’s test score is very stable over time (family influence remains constant, etc)

• Thus, we can reduce the noise in the test score by controlling for pre-test score

• We now ask whether the children in schools that got the Balsalkhi experienced a faster increase in test scores that the school that did not get the balsakhi.

Results Differences in differences estimates

Number of Observations Math Verbal Total

Mumbai and Vadodara together Year 1

Mumbai and Vadodara together Year 2

Pooling Both Standards Vadodara Year 1

12730

21805

8301

0.188 (0.047) 0.319

(0.067)

0.196

0.069 (0.056) 0.153

(0.050)

0.109

0.138 (0.047) 0.250

(0.059)

0.164

Vadodara Year 2 11819 (0.059) 0.342

(0.058) 0.225

(0.058) 0.309

Mumbai Year 2 9986 (0.077) 0.279

(0.064) 0.032

(0.073) 0.150

Mumbai Year 2 Specification Check 9986 (0.124) 0.285

(0.112)

(0.076) 0.063

(0.067)

(0.099) 0.173

(0.088)

Two-year effect(Mumbai only)

Number of Observations Math Verbal Total

Two Year 01-03 Mumbai Pre-test Year 1 to Pre-test Year 2 3124 -0.129 0.003 -0.066

(0.093) (0.100) (0.094) Mumbai Pre-test Year 1 to Post-test Year 2 3299 0.521 0.145 0.351

(0.142) (0.113) (0.115)

Comments on results

• Most results significant; same estimate with an oral test • Impact larger in second year than in first year • Impact on math always larger than in language • No measurable impact on language in Mumbai. • Impacts are large by the standard of the education

literature. – Many randomized evaluation find NO impact of resources on test

score – Those who do consider an increase of 0.2 standard deviation a

large effect. – This is the average effect on ALL children (irrespective of

whether or not child actually works with balsakhi): this needs to be taken into account when calculating the overall benefit.

• Impact seems to accumulate over time.

Effect on Specific CompetenciesEffect on specific competencies

Math Competencies for Verbal Competencies for Standard 1 Standard 2 Standard 3 Standard 4 Standard 1 Standard 2 Standard 3

Vadodara Year 1 Both Standards 0.050 0.011 0.021 - 0.039 0.040 0.014

(0.022) (0.007) (0.019) - (0.023) (0.019) (0.015) (0.031) (0.011) (0.033) - (0.028) (0.027) (0.022)

Year 2 Both Standards 0.064 0.063 0.022 - 0.019 0.073 0.031

(0.023) (0.019) (0.009) - (0.023) (0.021) (0.015) Mumbai Year 1 Standard Three 0.058 0.037 -0.003 - 0.004 0.045 0.007

(0.032) (0.030) (0.020) - (0.021) (0.030) (0.032) Year 2 Both Standards 0.088 0.058 0.078 0.094 -0.013 0.041 0.009

(0.038) (0.047) (0.025) (0.032) (0.030) (0.025) (0.037)

Effect by third of the distribution

Year 1 Year 2

N Math Verbal Total

Fraction of Children who go to the

Balsakhi N Math Verbal Total

Vadodara and Mumbai together Bottom Third

Middle Third

Top Third

4147

4271

4312

0.250 (0.055) 0.179

(0.057) 0.127

0.146 (0.061) 0.036

(0.068) 0.016

0.211 (0.057) 0.115

(0.054) 0.079

0.22

0.16

0.06

7293

7086

7426

0.507 (0.155) 0.319

(0.136) 0.039

0.133 (0.093) 0.034

(0.089) -0.038

0.316 (0.123) 0.167

(0.111) -0.003

(0.062) (0.072) (0.060) (0.151) (0.076) (0.106)

Results: CAL

• Pre-test score: – No difference in year 2, CAL children were

doing better in pre-test in year 3. • Post test score:

– Significant differences in Math in both years – Nothing in Language

• Difference in difference: Same conclusion

Test score difference: CAL

Pre test Mid test Post test CAL NO CAL DIFF CAL NO CAL DIFF

Year 2 Math -0.069 0.000 -0.069 0.739 0.705 0.034 1.094 0.805 0.289

(0.075) (0.078) (0.084) Verbal -0.025 0.000 -0.025 0.834 0.864 -0.031 0.688 0.702 -0.014

(0.081) (0.091) (0.090) Year 3

0.155(0.069)0.125

(0.076)

Math 0.155 0.000 0.235 -0.036 0.270 0.695 0.131 0.564 (0.072) (0.083)

Verbal 0.125 0.000 0.065 -0.033 0.098 0.168 0.009 0.159 (0.078) (0.076)

CAL: Diff in Diff

Table 7: Differences in differences estimate of the impact of the CAL program, by yearNumber of

Observations Math Verbal Total Vadodara Both Years 12562 0.358 -0.008 0.164

Vadodara Year 2 6079 (0.085) 0.343

(0.078) -0.006

(0.094) 0.192

Vadodara Year 3 6483 (0.083) 0.408

(0.073) 0.034

(0.079) 0.183

(0.073) (0.043) (0.053)

CAL: Differences in differences, by third in class

N Math Verbal Total Both Years Bottom Third 4310 0.400 0.063 0.227

Middle Third 4027 (0.105) 0.343

(0.091) -0.042

(0.110) 0.142

Top Third 4225 (0.093) 0.361

(0.082)

(0.083) -0.031 (0.083)

(0.100) 0.152

(0.093)

Table 8: Differences in differences for standard competencies, by yearMath Competencies for

Standard 1 Standard 2 Standard 3

Year 2 0.123 0.119 0.034(0.030) (0.026) (0.013)

Year 3 0.117 0.108 0.075(0.020) (0.018) (0.011)

CAL-BAL COMPARISON

Number of Observations Math Verbal Total

CAL Vadodara Year 2 6079 0.343 -0.006 0.192 -(0.083) -(0.073) -(0.079)

CAL Vadodara Year 3 6483 0.408 0.034 0.183 -(0.073) -(0.043) -(0.053)

BAL Vadodara Year 1 4123 0.205 0.122 0.175 (0.075) (0.075) (0.073)

BAL Vadodara Year 2 6080 0.276 0.217 0.265 (0.089) (0.073) (0.081)

Cost effectiveness

• Effect on math are of the same order of magnitude for BAL and CAL.

• Cost of CAL is Rs 722 per year, counting the cost of the computers and assuming they depreciate over 5 years(recurrent expenditures are Rs 367).

• Cost of BAL is Rs 107 per year. • CAL 6.7 times more expensive. • For math CAL is 26% more effective, for overall it is 30%

less effective. • CAL helps all children (it also affected all children).

Non-experimental resultsDirect and Indirect Effects

• Balsakhi was supposed to work with the low performing kids in the class.In practice, although allocation was free, it was largely the case.

• Assignment to the treatment group does not affect children only by increasing their probability to work with abalsakhi: removing some low scoring children from theclassroom may have a direct impact on children whoremain in the classroom, including high scoring children

• With additional assumptions, can we say something about how the balsakhi program decomposes into – Class size/tracking effect for remaining children – Direct program effect

The strategy• We do not exactly know who would have

been sent to balsakhi in control schools, if they had benefitted from the program.

• But the probability to be sent to the balsakhi was higher for children who were in the bottom 20 kids (since there were 20 seats in the balsakhi class)

• This allows us to look at the effect of balsakhi per se, and the effect of not being sent to balsakhi in a balsakhi class

Results: Comment

• Effect of balsakhi school is never significant for children who were not assigned to the balsakhi

• The effect is insignificant even in school where the non-balsakhi class is smaller than the balsakhi class.

• There seems to be no class size or not tracking effects: only a balsakhi effect!

Conclusions

• The remedial education intervention and the CAL interventions both appear to be veryeffective.

• Judged by the cost effectiveness to achieve a given increase in the average test score,Balsakhi is preferable.

• We find evidence that the entire effect of the balsakhi intervention is concentrated on the children who are affected, the effect on them isestimated to be between 0.6 and 1 standard deviation of test score (a very large impact).

What does this tell us about schools in developing world?

• Big effect of two interventions that is focusing on what kids do not know.

• No effect of just reducing class size • There are two differences:

– Incentives of the balsakhi to teach well, novelty of computer motivates kids

–Curriculum is adapted. • Subsequent research has shown that both elements

matter: motivated teachers matter, and focusing on basic skills matters too

MIT OpenCourseWarehttp://ocw.mit.edu

14.73 The Challenge of World Poverty Fall 2009 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms .