Information Campaigns to Strengthen Participation & Improve Public Schools: Experimental Evidence...

Information Campaigns to Strengthen Participation &

Improve Public Schools:

Experimental Evidence from Two Studies in

Uttar Pradesh and Madhya Pradesh

Stuti Khemani and Priyanka PandeyThe World Bank

Motivation Widening belief that encouraging “participation” by

citizens, or clients of services, can improve the quality of publicly provided education services

Education policies across states in India have created specific institutions for community participation to improve public schools

Emerging survey evidence that citizens are not informed about or aware of these institutions

We test (using the ‘gold standard’ of randomization) what impact providing information has on participation in and performance of public schools

Study 1 (with Pratham and J-PAL): Jaunpur district in Uttar Pradesh (UP)

Institutions of participation in UP Village Education Committee (VEC)

5 members: elected head of village government (Gram Pradhan), senior head teacher from village public schools, 3 parents

Roles of the VEC Monitoring: Visit schools and inspect records

Planning and Implementation

-- Receive and decide how to spend government grants

-- Participate in selection of Shiksha Mitras (contract teachers recruited from within the community) for government schools if there are not enough teachers.

-- Build or maintain school rooms

-- Manage mid-day meals, distribute scholarships

-- Raise more money for the school

-- Work with school teachers to improve education quality

-- Encourage parents to improve child attendance

Evidence on Participation: Baseline Survey Survey took place over March-June 2005

In each of 280 Villages (randomly selected), we collected data on: Learning Outcomes

30 households randomly selected all children between the ages of 7 and 14 are tested on basic

reading, writing, and math skills

Community Participation and Local Governance 10 households (of the above 30) randomly selected and surveyed All Government primary school head-teachers surveyed All VEC members surveyed

School Resources and School-Functioning All government primary schools surveyed

Parents don’t know that a VEC exists

Has Anyone Heard of the VEC?

1.5%

1.1%

5.0%

7.6%92.4%

Villagers Who Don't Know of aVillage Education Committee

Villagers Who Think there is aVEC

Villagers Who Believe there is aVEC, But Can't Name Any VECMembers

Villagers Who Can Name OnlyOne or Two VEC Members (the Pradhan and/orHeadmaster)

Villagers Who Can Name MoreVEC Members than Just thePradhan and Headmaster

* Based on 2,803 household surveys in 4 random blocks in the District of Jaunpur, UP. Each household is weighted by total number of households in village divided by number households surveyed in village.

VEC members don’t know their roles

Percent of VEC Members Who:

Don’t know that they are

members of the VEC

Have not heard of SSA*

Don’t know that funds are

provided to VECs to

improve schools*

Head Teachers 4.2% 0.5% 4.2%

Other VEC Members 22.7% 67.6% 73.6%

* Of those who know they are members of the VEC

When Asked What Are the Responsibilities of the VECWhat Do VEC Members Reply?

Percent of Following MembersWho List Item is a VEC Responsibility

Which Items are Part of Other VECVEC Responsibilities Headmasters Members

School inspection/visits 61% 30%Deciding how state money for the school is spent 20% 4%Authorizing additional Shiksha Mitras 9% 3%Hiring additional teachers from the community (not Shiksha Mitras) 5% 2%Prepared development plan for the village schools or village education plan 36% 14%Speaking to parents about child attendance 51% 14%Speaking to teachers or headmaster about teacher attendance 22% 14%Reporting school problems to higher authorities 32% 6%Raising money or materials from the community 9% 3%Monitoring distribution of textbooks 11% 5%Monitoring distribution of scholarships / grain 21% 16%Implementing the midday meal program 35% 30%None 0% 2%Other 18% 10%Don't know 1% 24%

Parents of children at low levels of learning are particularly unlikely to know this…

Perception versus Reality: Read

34%

6%

30%

22%

5%

10%

17%

11%

4%

10%

19%

23%

17%

8%

16%

36%

61%

78%

91%

1%1%

0%

1%0%1%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 1 2 3 4

Actual Read Level

Pa

ren

ts' P

erc

ep

tio

n o

f th

eir

Ch

ild

ren

's R

ead

Le

vel

Perceived Read Level = 4





* Data based on surveys of 2,803 households, and the testing of 5,377 children in 4 random blocks in the District of Jaunpur, UP. Child tests are weighted by number of children in village divided by number of children tested in surveyed households.

Parents of children at low levels of learning are particularly unlikely to know this…

Perception versus Reality: Math

27%

4%

34%

15%

6%

19%

31%

25%

11%

20%

51%

69%

87%

0%1% 2%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 1 2 3

Actual Math Level

Pa

ren

ts' P

erc

ep

tio

n o

f th

eir

Ch

ild

ren

's M

ath

Le

vel

Perceived Math Level = 3




* Data based on surveys of 2,803 households, and the testing of 5,377 children in 4 random blocks in the District of Jaunpur, UP. Child tests are weighted by number of children in village divided by number of children tested in surveyed households.

Interventions to Strengthen Community Participation: (1)Information about VECs

Small, informal meetings in each hamlet during 2 days in a village

Village-wide meeting on 2nd or 3rd day, with participation of key VEC members—Gram Pradhan and School Teacher

Distribution of pamphlets to VEC members listing and explaining their roles

Interventions to Strengthen Community Participation: (2) (1) + Testing Tool

In hamlet meetings, facilitators begin testing children; community invited to test children themselves and prepare hamlet-level “report cards”

In village-wide meeting, hamlet volunteers invited to present testing tools and “report cards”

Testing tool provides additional information (about learning), mobilizes community, and builds capacity in monitoring

Interventions to Strengthen Community Participation: (3) (2) + Teaching Tool of “Read India”

In village-wide meeting, Pratham facilitators present “Read India” teaching tool

Offer to train anyone who would like to hold reading classes

Teaching tool provides additional information (about how to improve learning), mobilizes community, (“come forward to make your village a reading village”), and builds capacity in teaching

Experience with Interventions 65 villages each received Interventions 1, 2, and 3 between

September and December 2005

Repeat visits in February to hand-out and explain pamphlets to VEC members

85 villages served as controls

215 village-wide meetings recorded in 195 “treatment” villages; on average, attendance of 108 villagers/meeting

(Village size: average 360 hshlds, min 146, max 1289)

Village Pradhan, and School Head-teacher attended 68.2% and 71.7% of meetings respectively

Experience with Interventions ”Read India” intervention received large response—local

youth volunteered to hold regular classes for 2-3 mths

No. of Class-Sequences in “Read India” Villages

11

1

2

4

6

3

10

7

3

4 4

2

3 3

2

0

2

4

6

8

10

12

0 1 2 3 4 5 6 7 8 9 10 11 12 13 16

No. of Class-Sequences

No. of Villages

Endline Survey Survey took place over March-June 2006

Impact evaluation using:

-- matched panel of 2500+ households

-- matched panel of 260+ govt. schools

-- matched panel of 16,400+ children

-- VECs: matched 240+ villages, with responses of all VEC members collapsed by village

Impact Evaluation Difference-in-Difference estimates

With and without controls

Standard errors clustered by village

Interventions did happen—impact on knowledge and training of VECs

(clustered std. errors in parentheses) T1 T2 T3 Any

Parents know about VEC

0.030(0.12)

0.019(0.014)

0.022(0.012)

0.023(0.009)

VEC member knows s/he’s a

member

0.106(0.05)

0.097(0.051)

0.068(0.048)

0.091(0.04)

VEC member has heard of

SSA

0.062(0.041)

0.069(0.04)

0.074(0.042)

0.068(0.03)

VEC member reports being

“trained”

0.161(0.054)

0.133(0.05)

0.219(0.054)

0.172(0.038)

No impact on self-reported VEC activity


VEC members report complaining

-0.012(0.029)

0.048(0.034)

0.007(0.028)

0.014(0.024)

VEC members report raising money

-0.005(0.011)

-0.01(0.011)

-0.007(0.01)

-0.007(0.009)

VEC members reported school inspections

-0.564(1.195)

-0.890(1.32)

-0.433(1.502)

-0.624(1.088)

Average over the three-0.193(0.395)

-0.284(0.434)

-0.144(0.496)

-0.206(0.36)

No impact on self-reported VEC activity No impact on

-- self-reported hiring of additional teachers,

-- distribution of scholarships,

-- implementation of mid-day meals,

-- various others

No impact on parent-reported participation


Visited school to monitor orcomplain

-0.016(0.025)

-0.040(0.025)

-0.014(0.025)

-0.023(0.020)

Donated to school-0.001(0.012)

-0.001(0.011)

-0.006(0.010)

-0.002(0.008)

Volunteeredat school

-0.008(0.010)

-0.020(0.009)

-0.010(0.010)

-0.013(0.008)

Complained or talked aboutproblems in school

0.028(0.019)

0.015(0.017)

0.019(0.019)

0.021(0.014)

Average over family of outcomes

0.001(0.009)

-0.011(0.009)

-0.003(0.010)

-0.005(0.007)

No impact on head-teacher reported parent participation

(clustered std. errors in parentheses)

T1 T2 T3 Any

Have parents visited theschool

-0.037(0.070)

-0.034(0.067)

-0.105(0.073)

-0.056(0.053)

Have you organizeda parents meeting

0.029(0.070)

0.118(0.060)

0.097(0.070)

0.082(0.054)

Did parents volunteerin the school

0.069(0.060)

0.045(0.060)

0.072(0.065)

0.061(0.047)

Did the school getan allocation from the

panchayat

-0.010(0.029)

0.009(0.029)

-0.006(0.031)

-0.002(0.023)

Did the school receiveparents' donations

-0.042(0.041)

-0.009(0.047)

-0.051(0.038)

-0.033(0.036)

Average over the familyof outcomes

0.002(0.029)

0.026(0.025)

0.001(0.027)

0.011(0.020)

No impact on head-teacher reported school resources


Textbooks0.026

(0.028)-0.013(0.034)

-0.013(0.036)

0.000(0.026)

Scholarships0.049

(0.048)-0.031(0.055)

0.054(0.049)

0.022(0.042)

Seating Furniture-0.063(0.052)

-0.057(0.053)

-0.040(0.044)

-0.054(0.038)

Maps and Charts0.005

(0.039)0.002

(0.039)0.029

(0.033)0.011

(0.031)

Electricity-0.031(0.027)

-0.022(0.030)

-0.014(0.022)

-0.023(0.021)

Water-0.054(0.033)

-0.071(0.038)

0.003(0.020)

-0.043(0.021)

Toilets-0.016(0.061)

-0.052(0.062)

-0.107(0.055)

-0.056(0.048)

Does the school servemidday meal

0.001(0.051)

0.059(0.048)

0.054(0.047)

0.037(0.043)

Average over the familyof outcomes

-0.005(0.013)

-0.028(0.014)

-0.017(0.013)

-0.017(0.011)

No impact on number & presence of teachers in public schools


Number of teachers (head-teacher report)

0.159(0.169)

0.116(0.152)

0.152(0.160)

0.141(0.127)

Teacher absence(head-teacher report)

-0.022(0.037)

-0.031(0.031)

-0.028(0.033)

-0.027(0.026)

Teacher absence(random check)

-0.007(0.028)

-0.011(0.025)

-0.046(0.025)

-0.021(0.021)

Teachers teaching/Teachers present

0.010(0.059)

-0.042(0.053)

-0.088(0.065)

-0.040(0.047)

Average effect overthe family of outcomes

0.005(0.038)

-0.008(0.031)

-0.041(0.033)

-0.014(0.027)

No impact on public school enrollment and attendance


Log(Boys Enrolled)(head-teacher report)

0.041(0.048)

0.027(0.050)

-0.020(0.069)

0.017(0.045)

Log(Girls Enrolled)(head-teacher report)

0.001(0.077)

0.020(0.074)

0.013(0.075)

0.012(0.071)

Boys present (direct observation)/Boys Enrolled

0.029(0.041)

-0.004(0.042)

-0.053(0.041)

-0.008(0.032)

Girls present (direct observation)/Girls Enrolled

0.053(0.043)

-0.006(0.035)

-0.027(0.035)

0.006(0.028)

Average effect overthe family of outcomes

0.031(0.027)

0.009(0.026)

-0.022(0.029)

0.007(0.023)

Impact on Learning(clustered std. errors in parentheses) T1 T2 T3

Reading level at endline0.009

(0.025)0.014

(0.025)0.064

(0.026)

Math level at endline-0.001(0.016)

0.014(0.016)

-0.005(0.017)

Writing level at endline0.009

(0.009)-0.005(0.008)

0.004(0.009)

N=15,500+; Controls include child age, sex, baseline learning levels, baseline school enrollment status; Reading levels: 0=nothing, 1=letters, 2=words, 3=paragraphs, 4=story; Math levels: 0=nothing, 1=numbers, 2=subtraction or division

Closer look at impact on readingChildren who could not read at Baseline (N=2288):

T1 T2 T3

First Stage: Attend Read Class

IV Impact of Read Class

read letters0.042(.031)

0.033(.034)

0.077(.035)

0.131(.023)

0.592(.303)

read paragraph

-0.006(.015)

-0.013(.012)

-0.007(.014)

-0.051(.106)

read story-0.006

(.01)-0.013(.008)

-0.008(.009)

-0.063(.074)

Closer look at impact on readingChildren who could read letters at Baseline (N=3539)

T1 T2 T3

First Stage: Attend Read Class

IV Impact of Read

Class

read letters-0.008(.016)

-0.015(.014)

0.022(.013)

0.132(.02)

0.163(.098)

read paragraph-0.010(.022)

-0.025(.021)

0.036(.022)

0.277(.171)

read story-0.001(.014)

-0.010(.014)

0.033(.017)

0.258(.135)

Closer look at impact on readingChildren who could read words or paragraph at baseline (N=3673)

T1 T2 T3

First Stage: Attend

Read Class

IV Impact of Read

Class

read letters-0.001(.006)

0.006(.004)

0.006(.004)

0.074(.012)

0.068(.065)

read paragraph

0.031(.019)

0.009(.019)

0.044(.017)

0.606(.271)

read story0.009(.026)

0.009(.025)

0.032(.027)

0.447(.388)

Improvement over time among illiterate children

Midline reading level: Children reading nothing at baseline

-20%0%

20%40%60%80%

100%120%

Con

trol

trea

tmen

t 1

trea

tmen

t2

trea

tmen

t 3

Afte

r re

adcl

ass

(est

imat

ed)

Letter

word or para graph

Story

Improvement over time among children who could recognize letters

Midline level: Children who could read letters at baseline

0%

20%

40%

60%

80%

100%

120%

Contr

ol

treatm

ent2

Aft

er

read

cla

ss

Letter

word or paragraph

Story

Improvement over time among children who could read words or paragraphs

Midline level: Children reading words or paragraph at baseline

0%20%40%60%80%

100%120%140%

Contr

ol

treatm

ent2

Aft

er

read

cla

ss

Letter

word or paragraph

Story

Summary of Impact Impact on learning driven by local youth volunteering to

hold reading classes

Children who enrolled in these classes made significant improvements in reading within a few months

No discernable impact on activity within public schools, or by VECs

No anecdotal evidence of VECs, or Pradhans, or school teachers, supporting these volunteer-led reading classes

Bottomline—evidence of participation to improve learning outcomes, but outside public schools

Information Campaigns to Strengthen Participation & Improve Public Schools: Experimental Evidence...

Documents

Transcript of Information Campaigns to Strengthen Participation & Improve Public Schools: Experimental Evidence...