Larysa Minzyuk - Felice Russo Department of Management and Economics - University of Salento (Lecce)

larissa.minzuk@unisalento.itfelice.russo@unisalento.it

“Improving Education through Accountability and Evaluation Lessons from Around the World”

Rome, 3-5 October 2012

The Causal Effect of Class Size on Pupils’ Performance: Evidence

from Italian Primary Schools

OUTLINES

Motivations and purposes Identification strategy

Regression Discontinuity design Data and procedure Results and conclusive remarks

Motivations and purposes • The relationship between class size and education attainment

has been widely explored, but the existing evidence on the class size effect is still contrasting and somewhat inconclusive

• In this paper, we estimate the class size effect in Italian public primary schools by using Regression Discontinuity (RD; Thistlewaite-Campbell, 1960, JEP) design, which has recently become a standard evaluation framework for solving causal issues with non-experimental data (not only in education)– in Italy a limited research has been done on this issue so far

e.g. data limitations; among the known studies, see Bratti et al. (2007, GE), Brunello and Checchi (2005, EER), Quintano et al. (2009, REST), Russo (2010)

• Our main results1. In Italian data, we do not find a significant evidence which

supporting class-size reduction policy2. There is an evidence of sorting of pupils’ characteristics

around cut-offs points (25 pupils): pupils with “unfavorite” socio-economic background are in smaller classes

Regression Discontinuity Design

1. An RD-based evaluation is appropriate when increases in grade-enrolment (forcing variable) are linked with jumps in class-size (treatment variable) as predicted by:

(a) the threshold rule generating a (b) discontinuous relation between the two variables

2. Individuals (schools and families) cannot precisely manipulate the grade-enrolment in order to receive or avoid treatment (i.e. to affect whether or not the fall on one side of the threshold or the other)

3. Smoothness condition: Other variables are smooth functions of the forcing variable conditional on treatment (i.e. the only reason pupils’ outcomes should jump at the cut-off is due to the discontinuity in the level of treatment)

If 1.,2.,3. jointly hold, effects of class size on pupils’ test

scores can be interpreted as the local average treatment effect of class size

Data and procedure (1)

We conduct our study using information from two sources:1. The first is the INVALSI test results of V grade pupils in primary

schools in 2008/09. These results are available for 150,000 pupils coming from 5,303 public and private primary schools (‘circoli didattici’) from all Italian regions– we restrict our analysis to the public schools whose testing

procedure was assisted by INVALSI supervisors 2. The second source of information is school-level administrative

data from the Italian Ministry of Education (MIUR; those data do not contain information about schools in regions with a special statute)– we matched the INVALSI data sample with the dataset on

school characteristics and class size coming from MIUR– because of missing data on pupils characteristics in INVALSI

data, we have 25,407 pupils coming from 1,561 school units

• Predicted class size (Angrist and Lavy, 1999)

Φis is the V grade enrolment at school s where the pupil i studied in 2008/09, int (•) is the function that takes the greatest integer less than the given argument

Average and Predicted

Class Size, 2008/09

Data and procedure (2)

1]25/)1int[(

is

isisPC

Compliance of Schools to the Rule, 2008/09

Data and procedure (3)• A standard model of fuzzy RD can be described as follows

(van der Klaauw, 2002, p. 1262):

Pis is the test score of pupil’s i in school s, CSis is average class size in V grade at school level, Φis is the V grade enrolment at school level, indicates the cut-off values of enrolment (multiples of 25), and α (•) and β (•) are functions of enrolment

• When enrolment is a discrete variable, class-size effect can be estimated only parametrically (Lee and Card, 2008)

we decide a linear specification for both control functions α (•) and β (•), choosing the piecewise linear splines whose kinks correspond to the values of cut-offs (Urquiola and Verhoogen, 2009, AER; Zada et al., 2009) (2SLS). For instance (1st stage; 2 knots):CSis = β + β1 1[Φis > 25] + β2 1[Φis > 50] + β3 Φis + β4 (Φis - 25)1[Φis > 25] + β5 (Φis - 50)1[Φis > 50] + μis

).()(1)|(

,)()|(

isisisis

isisisisis

CSE

CSEP

First stage and base IV specifications (+/- 3 pupils intervals), 2008-09

First stageIV stage IV stage with selected obs.Math Italian

language Math Italian language

Class size -0.005190 -0.003018 -0.002389 -0.0006561[Φis > 25] yes1[Φis > 50] 1.085096***1[Φis > 75] 0.670140**1[Φis > 100] -0.695183**Φis 0.024155***(Φi - 25)1[Φi > 25] -2.875964*** yes yes yes yes(Φi - 50)1[Φi > 50] -1.834880*** yes yes yes yes(Φi - 75)1[Φi > 75] -0.852245*** yes yes yes yes(Φi - 100)1[Φi > 100] -0.960405*** yes yes yes yesIsei index of father no no no 0.001056*** 0.001029***Isei index of mother no no no 0.000284*** 0.000270***Mother’s education no no no 0.015243*** 0.016755***Pupils with special needs no no no yes yesConstant 21.688432*** 0.710567** 0.698901* yes yesObservations 5,396 5,396 5,396 5,396 5,396Adjusted R-squared 0.2756 0.0066 0.0048 0.0508 0.0679

Note: In all regressions, standard errors are clustered by enrolment levels, see Lee and Card (2008). *** p<0.01, ** p<0.05, * p<0.1.

Reduced form of selected observables (+/- 3 pupils intervals), 2008/09

ISEI index father

ISEI index mother

Mother's education

Pupils with special needs

1[Φis > 25] yes yes yes -0.3069***1[Φis > 50] 2.5542*** -1.9496* yes 0.4053***1[Φis > 75] yes yes yes yes1[Φis > 100] yes yes yes -0.560***Φis yes yes yes -0.00296**(Φi - 25)1[Φi > 25] yes yes yes 0.067425***(Φi - 50)1[Φi > 50] -0.8283** yes yes -0.1689***(Φi - 75)1[Φi > 75] 2.031** yes yes(Φi - 100)1[Φi > 100] -0.6218*** -1.53963*** -0.2237*** 0.388616***

Constant 41.03482*** 29.28249*** 3.330471*** 0.690685***

Observations 5,396 5,396 5,396 5,396

Adjusted R-squared 0.0077 0.0064 0.0054 0.0327

Note: In all regressions, standard errors are clustered by enrolment levels, see Lee and Card (2008). *** p<0.01, ** p<0.05, * p<0.1.

First stageIV stage IV stage with selected obs.

Math Italian language Math Italian language

Class size -0.006296 -0.004871 -0.006244 -0.0051641[Φis > 25] yes1[Φis > 50] yes1[Φis > 75] yes1[Φis > 100] -0.935615**Φis 0.035858*** -0.016370* yes yes yes

(Φi - 25)1[Φi > 25] 0.035858*** yes yes yes yes

(Φi - 50)1[Φi > 50] -0.850894*** yes yes yes yes

(Φi - 75)1[Φi > 75] yes yes yes yes yes

(Φi - 100)1[Φi > 100] -0.850894*** yes yes 0.019211* yesIsei index of father no no no 0.000961*** 0.001014***Isei index of mother no no no yes yesMother’s education no no no 0.016281*** 0.017683***Pupils with special needs no no no 0.000394*** 0.000374***Constant 20.758594*** 0.727714*** 0.730816*** 0.626042*** 0.627706***Observations 10,685 10,685 10,685 10,685 10,685Adjusted R-squared 0.3791 0.003 0.0008 0.0527 0.0732

First stage and base IV specifications (+/- 5 pupils intervals), 2008-09

Note: In all regressions, standard errors are clustered by enrolment levels, see Lee and Card (2008). *** p<0.01, ** p<0.05, * p<0.1.

ISEI index father

ISEI index mother

Mother's education

Pupils with special needs

1[Φis > 25] yes yes yes -0.187210***

1[Φis > 50] yes -2.533225** yes yes

1[Φis > 75] yes yes yes yes

1[Φis > 100] -2.737642*** -4.413059*** yes -0.674291***

Φis 0.042128*** yes 0.003121** yes

(Φi - 25)1[Φi > 25] -0.492997** yes yes yes

(Φi - 50)1[Φi > 50] yes 0.658670* yes yes

(Φi - 75)1[Φi > 75] yes yes -0.116766** 0.386933***

(Φi - 100)1[Φi > 100] -0.646475*** -1.600302*** -0.225947*** yes

Constant 40.229238*** 26.607819*** 3.229377*** 0.636764***Observations 10,685 10,685 10,685 10,685Adjusted R-squared 0.0067 0.0043 0.0051 0.0131

Reduced form of selected observables (+/- 5 pupils intervals), 2008/09

Note: In all regressions, standard errors are clustered by enrolment levels, see Lee and Card (2008). *** p<0.01, ** p<0.05, * p<0.1.

Treatment effect (+/- 3 and 5 pupils intervals), 2008-091 cut-off (25) 2 cut-off (50) 3 cut-off (75) 4 cut-off (100)

Three-pupil intervalDep.Var: Math yes -0.0355* yes 0.1483**Average class-size (left side of cut-off) yes 23.078 yes 23.876Average class-size (right side of cut-off) yes 20.568 yes 21.470Obs. 2,703 1,544 786 363Three-pupil intervalDep.Var: Italian yes yes yes 0.1137**Average class-size (left side of cut-off) yes yes yes 23.876Average class-size (right side of cut-off) yes yes yes 21.470Obs. 2,703 1,544 786 363Five-pupil intervalDep.Var: Math -0.0133* 0.0728** yes 0.2193***Average class-size (left side of cut-off) 21.355 22.983 yes 23.612Average class-size (right side of cut-off) 16.556 20.028 yes 21.470Obs. 5,414 3,184 1,415 671Five-pupil intervalDep.Var: Italian yes 0.063** yes 0.2847***Average class-size (left side of cut-off) yes 22.983 yes 23.612Average class-size (right side of cut-off) yes 20.028 yes 21.470Obs. 5,414 3,184 1,415 671

Note: In all regressions, standard errors are clustered by enrolment levels, see Lee and Card (2008). *** p<0.01, ** p<0.05, * p<0.1.

Number of schools in enrolment intervals

(+/- 3 pupils), 2008/09

Number of schools in enrolment intervals

(+/- 3 pupils, sample compliant), 2008/09

Reduced form estimates of selected observables, sample of compliant schools (+/-

3 pupils intervals), 2008/09ISEI index father

ISEI index mother

Mother's education

1[Φis > 25] -7.773760*** -10.318900*** -0.763182***

1[Φis > 50] 1.930775* -4.608166*** yes1[Φis > 75] -11.345361*** yes -0.186474*

1[Φis > 100] -3.084358** yes yesΦis yes yes yes(Φi - 25)1[Φi > 25] 3.097997*** 1.748179*** 0.255036***(Φi - 50)1[Φi > 50] -0.677889* 0.654220** yes(Φi - 75)1[Φi > 75] 3.965961*** yes 0.095906***(Φi - 100)1[Φi > 100] -0.052756*** -2.039584*** -0.235058***

Constant 40.956966*** 29.367030*** 3.347153***

Observations 4,111 4,111 4,111

Adjusted R-squared 0.0102 0.0140 0.0093

Note: In all regressions, standard errors are clustered by enrolment levels, see Lee and Card (2008).

*** p<0.01, ** p<0.05, * p<0.1.

• In this paper we make an attempt to estimate the class-size effect on the pupils' performance using the data from Italian primary schools. We base our estimation strategy on RD design. To apply RD estimation strategy, practitioners have to test if the assumptions of RD analyses are not infringed, otherwise it would invalid to infer a “treatment” effect of class size on pupils' test results

1. In Italian data, we do not find a significant evidence which would strongly support class-size reduction policy

2. When focusing on small intervals (+/-3 and 5 pupils) around selected enrolment cut-offs, selection problem is not evident, as, on the one hand, we do not observe clear stacking behaviour of schools at the thresholds, and, on the other hand, pupils' characteristics result to be distributed smoothly in the large majority of cut-offs for this subsample

Conclusive remarks (1)

Conclusive remarks (2)

3. In contrast, as we observe in our data, we find that the stacking behavior is more evident in the reduced sample of compliant schools (+/-3 pupils) and there is a clearer evidence of sorting of pupils' characteristics around cut-offs points: in this sample, right sides of cut-off intervals include more pupils with “unfavorite” socio-economic background class size are largely used in primary public schools as a kind of compensatory policy (West-Woessman, 2006)

• Urquiola and Verhoogen (2009) have found an evidence that Chilean schools might exercise selection policy on enrolment. The authors suggest caution when using RD, especially in application to private schools that may have a better control over enrolment compared to public schools

• Zada et al. (2009) have found an evidence of “selection” policy in public secondary schools in Israel as well. It is worth noting that the authors have not found it in Israelian public primary schools.