OUTCOME-BASED AND COMPETITION-BASED POLICIES OF SCHOOL EVALUATION:a comparison of school performance (and

perverse effects) in two mid-sized Hungarian towns

HORN, Daniel BODIS, LajosEUI, Florence Corvinus, Budapestdaniel.horn@eui.eu bodis@uni-corvinus.hu

Presentation at the INVALSI-APPAM “Improving Education through Accountability and Evaluation Lessons from Around the World” conference

Rome, Italy, October 3-5, 2012.

Outline

• National background• “Parktown” and “Birdtown” – outcome-based and competition-based policies of

school evaluation

• Quantitative results (empirics not presented)• Qualitative speculations• Conclusion

03/10/2012 Horn-Bodis - Outcome and Competition Based Policies 2

National Background• (practically) Free school choice• Per-student lump-sum grant• Demographic decline• Teachers are public servants (hard to fire, fixed salary scale)

– Result: segregated school system(both student and teacher)(empirics for Hungary: Kertesi-Kezdi 2010, Varga 2011)

Gradual adaptation: „competition-based policy” (Birdtown)• National Assessment of Basic Competencies (NABC) since

2001– School assessment system (individual panel since 2010)

Reform: „outcome-based policy” (Parktown)

03/10/2012 Horn-Bodis - Outcome and Competition Based Policies 3

The two towns

• Birdtown– 75,000 inhabitants (county capital)– 2004: 14 schools – 7500 students– 2008: 11 schools – 6200 students (7,5% drop)

• Parktown– 50,000 inhabitants (not a county capital)– 2004: 10 schools – 4100 students– 2008: 5 schools – 3500 students (4,7% drop)

03/10/2012 Horn-Bodis - Outcome and Competition Based Policies 4

Reform in Parktown

• On the 1st July 2007.• Restructuring the whole system– Closing all 10 schools and opening 5 new

(2 closed, 2 merged into 2 other, and 1 handed over to the Church)

– Firing all and hiring 70% of teachers (performance based)

– Redrawing (and starting to use) catchment-areas

• NABC was a tool in the reform (see below)03/10/2012 Horn-Bodis - Outcome and Competition

Based Policies 5

School performance - Birdtown• Focus on “quality”– Any school can be good in one (or more) aspects.– Testing is important (but not that much)– School evaluations are based on “constructive way of life”,

“local patriotism” and “continuous professional development”

“It was around 2005 when the politicians and administrators here [in Birdtown] started to say that the parents are voting

with their feet. So we didn’t try to stop it. We said that if you can’t add a Chinese gymnastic course on top of

‘hottentot’ language course, and you can’t make them stay, we’re sorry.”

(an administrator at the local government, Birdtown)03/10/2012 Horn-Bodis - Outcome and Competition

Based Policies 6

School performance - Parktown• Focus on standardized test-scores

– Schools are compared on level scores annually– No-high stakes– Depends much on the (politically well imbedded and

charismatic) mayor, who used the testing to hammer through his reform in 2006

“Then we went through each school one-by-one, which created the first smaller turbulence. They [the General Assembly] did not like it. They did not like to face the fact that there was practically no school that showed above national average performance. Thus

we had no good schools. We had no exceptional primary schools.”

(the mayor of Parktown)

03/10/2012 Horn-Bodis - Outcome and Competition Based Policies 7

Quantitative results• National Assessment of Basic Competencies (NABC)

– School assessment based on standardized student test scores– Covering 6th, 8th and 10th grade– Math and reading– Since 2001, but panel since 2008-2010– Detailed background questionnaire– Administrative dataset on all students

• Panel data on 22 mid-sized towns between 2004-2006, 6th and 8th grade

• Results (diff-in-diff):– Parktown does not differ from the other towns in level. (For Birdtown

it is lower)– Value-added (VA) for Birdtown is non-significantly different from the

other 20 towns – VA for Parktown is lower(!) after the reform

03/10/2012 Horn-Bodis - Outcome and Competition Based Policies 8

Qualitative speculations

• Why have the VA dropped?1. Bad (?) outcome measure: focus on level results

a. Teachers can play the system i. Teach to the test (we saw that)ii. Reallocate resources (this too)iii. Selective testing (have not documented)

But no increase in level either! (not presented here)And the question is still valid: Why has VA dropped?

03/10/2012 Horn-Bodis - Outcome and Competition Based Policies 9

Qualitative speculations

• Why has the VA dropped?2. We are too early to judge (cf. Elmore 2003).

Well, yes.

3. There are important missing pieces in the reform in Parktown.

03/10/2012 Horn-Bodis - Outcome and Competition Based Policies 10

Missing pieces

• Introducing outcome regulation, inevitably puts a limit on the input as well as the process regulation (cf. Milgrom and Roberts 1992).– Parktown has changed input regulation.– but have not dealt with process measures:

a) Lack of „expert” teachersb) Lack of forums for deliberation and feedbackc) Lack of (uniformly) good teacher training

(participation)

03/10/2012 Horn-Bodis - Outcome and Competition Based Policies 11

Concluding thoughts• Doing a full-scale reform in a decentralized

system has its benefits:– decrease between school segregation– fame within the country

• but its effects on performance are questionable

03/10/2012 Horn-Bodis - Outcome and Competition Based Policies 12

Thank you for the attention!

daniel.horn@eui.eu

03/10/2012 Horn-Bodis - Outcome and Competition Based Policies 13