1

Impact Evaluation: The case of Bogotá’s concession schools

Felipe Barrera-OsorioWorld Bank

October 2010

2

Outline

1. Hypotheses2. Detour: Evaluation of Programs3. Empirical strategy4. The data5. Results

Why concessions (a PPP) may increase quality of education?

• An application of an already proven pedagogic model• Concession schools are financially stable, ensuring the

stability of the pedagogic model.• Freedom to choose (and fire) teachers and management

staff• Better infrastructure• Concession schools work actively with the families and the

community– “Affection deficit” and family problems are important issues in

these communities• Nutrition is an important variable in educational

outcomes

3

4

Three hypothesis

• Dropout rates are lower in concessions schools than in similar, public schools.

• Nearby schools have lower dropout rates than public schools outside the influence of concessions.

• Test scores are higher in concessions schools than in similar, public schools.

5

Detour: Evaluating Programs

What is Impact Evaluation?

• Impact evaluation is a set of methods to identify and quantify the causal impact of programs

• For example, What is the effect of a concession schools on drop-out rates and test scores?

Why are these effects difficult to estimate?

• The basic question: What would it happen in the absence of the program? (e.g., what is the contrafactual?)

• We need the same individual with and without the program– For example, we need to observe John attending one concession

school, and (the same) John attending a public school– …but it is impossible to observe the same individual in

both states!• Solution: “build” the correct contrafactual

– Find individuals who do not have the benefits of the program, but are very similar to the ones that have the program

– For example, find David who is very similar in all characteristics to John…

What is the problem?• In short, the problem is to find the correct comparison

(control) group– The control group and the treatment group should have

the same characteristics, observable and non-observable, before the beginning of the program

– External factors will affect in the same way control and treatment group

• An, usually, people self-select into programs, and therefore, the beneficiaries are (with high probability), different that the ones that did not enter into the program

The basic intuition: only data before and after the program

Y: Test scores

Timet = 0

before program

t = 1

after program

Impact

of the

program?

NO!

We need a contrafactual

Intervention

We need the right comparison groupY

Timet = 0

before program

t = 1

after program

Impact

of the

program

Intervention

Control

Treatment

Y: Test scores

The basic intuition 2: only data after the program….

Y

Timet = 0

before program

t = 1

after program

Impact

of the

Program?

Intervention

Control

Treatment

Y: Test scores

We need the right comparison group!Y

Timet = 0

before program

t = 1

after program

Impact

of the

Program?

NO!

At t=0, two groups were

very different…

Intervention

Control

Treatment

Y: Test scores

Four possibilities to find or construct the right control group

• Prospective evaluation– Randomization of benefits– Randomization of entry (phase-in approach)– Randomization of information (encouragement design)

• Retrospective evaluation– Regression discontinuity analysis– Instrumental variables– Differences in differences– Propensity and matching estimators

Randomization

• Lottery among individuals will separate the sample between winners and losers– Model of “over-subscription”

• A lottery will create homogenous control and treatment group: they will be very similar in all characteristics, observable and non-observable.

Regression discontinuity• When to use this method?

– The beneficiaries/non-beneficiaries can be ordered along a quantifiable dimension.

– This dimension can be used to compute a well-defined index or parameter.

– The index/parameter has a cut-off point for eligibility.– The index value is what drives the assignment of a

potential beneficiary to the treatment. (or to non-treatment)

Intuition

• The potential beneficiaries just above the cut-off point are very similar to the potential beneficiaries just below the cut-off point.

• We compare outcomes for individuals just above and below the cutoff point.

6065

7075

80O

utco

me

20 30 40 50 60 70 80Score

Regression Discontinuity Design - Baseline

Non-poor

Poor

6570

7580

Out

com

e

20 30 40 50 60 70 80Score

Regression Discontinuity Design - Post Intervention

Treatment effect

Difference in differences1. Estimating the impact of a “past” program2. We can try to find a “natural experiment” that allows us to

identify the impact of a policy. For example, An unexpected change in policy could be seen as “natural

experiment” A policy that only affects 16 year olds but not 15 year olds

3. We need to identify which is the group affected by the policy change (“treatment”) and which is the group that is not affected (“control”).

4. The quality of the control group determines the quality of the evaluation.

Intuition

• Find a group that did not receive the program• …with the same pattern of growth in the

outcome variable before the intervention– The two groups, treatment and control, have the

same profile before the intervention

Intuition: when it is right to use DD?

Timet = -1 t = 1

after program

Impact

of the

Program

Intervention

Control

t = 0

Slopes, before intervention

(between -1 and 0) were equal

Y: Test scores

Intuition: when it is right to use DD?

Timet = -1 t = 1

after program

Impact

of the

Program?

NO: wrong comparison

group

Intervention

Control

t = 0

Slopes, before intervention

(between -1 and 0) were different

Y: Test scores

Propensity and matching estimation

• Find the comparison group from a large survey• Each treatment will have a comparison that has

observable characteristics as similar as possible to the treated individuals

• This method assumes that there is not self-selection based on unobservable characteristics:– Selection is based on observable characteristics

How is this procedure done? Two steps

• Estimation of the propensity score: The propensity score is the conditional probability of receiving the treatment given the pre-treatment variables.– Estimate the probability of been treated based on the observable

characteristics• Estimation of the average effect of treatment given the

propensity score– match cases and controls with exactly the same

(estimated) propensity score;– compute the effect of treatment for each value of the

(estimated) propensity score– obtain the average of these conditional effects

25

…Back to Concessions

26

Empirical Strategy for Concessions: propensity and matching estimation

• Propensity score estimation:– “Find” controls that have similar characteristics as the

treatment one

• Matching estimation:– Once the controls are found, estimate impact:

)0,)(/()1,)(/(~

0~

1~

TXPYETXPYEIP

)()()/Pr( xfXPXT

27

The data

1. C100 and C600: Rich information about schools• For 1999 and 2003: Administrative personnel,

number and level of education of teachers, physicians and psychologists, students by grade and by age, students who failed a grade and dropped out.

• For 1999: infrastructure variables like computers, rooms, labs, etc.

2. ICFEX 2003: Standardized test scores• Test scores at the individual level• Some information at the individual level

28

Table 1. Basic stadistics (public schools)

Mean Std. Dev. Mean Std. Dev.A. CHARACTERISTICSLocalization

1 0.10 0.30 0.12 0.332 0.01 0.10 0.00 0.003 0.05 0.22 0.00 0.004 0.06 0.24 0.00 0.005 0.15 0.36 0.12 0.326 0.04 0.19 0.00 0.007 0.30 0.46 0.44 0.508 0.12 0.32 0.13 0.349 0.17 0.38 0.19 0.40

Type of the school Male 0.07 0.26 0.00 0.00 Female 0.03 0.16 0.00 0.00 Coed 0.90 0.30 1.00 0.00

Type of secondary Academic 0.63 0.48 0.93 0.26 Technical 0.29 0.45 0.00 0.00 Both 0.07 0.26 0.07 0.26

Schedule Complete 0.03 0.18 1.00 0.00 Morning 0.40 0.49 0.00 0.00 Afternoon 0.14 0.35 0.00 0.00 Evening 0.43 0.49 0.00 0.00

Number of students Total school 2954.60 1376.76 987.98 90.17 Taking exam 122.15 79.16 65.33 9.98 Old for grade 47.28 45.15 12.11 7.03 Repeating grade 103.30 73.72 47.56 35.93 Ratio teacher/stud. 28.44 6.20 27.20 3.48

Year of education, teachers 16.07 2.28 16.02 0.99

Non-concession Concession

29

Table 1. Basic stadistics (continuation)B. OUTCOMES

Dropout rates

Grade 1 0.040 0.05 0.020 0.02 Grade 2 0.031 0.04 0.023 0.02 Grade 3 0.026 0.04 0.019 0.02 Grade 4 0.026 0.03 0.015 0.02 Grade 5 0.020 0.33 0.018 0.02 Grade 6 0.071 0.09 0.021 0.02 Grade 7 0.040 0.04 0.025 0.02 Grade 8 0.038 0.04 0.032 0.03 Grade 9 0.039 0.05 0.036 0.03 Grade 10 0.033 0.04 0.038 0.04 Grade 11 0.012 0.02 0.021 0.05

Test scores Mathematics 42.08 5.13 41.68 4.77 Reading 50.99 7.13 51.04 6.79

Number of observations Students Max 36244 1056 Min 34218 1013

Schools 2790 17

Non-concession Concession

30

Flavor of the (strong) result

Concessions reduced the dropout rate significantely

Concessions reduce the jump in the desertion in grade 6

31

Test scores:

Public schools have lower test scores

Concessions and public non-concessions are “similar”

32

Table 2. Basic statistics, schools near concession versus public schools

A. CHARACTERISTICS OF SCHOOLS

1. "Before concessions" (1999) Mean Std.dev Mean Std.dev

Infrastructure Number of computers 17.82 16.75 24.96 22.69 Number of classrooms 21.40 10.58 22.24 12.93 Number of bathrooms 1.45 0.78 1.67 0.87 Number of sport facilities 1.78 0.54 2.59 1.52

Teachers Number of teachers 30.27 16.34 34.46 24.06 Number of admin. pers. 6.97 4.45 9.69 11.96 Number of physicians 0.06 0.24 0.45 1.10 Number of psychologists, etc 1.52 1.29 2.19 1.85 % teachers with second. educ 0.07 0.18 0.06 0.12 % teachers with college educ 0.60 0.23 0.60 0.62 % teachers with graduate. educ 0.25 0.17 0.34 0.34 Ration students / teachers 24.92 3.24 26.16 28.78

Dropout rates 0.09 0.07 0.07 0.06

Schools near a concession Schools far a way from concession

33

2. Follow up (2003)Localization

1 0.00 0.05 0.10 0.302 0.00 0.00 0.01 0.103 0.00 0.00 0.05 0.224 0.04 0.19 0.06 0.245 0.20 0.40 0.15 0.366 0.00 0.00 0.04 0.197 0.15 0.36 0.30 0.468 0.45 0.50 0.12 0.329 0.16 0.37 0.17 0.38

Gender of the school Male 0.00 0.00 0.07 0.26 Female 0.00 0.00 0.03 0.16 Coed 1.00 0.00 0.90 0.30Type of secondary Academic 0.80 0.40 0.63 0.48 Technical 0.06 0.24 0.29 0.45 Both 0.14 0.35 0.07 0.26Schedule Complete 0.04 0.20 0.03 0.18 Morning 0.28 0.45 0.40 0.49 Afternoon 0.07 0.25 0.14 0.35 Evening 0.61 0.49 0.43 0.49Number of students Total school 2544.18 786.25 2954.60 1376.76 Taking exam 136.97 69.02 122.15 79.16 Old for grade 35.52 35.30 47.28 45.15 Repeating grade 114.93 88.69 103.30 73.72 Ratio teacher/stud. 31.42 2.56 28.44 6.20Year of education, teachers 16.49 0.73 16.07 2.28

Dropout rates

Grade 1 0.035 0.04 0.040 0.05 Grade 2 0.030 0.03 0.031 0.04 Grade 3 0.030 0.03 0.026 0.04 Grade 4 0.018 0.02 0.026 0.03 Grade 5 0.021 0.02 0.020 0.33 Grade 6 0.054 0.06 0.071 0.09 Grade 7 0.029 0.03 0.040 0.04 Grade 8 0.051 0.04 0.038 0.04 Grade 9 0.046 0.03 0.039 0.05 Grade 10 0.040 0.03 0.033 0.04 Grade 11 0.012 0.02 0.012 0.02

Number of schools 23 416

Schools near a concession Schools far a way from concession

34

Dropout Results: Probit estimatesTable 3. Direct and indirect impact of concessions: effects over dropout rates

Probit estimatesDepended variable

Coefficient Std. Err. Coefficient Std. Err.

Total number of students -0.008 0.001 * -0.015 0.003 *Dispersion in the test -0.366 0.080 * -0.708 0.178 *Localization 4 13.472 8.162 *Localization 5 0.024 0.168 9.129 8.556 Localization 7 0.212 0.138 13.929 8.089 *Localization 8 -0.052 0.163 12.411 8.234 Localization 9 0.048 0.155 12.455 8.311 Schedule Evening -0.243 0.173 1.966 0.430 *Ration teacher / stud. -0.284 0.046 * 0.514 0.094 *Average year of educ. teac. -0.035 0.025 0.080 0.079 % of students repeating -1.723 0.351 * 1.407 0.502 *% of students old for grade 3.327 0.719 * 5.040 1.709 *Number of computers, 1999 0.027 0.020 Number of class rooms, 1999 0.053 0.038 Number of restrooms, 1999 -4.686 0.719 *Number of sport facilities, 1999 -1.723 0.260 *Number of teachers, 1999 -0.476 0.278 *Number of administrative p. 1999 0.202 0.050 *Number of psychologist, 1999 -0.069 0.187 Ration teacher / stud., 1999 0.059 0.038 % teachers with second., 1999 35.714 10.066 *% teachers with college, 1999 33.968 9.841 *% teachers with grad. st., 1999 25.912 8.898 *

Log likelihood -529 -231PseudoR2 0.1391 0.5714Number of obs. 2334 1217

Direct impact over dropout rate

Concession

Indirect impact over dropout rate

Schools near concession

35

Dropout results: impact

Matching

Varible: Dropout rate

Difference Treatment-ControlUnmatched -0,0130 -0,0069ATT -0,0173 -0,0082

Bootstrap statisticsRepetitions 1000 1000Standard error 0,0039 0,0096317Bias-corrected conf. interv. -0,0244 -0,0112 -0,0341 0,0004

Impact variable: Dropout rate Impact variable: Dropout rate

Direct impact over dropout rate Indirect impact over dropout rate

Matching:

• 10 nearest estimators, common support, balance groups

• Direct Effect: reduction in 1.7 points dropout rates

• Indirect Effect: reduction in 0.82 points

36

Test Results: Probit estimatesTable 4. Direct impact of concessions: effects over test scores (mathematics and reading)

Probit estimatesDepended variable

Coefficient Std. Err. Coefficient Std. Err.

Total number of students -0.018 0.001 * -0.018 0.001 *Dispersion in the test -0.351 0.033 * -0.197 0.029 *Localization 5 -0.441 0.079 * -0.542 0.080 *Localization 7 0.149 0.063 * 0.140 0.064 *Localization 8 0.073 0.075 0.079 0.076 Localization 9 -0.150 0.071 * -0.181 0.072 *Schedule Evening 0.102 0.072 0.068 0.073 Ration teacher / stud. -0.071 0.005 * -0.072 0.005 *Average year of educ. teac. -0.089 0.012 * -0.095 0.011 *% of students repeating -2.508 0.183 * -2.575 0.179 *% of students old for grade 19.165 0.779 * 18.789 0.774 *Constant 4.683 0.300 * 4.474 0.330 *

Log likelihood 2152.87 -2846.556Pseudo R2 0.2768 0.268Number of obs. 18629 18630

Direct impact over math tests Direct impact over reading tests

Concession Concession

37

Test results: impact

Matching

Variable: Test scores

Difference Treatment-ControlUnmatched -0.2099 0.6234ATT 0.9732 1.9364

Bootstrap statisticsRepetitions 100 100Standard error 0.522 0.748Bias-corrected conf. interv. 1.2684 1.4655 1.5501 2.8472

Impact variable: Mathematics Impact variable: Reading

Matching:

• 10 nearest estimators, common support, balance groups

• Effect over math test scores: improvement of 2.4%

• Effect over language test scores: improvement of 4%

38

Some conclusions (and some questions)

• Concessions delivers:– Strong evidence of differences in dropout rates– Some evidence of impact over test scores

• Colombian government is implementing the II stage of Concessions

• What makes concessions different from public?– Is it the work with families and community?– Freedom of hiring teachers? – Pedagogic method?

39

A discussion about the data

• Limits of the data:– Not a randomized experiment– Still some source of bias

• Ideally, we want to have more hard-evidence– Randomization of students who applied to

concessions

Why is this important?

• Impact evaluation can provide reliable estimates of the causal effects of programs

• Impact evaluation can potentially help improve the efficacy of programs by influencing design and implementation

• Impact evaluation can broaden political support for programs

• Impact evaluation can help in the sustainability of successful programs and the termination of programs that are a failure

• Impact evaluation can help expand our understanding of how social programs produce the effects that they do