The role of civil society to strengthen accountability and to build evaluation systemsThe experience of Fundación IDEA

November 2013

2

Content

• About Fundación IDEA• The Indigenous Education Gap Index• PEC FIDE (School based management program): evaluation method and results

• Civil society’s role in M&E• M&E and policy-making

Fundación IDEA is a non-partisan think tank that provides rigorous analysis and sound solutions.

3

Our vision: informed policy decisions Rigorous analysis and sound solutions 8 years of experience Projects in Mexico, Guatemala, Colombia, USA, Italy, Peru Impartial and non-partisan Work on a variety of topics: education, economic development…

4

“Evaluación Transversal: políticas y programas para el desarrollo de la juventud” “Estrategia nacional para la vivienda sustentable” “¿Qué hacer con la diabetes?” “¿Quiénes son los emprendedores innovadores mexicanos?” “Propuesta de programa de inclusión laboral para jóvenes”

“Estudio sobre calidad docente en zonas rurales e indígenas en México” “Índice compuesto de eficacia de los sistemas escolares” “Índice de equidad educativa indígena” “Evaluación de impacto PEC-FIDE (Fortalecimiento en inversión directa a escuelas)”

Fundación IDEA works on a variety of topics.

Fundación IDEA has developed expertise on education.

6

There is a gap of 34%.

Implications of the study: Showing the gap to push

for targeted policy.

Fundación IDEA found that indigenous people in Mexico have achieved 66% of the educational achievement compared to non-indigenous.

Impact evaluation of the federal School-Based Management ProgramPEC FIDE (PROGRAMA ESCUELAS DE CALIDAD-FORTALECIMIENTO EN INVERSIÓN DIRECTA A ESCUELAS)

Pilot variation of program School zone approach Focus on very low-income neighborhoods Focus on at risk students Focus on training (shift from insfrastructure) More money per school than average PEC

Dropout rates

Graduation rate

Learning outcomes

Relevant SBM indicators

Short-run Long-run

8

Fundación IDEA was the external evaluator.

PEC FIDE

Impartial, rigorous approach, publish results; Usefulness of recommendations; Implications of evaluation: design programs based on evidence, keep programs that

work, improve programs that do not work.

Evaluation context:• 50% of schools were beneficiaries (selection bias). • Info available: annual school census; ENLACE test; population census.

Methodology (mixed methods):• quantitative: diff-in-diff w/ propensity score matching.• qualitative: interviews, participant observation and document analysis in 10

schools.

Tasa deaprobación

Tasa deaprobación*

Efectos después de tres ciclos es escolares Efectos de un ciclo escolar

Resultados de Skoufias y Shapiro (2006a) Resultados de Murnane, Willett y Cárdenas (2006) 1.

6 pp

Tasa deaprobación

Efectos de un ciclo escolar

Resultados de Abreu y Santibañez (2010)

0.24 pp 0.11 pp

PEC FIDE PECvs

The evaluation showed positive impact for PEC FIDE (2008-09) in graduation rates.

Other results: Approx 20 more points in ENLACE test (reading comprehension); approx 1/3 of

standard deviation. Approx -4 percentage points of students at insufficient level in ENLACE test.

Avg ENLACE scores

PRIM - Tendencia generalPRIM - PEC-FIDEPRIM - PEC-FIDE sin PEC

Inicio de ciclo 2008-2009

Fin de ciclo 2008-2009

19.5 puntos de ENLACE/año

12.8 puntos de ENLACE/año

The evaluation showed impact on reading comprehension at the elementary school level (ENLACE test).

Fundación IDEA presented results to administrators of the program.

Qualitative analysis showed increased motivation.

Investment focused on infraestructure, hard shift to training.

Limited participation: how to promote more and better participation?

The importance of targeting neighborhoods? Importance on school zone program design? Targeting students at risk?

12

Content

• About Fundación IDEA• The Indigenous Education Gap Index• PEC FIDE (School based management program): evaluation method and results

• Civil society’s role in M&E• M&E and policy-making

13

Rigorous analysis

Credibility of conclusions

Usefulness of recommendations

Public policy based on evidence

Programs may be implemented because of:

- Evidence- Experience- Ideology- Politics- Budget- Technical capacity

Civil society needs to promote an evidence based policy culture.

Evaluation should be used as a tool for decision making and efficient spending. We are experiencing a paradigm shift in M&E.

Evaluation is a tool for accountability and transparency - programs use public resources and so society needs to know if objectives are accomplished and if programs are effective

Evaluation as a learning process- to improve design and operation of programs- to generate knowledge about complex problems and about interventions that work - to know if a program works or not- “bang for the buck”

15

M&E is used for decison making

t

No M&E

Some M&E activities

Good consumer and capacity to perform M&E activities

M&E, if appropriately used, can improve how programs are managed and the outcomes they can have.

The use of M&E by organizations

16

Impact eval

Implementation eval

Sound indicators

Needs assessment

There are some misconceptions on M&E practice.

Types of evaluation: Needs assessment Program theory (theory of change) Design evaluation Implementation or process evaluation Results (change) Impact (causality) Efficiency (CBA, cost effectiveness)

Impact evaluation can change policy if implemented based on a good understanding of M&E practice.

Key lessons for capacity building: Valid Counterfactual

- understand when to evaluate impact and when not- many organizations decide to evaluate before-after or compare beneficiaries with non-beneficiaries: not always good use of resources

Theory of Change- important for understaning what the organization is doing and how they are measuring it- sometimes there is no clarity/consensus of the purpose of a programamong stakeholders

Objectives Inputs Products Intermediate results

Impact

18

Thank you