Implementation Completion and Results Report (ICR) Document...United Mexican States Secretaria de...

Document of

The World Bank FOR OFFICIAL USE ONLY

Report No: ICR00004512

IMPLEMENTATION COMPLETION AND RESULTS REPORT

(LOAN 8446-MX)

ON A

LOAN

IN THE AMOUNT OF US$350 MILLION

TO THE

UNITED MEXICAN STATES

FOR A

SCHOOL BASED MANAGEMENT PROJECT

June 24, 2019

Education Global Practice Latin America And Caribbean Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

CURRENCY EQUIVALENTS

(Exchange Rate Effective January 29, 2019)

Currency Unit = Mexican Peso (MXN)

MXN 18.99 = US$1.00

FISCAL YEAR

July 1 - June 30

Regional Vice President: Axel van Trotsenburg

Country Director: Pablo Saavedra

Senior Global Practice Director: Jaime Saavedra Chanduvi

Practice Manager: Emanuela Di Gropello

Task Team Leader(s): Monica Yanez Pagans

ICR Main Contributor: Abril Alicia Ibarra Castaneda

ABBREVIATIONS AND ACRONYMS

AEL Local Education Authority (Autoridad Educativa Local) AGE School-based Management Support (Apoyo a la Gestión Escolar) CEAI Colegio de Estudios Avanzados de Iberoamérica CONAPASE Consejo Nacional de Participación Social en la Educación CONOCER Consejo Nacional de Normalización y Certificación de Competencias Laborales CPS Country Partnership Strategy DGAG Directorate of School Management (Dirección General Adjunta de Gestión) DGDGE General Directorate of School Management and Education (Dirección General de

Desarrollo de la Gestión Educativa) DGE General Directorate of Evaluation (Dirección General de la Evaluación) DGEI General Directorate of Indigeous Education (Dirección General de Educación

Indígena) ENLACE National Evaluation of Academic Achievement in Schools (Evaluación Nacional del

Logro Académico de Centros Escolares) FM Financial Management FY Fiscal Year GDP Gross Domestic Product GNI Gross National Income GoM Government of Mexico IBRD International Bank for Reconstruction and Development ICR Implementation Completion and Results Report IE Impact Evaluation ISR Implementation Status and Results Report IERR Internal Economic Rate of Return INEE National Institute for the Evaluation of Education (Instituto Nacional para la

Evaluación de la Educación) INEA National Institure for Adults Education (Instituto Nacional para la Educación de los

Adultos) IPP Indigenous People Plan ITEA Tlaxcalteca Institute for Adults Education (Instituto Tlaxcalteca para la Educación

de los Adultos) M&E Monitoring and Evaluation NAFIN Nacional Financiera, S.N.C., I.B.D. NDP National Development Plan (Plan Nacional de Desarrollo) OECD Organization for Economic Co-operation and Development OIC Internal Control Body (Órgano Interno de Control) PAD Project Appraisal Document PDO Project Development Objective PEC Quality Schools Program (Programa Escuelas de Calidad) PEC-FIDE Programa Escuelas de Calidad- Fortalecimiento e Inversión Directa a las Escuelas PED Dignified School Environment Program (Programa Escuelas Dignas) PEEARE Excelence School Program to Tackle Educational Gaps (Programa Escuelas de

Excelencia para Abatir el Rezago Educativo)

PETC Full Time Schools Program (Programa Escuelas de Tiempo Completo) PISA Program for International Student Assessments PLANEA National Plan for Learning Assessment (Plan Nacional para la Evaluación de los

Aprendizajes) PRE Education Reform Program (Programa de la Reforma Educativa) PSE Education Sector Program (Programa Sectorial de Educación) RCTs Randomized Control Trials SBM School Based Management s.d. Standard Deviation SEB Subsecretariat of Basic Education (Subsecretaría de Educación Básica) SEMS Subsecretariat of Upper Secondary Education (Subsecretaría de Educación Media

Superior) SEP Secretariat of Public Education (Secretaría de Educación Pública) SHCP Ministry of Finance and Public Credit (Secretaría de Hacienda y Crédito Público) SIGED Education Management Information System (Sistema de Información y Gestión

Educativa) SISAT Early Warning System (Sistema de Alerta Temprana) SPD Professional Teacher Service (Servicio Profesional Docente) WMS World Management Survey

TABLE OF CONTENTS

DATA SHEET .......................................................................................................................... 1

I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5 A. CONTEXT AT APPRAISAL .........................................................................................................5

B. SIGNIFICANT CHANGES DURING IMPLEMENTATION ................................................................9

II. OUTCOME .................................................................................................................... 10 A. RELEVANCE OF PDOs ............................................................................................................ 10

B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 11

C. EFFICIENCY ........................................................................................................................... 19

D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 19

E. OTHER OUTCOMES AND IMPACTS ......................................................................................... 19

III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 20 A. KEY FACTORS DURING PREPARATION ................................................................................... 20

B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 21

IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 23 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 23

B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 25

C. BANK PERFORMANCE ........................................................................................................... 26

D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 27

V. LESSONS AND RECOMMENDATIONS ............................................................................. 28

ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 29

ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 47

ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 49

ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 50

ANNEX 5. BORROWER COMMENTS ...................................................................................... 55

ANNEX 6. SUPPORTING DOCUMENTS .................................................................................. 57

ANNEX 7. THEORY OF CHANGE ............................................................................................ 59

ANNEX 8. CHANGES TO KEY INDICATORS ............................................................................. 61

The World Bank Mexico School Based Management Project (P147185)

Page 1 of 62

DATA SHEET

BASIC INFORMATION Product Information Project ID Project Name

P147185 Mexico School Based Management Project

Country Financing Instrument

Mexico Investment Project Financing

Original EA Category Revised EA Category

Not Required (C) Not Required (C)

Organizations

Borrower Implementing Agency

United Mexican States Secretaria de Educacion Publica

Project Development Objective (PDO)

Original PDO The objective of the Project is to improve schools' managerial capacity and parental participation to reduce dropout, repetition and failure rates among PEC Schools and PETC Schools. Revised PDO The objective of the Project is to improve schools' managerial capacity and parental participation to reduce dropout, repetitionand failure rates among Participating Schools.


Page 2 of 62

FINANCING

Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing IBRD-84460 350,000,000 350,000,000 322,064,712

Total 350,000,000 350,000,000 322,064,712

Non-World Bank Financing 0 0 0

Borrower/Recipient 469,950,000 469,950,000 469,950,000

Total 469,950,000 469,950,000 469,950,000

Total Project Cost 819,950,000 819,950,000 792,014,712

KEY DATES

Approval Effectiveness MTR Review Original Closing Actual Closing 24-Oct-2014 09-Jul-2015 10-Oct-2017 31-Dec-2018 31-Dec-2018

RESTRUCTURING AND/OR ADDITIONAL FINANCING

Date(s) Amount Disbursed (US$M) Key Revisions 26-Jul-2017 122.56 Change in Project Development Objectives

Change in Results Framework Change in Components and Cost Reallocation between Disbursement Categories Change in Legal Covenants Change in Financial Management Other Change(s)

15-Aug-2018 270.48 Reallocation between Disbursement Categories 29-Nov-2018 303.77 Reallocation between Disbursement Categories

KEY RATINGS

Outcome Bank Performance M&E Quality

Satisfactory Satisfactory Substantial


Page 3 of 62

RATINGS OF PROJECT PERFORMANCE IN ISRs

No. Date ISR Archived DO Rating IP Rating Actual

Disbursements (US$M)

01 12-Dec-2014 Satisfactory Satisfactory 0

02 22-Jun-2015 Satisfactory Moderately Satisfactory 0

03 15-Dec-2015 Satisfactory Moderately Satisfactory 70.00

04 21-Jun-2016 Satisfactory Satisfactory 122.56

05 23-Dec-2016 Satisfactory Satisfactory 122.56

06 30-Jun-2017 Moderately Satisfactory Moderately Satisfactory 122.56


08 26-Jun-2018 Moderately Satisfactory Satisfactory 270.48


SECTORS AND THEMES

Sectors Major Sector/Sector (%)

Education 100

Early Childhood Education 33 Public Administration - Education 1 Primary Education 33 Secondary Education 33

Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Social Development and Protection 0

Social Inclusion 49 Participation and Civic Engagement 49


Page 4 of 62

Human Development and Gender 0 Education 52

Access to Education 26

Education Financing 26

ADM STAFF

Role At Approval At ICR

Regional Vice President: Jorge Familiar Calderon Axel van Trotsenburg

Country Director: Gerardo M. Corrochano Pablo Saavedra

Senior Global Practice Director: Claudia Maria Costin Jaime Saavedra Chanduvi

Practice Manager: Reema Nayar Emanuela Di Gropello

Task Team Leader(s): Rafael E. De Hoyos Navarro Monica Yanez Pagans

ICR Contributing Author: Abril Alicia Ibarra Castaneda


Page 5 of 62

I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES

A. CONTEXT AT APPRAISAL

Context 1. At the time of Project Appraisal, Mexico's long-term economic growth was somewhat below expectations, limiting a potential rise in average living standards and more rapid progress on poverty reduction. In 2013, its Gross National Income (GNI) per capita was approximately 39 percent of the level observed in high income countries in the Organization for Economic Co-operation and Development (OECD),1 the same proportion observed two decades earlier, signaling a lack of progress in economic convergence. Over the past three decades, annual Gross Domestic Product (GDP) growth averaged 2.4 percent and only 0.8 percent per capita. Growth decomposition exercises pointed to insufficient average productivity growth as the main cause of growth performance. To raise productivity and unleash growth, the Government launched in 2013 an ambitious structural reform agenda and enacted legislative changes in the areas of labor market regulation, education, telecommunication and competition policy, financial sector regulation, energy, and fiscal policy. There was a clear potential for the reforms to enhance potential output growth, which at Project Appraisal was estimated in the range of 2.5-3.0 percent. 2. Low learning outcomes comprised the biggest challenge faced by Mexico’s education system. Student learning, as measured by the Program for International Student Assessment 2012 (PISA) placed Mexico in the last position among OECD countries and in 53rd of the 65 participating countries.2 In terms of education coverage, although Mexico was making significant progress in basic education, there were still students left out of school. At appraisal, net coverage was 87 percent for preschool (5 years-old), 96 percent for primary (6 to 12 years-old), and 87 percent for secondary (12 to 14 years-old).

3. The Education Reform was one of the 2013-18 administration’s priorities and aimed to accelerate growth and development by improving the quality of education while reducing access and achievement gaps between rich and poor. The 2013 Education Reform mainly consisted of: i) establishing a professional system for hiring, evaluating, training, and promoting teachers; ii) providing full autonomy to the National Institute for the Evaluation of Education (Instituto Nacional para la Evaluación de la Educación, INEE) to oversee all evaluation functions throughout the education system; (iii) establishing a federal census of education data and a national Education Management Information System; and (iv) fostering school autonomy and school-based management (SBM).

4. The Project supported the school autonomy component of the 2013 Education Reform. More autonomy at the school level sought to improve school infrastructure, provide relevant materials, and solve basic school operational problems, while promoting social participation so that students, teachers, and parents, under the school director’s leadership, could overcome their challenges. Changes to the General Education Law, derived from the Reform, stated that SBM programs should use evaluation results as feedback for continuous improvement and to develop annual school improvement plans (shared with the education authority and the school community), specifying their priorities and linking them to 1 Mexico is an upper middle-income country and a member of the OECD and the G20. 2 Mexico had slightly improved its Mathematics results in PISA over time (from 387 in 2000 to 413 in 2012), but reading results stagnated (from 422 in 2000 to 424 in 2012).


Page 6 of 62

verifiable targets. In addition, three of the seven strategies of the Programa Sectorial de Educación (PSE) 2013-18 focused on schools as the units of change and improvement. These included: (i) creating the conditions for schools to be at the center of the education system by allowing them to receive the support needed to achieve their objectives; (ii) increasing school management capacity to improve learning outcomes; and (iii) strengthening the relationship between the school and the broader community to promote social participation, transparency, and accountability. The school autonomy policy was mainly implemented through four programs: (i) the Schools of Excellence to End Educational Delay Program (Programa Escuelas de Excelencia para Abatir el Rezago Educativo, PEEARE); (ii) the Dignified Schools Environment Program (Programa Escuelas Dignas, PED); (iii) the Quality Schools Program (Programa Escuelas de Calidad, PEC); and (iv) the Full Time Schools Program (Programa Escuelas de Tiempo Completo, PETC). Under this framework, the World Bank loan was designed to support the school autonomy policy through PEC and PETC programs. 5. The financial and technical support to schools partially provided through the Bank-financed Project was the single most important means for schools to exert their autonomy, granted by the 2013 Education Reform. By supporting the school autonomy policy, the Project aimed to improve schools’ managerial capacity and parental participation to reduce dropout and improve the quality of education services in public basic education schools in Mexico. In the long run, the Project aimed to contribute to the Bank’s twin goals of eliminating extreme poverty and boosting shared prosperity by improving students’ future income through their staying in school longer and learning more. The Project was also an important component of services put forward in the World Bank Group’s Country Partnership Strategy (CPS) for FY14-FY19 (Report No. 80800-MX), discussed by the Executive Directors on December 12, 2013, which is fully aligned with the goals of Mexico's National Development Plan (NDP) for 2013-18.


Page 7 of 62

Theory of Change (Results Chain)

Project Development Objectives (PDOs)

6. The objective of the Project was to improve schools’ managerial capacity and parental participation to reduce dropout, repetition, and failure rates among Participating Schools.3

Key Expected Outcomes and Outcome Indicators 7. The PDO was to be measured through the following indicators:

a. Proportion of basic education schools in programs (PEC and PETC) to strengthen SBM with a director that has a “sufficient” level of managerial capacity.

b. Proportion of basic education schools in PEC with parent associations participating in the design, monitoring, and adjustment of the school improvement plan.

c. Dropout rate among basic education schools in programs to strengthen SBM (PEC and PETC). d. Gross failure rate among basic education schools in programs to strengthen SBM (PEC and PETC). e. Repetition rate among secondary school students in schools in programs to strengthen SBM (PEC

and PETC).

3 Participating Schools were clearly defined in the Project Restructuring, the Loan Agreement, and all other relevant Project documents as schools participating in the Programa de la Reforma Educativa (PRE) and in Programa de Escuelas de Tiempo Completo (PETC).


Page 8 of 62

Components 8. The Project included three components: 9. Component 1: Increasing School Autonomy and Parent Participation (Original allocation: US$342 million IBRD funding; Actual disbursed: US$321 million IBRD funding4) Provision of direct financial support to PEC Schools to implement School Improvement Plans (Ruta de Mejora) agreed with parent associations. The school improvement plan is a strategic plan whereby the school’s technical council (comprised by the school director and teachers) identifies the school’s main challenges and goals and the means to achieve them. The direct support amounts are defined by a resource allocation formula using two criteria: exogenous conditions (i.e. enrollment, level of marginalization) and performance in quality indicators (i.e. student learning, graduation rate). Priority is given to schools serving marginalized and vulnerable populations and those with low achievement levels in the national standardized test and high dropout rates. The resources are used to support activities related to the school improvement plans, as opposed to recurrent costs of the school’s everyday functioning. 10. Component 2: Improving Schools’ Managerial Capacity (Original allocation: US$3.5 million IBRD funding; Actual disbursed: US$0 IBRD funding5). Provision of technical assistance to PEC Schools and PETC Schools to strengthen school-based management through:

a. Support for the development of a school dashboard for school supervisors and directors including

performance indicators and best practices, among others. b. Support for: (a) the development and implementation of a capacity-building strategy for school

directors and supervisors on: (i) the use of the school dashboard and its role in improving school management practices; (ii) the use of a classroom observation tool; (b) the improvement of schools’ managerial practices through the development and implementation of capacity-building activities for parents; and (c) the development of the information system needed to keep the school dashboards running, relevant, and up to date, including the preparation of guidelines describing an implementation and maintenance protocol for dashboards.

11. Component 3: Research and Innovation (Original allocation: US$3.6 million IBRD funding; Actual disbursed: US$0 IBRD funding).

a. Provision of support for the development of a new instrument to measure Participating Schools’ managerial capacity, through the piloting, redesign (if necessary), implementation, and comparison of alternative questionnaires to measure managerial capacity.

b. Provision of support to States to improve adherence to PEC’s Operational Rules through the carrying out of workshops and knowledge exchange activities.

c. Provision of support for carrying out an assessment in at least two self-selected States which will estimate and quantify the effect of the intervention supported by the Project.

4 The disbursed portion of the loan allocated to Component 1 represents 39.9% of the original projection of the Component cost (US$804 million). The rest of the costs were financed by the GoM. 5 The GoM financed all costs for Component 2 (US$9.7 million) and Component 3 (US$5.4 million).


Page 9 of 62

B. SIGNIFICANT CHANGES DURING IMPLEMENTATION

Revised PDOs and Outcome Targets 12. The PDO was formally revised in the July 26, 2017 restructuring to replace the terminology “PEC and PETC schools” with “Participating Schools.” This change had no substantive impact on the Project; therefore, there were no major implications on the original theory of change or the originally expected outcomes and targets. The revised PDO reads: “the objective of the Project is to improve schools’ managerial capacity and parental participation to reduce dropout, repetition and failure rates among Participating Schools.” The change was needed since the PEC program, one of the programs supported by the Project and stated in the PDO, was consolidated into a meta-program called the Education Reform Program (Programa de la Reforma Educativa, PRE). For consistency, reference to PEC and PETC was replaced throughout the Project by "Participating Schools." Participating Schools were clearly defined in the Loan Agreement and other relevant Project documents as schools that are part of PRE and/or PETC.

Revised PDO Indicators 13. None of the PDO indicators suffered reductions in targets. While some indicators were replaced, their replacements were simply more accurate measures of the same outcomes. The PDO indicators were adjusted to reflect data availability and the change in the Project beneficiaries from "PEC and PETC Schools" to "Participating Schools” (mainly baseline conditions and progress among PRE beneficiaries). Of the 5 original PDO indicators, three were revised and two were replaced in the July 2017 restructuring. For a full explanation of the changes by indicator, please see Annex 8. The following are the revised key indicators:

i. Proportion of a representative sample of Participating Schools with a director that scores 3 or above in the World Management Survey (WMS). Added to replace original PDO Indicator 1.

ii. Proportion of a sample of Participating Schools that increase parental participation (score more than 1 in the School-based Management Support (Apoyo a la Gestión Escolar, AGE)). Added to replace original PDO Indicator 2.

iii. Dropout rate among (primary and secondary) Participating Schools. iv. Primary and Secondary school gross failure rate among Participating Schools. v. Secondary school repetition rate among Participating schools.

Revised Components 14. The three Project components were slightly revised in the July 2017 restructuring as follows:

i. Component 1. Increasing School Autonomy and Parent Participation was adjusted to support the provision of Direct Support to foster school autonomy and implement school improvement plans in “Participating Schools” (PRE and PETC) instead of PEC schools only. At the time of restructuring, the Government of Mexico (GoM) requested that direct support to PETC schools also be included under Component 1. Although this implied more complex financial management of Project funds (funding two government programs instead of only one), it also helped PETC schools benefit from Component 1 and helped disburse Project funds more promptly.

ii. Component 2. Improving Schools' Managerial Capacity was adjusted to support the provision of technical assistance to “Participating Schools” (instead of PEC and PETC schools) to strengthen SBM. The reference to the school dashboards was also removed because the Federal Government chose to have each state select the interventions to improve managerial skills of school directors and communities based on their specific context.


Page 10 of 62

iii. Component 3. Research and Innovation was adjusted to provide support to States to improve adherence to Operational Rules of "Participating Schools" (PRE and PETC) instead of PEC's Operational Rules.

Other Changes 15. The Project's Financial Management (FM), including flow of funds, was modified to reflect PRE arrangements, which differ from those under PEC. In addition, FM included the flow of funds to support SBM in PETC schools. The restructuring also included a reallocation between disbursement categories to provide more financing for Direct Support to Participating Schools (Component 1). Finally, the Loan Agreement was amended to reflect the above changes and make eligible expenses for the PETC retroactive as of January 2016.

Rationale for Changes and Their Implication on the Original Theory of Change

16. The July 2017 restructuring was needed to respond to the consolidation of national programs to strengthen SBM in Mexico. However, the description of the PDO, components and key activities mostly remained unchanged. The overall objective of the Project remained the same, since the Project continued supporting school autonomy and managerial practices of schools, but now under: (i) PRE's School Autonomy component by providing direct support to schools to implement school improvement plans; and (ii) PRE’s Managerial Practices component by developing and implementing capacity-building activities for school supervisors and directors. The PRE components supported by the Project follow the same approach as the former PEC program. Therefore, the theory of change remained unchanged.

II. OUTCOME A. RELEVANCE OF PDOs

17. The PDO is aligned to the current World Bank Group’s CPS for FY14-FY19. It aligns with the CPS Pillar 2 “Increasing Social Prosperity” as updated through the Performance and Learning Review (Report No. 104752-MX) which has the expected objective of “improved access and quality in target education programs”. The CPS and the PDO are also consistent with Mexico’s National Development Plan (NDP) 2013-18, which has five main components: Peace, Inclusion, Quality of Education, Prosperity, and Global Responsibility. All the NDP components emphasize improving the quality of education and reducing access and achievement gaps between rich and poor to increase productivity and long-term growth. In addition, the Project fully supported the school autonomy aspects of the 2013 Education Reform. Finally, the Bank’s engagement with and support to Mexico, an upper-middle income country, is highly relevant since around half of its population is still poor. In addition, the evidence of best practices in key education policies in Mexico potentially has a valuable demonstration effect for other lower-income countries.

Assessment of Relevance of PDOs and Rating Rating: High

18. The relevance of the PDO is rated High since the Project provided clear evidence of the alignment of the PDO to the expected outcomes of the current CPS and Mexico’s most current NDP (2013-18), as described above.


Page 11 of 62

B. ACHIEVEMENT OF PDOs (EFFICACY)

19. Even though the Project underwent a level 1 Restructuring in July 2017, this Implementation Completion and Results Report (ICR) does not conduct a split evaluation of the efficacy of the outcome indicators before and after the restructuring since the scope of activities and expected outcomes of the Project remained the same. The change in PDO merely reflects support to the same activities under different denominations in a recently consolidated government program. From the outset, the Project was designed to support the school autonomy policy as specific government programs evolve or shift. Given that PEC was consolidated into the PRE program, the Project was restructured to shift resources from PEC to PRE. In addition, the revised indicators reflected more detailed and accurate measures of the Project’s achievement without changing the Project’s scope or reducing its targets. Therefore, the achievement of each of the objectives is measured through the five PDO indicators revised at the July 2017 restructuring. 20. The Project’s expected outcomes were to improve managerial capacity and parental participation among Participating Schools to reduce dropout and improve quality of education, as measured by failure and repetition rates among Participating Schools. The ICR goes beyond the measurement of the PDO outcomes as stated in the Project Appraisal Document (PAD) by adding learning results as an expected outcome of this Project. Improved student learning (as measured by test scores) provides a broader view of the achievements of the Project in terms of its contribution to improving the quality of education. Although students’ learning results were not explicitly mentioned in the PDO, it was clear in both the PAD and the July 2017 restructuring paper, that the Project aimed to improve the quality of education, including learning results. For instance, the PAD mentions that “the two programs (initially supported by the Project) share the goal of contributing to students’ learning within a framework of more school autonomy, the use of evaluations to improve pedagogical strategies, better managerial capacities at the school level, and active social participation.” As explained in the objectives section (see paragraph 53), learning results were not included from the outset because: (i) the Bank was overly cautious of adding learning results as a key outcome indicator due, among others, to the short span of the Project to impact learning and to technical reasons (given the transition from the ENLACE national standardized test to the National Plan for Learning Assessment (Plan Nacional para la Evaluación de los Aprendizajes, PLANEA)) and the uncertainty on timing and availability of PLANEA results. Instead, failure and repetition rates were included as proxies for learning outcomes to measure the quality of education. In addition, it has become increasingly feasible and important to measure and include student learning outcomes as key outcome indicators to ensure children attain real learning in the classroom and are thus able to enter the job market as healthy, skilled, and productive adults (World Bank, Human Capital Project, 2018). Assessment of Achievement of Each Objective/Outcome Improve managerial capacity in Participating Schools. 21. This objective was measured through the key indicator “proportion of a representative sample of Participating Schools with a director that scores 3 or above in the World Management Survey (WMS).”6

6 The WMS is a well-known instrument developed by Bloom and others at the London School of Economics that has been widely used by the World Bank in several countries to measure managerial capacity in different areas, including education. The WMS has a scale from 1 to 5 to measure managerial capacity (1 being the lowest and 5 being the highest score). In the case of Mexico and other countries with low managerial capacity, there are two more intermediate scores, 1.5 and 2.5, to allow for a greater dispersion of the results. For more information, please visit: www.developingmanagement.org

http://www.developingmanagement.org/


Page 12 of 62

The share of Participating Schools with directors scoring 3 or more in the WMS increased from 21 percent in school year 2015-16 to 27 percent in 2017-18, exceeding the 25 percent target set at restructuring. School directors scoring 3 or more can use existing resources in an effective way to provide better education services. School directors’ managerial capacity was measured in a sample of 1,665 (PEC/PRE) schools. 22. Additional evidence from the results from 7 randomized control trials (RCTs) conducted in 7 of the 32 states in Mexico show that after the three years of implementation of the interventions supported by this Project, the school management index improved by 0.086 standard deviations (s.d.) per year7 in Participating Schools where Project activities were closely monitored and implemented. 23. The Project (under Component 1) contributed to improving the managerial capacity of Participating Schools by providing direct support (based on national criteria) to implement the school improvement plans agreed with parents. The provision of direct financial support to schools: (i) worked as an incentive for school directors and local actors (including teachers and parents) to establish a common dialogue on how to address school needs to improve student learning (through the school improvement plans); (ii) allowed schools to exert their autonomy; and (iii) helped increase local transparency and accountability, since the school directors needed to report on the use of funds received to both the school community and education authorities. 24. The development and implementation of the capacity-building strategy for school directors and supervisors (Component 2), helped improve school directors’ managerial capacity (De Hoyos, Silveyra and Yanez 2017). All the supervisors and over 96 percent of school directors in Participating Schools received capacity-building activities in SBM. The capacity-building strategy mainly consisted of two sets of interventions. The first set designed and provided cascade training in school management to school supervisors who, in turn, trained school directors. This training strategy was conducted by the Dirección General de Desarrollo de la Gestión Educativa (DGDGE) in all 32 States. Examples of the cascade training in management and tools to improve learning provided include the following:

a. Comprehensive school supervision course focused on how to help schools improve their student

learning outcomes. The course, titled “Una Supervisión Efectiva para la mejora del aprendizaje de nuestros alumnos,” benefited 18,447 school supervisors. This included all school supervisors in basic education in the country and those that were on a clear track to become school supervisors.

b. Training on the use of classroom observation tools to improve pedagogical practices and collection of performance indicators to assess basic reading, writing, and math skills. Overall, 7,694 supervisors were trained on a classroom observation tool and around 8,817 supervisors undertook the basic reading, writing, and math exploration tools training. Cascade training from supervisors to school directors benefited over 70,060 primary and secondary school directors (of 72,611).

c. Eight workshops for trainer of trainers to a core group of 1,000 school supervisors, including key topics on best practices in SBM.

d. Each school was also offered a paid time slot (one school day every month) to prepare, implement, and monitor its school improvement plan. This gave school directors and teachers a space to analyze school-level data and use it to improve students learning.

7 The school management index represents schools’ managerial capacity as measured by the 5 WMS aspects: leadership, operations, monitoring, setting of objectives, and personnel.


Page 13 of 62

25. The second set of interventions related to capacity-building under Component 2 consisted of training for school directors tailored to their specific management needs in the 7 States participating in the impact evaluation. This training aimed to help school directors improve their managerial capacity through a shared leadership approach, collaborative planning, and teamwork to improve learning. Examples of the training provided by the States include the following:

a. Tlaxcala offered the Diplomado en Liderazgo para la Autonomía de Gestión Escolar to improve school leadership and certify school directors on the competency of Liderazgo para la Autonomía de Gestión en Centros de Trabajo Educativos, validated by the Consejo Nacional de Normalización y Certificación de Competencias Laborales (CONOCER). Training was conducted by the Colegio de Estudios Avanzados de Iberoamérica (CEAI) and included hands-on and theoretical approaches under a blended modality (4 face-to-face sessions/24 hours, 13 online sessions/136 hours, and 90 practice hours). The training focused on sharing experiences on how to solve specific school needs among a diverse group of facilitators including teachers, school leaders, and education experts.

b. Puebla offered a Diploma program on school leadership imparted by WorldFund. Capacity-building activities under the Diploma program included 308 face-to-face training hours, 16 online hours, and personal coaching to school directors. Contents included the importance of building a school culture, understanding the role of teachers, parents, and students, and coaching to improve learning outcomes (including social and emotional skills).

26. The Secretariat of Public Education (Secretaría de Educación Pública, SEP) also developed a school dashboard information system called the Sistema de Alerta Temprana (SisAT). School directors and supervisors received training on the use of this dashboard. SisAT helped school supervisors and directors systematically collect relevant and up to date information on basic learning skills and performance indicators to monitor school improvement plans (i.e. students’ basic reading, writing and math skills). This was combined with the collection of frequent data on student attendance and student grades. Teaching practices (through the classroom observation tool), and the provision of feedback to teachers to improve their pedagogical practices was also a common practice at the school level, although this information was not always uploaded to the SisAT. The SisAT helped school directors access data to inform the monitoring of school improvement plans; these in turn helped them strengthen their managerial capacity. The SisAT is now working in 27 of 32 states in Mexico. 27. Finally, the research and innovation activities under Component 3 helped measure, monitor and thus improve schools’ managerial capacity (and assess the impact of the Project). Before the Project, there was no formal or robust measurement of school managerial capacity. The Project provided support to select and adjust an instrument to measure school managerial capacities in the Mexican context following international best practices. The adapted WMS provided, for the first time, a reliable measure of school managerial capacities in the country and their main dimensions (operations management, monitoring, target setting, people management, school leadership, and pedagogical practices). This measure helped fuel a discussion for schools to understand their specific challenges and some of the practices that effectively-managed schools follow.8

8 As previous research shows (i.e. De Hoyos, García, Patrinos, 2017), the sole provision of information to schools may have a positive effect on learning.


Page 14 of 62

Improve parental participation 28. This objective was measured through the key indicator “proportion of a sample of Participating Schools that increase parental participation.” Parental participation increased from 83.9 percent in the 2015-16 school year to 94.66 in 2017-18, exceeding the 90 percent target set at restructuring. Parental participation was measured with the School Management Autonomy questionnaire (Cuestionario de Autonomía de Gestión Escolar, AGE). The AGE is a national survey created to measure parental engagement and other aspects of SBM in 7 states. The baseline for this indicator was collected in a representative sample of 1,091 Participating Schools using the AGE questionnaire. 29. The Project (Component 1) contributed to increased parental involvement by providing direct support to Participating Schools to implement school improvement plans agreed with parents. Parents in schools that received support participated in the development of the school improvement plan and were involved in the monitoring process to ensure transparent use of the funds. 30. The Project (Component 2) also helped design capacity-building activities on SBM for parents to increase their participation. Although not all parents in Participating Schools were trained, 78.21 percent received some kind of SBM training. For instance, the DGDGE, in coordination with the Consejo Nacional de Participación Social en la Educación (CONAPASE), designed and delivered a capacity-building strategy for parents to increase their impact on students’ learning and development. The program, named Familias Educadoras, offered information and guidance for parents to effectively support their children at school. Familias Educadoras focused on: helping increase self-esteem and healthy emotional family relationships; guidance to set clear rules at home; specific techniques to support children with reading, writing and math at home; and parents’ role in school transparency and accountability. The results of an electronic beneficiary satisfaction survey of Familias Educadoras show that, overall, parents find this strategy useful and engaging. In terms of coverage, Familias Educadoras had only reached 3,000 schools by Project closing, but is expected to reach 18,386 basic education schools by 2018-19. 31. In addition, the 7 states participating in the impact evaluation designed tailored capacity-building activities to increase parental participation on school matters. For instance: 32. In Puebla, parents in Participating Schools received a 6-hour workshop (with two monthly sessions consisting of three hours each) called Familia Presente, delivered by school directors and SEP personnel. The workshop’s objective was to improve parental knowledge and practices that are effective to help strengthen student learning. Parents improved their communication skills and built confident relationships with their children, were informed of the importance of learning together with their children and understood the importance of having high expectations for their children’s performance. In addition, parents received information on the importance of participating in school activities and asking for support from teachers and other school actors when needed.

33. In Tlaxcala, intensive training was provided to a select group of 98 parents that were responsible for training and bringing in other parents to their children’s schools. The workshop, delivered by CEAI, lasted 20 hours (4 sessions of 5 hours each) and aimed to increase joint teacher-parent responsibility to improve students’ learning outcomes, as well as their involvement in the monitoring of the school improvement plans. Tlaxcala also developed a network called Un Domingo por Nuestros Hijos e Hijas,


Page 15 of 62

where parents met regularly over the weekends to brainstorm on how to improve their children’s learning and school autonomy from a social participation perspective. In addition, in 2015, the Instituto Tlaxcalteca para la Educación de los Adultos (ITEA) and Instituto Nacional de Educación de Adultos (INEA) helped design communication campaigns to help parents in Participating Schools meet their own learning needs, such as acquiring basic literacy and numeracy skills. 34. Finally, as mentioned in the previous PDO indicator, school supervisors and directors were also trained to increase parental involvement as part of the capacity-building strategy. Reduce Dropout Rates 35. This objective was measured through the key indicator “dropout rate among (primary and secondary) Participating Schools.” In primary, the dropout rate fell from 3.18 percent in 2014-15 school year to 2.56 in 2016-17 school year, exceeding the 2.91 target for 2017-18. In secondary schools, the dropout rate decreased considerably from 5.31 percent in school year 2014-15 to 2.12 percent in school year 2017-18, greatly exceeding the 4.80 percent target for the 2017-18 school year. 36. School grant schemes, such as the one supported under the Project, have shown positive impacts on lowering dropout rates (World Bank, 2014; Gertler, Patrinos and Rubio-Codina, 2012; Skoufias and Shapiro, 2006; Murnane, Willet and Cardenas, 2006).9 This Project, in particular, contributed to reducing dropout rates by increasing school autonomy, parental participation, and schools’ managerial capacity. The Project helped school directors, teachers, students, and parents identify and address their local priorities. The school dashboard information system (SisAT) also allowed directors and even teachers to detect students at risk of dropping out and to focus on different strategies to keep them in the system. Schools may have also absorbed some direct costs of schooling, reducing the financial parental contributions and, thus, decreasing dropout rates among children whose parents were unable to cover such direct costs. Parents (with higher opportunity costs of sending their children to school) may have also decided to keep their children in school due to their increased involvement in the school’s issues. Parents also perceived that the relevance and quality of education increased with the activities supported under the Project.10 For instance, parents perceived that there was more autonomy to make decisions at the school level and that school actors were better prepared to identify local needs and improve student learning outcomes. Reduce Gross Failure Rates 37. This objective was measured through the key indicator “primary and secondary school gross failure rate among Participating Schools.” For primary schools, the gross failure rate was calculated as the proportion of total students that failed the school year. For secondary schools, the gross failure rate was calculated as the proportion of total students that failed at least one subject in the school year.

9 The evidence cited comes from impact evaluations that isolated the impact of SBM interventions on the studied outcomes. However, since dropout rates are due to many other factors, it is hard to associate dropout to SBM interventions without an impact evaluation. 10 Evidence in Mexico suggests that school grants may contribute to improving learning results (Santibañez, Abreu and O'Donoghue, 2014; Cabrera and Campusano, 2016).


Page 16 of 62

Participating Primary schools’ gross failure rate slightly increased from 1.10 percent to 1.44 percent in the school year 2016-17, falling behind the 1.00 percent target. The increase in the gross failure rate for primary Participating Schools, however, should be taken cautiously. The changes observed are economically small and may be reflecting statistical noise, making them equivalent to no change11. In addition, gross failure rates are driven by many other factors outside of the Project’s scope; for instance, changes in the national failure policy for primary school children may have negatively affected the gross failure rate in Participating Schools. Impact evaluations taking into account SBM interventions have found positive impacts on failure and repetition rates in Mexico (i.e. Skoufias and Shapiro, 2006). 38. The Participating Secondary schools’ gross failure rate slightly increased from 7.79 percent in 2013-14 to 7.84 percent in 2017-18, falling behind the 3.26 percent target. The large difference between the current rate and the expected outcome is due to a mistake in the calculation of the baseline and target values during the Project restructuring (please see paragraph 63 and 68). The correct figures, however, show that secondary schools’ gross failure rate remained practically unchanged, with an increase of 0.06 percent throughout the span of the Project.

39. Although the Project’s activities may have a small positive impact on the overall gross failure rates, the activities under the Project are expected to mitigate in the short to medium term further increments in the gross failure rate among primary and secondary Participating Schools, making it likely to achieve the set targets in the future. As explained previously, the direct support to schools and capacity-building activities financed under the Project helped schools improve their managerial capacity and parental participation, which should increase schools’ responsiveness to local challenges, such as increments in gross failure rates (see theory of change). For instance, participating schools are now able to use data and design school improvement plans to support students that are more likely to fail the school year. Similarly, capacity-building interventions (such as training of school directors and supervisors in the classroom observation tools to provide feedback to teachers and monitor their school improvement plans), coupled with the collection and use of performance indicators through the SisAT, help identify low performers and find strategies to improve learning of students at a greater risk of failing the school year, which should contribute to reduced gross failure rates. Reduce Repetition Rates 40. This objective was measured through the key indicator “Secondary school repetition rate among Participating Schools.” Secondary school repetition rate among Participating Schools slightly increased from 0.63 percent (2014-15 school year) to 0.97 percent (2017-18), falling behind the 0.60 target. This result should be interpreted with caution given that the baseline values were already small and may fluctuate rapidly due to non-Project factors. The changes observed are economically small and may be reflecting statistical noise and are, therefore, equivalent to no change. In addition, repetition rates are driven by many other factors outside of the Project interventions. 41. Although the Project’s activities may have a small impact on helping decrease the overall repetition rates for participating secondary schools, Project activities are expected to contribute to achieving this

11 The last ISR for this Project suggested that the increase in primary gross failure rates could have been affected by the substantial reduction in dropout rates. However, there is not enough evidence to substantiate this argument. In addition, the increase is less than 0.4 percent and therefore the explanation in the main text seems more reasonable.


Page 17 of 62

objective in the medium term in the same way as they are expected to help reduce gross failure rates (see paragraph 39). 42. The indicators used to measure dropout, gross failure and repetition rates do not allow the establishment of a causal relationship between Project activities and the results obtained, mainly because these learning outcomes are affected by factors outside of the Project’s scope. Additional activities in both PRE and PETC may have also contributed to the results in dropout, gross failure and repetition rates. For instance, PRE had a large infrastructure component that helped schools with greater infrastructure needs to build classrooms, which could have contributed to helping parents think their children were better-off at school. Better infrastructure may have also contributed to improved learning. In the case of PETC, an impact evaluation (Silveyra, Yáñez & Bedoya, 2018) showed that the program had a positive impact on reducing dropouts and improving student learning outcomes. The impact evaluation cannot assess the mechanisms through which the different components of PETC, including school autonomy as described above, reduced dropouts and improved students’ learning outcomes. In addition to the extended school day and extra-curricular activities (i.e. music, English, etc.), PETC schools sometimes also include a meal program. The meal program, for example, could have worked as a magnet to retain students, and improve learning outcomes, especially among the poorest. Learning Results 43. This objective was measured through the key indicator “proportion of a representative sample of Participating Schools that improve their student’s learning,” measured by the National Standardized Test (Plan Nacional para las Evaluaciones de los Aprendizajes, PLANEA). Based on the overall results of impact evaluations conducted in 7 States, learning results were better in (T1) schools where Project interventions were closely monitored and implemented directly by the federal government (DGDGE) in coordination with the 7 States, as opposed to the (T2) schools that received similar interventions as part of the federal government’s national strategy to strengthen school autonomy (these schools may or may not have received the interventions, since the implementation was not overseen). In particular, students’ scores in language skills in T1 schools increased considerably and students’ learning improved on average (See Table 1). According to PISA equivalence scales, improving mathematics scores by 0.41 s.d. is equivalent to one additional year of education. This means that students in beneficiary schools accumulated 1.5 months of extra schooling per year.


Page 18 of 62

Table 1. Learning Results

44. The impact evaluations show that managerial capacity has a causal, positive but small impact on learning in Participating Schools in Mexico. Although the effects may not be transformational, they are cost-effective (see efficiency section) and given the impact evaluation results, are worth the investment. Furthermore, Project interventions, and in particular increased school managerial capacity, are expected to have lasting and larger effects in the future. As a meta-analysis12 of SBM interventions shows, results in students learning are seen after 5 to 8 years of implementation (Gertler, Partinos and Rubio-Codina 2007). Therefore, the Project’s long-term effects on the targeted populations are expected to increase with time. 45. The results of the impact evaluations show with certainty that the Project contributed to improved learning results in the short term (3 years) by increasing schools’ managerial capacity (parental involvement may have also contributed to student learning but was not part of the impact evaluation). Overall, school autonomy, as measured by the activities supported by this Project, directly impacted school management (and parental participation) which, in turn, improved student learning outcomes. Justification of Overall Efficacy Rating Rating: Substantial 46. The overall efficacy rating of this Project is considered Substantial since the operation has achieved improved managerial capacity and parental participation, while decreasing dropout rates and improving learning in Participating Schools. Gross failure and repetition rates did not change, but it is difficult to assess if this was due to the effects of Project activities because there are many other factors that affect these variables. This ICR considers that the Project almost fully achieved its objectives since the results of the impact evaluation (De Hoyos et al forthcoming) show that it contributed to improved student learning results, which are a more robust measure of the quality of education than failure and repetition rates.

12 Examination of data from a number of independent studies of the same subject, in order to determine overall trends.


Page 19 of 62

C. EFFICIENCY

Assessment of Efficiency and Rating

47. The Project’s benefits are Substantial. The economic analysis of the first two components of the Project shows positive results in both the cost-benefit analysis and the Internal Economic Rate of Return (IERR). After three years of implementation of Project interventions, the school management index improved by 0.086 s.d. per year, and student learning assessments in Math and Spanish improved, on average, by 0.054 s.d. per year. According to PISA equivalence scales, improving mathematics scores by 0.41 s.d. is equivalent to one additional year of education. This means that students in Participating Schools accumulated 0.132 more years of education per year of exposure to the program (0.054/0.41 = 0.132), which corresponds to 0.395 years over the duration of the program (0.132 × 3 = 0.395) or 1.5 months of extra schooling per year (0.132*12). Therefore, students who benefited from the Project will earn MX$784 more per year than those that did not through the effect of the Project on learning outcomes (50,771 × 0.117 × 0.132 = 784). This corresponds to an increase of 1.5 percent in per capita annual wages. (Please see Annex 4 for a complete explanation). 48. On average, the annual cost per student was of MX$291.2. Students who benefited from the Project earn MX$784 per year more than their peers. With an interest rate of 10 percent, the net present value of the income benefits of the average Project beneficiary is 6.7 times higher than its cost. Under these assumptions, the Project’s IRR is 21.5 percent. (Please see Annex 4 for a complete explanation and the methodology used).

49. DGDGE used existing systems and resources within SEP ensuring smooth project implementation, but disbursements faced a one-year delay given the consolidation of the SBM programs in Mexico, the fluctuations in the exchange rate, and the 2017 earthquake (please see section on Key Factors During Implementation). At closing, the Project had disbursed 92 percent of the funds; however, an extension to disburse the remaining funds until June 30, 2019 was granted. By the extended deadline, it is expected that about 99.4% of the Loan will be disbursed, leaving less than one percent undisbursed. D. JUSTIFICATION OF OVERALL OUTCOME RATING Rating: Satisfactory

50. The overall outcome rating of the Project is deemed Satisfactory. The relevance rating for the Project objectives is High, the efficacy rating is Substantial, and the efficiency rating is Substantial. There were only minor shortcomings in the operation’s achievement of its objectives. The combination of these ratings yields an overall Satisfactory rating. E. OTHER OUTCOMES AND IMPACTS Gender 51. Mexico has achieved gender parity in basic education. Female beneficiaries at the end of the Project accounted for 50.04 percent, while the set target was of 48.7 percent. Half of the student beneficiaries in Participating Schools were female, thus achieving gender parity.


Page 20 of 62

Institutional Strengthening 52. Since 2000, Mexico has provided direct support to schools to improve school autonomy and parental participation and the World Bank has supported SBM efforts since 2006. As the third World Bank Project supporting SBM in Mexico, this Project contributed to consolidating and strengthening a culture of school autonomy and SBM that are now embedded in the system. In particular, this Project helped reinforce the 2013 Education Reform and supported the institutionalization of the school autonomy policy. The policy has endured despite changes in the programs through which the provision of direct support to schools is implemented. The new administration has also continued to support the federal programs to improve school autonomy despite reducing its budget.

Poverty Reduction and Shared Prosperity

53. The Project contributed to the Bank’s twin goals of eliminating extreme poverty and boosting shared prosperity. The Project helped strengthen schools’ managerial capacity and parental participation which, in turn, contributed to reduced dropout rates in Participating Schools. The Project also improved learning outcomes in Participating Schools. All Participating Schools are public schools, many of which are situated in poor areas (on average, 78.33 percent and 51.28 percent of PRE and PETC schools are in highly and very highly marginalized areas). Because of the Project, students in these schools will stay in school longer and will learn more. As more Project beneficiaries enter the labor market (and the additional years of schooling and learning achievement of the workforce increases), the economic benefits of the Project are also expected to increase (see Annex 4). More human capital will translate to higher personal or household incomes, less poverty and, if the increases in human capital take place among disadvantaged households (which is the case in this Project), it will translate to lower income inequality in the long run. Other Unintended Outcomes and Impacts 54. The school autonomy and SBM policies in basic education have permeated into the upper secondary education system. The Secretaría de Educación Media Superior (SEMS) launched in recent years the Fondo para fortalecer la autonomía de gestión en planteles de Educación Media Superior to provide direct financial and technical support to schools following the school autonomy approach that has been undertaken for several years in basic education.

III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION

55. Objectives. The objectives laid out in the PDO were clear, measurable, and based on the available evidence on the impact of SBM initiatives. Based on the literature, the PDO aimed to improve schools’ managerial capacity and parental participation to decrease dropout, failure and repetition rates (World Bank, 2014; Gertler, Patrinos and Rubio-Codina, 2012; Skoufias and Shapiro, 2006; Murnane, Willet and Cardenas, 2006). Learning results were not included in the PDO from the outset because the Bank was cautious of adding learning results as key outcome indicators due to, among others, a short Project lifespan that would not allow enough time to impact learning. In addition, there was no baseline for test scores for Participating Schools. While the Project was being designed, the National Evaluation of Academic Achievement in Schools (Evaluación Nacional del Logro Académico de Centros Escolares, ENLACE) was under revision and there was no certainty on how learning results were going to be


Page 21 of 62

measured in the future,13 so gross failure and repetition rates were chosen as proxies to measure learning. However, dropout, failure, and repetition rates (as well as learning results) are education outcomes that may be affected by factors outside of the Project activities. The only way of measuring the real impact of the Project on such outcomes is through a rigorous impact evaluation, which was also conducted under the Project. 56. Design. The Project design was simple and sought a smooth implementation of the Project. The operational logic behind Component 1 was straightforward: the funds should reach and be administered directly by schools. Component 2 was in tune with the available international evidence and designed to ensure that schools were able to use the resources from Component 1 to improve education service delivery. Component 3 was crucial to estimate and quantify the effect of the Project interventions by including the design and implementation of a rigorous impact evaluation. 57. A drawback of the Project’s design was to allocate funds to Components 2 and 3. The two previous Projects’ experiences warned about the challenges associated with the procurement of technical assistance services, mainly related to the client’s preference to follow local procedures (see procurement section for a complete explanation). All the activities under Components 2 and 3 were undertaken using Government budget instead of the Loan as originally planned in the PAD. It would have been desirable to include Components 2 and 3 as financed with counterpart funding only from the outset. 58. Risk and Mitigation Measures. Few risks that materialized may have needed more attention at appraisal. Nevertheless, none of the materialized risks mentioned below jeopardized the achievement of the PDO: (i) The consolidation of PEC into PRE was not considered a major risk at Project design, but it materialized in 2016 requiring a Level 1 Restructuring (the risk of PEC being discontinued was rated as moderate); (ii) Macroeconomic risks were rated as moderate, but oil prices went down in 2015 and 2016, severely affecting the education budget for SBM programs and thus the number of Project beneficiaries. In addition, the exchange rate increased considerably from the beginning of the Project to closing. A clearer mitigation strategy for these risks would have been desirable. B. KEY FACTORS DURING IMPLEMENTATION Coordination and engagement 59. The three Government agencies involved in the implementation of the Project: Secretaría de Hacienda y Crédito Público (SHCP), National Financial Development Bank (Nacional Financiera, S.N.C., I.B.D, NAFIN), and SEP-DGDGE worked as a team to ensure that the Project was implemented as planned to achieve its development objectives. Disbursements slowed in 2016 due to the consolidation of programs to strengthen SBM in Mexico and the need to restructure the Project. This restructuring took almost a year to complete due to uncertainty from the Government on how and when the Project should finance the new program and if the Project should also finance direct support to PETC schools (given the increases in the exchange rate). Another challenge during Project implementation was that there were

13 In the June 2017 restructuring, the Bank team did not add learning outcomes as part of the PDO even though the new national standardized test (Plan Nacional para las Evaluaciones de los Aprendizajes, PLANEA) was already established. As previously stated, the Bank was cautious in using learning results as measured by test scores as a key outcome indicator. Measurement of learning outcomes, however, was maintained in the Project’s impact evaluation.


Page 22 of 62

cases where the direct support to schools arrived late in the school year, preventing schools from implementing their school improvement plan as planned. SEP addressed this by speeding up the process for transferring funds to schools, for example by keeping a trust fund to manage the direct support to schools’ funds and transferring resources directly from the federal government (bypassing the States). 60. Finally, the States’ uneven capacity to implement the national school director and supervisor training strategy was mitigated by SEP by having the DGDE carry out the coordination and provision of training for all school supervisors in the country. Nevertheless, the capacity-building for school directors was achieved at different levels and paces by states and districts given their own implementation capacity and limitations. For instance, in Tabasco and Durango, there is frequent rotation of personnel, affecting the implementation of Project activities. Therefore, it is important to provide each state with a suite of services to implement school autonomy and accountability initiatives, according to their particular needs and shortcomings. Factors subject to World Bank control Adequacy of Supervision 61. The Bank conducted regular supervision missions at least twice a year that included meetings with SHCP, NAFIN, and SEP, as well as field visits to Participating Schools, all of which helped identify implementation challenges in a timely manner. For instance, through the July 2017 Level 1 Restructuring, the Project was adjusted to respond to the consolidation of the SBM government programs in Mexico. 62. The Bank also sustained close collaboration with SEP in the design and implementation of the impact evaluation. For instance, the Bank provided: (i) experts’ time to ensure the robust design of the impact evaluation and the accuracy of the data collection and analysis; (ii) help to select an adequate instrument to measure schools’ managerial capacity; (iii) financial support for the piloting of the instrument, training of enumerators, and data analysis; (iv) continuous technical assistance tailored to the federal government and the 7 states participating in the impact evaluation, including the design and coordination of numerous conferences and workshops to share best practices and international experiences to inform the selection and implementation of adequate interventions to improve schools’ managerial capacity; and (iv) guidance to select/develop VALORA, an instrument to measure the degree to which the capacity-building strategy was implemented in the 7 states. Overall, the Bank accompanied the Government throughout the impact evaluation process, monitored its timeline closely, and helped ensure that any technical and administrative challenges were overcome. Furthermore, in coordination with SEP, the Bank created a webpage as a repository for the evaluation, including information on the: (i) design and preparation; (ii) lines of intervention; (iii) measurement instruments; (iv) implementation; (iv) results; and (v) good practices. The website is available under the domain: http://escuelaalcentro.com/. 63. The Bank also provided guidance and support related to M&E. For instance, after a mistake in the baseline data and target for secondary schools’ gross failure rates was reported by SEP, the Bank hired a consultant to review all data related to the Project indicators. SEP collaborated fully with the consultant, providing all relevant databases and information. After a detailed review, no further errors were detected.

http://escuelaalcentro.com/


Page 23 of 62

Factors outside the control of the Government or the World Bank

Macroeconomic Environment 64. As mentioned in the risks section, the fall in the world price of oil in 2015 and 2016 produced fiscal austerity in Mexico that resulted in, among others, the reduction of the education budget from 2016 onwards. The fiscal austerity affected the budget of SBM programs, including PRE and PETC and, although at the beginning of the Project a significant increase in students participating in SBM programs in Mexico was expected, the number of beneficiaries actually decreased. The exchange rate also increased considerably from the beginning to the end of the Project. At Project preparation, the Peso to Dollar exchange rate was around 13:1, while it was almost 20:1 by Project closing. In the July 2017 restructuring, the Government of Mexico decided to add PETC schools under Component 1 to ensure that all Project funds could be disbursed despite the changes in the exchange rate. Natural disasters 65. The earthquake that affected Mexico in September 2017 affected approximately 15,000 schools (5,000 had major damages), requiring a significant amount of resources to be redirected towards the reconstruction of the affected schools. The reduction in the amount of available resources from the Government significantly affected the projected number of beneficiaries of both the PRE and the PETC. To redirect existing limited resources, however, the Government adopted strict targeting criteria towards more vulnerable schools, including indigenous, rural, and those located in highly marginalized and marginalized communities. This focus had a significant impact on improving equity in allocating resources to schools. Although the targeted number of beneficiaries was not met, better-targeted schools received more resources. 66. The September 2017 earthquake also damaged several SEP buildings, including the data analysis buildings. This natural disaster further slowed M&E reporting since SEP needed to recover lost data and reallocate personnel responsible for analyzing and reporting indicator data.

IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E)

M&E Design 67. The operation’s theory of change was clear and based on available evidence on similar SBM initiatives (see Annex 7). Each of the key Project indicators were matched to monitor the achievement of each of the Project objectives. The M&E design mainly relied on official data from the Encuesta 911 (the main survey used by the Government to update official education indicators). The M&E design also contributed to incorporating a concrete measure of schools’ managerial capacity (one of the PDO objectives) into SEP’s M&E agenda. The Project provided support to choose an instrument through the piloting, redesign, implementation, and comparison of alternative questionnaires to measure schools’ managerial capacity. The M&E plan helped SEP systematically measure school directors’ managerial capacity for the first time in Mexico. 68. The results framework had some flaws, most of which were corrected in the 2017 restructuring (see Annex 8). Nevertheless, the following remained through the Project: (i) the target for the dropout


Page 24 of 62

rate for Participating Schools was underestimated; (ii) the baseline and targets for gross failure and repetition rates were already small from the outset and did not represent significant changes (for instance, the difference between the baseline and the target for participating secondary schools’ repetition rates was only of a 0.03 percent reduction); (iii) the participating secondary schools’ gross failure rate baseline and target were inaccurate in the 2017 restructuring;14 (iv) the number of project beneficiaries was not updated; and (v) learning results were not added as PDO indicators. M&E Implementation 69. The methodology to calculate each of the Project indicators was clearly described in the PAD and in the July 2017 restructuring paper. The methodology for most indicators followed conventional and local measurements of education indicators such as dropout, failure, and repetition rates. In the case of school managerial capacity, the methodology to collect and analyze data to calculate this indicator followed the WMS guidelines, which has been used in more than 15 different countries. The methodology to measure parental participation was adjusted to the AGE national survey guidelines in the July 2017 restructuring to make data collection for this indicator less expensive and more reliable. There were delays and a few inaccuracies in the reporting of information by SEP to the Bank, as well as high rotation of SEP personnel responsible for analyzing and updating data for Project indicators. Despite this, SEP was diligent in systematizing the information collected (i.e. keeping all relevant files together). M&E Utilization 70. For the first time, the SEP generated information related to the schools’ managerial capacity by using the WMS. The WMS results have been used to inform the school autonomy policy design at the federal, state, and school levels. Each of the 7 States that undertook the WMS received a diagnostic with specific feedback pointing to management areas for improvement, which helped them adjust and tailor capacity-building strategies for schools principals and supervisors. A similar diagnostic has been received by each of the schools participating in the WMS so that each school director can identify and address its managerial shortcomings. In addition, SEP has used the WMS results to shape the national school autonomy policy. Similarly, the information on the AGE questionnaire has been useful to adjust the strategy to increase parental participation and engagement at federal, state, and local levels. Justification of Overall Rating of Quality of M&E Rating: Substantial 71. The rating for the overall quality of M&E is considered Substantial given the moderate shortcomings in the M&E system’s design and implementation. For instance, there were delays in the collection and analysis of the data to update PDO indicators. Nevertheless, the M&E system was generally sufficient to assess the achievement of the objectives and test the links in the results chain.

14 The baseline at restructuring was calculated by SEP at 3.49 percent for PRE and PETC schools for the 2014-15 school year but in March 2018, SEP reported that the correct baseline figure for the 2014-15 school year was 7.79 percent instead. Since the PAD baseline was reported at 9.64 percent for PEC and PETC schools, it seems feasible that the 3.49 percent reported at restructuring was incorrect.


Page 25 of 62

B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE

72. This was a category C Project with a low Social Risk Rating. The Project triggered OP/BP4.10 Indigenous Peoples. Therefore, an Indigenous Peoples Plan (IPP) was approved and published on the Bank and SEP webpages in August 2014. The IPP was updated at the mid-term review (2016), in consultation with the Dirección General de Eduacación Indígena (DGEI), teachers and indigenous leaders, and published in early 2017. The overall implementation of the IPP was satisfactory, especially among school communities that speak Indigenous languages (in Mexico, there are 68 indigenous languages and 364 variants). 73. The main results of the IPP were:

i. Increased awareness of SEP staff at the central level, as well as supervisors, directors and teachers, that Indigenous Peoples may be bilingual and require differentiated strategies to ensure their participation in SEP Programs.

ii. Promotion of the coordination of DGDGE with other departments within SEP around the topic of Indigenous schools and the pertinence of processes and instruments used for M&E and Supervision (DGDGE coordinated with the DGEI, the General Directorate of Curriculum (DGDC), and others).

iii. Improved communication of guidelines, norms, plans, and other important messages by addressing communications in the language of the Indigenous communities to reach parents and students in their native language – i.e. (a) in the SisAT Program, the instructions given to students for reading and writing exercises were translated into two main Indigenous languages (Tutunacú and Náhuatl) thus enabling students to understand instructions well before performing the exercise in Spanish; (b) instruments to monitor the Indigenous PRE and PETC schools were improved. Monitoring of Indigenous schools increased from 8 percent in 2016 to 16 percent in 2017; (c) translation of administrative information (banners/posters) to local languages to improve governance and fiduciary transparency is underway.

iv. Yearly published PRE guidelines now include a clause to alert education authorities (Autoridades Educativas Locales, AEL) that information on Programs, processes, and plans must be communicated to the community in the local language.

v. Indigenous schools were incorporated into the pilot program of a school-based strategy. 104 of 997 schools in this program were Indigenous.

74. Financial Management (FM) and audit. Project Aide Memoires indicated a consistent and satisfactory FM performance, except for the FM supervisions that took place from November 2016 through October 2017, where performance was rated Moderately Satisfactory (MS). FM performance was downgraded during this period mainly due to a lack of disbursements while the Project was restructured to disburse funds to schools in the newly created PRE program. 75. The audit performance was uniformly strong. Annual, independent audits of Project accounts were delivered on time and produced in general clean, unqualified opinions for all years of implementation, except for the 2017 audit report where the opinion was qualified due to observations identified by the auditors (Órgano Interno de Control, OIC) from SEP, which were promptly addressed by the Project. The final Implementation Status and Results Report (ISR) rated FM performance as Satisfactory.


Page 26 of 62

76. FM supervision was conducted in accordance with arrangements stipulated in the legal agreement. The FM risk rating remained Substantial throughout Project implementation due to the Project’s size, scope and complexity. The Bank was able to dedicate greater time and attention to FM oversight as implementation progressed, which translated into visible results. 77. Procurement. As per the Project Procurement visits to participating states, school-level procurement was conducted as agreed in the Operational Manual. SEP could not use the funds allocated to Categories 2 and 3 mainly due to budgetary and regulatory reasons related to the internal budgetary calendar and budgetary constraints that were aggravated by the state of emergency caused by the September 2017 earthquakes. 78. Project design in Mexico should consider the challenges associated with the procurement of technical assistance services with Bank financing. SEP, as other federal entities, is required to follow local budgetary procedures that are difficult to reconcile with national procurement procedures, making procurement of capacity-building activities burdensome for SEP. SEP could thus not finance training or impact evaluation expenses with Project funds as they are not considered “investments” and by Law are not subject to be financed with foreign debt.

C. BANK PERFORMANCE Quality at Entry 79. The Bank provided technical assistance to the Government to prepare a Project with strategic relevance to strengthen schools’ managerial capacity and parental participation, in line with the school autonomy component of the 2013 Education Reform. The Bank’s contribution for quality at entry included sharing international best practices and available evidence on SBM initiatives to inform Project preparation. For instance, the Bank conducted research on available instruments to measure schools’ managerial capacity to help the Government select an adequate instrument for the Mexican context and measure schools’ managerial capacity for the first time. The Bank also provided information on relevant interventions, such as the use of classroom observation tools and students’ performance data to improve teaching practices. Project design considered poverty and social development aspects by targeting public marginalized schools and preparing a plan to address indigenous populations’ specific needs. In terms of FM, the Project used the supported programs’ flow of funds. The safeguards and fiduciary teams were continuously involved in relevant aspects of the Project design. On the other hand, the Bank missed the opportunity to assign counterpart funding instead of loan funds to capacity-building and research activities under the Project, even though lessons learned from the two previous Projects warned of the challenge of using loan funds for these activities. While the Project’s M&E framework had moderate shortcomings, including the underestimation of school dropouts and overestimation of gross failure and repetition rates, it was sufficient to measure the Project impacts. Quality of Supervision 80. The Bank sustained a continuous dialogue with the client, which facilitated the close monitoring of Project implementation. For instance, the Bank began a Level 1 Restructuring in 2016 as soon as the PEC program was consolidated into the PRE program, and disbursements were put on hold until the restructuring was completed. If the Bank had not started the restructuring as soon as the changes in the SBM programs occurred, disbursements would have lagged even further.


Page 27 of 62

81. The Bank also helped overcome technical, administrative, and procurement challenges to undertake the impact evaluation of the Project. Under the impact evaluation, the Bank helped ensure that the interventions to improve schools’ managerial capacity were undertaken as designed at Project appraisal. This engagement was crucial to the successful completion of the impact evaluation, which was a major achievement of this Project since the two previous projects were unable to carry out or finalize the planned impact evaluations.

82. The safeguards and fiduciary teams continuously supervised and developed follow-up reports on the implementation of the IPP and followed up on detailed aspects of the direct support to schools’ disbursements, audits, etc. This was coupled with building staff capacity in the Directorate of School Management (Dirección General de Desarrollo de la Gestión Educativa) on both fiduciary and technical aspects. On the other hand, there were moderate shortcomings. For instance, the above-mentioned restructuring missed the opportunity to update the number of beneficiaries and further adjust the Project’s indicators (i.e. removing gross failure and repetition rates and adding learning r

Implementation Completion and Results Report (ICR) Document...United Mexican States Secretaria de...

Documents

Transcript of Implementation Completion and Results Report (ICR) Document...United Mexican States Secretaria de...