Implementation Completion and Results Report (ICR)...

Document of

The World Bank FOR OFFICIAL USE ONLY

Report No: ICR00004829

IMPLEMENTATION COMPLETION AND RESULTS REPORT

(IDA 5218-PK)

ON A

CREDIT

IN THE AMOUNT OF SDR 259.6 MILLION

(US$ 400 MILLION EQUIVALENT)

TO THE

Islamic Republic of Pakistan

FOR THE

Pakistan: Second Sindh Education Sector Project

June 21, 2019

Education Global Practice

South Asia Region

This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization.

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

CURRENCY EQUIVALENTS

(Exchange Rate Effective {Jun 15, 2019})

Currency Unit = Pakistani Rupee (PKR)

PKR 140.57 = US$1

US$1.39 = SDR 1

FISCAL YEAR

July 1 - June 30

Regional Vice President: Hartwig Schafer

Country Director: Patchamuthu Illangovan

Senior Global Practice Director: Jaime Saavedra Chanduvi

Practice Manager: Mario Cristian Aedo Inostroza

Task Team Leader(s): Shinsaku Nomura, Umbreen Arif

ICR Main Contributor: Tashmina Rahman

ABBREVIATIONS AND ACRONYMS

ASC Annual School Census

ASER Annual Status of Education Report

CPF Country Partnership Framework

DG Directorate General

DLI Disbursement-linked Indicator

ECE Early Childhood Education

EOP End-of-Project

ESMF Environment and Social Management Framework

GDP Gross Domestic Product

GoP Government of Pakistan

GoS Government of Sindh

GPE Global Partnership for Education

ICR Implementation Completion and Results Report

ISR Implementation Status and Results Report

IT Information Technology

KPI Key Performance Indicator

M&E Monitoring and Evaluation

NER Net Enrollment Rate

NPV Net Present Value

PAD Project Appraisal Document

PDO Project Development Objective

PDV Present Discounted Value

PPP Public-Private Partnership

PPRS Promoting Private Schooling in Rural Sindh

PSLM Pakistan Social and Living Standards Measurement

RSU Reform Support Unit

SAT Student Achievement Test

SDG Sustainable Development Goal

SEF Sindh Education Foundation

SELD Sindh Education and Learning Department

SEMIS Sindh Education Management Information System

SERPII Sindh Education Sector Reform Program II

SESP Sindh Education Sector Plan

SESPII Second Sindh Education Sector Project II

SID School Infrastructure Development

SMC School Management Committee

SSB School- Specific Budget

SPR School Participation Rate

TA Technical Assistance

US$ United States Dollar

TABLE OF CONTENTS

DATA SHEET .......................................................................................................................... 1

I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5

A. CONTEXT AT APPRAISAL .........................................................................................................5

B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) ..................................... 11

I. OUTCOME .................................................................................................................... 15

A. RELEVANCE OF PDOs ............................................................................................................ 15

B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 16

C. EFFICIENCY ........................................................................................................................... 21

D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 22

E. OTHER OUTCOMES AND IMPACTS (IF ANY) ............................................................................ 23

II. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 25

A. KEY FACTORS DURING PREPARATION ................................................................................... 25

B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 25

III. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 27

A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 27

B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 29

C. BANK PERFORMANCE ........................................................................................................... 30

D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 32

IV. LESSONS AND RECOMMENDATIONS ............................................................................. 32

ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 35

ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 44

ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 46

ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 47

ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 50

ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 51

ANNEX 7. OTHER MEASURES OF SCHOOL PARTICIPATION IN SINDH ..................................... 54

ANNEX 8. SUMMARY OF IMPACT EVALUATION ON SCHOOL MANAGEMENT COMITTEES ..... 55

ANNEX 9. STUDENT LEARNING OVER SESP II PERIOD ............................................................ 56

ANNEX 10. SUMMARY OF BORROWER’S ICR ........................................................................ 58

The World Bank Pakistan: Second Sindh Education Sector Project (P125952)

Page 1 of 59

DATA SHEET

BASIC INFORMATION

Product Information

Project ID Project Name

P125952 Pakistan: Second Sindh Education Sector Project

Country Financing Instrument

Pakistan Investment Project Financing

Original EA Category Revised EA Category

Partial Assessment (B) Partial Assessment (B)

Organizations

Borrower Implementing Agency

Islamic Republic of Pakistan Reform Support Unit, School Education and Literacy

Department

Project Development Objective (PDO) Original PDO

The project supports the Sindh Government’s Second Sindh Education Sector Reform Program (SERP II). The Project Development Objective is to raise school participation by improving sector governance and accountability and strengthening administrative systems, and measure student achievement.


Page 2 of 59

FINANCING

Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$)

World Bank Financing IDA-52180

400,000,000 361,391,511 332,555,053

Total 400,000,000 361,391,511 332,555,053

Non-World Bank Financing 0 0 0

Borrower/Recipient 2,300,000,000 4,460,493,658 4,460,493,658

Total 2,300,000,000 4,460,493,658 4,460,493,658

Total Project Cost 2,700,000,000 4,821,885,169 4,793,048,711

KEY DATES

Approval Effectiveness MTR Review Original Closing Actual Closing

14-Mar-2013 19-Mar-2014 30-Jun-2017 31-Dec-2018

RESTRUCTURING AND/OR ADDITIONAL FINANCING

Date(s) Amount Disbursed (US$M) Key Revisions

10-Feb-2014 0 Change in Results Framework Change in Legal Covenants

23-Jun-2017 321.43 Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Reallocation between Disbursement Categories Change in Implementation Schedule

21-Dec-2018 321.43 Change in Components and Cost Cancellation of Financing Reallocation between Disbursement Categories

KEY RATINGS

Outcome Bank Performance M&E Quality

Satisfactory Moderately Satisfactory Modest


Page 3 of 59

RATINGS OF PROJECT PERFORMANCE IN ISRs

No. Date ISR Archived DO Rating IP Rating Actual

Disbursements (US$M)

01 24-Jun-2013 Satisfactory Satisfactory 0

02 08-Jan-2014 Satisfactory Moderately Satisfactory 0

03 29-Jul-2014 Satisfactory Satisfactory 79.73

04 02-Mar-2015 Satisfactory Moderately Satisfactory 87.92

05 08-Oct-2015 Moderately Satisfactory Moderately Satisfactory 182.39

06 25-May-2016 Moderately Satisfactory Moderately Satisfactory 182.39

07 27-Jan-2017 Moderately Satisfactory Moderately Satisfactory 296.09

08 05-Jun-2017 Moderately

Unsatisfactory Moderately Satisfactory 296.09

09 10-Jan-2018 Moderately

Unsatisfactory Moderately Satisfactory 321.43

10 25-May-2018 Moderately Satisfactory Moderately Satisfactory 321.43

11 26-Dec-2018 Moderately Satisfactory Moderately Satisfactory 321.43

SECTORS AND THEMES

Sectors

Major Sector/Sector (%)

Education 100

Public Administration - Education 20

Primary Education 40

Secondary Education 40

Themes

Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Private Sector Development 10

Public Private Partnerships 10


Page 4 of 59

Public Sector Management 20

Public Administration 20

Transparency, Accountability and Good Governance

20

Human Development and Gender 80

Education 80

Access to Education 40

Education Financing 40

ADM STAFF

Role At Approval At ICR

Regional Vice President: Isabel M. Guerrero Hartwig Schafer

Country Director: Rachid Benmessaoud Patchamuthu Illangovan

Senior Global Practice Director: Jesko S. Hentschel Jaime Saavedra Chanduvi

Practice Manager: Amit Dar Mario Cristian Aedo Inostroza

Task Team Leader(s): Umbreen Arif Shinsaku Nomura, Umbreen Arif

ICR Contributing Author: Tashmina Rahman


Page 5 of 59

I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES

A. CONTEXT AT APPRAISAL

Context

1. Country and provincial context. At the time of appraisal in 2012, Pakistan was the sixth most populous country with 179 million inhabitants. It was a lower-middle-income country according to the World Bank’s classification system, with a per capita income of US$1,050 in 2011. The country was experiencing a slowdown of economic growth and poverty reduction. The pace of gross domestic product (GDP) growth had been slowing down over the past decade, averaging around 3.5 percent in 2012. The economic turmoil was from a confluence of factors including adverse macroeconomic, fiscal, political, and security challenges. Added to this, inadequate governance and management capacities were hampering public provision of social services at all levels.

2. Sindh, the second-largest province by population and with the second-highest per capita income level out of the four provinces, was no less insulated from the economic turmoil. With a population of 40 million, the province was experiencing declining economic growth and stagnated household incomes over the past decade. The economic slowdown of the province was underpinned by several key structural challenges including (a) widening urban-rural disparities in welfare and other social and economic segmentations; (b) poor law and order conditions; (c) low-quality provision of social and economic services; and (d) shortages of skills, low productivity, and outputs. Moreover, a disproportionate share of natural disasters and some of the worst power crises adversely affected rural households in the province. Given the situation at the time, the near-term economic outlook for Pakistan, and Sindh in particular, was bleak with heightened risk of further economic deterioration without strong political commitment and interventions to support critical governance reforms. The Government of Pakistan (GoP) took on the ambitious task to regain macroeconomic stability and focus on structural reforms in efforts to steer Pakistan’s GDP growth back to the range of 5–7 percent per year.

3. Sector context. In 2012, school participation in Pakistan was low and trailing behind most other countries in the region. The country was likely to not achieve the Millennium Development Goals of universal primary education by 2015. In Sindh, the Government has been the main provider of education service, with around one-quarter of total public expenditures dedicated to primary and secondary education in 2012. Around 3.56 million students were enrolled in Grades 1–12 in the government school system, which comprised roughly 43,000 functional schools and 148,000 teachers. At the time of appraisal, there was an observed long-term enrollment shift from government to private schools. The proportion of students enrolled in government primary schools was 74 percent in 2004/05, but it declined to 59 percent in 2012/13.1

4. School participation rates (SPRs) for Sindh were low—the primary, middle, and matriculate levels stood at 65.1 percent, 65.6 percent, and 49.8 percent, respectively. Gender parity index stood at 69 percent at the primary level in government schools. In addition to low school participation, learning outcomes were also very poor with Grade 4 students achieving an average score of less than 50 percent in mathematics and local language in government

1 Pakistan Social and Living Standards Measurement survey.


Page 6 of 59

schools.2 One of the major reasons for the low education outcomes in the province was the underperformance of the government school system. One indication for the underperformance was that nearly 90 percent and 55 percent of households reported living approximately 15 minutes away from the nearest primary and middle school respectively, yet SPRs for children ages 6–10 and 11–13 in these households were only 65 percent and 78 percent, respectively. The low participation continued despite free government schooling.

5. The underperformance of the government schooling system resulted from several factors, including (a) poor human resource management and weak planning and implementation capacities at all levels, (b) inadequate monitoring and accountability in the service delivery system, and (c) system captures by vested interest groups. Poor governance and management in the sector created challenging school environments that adversely affected the quality of education delivery. For instance, 72 percent of primary schools had one or two teachers while 52 percent had poor quality or no teaching-learning facilities apart from teachers and textbooks.3 Without improvements in the quality of school education and service delivery, households had less incentive to send children to the schools.

6. Rationale for World Bank assistance. The rationale for World Bank assistance to support the Government’s Second Sindh Education Sector Reform Program (SERP II) was strong. The World Bank had a long-standing partnership with the Government of Sindh (GoS) in education sector development through support to the first Sindh Education Sector Reform Program (SERP I). The World Bank provided financial support to the Government’s reform program through a development credit policy (2007–2009) and a results-based financing operation (2009–2012). Under the results-based operation, the Government complied with all legal covenants and met all the agreed disbursement-linked indicators (DLIs), achieving a rating of Moderately Satisfactory at the Project Development Objective (PDO) level and World Bank and borrower’s performance as Satisfactory according to the Sindh Education Sector Project (SESP I) Implementation Completion and Results Report (ICR). Building upon the lessons of SERP I, the GoS embarked upon a next-level ambitious reform agenda to further strengthen governance and management in the sector through SERP II. Established as a credible and reliable long-term partner in education to the GoS, the World Bank’s sustained and strategic engagement through SESP II was viewed critical to (a) promote and protect continuity of politically challenging but crucial governance reforms for improving service delivery, (b) encourage stronger results orientation, (c) strengthen program and implementation design, and (d) provide important technical assistance (TA) and facilitate systematic knowledge generation on innovative interventions and strategies.

2 Provincial Educational Assessment Center Diagnostic Test 2009. 3 Annual School Census 2011/12.


Page 7 of 59

Theory of Change (Results Chain)

7. Figure 1 illustrates the Theory of Change for the Second Sindh Education Sector Project (SESP II). SESP II’s development outcomes would ultimately contribute to achievement of the national and provincial development priorities in primary and secondary education. The project aimed to achieve two main objectives: (a) raise school participation by improving sector governance and accountability and strengthening administration systems and (b) measure student achievement. To achieve these objectives, the project included a set of critical governance and management reforms to address the issues of underperformance of service delivery at the provincial education office and schools. These reform areas were supported by 11 results-based activities, which were part of the GoS’ SERP II, ensuring sustainability of the reform initiatives.

8. To support the objective of raising school participation, SESP II comprised 10 DLI-supported reform activities4 to improve accountability and governance and strengthen administrative system for improved education service delivery. The DLI-supported activities focused on the following areas: (a) enhanced planning, budget preparation, and execution capacities at the central and school levels to implement education development activities timely and with quality (DLI 1 and DLI 4); (b) strengthening annual school census (ASC) systems to provide reliable and timely information to the Government and households for decision making (DLI 2); (c) strengthening teacher management system and professionalization of education management to enhance human resource in the sector (DLI 9 and DLI 10); (d) implementation of school budgets and enhanced School Management Committees (SMCs) to promote quality improvements through community engagement and school-level leadership (DLI 4 and 5); (e) infrastructure development and promoting school consolidation for improved rationalization of resources for increasing the number of functional quality schools (DLI 4, DLI 6 , DLI 7, and DLI 11); and (f) strengthening public-private partnerships (PPPs) in education to improve access to education in underserved rural areas in the province (DLI 8).

9. To support the objective of measuring student achievement, SESP II supported a DLI activity (DLI 3) that aimed to strengthen the student achievement test (SAT) for measuring student learning reliably and on time and inform the Government, schools, and households for policy and decision making.

4 The 11th DLI was introduced during the restructuring in June 2017.


Page 8 of 59

Promoting a strengthened SAT system

Supporting sound budget preparation and execution on time in primary and secondary education

Promoting the professionalization of education management at all levels

Promoting sound standardized recruitment, performance management, guidance, and support to contracted teachers

Strengthening administration of the ASC for government schools

Promoting school level salary and non-salary budgeting, and proper management & use of non-salary budgets

Promoting strengthened due diligence, fiduciary management and support for SMC grant provision and use

Promoting the rationalization of government school facilities and resource and operation of consolidated schools as single schools

Promoting strengthened rules, procedures, and practices in school rehabilitation and school upgradation

Supporting expansion of a strengthened accountability-based PPP in education service delivery

Upgrading target schools to basic quality standard schools

Intermediate Results

Outcomes

Strengthened budget system in primary and secondary education and SERP II initiatives (DLI 1)

More accountable staff management and recruitment system. Better and more accountable school management. Number of additional qualified teachers at the primary level in government schools

More efficient and better quality education in campus schools established

Improved school infrastructure through additional classrooms built or rehabilitated at the primary level, government schools

Increased provision of schooling in unserved rural communities through strengthened PPP (DLI 8)

To measure student achievement

Improved accountability

and governance

and strengthened administrative

system

Improved service delivery

Increased school

participation

Activity

Regular assessment of students’ learning through SAT

Enhanced planning, budget preparation and implementation capacities at schools (DLI 4)

Annual School Census implemented each year using strengthened administrative procedures practices (DLI 2)

Figure 1: Theory of Change: Second Sindh Education Sector Project II


Page 9 of 59

Project Development Objectives (PDOs)

10. The PDO is to raise school participation by improving sector governance and accountability and strengthening administrative systems, and measure student achievement.

Key Expected Outcomes and Outcome Indicators

11. The PDO has two objectives with the following outcome indicators:

Objective 1: To raise school participation by improving sector governance and accountability and strengthening

administrative systems

• Key Performance Indicator (KPI) 1: Net Enrollment Rate

(a) Net enrollment rate, primary level, ages 6–10 years

(b) Net enrollment rate, secondary level, ages 11–13 years

(c) Net enrollment rate, matriculate level, ages 14–15 years

Objective 2: To measure student achievement

• KPI 2: Annual administration of the Student Achievement Test (SAT) in Grades 5 and 8 in all government

schools

Components

12. The SESP consisted of two components: (a) a results-based component that supported the GoS’s SERP II and (b) a TA component. The descriptions are derived from the Financing Agreement and the Project Appraisal Document (PAD).

Component 1: Result-based Financing of the Second Sindh Education Sector Reform Program (Estimated Cost: SDR 255 million; Actual Cost: SDR 226.64 million) 13. According to the PAD, Component 1 comprised 10 DLIs for each fiscal year with the objective to ‘address

gaps in and/or current poor practices in sector management and governance’. The 10 DLIs are the following:

• DLI 1: Program Budget and Expenditure Management. The aim of this DLI was to support the preparation of sound budgets for the primary and secondary education sector and ensure timely and full budgetary releases and payments for incurred expenditures.

• DLI 2: Annual School Census (ASC). This DLI aimed to support regular, reliable, and timely data collection through the ASC for effective decision making. This activity entailed administration of the ASC in government schools on a regular basis in an expanding set of districts through two distinct channels (the Government and third party) accompanied by strengthened administrative procedures, and practices,


Page 10 of 59

and utilization of a web-based direct reporting information technology (IT) platform by secondary and higher secondary schools to improve the regularity, relevance, and reliability of ASC data.

• DLI 3: Student Achievement Test (SAT). The DLI aimed to ensure administration of SAT on a regular basis, accompanied by third-party review to strengthen the reliability and validity of the test results and the responsible use and dissemination of test results.

• DLI 4: School Budgets. This DLI aimed to promote the preparation of school budgets, both salary and non-salary components, following transparent, objective, and needs-based criteria. The activity also facilitated third-party support to the districts for managing and monitoring the flow and use of school budgets in line with applicable rules and regulations.

• DLI 5: School Management Committees (SMCs). This DLI aimed to strengthen the SMCs through stronger due diligence and financial management controls; improve communication of expectations, roles, responsibilities, and redressal avenues to all key parties; support services offered by regions and districts; and increase grant use in line with applicable rules and regulations.

• DLI 6: School system consolidation. This DLI aimed to improve resource efficiency through merging of distinct government schools that share the same building, same compound, or are in close proximity and cater to the same local child population into single schools and their reorganization and strengthening to function as single schools.

• DLI 7: School infrastructure development. The DLI aimed to enhance school teaching-learning facilities through investments which support (a) whole school development of schools with infrastructure deficiencies; (b) upgradation of primary schools to elementary schools; and (c) consolidated schools, following objective, transparent, and needs-based criteria compliance with stipulated construction quality and school design specifications. The activity also supported third-party screening, assistance, monitoring, and certification of construction completion and quality and pilot interventions on specific environmental enhancements in schools.

• DLI 8: Incentive- and accountability-based public financing of the private provision of schooling. The objective of this DLI was to support expansion of the Promoting Private Schooling in Rural Sindh (PPRS) Program of the Sindh Education Foundation (SEF) through regularization of program financing, strengthened administrative and monitoring systems, strengthened design and enforcement of incentive and accountability conditions to improve school performance, and strengthened program qualification criteria and the rigorous application of rules and procedures to screen rural communities for program school placement.

• DLI 9: Education management. This DLI aimed to strengthen education management. The subcomponent supported activities including (a) appointment of specialized cadres of education managers and school headmasters following transparent, objective, merit-based criteria, and rigorous mechanisms; (b) performance-based contracts; (c) induction training, job guidelines, management materials, and tools; and (d) and a tailored annual performance evaluation process.


Page 11 of 59

• DLI 10: Teacher management. The DLI aimed to strengthen teacher management system including (a) merit- and needs-based teacher recruitment arrangements, procedures, and practices; (b) teacher contracts with performance terms and conditions; (c) induction training, job guidelines, teaching materials, and tools; and (d) a tailored annual performance evaluation process.

Component 2: Technical Assistance (Estimated Cost: SDR 4.60 million; Actual Cost: SDR 5.04 million)

14. The objective of this component was to finance important technical, advisory, and capacity-building support to strengthen fiduciary and environmental management, administrative, and monitoring and evaluation (M&E) activities under the project. The component also encompassed selected activities that supported program implementation progress and performance including the achievement of DLIs.

B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE)

Revised PDOs and Outcome Targets

15. The PDOs were not revised during implementation. However, the outcome indicators and targets were revised for refinement and to account for expanded scope and project extension during a restructuring in June 2017.

Revised PDO Indicators

16. Refinement of KPIs. Key changes include (a) outcome indicator changed from net enrollment rate (NER) to SPR, (b) introduction of three new outcome indicators related to Objective 1 (school participation), and (c) change in unit of measurement and revised target for SAT to include another round related to Objective 2 (student achievement). The changes in outcome indicators and rationales during the restructuring are discussed in the following sections.

17. Change in outcome indicator from NER to SPR. Before restructuring in June 2017, Objective 1 (school participation) was tracked using a single KPI on NERs at primary, middle, and matriculate levels based on the Pakistan Social and Living Standards Measurement (PSLM) household survey implemented intermittently by the GoP. PSLM 2015/16 showed a decline in NER in Sindh and other provinces,5 which led the Pakistan Bank education team to investigate the reliability of the indicator (table 1). A review showed that NER was not a robust and reliable measure for school participation mainly due to two reasons: (a) NER captures only students enrolled in age-relevant grades while late entry into the school system in Pakistan and Sindh, in particular, was prevalent and the norm6 and (b) an analysis showed potential inaccuracy of age data in household surveys and hence lesser consistency in NERs compared to other measures such as SPR.7 On the other hand, SPR captures enrollment of the specified age group irrespective of the level of education they are enrolled in, thereby making it more reliable and relevant for measuring school

5 Based on PSLM surveys between 2011/12 and 2014/15, similar decline in school NER was observed for Punjab and Khyber Pakhtunkhwa as well as a decline in national NER at primary level for children ages 6–10 years. 6 UNESCO-IIEP (United Nations Educational, Scientific, and Cultural Organization-International Institute for Educational Planning). 2018. Situational Analysis of the Education Sector in Sindh. 7 World Bank. 2017. SESP II Restructuring Paper.


Page 12 of 59

enrollments in the province. Hence, it was the Pakistan education team’s strategic shift across the board to adopt the use of SPR as an appropriate and better measure of the project’s achievement in student participation.8

Table 1: NER and SPR (in percentage, %)

Level Baseline

(2011/12) End Line

(2015/16) Target

Primary NER (ages 6–10 years) 61.6 61.0 69.7

Middle NER (ages 11–13 years) 35.7 34.0 40.9

Matriculate NER (ages 14–15 years) 23.1 25.0 26.0

Primary SPR (ages 6–10 years) 65.1 68.0 69.0

Middle SPR (ages 11–13 years) 65.6 66.8 67.7

Matriculate SPR (ages 14–15 years) 49.8 55.0 55.8

Source: PSLM surveys (2012 and 2015).

18. Inclusion of new PDO indicators. At the time of restructuring in June 2017, the task team reported in the SESP II Restructuring Paper that SPR would not be available for tracking progress on or after 2017/18 due to unavailability of household survey data beyond 2015/16. On the other hand, the Sindh Education Management Information System (SEMIS) covered only government schools despite the continuous outflow of students from the public to private system and therefore did not allow for a complete picture of sector enrollments. In this context, the SESP II Restructuring Paper reports the inclusion of annual retention rate (Grade 1 to Grade 2) calculated from SEMIS data to be used as an alternative measure of immediate outcomes of school participation in 2017/18. In addition, annual transition rate (Grade 5 to Grade 6) based on SEMIS was included as an outcome indicator for school participation. The choice of the indicators was based on two points: (a) these indicators are directly influenced by the project intervention and (b) the rationale to support the GoS in tackling the high dropout rates at primary levels and during transition from primary to middle school. The GoS in its Sindh Education Sector Plan 2014–2018 highlighted the alarming dropout of children at the primary level with only 50 percent of children surviving primary school. The transition rate for children from primary to middle school (that is, how many children enrolled in Grade 5 get to Grade 6) was only around 56 percent in 2011/12, indicating that more than one-third of children did not continue their education after primary school. The selection of these two indicators as alternative measures was strategically relevant for the provincial policy and valid under the data constraints.9

19. Additionally, a new outcome indicator on basic quality standard schools was added under Objective 1 to account for expanded project scope under a new activity on basic quality standard schools (discussed in the section Rationale for Changes and Their Implication on the Original Theory of Change, paragraph 28). As noted in the PAD, the low quality of schooling was considered one of the contributing factors for families not sending children to school. The

8 Pakistan Development Update published in November 2017 also used SPR and explains the reason for using this indicator as follows: “The SPR allows greater flexibility for late entrants into the school system. Although the official primary school entrance age is 5 years, late entry into the schooling system is common in Pakistan as approximately one fourth of all children begin schooling at the age of 6 or older (page 75 of World Bank Pakistan Development Update November 2017).” 9 The end-of-project (EOP) targets set for retention and transition rates were based on the incremental improvement over the past trends. The EOP target for retention was set to achieve an increase in 5.5 percentage points from the baseline, while the target for transition rate was set to achieve an increase of 3 percentage points from the baseline.


Page 13 of 59

rationale for inclusion of this new PDO-level indicator was to capture the reform outcome in raising school participation through improved quality of school environment supported by various DLI activities.

20. Expanded target under Objective 2. A change in unit of measurement of indicator (SAT) from text response (Yes/No) to cumulative number of rounds and inclusion of one more round to account for project period extension was made to better track the SESP II achievement in measuring student achievement.

21. The revised outcome indicators following the restructuring in 2017 are summarized below:

Objective 1: To raise school participation by improving sector governance and accountability and strengthening administrative systems

• KPI 1: School Participation Rates

(a) School participation rate, primary level, ages 6–10 years

(b) School participation rate, secondary level, ages 11–13 years

(c) School participation rate, matriculate level, ages 14–15 years

• KPI 2: Number of basic quality standard schools

• KPI 3: Annual Retention Rate between Grade 1 and Grade 2

• KPI 4: Annual Transition Rate from Grade 5 to 6

Objective 2: To measure student achievement

• KPI 5: Cumulative number of Student Achievement Test (SAT) conducted

22. Following the restructuring, most of the sector reforms and intermediate outcomes were accounted through introducing improved intermediate indicators. Capturing qualitative changes in education service delivery from governance reforms through indicators is challenging as changes in administrative processes and management systems are less visible than physical outputs. However, intermediate-level indicators were included to track outputs related to the governance and management reforms, including the number of additional classrooms built or rehabilitated, additional qualified teachers in primary schools, schools with head teachers, number of campus schools, and number of schools with biometric teacher attendance monitoring system. These were reasonably good indicators of improvements in service delivery at the provincial education office and school levels.

Revised Components

23. The SESP II activities were implemented through two components. During restructuring in June 2017, activities under both components were scaled up without cost escalation to (a) more effectively translate the reform actions achieved through met DLIs10 into outcomes, (b) deepen effectiveness of project operation by diverting more fund to

10 At the time of restructuring in 2017, out of the 30 DLI targets for three years (2013/14, 2014/15, and 2015/16), 24 DLIs were fully met and 4 partially met (out of 30)—which had contributed to several reform initiatives undertaken by the GoS.


Page 14 of 59

high-performing SAT activity, and (c) generate critical knowledge to inform strategy and operations through pilot interventions and studies and assessments under TA.

24. Under the results-based Component 1, the following changes were made: (a) scale-up of DLI 3 to include an additional round of SAT in school year 2017/18, (b) revision of achievement date for DLI 10 on teacher management, and (c) introduction of DLI 11 on basic quality standard schools. The DLI protocols and verification conditions remained unchanged for the 10 DLIs under the original project.

25. Under Component 2, studies and evaluations were included for knowledge generation and documentation of activities under the project. The revised TA component included five new analytical studies, including education sector assessment and support to the preparation of the next phase of Sindh Education Sector Plan (2019-2024). A new pilot activity on the feasibility of single-teacher school rationalization was also included to find solutions to the quality degradation in vast single-teacher schools in Sindh.

Other Changes

26. The project underwent restructuring three times, as per the following details:

• Level 2 restructuring approved on February 10, 2014. Under this restructuring, the main changes

were a change in due dates of two legal covenants, a correction of academic years referred in DLI 5

(school management), and a correction of account code used for fund release from the Finance

Division to the Education Department.

• Level 2 restructuring approved on June 23, 2017. Under this restructuring, the other changes were

an additional 18-month extension in the closing date to December 31, 2018, to allow for completion

of project activities and a reallocation of funds in disbursement categories to support enhanced

activities under both components.

• Level 2 restructuring approved on December 22, 2018. A total of US$38 million in IDA funds were

cancelled for three unmet annual targets of DLI 10 and DLI 11 and for uncommitted amounts of TA

(discussed in detail in the section Rationale for Changes and Their Implication on the Original Theory

of Change, paragraph 29).

Rationale for Changes and Their Implication on the Original Theory of Change

27. All refinements to the outcome indicators ensured that the two objectives of the PDOs were well accounted for by one or more relevant KPIs and reflected improved measures of the project’s achievement.

28. At the time of the original project closing date of June 30, 2017, the disbursement was US$205 million (79 percent) of credit proceeds. The remaining balance was the result of two factors: (a) a one-year delay in project effectiveness which adversely affected implementation progress of teacher management activity and (b) undisbursed funds from four partially met DLIs (1,2,7, and 9) and the unmet DLI on teacher management (DLI 10). The teacher management DLI involved the development of a teacher recruitment policy followed by teacher recruitments based on the new


Page 15 of 59

policy. Teacher recruitment reform is a highly politically challenging agenda that takes time for consensus building and implementation. At the same time, many schools had already benefitted from sector-level reform activities under SESP II but required additional school-specific support to upgrade to basic quality standard schools11 for ensuring functionality and enabling greater school participation. The GoS requested that SESP II be extended by 18 months and remaining funds be used to support a new activity on upgrading target schools to basic quality standard schools and also another round of the well-performing SAT activity to deepen development effectiveness of the project. The Government also requested the World Bank to support the preparation of the next Sector Plan through background assessments and studies using increased TA under Component 2. The rationale for the World Bank to support these changes was to (a) build on the sector-level reforms and intensify impact outcomes at the school level through upgrading schools to basic quality functional schools, contributing to Objective 1 and (b) enable the completion of the implementation of remaining activities through the extension of the closing date up to December 31, 2018.

29. The cancellation of funds (5 percent of total allocation) was only at EOP. The cancellation comprised missed targets under DLI 10 and DLI 11 and dropped studies and evaluations under the TA component. The two missed DLIs did not achieve their full targets; however, the progress made with these DLIs contributed to important reforms supporting the project objective of raising school participation. DLI 10 contributed to the introduction of a new merit-based teacher recruitment policy, which is an important reform for enhancing quality of teaching and learning and encouraging families to send children to schools. On the other hand, DLI 11 though could not achieve its EOP target but enabled 1,366 schools to meet the basic quality standards, increasing capacity to enroll more children in these schools with better school environment. Both DLI activities are progressing under the GoS parent program with dedicated budget, and hence, their objectives envisioned under SESP II continue to be pursued.

30. Based on this discussion, the changes in outcome indicators and components did not have any impact on the original Theory of Change. The changes were based on strong rationales to adopt appropriate and better measures of project objectives and to deepen the impact of reforms through revised activities supporting the project objectives.

I. OUTCOME

A. RELEVANCE OF PDOs

Assessment of Relevance of PDOs and Rating

31. The relevance of PDOs is rated High. At the time of closing, the PDOs remain fully aligned with the Government’s strategies and the World Bank’s Country Partnership Framework (CPF) priorities. The attainment of universal access to primary education (according to the Constitution Article 25A), expansion of access to secondary and higher levels of education, and improved quality of education service delivery continue to remain priority areas in the National Five-Year Plan (2013–2018) and the Sindh Education Sector Plan (2014–2018). The project addressed important governance concerns related to recruitment of qualified school teachers, inadequate and substandard school facilities, inadequate implementation and management capacities at all levels, and deficiencies

11 The GoS identified seven criteria that were needed to ensure basic quality standard in schools. The Government identified 992 schools that met five criteria and another 2,080 schools that met three to four criteria out of these seven quality criteria and aimed to provide the additional support to these ‘target schools’ and transform them into fully functional ‘basic quality schools’.


Page 16 of 59

in student assessment system, which were considered detrimental to school participation and quality learning by the Government as stated in the National Education Policy 2017 and the SESP (2014–2018). Furthermore, the current CPF FY15–FY19 of Pakistan identifies improved service delivery in education as a foundational priority and prerequisite for improved human development outcomes. Under Results Area 4 (Service Delivery), the CPF promotes improved school enrollments and adoption of education quality assessment to measure student learning (Objective 4.3). The SESP II development objectives—to raise school participation by improving sector governance and accountability and to measure student achievement—were fully aligned with the CPF priorities in education.

32. The SESP II development objectives also remained highly relevant to the Sustainable Development Goal (SDG) 4, specifically on ensuring universal primary and secondary education. The availability of qualified teachers is essential to ensure functionality of schools and raise quality of learning. SESP II specifically supported the SDG implementation action (4c) of increasing supply of qualified teachers in schools through the teacher management, basic school quality, and the biometric teacher attendance monitoring activities.

B. ACHIEVEMENT OF PDOs (EFFICACY)

Assessment of Achievement of Each Objective/Outcome

33. As discussed, there are two main outcomes that the project set out to achieve. Ratings of achievements of

each objective and their justifications are discussed in the following paragraphs.

34. To raise school participation by improving sector governance and accountability (Objective 1): Achievement of this objective is rated Substantial. The KPIs relevant to this objective are KPI 1 (school participation rate), KPI 2 (basic quality standard schools), KPI 3 (retention rate), and KPI 4 (transition rate).12 Each outcome indicator is reviewed against its target followed by a discussion on the contribution of SESP II-supported activities toward the PDO of raising school participation.

Table 2. Indicators and the Achieved Outcomes

Indicator Baseline Target Actuala Achievement (%)

Outcome Indicators

KPI 1: School Participation Rates (%)

Primary SPR (ages 6–10 years) 65.1 69.0 68.0 74

Middle SPR (ages 11–13 years) 65.6 67.7 66.8 57

Matriculate SPR (ages 14–15 years) 49.8 55.8 55.0 87

KPI 2: No. of Basic Quality Standard Schools 992 2,500 1,366 25

KPI 3: Annual Retention Rate between Grade 1 and Grade 2 (%) 76.0 82.0 87.6 167

KPI 4: Annual Transition Rate from Grade 5 to 6 (%) 56.5 60.0 66.8 294

12 The end line statistics are calculated based on the ASC of 2018/19 (which is named as the School Profiling Survey). The aggregated preliminary statistics were shared on May 12, 2019. As the ASC did not take place for 2017/18, the KPI on retention and transition rate were calculated by using 2018/19 data and interpolated 2017/18 based on the past trend.


Page 17 of 59

Intermediate Indicators

IR Indicator 2: Number of additional classrooms built or rehabilitated at the primary level, government schools

0 2,400 1,808 75

IR Indicator 4: Number of additional qualified teachers at the primary level, government schools

0 24,000 18,032 75

IR Indicator 5: Number of campus schools 416 1,500 1,350 86

IR Indicator 6: Number of schools with head teachers in place

4,062 4,900 6,299 267

IR Indicator 7: Number of schools with biometric teacher attendance monitoring system

0 40,000 42,383 106

Note: a. The actual achievement time line for SPR (KPI 1) is as of 2015/16; for all other indicators it is 2017/18.

35. School participation rates (KPI 1). The EOP target for school participation has been almost met as of 2015/16. SPR for primary, middle, and matriculate levels stood at 68 percent (EOP target: 69), 66.8 percent (EOP target: 67.70), and 55 percent (EOP target: 55.80), respectively. Compared to the baseline in 2012, SPRs increased by 3 percentage points for primary level, 1.2 percentage points for secondary, and 5.2 percentage points for matriculate levels over the three-year period. It is plausible that the positive trends in SPR exhibited up to 2015/16 continued till the EOP. For instance, administrative data from the SEF show a substantial increase in student enrollment in the government-assisted PPRS schools from 55,256 to 303,405 between 2013 and 2017, which has been directly supported by SESP II.

36. For triangulation of information, few other external evidences measuring various aspects of school participation show positive trends post 2015/16:

• First, the Annual Status of Education Report (ASER) national survey on rural households in Pakistan shows that NER among children ages 6–16 years in Sindh increased during the SESP II period—from 67.6 percent in 2012 to 86.1 percent in 2018 in the rural schools.13 Although the coverage is only rural schools, this indicates that enrollment increase in rural areas was substantial.

• Second, although the SEMIS data show a slight decline (by 0.6 percent) of enrollment in government schools over the project period, there has been an observed long-term enrollment shift from public to private schools before SESP II. While the share of student enrollment in the government primary schools declined on average by 1.8 percentage points annually between 2007/08 and 2012/13, the rate of decline slowed down during the SESP II period to 0.8 percentage points annually.14

• Third, student survival rate15 from Grade 1 to Grade 10 increased from 20.2 percent in 2012/13 to 29.7 percent in 2016/17 school years, indicating improved retention of students, substantial increase in

13 Includes both government and all other types of schools. 14 However, one challenge with the SEMIS data is that it does not collect information on private schools directly, but uses a rough prediction of enrollment growth based on 2002 private enrollment data, and hence, may not precisely capture the enrollment statistics. As private schools have remained much out of Government monitoring (with the exception of PPRS schools), there are no other available sources on the status of enrollments in these schools. 15 Percentage of students who enter Grade 1 reaching Grade 10.


Page 18 of 59

secondary school completers, and improved school participation in government schools. Annex 7 provides a table summary of these other measures.

37. Basic Quality Standard Schools (KPI 2). The number of the schools upgraded with needs-based facilities across seven criteria required essential for functionality of schools reached 1,366 in 2018 from a baseline16 of 992 in 2017, achieving 25 percent of the EOP target. This target was not met mainly due to the implementation challenges faced in absence of dedicated project leadership in the last year of project life. Nonetheless, this DLI activity helped upgrade 1,366 into functional schools with the adequate resources (including teachers, textbooks and learning materials, classrooms, furniture, and so on) improving quality of school environment and capacity to enroll more students.

38. Annual Retention Rate (KPI 3). The annual retention rate between Grade 1 and Grade 2 reached 87.5 percent in 2018 exceeding the EOP target of 82 percent and exhibiting an increase in student retention of 11.6 percentage points compared to the baseline in 2012. This indicates a significant achievement in reducing dropouts within the primary grades and raising primary school participation.

39. Annual Transition Rate from Grade 5 to Grade 6 (KPI 4). The EOP annual transition rate from Grade 5 to Grade 6 reached 66.8 percent in 2018, which is considerable gain of 10.3 percentage points compared to baseline in 2012 and surpassing the EOP target. The rise in transition rate between Grade 5 and Grade 6 indicates increased middle school participation among primary school completers, contributing to the objective of improving school participation in the province.

40. As noted in the PAD, improved sector governance, accountability, and strengthened administrative systems were important to raise the quality of education service delivery and motivate families in sending children to schools. The reasonable achievement of the above project outcomes and indicator targets was supported by satisfactory progress in key activities linked to 10 DLIs in the results chain. Out of 28 annual DLI targets, 21 were fully achieved, 4 partially achieved, and 3 not achieved. As described in the Theory of Change, the achieved DLI targets facilitated critical governance reforms at the provincial Education Department and the schools, which contributed to improve the quality of education service delivery through provision of school facilities and infrastructure (DLI 1, DLI 4, DLI 6, and DLI 7); increased teaching strength through introduction of merit and needs-based recruitment teacher recruitment policy and teachers’ professional development opportunities (DLI 10); improved school sector planning and management through head teacher staffing, strengthened SMCs, and ASC implementation (DLI 3, DLI 5,and DLI 9); and improved access to education in rural and underserved areas through strengthened PPPs in schooling (DLI 8).

41. Basic school facilities and infrastructure development under the project exhibited satisfactory progress contributing to the objective of improving school participation. SESP II financed need-based school infrastructure development in 284 schools, with 908 classrooms built, 1,566 washrooms constructed, and 316 water filters installed under the DLI-supported school infrastructure development activity (DLI 7). The project supported the establishment of 1,350 campus schools under the DLI-supported school consolidation activity (DLI 6). This allowed for better resource utilization and reduced basic maintenance expenditure as it enabled teacher rationalization and shared facilities and infrastructure, contributing to better-resourced schools. Additionally, fund allocations directly to schools through school-specific budgets (DLI 4), SMC grants (DLI 5), and school consolidation grants (DLI 6) financed teaching-learning materials, stationary, co-curricular activities, and community-identified small investments for

16 According to the Restructuring Paper, the baseline of 992 schools met 5 (out of the 7 criteria) and required to meet two more criteria on textbook delivery and furniture delivery to become fully functional schools.


Page 19 of 59

school improvements (such as classroom refurbishment, lab upgradation, washrooms, and so on).17 An additional 1,808 classrooms were built or rehabilitated against the project target of 2,400. The target for classroom built or rehabilitated was not met to the slow progress of the basic quality standard school activity during the last year of the project. Nonetheless, all these activities together upgraded 1,366 schools to functioning schools (DLI 11), contributing to better quality of school environment and making it possible to retain and enroll more students in fully functional government schools.

42. The PAD also highlights that low quality of schooling was one of the main reasons for families to be less likely send children to school. SESP II addressed quality challenges through DLI 10 which focused on teacher recruitment and teacher performance management. The project supported training of 18,032 government primary school teachers in the areas of lesson planning, teaching methodologies, and pedagogical content knowledge. New teacher training programs were developed and delivered based on the findings of SAT. The teacher training activities were particularly important as the low quality of teaching in primary levels was considered to contribute to high dropout in the schools. Despite the substantial progress, the project target for additional qualified primary school teachers could not be met mainly due to the slow progress of activities under the DLI on teacher management (DLI 10). As discussed earlier, the teacher recruitment and management policy is a highly political and challenging reform agenda in Pakistan, taking time to build consensus among stakeholders. A new teacher recruitment policy was developed and approved in March 2018 to institutionalize the system of merit and needs-based recruitment under a third party, with the first round of teacher recruitment initiated in November 2018 under the DLI support. Though the teacher recruitment could not be completed within the project life, the GoS has shown strong commitment to continue with the implementation of the activity with dedicated budget allocation under its education sector program for FY19.

43. Another important contributing factor to improving education delivery in the schools has been strengthening of school management and governance. The DLI on education management (DLI 9) helped fill 6,299 head teacher positions and recruit 957 head teachers through a new competitive mechanism conducted by a third party, addressing critical leadership and management gaps in these government schools. Under the DLI, job description and key performance indicators were developed and notified to track head teacher performance with the aim to improve school quality. Schools with the competitively selected head teachers were high performers in the latest SAT round, indicating improved management and quality of learning in these schools.18 The DLI on SMCs helped facilitate deeper community engagement in school management by bringing 33,493 government schools under the strengthened grant system. The DLI also supported the revision of granting fund formula from flat rate to composite rate-based school-level indicators, such as enrollment, classroom-student ratio, and school level. This improved equity among the schools and incentivized the SMCs in taking lead in increasing the enrollment in the schools. An independent impact evaluation found that intensifying community-led engagement through the SMCs has led to substantial gains in school access, primary school retention, and quality in rural Sindh (see study abstract in annex

17 One caveat is that the school-specific budget, though allocated fully to schools with technical support for non-salary expenditures, was not satisfactorily used. In 2013/14, only 12 percent of the total non-salary school budget was used and continued to remain low as documented in the Aide Memoires. The low fund utilization was mainly due to lack of financial authority of the principals/head teachers in primary, middle, and elementary schools to expend the funds, leading to a long approval process involving the district education office. By contrast, higher fund utilization rates were achieved for SMC grants, where the principals and SMCs (along with capacity building in planning, budgeting, and fiduciary management) were fully empowered to withdraw funds as and when needed, without seeking any authorization from education management authorities. 18 The SELD organized a recognition ceremony on January 28, 2019, for the competitively recruited headmasters, who were awarded certificates of excellence for “their highly admirable performance and hard work at their schools” (Source: SELD website).


Page 20 of 59

8).19 The DLI on the ASC published five rounds of ASC reports that generated useful school data for decision making among policy makers, school management, and other stakeholders. Moreover, school accountability improved due to the implementation of biometric teacher attendance monitoring system in 42,383 schools (against EOP target of 40,000) and monthly school monitoring, curbing teacher absenteeism and the prevalence of ‘ghost teachers’.20

44. A game-changing PPP scheme was introduced to foster educational provision in underserved areas. Through the PPRS Program of the SEF, an additional 804 PPRS schools were brought under Government support (565 of which were newly established) in 18 educationally disadvantaged districts and all PPRS schools brought under the Government’s recurrent budget (DLI 8), resulting in an enrollment increase by more than five-fold from 55,256 in 2012 to 303,405 in 2017.21 More importantly, the project introduced quality assurance and accountability framework in these private schools, with PPRS schools participating in SAT to track and report student learning outcomes as part of the performance agreement under the partnership. The quality of PPRS schools was further reinforced with additional grants from a separate Government scheme and matched private funds that equipped these schools with IT labs, science labs, and virtual classrooms. 22

45. To measure student achievement (Objective 2). Achievement of this objective is rated High. The achievement of this objective was measured through KPI 5 that tracked the cumulative number of SAT conducted (as shown in table 3).

Table 3. KPI 5

KPI Indicator Baseline Target Actual Achievement

Cumulative No. of Student Achievement Test (SAT) conducted

0 6 6 100%

46. Six rounds of SAT were conducted for mathematics, science, and language subjects in government schools across all districts in Sindh between academic years 2012/13 to 2017/18, with the last round involving participation of 319,000 students in 6,030 public schools.23 The student participation rate in SAT gradually increased over time to 62 percent and 39 percent for Grade 5 and Grade 8, respectively, in government schools in 2018.24 Based on the SAT results, the Sindh Education and Learning Department (SELD) placed recruitment of more qualified teachers for science, mathematics, and English as a priority under the new teacher recruitment policy. Additionally, a teacher training module was developed based on the SAT findings and training provided to 2,392 science and mathematics teachers (49 percent women) in primary, secondary, and higher secondary schools. Capacity building on monitoring learning achievement was also built. The activity supported schools to strengthen their own assessment systems that oriented teachers on objective assessment and to better measure student achievement on a regular basis. A test

19 Asim, Salman, and Amina Riaz. 2019. Community Engagement in Schools: Evidence from a Field Experiment in Pakistan (English). Washington, DC: World Bank Group. 20 Disciplinary action has been initiated against 40,000 absent teachers and 6,000 absconders by February 2017. 21 SEF administrative data. 22 Interviews with SEF officials suggests that the capacity developed in the schools under SESP and SESP II support were critical to attract more investments and enable more socioeconomically disadvantaged students, especially girls, to enroll. 23 SAT Report 2018. 24 Several factors which affected student participation, including unfavorable timing of assessment during peak harvesting season, lack of interest among students and parents, and inadequate classroom facilities in many schools creating an uncongenial environment for assessment. The SELD and RSU have taken steps to change the SAT administration timing and are working closely on building greater awareness on the importance of SAT among school community.


Page 21 of 59

item bank was developed for different grades and hosted on the SAT website, enabling teachers to access these resources for improving their own classroom assessments. The findings from each round of SAT were well disseminated through publication of annual reports on a dedicated website.25 School-specific performance was also disseminated through school profiles, while parents received individual student reports cards based on the SAT results during each round.

Contributing to Higher-Level Objective

47. Though not part of the SESP II development objective, an analysis based on the comparison of SAT scores between 2014 and 2018 using fixed effect regression shows that students’ learning has improved over the project period. Specifically, there has been improvement in the learning of science for Grades 5 and 8, mathematics in Grade 5, and language in Grade 8 for both boys and girls. Substantial progress was observed in the learning of science across gender and grades—the SAT scores for science have increased between 5 and 6 percentage points in 2018 compared to the level in 2014. For mathematics, there was large improvement for students of Grade 5 with an increase of about 4 percentage points, while for language, improvement for Class 8 for both genders is found between 5 and 6 percentage points; however, there has not been much change in learning of language for Grade 5 students. Annex 9 provides details of the analysis on the student learning improvement.

Justification of Overall Efficacy Rating

48. Based on the rating of Substantial for Objective 1 on raising school participation and the rating of High for Objective 2 on measuring student achievement, the overall achievement of the PDO is rated Substantial. The achievement of the objectives discussed earlier has been based on the quantitative and qualitative achievements documented by third-party reports, monitoring reports, and administrative data of the various project subprogram units.

C. EFFICIENCY

Assessment of Efficiency and Rating

49. Efficiency of the project investment is rated Substantial based on high economic returns and cost-efficient project design, which is offset by delayed implementation.

50. Economic analysis. The economic analysis at the appraisal indicated healthy high returns on investment for SESP II. At project appraisal, the net present value (NPV) of costs and benefits was estimated to be at PKR 1,557 (US$11) per child26 (benefit-cost ratio of 1.46) based on expected benefits from increases in secondary education completers and future wages. This ICR conducted a cost-benefit analysis using the same approach and with updated information on project costs, education profile, lifetime earnings profiles, and actual survival rates from Grade 1 to Grade 10. The NPV of SESP II at completion is calculated to be PKR 20,157 (US$143) per child (benefit-cost ratio of this case is 6.21), which is far greater than the base case at the appraisal. The large increase of net benefits compared

25 http://www.satsindh.net.pk/Downloads/Reports 26 US$1= PKR 140.87 as of April 1, 2019

http://www.satsindh.net.pk/Downloads/Reports


Page 22 of 59

to the base case is mainly contributed by a large increase in secondary school survival rates of girls. The detailed economic analysis is provided in annex 4.

51. Efficiency in design aspects. The partnership with private sector and specialized agency to support implementation of project activities boosted efficiency. The PPP with the SEF helped increase student enrollments and improve access to quality in underserved and hard-to-reach communities through establishment and management of public-assisted private schools. This was cost-effective with the Government’s per student per month subsidy at PKR 750 (US$5.32) under the partnership as compared to an average public spending of PKR 1,222 (US$8.67) to PKR 6,373 (US$45.2) per student per month under the regular government schooling system.27

52. Efficiency in implementation. The initial delay in project effectiveness by one year hampered the implementation progress of the teacher management activity, requiring an extension of the project by 18 months. Frequent turnover of project leadership, especially during the last year of the project, adversely affected the achievement of the DLIs on teacher management and basic school quality under the results-based component while several studies and assessments were not implemented under the TA component.

53. An amount of US$38 million in IDA funds (5 percent of total credit allocation) was cancelled at project closing for three unmet annual targets of the DLIs on teacher management and basic school quality and for uncommitted amounts of TA. The targets for teacher recruitment and the number of schools upgraded could not be achieved within the last year of the project life, but these activities continue to progress with dedicated budget allocation under SERP II. The SELD completed the recruitment test through a third party for 7,190 teachers based on need-based vacancy positions according to the revised recruitment policy along with the revised recruitment rules as of February 2019. Moreover, the basic school quality activity (DLI 11) was scaled up to a total of 4,560 schools under the tentative name of ‘model schools’ with dedicated budget allocation up to FY2019/2028.

D. JUSTIFICATION OF OVERALL OUTCOME RATING

54. As discussed in preceding sections (paragraph 27–30), as the changes in the outcome indicators reflect an improved measure of SESP II achievement and the changes in components were an expansion of activities, SESP II’s scope or ambition was not unfavorably affected following the restructuring in June 2017. Therefore, a split rating is not triggered based on the latest ICR guidance.

55. SESP II is rated ‘Satisfactory’ for its overall outcome rating based on High relevance, Substantial efficacy, and Substantial efficiency ratings.

27 World Bank. 2017. Pakistan Sindh. Public Expenditure Review. 28 As reported in Aide Memoire (March–April 2018)


Page 23 of 59

E. OTHER OUTCOMES AND IMPACTS (IF ANY)

Gender

56. SESP II targeted some aspects of gender outcomes in education primarily through the school infrastructure development and PPRS schools. In Pakistani communities, girls are particularly vulnerable to poor conditions of school sanitation and water facilities. Badly maintained toilets and lack of access to clean water are often a huge hindrance for girls and discourage them from attending schools regularly. The School Infrastructure Development program of the project contributed to improving school environment including sanitation facilities, common rooms, and water filters at most disadvantaged schools. Female students and parents expressed satisfaction over the provision of these facilities during beneficiary interviews.

57. As discussed in the PAD, the probability for school participation at upper grades declines faster for girls than boys in rural areas in Sindh. PPRS schools were established in underserved rural communities, where one of the criteria included localities with high gender disparity in primary school participation. The opening of these schools helped boost enrollments in these areas, especially among girls. SESP II encouraged female teacher recruitment in the schools with the aim to improve girls’ enrollment. The number of female teachers in government schools slightly increased from 44,350 in 2012 to 47,365 in 2017, with most female teachers recruited in secondary schools. This has likely contributed to higher gender parity in secondary schools—for every 100 boys, 62 girls are enrolled in primary schools and 69 girls in secondary schools.29

58. In terms of gender outcomes in enrollments, there was a decline in gender parity in primary education from 69 percent to 62.4 percent between 2012 and 2018 in government schools. While there has been no significant analysis on the reasons for this decline, one reason may be the shift of enrollments to private schooling. This trend has been specifically observed in the case of schooling among children ages 6–16 in rural Sindh. The ASER reports show the female share of total enrollment in private schools in rural Sindh increased from 36 percent in 2013 to 41 percent in 2018. On the other hand, female secondary school completion increased considerably following increased survival rates for girls from Grade 1 to Grade 10 in government schools.30 Specifically, the survival rates from Grade 1 to Grade 10 for girls increased from 15.6 percent to 25.7 percent between 2012/13 and 2016/17 school years. The SAT reports also discuss learning outcomes by gender, which has been critical for regular tracking of girls’ performance in mathematics, science, and language in the government schools.

Institutional Strengthening

59. Institutional strengthening was built into SESP II whereby the PDO of the project explicitly focused on improving governance and accountability and strengthening administrative systems. The project improved budget planning and management capacities at the Education Department and schools through reform activities (DLI 1 and DLI 4). The human resource system of the SELD was strengthened through introducing reforms that helped professionalize the education management cadre, implement merit and need-based teacher recruitment, and rationalize teacher placement in schools. Furthermore, SESP II has supported strengthening of the SELD, RSU, and district education offices through technical

29 Annual School Census 2017. 30 Annual School Census, various years.


Page 24 of 59

support, especially in improving implementation capacities for large scale activities such as school infrastructure development, ASC, and SATs.

Mobilizing Private Sector Financing

60. SESP II continued to support PPPs in school provision in rural and hard-to-reach communities with the SEF. Under SESP II, an additional 804 private schools were brought under the PPRS Program in 18 districts of Sindh and subsidy per child per month increased to PKR 750, which was originally PKR 350 under SERP I. The SEF complemented the subsidy with investments in establishing IT and science labs and mainstreaming quality assurance activities, including increasing M&E of schools, conducting and publishing student assessment tests, and strengthening implementation arrangements.

Poverty Reduction and Shared Prosperity

61. SESP II mainstreamed poverty reduction and shared prosperity into the design of its various activities. The school infrastructure development helped rehabilitate dilapidated schools covering all districts, including vulnerable districts and hazardous areas. PPRS schools were opened in rural and underserved localities in 18 districts of Sindh that rank poorly along three indicators: the size of the out-of-school children population (6–10 years), distance to the nearest primary school, and gender disparity in primary school participation. These activities under the project were critical in developing infrastructure and facilities for access to better quality education among underserved communities of Sindh.

Other Unintended Outcomes and Impacts

62. Improved efficiency in education sector monitoring. The project used TA for completing biometric registration of all government teachers, which significantly improved efficiency of monitoring teachers’ attendance under the biometric system across the government schools in the province. The use of a web-based system for data collection from secondary and higher secondary schools moreover supported systematic and timely collection of ASC data, which improved efficiency in managing the activity.

63. Development partner coordination. During project implementation, coordination with other DPs involved in SERP II was well maintained to ensure harmonization and effectiveness of efforts. Joint implementation support missions were fielded with the Global Partnership for Education (GPE) and other DPs, which ensured concerted efforts in the sector and lower transaction costs. In terms of operational coordination, while the GPE financed the setup of the biometric teacher attendance monitoring system, SESP II provided TA support in the implementation of the activity.


Page 25 of 59

II. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME

A. KEY FACTORS DURING PREPARATION

64. Well-aligned project objectives and well-informed choice of interventions. The choice of project objectives

was well aligned to the national and provincial sector education plans that prioritized improving school participation and measuring student performance. The design was further strengthened by ensuring synergy with the GoS’s SERP II, thereby advancing the GoS’s commitment to take up politically challenging reforms in the sector. The design was also informed by the lessons learned and experiences of interventions under SESP I. Several interventions under SESP I were scaled up and fine-tuned, including expanding access to schooling through PPPs, strengthening administration of the ASC and SAT and increasing community outreach and engagement. However, the PAD could have benefited from a more detailed description on how the governance-based interventions supported the objective of the project in improving school participation.

65. Appropriate choice of lending instrument. The choice of the lending instrument, a results-based Investment Policy Financing, consisting of a results-based component with DLIs and a TA component, was appropriate. As the project focused on politically unpopular reforms (such as teacher and school management recruitment, teacher rationalization, and school consolidation), DLIs were appropriate mechanism to incentivize the client for achieving results in these priority areas.

66. Clearly identified project risks and mitigation measures placed. SESP II included a comprehensive risks assessment of the project design and incorporated mitigation measures into the activity designs. Strategic use of TA in executing project activities (such as preparing and implementing school budgets and grants and conducting the ASC and SAT) was critical to overcome and develop implementation capacities within the Education Department and at the schools. The safeguards and fiduciary risks were also well addressed through detailed risk management plans.

67. Significant scope remained for better M&E design. The country and provincial contexts—where late entry into

schools is common among children—could have been factored when choosing the original PDO indicator of NER. The original Results Framework, including intermediate indicators, did not sufficiently cover all aspects of the project activities to better track reform outputs or outcomes. Both issues were later addressed through the second restructuring. As indicated in the PAD, the M&E reporting for SERP II was divided across the different entities. The updating of the Results Framework was assigned as the responsibility of the World Bank while the project’s activity-specific M&E responsibilities were assigned to respective subprogram units. A better integrated M&E arrangement, for example, through a dedicated project M&E Unit and mainstreaming of the Results Framework, would have been useful for the Government and the World Bank to ensure more coordinated reporting and evaluation of project performance as a whole. The inclusion of evaluations in the M&E plan during preparation would have been useful to assess the impacts of project reforms on outcomes.

B. KEY FACTORS DURING IMPLEMENTATION 68. Overall, all project activities were implemented successfully with the exception of Year 3 DLI on teacher

recruitment, DLI 11 on quality basic schools, and some studies and assessments planned under the TA. 69. Key factors that facilitated implementation include the following:


Page 26 of 59

• Strong Government commitment to achieving politically challenging reforms. The GoS successfully executed several politically challenging reforms. The GoS adopted education management reforms by separating the teaching and administrative process to ensure appropriate human resource management expertise in the schools, successfully contesting political resistance against the implementation of the new education manager cadre system. A new teacher recruitment policy was developed and approved in March 2018, which introduced the recruitment of teaching in all public schools on merit through a third-party firm to improve the quality of education across the province. Around 1,350 campus schools were established through the school consolidation efforts, which involved either closing or merging nonfunctional or nearby schools into a single campus for efficiently utilizing resources and improving school quality. The consolidation also implied a rationalization and redistribution of management roles, which is often met with high contestation among school management and staff.

• Strategic use of TA to strengthen implementation. The project used competitively selected local expertise to supplement SELD/RSU efforts in timely completing four rounds of the ASC and six rounds of SAT. Third-party verification also helped ensure the quality and reliability of the data.

• Continuous engagement with private sector and community to strengthen implementation. SESP II used the strength of the SEF’s extensive network of schools for expanding provision of education in rural and underserved communities in the province. During implementation, the project continued to engage the SEF effectively and hold it accountable for results by requiring submission of progress and student assessment reports. The expansion of SMCs in government schools, financial authority to SMCs in the implementation of SMC grants, and support services (including communication, awareness-raising, grievance management, and facilitation services) also supported community engagement in the school development process.

• Use of technology to improve efficiency in monitoring and data collection. SESP II developed and used a web-based system for more systematic and efficient data collection from secondary and higher secondary government schools, enabling the timely execution and publication of the ASC each year. The GoS has now taken the initiative to expand the web-based data entry system to primary schools, which is expected to reduce costs significantly. The GoS also used TA for completing biometric registration of all government teachers, which complemented GPE efforts in establishing the teachers’ biometric attendance monitoring system and significantly improved efficiency of monitoring teachers’ attendance across the government schools in the province.

70. Key factors that slowed project implementation include the following:

• Challenges with outcome indicator. Before the restructuring in 2017, the PDO rating was downgraded to Moderately Unsatisfactory reflecting the slow progress against NER targets as documented in the Implementation Status and Results Report (ISR) 8. PSLM 2015/16 survey showed a decline in NER, which at the time was the only KPI measuring the PDO of raising school participation. As discussed, the World Bank team investigated the reliability of the indicator given the country contexts of late school entry among children and agreed on the adoption of SPR throughout the board. The adoption of SPR using the same survey showed upward trend in school participation as of 2015/16. Alternate KPIs on retention and transition rates and the number of basic standard quality schools were included to track progress of school


Page 27 of 59

participation post 2015/16, which also showed improvements. Based on these improvements, the PDO rating was upgraded to Moderately Satisfactory in May 2018.

• Absence of fully committed project leadership, particularly during the last year. Following restructuring in 2017, the changes in leadership in the SELD and the RSU slowed the progress of implementation of project activities under the remaining two DLIs and the TA. The chief program manager position changed four times and the chief program manager who served during the last six months before closing took on the responsibility of SERP II as an additional charge. High turnover and absence of fully committed project leadership therefore made project implementation challenging in the last year with delays in decision making and procurement process.

• Frequent turnover of fiduciary staff. As noted in different Aide Memoires, frequent turnover and absence of fiduciary staff, especially in the case of a procurement specialist, at the RSU and SEF throughout the project life created delays in procurement activities and weakened fiduciary capacities, specifically in areas of contract management and asset management.

• Inadequate coordination to address emerging challenges quickly. The coordination between the Finance Division, SELD, subprogram-level implementation units, and district officers could have improved through more regular coordination meetings (as recommended in Aide Memoires) to seek progress of achievement, resolve delays, and track targets. Lack of effective coordination, often reflected in slow fund releases within the departments of the SELD, affected the pace of implementation of school infrastructure development program, especially during the first year of the activity.

III. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME

A. QUALITY OF MONITORING AND EVALUATION (M&E)

M&E Design

71. The M&E design of SERP II relied on the existing monitoring arrangements of the GoS under the parent program in addition to third-party assessments. The design of the monitoring system was at the school sector level and at the project level. At the school sector level, two key system monitoring activities under SESP II were included: (a) the ASC conducted by jointly by the SELD/RSU and third party TA and (b) SATs administered and verified by a third party. The sector-level monitoring systems, supported by DLIs, were important inputs to also track the progress of SESP II achievement, which is discussed later in the M&E utilization section. 72. At the project level, the M&E design involved (a) biometric teachers’ attendance monitoring, (b) monthly monitoring school visits conducted by district officers through Directorate General (DG) M&E Office, (c) administrative data collection by subprogram units, and (d) third-party assessments implemented to evaluate performance of activities and support DLI verification. The monitoring design revolved around the SELD/RSU, subprogram implementing units, DG M&E Unit, and the SEF program administrative data. The M&E Results Framework included five PDO-level indicators and seven intermediate indicators (after restructuring in 2017), which covered most aspects of the activities under the project. One caveat is that, although the choice of the PDO indicator of NER during project design was consistent with


Page 28 of 59

the SESP 2014–2018 monitoring framework, it was not context appropriate for measuring school participation. The issue was however addressed through a change in measurement to SPRs. The SPR indicator yet relied on the GoP’s national household survey, which could not be updated after 2015 because the survey did not occur regularly. For this purpose, new PDO indicators were added to complement tracking on the progress of school participation. Other indicators were to be updated based on the annual school census, which were conducted on a regular basis under the DLI and TA support of the project. 73. The M&E arrangement assigned monitoring and data collection responsibilities across subprogram activities without a mechanism to consolidate findings for regular tracking of the project performance as a whole. An integrated approach for M&E, such as through a dedicated project-level M&E Unit and mainstreaming of the Results Framework across project activities, would have been useful to better coordinate between different monitoring systems and better evaluate project performance during implementation. While third-party assessments under DLI activities validated the procedures of the activities implemented and the reliability of the data collection, the SERP II design could have nonetheless benefitted from additional assessments and evaluation on project activities. Though this was realized during the later stages of the project, and an ambitious plan for conducting several studies was included under TA during the second restructuring, these could not be completed within the remaining project time.

M&E Implementation

74. The M&E activities designed for the project were implemented satisfactorily. As part of school sector monitoring, the SELD and RSU successfully implemented (a) four rounds of the ASC conducted covering government primary, secondary, and higher secondary schools across all districts and (b) six rounds of SATs implemented with increasing student participation among Grade 5 and Grade 8 students in government and PPRS schools. At the project level, key M&E achievements include (a) monthly reports based on the school monitoring system, including biometric teachers’ attendance checks; (b) third-party verifications under project activities to ensure reliability of data and processes; and (c) third-party assessments that were also successfully conducted, as part of the project, to verify the achievement of various DLI-supported reforms. The M&E implementation could have benefitted from better coordination between the SELD, RSU, SESP II Project Unit, and subprogram-level units, especially with regard to school-level monitoring. Both SEMIS under the RSU and the school monitoring system of DG M&E under the SELD collected school-level data on similar parameters, leading to some duplication of supervision efforts in the sector.

M&E Utilization

75. Utilization of M&E data was actively pursued at the school sector and project levels during the implementation. Major achievements in terms of utilization of sector-level M&E include (a) dissemination of findings from six rounds of SAT to policy makers, schools, parents, and students through online published reports, school performance profiles, and student report cards; (b) dissemination of the ASC through published reports and school profiles based on census findings; (c) use of the ASC for identifying the needs for ‘basic quality schools’ during the restructuring design, and (d) use of SAT results to inform policy decision on teacher recruitment and teacher training in subject areas of Science, mathematics, and language. 76. At the project level, M&E data were used intensively in the following areas: (a) use of ASC results to identify school infrastructure needs and channel investments accordingly under the basic quality standard school activity (DLI 11); (b)


Page 29 of 59

use of DG M&E monthly report to track progress of the basic quality standard school activity; (c) utilization of SAT results to develop and implement teacher training module in science and mathematics; (d) use of teachers’ biometric attendance monitoring results to take official measures against high absentee teachers and ‘ghost teachers’; (e) use of study findings on viable and nonviable schools to inform the school consolidation activity; and (f) knowledge generated and lessons learned from SESP II used to inform further analytical work, including the Sindh Education Sector Plan (2019–2022) and the Education Sector Analysis.

Justification of Overall Rating of Quality of M&E

77. Based on the above discussions, the overall quality of M&E is rated Modest. It merits attention that the strong validation and verification process under DLI-supported project activities were useful to achieve reliability of data used for updating the results indicators and producing subprogram administrative reports, which have informed the ICR assessment of the achievement of project objectives. However, the main shortcoming has been in the M&E design and arrangement. Consideration of country contexts may have better informed the selection of the original PDO indicator of NER, which was nonetheless rectified by the team during the restructuring. The absence of evaluations in the M&E plan during project design was a shortfall—though later addressed during the restructuring of the TA component—the incompletion of the planned studies provided a missed opportunity for rigorous assessment and knowledge generation on the impacts of governance reforms on project objectives. The M&E arrangement could have benefitted from more integrated approach through better coordination across the different project entities in tracking the overall performance of SESP II activities.

B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE

78. Environment and social safeguards. SESP II was rated a Category B project in accordance with World Bank OP 4.01 (Environmental Assessment), given the school civil works, rehabilitation, and/or upgrading activities were planned for existing eligible government schools. The Environmental and Social Management Framework (ESMF) developed under SESP I was revised for SESP II and included mitigation measures, implementation and institutional arrangements, monitoring requirements, and supervision roles and responsibilities. An environmental coordinator was hired by the project to support the implementation of the ESMF; however, the position did not continue toward the end of the project.

79. SESP II undertook several important tasks under the ESMF, including (a) development of a GIS-based school hazard mapping of the province, (b) completed needs-based assessment for environmental-friendly school infrastructure development, (c) ensured provision of safe drinking water and sanitation facilities in upgraded schools, and (d) conducted master trainer training of 46 head teachers and teachers of government schools covering all districts on personal hygiene and disaster preparedness. At project closing, the World Bank’s environmental safeguard review found that the upgrading and rehabilitation work completed under the project has significantly improved the overall environment of schools with provision of basic infrastructure facilities. However, general operations and maintenance mechanism of facilities developed in some schools remain a concern, suggesting the need for the GoS’s continued field-level monitoring in this aspect. The absence of an environmental coordinator toward the project end and the inadequate O&M of facilities in some schools led to a rating of Moderately Satisfactory for safeguard compliance according to the last ISR.


Page 30 of 59

80. Financial management. The financial management was rated Moderately Satisfactory in the ISRs for most of the project life. Financial management capacity and practices of SESP II were adequate. Financial management function of project was mainly carried out by an experienced senior FM manager, supported by a finance associate, finance assistant, and a data processing assistant. Throughout the project life cycle, accounting records were found recent and up-to-date, and the RSU complied with reporting requirements of the Legal Agreement. External audit was conducted by the Auditor General of Pakistan through its DG Audit Sindh each year and the audited financial statements were submitted to the World Bank within six months of the end of fiscal year. However, delays in budget releases with consequent payment lags were observed during various points of the project life due to the overall delays in the process at the start of the fiscal year, which is a common challenge in the country.

81. Procurement. Procurement was rated Moderately Satisfactory in the ISRs for most of the project life. SESP II maintained the procurement arrangements under SESP I. Measures taken to strengthen procurement capacity at the project and field levels include (a) revamp of a Procurement Unit at the RSU with a procurement specialist, a contract management specialist, and a procurement assistant using project TA funds; (b) training for capacity building of the RSU procurement staff; (c) training for procurement manager and the SEF staff under the PPRS Program; and (d) procurement guidelines prepared and distributed to schools and districts. However, frequent turnover of procurement staff at the RSU and SEF throughout the project life created delays in procurement activities and weakened capacity of procurement and contract management. Despite these challenges, the last post procurement review did not find any major issue, despite staff turnovers and capacity constraints.

C. BANK PERFORMANCE

Quality at Entry

82. The PDO of the project remained highly relevant to development agenda of the country and province as well as the priorities of the World Bank throughout the project life cycle. The latest Sindh Education Sector Plan (2014–2018) highlights the importance of increasing school participation at primary, secondary, and higher secondary levels with set targets, especially through governance reforms and measuring student achievement to support human development in the province. The Sindh Right to Free and Compulsory Education Act 2013 also provided strong legislative framework for efforts to bring free and compulsory education for all children ages between 5 and 16 years.

83. The project was designed in synergy with the sector program (SERP II) of the GoS, garnering strong commitment of the Government to undertake politically challenging governance reforms in education (including education management, teacher recruitment, teacher rationalization, and school consolidation). The design also incorporated lessons learned from the previous World Bank-supported activities under SESP I, including strengthening school census and student assessments, enabling strategic partnerships with the private sector in provision of schooling, and engaging the community in the school development process. As most DLIs were designed to support governance reforms through improvements in processes and procedures, a gap often remained in reform execution and subsequent implementation at the field level. This was specifically in the case of school budgets and grants, where DLIs may have ensured allocation of budget to schools, but these allocations did not necessarily translate into satisfactory fund utilization. Considerations into the reform design regarding the financial power limitations and inadequate fiduciary and management capacities at school levels may have supported better reform implementation.


Page 31 of 59

84. The project clearly defined risks associated with the project along with appropriate mitigating measures, as well as safeguards and fiduciary related responsibilities. A detailed Environmental and Social Management Plan was prepared, with clearly defined roles and responsibilities, along with the provision of an environmental coordinator to support school infrastructure activity.

85. As discussed earlier, the design of the M&E arrangement may have benefitted from utilizing an integrated approach in performance tracking and reporting of the project. Third-party assessments and verifications helped validate processes and achievements of reforms, but more strategic use of studies and evaluations in the M&E design and plan would have been useful for assessment of the impacts and outcomes of project reforms. The original Results Framework and results indicators were not aligned adequately to the project activities, limiting the tracking and capturing of the reform outputs or outcomes. The original PDO for school participation faced difficulties, whereby consideration of country contexts would have been useful in the choice of indicator during project preparation.

Quality of Supervision

86. The quality of supervision by the World Bank team has been satisfactory, and the World Bank team proactively supported the project to find solutions to implementation issues. SESP II program officers noted that the World Bank team was easily accessible for consultations and the technical advice and suggestions from the World Bank were helpful in improving operational procedures and solving problems.

87. The World Bank team fielded implementation support missions in Karachi on a regular basis with a multidisciplinary team consisting of education, procurement, financial management, and environment and safeguards specialists. The minimal turnover of task team leaders contributed to consistency in project leadership and continuity of dialogue that was conducive to effective implementation. The right mix of expertise helped ensure that implementation challenges were addressed effectively, especially during the second restructuring. Most responsibility turnover among World Bank project team members occurred during the last two years. Interviews suggest that transition of responsibilities could have been better supported to sustain institutional memory of early project implementation experiences. With the Project Management Unit located in Karachi, the availability of more on-the-ground supervision support could have benefitted the team, especially with regular follow-up and reporting.

88. There were some delays in responding to the challenges with the PDO-level indicator on NER, which was first documented to be declining in the ISR in October 2015. The team addressed this issue through a restructuring in June 2017, whereby NER was replaced with a more appropriate indicator of SPR. During this restructuring, the Results Framework was also refined and made more comprehensive to include new outcome-based indicators. Several studies and evaluations were also included under the TA component to measure and document the outcomes of the governance reforms, which could not be completed within the remaining project life.

89. The Aide Memoires were detailed and candid with descriptions of the progress, challenges, and recommendations across each activity. Changes in ratings were frequent and used effectively in the ISRs to flag critical issues that required management attention and urgent action.


Page 32 of 59

Justification of Overall Rating of Bank Performance

90. Based on the above discussions, the World Bank performance is rated Moderately Satisfactory. The main shortcoming has been during project preparation, whereby technical issues remained with the design of DLIs on school budget and grants, as well as with the M&E arrangement.

D. RISK TO DEVELOPMENT OUTCOME

91. The risk to development outcomes is deemed Low to Moderate. The objectives to improve school participation and measure student performance continue to be development priority areas for the province and the country as indicated in the current legislation, policy, and plans. Most of the activities conducted under SESP II are continuing under the sector program of the GoS with some activities expanded. At the time of project completion, the SELD and RSU had already initiated the implementation of the ASC and SAT, while activities such as school infrastructure and basic school quality were expanded, demonstrating strong political commitment to continue with the development objectives. The GoS is now developing the next Education Sector Plan, with TA from the GPE, World Bank, and Local Education Group expected to support further interventions in achieving goals in school participation and student achievement.

IV. LESSONS AND RECOMMENDATIONS

92. A results-based lending instrument and specifically DLIs are highly effective in advancing the challenging reform agenda of governments. SESP II aligned with the sector reform program of the GoS and provided incentives through the use of DLIs that provided strong motivation to the Government in pursuing politically challenging reforms in areas of school management cadre, teacher recruitment and rationalization, and school consolidation. Feedback from the central stakeholders indicates that having DLIs did help in significant ways to build up consensus among different stakeholders on expected results and provide compulsion to Government officers to achieve the agreed results.

93. Strategic use of TA is useful in developing capacities and implementing education reforms. The TA used in the implementation of SAT and ASC were critical to ensure that the right expertise was available to improve the quality of the design and implementation of the activities for generating reliable and timely data. At the project level, TA was critical in supporting fiduciary management, especially in contract management, asset management, record keeping, and documentation, essential to effective project management. At the district and school levels, TA was placed strategically for capacity building of district education officers and school management in areas of planning, budgeting, and fiduciary management for effective implementation of reforms. However, measures should be taken to ensure that a good capacity development plan is also in place for the implementing agency with the aim that the system capacity would be enhanced, and the main agency would gradually become capable of taking over the responsibility, should that be appropriate.


Page 33 of 59

94. National assessment of student learning on a regular basis is a challenging task but a big step toward quality improvement, and efforts need to be ensured for effectively utilizing assessment results in promoting policy dialogue and public discussion. Conducting SAT annually by the SELD and RSU was challenging as it involved various stakeholders, ranging from the Education Department to the district offices to schools and to students and parents across the province. Proper dissemination of SAT results at all levels worked as a quality assurance mechanism. Official recognition by the Government of good performance of head teachers and schools based on SAT results is a good means to mobilize communities and the school management in lower-performing schools to strive for better outcomes. The SAT findings were also useful for the Government to inform policy and operations in teacher management. It would be important for the GoS to now go beyond measuring student achievement and maximize effectiveness of the SAT activity through effectively utilizing the assessment results to inform and generate public discussion and policy dialogue around strategies for improving student learning outcomes with time-bound targets.

95. Leveraging the capacity of the private sector through PPPs is a good strategy to expand access to schooling in rural and underserved communities, provided quality assurance and accountability systems are in place. The GoS was able to expand schooling in rural and underserved communities where public provision was low through expanding and strengthening the PPP with the SEF. Performance-based partnership linked to student assessments results is a useful means to monitor and track quality of education in these schools and hold the partnering agency accountable for results.

96. Digital tools can be an effective means to improve efficiency in monitoring and data collection. SESP II developed and used a web-based system for more systematic and efficient data collection from secondary and higher secondary government schools for timely completion of the ASC work each year. The system is now being expanded to primary schools, which would reduce implementation costs significantly. The biometric monitoring system of teachers’ attendance across government schools in the province was also a powerful digital tool used to overcome challenges of teacher absenteeism and ghost teachers. The power of digital tools needs to be further harnessed for efficient operations and governance of the education sector.

97. A robust integrated M&E design and plan are important to ensure effective monitoring and knowledge development around project interventions. While SESP II used several M&E activities to monitor program-level performance of various interventions, there was a need for a more harmonized approach in data collection across the various entities and to enhance utilization of data to support effective reporting of project-level activities. Strategic use of TA to support studies and evaluations in the M&E design and plan is helpful for rigorous assessment of the impacts and outcomes of reforms. In the choice of PDO and results indicator, country contexts remain a crucial element to factor, especially as seen in the case of NER.

98. While DLIs are effective in pushing forward challenging reforms, the development effectiveness of reforms may be undermined in contexts of inadequate implementation capacities, inconducive management structures, and inadequate communication. SESP II used DLIs that supported process-based improvements in fund allocation to government schools. While funds were allocated to government schools against DLI targets, the utilization of funds faced some challenges, especially in case of the school-specific budget activity, where governance regulations placed financial authorization with district education officers instead of school principals, leading to a prolonged bureaucratic approval process. Capacity building of school management, as earlier explained, was also critical for progressing implementation of school budgets and grants. A comprehensive communication plan on the purpose and advantages of the reforms being carried out for stakeholders at the implementation level, especially among district education offices and the schools, is important for effective implementation of education reforms.


Page 34 of 59

99. A whole school development approach is important for effectively affecting and raising the overall quality of schools. Implementation experience from the first three years of SESP II shows that separately designed activities for school development are less likely to be effective, particularly when implemented by different departments using different school selection criteria. In the absence of a systematic and consolidated investment approach to school development, efforts may inadvertently lead to partially developed and sporadically improved schools. To address this, the basic quality of standard school activity introduced during the restructuring of SESP II provides an important lesson that concerted and consolidated efforts across all areas—including infrastructure, teachers, management, equipment, textbook, and so on—placed within a school are important to significantly affect the school quality and ensure that the school is fully functional.

.

100. Effective project leadership remains central to implementation and success of operations, and therefore this aspect deserves more attention through high-level communication, if needed. Prolonged absence of effective project leadership, especially during the past year of the project implementation, adversely affected the implementation of teacher recruitment and basic quality school activities under SESP II. High-level project performance reviews and sustained communication among the Government, implementing agencies, project units, and the World Bank may be a means to bring greater attention to this issue.

.

.


Page 35 of 59

ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS

A. RESULTS INDICATORS A.1 PDO Indicators

Objective/Outcome: To raise school participation by improving sector governance and accountability

Indicator Name Unit of Measure Baseline Original Target Formally Revised

Target

Actual Achieved at Completion

School Participation Rate (SPR), primary level, ages 6-10 years

Percentage 65.10 69.00 68.00

30-Jun-2012 30-Jun-2017 30-Jun-2017

Comments (achievements against targets):

74 percent achieved against the EOP target


Target


School Participation Rate (SPR), matriculate level, ages 14-15 years

Percentage 49.80 55.80 55.00

30-Jun-2012 30-Jun-2017 30-Jun-2017


Page 36 of 59


87 percent achieved against EOP target


Target


School Participation Rate (SPR), middle level, ages 11-13 years

Percentage 65.60 67.70 66.80

30-Jun-2012 30-Jun-2017 30-Jun-2017




Target


Number of basic quality standard schools

Number 992.00 2500.00 1366.00

30-Jun-2017 31-Dec-2018 29-Nov-2018




Page 37 of 59


Target


Annual retention rate between grade 1 and 2

Percentage 76.00 82.00 87.60

18-Apr-2012 31-Dec-2018 31-Dec-2018




Target


Annual transition rate from grade 5 to 6.

Percentage 56.50 60.00 66.80

18-Apr-2012 31-Dec-2018 31-Dec-2018



Objective/Outcome: To measure student achievement


Target


Cumulative number of Number 0.00 6.00 6.00


Page 38 of 59

Student Achievement Test (SAT) conducted

18-Apr-2012 31-Dec-2018 22-May-2018



A.2 Intermediate Results Indicators

Component: Result-based Component


Target


Students benefiting from direct interventions to enhance learning

Number 0.00 8600000.00 8505806.00

01-Jul-2012 31-Dec-2018 31-Dec-2018

Students benefiting from direct interventions to enhance learning - Female

Number 0.00 3500000.00 3257827.00


99 percent achieved against the EOP target for total beneficiaries; 93 percent achieved against the EOP target for female beneficiaries.


Page 39 of 59


Target


Number of additional classrooms built or rehabilitated at the primary level, government schools

Number 0.00 2400.00 1808.00

01-Jul-2012 31-Dec-2018 31-Dec-2018


75 percent achieved against target.


Target


Gender Parity Index (GPI), primary level, government schools

Number 69.10 76.00 62.40

01-Jul-2012 31-Dec-2018 31-Dec-2018


- 97 percent achieved against EOP target


Page 40 of 59


Target


Number of additional qualified teachers at the primary level, government schools

Number 0.00 24000.00 18032.00

01-Jul-2012 31-Dec-2018 31-Dec-2018


75 percent achieved against the EOP target.


Target


Number of campus schools Number 416.00 1500.00 1350.00

30-Jun-2013 31-Dec-2018 03-May-2018


86 percent achieved against EOP target.


Target


Number of schools with head Number 4062.00 4900.00 6299.00


Page 41 of 59

teachers in place 30-Jun-2015 31-Dec-2018 03-May-2018



Component: Technical Assistance Component


Target


Number of schools with biometric teacher attendance monitoring system

Number 0.00 40000.00 42383.00

30-Jun-2012 31-Dec-2018 03-May-2018




Page 42 of 59

A. KEY OUTPUTS BY COMPONENT

Objective/Outcome 1: To raise school participation by improving sector governance and accountability and strengthening administrative systems

Outcome Indicators

1. School Participation Rates (at primary, middle, and matriculate levels) 2. Number of basic quality standard schools 3. Annual retention rate between grade 1 and 2

4. Annual transition rate from grade 5 to 6

Intermediate Results Indicators

1. Students benefitting from direct interventions to enhance learning 2. Number of additional classrooms built or rehabilitated at the primary level, government schools 3. Gender Parity Index (GPI), primary level, government schools 4. Number of additional qualified teachers at the primary level, government schools 5. Number of campus schools 6. Number of schools with head teacher in place

Key Outputs by Component (linked to the achievement of the Objective/Outcome 1)

1. 1,366 schools upgraded to meet quality standards and transform into functional schools 2. 1,808 additional classrooms built or rehabilitated at the primary level government schools 3. 18,032 additional qualified teachers at the primary level, government schools 4. 1,350 schools consolidated to support for efficient allocation and use of resources 5. 6,299 head teachers placed to strengthen school administration 6. Implemented biometric teacher attendance monitoring system to tackle teacher absenteeism


Page 43 of 59

Objective/Outcome 2: To measure student achievement

Outcome Indicators 1. Cumulative number of Student Achievement Test (SAT) conducted

Intermediate Results Indicators n.a.

Key Outputs by Component (linked to the achievement of the Objective/Outcome 2)

1. Six rounds of SAT completed, and results published 2. Teacher training module based on SAT results developed and implemented 3. A test bank developed and hosted online to support teachers in classroom assessments


Page 44 of 59

ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION

A. TASK TEAM MEMBERS

Name Role

Preparation

Umbreen Arif Task Team Leader

Dhushyanth Raju Task Team Leader

Uzma Sadaf Sr. Procurement Specialist(s)

Akram Abd El-Aziz Hussein El-Shorbagi Financial Management Specialist

Ernesto Sanchez-Triana Lead Enviromental Specilsit

Chaohua Zhang Lead Social Development Specialist

Luis Caetano Porto Miglino Enviroment Consultanr

Andaleeb Jahan Alam Social Consultant

Surendra K. Agarwal Education Consultant

Anwar Ali Bhatti Financial Analyst

Hanid Mukhtar Senior Economist

Amna W. Mir Program Assistant

Mohammad Khalid Khan Program Assistant

Chau-Ching Shen Senior Finance Officer

Mosuf Ali Enviroment Consultant

Eliana Carranza Young Profesional

Martin M. Serrano Senior Legal Counsel

Javaid Afzal Senior Enviromental Specialist

Saeeda Sabah Rashid Senior Financial Management Specialist

Khalid Bin Anjum Procurement Specialist

Luis C.P. Miglino Enviroment Consultant

Salman Asim Education Consultant

Andaleeb Alam Education Consultant

Quynh T. Nguyn Education Consultant

Aristeidis I Panou Legal Consultant

Lorenza De Icaza ET Consultant

Mariam Nusrat Adil ET Consultant

Malik Mirza Financial Management Consulatnt

Robin S. Horn Advisor/Peer Reviewer

Reema Nayar Sector Manager/ Peer Reviewer

Dilip Parajuli Education/QER Panelist

Naveed Hassan Naqvi Education/QER Panelist


Page 45 of 59

Supervision/ICR

Shinsaku Nomura, Umbreen Arif Task Team Leader(s)

Haider Raza Procurement Specialist(s)

Mirza Omer Baig Financial Management Specialist

Syed Farrukh Ansar Team Member

Nathalie Lahire Team Member

Rahat Jabeen Environmental Specialist

Ali Inam Team Member

Babar Naseem Khan Social Specialist

Maliha Umer Team Member

A. STAFF TIME AND COST

Stage of Project Cycle Staff Time and Cost

No. of staff weeks US$ (including travel and consultant costs)

Preparation

FY12 10.675 146,689.52

FY13 52.330 359,606.51

FY14 29.962 160,773.86

FY15 0 0.00

FY16 0 1.25

Total 92.97 667,071.14

Supervision/ICR

FY13 0 0.00

FY14 14.475 68,275.54

FY15 18.862 128,552.32

FY16 1.022 20,162.48

FY17 6.441 57,832.21

FY18 14.937 94,604.48

FY19 30.771 184,816.97

Total 86.51 554,244.00


Page 46 of 59

ANNEX 3. PROJECT COST BY COMPONENT

Components Amount at Approval

(SDR, millions) Actual at Project

Closing (SDR, millions) Percentage of

Approval

Results-based component: Financing of the Sindh Government's Second Sindh Education Sector Reform Program (SERP II)

255.00 226.64 89

Technical Assistance (TA) component: Financing of essential technical, advisory, capacity building support for SERP II

4.60 5.04 110

Total 259.6 231.6 89.2


Page 47 of 59

ANNEX 4. EFFICIENCY ANALYSIS

1. The net effects of SERP II are estimated as the expected real lifetime labor earnings to a representative child resulting from the effects of SERP II. This is estimated using present discounted value (PDV) method. The PDV of expected economic benefits of primary and secondary schooling is derived by estimating real lifetime labor earnings based on age-earning profiles and education profiles for 15–65-year-olds. The PDV of expected economic costs comprised both direct (public expenditures on public and primary schooling) and indirect (forgone labor earnings of children ages 10 and above) costs of attaining each school completion level.

Economic Analysis at Appraisal

2. Basic approach. The current and future economic benefits and costs incurred for a representative child age 6 (the official age for entering primary school in Grade 1 at appraisal) given a projected average school attainment profile of a representative child (16–20-year-olds) is estimated for two cases: (a) without SERP II, where the projected average school attainment profile is identical to the average school attainment profile over a recent period between 2004 and 2010 and (b) with SERP II, where essentially the projected average school attainment profile is adjusted by raising the annualized growth rate in the probability of completing secondary school by a factor of 20 percent (base case) for male-urban, male-rural, female-urban, and female-rural population groups.

3. Inputs and assumptions. The present values of key variables used in the cost-benefit analysis are estimated based on 2010/11 PSLM survey data (province-representative household survey) that covered 19,622 households in Sindh. Education expenditures and students in the Sindh public education system are obtained from GoS sources (2010/11 education expenditure). Private expenditures on schooling are obtained from 2007/08 PSLM survey data which covered 3,765 households in Sindh.

• Urbanization rate. The forecasted annualized urbanization rate is assumed to be equal to the annualized population urbanization rate over 2004/05–2010/11, which stood at 0.8 percent.

• Discount rate. The rate at which expected economic costs and benefits are discounted to the present period is assumed to be 10 percent. Pakistan deposit interest rates adjusted for inflation have remained above 10 percent over 2007/08 to 2010/11 and are expected to continue at this level over the long term.

• Survival probabilities. Estimates of age group-specific adult mortality rates for women and men ages 15–49 using Pakistan Demographic and Household Survey (2006/07) data are linearly extrapolated to impute survival rates for ages 50–64.

• Unpaid family workers. The ‘age, education level, and earnings’ profile of unpaid workers is artificially constructed by applying a reduction factor of 75 percent to the corresponding profiles for self-employed workers. For children ages 10 and above, a reduction factor of 50 percent is applied to unpaid child workers.


Page 48 of 59

• Real annual earnings. The forecasted annual growth rates in real annual earnings in wage and self-employment are assumed to be equal to the annualized growth in real annual earnings in the respective employment type over 2004/05–2010/11 for workers ages 15–64. The growth rate for self-employment earnings is applied to the constructed earnings in unpaid employment.

• Schooling length. For each school completion level, the length is assumed to be equal to the modal number of grades completed—for children ages 16 and above, the modal grades completed are 0, 5, and 10, corresponding to less than primary school completion, at most primary school completion, and at least secondary school completion, respectively.

• Secondary school completion rates. With SERP II, the forecasted annualized change in the probabilities of completing at least secondary school is 2.4 percentage, 1.5 percentage, 4.1 percentage, and 5.4 percentage points for the male-urban, female-urban, female-rural, and male-rural population groups, respectively.

4. Results. Under the base case, the NPV of costs and benefits is discounted at 10 percent and stood at PKR 1,557 (2010/11 PKR) per child at project appraisal.

Economic Reevaluation and Data Inputs

5. Effect of SERP II. Cost-benefit analysis is reevaluated with two scenarios: (a) Scenario 1: it is equivalent to the base case at the appraisal, which is 20 percent increase in forecasted growth rate in the probability of secondary school completion, and (b) Scenario 2: the annual growth rates in the probability of secondary completion are estimated based on the actual improvement of education survival rates from Grade 1 to Grade 10. At the end of the SERP II program period, compared to the pre-SERP II, the annualized growth rates of survival rates (Grades 1 to 10) are 10 percent for boys and 18 percent for girls. With assumption that 10 percent increase in secondary schooling survival rate would result in 3 percent of secondary school completion rate, the annualized increase in the probability of secondary completion would be 3 percent for boys and 6 percent for girls. Table 4.1 shows the secondary school probability for each case scenario.

Table 4.1. Secondary School Completion Rates

Without SERP II

Scenario 1: 20% Increase in Growth Rate of

Secondary Completion Probability

Scenario 2: Based on Actual Improvement of Survival Rate (Grades 1 to 10) Due to SERP II

Male urban 0.56 0.57 0.58

Male rural 0.28 0.28 0.31

Female urban 0.58 0.60 0.68

Female rural 0.11 0.12 0.13


Page 49 of 59

6. Data and inputs. The model is reevaluated using the most recent data. PSLM 2014/15 is used to calculate the lifetime earnings profiles and education profiles of the working population (ages 16–65) in 2013/14. Actual education expenditure and the number of students in Sindh Province in 2013/14 are derived from the Public Education Review 2017 (World Bank). Model parameters for urbanization rate, earning’s growth rate, and demographic share of male-urban, male-rural, female-urban, and female-rural population groups are also updated using PSLM 2014/15. All other model parameters remain identical to the ones at the appraisal.

7. Results. The NPV is estimated at PKR 3,096 per child, with the case of 20 percent increase in the growth rate of secondary completion probability, and PKR 20,157 per child with case of 3 percent and 6 percent annual increase in secondary completion probability.

Table 4.2. NPV of Benefits and Costs (in PKR at 2010/11 prices, 2013/14 prices)

Scenario Benefits Costs Net Benefits Benefit-Cost

Ratio

Base scenario: 20% increase in growth rate of secondary completion probability

At appraisal (2010/11 prices) 4,917 3,359 1,557 1.46

At completion (2013/14 prices) 6,494 3,398 3,096 1.91

Scenario 2: based on actual improvement of survival rate (Grades 1–10) due to SERP II

At completion (2013/14 prices) 24,024 3,867 20,157 6.21

Comparison

8. At completion, with the base scenario, the benefit-cost ratio of SERP II is 1.91, which is higher than the one at the appraisal (1.46). The difference is mainly attributed to the reduced per student real cost of SERP II; at the completion, the cost of the SERP II program amounts to 80 percent of original project cost at the appraisal. In addition, the team estimated the NPV of the benefits using the actual results from the improvement of survival rates over the SERP II period compared to the pre-SERP II. The benefit-cost ratio of this case is 6.21,31 which is far greater than the base case at the appraisal. The large increase of net benefits compared to the base case is mainly contributed by a large increase in secondary school survival rates of girls. Accounting for the social benefit coming from large improvement in girls’ education outcome, the total benefit of SERP II would be higher.

31 This is within a reasonable range of benefit-cost ratios for basic education program. For comparison, the benefit-cost ratio of secondary education quality improvement program in Afghanistan (completed in 2018) was estimated to be between 2.31 (worst scenario) and 7.4 (best scenario).


Page 50 of 59

ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS

NOT APPLICABLE


Page 51 of 59

ANNEX 6. SUPPORTING DOCUMENTS (IF ANY)

Documents

Strategies

World Bank. 2010. Country Partnership Strategy for Pakistan for FY10–13.

World Bank. 2014. Country Partnership Framework for Pakistan for FY15–20.

Project Documents

World Bank. 2013. Project Appraisal Document on a Proposed Credit for the Pakistan Second Sindh

Education Sector Project.

World Bank. 2017. Project Restructuring Paper of the Pakistan Second Sindh Education Sector Project

Aide Memoires

Aide Memoire for Implementation Support Mission (October 3–11, 2013)

Aide Memoire for Implementation Support Mission (February 3–7, 2014)

Aide Memoire for Implementation Support Mission (November 24–28, 2014)

Aide Memoire for Implementation Support Mission (May 11–20, 2015)

Aide Memoire for Implementation Support Mission (April 17–29, 2016)

Aide Memoire for Implementation Support Mission (March 14–15, 2018)

Aide Memoire for Implementation Support Mission (April 19–20, 2018)

Aide Memoire for Implementation Support Mission (November 12–29, 2018)

Implementation Status and Results (Archived Date)

ISR #1 (June 23, 2013)

ISR #2 (January 8, 2014)

ISR #3 (July 29, 2014)

ISR #4 (February 18, 2015)

ISR #5 (October 1, 2015)

ISR #6 (May 13, 2016)

ISR #7 (January 13, 2017)

ISR #8 (December 29, 2017)

ISR #9 (May 22, 2018)

ISR #10 (December 21, 2018)

Reports

Strategies

Government of Pakistan. 2013. National Five Year Plan 2013–2018. Ministry of Planning. Islamabad


Page 52 of 59

Government of Sindh. 2014. Sindh Education Sector Plan 2014–2018.

Policy

Government of Pakistan. National Education Policy 2017. Ministry of Federal Education and Training.

Islamabad

Project Reports

Government of Sindh. 2019. Project Completion Report for SESP II.

Institute of Business Administration. 2017. E-Assessment Report of Promoting Private Schooling in Rural

Sindh. RSU: Karachi.

SEF. 2014 and 2015. Promoting Private School in Rural Sindh. E-Marketing Report 2016.

SEF. 2016. Promoting Private School in Rural Sindh. E-Marketing Report 2016.

Safeguard Assessments

SELD. 2013. Environmental Impact Assessment II and Environmental and Social Management Framework

II. Reform Support Unit. Karachi.

SELD. 2015. School Needs Assessment Report 1–5. Reform Support Unit. Karachi.

SELD. 2015. Report on School Siting Manual 7 Guidelines. Reform Support Unit. Karachi.

SELD. 2015. Post Training Report on Training Seminar for Sindh Schools on Natural Hazard Preparedness

and Public Health and Hygiene. Reform Support Unit. Karachi.

Census and Surveys

Annual School Census

SELD. 2013. Sindh Education Profile 2012–2013. Reform Support Unit. Karachi





Student Achievement Test

SELD. 2013. Student Achievement Test 2012–13. Reform Support Unit. Karachi



SELD. 2016. Student Achievement Test 2015-16. Reform Support Unit. Karachi




Page 53 of 59

Analytical Studies

Asim, Salman and Amina Riaz. 2019. Community Engagement in Schools: Evidence from a Field

Experiment in Pakistan (English). Washington, DC: World Bank Group.

UNESCO-IIEP. 2018. Situation Analysis of the Education Sector in Sindh.

World Bank. 2017. Pakistan Sindh. Public Expenditure Review. World Bank: Washington, DC.


Page 54 of 59

ANNEX 7. OTHER MEASURES OF SCHOOL PARTICIPATION IN SINDH

2007/08 2012/13 2013/14 2014/15 2015/16 2016/17 2017/18 2018/19 Source

Government school enrollments (Grades 1–10) (in millions)

— 3.56 3.44 3.37 3.40 3.38 3.46 3.54 SEMIS

PPRS school enrollments

— 55,265 65,432 97,986 128,544 303,405

— — SEF

NER in rural Sindh for children ages 6–16 (%)

— 67.6 70.9 72.8 75.9 78 86 — ASER

Share of enrollments in government primary schools (Grades 1–5)

70 59 — — — 55 — — SEMIS

School survival rates Grades 1–10 in government schools (Total/Boys/Girls) (%)

— 20.2/24.3/15.6 — — — 29.7/32.7/25.7 — — SEMIS

Source: SEF administrative data. Note: The large increase in enrollments in PPRS schools observed between 2013 and 2017 was due to the addition of new schools: 318 schools in 2014, 247 schools in 2015, and 617 schools in 2017.


Page 55 of 59

ANNEX 8. SUMMARY OF IMPACT EVALUATION ON SCHOOL MANAGEMENT COMITTEES

The study presents findings from a field experiment in rural Sindh where half of school-age children (6–10 years) are out of school. The study tests alternative approaches to strengthen engagement of communities with schools through SMCs using: face-to-face dialogue at externally facilitated community meetings and ongoing, anonymous dialogue through text messages. The interventions increased communities’ interest in education as measured through an improvement in the number of functioning schools and, in the case of text message treatment, substantial gains in retention of students in Grades 2, 3, and 4. On the supply side, schools were able to significantly increase staffing and reduce the share of one-teacher schools; however, teacher absenteeism increased, and there was no substantial impact on basic school infrastructure. An additional reform to an existing institution, elections, and capacity building for school committees was implemented in a cross-over experimental design; the intervention undermined the participation of communities in meetings and reduced impacts on all indicators except new admissions and availability of toilets in schools. Overall, there is no evidence of an impact on measured test scores for any intervention. The findings suggest that community engagement can achieve some impacts on school access and quality, but even when communities are engaged with schools, there is a limit to what they can do to fill in the administrative gaps left by the state.


Page 56 of 59

ANNEX 9. STUDENT LEARNING OVER SESP II PERIOD

1. Based on SAT reports, table 9.1 provides a summary on the average score achieved across subjects and grades between 2014 and 2018. For Grade 5, the average score improved for mathematics and science, while there has not been much change in the average score for language. For Grade 8, the average score for science improved while for mathematics, it slightly declined in 2018 after some improvement in 2017. The average score in language for Grade 8 improved by 6 percentage points between 2014 and 2018.

Table 9.1. Average SAT Scores across Subjects and Grades

Class 5 Class 8

Variable Mean Score Mean Score

Language (% mark)

2014 SAT 33 40

2016 SAT 32 38

2018 SAT 33 46

Mathematics (% mark)

2014 SAT 18 18

2016 SAT 24 22

2018 SAT 22 16

Science (% mark)

2014 SAT 15 17

2016 SAT 23 25

2018 SAT 19 22

2. To assess whether students’ learning has improved over the project period, the comparison of SAT between 2014 (beginning of SESP II) and 2018 (end of SESP II) is conducted using fixed effect regression32 analysis, controlling for time-unvarying school-level information. The fixed effect regression analysis is repeated over classes (Classes 5 and 8), genders (boys and girls), and subjects (language, mathematics, and science). The indicator variable on 2018 is included, which captures the difference of SAT scores between 2014 and 2018.

3. Over the project period, there has been improvement, especially in the learning of science in Grades 5 and 8, mathematics in Grade 5, and the language in Grade 8 for both boys and girls. A significant improvement is observed for the learning of science across the gender and the grades: in 2018, compared to the level in 2014, the SAT scores for science have increased between 5 and 6 percentage points (see column 3 in table 9.2). For mathematics, there was large improvement in students of Grade 5, increase in about 4 percentage points, but, for students in Grade 8, there was a little decrease, about 2 percentage points (column 2). For language, improvement for Grade 8 for both genders is found, between 5 and 6

32 Results from Ordinary Least Squares also show a similar trend. The ICR provides the result using school-level fixed effect because fixed effect is more robust in that it captures any time-unvarying school-level effect, which is not relevant to SESP II or other programs that happened during the project period.


Page 57 of 59

percentage points; meanwhile, there is not much change in learning of language for Grade 5 students, showing a difference of plus and minus 1 percentage mark (Column 1).

Table 9.2. SAT Scores for Girls, School-Level Fixed Effect Regression

Language (Column 1) Mathematics (Column 2) Science (Column 3)

Class 5 Class 8 Class 5 Class 8 Class 5 Class 8

Difference between 2014 and 2018 −0.690*** 5.897*** 3.866*** −1.852*** 4.417*** 5.707***

(−6.37) (45.58) (57.08) (−28.27) (71.34) (87.27)

Constant 35.12*** 42.69*** 17.75*** 18.15*** 15.28*** 17.95***

(497.92) (476.82) (402.34) (400.40) (378.73) (396.67)

N 132,226 93,058 132,226 93,058 132,226 93,058

Table 9.3. SAT Scores for Boys, School-Level Fixed Effect Regression

Language (Column 1) Mathematics (Column 2) Science (Column 3)

Class 5 Class 8 Class 5 Class 8 Class 5 Class 8

Difference between 2014 and 2018 1.015*** 5.351*** 4.026*** −1.772*** 3.856*** 4.223***

(13.49) (48.34) (80.22) (−33.45) (89.60) (86.05)

Constant 31.65*** 39.05*** 18.28*** 17.29*** 15.10*** 16.59***

(620.08) (494.95) (536.70) (458.10) (517.05) (474.20)

N 242,997 138,613 242,997 138,613 242,997 138,613

Note: t statistics in parentheses; the results are significant at p < 0 .05 (*), p < 0.01 (**), p < 0.001 (***).


Page 58 of 59

ANNEX 10. SUMMARY OF BORROWER’S ICR

1. SERP II was aimed at revamping the education sector of Sindh Province. Each DLI was aimed to strengthen the governance and management aspect of education activities and strengthen the administrative system of the education sector. The PDO was to raise school participation by improving sector governance and accountability, strengthening administrative systems, and measuring student achievement. Based on the analysis of DLIs, the project has contributed significantly in developing the foundation for improved service delivery. The SELD, in alliance with the World Bank, worked to achieve the identified targets to improve the administrative systems and enrollment and measure students’ learning achievements. Apparently, the project was a success in terms of the completion of the deliverables because it laid the foundation to resolve the prevailing pertinacious problems in the education system.

2. The project enabled a series of reforms through DLI-supported activities, including (a) merit and need-based recruitment and training of headmasters/headmistresses (DLI 9) and teachers (DLI 10) at different levels through a competitive process; (b) teacher rationalization to cater to the needs of schools, reopening of the viable nonfunctional schools, and effective utilization of physical, human, and financial resources through consolidating the schools (DLI 6); (c) aiding of the schools through school-specific budget and SMC funds (DLI 4 and DLI 5); (d) provision of basic quality standards and missing facilities through school infrastructure development (DLI 7 and DLI 11); and (e) reduction in teacher absenteeism through the introduction of biometrics, establishment of direct communication channels with communities and beneficiaries through SMS and phone calls, and so on.

3. Although SERP II focused on improving service delivery by addressing the issues related to participation, operations, and governance and accordingly formulated the deliverables to overcome the trickle-down effects, the outcomes of reforms were not adequately captured due to lack of assessments. As the studies and evaluations planned under the TA could not be completed, the project post-completion scenario suffered in terms of documenting the impact of reform outcomes. Establishing an office of the DG (M&E) was a commendable step; however, the budgetary allocation with regard to the Service-Level Agreement did not allow for expanding its roles and activities. Similarly, there was a need to conduct evaluations on the trainings imparted to the teachers as well as for in-depth assessment of overtime outcomes in student learning using the six rounds of SAT that were conducted successfully.

4. The induction or the preservice training of the newly recruited teachers would be more beneficial if offered over a few months rather than a few weeks. This part of teachers training management requires attention in future operations.

5. Infrastructure development is an ongoing process. Apart from building new schools, the component of repair and maintenance is equally important. The repair and maintenance work is carried out through an Education Works Department with resource constraints as well as low capacity and expertise. In future, the repair and maintenance activity is required to be outsourced to a third party or an engineering firm that will act on the request of the concerned school, send its team, assess the work, and execute it accordingly. This will not only expedite the repair work but will improve the quality of work also. World Bank financing can support the repair and maintenance purpose. Likewise, the Education Department may place its regular budgetary allocation for repair and maintenance in the same


Page 59 of 59

imprest/escrow account or some other financial arrangement. This will result in a revolution with regard to maintaining school buildings across the province.

6. Alfred Lord Tennyson in one of his poems said, “Though much is taken much abides.” This line completely applies to SERP II. SERP II through the World Bank has accomplished significantly for the education sector of Sindh, but it has yet to do a lot in the areas ignored in SERP I & II. Early childhood education (ECE) is a component that was missed in both SERP I and SERP II. The Education Department is now focusing more on the ECE component; hence, creating access to ECE shall be the major initiative to increase the in-school enrollment. Besides, for the very first time, the ECE-trained teachers are employed by the Government. This calls for a thoughtful continuous professional development program for newly inducted ECE teachers. This will lead the SELD toward sustainable development that will have long-lasting effect on the children, teachers, and school system as well.

7. The World Bank has a great opportunity to contribute in the area of ECE, which is for the first time focused on the public sector schools in Sindh. The World Bank either through SERP III or otherwise may invest in ECE teacher training, by providing teaching aids in the schools where ECE is in operation. Besides, the World Bank has an equal opportunity to invest in nonformal education and the out-of-school children, which was not taken up in SERP II and not even considered as a priority. This is the right time for the World Bank to address these important areas through SERP III.

Implementation Completion and Results Report (ICR)...

Documents

Transcript of Implementation Completion and Results Report (ICR)...