World Bank Documentdocuments.worldbank.org/curated/en/... · National Development Planning Agency...

Document of

The World Bank FOR OFFICIAL USE ONLY

Report No: PP2940

INTERNATIONAL BANK FOR RECONSTRUCTION AND DEVELOPMENT AND/OR INTERNATIONAL DEVELOPMENT ASSOCIATION

PROJECT PAPER

ON A

PROPOSED GRANT

IN THE AMOUNT OF

US$ 1,348,000 EQUIVALENTS

TO THE

BURSA PENGETAHUAN KAWASAN TIMUR INDONESIA (BAKTI)

FOR A

INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY IN URBAN PRIMARY SCHOOLS (KIAT GURU URBAN PILOT)

FEBRUARY 2019

Social, Urban, Rural And Resilience Global Practice

East Asia And Pacific Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

CURRENCY EQUIVALENTS

(Exchange Rate Effective {January 18, 2018})

Currency Unit = Indonesian Rupiah

IDR 14,182 = US$1

FISCAL YEAR

July 1 - June 30

Regional Vice President: Victoria Kwakwa

Country Director: Rodrigo A. Chaves

Senior Global Practice Director: Ede Jorge Ijjasz-Vasquez

Practice Manager: Nina Bhatt

Task Team Leader(s): Dewi Susanti, Javier Luque

ABBREVIATIONS AND ACRONYMS

ASA BAPPENAS BETF CEM CPF CSC EAP GoI GP IE KGP1 KGP2 LSDTF MIS MoEC MoF MoVDAT NCT NGO PDO PforR POM PPM RETF SA SC SCT SLA TA TAS TNP2K TKG TPG UC USAID WB WDR

Advisory Services and Analytics National Development Planning Agency (Badan Perencanaan Pembangunan Nasional) Bank Executed Trust Fund Community Empowerment Mechanism Country Partnership Framework Community Score Card East Asia and Pacific Government of Indonesia Global Practice Impact Evaluation KIAT Guru Phase 1 (mid-2016 to March 2019) KIAT Guru Phase 2 (this Project) Local Service Delivery Trust Fund Management Information System Ministry of Education and Culture Ministry of Finance Ministry of Village, Disadvantaged Areas, and Transmigration National Coordination Team for KIAT Guru Non-Government Organization Project Development Objectives Program for Results Project Operational Manual Pay for Performance Mechanism Recipient Executed Trust Fund Service Agreement School Committee Sub-National Coordination Team for KIAT Guru Student Learning Assessment Technical Assistance Teacher Absence Survey National Team for Acceleration of Poverty Reduction, Secretariat Vice President of Indonesia (Tim Nasional Percepatan Penanggulangan Kemiskinan) Teacher Special Allowance (Tunjangan Khusus Guru) Teacher Professional Allowance (Tunjangan Profesi Guru) User Committee United States Agency for International Development World Bank World Development Report

The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY IN URBAN PRIMARY SCHOOLS (KIAT Guru Urban Pilot) (P167216)

Page 1 of 42

BASIC INFORMATION

Is this a regionally tagged project? Country (ies)

No

Financing Instrument Classification

Investment Project Financing Small Grants

[ ] Situations of Urgent Need or Assistance/or Capacity Constraints

[ ] Financial Intermediaries (FI)

[ ] Series of Projects (SOP)

OPS_BASI CINFO_TABLE _3

Approval Date Closing Date Environmental Assessment Category

25-Feb-2019 31-Dec-2019 B-Partial Assessment

Approval Authority Bank/IFC Collaboration

CD Decision No

Please Explain

Proposed Development Objective(s) The Project Development Objective (PDO) is to improve teacher presence and teacher service performance in pilot schools.

Components

Component Name Cost (USD Million)

Project Implementation 1.35

Note to Task Teams: The following sections are system generated and can only be edited online in the Portal.


Page 2 of 42

Organizations Borrower :

Bursa Pengetahuan Kawasan Timur Indonesia (BaKTI)

Implementing Agency : Yayasan BaKTI

PROJECT FINANCING DATA (US$, Millions)

SUMMARY-NewFin1

Total Project Cost 1.35

Total Financing 1.35

Financing Gap 0.00

DETAILS-NewFinEnh1

Non-World Bank Group Financing

Trust Funds 1.35

Indonesia - Program for Community Empowerment 1.35

Expected Disbursements (in USD Million)

Fiscal Year 2019 2020

Annual 0.74 0.61

Cumulative 0.74 1.35

INSTITUTIONAL DATA

Practice Area (Lead)

Social, Urban, Rural and Resilience Global Practice

Contributing Practice Areas

Education


Page 3 of 42

Private Capital Mobilized

No

Gender Tag Does the project plan to undertake any of the following? a. Analysis to identify Project-relevant gaps between males and females, especially in light of country gaps identified through SCD and CPF Yes b. Specific action(s) to address the gender gaps identified in (a) and/or to improve women or men's empowerment No c. Include Indicators in results framework to monitor outcomes from actions identified in (b) No

OVERALL RISK RATING

Risk Category Rating

Overall ⚫ Moderate

COMPLIANCE

Policy

Does the project depart from the CPF in content or in other significant respects?

[ ] Yes [✔] No

Does the project require any waivers of Bank policies?

[ ] Yes [✔] No

Safeguard Policies Triggered by the Project Yes No

Environmental Assessment OP/BP 4.01 ✔

Natural Habitats OP/BP 4.04 ✔


Page 4 of 42

Forests OP/BP 4.36 ✔

Pest Management OP 4.09 ✔

Physical Cultural Resources OP/BP 4.11 ✔

Indigenous Peoples OP/BP 4.10 ✔

Involuntary Resettlement OP/BP 4.12 ✔

Safety of Dams OP/BP 4.37 ✔

Projects on International Waterways OP/BP 7.50 ✔

Projects in Disputed Areas OP/BP 7.60 ✔

Legal Covenants

Conditions

PROJECT TEAM

Bank Staff

Name Role Specialization Unit

Dewi Susanti Team Leader(ADM Responsible)

GSU21

Javier Luque Team Leader Co-TTL GED02

Achmad Zacky Wasaraka Procurement Specialist(ADM Responsible)

GGOPG

Andy Chandra Firdana Procurement Specialist GGOPG

Unggul Suprayitno Financial Management Specialist(ADM Responsible)

GGOEA

Tatong Permana Anggrimulja

Financial Management Specialist

GWA02

Fajar Argo Djati Social Specialist(ADM Responsible)

GSU21

Krisnan Pitradjaja Isomartana

Environmental Specialist(ADM Responsible)

GENE1

Chatarina Ayu Widiarti Team Member GSU21

Daniel Seno Yusanto Team Member GSU21


Page 5 of 42

Dea Widyastuty Team Member EACIF

Evarist F. Baimu Counsel LEGES

Fazlania Zain Team Member GSU21

Gregorius Kelik Agus Endarso

Team Member Analytics GSU21

Lily Hoo Team Member Analytics GSU21

Megha Kapoor Team Member App & MIS GSU21

Noah Bunce Yarrow Team Member Education GED02

Noto Luciana Team Member EACIF

Petra Wiyakti Bodrogini Prakosa

Team Member App & MIS GSU21

Ria Nuri Dharmawan Counsel LEGES

Seble Berhanu Counsel LEGES

Vivianti Rambe Environmental Specialist GENE1

Yogana Prasta Team Member EACIF

Extended Team

Name Title Organization Location

Note to Task Teams: End of system generated content, document is editable from here.


Page 6 of 42

INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY IN URBAN PRIMARY SCHOOLS (KIAT Guru Urban Pilot)

TABLE OF CONTENTS

I. STRATEGIC CONTEXT................................................................................................................7

A. Country Context ............................................................................................................................ 7

B. Sectoral and Institutional Context ............................................................................................. 7

C. Higher Level Objectives to which the Project Contributes .................................................... 9

II. PROJECT DEVELOPMENT OBJECTIVES ................................................................................... 10

A. PDO ............................................................................................................................................... 10

B. Project Beneficiaries ................................................................................................................... 10

C. PDO-Level Results Indicators .................................................................................................... 10

III. PROJECT DESCRIPTION........................................................................................................... 10

A. Project Components ................................................................................................................... 10

B. Project Cost and Financing ........................................................................................................ 11

C. Project Intervention Design ...................................................................................................... 11

IV. IMPLEMENTATION ................................................................................................................. 17

A. Institutional and Implementation Arrangements ................................................................. 17

B. Results Monitoring and Evaluation ......................................................................................... 17

C. Sustainability ............................................................................................................................... 21

V. KEY RISKS ................................................................................................................................ 21

A. Overall Risk Rating and Explanation of Key Risks ................................................................. 21

VI. APPRAISAL SUMMARY ........................................................................................................... 23

A. Economic and Financial Analysis .............................................................................................. 23

B. Fiduciary ....................................................................................................................................... 24

C. Other Safeguard Policies (if applicable) .................................................................................. 24

D. World Bank Grievance Redress ................................................................................................ 26

ANNEX 1. RESULTS FRAMEWORK AND MONITORING ................................................................ 27

ANNEX 2. PROJECT INTERVENTION DESIGN RATIONALE ............................................................. 31

ANNEX 3: IMPLEMENTATION ARRANGEMENTS .......................................................................... 37


Page 7 of 42

I. STRATEGIC CONTEXT

A. Country Context

1. In the past decade, while Indonesia’s national poverty rate has fallen by more than half from 24 percent in

1999 to 10 percent in 2018 (BPS, 2018), its inequality has increased. “Indonesia’s Rising Divide,” a World Bank publication, noted that the pace of poverty reduction has slowed since 2012. The sustained period of growth and macroeconomic stability experienced over the past decade was accompanied by one of the fastest increases in inequality in the East Asia and Pacific (EAP) region. Over one-third of increase in inequality from 2002 to 2012 can be explained by where one is born and who one’s parents are. Among its key recommendations, the publication identified improving local service delivery in education as a potential solution to reduce inequality (World Bank, 2016).

2. The Government of Indonesia (GoI) recognizes the importance of education in reducing poverty and inequality and has introduced reforms and allocated a substantial amount of resources to increase education access and quality. The GoI budget has more than doubled in real terms since 2001, with education budget tripled during the same period. Since 2009, 20 percent of national and district government budgets have been allocated for education, as per Law 20 passed in 2003.

B. Sectoral and Institutional Context 3. While education access has improved, quality and equity remain a challenge. Recent international

assessments show that Indonesian student learning outcomes remain at the bottom rank of participating countries (WB, 2013; OECD 2016). Using the assumptions of improvement rates on PISA tests from 2003 and 2015, the “World Development Report 2018” (hereafter, “WDR 2018”) calculated that it would take Indonesia 48 years to achieve the current OECD average score in mathematics, and 73 years in reading if education practices do not change (WB, 2017).

4. GoI has significantly increased financing of teacher salaries and allowances, but this has not led to better

teacher performance or student learning outcomes. Half of the national education budget has been allocated for payment of close to three million teachers’ salaries and allowances (Figure 1), which in 2018 amounted to US$ 16.1 billion. Teachers have also received additional allowances depending on their status and assignment location. Certified teachers1 receive a Tunjangan Profesi Guru (teacher professional allowance, hereafter TPG), in the amount up to one times (100%) their base salary. In 2018, close to 1.9 million of elementary and secondary school teachers were TPG recipients, with an annual budget of US$ 5.6 billion. Those working in special areas (including remote) receive a Tunjangan Khusus Guru (teacher special allowance, hereafter TKG), also up to one times the base salary. In 2018, close to 69,000 elementary and secondary school teachers were TKG recipients, with an annual budget of US$ 183 million. Nonetheless, TKG recipients had a higher teacher absenteeism rate compared to non-TKG recipients in the same schools (SMERU, 2010). Similarly, the introduction of TPG did not increase student learning (de Ree et al, 2018). Table 1 shows GoI’s spending in 2018 on TKG, TPG, teachers’ salary and other allowances.

1 Teacher certification process requires that teachers hold an undergraduate degree, submits portfolio of their teaching experiences, and passes a competence test. Teachers are currently certified for life, with no recertification process in place.


Page 8 of 42

Figure 1. GoI spending on teachers’ salary and allowances, compared to total education budget from 2012 to 2017.

Table 1. GoI spending on teachers’ salary and allowances, TPG, and TKG in 2018

Teachers Income Teacher Special Allowance (TKG)

Teacher Professional Allowance (TPG)

Salary and Allowances

GoI Spending in 2018 US$ 183 million US$ 5.6 billion US$ 16.1 billion

5. GoI has recently begun putting greater emphasis on teacher performance to improve education quality. Their

current strategy moves away from input-based teacher allowance payments to performance-based payments. Recent legal and regulatory reforms have created significant opportunities to expand this approach. One important reform is the introduction of the Civil Servant Law (UU 5/2014), which recommends performance-based evaluation and performance-based incentives for civil servants. Since the Civil Servant Law was issued, it has not yet been adapted and applied to education service providers, the largest category of civil servants in Indonesia. Another reform is the Village Law (UU 6/2014), which transfers a substantial amount of resources to villages and empowers villages to pursue local development and community empowerment initiatives. The Village Law thereby provides the legal and regulatory framework that enables communities to directly support improvements in basic service delivery.

6. The Ministry of Education and Culture (MoEC), under the Directorate General for Teachers and Education

Personnel has also continuously attempted to reform teacher management. This includes improving teacher intake quality, tracking and increasing teacher competence, matching teacher competence with placement, improving teachers’ distribution, developing pre-service and on-the-job trainings and continuous professional development, diversifying teacher accountability measures and teacher performance evaluation methods, and making teacher allowances performance based. The World Bank is currently providing MoEC with Technical Assistance (TA) through the Programmatic Approach on “Supporting 12 years Quality Education for All” (P157380) and the “KIAT Guru: Improving Teacher Performance and Accountability”


Page 9 of 42

project (P159191), hereafter referred to as KIAT Guru Phase 1 (KGP1). 7. The KGP1 has been implemented since 2016 to test two mechanisms to improve teacher presence, teacher

service performance, and student learning outcomes. A Community Empowerment Mechanism (CEM) provides community members with an explicit role to monitor and evaluate teacher service performance and to ensure teacher accountability. There is also a Pay for Performance Mechanism (PPM), which links the payment of TKG with either teacher presence or teacher service quality. The efficacy of the two mechanisms has been tested by combining them into three intervention groups i.e. (1) CEM; (2) CEM + PPM based on teacher presence; and (3) CEM + PPM based on a broad measure of the quality of teacher service performance. KGP1 covers 203 primary schools in five disadvantaged districts, and it was implemented through a Recipient Executed Trust Fund (RETF) by Bursa Pengetahuan Kawasan Timur Indonesia (BaKTI), a national Non-Government Organization (NGO), with directions from the Steering Committee chaired by MoEC and the National Team for Acceleration of Poverty Reduction, under the Secretariat Vice President Office (TNP2K).

8. The World Bank conducted an Impact Evaluation (IE) to identify which KGP1 intervention was most effective

in achieving the outcome indicators. A total of 270 schools were randomly assigned into three intervention groups and compared to a control group. The IE analysis found statistically significant positive impacts. The CEM combined with the PPM based on teacher presence (“Group 2”) had the strongest positive effects on student learning outcomes in mathematics and Indonesian language (at 0.19 and 0.17 standard deviations respectively), as it increased the presence of TKG-recipient teachers in classrooms and improved parental involvement in meeting with teachers and in supervising learning at home (Gaduh, et al, forthcoming).

9. Based on findings and lessons learned from KGP1, GoI plans to expand the scope of implementation and

apply a similar intervention to make TPG performance-based. The five pilot district governments have also committed their own funding (US$ 700,000) in 2019 to sustain “Group 2” intervention in 68 schools, convert 135 schools in “Group 1” and “Group 3” to “Group 2” schools, and expand field implementation to 183 additional schools in the same five districts. In November 2018, MoEC requested Bank support to expand the scope as proposed by the five districts, and to test the mechanisms for implementation in secondary education and using TPG.

10. The KIAT Guru Phase 2 (KGP2) will provide evidence-based recommendations to inform the policies of MoEC,

National Planning Agency (BAPPENAS), and the Ministry of Finance (MoF) on government-led cost-effective up-scaling of a performance-based TKG and a performance-based TPG. The GoI plans to make TKG and TPG performance-based as national policy starting in 2020 and will consider the results of KGP2 as policy inputs. It is also anticipated that KGP2 will inform the design of mechanisms for adapting the Civil Servant Law for the education sector, and the utilization of village fund for community participation in improving basic education service delivery.

C. Higher Level Objectives to which the Project Contributes 11. The project aligns with GoI’s “frontline” approach to improve service delivery in its 2015-2019 National

Medium-term Development Plan. The approach is based on the following lessons: (a) fragmented reform efforts are likely to fail; (b) Indonesia’s size and diversity preclude the possibility of success through one-size-


Page 10 of 42

fits all solutions; and (c) local experimentation, adaptation, and knowledge-sharing. Specifically, the frontline approach aims to: (a) remove bottlenecks and constraints within the service delivery chain that prevent frontline service providers from doing their jobs; and (b) promote mutual accountability and joint problem-solving at the point of service delivery.

12. The World Bank’s Country Partnership Framework (CPF) emphasizes six engagement areas across two supportive beams. This project falls under the fourth engagement area on delivery of local services and is in line with the CPF’s strong focus on supporting local governments in efforts to establish performance-based fiscal transfer systems and strengthening the capacity of central government agencies to support and oversee the performance of local governments. KGP2 is a collaboration between the Social, Urban, Rural, and Resilience and Education GPs. KGP2 will also inform broader teacher management reforms supported through an upcoming World Bank operation with MoEC (Efficient Systems for Education Excellence and Equity Program - P163217), led by Education Global Practice (GP) using the Bank’s Program for Results (PforR) financing instrument. The PforR will integrate at least one Deliverable Linked Indicator (DLI) on teacher pay for performance.

II. PROJECT DEVELOPMENT OBJECTIVES

A. PDO

13. The Project Development Objective (PDO) is to improve teacher presence and teacher service performance

in pilot schools. B. Project Beneficiaries 14. The project beneficiaries in target schools are 386 school principals, 386 village heads, 549 village cadres,

and 1,755 village community members who will conduct performance evaluations for approximately 3,000 primary school teachers to potentially benefit 27,000 primary school students with improved learning activities.

C. PDO-Level Results Indicators 15. The PDO-level results indicators are:

i. Reduced percentage of teacher tardiness in participating schools. ii. Reduced percentage of teachers with unexcused absence in participating schools.

iii. Improved teacher performance score.

III. PROJECT DESCRIPTION

A. Project Components

16. There will be one project component, which will cover the activities below:

i. Technical Assistance (TA) for MoEC and five district governments to revise the regulatory umbrella to


Page 11 of 42

tie TKG with teacher presence. ii. TA for MoEC and five district governments to enhance the support for KGP1 schools and to conduct

implementation of KGP2 interventions in additional schools. iii. Develop training modules and provide capacity development trainings for stakeholders from 386

schools. iv. Develop, pilot, and provide capacity development training of upgraded KIAT App and MIS. v. Testing of mechanisms to expand KIAT Guru in up to 40 secondary schools and using TPG in urban

locations of the five districts. vi. Management and monitoring of project implementation.

B. Project Cost and Financing 17. Funding for the Project Components is proposed to come from the United States Agency for International

Development (USAID) through a Local Solutions to Development Single Donor Trust Fund.

Table 3. Project Cost and Financing

Category Description Amount of Grant (US$) % of expenditures to be financed

Goods, Non-consulting services, Consulting services, Training & workshop, and Incremental operational costs2

1,348,000 100%

TOTAL 1,348,000

C. Project Intervention Design 18. A key principle to the intervention design of KGP2 is the scalability of the mechanism for government-led

nation-wide policy implementation. KGP1 IE, qualitative research, and process monitoring attributed the success of the interventions to four key elements: (a) actively engaging external stakeholders in monitoring and evaluating teacher performance; (b) increasing parental involvement in learning; (c) keeping teacher performance evaluation to a few simple and objective indicators; and (d) paying teacher allowance based on objective performance indicator (Gaduh et al, forthcoming). This finding is in line with international evidence, including in Indonesia, which show that making teachers accountable to different groups of education stakeholders, as opposed to solely being accountable to higher level supervisors within the education system, can be effective in improving education service delivery (Pradhan et al, 2014; Brinkerhoff & Wetterberg, 2013; Joshi, 2013; Barr et al, 2012; Ringold et al, 2012; WB, 2004). In addition, KGP1 is also in line with key recommendations from the WDR 2018, which identifies the use of both pecuniary and nonpecuniary incentives to improve teachers’ motivations and align teaching with learning to improve student learning

2 Incremental operational costs include: local contractual support staff salaries (implementation support personnel); travel and other travel-related expenditures; equipment rental and maintenance; vehicle operation, maintenance and repair; office rental and maintenance; materials and supplies; utilities and communications expenses; consumables; bank charges; and advertising expenses.


Page 12 of 42

outcomes. These findings were also shared in “Growing Smarter,” the World Bank EAP Education Flagship Report (WB, 2018).

19. While GoI is encouraged by KGP1 results in improving learning outcomes, it seeks to test several improvements for government-led scalability through KGP2. These improvements include refinements to improve the simplicity, sustainability, and scalability of the KGP1 design. KGP1 was implemented by an NGO and involved a long process of Community Empowerment Mechanism (CEM) led by NGO facilitators. In addition, the Pay for Performance Mechanism (PPM) created an administrative burden for government officials. The section below will present first the proposed support for GoI to further study and sustain implementation in KGP1 schools, and second the rationale and intervention design for the KGP2.

20. International evidence shows that interventions found to be successful at a smaller scale and implemented by NGO pilots could not be sustained when later adopted by the government (Bold et al, 2018; Banerjee et al, 2017; Banerjee, Glennester & Duflo, 2008; Banerjee & Duflo, 2006; Duflo et al, 2012). Two reasons are often put forward. First, government workers face a different set of incentives and operational budgets compared to those working for NGO’s. This led to weaker implementation arrangements. Second, over a longer period of time, the stakeholders’ behavior may change, and they may learn how to work the system, the initial effectiveness of the interventions became muted. In the case of KIAT Guru, parents and community members may lose interest in the monthly teacher-evaluation meetings or reduce their efforts to support learning at home. Teachers could realize, perhaps, that they could use a portion of their allowance to incentivize parents and community members to provide good scores. In other words, the stakeholders will likely adjust their behaviors over time.

21. While little evidence exists on how to scale up successful education pilots, a few related literatures offers important lessons. As a pilot expands its scope, it is very important to continue evaluating the key mechanism behind the successful intervention and identify and implement more cost-effective variations that can be applied by the government staff who will scale the intervention. Initial support by external actors will likely be needed, but by design, this should be reduced over time, while continuing to monitor for administrative, institutional, and political challenges (Kerwin & Thornton, 2018; Banerjee et al, 2017; Muralidharan, 2017). KGP1 IE provided proof of concept for the efficacy of the intervention. GoI support for the implementation expansion in five districts provides a rare opportunity to identify more cost- effective implementation strategies.

22. The World Bank will implement an integrated Advisory Services and Analytics (ASA) produce to accompany the KGP2 implementation. The ASA firstly proposes to conduct a longer-term IE of KGP1, to identify if the impacts could be sustained over a longer period. KGP1 implementation was handed over from NGO to government at end 2017, with process monitoring until end 2018 indicating a strong sustainability. Analysis of KGP1 MIS trends between 2017 and 2018 indicate stability in several key indicators. Monthly meetings continued to happen in all the 203 treatment schools throughout 2018, with 88% of the schools conducting these in a timely manner (compared to 80% in 2017). The completed KGP1 IE measured outcomes of the NGO-led implementation. It found the strongest positive impacts for Group 2 intervention (CEM combined with the PPM based on teacher presence), with overall positive impacts for Group 1 intervention (CEM only)3.

3 Group 1, which implemented only the CEM without performance-based payment of teacher allowance, achieved statistically significant impacts on teacher presence in school, parental involvement, and language score. The IE


Page 13 of 42

The longer-term IE of KGP1 proposes to measure the outcomes in both Group 1 and Group 2, and compare them to the control group, after the NGO handed over the implementation to the government. A total of 203 KGP1 schools (control, Group 1, and Group 2) will be surveyed in the first half of 2019.

23. After the survey for the longer-term IE of KGP1 has been completed, 135 of Groups 1 and 3 KGP1 schools will be converted to Group 2. Through the KGP1, which will end in March 2019, BaKTI as the Grant Recipient provides TA for the GoI to revise the Ministerial and Head of District regulations, decrees, and technical guidelines so that the payment of TKG can be tied with teacher presence. The Grant Recipient also prepares capacity development training modules for the schools’ stakeholders, which will be disseminated through government-funded socialization events at the district and village levels. Key stakeholders from 68 KGP1 Group 2 schools will be selected to provide peer-to-peer mentoring and supports for neighboring 135 KGP1 Group 1 and Group 3 schools in the process of converting the interventions. Key stakeholders from all KGP1 schools will be selected to provide further peer-to-peer mentoring and supports for the expansion to 183 new schools that are nearby.

24. The KGP1 CEM consisted of three phases. The initial phase consisted of eight meetings to produce a bottom-up Service Agreement (SA) between teachers and User Committee4 (UC) and a Community Score Card (CSC), with five to eight service indicators for teachers to deliver, including teacher presence. In the implementation phase, the stakeholders reviewed the implementation of SA and the UC evaluated each teacher’s CSC monthly. During evaluation phase, at the end of every semester, a village-wide meeting was held to evaluate the SA, CSC, and the membership of the UC. A few stakeholders at the village level were trained to administer an adaptive Diagnostic Student Learning Assessment (Diagnostic SLA). Results of the Diagnostic SLA were shared during the evaluation meeting, to better informed the stakeholders on learning outcomes. For more details on the KGP1 CEM, see Annex 2.

25. The initial set up phase of KGP1 was led by NGO facilitators, each covered between five to six villages. In each village the NGO facilitators trained and provided a Village Cadre with on-the-job mentoring. Over time, during implementation phase, the Village Cadres took over the role of the NGO facilitators, and by end 2017, the NGO facilitators were discharged. There was clearly a lot of hand-holding in KGP1 that would be difficult to scale up through a government-led expansion.

26. Therefore, secondly, the ASA proposes to test two mechanisms for government-led implementation expansion to new schools: with and without project facilitators. As KGP2 will focus the PPM on teacher presence (in line with Group 2 of KGP1), various aspects of KGP1 implementation can be streamlined. The development of SA can be simplified by providing the stakeholders with a consolidated list of KGP1 indicators that have been analyzed as the most effective in improving learning outcomes. For each stakeholder, only three indicators will be agreed upon, with a total score of 100. To optimize government-led expansion, we

found no negative impact for Group 1. Given that many developing countries with teacher absenteeism problem may not be able to tie payment of teacher allowance with performance (due to strong teacher unions or other political economy challenges), Group 1 is a strong contender for potential adaptation in other contexts. Adding a survey of all Group 1 schools cost an additional US$ 33,000, a negligible amount compared to the quality gain for policy recommendation and knowledge production. 4 The UC consists of a minimum of 9 elected members, 3 community or religious leaders, and 6 parents from each grade level. At least half of the UC members should be female.


Page 14 of 42

propose two CEM mechanisms for KGP2 (Group A and Group B). The two intervention groups were developed based on consultations with MoEC and five district governments, and were presented at the KIAT Guru Steering Committee meeting on December 13, 2018.

27. The first mechanism, KGP2 Group A, will initially be implemented by NGO facilitator, and later handed over to village stakeholders. The NGO facilitator will identify in each village three village cadres who will be trained along with representatives from village government, school, parents, and community members. However, the NGO facilitator’s visits to the village will be limited to a maximum of three visits (instead of an average of 14 visits during the KGP1), with each covering seven villages (instead of five villages during the KGP1). The NGO facilitators will set up the UC, membership to which will now include the School Committee to formally link the UC with school governance. Three Village Cadres (instead of one during KGP1) will be appointed to reduce the risk of personnel turnover and improve sustainability of capacity building and peer-to-peer supports. Per KGP1 lessons learned, the Village Cadres will be selected based on consultations with community leaders and members, enacted with Village Head decree, and trained by the NGO facilitators. Like in KGP1, the UC will be established by a Village Head decree and its institutional arrangement will be under the village government (instead of with the school governance), who will provide supports and funding. KGP2 Group A aims at maintaining the key elements of KGP1, particularly on engaging external stakeholders in monitoring and evaluating teacher performance and increasing parental involvement in learning. In this mechanism, the Diagnostic SLA that was implemented as part of the KGP1 will be digitized into the upgraded KIAT App. Similar to KGP1, the Village Cadres will be trained to administer the Diagnostic SLA and share results with the stakeholders, this time at the beginning of the meeting where they will select the SA indicators, and also prior to the semi-annual evaluation meeting.

28. Meanwhile, KGP2 Group B will not be implemented by an NGO facilitator. Rather, the government will identify which of the consolidated list of KGP1 SA indicators should be adopted by the school’s stakeholders. The district government will conduct socialization meetings, inviting school principals and village heads. The latter will lead socialization at the school level and revitalize the School Committee (SC), in line with the MoEC Ministerial Regulation 75/ 2016. The selection criteria for the SC was influenced by KGP1 UC. The role and responsibilities of the KGP2 Revitalized SC will be expanded to include the roles of the KGP1 UC. Unlike in the KGP1, the KGP2 Revitalized SC will be established by the School Principal Decree and its institutional arrangement will be under the school governance. KGP2 Group B therefore represents a more classical top-down government decision making and implementation approach that can be implemented strictly under the education system. Local-level decision making will be limited to the specifications of the given service performance indicators and how much it should weight5. This intervention group will be the easiest for a national scale-up since it involves an implementation mechanism that is highly standardized. To enforce engagement of external stakeholders, the results of monthly evaluation and teacher presence verification will still need to be signed off by the Village Head and the School Committee Chair, before TKG based on teacher presence can be disbursed.

5 For example, if one of the three indicators is teacher provides additional lessons for students who fall behind in reading and math, the specification needs to make the indicator quantifiable by identifying how many lessons per week or per month need to be provided by a teacher. As the total weight for all three indicators will be 100, at the local level, the weight of each indicator should be agreed upon.


Page 15 of 42

Table 2. Comparison of CEM for KGP1, KGP2 Group A and KGP2 Group B

KGP1 CEM with Facilitator

KGP2 Group A CEM with Facilitator in key meetings (*)

KGP2 Group B CEM without Facilitator

Implemented by

NGO facilitators (14 meetings), hand over to Village Cadre

NGO facilitators (maximum of 3 meetings), hand over to 3 Village Cadres

School principal

Initial Phase Meetings

Village socialization Village socialization* School socialization

Three meetings (children, parents, teachers)

- -

Service Agreement (bottom up indicators)

*School stakeholders select three Service Agreement indicators from a consolidated list, and define the specifications and weights of the Teacher Score Card indicators

School stakeholders will be informed by a given three Service Agreement indicators (selected by district government from a consolidated list), and define the specifications and weights of the Teacher Score Card indicators

Setting up of User Committee

Setting up of User Committee*

Revitalizing School Committee

Community score card evaluation matrix

- -

Implementation Monthly meeting to discuss SA and evaluate CSC

Monthly meeting to discuss SA and evaluate CSC (with facilitator one time only*)

Monthly meeting to discuss SA and evaluate CSC

Evaluation Semi-annual meeting Semi-annual meeting Semi-annual meeting

29. The two proposed KGP2 mechanisms may bring some valid concerns on the scalability of Group A, and on the

effectiveness of Group B. The use of NGO facilitator for KGP2 Group A may bring up a question of scalability through government-led intervention. Lessons learned from implementation of Village Law in Indonesia indicates various challenges for government-managed community development facilitators. To address this issue, some community-driven and participatory development projects test mechanisms to engage with the private sector and NGOs. A recent Presidential Regulation 16/ 2018 enables the procurement of NGOs to conduct services for government works. Should KGP2 Group A find stronger impacts, this regulation allows government-led expansion with the support of NGO. A follow up to this would be to help government set up a procurement process to identify a pool of eligible NGOs along with a certification process for NGO facilitators to expand implementation at scale.

30. On the other hand, the top-down mechanism of Group B may not be effective enough to engage external


Page 16 of 42

stakeholders in improving learning. Group B mirrors a typical pay-for-performance scheme that have been implemented in other developing countries and found to improve student learning outcomes (Barrera-Osoria & Raju, 2017; Duflo et al, 2012; Muralidharan, 2012; Muralidharan & Sundararaman, 2011; Glewwe et al, 2010). However, a purely top-down version may not be as effective as participatory decision making (Barr et al, 2012; Brinkerhoff & Wetterberg, 2013; Joshi, 2013; WB, 2004). Together, these studies raise an important question on how government-led education service delivery improvement can be made more participatory and effective. To this end, and different from previous studies, KGP2 Group B design inserts a required social accountability mechanism, where Village Head and School Committee Chair sign off on teacher presence records which became the basis of payment for the teacher’s TKG. The technical guideline will assert that Village Heads send invitation for the monthly evaluation meetings to invite broader community participations. However, this approach will rely completely on the government’s established system. The Task Team will discuss with district governments on possibility to have Sub-district Heads and school supervisors involve in supervising implementation by making calls to Village Heads and school principals respectively to reinforce.

31. It should be noted that while all teachers in KGP2 Group A and Group B schools will be affected by the interventions, the TKG based on presence will only affect TKG-receiving teachers. Technical workshops will be conducted with MoEC to determine whether the formula for TKG based on presence need to be changed. During KGP1, the formula was developed based on government regulation for civil servants (Perka BKN 12/ 2016). Evaluation is based on a full day presence (no allowance cut), partial presence (cut by up to 1.5% daily), excused leaves (cut by 2%), and unexcused absences (cut by 5%). Teachers whose presence fall below 85% in a month do not receive their TKG at all.

32. The RETF for KGP2 will support government-led implementation of the two intervention groups in 183 primary schools in five districts. In addition, the RETF will upgrade the KIAT App and MIS, with implementation support from the Bank’s Task Team. The upgrade aims to digitize the monthly administration of teachers’ CSC scores and verify teacher presence, with all school-level clearance by principal and village-level approvals by the Village Head and UC/ SC to be completed through the KIAT App. The App will include a digitized version of the Diagnostic SLA for the Village Cadres and UCs in Group A to administer to randomly selected students. The upgraded MIS will compile village-level inputs from KIAT App and aggregate them at the district level, with web-based and phone-based dashboards for schools and district and national level governments to monitor progress, process TKG payment based on verified teacher presence, and analyze trends over time. The App will also provide a prognosis to inform TKG-recipient teachers of the amount of TKG due to them every month, based on their verified attendance. Both the KIAT App and MIS will also integrate the complaint handling and redress mechanism, whereby school- and village-level stakeholders can register their complaints through the KIAT App and monitor the progress directly, while the MIS system will compile and manage the handling of the complaints, with automatic notifications to a higher-level government staff if the complaints are not resolved according to agreed procedure. The KIAT App and MIS will need to have an offline modality, as some of the schools have no or unstable telecommunication signal. The KIAT App and MIS ideally should be linked with the DAPODIK system, which is utilized by MoEC to collect and administer school and teacher data. In addition, possibility of linking the App and MIS with the village-level data system will be explored with the Ministry of Home Affairs and Ministry of Villages, Disadvantaged Areas, and Transmigration.


Page 17 of 42

IV. IMPLEMENTATION

A. Institutional and Implementation Arrangements 33. A Steering Committee, also known as National Coordination Team (NCT), was established for the KGP1. The

NCT is chaired and co-chaired by Echelons 1 at MoEC and TNP2K respectively. The NCT members consist of the BAPPENAS, MoF, Ministry of Civil Servants and Bureaucracy Reform, Ministry of Home Affairs, and Ministry of Village, Disadvantaged Areas and Transmigration. Sub-National Coordination Teams (SCT) were established in each of the five pilot districts, under the leadership of the District Head. The roles of the NCT and SCTs were instrumental in: (i) the issuance of national and district government regulations that enabled community participation in improving education service delivery, (ii) appointing schools for expansion, (iii) making the performance-based payment of TKG, (iv) conducting project monitoring, and (v) developing policy recommendations at the end of the project. The NCT and SCTs will be continued for KGP2.

34. The World Bank has selected an NGO, BaKTI, as a Grant Recipient to implement this activity. The Task Team considers the NGO option as the better candidate compared to a government recipient. Providing the grant to MoEC will require longer and more complicated processing time that will not meet the required timeline. BaKTI was selected through a limited selection procedure, where eight NGOs were invited to attend an information session on the project, and to submit a proposal on providing information on managerial and operational experiences and other administrative requirements. Four NGOs submitted proposals, of which only two met the selection criteria. The two NGOs were invited to attend a discussion with the Task Team members, in which a set of the same questions were given, ranging from overall project management, fiduciary management, and procurement of personnel and services. The detailed selection process and the rationale for selecting BaKTI is in the attached “Worksheet for Selection of an NGO as a Recipient of Grant,” as part of the Appraisal Package for the RETF. Both MoEC and TNP2K have provided No Objection Letters to the selection of BaKTI as Grant Recipient. The Grant Recipient will continue to conduct capacity development for MoEC and district government officials involved in the project. During KGP1, there was a continuous support to MOEC and local governments to implement the pilot, which led to the development of local capacity. The RETF will provide support to the MOEC and districts to sustain the implementation of KGP1 and expansion to KGP2.

35. The World Bank will implement the ASA, covering intervention and evaluation designs, impact evaluation, and policy recommendations. Core survey teams will be hired directly by the World Bank, to provide quality assurance of survey works and data collection and conduct training of enumerators and qualitative researchers. In addition, a consulting firm will be procured through a limited bidding process to contract the enumerators, administer training events, and collect data during field missions. Since the timeline for implementing the surveys in early 2019 is tight, contracting of data collectors may be conducted directly by the Bank.

B. Results Monitoring and Evaluation 36. The theory of change that informs the design of KGP2 is slightly modified based on evidence from KGP1. In

the current system, the TKG is not tied with any teacher performance measure. KIAT Guru hypothesizes that if TKG is tied with teacher presence, parents are engaged in education, and broader education stakeholders


Page 18 of 42

are committed to improve student learning through collective action, student learning outcomes will improve. During KGP1, monthly meetings between teachers and external stakeholders (village government, community leaders, and community members) generated pressure for teachers to be more disciplined and perform better on service agreement indicators set with community representatives, and for parents to get more involved in supervising their children’s learning at home. Figure 2 describes the theory of change on how the activities and outputs will lead to desired outcomes and impact.

Figure 2. KGP2 Theory of Change

37. The project monitoring, learning, and evaluation design is developed based on this theory of change. Through

the ASA, the World Bank will collect data through surveys on the KGP1 and KGP2 schools and the district characteristics in intervention and control schools. For both studies, data will be collected to identify impact on teacher presence, teacher service performance, parental involvement, parental satisfaction on education, and student learning outcomes. The following survey instruments from KGP1 will continue be used: Teacher Absence Survey (TAS), Student Learning Assessment (SLA), and questionnaires administered to key stakeholders at the village level (see Annex 2 paragraph 73 for details). However, the questionnaires will be reduced to the ones proven essential for analysis during KGP1, while adding those related to administrative, institutional, or governance issues and challenges for government-led CEM and PPM. A total of 203 KGP1 schools – consisting of Group 1, Group 2, and control schools – will be surveyed in the first half of 2019.

38. The ASA for KGP2 will compare Group A, Group B and a control group. The main outcome indicators and the

survey instruments will be the same as KGP1. The ASA will also evaluate the cost-effectiveness of each intervention arm. A total of 204 schools will be surveyed, with a baseline in the first half of 2019 and the


Page 19 of 42

endline in the first half of 20206. The 204 schools will be randomly selected from a list of eligible schools7, and randomly assign them to 3 groups (2 treatments and 1 control). The two treatment groups (136 schools) will be implemented using the district government funding. Since the government have funding for 183 schools, they will select additional 47 schools, which will be randomly divided into two treatments. In these 47 schools, IE surveys will not be conducted, but process monitoring will be conducted to collect data.

39. For both KGP1 and KGP2, the IE and process monitoring aim to respond to the following research questions:

a. What are the effects of each intervention group on teacher presence, teacher service performance, parental efforts and student learning outcomes?

b. Which factors contribute to the effectiveness/ impact of the intervention groups? i. What are the incentives and constraints to utilizing the tools, instruments, and mechanisms

provided to evaluate teachers? ii. How do the shortlisting and decision-making mechanisms correlate with implementation of each

intervention group? iii. How can the interventions be simplified for potential nation-wide implementation by the

government? 40. The baseline and endline surveys for KGP2 IE will be analyzed to: (a) verify that the characteristics of schools

and teachers, and ex-ante outcome variables, are balanced between intervention and control groups; (b) identify the intervention effects on teacher presence, teacher service performance, parental efforts, and student learning outcomes; and (c) identify factors influencing effectiveness of the intervention groups. The process monitoring will complement the baseline and endline surveys to identify mechanism and heterogeneous treatment effects as mentioned above.

41. For both the conversion and the expansion, the RETF Grant Recipient will conduct spot checks to monitor

implementation by the districts/ municipalities and the schools and write monthly reports to document the implementation processes. In addition, the process monitoring for KGP2 will focus on: a. For both Group A and Group B: SA indicators selected and weights agreed, meetings conducted and

attendees (number, gender), monthly evaluation results as recorded through KIAT App and MIS, and quarterly payment data for all TKG recipients.

b. For all schools in five districts: annual data on teacher deployment, teacher allowances, school funds and expenditures.

Section III.C. and the Results Framework in Annex 1 define the main outcome indicators and how the data will be collected.

42. Progress reports will be presented to the Steering Committee8 towards the end of every calendar year. To

6 The timing of the endline survey is currently constrained by the end date of the LSDTF. On-going discussion with USAID aims to propose a No Cost Extension by one-and-a-half year, when ideally the endline survey can be implemented during to the first half of 2021 instead. 7 Eligible schools are those located in very remote villages per government’s regulation, with a minimum of three teachers receiving TKG allowance. To reduce the cost of survey, villages that are too remote will not be included. 8 The Steering Committee/ National Coordination Team (NCT) will be chaired and co-chaired by Echelon 1 at MoEC and TNP2K

respectively. The members consist of the BAPPENAS, MoF, Ministry of Civil Servants and Bureaucracy Reform, Ministry of Home Affairs, and Ministry of Village, Disadvantaged Areas and Transmigration. See Section IV.A. for Institutional and Implementation Arrangements.


Page 20 of 42

this end, while data is collected on the longer-term impacts of the interventions, the progress report will present analysis of data collected, progress on implementation and outputs, and lessons learned and findings. These progress reports will be used by the Steering Committee for immediate policy inputs and by RETF for improving on-going program implementation.

Figure 3. KGP2 Timeline

43. Policy recommendation by the World Bank will be presented to the government at crucial junctures for policy inputs and budgeting cycle. The KGP1 longer-term survey will be conducted between March and April 2019. Quick analysis from this survey will feed into the design and implementation of KGP2 (scheduled to start in mid-2019). More careful analysis will feed into stronger policy recommendation to strengthen institutional mechanism and governance of KIAT Guru approach and will be presented to the new cabinet in October 2019. Final KGP1 IE report and policy brief will be disseminated in December 2019, along with process monitoring results of KGP2. Should the new cabinet support the scaling up of KIAT Guru in 2021 to other disadvantaged regions, to secondary education level, or using the TPG, this timing will allow enough time for government budget allocation.

11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12

Regulation & National WorkshopRegulations issuance TKG TKG & TPG

National workshops

Conversion of KGP1 SchoolsRevision of KIAT App and MIS

Socialization and training for converted schools

Monthly meetings KGP1 formula KGP2 formula independent by district government

Payment of TKG based on presence

Expansion to KGP2 SchoolsSchool selection and randomization

IE surveys

Socialization at the district level

Socialization at the village level

Training of school stakeholders

Initial set up meeting

Monthly meetings independent by district government

Payment of TKG based on presence

Process monitoring

Testing of mechanisms in secondary

level and using TPGDiscussion on TPG formula and mechanism

Testing of mechanisms in schools

Recommedation and revision

Implementation of performance-based TPG independent by MoEC

Payment of TPG based on performance

Process monitoring

MoEC & district government

MoEC & district government (independent)

RETF by BaKTI

BETF by World Bank

2019 2020


Page 21 of 42

44. The project and evaluation timeline for KGP2 is currently constrained by the end of the two Trust Funds, which financing need to be completed by end December 2019 (see Figure 3). The KGP2 endline survey is currently planned for implementation in February and March 2020, just a little over half-a-year after KGP2 will start implementation. A request for a No Cost Extension is under consideration by USAID, to add one-and-a-half year so the endline can be implemented in early 2021 instead. The No Cost Extension will be signed only if sufficient funds are available for the Bank to adequately manage the activities. Initial findings on the effectiveness of the two expansion models will be presented by mid of the year, with final policy recommendation and policy brief to be published by end of the year. For both KGP1 and KGP2, data on teachers, teacher performance, students, and student learning outcomes in the participating schools will be disaggregated by gender.

C. Sustainability 45. The conversion of all 203 KGP1 schools to Group 2 intervention and expansion of this intervention to new

183 KGP2 schools will be government-led implementation with limited support from the Grant Recipient. By the end of 2019, the five district governments will have implemented KIAT Guru in 51% (386 of 750) of very disadvantaged villages in these districts. The Grant Recipient will assist the district governments to develop medium- and long-term plans to expand coverage to all schools in very disadvantaged villages. Based on KGP2 IE and process monitoring, the World Bank will provide GoI with policy recommendation on which expansion mechanism is the most effective in improving learning outcomes. The policy recommendations will include suggestions to revise the technical and implementation guidelines that detail the roles and responsibilities of government officials and school stakeholders. Further support for TA in scaling up the mechanism to secondary level and using TPG will be provided per government requests.

46. GoI is also considering expanding the KIAT Guru approach to other disadvantaged districts. In addition to MoEC, the Directorate General for Disadvantaged Areas from the Ministry of Village, Disadvantaged Areas, and Transmigration (MoVDAT) is keen to champion the expansion of KIAT Guru to 117 other disadvantaged districts in Indonesia. In November 2018, MoVDAT invited 5 KIAT Guru district governments to share their experiences with 13 other disadvantaged districts. The KGP2 results will also inform GoI on how the approach could be adopted and expanded by these other districts.

V. KEY RISKS

A. Overall Risk Rating and Explanation of Key Risks

Table 4. Risk Matrix and Ratings

Risk Categories Rating

Political and Governance High

Macroeconomic Low

Sector Strategies and Policies Moderate

Technical Design of Project Low

Institutional Capacity for Implementation and Sustainability Moderate

Fiduciary Moderate


Page 22 of 42

Environment and Social Low

Stakeholders Moderate

Overall Moderate

47. The Task Team accords a “Moderate” risk rating for this project.

a. Political and Governance and Sector Strategies and Policies. The project can only be implemented with

national and district governments issuing regulations to tie TKG with teacher presence. With the upcoming presidential election in April 2019, there is a lot of uncertainty within MoEC on the future direction of teacher pay for performance, as the implementation of this policy will be contingent upon the result of the election. To manage this potential risk, in December 2018, the Executive Secretary of TNP2K sent a letter to the Coordinating Ministry for Human Development, and the Ministers of MoEC, MoF, BAPPENAS to inform them about KIAT Guru and the results of KGP1. The five district governments also have strong ownership of the project, as proven by the funding they have allocated to sustain KGP1 and expand it to KGP2 schools, which needs to be disbursed in 2019. As such, the Grant Recipient, through KGP1, will work with the national- and district-level governments, so that the regulations needed to tie TKG with teacher presence can be issued prior to the presidential election. During the cabinet changes after the election, KGP2 will be implemented under the leadership of the district governments, and once the reshuffling has been completed, the policy recommendations for the national government will be ready. In addition, the government may also use the findings of the IE in new context without sufficient evidence of effectiveness. To mitigate this risk, upon completion of the endline survey, the government will be provided with a clear description on the specific conditions to be applied to ensure that the proposed modalities will work at scale up.

b. Institutional capacity for implementation and sustainability. Since the implementation of the project is dependent largely on the leadership, capacity and buy-in from the district governments’ staff and schools’ stakeholders, variations in the quality of implementation are expected to be high. The Grant Recipient will provide capacity building trainings for all the KGP1 and KGP2 schools. District governments will conduct quarterly meetings to analyze teacher performance reports and identify schools that may need additional supports from the school supervisors. The TA will support districts to identify lagging schools, and support lagging districts. The Grant Recipient will also assign several full-time staff, to be stationed at and hosted by the district government offices, to provide on-the-job mentoring and TA for the district government officials throughout the timeline of the project. In two districts, Manggarai Timur and Manggarai Barat, the governments have also appointed staff whose specific job descriptions will be to coordinate and administer the project’s expansion. The Grant Recipient will advocate for the other three districts to appoint staff specific for the project, so that expansion to the rest of disadvantaged villages in the districts can be achieved and sustained.

c. Stakeholders. In our understanding, MoEC’s Director General for Teachers and Education Personnel has discussed the plan to tie TPG with performance with PGRI, the biggest and most conservative teacher union. However, precisely because this is a presidential election year, MoEC has decided to expand KGP2 within the current five districts and keep a low profile. We agree with this approach, as the KGP2 will provide more robust evidence and enlarge the scope of the pilot, while testing mechanisms for scaling. The TKG is currently paid unconditionally to teachers located in very disadvantaged villages. TKG-


Page 23 of 42

recipient teachers in intervention schools will be subjected to performance-based TKG payment and will likely consider this treatment as unfair. However, the five district governments now have the experience to manage possible complaints from teachers, and they have also collaborated with village governments for the expansions. Lessons from KGP1 show that community involvement provides a balancing force to potential rejections by the teachers. The dissemination of World Bank’s policy recommendations as previously described are scheduled to provide timely evidence to support MoEC’s plan to scale performance-based teacher pay.

VI. APPRAISAL SUMMARY

A. Economic and Financial Analysis

48. The budget for KGP2 is developed based on KGP1, adjusted to standard GoI costs (Standar Biaya Umum/ SBU) for 2018. As of December 2018, it is estimated that MoEC will allocate a minimum of IDR 35 billion (USD 2.5 million) for TKG in 386 schools to be tied with teacher presence in 2019. The five pilot districts have committed IDR 3.65 billion (USD 260,000) to pay for implementation activities at the district level. The village governments will allocate IDR 6.25 billion (USD 450,000) for implementation activities at the village level.

49. Every dollar spent on KGP2 implementation is matched by 2.6 dollars of GoI investment (compared to 0.95 dollars for KGP1). The cost of intervention per village is around IDR 47 million (USD 3,390) per year, one third of the cost for KGP1 (at USD 9,000), and 5.9% of village fund allocation in one year. This one-off investment cost is needed only at the initial phase of the intervention in a new village (to introduce the mechanisms and develop capacity for the stakeholders) and cover monthly meetings expenses for one year. After the first year, the cost of maintaining the activity will drop to IDR 17.8 million (USD 1,270), or a mere 2.2% of village fund. Figure 3 summarizes the total costs compared to the costs per village, with the first two years covering KGP1 implementation, the third year for KGP2 implementation, the fourth year for district and village governments to sustain the activities, and the fifth year for village governments to continue independently. Figure 4. Total cost compared to cost per village over five years


Page 24 of 42

B. Fiduciary

50. The Bank carried out a Financial Management (FM) assessment in accordance with BP/BD for IPF on November 10, 2017. Overall, the financial management risk is assessed as moderate. With the implementation of the agreed action plan, the proposed financial arrangements will satisfy the Bank’s minimum requirements and are adequate to provide, with reasonable assurance, accurate and timely information of the grant required by the Bank. See Annex 3 for details.

51. The proposed grant is mainly expected to include technical assistance to support implementation of the project through hiring of various consulting services assignments. For operational of the grant, it is also anticipated there will be procurement of small value goods and non-consulting services. Procurement under the proposed grant shall be carried out under the World Bank’s Procurement Framework in accordance with the Procurement Regulations for IPF Borrowers dated July 2016 (revised November 2017 and August 2018) and the provisions of the Financing Agreement.

52. The procurement capacity assessment identified that the proposed Grant Recipient, BaKTI, has adequate experience in carrying out procurement under previous Bank-financed projects however there was lacking resources assigned as designated procurement staff to efficiently manage procurement of the expected contracts identified under the current Bank-financed project.

53. As has also been agreed by the Bank in the case of the previous and current projects executed by BaKTI (i.e. Barefoot Engineers Wave III, MELAYANI, and KGP1), BaKTI may use its own procurement manual and standard procurement forms, which have been found to be in broad consistency with the Bank's Procurement and Consultants Guidelines. Further improvement may be made to ensure consistency with the Bank’s Procurement Regulations apply to this Project. An initial draft Procurement Plan, identifying the proposed activities, method, and estimated cost and review requirements has to be agreed with the Bank as priority before the signing of the proposed grant, which shall be processed through the Bank’s online procurement planning and tracking tool i.e. STEP (Systematic Tracking of Exchanges in Procurement) – also applied to the KGP1. As part of the project preparation BaKTI will be required to prepare the Project Procurement Strategy for Development (PPSD) in accordance with these regulations which will facilitate preparation of the procurement plan. Based on which the assigned procurement risk rating is considered as “Moderate.”

C. Other Safeguard Policies (if applicable)

54. Environmental (including Safeguards). The environmental risks are low. Following the advice provided by the RSS, the project is categorized as Category B due to the triggering of the World Bank OP 4.10 on Indigenous Peoples (OP 4.10 is triggered as a precautionary approach as per below explanation on social safeguards).

55. As an overarching umbrella policy, OP/BP 4.01 on Environmental Assessment is triggered to lay out key safeguard principles and measures necessary to manage and mitigate potential adverse environmental and social impacts, especially in addressing OP 4.10. Since the project will not finance any physical/ civil work activities nor support TA that will lead to future physical/ civil work activities or other activities with environmental implications, no stand-alone environmental assessment or environmental risk mitigation


Page 25 of 42

instrument is required. The activities supported by the project are expected to result in positive outcomes, which include, among others, improved teacher presence and teacher service performance. The previous Project Operational Manual (POM) for KGP1 to establish screening procedures and negative lists will remain relevant for activities financed under KGP2. Technical guidelines developed under RETF component to be implemented by schools will be reviewed by the Bank’s Safeguards Specialist. BaKTI has previous experience with the Bank’s project through KGP1 and has developed safeguards procedures in line with the Bank’s safeguards and good international practices. These include Occupational, Health and Safety (OHS), child protections, consultations and community engagement. The ToR for BaKTI will be reviewed to ensure that the above requirements are adequately resourced. Periodic review will be carried out during implementation to ensure that the project’s safeguards management remains functioning and relevant to identify and address emerging risks. Project facilitators will receive training support on safeguard related aspects. All TA activities, recommendations, and advice, will be consistent with the objectives of World Bank environmental and social safeguard policies.

56. Social (including Safeguards). The overall social risk for the project is rated as low. The project triggers OP 4.10 due to the presence of community groups with Indigenous Peoples’ criteria as per the policy in the target locations. No adverse impacts on these communities are envisaged. A similar approach under KGP1 for community engagement and consultations to secure broad community support will continue to be adopted. Furthermore, the project will ensure measures are in place to ensure that such consultations are inclusive and accessible, particularly to vulnerable groups in the target communities. Existing procedures for community engagement and consultations in the POM will be revisited to better reflect feasibility as well as challenges on the ground based on lessons-learnt from KGP1. It is expected that when the MoEC and target district governments undertake broader engagement for KGP2, civil society organizations including groups representing Indigenous Peoples will be consulted in line with OP 4.10.

57. There are no anticipated changes in safeguards risks, similar to KGP1 where such risks were assessed as low (EA Category C). The project is expected to generate positive social impacts by enhancing teachers’ performance and accountability through increasing community and parental involvement in student learning and providing oversight. Taking stock of the previous KGP1 implementation, key risks are often associated with TKG payments (and deduction), lack of stakeholders’ buy-in and understanding of the project’s objectives, human resource constraints to ensure oversight and communication gaps between parents (user committees) and teachers and principals (service providers) in deciding performance scores, which occasionally resulted in disputes. There are no safeguards risks under the BETF Advisory Services and Analytics (ASA P167281) that is being prepared in parallel to complement the RETF.

58. While previous safeguards implementation has demonstrated a positive track record and capacities are adequate within BaKTI, an additional scope and gradual phase out of external facilitation and hand-holding support to enable government-led transition and scale up may increase the likelihood that risks may not be managed in a cautious manner as it is currently being provided. The existing institutional capacities to manage risks will be revisited on a regular basis during project implementation, particularly in light of addressing emerging risks. Under KGP1, a Complaint Handling System (CHS) has been developed using an Short Message Service (SMS) gateway platform, which is being operated by BaKTI. The CHS has been used to identify systemic issues to enable coordination and troubleshooting. To complement the CHS, further facilitation to enable proactive identification of emerging issues and complaints will be embedded under the project’s facilitation design. It is expected that the same CHS will be grandfathered to support KGP2 and


Page 26 of 42

further improvement to integrate the CHS into the upgraded KIAT App and MIS will be conducted during project implementation. Such efforts to upgrade and streamline the existing CHS is expected to enable systematic tracking of complaints and an automated process to dispatch relevant issues to the district governments’ high-ranking officers with relevant mandates if no resolution can be reached. Progress reports provided by BaKTI to the World Bank will document the status of all complaints received.

D. World Bank Grievance Redress

Communities and individuals who believe that they are adversely affected by a World Bank (WB) supported project may submit complaints to existing project-level grievance redress mechanisms or the WB’s Grievance Redress Service (GRS). The GRS ensures that complaints received are promptly reviewed in order to address project-related concerns. Project affected communities and individuals may submit their complaint to the WB’s independent Inspection Panel which determines whether harm occurred, or could occur, as a result of WB non-compliance with its policies and procedures. Complaints may be submitted at any time after concerns have been brought directly to the World Bank's attention, and Bank Management has been given an opportunity to respond. For information on how to submit complaints to the World Bank’s corporate Grievance Redress Service (GRS), please visit http://www.worldbank.org/en/projects-operations/products-

and-services/grievance-redress-service. For information on how to submit complaints to the World Bank Inspection Panel, please visit www.inspectionpanel.org.

.

http://www.worldbank.org/en/projects-operations/products-and-services/grievance-redress-service

http://www.worldbank.org/en/projects-operations/products-and-services/grievance-redress-service

http://www.inspectionpanel.org/


Page 27 of 42

ANNEX 1. RESULTS FRAMEWORK AND MONITORING

Results Framework

COUNTRY : Indonesia INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY (KIAT Guru) Phase 2

Project Development Objectives

The Project Development Objective (PDO) is to improve teacher presence and teacher service performance in pilot schools.

Project Development Objective Indicators

Indicator Name Corporate Unit of Measure

Baseline End Target Frequency Data Source / Methodology

Responsibility for Data Collection

Name: Reduced percentage of teacher tardiness

Percentage

4.00 2.00 Monthly

Project MIS

Grant Recipient, District Government

Description: Teacher tardiness as recorded in KIAT App and verified by UC/ SC. During KGP1, percentage at baseline was 5. This value is assumed for KGP2, to be confirmed after baseline survey has been implemented.

Name: Reduced percentage of teachers with unexcused absence

Percentage

15.00 5.00 Monthly

Project MIS


Note to Task Teams: The following sections are system generated and can only be edited online in the Portal.


Page 28 of 42




Description: Teacher performance indicators from Service Agreement, evaluated by UC/ SC. During KGP1, percentage at baseline was 20. This value is assumed for KGP2, to be confirmed after baseline survey has been implemented.

Name: Improved teacher performance score

Percentage

0.00 85.00 Monthly

Project MIS


Description: Teacher performance score evaluated by UC/ SC and recorded in KIAT App.

Intermediate Results Indicators




Name: Relevant government regulations issued

Yes/No N Y Annual

Regulations

MoEC; District Government

Description: Ministerial and District Head regulations and decrees to tie TKG with teacher presence in pilot schools

Name: Monthly teacher evaluation meetings conducted

Number 0.00 2800.00 Monthly

Project MIS

Village government; school principal


Page 29 of 42




Description: The meetings are expected to happen in 80% of 203 KGP1 schools for 12 months, and in 183 KGP2 schools for 6 months.

Name: Number of MoEC and district government officials received capacity building (disaggregated by gender)


Project MIS

Grant Recipient

Description: The capacity building activities are targeted for at least 10 people/ district on a quarterly basis.

Name: Number of school stakeholders received capacity building trainings (disaggregated by gender)


Project MIS


Description: Trainings for stakeholders from 135 KGP1 schools and 183 KGP2 schools, represented by 4 person each.

Name: Percentage of TKG paid based on teacher presence

Percentage

0.00 85.00 Quarterly

Project MIS and government's financial reports

MoEC; District Governments

Description: Percentage is based on the total number of TKG-recipients in the pilot schools.


Page 30 of 42

Target Values Project Development Objective Indicators FY

Indicator Name Baseline End Target

Reduced percentage of teacher tardiness 4.00 2.00

Reduced percentage of teachers with unexcused absence 15.00 5.00

Improved teacher performance score 0.00 85.00

Intermediate Results Indicators FY

Indicator Name Baseline End Target

Relevant government regulations issued N Y

Monthly teacher evaluation meetings conducted 0.00 2800.00

Number of MoEC and district government officials received capacity building (disaggregated by gender)

0.00 200.00

Number of school stakeholders received capacity building trainings (disaggregated by gender)

0.00 1200.00

Percentage of TKG paid based on teacher presence 0.00 85.00

The World Bank INDONESIA: IMPROVING TEACHER PERFORMANCE AND ACCOUNTABILITY IN URBAN PRIMARY SCHOOLS (KIAT Guru Urban Pilot) ()

Page 31 of 42

ANNEX 2. PROJECT INTERVENTION DESIGN RATIONALE

59. Improving education quality in poor rural and remote areas is a complex endeavor influenced by myriad factors. International experience shows that demand-side only interventions are less likely to improve service delivery outcomes compared to those that are combined with supply-side reforms. Hence, the interventions are designed around national and district governments’ decisions to link teacher performance evaluations with feedback generated through participatory approaches. KIAT Guru Phase 1 (KGP1) interventions combine:

i. Community Empowerment Mechanism (CEM), which provides community members with an explicit role to monitor and evaluate teacher service performance, and to ensure teacher accountability; and

ii. Pay for Performance Mechanism (PPM), which ties payment of TKG with teacher presence, as recorded by KIAT App and verified by the User Committee (UC), or with teacher service performance, as evaluated by the UC.

A. Community Empowerment Mechanism (CEM)

60. International evidence, as well as evidence from Indonesia, shows that making teachers accountable to their

community as opposed to solely being accountable to higher levels of the education system are effective in improving education service delivery (Joshi, 2010; Ringold et al, 2012). The state of education in poor and remote areas indicates that ongoing reform efforts may be insufficient, or worse, ineffective. The 2004 World Development Report: Making Services Work for Poor People identified two routes to improve accountability of service delivery: the long route through citizen electing, influencing, demanding and pressuring policy makers, and the short route through communities directly influencing, demanding, and pressuring service providers. In Indonesia, learning outcomes improved when village governments played a greater role in basic education, which in turn motivated teachers to treat the local community as a client (Pradhan et al, 2014).

61. MoEC has made several attempts to include community participation in management and delivery of

education services. These efforts have not resulted in positive outcomes due to capacity constraints. In 2002, MoEC issued a decree stating that every school must form a School Committee (MoEC Decree 44/ U/ 2002). The decree stipulates community participation in school-based management. Through a Ministerial Regulation 75/ 2016, MoEC renew the criteria for School Committee, which no longer allow principal and teachers as members, and therefore allow the SC to be more independent and demand accountability if needed. In addition, the Village Law provides an opportunity for villages to assert stronger demands to improve basic education service delivery. However, village governments and community members need mechanisms, tools, and support to do so.

62. Communities also lack the means to hold teachers accountable, to voice their concerns, or to sanction poorly

performing teachers. Drawing from experience in Indonesia and internationally, several elements are required to involve communities in improving accountability of service providers, including: (a) having a

Note to Task Teams: End of system generated content, document is editable from here.


Page 32 of 42

standard to which the service providers will be accounted for; (b) improving communities’ access to information, including their basic rights to services; (c) giving communities the means to influence and voice concerns to service providers; and (d) providing routes to sanction poorly performing service providers (Joshi, 2010; Ringold et al, 2012). Brinkerhoff and Wetterberg (2012) provide a useful pathway which includes standard-setting, results-based management, performance-based payment, and increased information flows and transparency.

63. The KGP1 CEM addressed capacity constraints and strengthened incentives for schools to service villages and

for villages to contribute to improvements in education service delivery. The community facilitation processes consisted of three main phases:

i. Initial Phase to set up Service Agreement (SA), Community Score Card (CSC), and User Committee (UC). This stage consisted of eight meetings, drawing participants that represent two groups: (i) the school provider group, consisting of the teachers and principal, and (ii) the user group, consisting of community leaders, community members, parents, student alumni, upper-grade students, and education stakeholders. The earlier meetings had the two groups separated, and in later meetings the two groups reconvene. The UC was elected by the user group and represented them in filling out the CSC in later stages.

ii. Implementation Phase, where the school provider and the user groups implemented the agreed indicators on SA and CSC. At the end of every month, an interface meeting between the two groups reviewed progress and challenges in meeting the SA. The UC filled out CSC for each of the school providers.

iii. Evaluation Phase, where the school provider and user groups evaluated the implementation of the agreed SA and CSC indicators over a semester. Diagnostic Student Learning Assessment (Diagnostic SLA) from a randomly sampled group of students per grade was administered and disseminated to the stakeholders. At this stage, amendments to the SA and CSC indicators and UC members could be proposed and agreed upon.

64. The KGP1 CEM included:

i. Providing communities with information on their rights to education and on student learning outcomes to increase their demands for better education service delivery. Communities are also informed on their rights to education and to participate in education service delivery as stated in Village Law, and children’s right to education and protected environment. Information on minimum teacher service standards was also shared with both teachers and communities.

ii. Facilitated meetings to develop SA and CSC. KIAT Guru facilitated a locally-developed SA between teachers and community members and assigned communities with the role to monitor and evaluate teachers’ service performance using a CSC (adapted from Bjorkman & Svensson, 2009). The CSC included between 5 to 7 indicators, one of the indicators must be teacher presence in school as scheduled formally.


Page 33 of 42

iii. Assisted communities to establish a UC to evaluate education service delivery. A study on School Committees (SC) in Indonesia suggested that election of committee members and linkage to the village council are instrumental (Pradhan, et al, 2014). The UC is based on this finding and the SC criteria9. The UC was ratified by a Village Head Decree and acknowledged in the District Head Decree, which provided the legitimacy for the UC to hold school providers directly accountable to them.

iv. Providing communities with tools and mechanisms to monitor and evaluate teacher presence and service quality. Teacher presence in schools is usually self-reported manually, and often is required by the Dinas Pendidikan for administrative purpose. In all schools, the filling out of a Teacher Manual Presence Form (TMPF), as mandated by the district governments, were still be implemented. In Group 2 intervention schools, the TMPF was combined with a daily record of teacher presence using KIAT Kamera, a tamper-proof Android-based camera application (adapted from Duflo et al, 2012). At the end of each month, for each teacher in Group 2, the UC filled out a Teacher Presence Verification Form (TPVF) that cross checked the results from KIAT Kamera against TMPF10. For service quality indicators, the UC were informed with three means to monitor teacher service: by interviewing students, through direct observations, or by examining written documents.

v. Putting in place institutional arrangements to bring community inputs to village, sub-district, and district governance. Results of the CSC, SPVF, KIAT Kamera, and TMPF were reported to village, sub-district, and district governments monthly. Meetings on a quarterly basis were conducted at the district levels to evaluate progress.

vi. Conducted Diagnostic SLA to identify where students are along a learning continuum of functional literacy and numeracy skills. The Diagnostic SLA11 was developed to identify students’ skills along a learning continuum of the national curriculum (KTSP 200612). KGP1 trained the Village Cadre and representatives of the UC to administer the Diagnostic SLA, with a random sample of six students per

grade level.

B. Pay for Performance Mechanism (PPM)

65. International experience suggests that incentivizing teachers, whether through social rewards and sanctions, and/or financial rewards and sanctions is likely to improve education quality. Reviews of interventions to improve education and learning in developing countries reveal that changes in incentives and pedagogy are more effective than providing access to education, improving educational inputs (e.g. textbooks, libraries, and grants), and implementing school-based management (Banerjee & Duflo, 2011; Kremer, et. al., 2013;

9 The User Committee is set up based on existing committees in the village (e.g. School Committee, ex-TPMD Committee), although reelection will be necessary to meet the User Committee criteria of: a minimum of 9 elected community members representing a gender-balanced composition, 1/3 parents, 1/3 of minority group representatives, and 1/3 community and religious leaders to provide linkage to the village council. 10 The TMPF is also needed to encounter possible malfunctioning of KIAT Kamera. 11 The diagnostic tools were adapted mostly from ASER (2014), which is administered on a yearly basis in India by volunteers from partner organizations (e.g. universities, government departments, youth clubs). The same tools are provided for community members to check on student learning outcomes progress. 12 The SLA is based on KTSP 2006, which is an older curriculum still widely utilized in most of the remote schools.


Page 34 of 42

Evans & Popova, 2015). In rural and remote areas of Indonesia, it is important to urgently improve basic service delivery first, before changes in pedagogy could be attempted. A meta-analysis of 64 studies found financial incentives to be more effective in comparison with non-financial rewards, but another meta-analysis of 72 empirical studies showed that financial incentives, when combined with forms of non-financial rewards, resulted in better improvements in comparison with financial incentives alone (Hasnain, et al, 2012).

66. KIAT Guru’s PPM, which is a top-down intervention and is a form of financial reward and sanction, is implemented in conjunction with the CEM, which is a form of social reward and sanction. Key features of the PPM include:

i. Regulations at the national and district levels to enable PPM and transparent payment of the teacher special allowance (Tunjangan Khusus Guru, hereafter TKG). MoEC, TNP2K, and Head of Districts signed Cooperation Agreement. Minister and Director General of MoEC, and the Head of Districts also issued regulations and technical guidelines for implementation.

ii. Linking payment of TKG to teacher presence (adapted from Duflo et al, 2012). During KGP1, the formula was developed based on government regulation for civil servants (Perka BKN 12/ 2016). Evaluation is based on a full day presence (no cut), partial presence (cut by up to 1.5% daily), excused leaves (cut by 2%), and unexcused absences (cut by 5%). Teachers whose presence fall below 85% in a month do not receive their TKG at all.

iii. Linking payment of TKG to teacher quality of service, based on the CSC as scored by the UC. Teachers who score 70 receive 70% of their allowances; those who score 85 receive 85% of their allowances.

As elaborated in the main document, the KGP2 will implement two different versions of the CEM, and its PPM will focus on tying TKG with teacher presence.

C. Impact Evaluation Instruments and Sample Size Calculation

67. For the proposed IEs, the surveys will include Teacher Absence Survey (TAS), quantitative questionnaires,

and student learning assessment. These instruments will be administered for longer-term KGP1 survey and KGP2 baseline and endline surveys by a survey team or firm hired directly by the World Bank, and as such, independent from the RETF implementation team. Details of each of the instruments are described below.

i. Teacher Absence Survey (TAS). The instrument originated from the World Bank’s multi-country teacher absence survey (Chaudury, et.al., 2006) which calls for an unannounced visit to schools during normal school hours to obtain a representative estimate of teacher absence from school. The instrument has since been adapted for various TAS implementations in Indonesia. The design and methodology of the KGP1 TAS were mainly adapted from ACDP (2014) study in Indonesia, with additional input from instrument used in UNICEF (2012) study in Papua and West Papua. The KGP1 TAS gauges the rate of teacher absence from the pilot schools and all classrooms in the schools.


Page 35 of 42

ii. Quantitative Questionnaires. Questionnaires will be administered to capture information from key respondents, namely: (i) the school principal, (ii) all teachers, (iii) village government official, (iv) village cadres, (v) school committee representative, (vi) twenty parents’ representatives; and (vii) user committee representatives. The questionnaires were adapted from previous surveys conducted by the World Bank and others (WB, 2013; ACDP, 2014; WB, 2015a & 2015b; WB, 2016). Items included in the KGP1 questionnaires will be modified for KGP2 to add questions on institutional arrangements and governance supports for sustainability.

iii. Student Learning Assessment (SLA). The SLA was developed, pre-tested, and implemented to assess basic functional literacy (in Bahasa Indonesia) and numeracy competencies along the learning continuum standards set in the 2006 national curriculum (KTSP 2006). Designed based on frameworks and findings from other assessment tools (ASER, 2014; UWEZO, 2012; Gove & Wetterberg, 2011; and Platas et. al., 2014), the SLA will be administered for all students in grade 1 to 6 in participating schools, on a one-on-one basis for 15 students in each grade 1 and 2, and on a group basis for grades 3 to 6. Data will be analyzed based on changes in students’ score, controlling for other background characteristics. All students tested during baseline will be tracked for endline (including if they move to another treatment school). The KGP1 results were estimated for the panel students, although results remain the same when all students were considered. There was no evidence of systematic attrition of students. The same approach will be used in KGP2.

68. We use the standardized test scores of children in grades 4, 5, and 6 as the outcome variable to conduct the power calculation. We used the sample from the teacher certification study (de Ree et al, 2018). The data has a panel structure, which allows us to take account of the residual variance that can be reduced by including lagged test scores in the equation. The lagged test scores were highly correlated with the current test score (correlation coefficient 0.59). Table 5 indicates that with an effect size of 0.15, a power of 80% is reached with 60 schools in one cell. If the effect size falls, the sample size requirements increase. For example, to detect an effect of 0.1 standard deviations, we need a sample size of just over 100 schools per cell.

Table 5. Power with different effect sizes and sample sizes

Effect size Sample size

0.1 0.15 0.2

40 0.36 0.625 0.825 60 0.53 0.795 0.975 80 0.62 0.945 0.995

100 0.775 0.975 1 120 0.925 1 1

69. In the teacher certification study, there were 228 primary schools stemming from 19 districts. The average number of students per school (in classes 3, 4, 5, and 6) combined was 450. So these power calculations are based on rather large samples per school. Table 6 shows the results if we restrict ourselves to one class (we choose class 5). In that case, the average number of children per school drops to 112. The power slightly


Page 36 of 42

declines, but not by much.

Table 6. Power with different effect sizes and sample sizes using grade 5 students only

Effect size Sample size 0.1 0.15 0.2

40 0.3 0.53 0.735

60 0.46 0.715 0.905

80 0.62 0.86 0.99

100 0.715 0.97 1

120 0.865 0.995 1

70. In KGP1, we had 67 schools per cell, which provided us power of close to 80% with a hypothesized effect size of 0.15 standard deviation change in test score. The minimum detectable effect size is 0.165 standard deviation with a sample size of 67 per cell and 80% power.


Page 37 of 42

ANNEX 3: Implementation Arrangements

A. Project Institutional and Implementation Arrangements Project administration mechanisms

71. BaKTI will have responsibility for administering and executing funding allocated according to the pre-agreed

work plan and targets set by the World Bank and Steering Committee. Additionally, BaKTI will be guided by the draft Procurement Plan that outlines estimated annual expenditure envelopes for the grant. Flexibility is built into this arrangement through the Steering Committee meetings where performance will be reviewed, priorities will be clarified, and amendments to pre-agreed activities can be proposed.

72. It is expected that BaKTI will maintain up-to-date records on each activity funded through the grant and will provide this information on a monthly and quarterly basis to the World Bank, which will in turn use this information to prepare performance reports for the Steering Committee. The monthly and quarterly reports prepared by BaKTI for the World Bank will include a detailed update on the progress of the various grant-funded activities under implementation as well as all necessary financial information concerning the administration and use of the grant. The World Bank, in conjunction with BaKTI will agree to a reporting template for the biannual reports, including on all reportable results indicators and targets, once the Grant Agreement (GA) is signed.

73. The grant, while executed by BaKTI, is subject to the rules, procedures, policies and guidelines governing the identification, procurement and selection of external vendors be it an individual or firm. Moreover, the procurement of goods and administration of all funding are subject to the requisite policies of the World Bank (per the GA). The World Bank will provide advice and guidance on these matters to BaKTI in a just-in-time manner throughout the implementation of the grant.

B. Financial Management, Disbursements and Procurement Financial Management

74. The Bank carried out a Financial Management (FM) assessment for the Recipient Executed Trust Fund (RETF)

in accordance with BP/BD for IPF on November 10, 2017. Overall, the financial management risk is assessed as moderate and with the implementation of the agreed action plan, the proposed financial arrangements will satisfy the Bank’s minimum requirements and are adequate to provide, with reasonable assurance, accurate and timely information of the grant required by the Bank.

75. Budgeting. The project budget will be included in BaKTI’s budget and become effective after the grant signing. Budgeting will follow project budgets and activity plans as agreed between BaKTI and the Bank in the project appraisal documents.

76. Accounting and Reporting. Project accounting will be integrated into existing BaKTI accounting system (Quickbook). In addition, BaKTI will prepare quarterly Interim un-audited Financial Report (IFR). IFR should


Page 38 of 42

be submitted to the Bank no later than 45 days after the period end.

77. Internal Control & Internal Audit. BaKTI has adequate internal control system. Segregation of duties has taken place and work properly. BaKTI Headquarter has conducted internal audit function to the satellite office during KGP1 and will continue to conduct such function during KGP2.

78. External Audit. BaKTI is required to submit an entity audited financial report to the Bank annually. The financial report should include the project’s expenditures in such period with notes on the information of the funds received and expenditures of KGP2. The audits will be conducted based on TOR approved by the Bank. Audit reports and audited financial statements will be furnished to the Bank by not later than six months after the end of the fiscal year and shall be made available to the public.

79. Disbursement arrangement. A Designated (Special) Account (DA) denominated in Indonesia Rupiah will be opened. The DA will be a segregated account with a ceiling of IDR 6,000,000,000 (Six Billion Indonesia Rupiah) and will be used for financing eligible expenditures from this Grant. Applications for the replenishment of DA advance may be submitted on quarterly basis together with the report of use of the DA fund, supported by: (i) list of payments for contracts under Bank’s prior-review and records evidencing such expenditures, or (ii) statement of expenditures (SOEs) for all other expenses; (iii) DA reconciliation statement; and (iv) forecast cash requirement for the next six months.

80. Allocation of grant proceeds is as follows:

Category Description Amount of Grant (US$) % of expenditures

to be financed

Goods, Non-consulting services, Consulting services, Training & workshop, and Incremental operational costs13

1,348,000 100%

TOTAL 1,348,000

81. Supervision Plan. Risk-based supervision of project financial management will be conducted. This will involve

desk supervision, including review of IFRs and one supervision mission in one year. Financial management supervision will be conducted by FMS or Bank consultants under Financial Management Specialist direction.

Procurement

82. The following mitigation actions will be completed before the signing of the proposed financing agreement,

which includes support for application of the Bank’s Procurement Framework according to the Procurement Regulations for IPF Borrowers dated July 2016 (revised November 2017):

13 Incremental operational costs include: local contractual support staff salaries (implementation support personnel); travel and other travel-related expenditures; equipment rental and maintenance; vehicle operation, maintenance and repair; office rental and maintenance; materials and supplies; utilities and communications expenses; consumables; bank charges; and advertising expenses.


Page 39 of 42

i. Facilitate updates of actual procurement needs of the project by BaKTI through setting of appropriate

prior review requirements based on project procurement capacity and risk level to be specified in the procurement plan.

ii. Proper planning of procurement activities to balance the work load of the designated procurement staff.

iii. Deliver trainings on the Bank’s procurement procedures under the Procurement Framework, including for preparation of the short-form Project Procurement Strategy for Development (PPSD), procurement plan and use of the updated version of the Bank’s new online procurement planning and tracking tool (STEP) for all procurement transactions.

83. Procurement implementation support will be part of the Bank’s regular implementation support mission.

The Bank will conduct implementation support missions for the project every six months, and this will be followed up by the ex-post review of samples of contracts subject to post-review (as indicated in the agreed Procurement Plan), which will be reported in June of every calendar year. The procurement implementation support will include advisory support to BaKTI and hands-on training on procurement arrangement and contract administration for the project. The procurement plan will be made available on the project’s website and on the Bank’s external website. The Procurement Plan would be updated at least once a year or as required to reflect the actual project implementation needs and improvements in institutional capacity.

Environmental and Social (including safeguards)

84. BaKTI, as the Grant Recipient for the project’s RETF will implement the overall safeguards requirements,

including among others, provision of operational support and technical assistance to ensure that the project interventions can be implemented in accordance to environmental and social good practices and in a manner appropriate and acceptable to local contexts. The Bank’s safeguards specialists will review the Project Operational Manual (POM) to ensure that they adhere to the World Bank’s environmental and social safeguards policies and participate in regular supervision to ensure the implementation of necessary due diligence processes as prescribed in the POM. Prior to project implementation, BaKTI will provide training to project staff and facilitators on relevant World Bank’s safeguards and child protection policies where applicable.

85. A Complaint Handling System (CHS) has been developed earlier under KGP1 using an Short Message Service (SMS) gateway platform and is being operated by BaKTI. The CHS has been used to identify systemic issues to enable coordination and troubleshooting. To complement the CHS, further facilitation to enable proactive identification of emerging issues and complaints will be embedded under the project’s facilitation design. It is expected that the same CHS will be grandfathered to support KGP2 and further improvement to integrate the CHS into the upgraded KIAT App and Management Information System (MIS) will be conducted during project implementation. Such efforts to upgrade and streamline the existing CHS is expected to enable systematic tracking of complaints and an automated process to dispatch relevant issues to the district governments’ high-ranking officers with relevant mandates if no resolution can be reached. Progress reports provided by BaKTI to the World Bank will document the status of all complaints received.


Page 40 of 42

Monitoring & Evaluation

86. BaKTI will be responsible for actively monitoring and documenting lessons learned in each activity funded through the grant against a set of approved results indicators. BaKTI will collate results across activities into monthly and quarterly summary reports to be shared with the World Bank. The time-table for completion and submission of these progress reports will be determined after the GA has been signed. Major milestones from the work plan will be presented to the Steering Committee in its inaugural meeting to get their inputs and approval.

87. To supplement the results monitoring and evaluation carried out under the Project, the World Bank will support GoI to implement a rigorous evaluation program. The monitoring, learning, and evaluation reports will be presented to the Steering Committee biannually to provide more intermediate policy inputs, while data is collected on the longer-term impacts of the interventions.

Role of Partners (if applicable)

88. KIAT Guru is a program for reform financed by MoEC, five district governments, and the World Bank through

the LSDTF. It is overseen by the National Coordination Team, which provides policy and technical advices and directions for the intervention and evaluation designs.


Page 41 of 42

REFERENCES ACDP (2014). Study on Teacher Absenteeism in Indonesia 2014. Jakarta, Indonesia: Education Sector

Analytical and Capacity Development Partnership. ASER. (2014). Annual Status of Education Report (Rural) 2013. New Delhi, India: ASER Centre. Banerjee, A., Banerji, R., Berry, J., Duflo, E., Kannan, H., Mukerji, S., Shotland, M., & Walton, M. (2017).

From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application. NBER Working Paper No. 22931.

Banerjee, A., Banerji, R., Duflo, E., Glennerster, R., & Khemani, S. (2010). Pitfalls of participatory programs: Evidence from a randomized evaluation in education in India. American Economic Journal: Economic Policy, 2010, 2(1): 1-30.

Banerjee, A., Glennerster, R., & Duflo, E. (2008). Putting a band-aid on a corpse: Incentives for nurses in the Indian public health care system. Journal of European Economic Association, 2008, 6(2-3): 487-500.

Banerjee, A., & Duflo, E. (2006). Addressing absence. Journal of Economic Perspectives—Volume 20, Number 1—Winter 2006—Pages 117–132.

Barrera-Osoria, F. and D. Raju (2017). Teacher performance pay: Experimental evidence from Pakistan. Journal of Public Economics Vol. 148, pp. 75-91.

Barr, A., Mugisha, F., Serneels, P., & Zeitlin, A. (2012, draft). Information and collective action in community-based monitoring of schools: Field and lab experimental evidence from Uganda. Retrieved from https://sites.google.com/site/andrewzeitlin/research/m%26m.pdf?attredirects=0

BPS (Indonesian Center of Statistics Agency). (2018). Profil Kemiskinan di Indonesia, Maret 2018. (Poverty Profile in Indonesia, Maret 2018). Jakarta: BPS.

Bold, T., Kimenyi, M., Mwabu, G., Ng’ang’a, A., & Sandefur, J. (2018). Experimental Evidence on Scaling Up Education Reforms in Kenya. Journal of Public Economics, Volume 168, December 2018, pages 1-20.

Brinkerhoff, D.W. & Wetterberg, A. (2013, draft). Performance-based public management reforms: Experience and emerging lessons from service delivery improvement in Indonesia.

Chaudhury, N., Hammer, J., Kremer, M., Muralidharan, K., and Rogers, F.H., Missing in Action: Teacher and Health Worker Absence in Developing Countries. Journal of Economic Perspectives, February 2006, 20 (1), 91–116.

De Ree, J., Muralidharan, K., Pradhan, M., and Rogers, H., (2018). Double for Nothing? Experimental Evidence on an Unconditional Teacher Salary Increase in Indonesia. Quarterly Journal of Economics 133 (2): 993–1039.

Duflo, E., Hanna, R. and S.P. Ryan (2012). Incentives Work: Getting Teachers to Come to School. American Economic Review Vol. 102(4), pp. 1241-1278.

Gaduh, A., Pradhan, M., Priebe, J., & Susanti, D. (Forthcoming). Scores, Camera, Action? Incentivizing Teachers in Remote Areas.

Gove, A. & Wetterberg, A. (2011). The Early Grade Reading Assessment: Applications and Interventions to Improve Basic Literacy. Research Triangle Park, NC: RTI Press.

Glewwe, P., Ilias, N. and M. Kremer (2010). Teacher incentives. American Economic Journal – Applied Economics Vol. 2(July), pp. 205-227.

Hasnain, Z., Manning, N., & Pierskalla, J.H. (2012). Performance-Related Pay in the Public Sector: A Review of Theory and Evidence. Washington, D.C.: The World Bank.

https://sites.google.com/site/andrewzeitlin/research/m%26m.pdf?attredirects=0


Page 42 of 42

Kremer, M., Brannen, C. & Glennester. (2013). The challenge of education and learning in developing world. Science 340, 297. DOI: 10.1126/science.1235350.

Joshi, A. (2013). Do they work? Assessing the impact of transparency and accountability initiatives in service delivery. Sussex, UK: Institute of Development Studies.

Muralidharan, K. (2017). Field experiments in education in developing countries. In Banerjee, A., & Duflo, E. (2017). Handbook of Economic Field Experiments, Volume 2. North-Holland.

Muralidharan, K. (2012). ‘Long-Term Effects of Teacher Performance Pay: Experimental Evidence from India’. Working paper.

Muralidharan, K. and V. Sundararaman (2011). Teacher Performance Pay: Experimental Evidence from India. Journal of Political Economy, Vol. 119(1), pp. 39-77.

OECD. (2016). PISA 2015 Results in Focus. Paris, France: OECD. Perez-Alvarez, M., Priebe, J., & Susanti, D. (Forthcoming). Teacher Accountability and Pay-for-

Performance Schemes in (Semi-) Urban Indonesia: What do Education Stakeholders Think? Platas, L. M., Ketterlin-Gellar, L., Brombacher, A. & Sitabkhan, Y. (2014). Early Grade Mathematics

Assessment (EGMA) Toolkit. North Carolina, USA: Research Triangle Park Press. Pradhan, M., Suryadharma, D., Beatty, A., Wong, M., Alishjabana, A., Gaduh, A., & Artha, R. (2014).

Improving educational quality through enhancing community participation: Results from a randomized field experiment in Indonesia. American Economic Journal: Applied Economics, Vol 6(2), April 2014, pp. 105-26.

Ringold, D., Holla, A., Koziol, M. & Srinivasan, S. (2012). Citizens and service delivery: Assessing the use of social accountability approaches in human development. Washington, D.C.: The World Bank.

SMERU. (2010). Teacher Absenteeism and Remote Area Allowance: Baseline Survey. Jakarta, Indonesia: SMERU Research Institute.

UWEZO. (2012). Are Our Children learning? Annual Learning Assessment Report. Uganda: UWEZO. World Bank. (2004). World Development Report 2004: Making services work for poor people.

Washington, DC: The World Bank & Oxford University Press. World Bank. (2016). Indonesia's Rising Divide. Washington, D.C.: World Bank Group.

http://documents.worldbank.org/curated/en/267671467991932516/Indonesias-rising-divide World Bank. (2017). Learning to realize education’s promise. World Development Report 2018.

Washington, D.C: World Bank Group. World Bank. (2018). Growing Smarter: Learning and Equitable Development in East Asia and Pacific.

Washington, D.C: World Bank Group.

World Bank Documentdocuments.worldbank.org/curated/en/... · National Development Planning Agency...

Documents

Transcript of World Bank Documentdocuments.worldbank.org/curated/en/... · National Development Planning Agency...