Document of The World Bank - Documents & Reports -...

Document of The World Bank

Report No: ICR00003507

IMPLEMENTATION COMPLETION AND RESULTS REPORT

(TF-14360)

ON A

PNPM SUPPORT FACILITY (PSF) GRANT

IN THE AMOUNT OF USD 1,665,000

TO THE

KEMITRAAN BAGI PEMBARUAN TATA PEMERINTAHAN (KEMITRAAN)

FOR A

PRO-POOR PLANNING, BUDGETING AND MONITORING (P3BM) PROJECT

December 17, 2015

Global Practice for Social, Urban, Rural and Resilience Indonesia Country Management Unit East Asia and Pacific Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

i

CURRENCY EQUIVALENTS

(Exchange Rate Effective June 30, 2015)

Currency Unit = Rupiah USD 1.00 = IDR 13,333

FISCAL YEAR

ABBREVIATIONS AND ACRONYMS

ADB APBD APBN

Asian Development Bank Anggaran Pendapatan dan Belanja Daerah (Local Government Budget) Anggaran Pendapatan dan Belanja Negara (State Budget)

BAPPEDA BAPPENAS BETF

Badan Perencanaan Pembangunan Daerah (Regional Development Planning Agency) Badan Perencanaan Pembangunan Nasional (National Development Planning Agency) Bank Executed Trust Fund

CBO CC CIDA CPS CR CSO CSR

Community Based Organization Creative Communities Canadian International Development Agency Country Partnership Strategy Completion Report (prepared jointly by executing and implementing agencies) Civil Society Organization Corporate Social Responsibility

DFAT DO DPRD

Australian Department of Foreign Affairs and Trade Development Objective Dewan Perwakilan Rakyat Daerah (Regional House of Representatives)

EIRR ET

Economic Internal Rate of Return Extended Term

FM FY

Financial Management Fiscal Year

GDP GIS GoI

Gross Domestic Product Geographic information system Government of Indonesia

HDI Human Development Index ICR IDE IDR IFR IP ISR

Implementation Completion and Results Institute for Democracy Education Indonesian Rupiah Interim Financial Report Implementation Performance Implementation Status Results

JMC Joint Management Committee Kabupaten Kota

District Municipality

LG LGCD

Local Government Local Governance Capacity Development Project

KPI Kemitraan

Key Performance Indicator Kemitraan Bagi Pembaruan Tata Pemerintahan (The Partnership for Governance Reform)

M&E MDG MDTF

Monitoring and Evaluation Millennium Development Goals Multi Donor Trust Fund

ii

MIS MoF MoHA MTR Musrenbang

Management Information System Ministry of Finance Ministry of Home Affairs Mid-Term Review Musyawarah Perencanaan Pembangunan (Community Planning Meeting)

NGO NSC NPV

Non-government organization National Steering Committee Net Present Value

P3BM PAD PID PDO PMU PNPM PP PSF Pusbindiklatren

Pro-poor Planning, Budgeting and Monitoring Project Appraisal Document Project Information Document Project Development Objective Project Management Unit Program Nasional Pemberdayaan Masyarakat (National Community Empowerment Program) Project Paper PNPM Support Facility Pusat Pembinaan, Pendidikan dan Pelatihan Perencanaan (The Center for Planners Development, Education and Training under Bappenas)

QAG QEA QPR QSA

Quality Assurance Group Quality at Entry Quarterly Progress Report Quality at Supervision

Renstra RETF RPJMN

Rencana Strategis (Strategic plan) Recipient Executed Trust Fund Rencana Pembangunan Jangka Menengah Nasional (National Medium-Term Development Plan)

SDG Seknas SIMPADU SK SKPD SME

Sustainable Development Goals Sekretariat Nasional (National Secretariat of P3BM, within Bappenas) Integrated Management Information System (MIS) for PNPM Mandiri Surat Keputusan (Decree) Satuan Kerjan Perangkat Daerah (Regional Working Unit) Small Medium Enterprises

TA TKPKD TF ToT TNP2K TTL

Technical Assistance Tim Koordinasi Penanggulangan Kemiskinan Daerah (Regional Coordinating Team for Poverty Reduction) Trust Fund Training of Trainers Tim Nasional Percepatan Penanggulangan Kemiskinan (National Team for the Acceleration of Poverty Alleviation) Task Team Leader

UNDP UU

United Nations Development Programme Undang-Undang (Law)

Senior Global Practice Director: Ede Jorge Ijjasz-Vasquez

Country Director Rodrigo A Chaves

Program Manager: Kevin A Tomlinson

Project Team Leader: Bambang Soetono

ICR Team Leader: Bambang Soetono

iii

INDONESIA Pro-Poor Planning, Budgeting and Monitoring (P3BM) Project

CONTENTS

Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph

1. Project Context, Development Objectives and Design ............................................................ 1 2. Key Factors Affecting Implementation and Outcomes ........................................................... 3 3. Assessment of Outcomes ....................................................................................................... 10 4. Assessment of Risk to Development Outcome ..................................................................... 16 5. Assessment of Bank and Borrower Performance .................................................................. 17 6. Lessons Learned .................................................................................................................... 19 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners ....................... 20 Annex 1. Project Costs and Financing ....................................................................................... 21 Annex 2. Outputs by Component .............................................................................................. 22 Annex 3. Economic and Financial Analysis .............................................................................. 24 Annex 4. Bank Lending and Implementation Support/Supervision Processes ......................... 26 Annex 5. Beneficiary Survey Results ........................................................................................ 28 Annex 6. Stakeholder Workshop Report and Results ............................................................... 29 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR .................................. 30 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders .................................... 32 Annex 9. List of Supporting Documents ................................................................................... 33

MAP

iv

A. Basic Information

Country: Indonesia Project Name: ID-TF PNPM Pro-Poor Planning, Budgeting and Monitoring Program

Project ID: P131815 L/C/TF Number(s): TF-14360

ICR Date: 05/05/2015 ICR Type: Core ICR

Lending Instrument: TAL Grantee: NGO

Original Total Commitment:

USD 1.67M Disbursed Amount: USD 1.66M

Revised Amount: USD 1.67M

Environmental Category: C Implementing Agencies: KEMITRAAN Co-financiers and Other External Partners:Bappenas

B. Key Dates

Process Date Process Original Date Revised / Actual

Date(s)

Concept Review: 08/08/2012 Effectiveness: 06/26/2013

Appraisal: 12/19/2012 Restructuring(s): 12/2/2013

Approval: 03/04/2013 Mid-term Review: 07/14/2014

Closing: 12/31/2013 06/30/2015 C. Ratings Summary C.1 Performance Rating by ICR

Outcomes: Moderately Unsatisfactory

Risk to Development Outcome: Significant

Bank Performance: Moderately Unsatisfactory

Grantee Performance: Moderately Satisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings

Quality at Entry: Moderately Satisfactory Government: Moderately Satisfactory

Quality of Supervision: Moderately Unsatisfactory

Implementing Agency/Agencies:

Moderately Satisfactory

Overall Bank Performance:

Moderately Unsatisfactory

Overall Borrower Performance:


v

C.3 Quality at Entry and Implementation Performance IndicatorsImplementation

Performance Indicators

QAG Assessments (if any)

Rating

Potential Problem Project at any time (Yes/No):

No Quality at Entry (QEA):

None

Problem Project at any time (Yes/No):

No Quality of Supervision (QSA):

None

DO rating before Closing/Inactive status:


D. Sector and Theme Codes Original Actual

Sector Code (as % of total Bank financing)

Other social services 40 40

Sub-national government administration 60 60

Theme Code (as % of total Bank financing)

Other rural development 50 50

Other social development 50 50 E. Bank Staff Positions At ICR At Approval

Vice President: Axel van Trotsenburg Axel van Trotsenburg

Country Director: Rodrigo A. Chaves Stefan G. Koeberle

Practice Manager/Manager: Kevin A. Tomlinson Jan Weetjens

Project Team Leader: Bambang Soetono Hans Antlov

ICR Team Leader: Bambang Soetono

ICR Primary Author: Julie Klugman F. Results Framework Analysis

Project Development Objectives (from Project Appraisal Document)The objective of the project is to improve poverty targeting of local government expenditures and local government responsiveness to community needs through the scale-up and institutionalization of the Pro-poor Planning, Budgeting and Monitoring (P3BM) program. Revised Project Development Objectives (as approved by original approving authority) N/A

vi

(a) PDO Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised

Target Values

Actual Value Achieved at

Completion or Target Years1

Indicator 1 : # of local governments that have adopted poverty targeting in annual development planning

Value quantitative or Qualitative)

0 60 73

Date achieved 11/23/2012 06/12/2013 06/30/2015

Comments (incl. % achievement)

Of the 85 LGs who have adopted the P3BM tools, 73 LGs have used it to set the local budget plan and 12 LGs used it to analyze the local MDGs data. The measure used for “adoption” was not standard. In some cases this meant using several of the P3BM tools, in others, one aspect of the toolkit only.

Indicator 2 : Increased percentage of direct LG budgets allocated for MDG targeting and poverty-related spending

Value quantitative or Qualitative)

0 7.5% 9.3%

Date achieved 11/23/2012 06/12/2013 06/30/2015


This figure of 9.3%, provided in the Completion Report (C.R), is based on the Seknas analysis of 15 local governments that received P3BM training and technical assistance in early 2013, and compares the MDG-related spending in APBDs pre-P3BM (2013) and post-P3BM (2014),2 in both target and non-target areas. It should be noted that these figures differ to those provided in the final ISR, where the increase is reported as 6.15% in local government budget allocations, which is less than the target. It is also interesting to note that if “target” areas only are considered (7 locations), the percentage increase is reduced, closer to 5.5%.

(b) Intermediate Outcome Indicator(s)



approval documents)

Formally Revised Target

Values


Completion or Target Years

Indicator 1 : # of master trainers trained Value (quantitative or Qualitative)

0 280 284

Date achieved 11/23/2012 06/12/2013 06/30/2015 Comments (incl. % achievement)

There are 679 people who participated in master trainer training and certified. There was no rigorous assessment of participants’ capacity to then become “master trainer”. Implementing and executing agencies estimate that of those who had participated in this

1 Figures as provided in the Completion Report, Kemitraan, September 2015. 2 The 14 locations in the sample were reportedly chosen because of availability of data.

vii



approval documents)

Formally Revised Target

Values


Completion or Target Years

training only approximately 41%, that is, 284 people, are currently capable (and available) to act as master trainers.

Indicator 2 : # of P3BM provincial secretariat/clinic established Value (quantitative or Qualitative)

0 12 15

Date achieved 11/23/2012 06/12/2013 06/30/2015


The CR notes the establishment of 15 secretariats/clinics. However, Seknas subsequently advised that only 10 of the provincial secretariats/clinics obtained official letter (SK/decree) of support from Governor or head of Bappeda to confirm their existence.

Indicator 3 : # of CSOs and community groups that use P3BM instruments to track district pro-poor expenditures

Value (quantitative or Qualitative)

0 60 9


The figure of 9 represents the number involved in piloting of training for CSO monitoring/tracking. It is not clear if any have actually used P3BM instruments to track expenditure. This is substantially less than the agreed target.

Indicator 4 : Business plan developed for the long-term sustainability of P3BM Value (quantitative or Qualitative)

N/A Business plan implemented

Draft business plan prepared.


The business plan has been drafted but not officially finalized or implemented. Therefore the target has not yet been achieved.

Indicator 5 : District-level MDG data improved Value (quantitative or Qualitative)

N/A Database used 74 LGs used P3BM database


Although not one of the official KPIs, and not in the Grant Agreement, this indicator had been included as an indicator (against component 3) in the Bank’s ISRs. This figure comes from the final ISR, and is provided here for consistency.

G. Ratings of Project Performance in ISRs

No. Date ISR Archived

DO IP Actual Disbursements

(USD millions) 1 09/24/2013 Satisfactory Satisfactory 0 2 06/20/2014 Moderately Satisfactory Satisfactory 1.41 3 12/30/2014 Moderately Satisfactory Satisfactory 1.62

viii

No. Date ISR Archived

DO IP Actual Disbursements

(USD millions) 4 06/23/2015 Moderately Satisfactory Satisfactory 1.66

H. Restructuring (if any)

Restructuring Date(s)

Board Approved PDO Change

ISR Ratings at Restructuring

Amount Disbursed at

Restructuring in USD millions

Reason for Restructuring & Key Changes Made

DO IP

12/05/2013 S S 0.18 Extension of closing date I. Disbursement Profile

1

1. Project Context, Development Objectives and Design 1.1 Context at Appraisal After almost fifteen years of decentralization, subnational governments in Indonesia today are responsible for around one third of all public expenditure and more than half of the spending on basic services such as health, education and sanitation. However, service quality is still generally low and local governments are not responding to community priorities. Indonesia is lagging behind in several MDGs and HDI indicators. Many communities, especially in disadvantaged areas, lack access to basic services. The capacity of district governments to provide equitable and accessible services targeting the poor varies considerably across regions. There are several contributing factors to this: local governments lack the latitude to reward staff for enhanced performance, and government officials lack the incentives to respond positively to community priorities or embrace pro-poor planning. Decades of hierarchical bureaucratic culture have also meant that a reliance on formal, upward accountability have constrained reforms. Prior Public Expenditure Reviews undertaken by the World Bank in Indonesia3 had shown that the efficiency of local government spending had been constrained by negative incentives from the national level (e.g. over-staffing driven by inter-governmental transfer formulas); limited discretionary funds for service delivery; and an over-engineered system hampered by late approvals of budgets and a lack of capacity to disburse. Although most local governments in Indonesia have sufficient funds to deliver quality services, they are not spending their funds efficiently, contributing to the low quality of public expenditure and service provision. The hypothesis behind the Pro-poor Planning, Budgeting and Monitoring (P3BM) project is that improved, better targeted, service provision can be achieved if district governments would use improved poverty data and reference pro-poor achievements in their planning and budgeting process. Many local governments have an interest in doing so, but lack the references, capacity and skills. Officials need the tools to monitor and evaluate local government performance related to the provision of public services for the poor, while citizens need access to monitoring tools to ensure that such plans are implemented appropriately. The P3BM was originally designed by the UNDP and ADB in collaboration with Bappenas in between 2008 – 2010, during which time the approach and training were piloted in a number of locations4. In 2010, P3BM was taken over by Bappenas with support from PSF, and support was provided for further capacity building and pilots during 2010-2012 under phase 1 of the Local Government Capacity Building Project (LGCD). During this phase, training in the P3BM toolkits was provided to two batches of master trainers, who in turn trained 210 local officials from 29 local Governments5. A local organization IDE6, working with a team from Bappenas, was contracted by the Bank to provide interim support on a range of activities relating to the toolkit and training during the transition period prior to the scale up of this most recent phase. 1.2 Rationale for Bank Assistance The project is consistent with the FY2013-15 Country Partnership Strategy (CPS) for Indonesia, which emphasizes engagement with government counterparts (including counterparts at the subnational level) and other stakeholders to address critical governance and institutional challenges. In addition to cross-cutting engagement to strengthen governance, the objective of the P3BM was consistent with and able to contribute

3 World Bank (2011a). “Sub-National Government Spending Trends and Impact on Service Delivery Outcomes”, Indonesia Sub-National Public Expenditure Review, Policy Note 1. Jakarta: World Bank, 2011. 4 The project was piloted in 11 districts. At that time 2 donors, UNDP and CIDA implemented in 4 provinces and Sulawesi, respectively. http://www.adb.org/sites/default/files/project-document/67322/39063-ino-tcr.pdf 5 P3BM Project Paper, June 2013. 6 Support for IDE (The Indonesian Institute for Democracy Education) was provided through the Bank Executed Trust Fund.

2

to three of the four thematic areas that form the CPS, those of: (i) Promoting Prosperity through Pro-Growth Policies; (ii) Enhancing Skills and Technology, and Improving Social Protection through Pro-Jobs Policies; and (iii) Promoting Communities, Protecting the Vulnerable and Improving Health Outcomes through Pro-Poor Policies. The P3BM project addresses core concerns in the CPS in terms of poverty targeting, local government capacity and linkages between levels of government. 1.3 Original Project Development Objectives (PDO) and Key Indicators The Project Development Objective is to “improve poverty targeting of local government expenditures and local government responsiveness to community needs through the scale-up and institutionalization of the P3BM program”. The underlying development assumption is that the availability and use of reliable evidence and data can make planning decisions more pro-poor and responsive to local needs. Key performance indicators7 and compounded targets as described in the Project Grant Agreement were (i) 280 provincial master trainers trained; (ii) 60 local governments adopting P3BM in annual development planning; (iii) a 7.5 percent increase of direct local government budgets allocated for MDG targeting and poverty-related spending; (iv) 12 provincial secretariats/clinics established; (v) 60 civil society organizations (CSOs) and community groups that use P3BM instruments to track district pro-poor expenditures. 1.4 Revised PDO and Key Indicators The Project had only a level two restructuring, extending the project period for six months (at no additional cost). The PDO and the KPIs remained unchanged throughout the project. 1.5 Main Beneficiaries The primary target groups of the project were (a) national and provincial master trainers, who were to be given advanced skills in different aspects of P3BM toolkit, to support them in their role of trainers of local government officials involved in planning and budgeting; (b) staff from provincial governments that were to be provided with direct technical assistance on the P3BM toolkit to support their own planning (as well as their and capacity to support planning of district level governments). Secondary beneficiaries included district governments to which the provincial trainers would provide capacity development and subsequently communities, who in turn would benefit from improved planning and quality of expenditures. There was no revision to the definition of the main beneficiaries. 1.6 Original Components The project consisted of three components, which are described below, with principal financing from the Australian Government through NGO Executed Trust Fund from the PNPM Support Facility. The project was to be implemented by the Indonesian CSO Kemitraan. Kemitraan was provided with a grant to administer the national P3BM secretariat hosted by Bappenas, the executing agency. The specific components were as follows: Component One: Provincial Cascade Training (ToT) for Master Trainers (USD 171,000). The scale-up of P3BM continued the training of master trainers’ approach which began under the LGCD project mentioned above. Through a national cascade approach, national master trainers would support 280 provincial master-trainers through larger regional events, who in turn would provide training and technical assistance to targeted subnational governments.

7 It should be noted that during the project, the reported indicators were changed at different times. For example, indicator (ii) on number of local governments that have adopted P3BM in annual planning was in some reports changed to “that have adopted poverty targeting”; indicator (v) on number of CSO using P3BM to track expenditure was changed for a period to number of “CSO trained” in P3BM; and indicator (iv) number of provincial secretariats established was in some documents changed to “number of clinics established”.

3

Component Two: Technical Assistance for Local Governments (USD 660,000). The P3BM master trainers (national and provincial) were to support local governments in improving their planning, budgeting and public service monitoring, providing continuous technical assistance to selected and targeted district governments. 12 provinces were selected for enhanced technical assistance. In these provinces, the project established P3BM Secretariats/clinics within Bappeda consisting of a Coordinator and a small pool of ad hoc Provincial Master Trainers, supported by a small secretariat. Activities to be managed by these provincial secretariats, with support from Seknas, included the establishment of a P3BM Provincial Secretariat/Clinic, a P3BM Forum Data and Information Office, as well as the capacity building support in relation to SKPD, Civil Society and Community Monitoring. Component Three: Institutional Support and National Secretariat (USD 834,000). This component revolved around supporting a system to institutionalize the P3BM program. The sub-components included the (i) Core support for the National Secretariat (ii) Information and knowledge management (iii) Database Management (iv) Collaboration with TNP2K on poverty data, poverty reduction outreach (v) support to the TKPKD and (vi) Analytical studies and monitoring and evaluation. 1.7 Revised Components There was no revision of components. 1.8 Other significant changes The project was extended in December 2013 until June 2015, making the duration of this phase 30 months. The project design called for 24-month implementation; however, while the parent trust fund (PSF Multi-Donor Trust Fund no TF070967) at the time of project effectiveness (June 2013) had 30 June 2014 as the closing date; the Legal Agreement with Kemitraan was signed until December 2013. Once the MDTF was extended, the task team sought approval for a no-cost extension of TF014360 until June 30, 2015. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry Rating: Moderately Satisfactory 2.1.1 Soundness of Background Analysis. Given that P3BM was not required to prepare a PAD, but instead only a Project Concept Note, Project Paper and PID, the length of the document and background analysis required was quite limited. Drawing upon the substantial experience gained from previous phases of P3BM, the Banks background analysis identified8 the following important lessons learned, which were highlighted as key areas requiring increased attention during implementation: The need to increase attention to and capacity building support for Community tracking of expenditure. The need to discuss the issue of “incentives” with local governments The need to improve district-level poverty data and information The need to provide support to District Coordination Teams for Poverty Alleviation (TKPKD).

The issue of the need to better engage with DPRDs was also touched on, but not any detail. There were a number of additional important lessons which were missing from the background analysis. These issues are outlined below, and discussed further in the next sections. The importance of providing a legislative/regulatory basis in order to “institutionalize” P3BM;

8 P3BM Project Paper, June 2013

4

The need to look to ways in which to integrate P3BM within existing local government agencies currently mandated to address/coordinate approaches to poverty alleviation;

Related to both of the above, an analysis of the constraints to internalization; The need to consider issues of gender within P3BM (in data collection, analysis, planning and

monitoring - gender budgeting); The implications on the quality of support available for P3BM target areas if the national training team

is also responding to requests for support to non-target areas. 2.1.2 Assessment of Project Design Diagram 1. Project Design – Outcomes and Components9 As the diagram above shows, the project’s overarching design was largely appropriate, with attention to sustainability plans, engagement of CSOs, and knowledge management. However, it has generally been interpreted as one of a capacity building program by the national and provincial secretariats/clinics, even though the project concept was more far reaching. This is demonstrated by the fact that the main inputs of the design revolved around a series of training, accompanied by required personnel and hardware support. The approach seemed to assume that sufficient training would influence ways of thinking and working of individuals, which in time would result in changes in Government systems and processes. This emphasis on awareness raising and training may be appropriate for a new project but in this case, where the methodology has been available and training has been provided for nearly eight years, more attention should have been given to institutionalization of the tools and approach as part of the design in order to make progress towards the PDO.

9 Diagram taken from the Kemitraan Completion Report, September 2015

Component 1 Component 2 Component 3

# of master trainers trained

# of P3BM provincial clinics established

# of CSOs and community groups that use P3BM instruments to track district pro-poor

expenditures

Business plan developed for the

long term sustainability of P3BM

#LG that have adopted poverty targeting in annual development

planning

Increased percentage of district LG budgets allocated for MDGs targeting

and poverty related spending

5

The design included assumptions around the links between: Delivery of training and adoption of new planning and budgeting processes by local governments (in

the absence of enabling national policy environment); Provision of training to selected individuals at the provincial level and their availability to become

master trainers in district training (in the absence of formal agreements releasing them from their jobs/full time positions);

Provision of training to community groups/CSO and their monitoring/tracking of local Government expenditure (in the absence for resources to support monitoring10);

The assumption that an increase in budget allocations to MDG related areas of government would result in improved poverty related spending and improved pro-poor service delivery by local Government.

To make the above assumptions work, greater attention should have been given to the following elements. Enabling policy umbrella. Previous reports indicate that after several phases of piloting, training and in some areas implementation, there was a need to focus attention on the issue of institutionalization of P3BM within national legislation. This was one of the key recommendations of the 2011 study11. A component and clear indicators relating to the extent to which P3BM becomes officially recognized – under the necessary legal or regulatory framework at the different levels of Government – would have been a valuable in emphasizing the importance of this issue. Integration within relevant areas of Government. P3BM was designed to be managed by provincial P3BM secretariats/clinics attached to the local Bappeda office. Efforts were reportedly made during the preparation phase to look at opportunities to better integrate the project within existing official regulated Government bodies, ideally the TKPKD12. This proposal was apparently not supported by Bappenas13, with the reasoning that the TKPKD were not yet capable of managing P3BM. The resulting stand-alone project funded secretariat reduced the opportunity for improved local ownership and “buy in” from other areas of government14, and therefore sustainability. Capacity building strategy - The design focused resources on a number of “target” locations across Indonesia. This targeting of resources was linked to the “cascade” training approach, maximizing opportunities for skills transfer and therefore the likelihood that selected provincial master trainers could themselves provide required support to district governments. At the same time however, the project was designed to allow for support for non-target locations where there was demand, with the result that significant time and energy was spent by the national trainers in non-target areas.15 It appears that the focus on quality (building capacity of provincial master trainers) has been compromised by the desire to increase quantity (numbers of locations). Support for engagement with CSOs. The background analysis and design would have benefited from greater consideration of any potential political sensitives as well as the issue the issue of longer term technical support and resourcing to support (or at least compensate) the CSOs in playing their envisaged role of monitoring of local government resources/programs allocated for poverty reduction.

10 Assuming that training was provided to CSOs/CBOs, and was of reasonable quality. 11 Strengthening effective participation of civil society and accountability mechanisms in provincial planning, budgeting, monitoring and evaluation (P3BM), Titik Hartini and Yusuf Murtiono, September 2011. 12 The TKPKD (provincial and district poverty reduction commissions), are recognized as the government body mandated to improve coordination and planning between government agencies on issues of poverty alleviation. 13 No reasons were provided in project documentation. 14 In provincial and district field visits, other staff within local Bappeda offices themselves seemed unaware of P3BM, including the Head of one Provincial office, who said he wasn’t aware of P3BM because he had been in the “head position for just over a year”. 15 Several of the KPI incorporate data from non-target area activities, which has caused some confusion.

6

Other pertinent issues that would have benefitted consideration in the design include greater attention and a more in-depth analysis on gender, and in particular discussion of the issue of gender budgeting, 16 and the role and potential influence of key stakeholders, and the incorporation an action plan/strategy for engagement with and influencing of DPRD, the local media, community at large as part of component 3. 2.1.3 Adequacy of government commitment. There is a high level of commitment from Bappenas for the roll out of P3BM, who have been the key player and “driver” of P3BM. Bappenas was actively involved in the design process for this phase, with the Seknas within Bappenas playing the primary role in the execution and implementation (through hosting and managing the PMU), and collection of data for monitoring and evaluation purposes. Whilst the support for the further development and national roll out of P3BM is admirable, the necessary attention and resources required to improve inter-governmental collaboration at the national and local levels to promote the institutionalization of P3BM was not available (refer section 5.2a below). 2.1.4 Assessment of risks and mitigation measures. A brief overview of perceived risks, their ratings (moderate) and mitigation measures was provided in the Project Paper. Though identifying a number of important issues, there were a number of gaps in the risk assessment analysis (and mitigation measures), several of which related to comments on gaps in the design noted above. The design would have benefited from greater attention to the significant risks posed by the lack of attention or commitment to institutionalization of P3BM; the project sitting outside of existing local Government bodies mandated specifically to support pro poor planning; the risks posed by lack of clarity in the design on breadth/geographic scope of the project, and how the dilution of support from the national trainers to non-target areas would impact on capacity development efforts in relatively well resourced “target areas”. 2.2 Implementation Rating: Moderately Satisfactory 2.2.1 Project launch and early implementation. When the grant agreement became effective in June 2013, P3BM was not new to the Bank or Bappenas: the UNDP/ADB project and subsequent support under the LGCD trust fund (TF098107) had developed tools and begun the piloted the approach and trained staff from a number of local governments in its use. To help bridge activities during the transition from the LGCD trust fund to the new project, the Institute for Development and Education (IDE) was contracted during July-December 2012 (with funding through the LGCD) to support the revision and finalizing of the P3BM training modules and field-test the P3BM monitoring modules. Building on these activities, the project could mobilize swiftly. At the start of this phase (January 2013), the National Secretariat was staffed and operational. Little attention was given to setting the ground work for institutionalization during this launch/early implementation period. The challenges to coordination between different areas of Government at national level were demonstrated early on by the fact that the planned National Steering Committee (NSC)17 for P3BM did not form, nor meet during the project lifetime. If established as planned during the initial stage,

16 There has been substantial work in this area in Indonesia, in particular with the national level to the Ministry of Finance, and some high quality local facilitators would have been available to assist with facilitating discussion on how gender can be better integrated into the P3BM. 17 . The NSC proposed in the design was to involve key national stakeholders, including Bappenas, TNP2K, MoHA, MoF, universities and others working on MDG targets such as UNDP.

7

it is likely that the NSC could have encouraged a sense of collaboration and joint ownership of P3BM, which could have help stimulate future efforts to institutionalization P3BM as a national program. At the sub-national level during early implementation, letters were sent to each of the local governments in the selected “target” locations inviting them to be involved in P3BM training. There was no discussion or official agreement between national and local governments in relation to respective commitments, roles and responsibilities in the implementation of P3BM. 2.2.2 Project implementation. The organizational arrangements for implementation consisted of a National Secretariat within Bappenas, with administrative support provided by Kemitraan. At the subnational levels, the project established Provincial P3BM Secretariats led by Bappeda. The Project first produced and/or built upon an existing pool of national and then provincial master trainers, consisting of Bappenas and provincial Bappeda officials, technical agency staff and university representatives. The plan was that these “master trainers” within the provincial P3BM Secretariats/Clinics (in the target areas) would serve as focal points for P3BM activities at the district level, and provide training to district technical agency staff and providing TA to local government in the adoption of pro-poor analysis. The implementation phase largely focused on the national roll-out of training. Regular coordination meetings were held between Bappenas, the P3BM national secretariat and Kemitraan to provide operational support for the training. The training was generally both well targeted and well received18. Training reports indicate a consistent high level of satisfaction with the training19. Seknas reported that the majority of participants came from staff of planning bureaus of the MDG-relevant agencies that were (at the time) tasked with drafting and revising budgets, so they need the skills and data to improve poverty targeting. Table 1 below provides an overview of participants of training during this phase of P3BM.

Training type Number of participants Comments

Master training (training of “master trainers”)

28420 This involved a series of cluster training. The first including participants from the national level government agencies, subsequent trainings included participants from provincial government and some CSOs/universities. Ratio of men to women was not provided.

District level training

423221 Participants primarily from district level government departments (approx. 92%), but also included local CSOs (6%) and universities (3%). Approximately 33% of participants female.

Throughout the project, much effort went into stimulating interest and building local government demand for P3BM. During 2014, approximately 50% of local governments trained were funded by local funds. Whilst this is admirable, due to a desire to reduce costs, training days were generally less than the one week provided under project-supported training22.

18 Paparan Capaian Evaluasi P3BM 2015. 19 Bappenas, Paparan Capaian Evaluasi P3BM 2015. 20 There are 679 people who participated in master trainer training and certified. Of which only 284 master trainers are currently capable (and available) to act as master trainers. Kemitraan Completion Report, Sept 2015, Paparan Capaian Evaluasi Seknas, April 2015. 21 Paparan Capain Evaluasi, Seknas, April 2015. 22 IDE22 had recommended initial P3BM training should include more practical sessions, and therefore should be longer than one week, and should include follow up trainings.

8

There were a number of obstacles to implementation that affected the achievement of PDO. The key obstacles, noted below, have been touched on previously in relation to gaps in the design (and are mentioned again in relation to effectiveness), so they will not be detailed again here. The limited attention given to more strategic issues of institutionalisation and internalisation, and the

inclusion of other areas of government to promote broader commitment to and ownership of P3BM; The limited progress in relation to integration of monitoring into the P3BM; The impact on the quality of support that can be provided by Seknas to target areas as a result of

responding to increased demands for support from non-target areas. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization Rating: Moderately Unsatisfactory 2.3.1 M&E Design. For a number of reasons, the KPI’s provided in the design were not very useful in tracking progress towards the PDO. There was also confusion in relation to the meaning of several of the indicators and how they would be measured. The following provides brief comments against each of the KPIs. a. 280 provincial master trainers trained – an output indicator, which has limited use in measuring

progress towards the PDO. This indicator did not consider capacity, as everyone who attended training were included in indicator reporting.

b. 60 local governments adopting P3BM in annual development planning – there is some confusion in relation to how this KPI is measured. In some districts, LGs adopted the bulk of the planning and budgeting tools (though none in relation to monitoring); in others just one tool or aspect of P3BM was “adopted”.

c. A 7.5 percent increase of direct local government budgets allocated for MDG targeting and poverty-related spending. This KPI was reportedly measured by changes in a limited number of “sample” locations23. It compared allocations for MDG-oriented government agencies (SKPD) with those of non-MDG SKPDs, with the assumption that the increase for MDG related agencies necessarily means budget is used for MDG and/or pro poor services delivery. It therefore does not consider how or where the budget increase is used, nor does it factor in the very large question of attribution. The question of attribution is especially difficult here given the National Government legislation (UU 20/2003 article 4924 and UU No.36/2009 article 17125) which states that all local Governments must progressively increase their budget allocations to health and education by 10% and 20% respectively, and that funding increased budgets are to be used for increased spending on health and education. It should also be noted there are other CSO and Government programs advocating for increased support for poverty reduction programs.

d. Establishment of 12 provincial secretariats/Clinics. The reporting against this indicator was the number of project funded the provincial secretariats/clinics and a staff member (provincial coordinator). The indicator should have been the existence of an official supporting letter from the provincial government supporting the establishment and on-going resourcing of the secretariat26.

e. 60 civil society organizations and community groups that use P3BM instruments to track district pro-poor expenditures. Bappenas changed this indicator to “number of CSOs trained”, and reported

23 Seknas reported that the 15 locations in the sample where chosen based on considerations of data availability. 24 Article 49, Law No. 20/2003 on National Education System states that “Funding for education should make up at least 20% of the national budget allocation and 20% of local budget allocations (in addition to funds allocated for salaries)”. 25 Article 171, Law No. 36/2009 on Health states that “Funding for health should make up at least 5% of the national budget allocation and 10% of local budget allocations (outside of/in addition to funds allocated for salaries)”. 26 The P3BM Secretariat needed an official letter from the Governor or head of Bappeda to officially confirm their existence.

9

against this changed indicator. Approval was not requested for the change, and it reverted back to the original.

2.3.2 M&E Implementation. The core P3BM monitoring and evaluation tools were quarterly reporting in order to provide an overview of implementation progress and to track progress of project interventions against the above KPIs. There was a strong focus in reports on numbers of people trained and location of training. After the full cycle of capacity development activities was carried out in the jurisdictions during preparation for the 2014 budget, and with the district budget approved in December 2013, the project also started to track budgets in selected locations in order to analyze the increased percentage of local government budgets allocated for MDG targeting and poverty-related spending (KPI 3 above). As well as regular meetings, the Bank fielded several missions to P3BM activities, producing 4 ISRs27. The main conclusions of these missions were: (i) the project was on track to meet its development objectives; (ii) training of trainers need to be more targeted to staff that can be deployed as trainers and (iii) the development of the Community Monitoring module should continue, in spite of some Bappenas hesitation, since this is an important post-planning tracking of expenditure. The design anticipated a significant role for Kemitraan in P3BM monitoring. However Kemitraan reportedly did not play a role in the monitoring/collection of “raw” indicator related data, instead relying on Bappenas for data on indicators for their progress reporting. Despite being relegated an administrative role, there were reportedly some discrepancies in figures provided by the National Secretariat and Kemitraan, which required follow up by the Bank for clarification and subsequent amendment. There was confusion amongst different stakeholders around the details of some of the indicators and how they were to be measured, which as noted above, lead to some indicators being (unilaterally) changed in reporting. As can be seen in Annex 2 below, a number of activities planned under Component 3 related to monitoring and evaluation did not happen as planned. These included the planned regular tracking improvements in quality of expenditures with regards to MDGs; a mid-term review and a survey to allow for the tracking of master trainers. It is not clear why these did not take place28, but it appears that despite being planned in the PCN, the initiative was considered too small to warrant a MTR (Bank), and in terms of the tracer study not taking place, there was reportedly difficulty in identifying a suitable consultant, so this did not proceed (Seknas). 2.3.3 M&E Utilization. It appears that little of the information/learning from monitoring was used to reflect on or revise approaches to implementation. More attention to addressing issues emerging through monitoring would have been useful in addressing some of the significant operational obstacles faced by provincial teams (which were raised during the ICR mission). This could have included for example: Providing support to discuss and clarify confusion within Provincial Secretariats/Clinics and other

key local stakeholders in relation to their expected roles and responsibilities, and according them some official status/title, which may have assisted them in local coordination and advocacy efforts. This may

27 http://www.worldbank.org/projects/P131815/id-tf-pnpm-pro-poor-planning-budgeting-monitoring-program. 28 At the time of ICR mission, a new TTL had just been appointed who had not been involved in implementation phase, and none of the “Seknas” team were available for any meetings (they were no longer paid for by the project). Therefore much of the communications with those involved in direct implementation and decision making on various issues had to be via email.

10

have also help address problems in some locations relating to Provincial Secretariats/Clinics’ abilities to access data from line agencies.

The issue of both availability and capacity of provincial master trainers, and the need for more targeted refresher training and mentoring support for target areas. This may have helped improve the effectiveness of the planned “cascade training”, building the capacity of provincial teams and therefore reducing their reliance on Seknas support.

Given that it became clear early on that Kemitraan would not be playing its envisaged role in monitoring (nor providing strategic advice), the project would have benefitted from both Seknas and the Bank team taking a more active role in monitoring progress - assuming Bank time and resources could allow this. This would have been useful both in relation to field visits and discussions with LGs and Seknas, and in ensuring that results of monitoring are discussed and subsequent approaches amended to reflect the results of monitoring.

Any follow up initiatives would benefit from a thorough review of the M&E system, ensuring that the indicators used are meaningful (in terms of contribution to the PDO), practical, and so can form the basis for reflection on progress and where appropriate revision of project strategies. It is also important that the indicators and objectives of the M&E framework are well understood and agree by all stakeholders, and that relevant staff have sufficient capacity to undertake the planned monitoring.

2.4 Safeguard and Fiduciary Compliance Being largely a capacity development and training program, there were no significant long term potential safeguard impacts triggered by this project and the project was rated as Category “C.” Compliance with Current Environmental Assessment Category was not required The projects fiduciary and governance requirements were sound. An agreed Standard Operations Manual was developed which clearly outlined operational processes and responsibilities, and Kemitraan finance staff were tasked with managing and monitoring expenditure. Independent audits were to take place annually. Only one actually took place, and not until after the end of the project (4 months overdue), so the results were not available at the time of the ICR. Overall financial management was reportedly in moderately satisfactory, due to delays in several of the IFRs and the project audit(s). 2.5 Post-completion Operation/Next Phase As part of the broader PSF transition strategy, P3BM transferred out of the Bank for consideration by the Australian Department of Foreign Affairs and Trade (DFAT) at the end of the grant agreement on June 30, 2015. This transition reportedly provided the opportunity to review the project’s design and achievements to date, as well as how P3BM could be adapted to the new Village Law framework29. The Bank task team assisted Kemitraan and the National Secretariat in drafting a business plan and possible funding under DFAT, and was in consultation with PSF's JMC and DFAT to ensure the orderly transition of the project out of the World Bank. Given that the business plan is still in draft form, there is an opportunity to ensure it provides practical steps and timelines towards institutionalisation, in view of P3BM’s eventual mainstreaming and recognition within government regulations, systems, or procedures. Having this supporting institutional framework in place should ostensibly ensure that any future phases of P3BM do not face the same obstacles and limitations faced by previous phases of the project.

29 There has been talk of training villages in the use of P3BM, in particular for data entry (and analysis) so as to track the use of village funds. This seems both impractical and an unnecessary bureaucratic burden for village governments (especially considering issues of computer literacy, access to electricity, internet in many villages). This data will be available for entry by district level if considered useful.

11

3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation Rating: Modest As noted above in section 1.2, the Project Development Objective (PDO) is considered relevant to the CPS. The PDO was however too ambitious for a relatively small and short term initiative which focused largely on a series of trainings. The link between activities and outcomes was tenuous and (as noted in 2.1.2) involved a range of assumptions that were either not addressed and/or unrealistic. The design (or methodology) used in delivering the training program is considered largely relevant and appropriate to decentralized Indonesia. It was framed around a cascade training approach (using national and local trainers), and involved bringing together key local government stakeholders to reflect on and plan local development initiatives. One of the obstacles to efforts to make local governments more responsive to the poor is the lack of evidence and practical tools they could use to both argue for an actually provide alternative budgets that better reflect poverty priorities. The approach to providing practical toolkits and software to local government agencies to allow them to come together to assess whether plans and budgets was highly relevant, and provided much-needed methodologies and rationale to allocate more funds for MDG-related activities. Using MDGs as the benchmark towards which poverty was measured meant that local governments could be measured against each other and national targets. This was to be further enhanced by the incorporation of the lessons from the LGCD pilots were incorporated into the design, in particular the roll out of the expenditure tracking modules. The focus on training that characterized earlier phases was to be strengthened by making it possible to track the actual expenditures of the MDG sectors, making sure that the local P3BM indicators were used to target the poor. This phase of P3BM continues to demonstrate that reform-minded local governments, given evidence and support, could reallocate budgets to better match the needs of the poor. Where there is sufficient political will, the toolkit is able to be put to good use, providing committed governors and mayors (and Bappeda staff) with the data, evidence, and arguments required to advocate for increased allocate funds of funds for health and education.. In the absence of a policy umbrella, the project was thus most successful when it could be combined with reform-minded local leadership. In terms of implementation there was a lack of due attention on the part of the Bappenas/the Bank in relation to how the project could interact with, and be integrated into, local government planning. There was also limited understanding of the challenges posed by a deeply entrenched bureaucracy and rent-seeking political economy, in the absence of a policy or firm regulatory framework at its basis. The Presidential Regulation No. 5/2010 on Midterm Development Plan 2010-2014 had adopted mainstreaming MDGs as key development target. Based on this, it would have been possible for Bappenas to issue a circular letter to emphasize the responsibility of local Governments to adopt P3BM in planning and budgeting. Whilst the PDO remained highly relevant, the mechanism used to reach to objective - the P3BM toolkit - which is framed around MDGs, has become less relevant with the change to SDGs. 3.2 Achievement of Project Development Objectives Rating: Modest There are three aspects to the PDO - to improve poverty targeting of local government expenditures and local government responsiveness to community needs through the scale up and institutionalization of the P3BM program. Each of these areas are addressed below.

12

Poverty targeting of local government expenditures has improved in many locations by allocating more funds for core MDG related public services. Based on analysis of 15 pilot districts that received P3BM training in 2013-14 (from both “target” and “non-target” locations), the project could track an average increase of 9.3%30 increase on direct spending by MDG-related SKPD31 in comparison with non-MDGs spending (see Annex 2 for a summary). Setting aside the significant question of attribution noted previously, this is a strong increase, showing that after receiving the training and using the P3BM tools, significantly larger allocations were made to MDG-related SKPDs compared with previous years and non-MDG agencies. However in terms of “poverty targeting”, as the project did not track details of these increased allocations, and as the planned attention to SKPD/CSO monitoring/tracking of expenditure did not take place, it is not known whether an increase in funding has actually resulted in better SKPD targeting of poorer sub-districts, or service delivery for poorer people within the district generally. A significant strength of P3BM was in promoting the recognition of the importance of the availability of reliable data as the necessary basis for sound planning. It has supported significant progress in data collection (and to some extent analysis) through data forums and “database launching”. An assessment of the extent to which this PDO has been achieved is hampered by the limited availability of relevant data. This is due to both the small sample size (and advice that one of the selection for criteria for districts inclusion in the sample was data availability), and the lack of monitoring data from the rest of the districts involved in the program. Local government responsiveness to community needs can be said to have improved by making local government agencies more aware of an attentive to poverty data and expenditure on MDG related areas of government. Of the 85 LGs who have adopted the P3BM tools, 73 LGs have used it to set the local budget plan and 12 LGs used it to analyse the local MDGs data. The project stimulated coordination between Bappeda and SKPD from provincial level down to sub-districts through data collection activity and management of local database system with the establishment of MDG Data Coordination Forums led by Bappeda. The Bank reports that observations during missions and post-training assessments showed that the government staff trained had the capacity and ability to target budgets to meet MDGs. P3BM training is considered practical in the sense that during the training, each participant completed an analysis of their local budget and provided recommendations for possible budget re-allocations. They showed increased awareness of the drivers of inequality, and the ability to allocate money to solve those bottlenecks – whether to address pockets of tuberculosis in one particular sub-district or the prevalence of primary school pupils dropping out in another. The P3BM tools were able to provide them with the data, analysis and tools on how to reallocate funds to address poverty in their jurisdictions. Lack of availability of data also hampers more rigorous assessment of this aspect of the PDO. The failure to roll-out the Community Monitoring module, which was reportedly resisted by Bappenas, limited the ability to verify the extent to which “community needs” are addressed. Nevertheless, more than 100 local CSOs staff were involved in general P3BM training, with a portion of them likely to be interested in engaging with P3BM in the future. The design envisaged PNPM facilitators playing a significant role in supporting community linkages. This may have provided a significant potential resource for the program if the linkages could have been

30 This figure differs from the 6.5% provided in the final ISR, slightly less than the 7.5% target. As noted above, this figure is reduced further to approximately 5.5% if limited to the 7 target areas within the sample. 31 That is health, education, community empowerment, water and sanitation, and environmental services.

13

established. Where linkages were made, this created a very useful asset for the work of P3BM, both in terms of “reach” and in supporting P3BM advocacy efforts. The scale-up of the P3BM program was supported by the training of significant numbers of (primarily) local government staff across Indonesia, both using project funds and/or combined with local governments own budgets/resources. There is reportedly still a strong demand for additional support from the P3BM Seknas which has been providing the training, but it is unlikely that these requests will be accommodated without additional ongoing donor funding to support the operations of Seknas. There were however negative impacts of the scale up relating to a diluting of time and resources available to build a competent cadre of master trainers, who would be able to provide training without continued assistance from national level trainers. One indication of “institutionalization” of aspects of PP3M may be found in the results from a rapid assessment conducted in 2014. The assessment concluded that of the local governments that received basic P3BM training since 2008, over 80 percent of them were using at least some part of the tools to help them conduct their planning. As noted above however, lessons from previous phases confirmed that a national policy umbrella is necessary before any real institutionalization can take place. To summarize, where it has worked, P3BM has been effective in improving national and local government awareness of local development progress and needs against MDGs; has improved coordination across different areas of local government and the skills, capacity and commitment of selected/committed individuals. There are examples where it has been effectively used as an advocacy tool for government, to encourage greater attention and resource allocation to issues of poverty (as identified in MDGs). A strong foundation has been laid to take this approach to the necessary next step, a national level policy umbrella. 3.3 Efficiency Rating: Substantial Attempting to rate the efficiency of initiative such as P3BM is difficult, as it has been primarily a capacity development and training program, and as such, there are no physical outputs to which a financial analysis of efficiency can be done. Comparison with other capacity building projects is not useful, as it ignores the issue of quality of training, and the extent to which training is /can be used, and to what effect32. A number of issues should be considered in judging the efficiency of P3BM. These issues, noted below, are touched on in other areas of the report, and further detailed in annex 3.

- The number of people trained (substantial); - The participants level of satisfaction with the training (reportedly very high); - The capacity of trainees to perform their envisaged role post training (not quantified, reportedly

highly varied); - The proportion contribution of funds for the training from the project compared to local government

budgets (reportedly 1:3); - The outcomes achieved in well resourced “target” locations compared with “non-target” locations

(negligible difference);

32 This type of analysis would be further confused (and less useful) because of the different types of training support provided varied between target and non-target locations, and the depended also on the availability of provincial level master trainers to co-facilitate district level trainings (some of which could have been attributed to previous phases of P3BM).

14

- The efficiency of the management structure/arrangements (limited, due to duplication/overlap of some roles of staff in Seknas and Kemitraan, and the resulting waste of resources);

- The attention given to creating an enabling environment in which new skills and processes can be sustained in the future (limited).

Based on the above concerns, it is considered that P3BM provided moderately satisfactory value for money. With an investment of USD1.67million over a period of 2½ years, the program provided at least introductory training to significant numbers of local government staff, with significant cost-sharing from local governments. 3.4 Justification of Overall Outcome Rating Rating: Moderately Unsatisfactory Combining the three elements of outcome, based on relevance (modest), achievement of PDO (modest), and efficiency (substantial) as described above, the rating of moderately unsatisfactory reflects the Project’s overall outcome33. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development P3BM has been operating as a local government capacity building program, and as such, it is not possible to draw direct links between the effects of P3BM and poverty levels. The Project can however be said to have contributed to increasing attention to some of the key social development challenges in Indonesia relating to improved basic public service delivery. This has been done by encouraging analysis of existing budgets, and arguing for and encouraging higher budget allocations for basic services such as school education, hospitals and health clinics. The lack of monitoring data, or any analysis of use the actual use of any additional funds allocated for “MDG related” areas of government, means that it is not possible to determine the extent to which the project has benefitted the poor (has been pro-poor), women, or other traditionally disadvantaged groups. In relation to gender, as several of the MDGs are important in terms of disproportionate impact on women (i.e. maternal mortality), P3BM could be said to have contributed to increased attention to an issue that impact more heavily on women. The local women’s empowerment agencies were involved in trainings, but it is unclear the extent to which they have been able to use the methodology to advocate for increased support for women’s empowerment, as no data was available. The methodology did not directly address MDG indicators such as women’s political representation. It was a missed opportunity that the P3BM methodology did not integrate aspects of gender budgeting, to encourage local governments to review their allocations and planned factoring in issues of gender34. The P3BM toolkit, including the recently finalized monitoring tools, are largely “gender blind”, in that little attention is given to the need to monitor and analyze service delivery in relation to access and benefits for women and men. (b) Institutional Change/Strengthening

33 Overall rating is based on guidance provided in the ICR Guidelines Updated 7-22-2014, Appendix J. 34 In the past there has been some quality initiatives in support of improving GOI capacity in relation to gender budgeting, much of it with CIDA funding (with Asia and Ford Foundations), working with the Ministry of Finance.

15

The project has reportedly strengthened the capacity of selected staff from the subnational development planning agencies - Bappeda and TKPKD. Where it worked well, it also improved coordination on planning. Where it didn’t work well, roles and responsibilities of different actors were unclear, staff trained as trainers were either not capable and/or not released following the training, required data was not provided. The lack of willingness and/or capacity to engage different government stakeholders in cooperating to progress an policy framework or umbrella for P3BM calls into question the breath of support for P3BM outside the Bappenas executing Division of Poverty, Social Affairs and SMEs. (c) Other Unintended Outcomes and Impacts Although not explicitly stated in project documents, by providing a framework and toolkits, P3BM is influencing the “mindset” of local government officials involved in planning. As noted above, in some areas P3BM is being used as an “advocacy tool” for committed elected representatives and concerned local government staff, allowing them to argue for more for budget allocations based on data Government agencies themselves have provided. In many areas, the project has reportedly resulted in substantial improvements in communication, collaboration and joint analysis amongst individuals from different local government agencies around issues causes of poverty and how these can be addressed. This is a significant achievement which is considered a noteworthy outcome of the project. Another positive unexpected outcome was reported in Banten, West Java, an area with a considerable manufacturing sector. Local Bappeda staff reported using P3BM data in their discussions with the private sector in relation to possible uses for their CSR funds. An unexpected negative impact is that P3BM is reportedly causing confusion in local governments, with regards to what is the official government requirement for reporting against MDGs (the local TPKD used one reporting process and supporting software, which was different from that of P3BM – although Bappeda is the secretary for TPKD, and hosts the P3BM secretariat). Improved coordination within Bappenas/Bappeda and with other areas of Government will help to reduce this confusion. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops There was no beneficiary survey undertaken, and no issues raised at the final meeting of national stakeholders that have not been included in this document. 4. Assessment of Risk to Development Outcome Rating: Significant The “Risk to Development Outcome” is defined as the risk that the development outcomes of the project––at the time this ICR was prepared––will not be maintained. As noted above, official project/Seknas data highlights the fact that 67935 people at the province level have been provided with training to become P3BM “master training”, and significant numbers of district government officials have been trained. The P3BM has also now been included in the curriculum of Bappenas training institute (Pusbindiklatren), and so can be incorporated in routine Bappenas staff training. The project has also supported the establishment of 15 provincial secretariats/clinics, which included

35 Seknas estimates that approximately 41% of these, 284 master trainers, are now available to support/co-facilitate district training.

16

support for the salary of a provincial coordinator (recruited on contract from outside of the public service). There is a large demand for P3BM services, measured by the fact that in 2014 and 2015, more than half of project interventions carried out were funded by local governments directly. Senior leadership figures in Bappenas – including the Minister and several Deputies – have taken part in a number of P3BM national and provincial events. A key risk to development outcome is the institutional issue raised a number of times previously, the lack of progress towards a supporting policy umbrella for P3BM, so that P3BM becomes officially recognized as a part of planning systems and processes. As has been noted in previous reviews and again during stakeholder consultations for this ICR, though P3BM may be considered useful and relevant to local government development planning needs, as long as it is not official sanctioned, and has no legislative basis, it will have limited sustained influence on local government ways of working. A second important risk relates to the lack of capacity of provincial level “master trainers”. Despite the large number of people trained, it is generally agreed that none of the provincial teams are yet able to independently provide the required training for district governments without support from national secretariat. A third risk relates to the change from the MDGs to SDGs for the future value of P3BM. The implications of this change to P3BM methodology is yet to be discussed. Finally, the new policy environment of the enactment of Village Law is also likely to influence the P3BM approach. Under the Law, there is a significant change in the division of tasks and responsibilities in local development management, including in relation to achieving MDGs/SDGs and other pro-poor budget policy. Under the Village Law, 10% of national budget would be allocated and transferred directly to Villages. This will allow much greater autonomy and authority for villages themselves to determine use of their development funds. Taken these important factors into account, the risk rating is considered significant. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory The strengths of the Bank in terms of ensuring quality at entry included: Responding quickly when a request came from Government for a separate RETF, acting to ensure pre-

financing for the project, and ensuring that key actors (trainers) were continued through the transition phases;

Project design was conceptualized in response to some of the findings and lessons from the BETF-managed pilots;

The identification of the need for a national Steering Committee, and the development of a P3BM “business plan”, to being to address issues of institutionalization;

The Bank worked closely with Bappenas to select a highly capable non-governmental organization to implement the program;

The project contributed to key GoI objectives at the time, including building capacity of local governments.

17

As noted above, Bank staff advised that due to the relatively small size of the grant for this phase of P3BM, a PAD was not required for this project36. A Small Grant Project Concept Note was prepared for internal bank appraisal in August 2012, and a Concept Paper Decision Note provided in November, 2012. Whilst giving in principle approval to proceed, the decision note highlighted three areas requiring attention: 1. Implementing Agency – the Bank reviewers noted that further clarity was needed in regards the

working relationship between Kemitraan as grant recipient and Bappenas as implementing agency; 2. Results Indicators – the reviewers noted that the proposed results indicators are output rather than

outcome orientated, and needed to be better aligned with the PDO; 3. Country Context – There was a need to provide better elaboration on the overall strategy in raising local

government’s service delivery capacity, and the role and achievements of related projects. It appears that these revisions did not take place, as there are no changes reflected in the contents of the final Project Paper (June 2013) or Grants Agreement – with the project being effectively as it was when submitted in the Project Concept Note. It also appears that some of the questions raised (internally) in relation to data provided in the ISR were not followed up on. (b) Quality of Supervision Rating: Moderately Unsatisfactory Throughout project implementation, the Bank communicated regularly with the implementing agency and/or Seknas. Supervision was reportedly frequent, funded in part through a BETF at the PNPM Support Facility, and ISR reports prepared in a timely manner. As noted above, towards the end of the project period, Bank staff worked closely with the project national secretariat to ensure the smooth transition of the Project out of the Bank. While able to maintain a high level of communication during implementation, the rating has been lowered due to the fact that the Bank failed to follow up on a number of important issues, several of which were identified during ISRs. These included the delays in the preparation of the baseline study, not carrying out the tracer study of master trainers, or supporting the conduct of a mid-term review as planned. There seemed also to be no follow up in relation to the delays in drafting of the business plan, or in relation to identifying risks associated with the ongoing delays in the finalization and training in the Community Monitoring module37. The Bank, Bappenas, or Kemitraan do not appear to have been able to adequately address the various problems raised by provincial master trainers in relation to the difficulties they faced as a result of lack of official status of the P3BM,38 or other difficulties which impacted negatively on project efficiency and effectiveness. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Unsatisfactory

36 In accordance to Procedures for Small Recipient-Executed TR Grants issued on March 30, 2012, small grant project required to prepare PID (not PAD). For the purpose of this document, the Project Paper will be referred to as the design. 37 This included the use of the associated KPI revised (unilaterally) in the implementation progress reports to number of CSOs trained in the toolkit rather than the original/agreed indicator of the number of CSO using toolkit for monitoring. During early ISR the task team also adopted/used the number of CSO trained in the toolkit, an oversight which as only rectified in the last ISR. 38 The range of difficulties noted previously, including problems in getting release from current jobs to be able to act as “master trainers” as planned, the difficulties in accessing data from line agencies in the absence of any official authorization, policy/legislative backing for P3BM.

18

Whilst providing quality input at various stages during preparation and implementation, the weakness above lowers the rating to moderately unsatisfactory. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Satisfactory The staff within Bappenas/Bappeda who have been involved in P3BM have been able to effectively use the toolkit to build increased attention and skills in using MDG indicators to guide local budget planning and budgeting. There is a high level of enthusiasm and commitment to P3BM within Seknas which is highly commendable. The focus on training however seems to have come at the expense of attention to addressing more strategic issues required after many years of piloting and implementation of training. The moderately satisfactory rating results from a lack of attention and/or action in a number of important strategic areas, including the formation of the planned National Steering Committee39, attention/progress towards the integration of P3BM into national policy or regulations, formal agreements with local governments in relation to roles and support mechanisms (and the resulting problems in several areas of lack of master trainers and their access to data), as well as lack of attention or progress on the issue of incentives40. The reluctance to progress with the “CSO monitoring” aspect of the project was also considered a missed opportunity, as was the lack of cooperation with Kemitraan in this (and other) issues where they will have been able to provide strong strategic input. The final area worth noting was the belated attention to the development of the P3BM “business plan”, and seeming lack of ownership of this potentially important document (now still in draft form). (b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory Kemitraan was competitively awarded the grant to implement P3BM. The organization had a previous track record of working with the Bank, with a focus on pro-poor governance. Unfortunately the full range of Kemitraan’s capabilities was never brought to the Project. Instead, Kemitraan kept the project at a hand-length distance, mainly fulfilling its fiduciary and reporting responsibilities. There were also some discrepancies between Kemitraan and the National Secretariat for KPI and financial reporting that, although ultimately reconciled, caused some tensions. Kemitraan provided pre-financing for the project when it was awarded, this meant that there we no delays in project implementation. Procurement was generally on time but there were delays in a number of financial reports. As noted above, there were also slippages and/or delays in auditing41. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory Based on the issues summarized above, the overall “borrower” performance is considered moderately satisfactory.

39 It is not clear why this did not take place, was not prioritised. The funds allocated to support the national steering committee were apparently reallocated to training. It is not clear whether this had been discuss or approved by Bank staff. 40 This, as noted above, was flagged as a lesson learned in the project paper, which would be given attention during this phase. 41 Only one of the two planned audits took place, and this has not yet been completed.

19

6. Lessons Learned Below provides a brief overview of selected lessons learned from this phase of P3BM. Background Analysis

There is a need to ensure adequate attention is given to potential institutional obstacles which may impact implementation, as well as a discussion of possible ways these can be addressed. This analysis, and the resulting strategies, should be clearly integrated into project design and resourcing.

There is a need to increase attention to gender issues in background analysis, and the extent to which the project can contribute to efforts to promote gender equality. This could include for example opportunities to integrate aspects of gender budgeting into different functions of the toolkit, both in terms of budget analysis and monitoring of local government expenditures.

Design (and implementation)

Official agreements/MOUs between different stakeholders should be made prior to commencement of initiatives. This should include agreements between national Government and LGs, which clearly outline agreed roles, responsibilities, and resource allocations, including commitments to staff capacity building, the release of staff to support P3BM implementation, and support for local data forums. This may lessen the impact of problems causes by the frequent circulation of government staff. These negotiations should also include discussion (and agreement) on the most appropriate location for a P3BM secretariat/clinic within each local government, which may increase the opportunity for P3BM to be integrated/internalized within existing government bodies, for example as a working group within existing, multi-sectoral local government poverty reduction task forces - TKPKD, which have the official recognized and mandated role in relation to coordination of local poverty reduction efforts.

Engagement and collaboration with CSO as national level implementing agency is an interesting model, which had the potential to greatly contribute to the development objective. Experience has demonstrated that is necessary to ensure that all parties are clear on roles and responsibilities, with attention to respective strengths of each stakeholder, and where they are best able to add value.

Monitoring and Evaluation

Greater attention should be given to undertaking an in-depth analysis of the key factors or assumptions that could impact the achievement of PDOs. These factors should be used as a basis for determining the timing and staging of implementation. For example, a key factor would be the need to progress the development of a regulatory framework for P3BM. The steps towards this should be articulated, with evidence of progress forming necessary milestones required before further roll out of trainings.

7. Comments on Issues Raised by Grantee/Implementing Agencies/Donors (a) Grantee/Implementing agencies The Completion report of the implementing partner, which was prepared in collaboration with the executing agency, highlighted a number of important issues based on lessons learned from implementation. These

20

recommendations42 included 1) the need for a policy umbrella for P3BM; 2) the need to address the confusion caused by multiple planning tools provided to local governments 3) the need for greater attention to data (availability and accuracy, and weaknesses in capacities to analyze data); 4) the need to review and revise the capacity building strategy. (b) Co-financiers/Donors N/A (c) Other partners and stakeholders N/A

42 Kemitraan Completion Report, September 2015.

21

Annex 1. Project Costs and Financing

(a) Project Cost by Component (in USD Million equivalent)

Components Appraisal Estimate

(USD millions)

Actual/Latest Estimate (USD

millions)

Percentage of Appraisal

Provincial Cascade Training 0.17 0.18 106.04% Technical Assistance for Provinces and Districts

0.66 0.63 95.39%

Institutional Support and National Secretariat

0.83 0.85 102.01%

Total Baseline Cost 1.67 1.66 99.80% Physical Contingencies - - Price Contingencies - - Total Project Costs 1.67 1.66 99.80% (b) Financing

Source of Funds Type of

Cofinancing

Appraisal Estimate

(USD millions)

Actual/Latest Estimate

(USD millions)

Percentage of Appraisal

Trust Funds 0.00 0.00 Indonesia - Program for Community Empowerment

1.67 1.66 99.80%

22

Annex 2. Outputs by Component This annex summarizes the outputs of the project, and is largely based on a review of the achievements of the outcome and intermediate results indicators as provided in the final ISR and Completion Report of the implementing agency. Table A2.1 below provides a complete overview of targeted and achieved values for KPIs. Component One: Provincial Cascade Training (ToT) for Master Trainers. This component financed the provision of: Training of Master Trainers. 679 people from across Indonesia have been trained in the P3BM methodology and toolkit. Of which only 284 master trainers are currently capable (and available) to act as master trainers The training started in 7 regional 5-day events organized by the National Secretariat. Provincial training were conducted in 18 provinces, bringing together a provincial average of 60 district government officials from Bappeda and MDG-relevant technical SKPD agencies to provide hands-on training on how to make budgets more sensitive to MDG objectives. Component 2: Technical Assistance for Local Governments. This component financed: Provincial P3BM Secretariat/Clinics: The project has established 15 provincial P3BM secretariat/clinics that have provided training to more than 4,232 (primarily) local government staff. Of which, only 10 of the provincial secretariats obtained official letter (SK/decree) of support from Governor or head of Bappeda In addition, 51 districts and cities funded P3BM training using their own APBD budgets. Together with the MDG Provincial Offices, Provincial P3BM Secretariat/clinic supported an MDG database and other poverty-related data needed for the implementation of P3BM and the roll out the compulsory “Regional MDG Action Plan” (Rencana Aksi Daerah MDG). Eventually, districts would independently manage the local database through “Data Coordination Forums (“MDG Forum Data”) with the central coordination at the Local Development Planning Agency (Bappeda). Civil Society and Community Monitoring. Under this sub-component, two new modules in public expenditure MDG monitoring were finalised: one for SKPD (internal government) tracking and monitoring of plans vs. actual expenditure; and one on community monitoring, through which NGOs and community groups would monitoring the quality of public expenditure related to MDG targets. The former has been incorporated in the full “P3BM Package”, while training was provided to more than 100 civil society organizations on how to use the community monitoring module, no CSO tracking has taken place. Component 3: Institutional Support and National Secretariat National Secretariat consisted of 6 professional and 6 administrative staff was established in an office space provided by Bappenas. Core support for the National Secretariat, to be hosted at Bappenas, consisting of six technical staff and a small secretariat. Information and knowledge management. This involved tracking improvements in quality of expenditures with regard to MDGs, documenting best practices, and regular publishing P3BM newsletters and updates on national MDG/poverty targets. Database Management. A pilot project has been conducted during Phase 1 to build a database and identify sources of data for districts targeted to receive P3BM training. During Phase 2, this database system will be refined and populated in collaboration with the SIMPADU team. The database will be hosted by Bappenas and be made available to P3BM cadres and local governments. The operational system is

23

elaborated in the third P3BM book on MDG Database. The project will link to the P3BM database system with the MDG Action Plan Database system existing at the province level. Collaboration with TNP2K on poverty data, poverty reduction outreach, and support to TKPKD. The aim is to encourage a “one stop shop” for poverty reduction data at the district and provincial levels. Analytical studies and monitoring and evaluation. This will consist of a base-line, a mid-term and end of project surveys that will allow for tracking of outcomes of key performance indicators on the project. The following table provides an overview of achievements versus targets, and is based on data from the Kemitraan Completion Report, version September 2015. The data was reportedly provided by Seknas. Table A2.1: Key Performance Indicators

No INDICATORS Target Achievement

(30 June 2015) PDO LEVEL RESULT INDICATORS

1) # of local governments that have adopted poverty targeting in annual development planning

60 73

2) Increased percentage of direct LG budgets allocated for MDG targeting and poverty-related spending

7.5% 9.3%43

INTERMEDIATE RESULTS INDICATORS

Component 1 Provincial cascade training for Master Trainers

a) # of master trainers trained 280 284

Component 2: Technical Assistance for Provinces and Districts

a) # of P3BM provincial secretariat/clinic established 12 15 b) # of CSOs and community groups that use P3BM

instruments to track district pro-poor expenditures 60 944

Component 3: National Secretariat and Implementation support

a) Business plan developed for the long-term sustainability of P3BM

Business plan implemented

Draft business plan prepared.

b) District-level MDG data improved Databased used 74 LGs used P3BM database

43 The figure provided in the last ISR seq. 4 was 6.5%. 44 Completion report provides this figure. However it appears this indicates the number involved in practical/hands on training during the piloting which took place towards the end of the project (not in actual, independent, and tracking of expenditure as planned).

24

Annex 3. Economic and Financial Analysis The project financed investments in capacity building of local government officials and the establishment of provincial P3BM secretariats/clinics, who would provide technical facilitation to the local government planning and budgeting process to ensure that investment were made in pro-poor and MDG public services. Because most of the benefits of such investments cannot be measured in monetary terms, it was not deemed useful to estimate indicators such as NPVs or EIRRs. Following Bank guidelines, the economic benefits of the project were therefore analyzed with respect to: (i) the project development objective––which was considered highly appropriate––and (ii) net benefit analysis, to show to what extent the project represented the least-cost way of attaining its stated objective.45 A budget tracking analysis was carried out of 15 districts that in 2013 received the P3BM training. In accordance with the key performance indicator, this analysis measures increased allocations for MDG and pro-poor services by comparing MDG-oriented SKPD with non-MDG SKPD, with the assumption that the increase for MDG agencies are higher (the target is a 7.5 percent increase). The MDG SKPDs are:

1. Forestry, Agriculture, Fishery, Plantations (Goal 1) 2. Education (Goal 2) 3. Community Empowerment, Family Planning (Goal 3) 4. Health, Sanitation, Water Supply (Goal 4, 5, 6) 5. Environment, Disaster Prevention (Goal 7)

The analysis provided in table 2 below is taken from Kemitraan/Seknas Completion Report (p12). It compares the average budget increase between 2013 and 2014 of the two sets of local government agencies (SKPD). A few outliers have been disregarded, when for instance a small SKPD re-organised and received significantly larger allocations. Table 2. The Increase of MDGs Budget in 15 Sampling Areas (APBD Year 2013- 2014)

45 In accordance with section 4 of OP 10.04 - Economic Evaluation of Investment Operations, September 1994.

25

Unfortunately, this analysis included location from target and non-target areas, and increase (and decreases) occurred in both target and non-target areas. It is there difficult to estimate the benefits of additional resources (secretariats/clinics, staff etc) provided to target areas. A number of additional factors relating to project efficiency should also be considered in an analysis of the financial or economic benefits of the project. These issues, touched on in 3.3 above, are further elaborated below. It could be argued that P3BM is quite efficient because of the large numbers of people involved in training for what was considered relatively moderate cost46. In judging the efficiency of the training, it is necessary also to consider the effectiveness (outcome) of the training, that is, whether the training has provided the necessary skills and capacity to do perform the given task/role, as well as the extent to which the operating environment allows for those with improved capacity to use their new skills effectively. Whilst there are reportedly high levels of satisfaction amongst district level participants, Seknas has estimated that only approximately 41% of those involved in training of “master trainers” would have capacity to provide training. All provinces still require support from national level for any district level training, which involves significant additional costs such as those related to travel and per diems, national facilitator fees etc. It is also not clear how many of the significant numbers of district government official trained are actually then able to integrate P3BM tools and principles into their work (in the absence of supporting policy/regulations). There has been substantial demand for P3BM training from “non-target” areas for training. Of the number of districts involved in the training, the ratio of target to non-target areas is 1:3 (in non-target area, project funds pay salaries and travel costs of national trainers, but local governments cover local costs for venue and participants etc). APBD budgets have funded training in 48 locations (IDR 8.7M), compared to the 31 locations trained with P3BM funds (IDR 6.2M). Whilst the demand is clearly commendable, the desire to respond to demands from non-target area has meant that support from Seknas (national training team) has been diluted. As a result, none of the “master training teams” in the target provinces yet have the capacity to train without significant support from national level – and with that a significant additional expense. An additional point should be made in relation to the relative efficiency of the additional resources provided to target areas - staff, hardware, secretariat and clinic support. The data used for measuring the project KPI of increased budget allocation for MDG targeting was based on a sample districts from target and non-target areas (ref table in annex 3). Interestingly, whilst the increases (and the decreases) in MDG related budgets allocations occurred in both target and non-target areas, the average increase in projects target areas (5.5%) were less than the increase in non-target areas. Assuming this information is correct, it calls into question the benefits of the additional resources provided to target areas, and the relative efficiency of the projects capacity building model. Outside of the issue of effectiveness, it is also worth questioning the efficiency of engaging and paying for staff and overhead costs of a third party (Kemitraan) to effectively provide administration for the project, particularly given the project was also paying for administration staff within the national secretariat.

46 A recent project, similar in scale and with significant attention to capacity building, CC2 (Creative Communities II), disbursed a total of USD 2.112mil for 4,800 beneficiaries or cost about USD 440 per capita, over a 4 year period.

26

Annex 4. Grant Preparation and Implementation Support/Supervision Processes (a) Task Team members

Names Title Unit Responsibility/

Specialty Lending/Grant Preparation Jan Weetjens Lead Social Development Specialist EASID Sector Manager Kevin A Tomlinson Senior Operations Officer EASID Portfolio Management Hans Antlov Sr Social Development Specialist EASID Task Team Leader Yogana Prasta Operations Advisor EACIF Operations Imelda Luciana Noto Trust Fund Coordinator EACIF Trust Fund Melinda Good Senior Counsel LEGES Legal Unggul Suprayitno Sr Financial Management Specialist EAPFM Financial Management I Gusti Ngurah Wijayakusuma FM Analyst EASFM Financial Management Ahsan Ali Lead Procurement Specialist EAPPR Procurement Achmad Zacky Wasaraka Procurement Analyst EAPPR Procurement Dennie S Mamonto Consultant EASID Safeguards Juan Martinez Senior Social Scientist EASIS Safeguards Muhammad Fithriadi E T Consultant EASID Project Management Purri Andriaty Subana Consultant EASID Project Management Rani Dewi Maharani Team Assistant EACIF Operations Chatarina Ayu Widiarti Program Assistant EACIF Administration Supervision/ICR Kevin A Tomlinson Acting Program Manager GSURR Program Manager Hans Antlov Sr. Social Development Specialist GSURR Task Team Leader Bambang Soetono Social Development Specialist GSURR Task Team Leader Julie Klugman Consultant GSURR ICR consultant Yogana Prasta Operations Advisor EACIF Operations Imelda Luciana Noto TF Coordinator EACIF Trust Fund Jana Halida Uno Operations Officer EACIF Operations Mariangeles Sabella Senior Counsel LEGES Legal Ria Nuri Dharmawan Associate Counsel EACIF Legal Unggul Suprayitno Sr. Financial Management Specialist GGODR Financial Management I Gusti Ngurah Wijaya Kusuma FM Analyst GGODR Financial Management Tatong Permana Anggrimulja Consultant GSURR Financial Management Achmad Zacky Wasaraka Procurement Analyst GGODR Procurement Ahsan Ali Lead Procurement Specialist GGODR Procurement Patricia Astiani Team Assistant GSURR Administration Chatarina Ayu Widiarti Program Assistant EACIF Operations Narae Choi Young Professional YPP Portfolio Management Niruban Balachandran Operations Officer GSURR Portfolio Management Juan Martinez Senior Social Scientist GSURR Safeguards Hanggar Irawan Operations Analyst GSURR Financial Management Festina Lavida Operations Officer GSURR Financial Management

27

(b) Staff Time and Cost

Stage of Project Cycle Staff Time and Cost (Bank Budget Only)

No. of staff weeks USD Thousands (including travel

and consultant costs) Lending

FY12* 8 242,794 Total: 8 242,794

Supervision/ICR** FY13 34 133,105 FY14 25 102,136 FY15 31 80,257 FY16 3 29,788

Total: 84 345,286 * In FY12, P3BM was one of studies under LGCD project (BETF - TF098107). LGCD project conducted P3BM Impact Assessment in 10 districts and piloting P3BM training for 32 national master trainers and 60 provincial master trainers. ** The project has no specific BETF for supervision. The monitoring and supervision by task team was financed by PNPM Supervision and Monitoring and PNPM Field Operations, and the ICR was financed by PSF Operations.

28

Annex 5. Beneficiary Survey Results No beneficiary survey was carried out.

29

Annex 6. Stakeholder Workshop Report and Results A stakeholder meeting/workshop was held on 22 October 2015 to present an overview of preliminary findings, and to seek input from key stakeholders on the preliminary findings as input into the finalization of this ICR. The meeting was held in Bappenas, and included representatives from PSF, Bappenas, Kemitraan and the DFAT funded KOMPAK project (who are considering future funding for P3BM). It was very unfortunate that there were no staff from the project funded P3BM national secretariat in attendance, as they had not been available for any meetings during the mission47. The planned method for the workshop was revised to reflect the lack of attendance of any representatives from the implementing team. The general findings of the mission were presented - achievements, challenges, and lessons learned - and input sought in relation to the preliminary findings. The participants were in general agreement with the findings and lessons learned, which have since been reflected in this report. A number of points were raised by participants, amongst them – General agreement on the need for a legislative basis to support P3BM There is substantial demand for training in the non-target areas. Each location is different, has different needs and capacities. The process encourages improved openness and sharing of data (though not always), and is very

dependent on support from local leaders. P3BM would be well placed as a working group within TKPDs. Comment that in field visits to discuss P3BM, there were locations where TKPD not aware of P3BM

(KOMPAK representative). There is a need to re-think the role and composition of Seknas (I understood this query related to the

need to consider required key competencies of Seknas team – should that be as trainers, and/or experience and skills in relation to advocacy, legislative reform, good governance etc).

That the indicators were not well understood. The Bappenas representatives committed to follow up on a number of issues where data was not available, and expressed enthusiasm to learn from this process so that future phases can be more effective.

47 Bappenas staff tried numerous times to contact various Seknas staff by phone just prior to the meeting to confirm their attendance. There was no response.

30

Annex 7. Summary of Grantee's ICR and/or Comments on Draft ICR The following is an exert from the Kemitraan P3BM Completion Report, which was jointly prepared by Kemitraan and Seknas (seknas providing data against indicators). It provides an overview of background to this phase, achievements against indicators and recommendations for the future. Pro Poor Planning, Budgeting and Monitoring (P3BM) is a set of analytical and monitoring tools developed to help local government officials increase the quality of planning, budgeting and monitoring in eradicating poverty. P3BM as a program is managed under Deputy of Poverty, Manpower and Small and Medium Enterprise in Bappenas. A P3BM’s National Secretariat has been set up to implement the program. The P3BM program “Phase 1” in 2010 - 2011 was supported by PSF-World Bank. The project execution of P3BM “Phase 2” was granted by PSF-World Bank to Kemitraan in January 2013 – December 2014 (PSF TF Grant No.TF014360) and received no cost extension until June 2015. The objective of the program, as stated in the project agreement is to improve poverty targeting local government expenditure and local government responsiveness to community needs through the scale up and institutionalization of P3BM program. The program framework has defined the Key Performance Indicator for the implementation of the P3BM “Phase 2”: At least 7.5% increase in the direct local government budgets allocated for MDGs targets and poverty

related expenditure; At least 280 provincial master trainers trained; At least 12 provincial P3BM secretariats/clinics established; At least 60 local governments adopted P3BM to annual development planning; At least 60 CSOs and community groups used P3BM to track pro-poor expenditure at district level. To support the objective, there are 3 components that should be conducted in the program implementation: 1) Provincial Cascade Training for master trainers, 2) Technical Assistance for local government and 3) Institutional support for national secretariat. The P3BM program was implemented in tree levels: national, provincial and district/municipals. In 2013, there were 12 provinces and 12 districts/municipals that have been determined as target areas. However, in 2014, there were 3 provinces and 3 districts added to the target locations. It means that the number of local governments (provinces and districts/municipals) who received the P3BM package increased from 24 in 2013 to 30 in 2014. All target locations have received a training, database launching, socialization, and P3BM Secretariat/Clinic establishment. Beside above target location, there are others provinces which asked the National Secretariat for P3BM Training, using APBD budget. The program monitoring and evaluation of P3BM Phase 2 demonstrated that this program has successfully achieved its objectives. Through the technical assistance, the local government has reached the public expenditure on health and education to 9.31% in 2015. According to baseline survey of P3BM, between 2011-2014 MDGs allocation for education sector in Indonesia in most provinces has decreased except than West Java and Papua. This achievement was higher than the target of 7.5%. Seventy three (73) local governments of 85 LGs (target and non-target) adopted P3BM or 122% higher than the cumulative KPI’s Target of sixty (60). In these local governments, P3BM tools have been used in

31

musrenbang, SKPD forum, TKPKD in the province and district/municipal level. The utilization of P3BM has given a result of MDGs document in 12 districts/municipals. More than this, the MDGs data were also used to increase the quality of the local planning document, such as RPJMD and Renstra. The number of local governments which have been trained on P3BM is 85 LGs, of which 12 LGs have not used P3BM tools in developing budget plan as they were trained after musrenbang. Kemitraan has supported the National Secretariat program by conducting P3BM training not only in the target areas, but also in other non-target areas. The cascade trainings have been conducted in 19 provinces and produced 679 master trainers – of which 284 master trainers are ready to act as trainers, higher than KPI’s target of 280 persons. For the sustainability of the P3BM program, the P3BM Provincial Secretariat/Clinic has been set-up. At the end of the project implementation in all 15 provinces of P3BM, there are 15 provincial secretariats/clinics. Ten (10) from these 15 offices have been endorsed by Governor or Bappeda Decree (SK Gubernur/Bappeda). P3BM as a planning process has used the multi-stakeholder approach. The CSOs was included in the master training, data analysis training, and monitoring training. There are 9 CSOs which have taken part in P3BM Secretariat/Clinic as the coordinator. This achievement on CSO reached 15% of the target (60 CSOs). The community based monitoring has been developed and trained in the community. From the evaluation performance, P3BM is highly relevant with the national program and the demand of the local government. The former Minister of Bappenas Mrs. Armida Alisyahbana highlighted in the launching of P3BM data in the province of Banten 2013, that by using the P3BM tools the development planning could run more efficient to meet targets. However, in term of effectiveness, the training method was less effective, as only 41.83% of the master trainers have the capacity to conduct the training. Based on the experience of the P3BM phase 2, the recommendations for future P3BM program are: One of P3BM components, poverty mapping, requires GIS software to produce visual images. Initially,

P3BM used licensed ArcGIS software. The type of ArcGIS software utilized by the local governments was the software with annual license. As the consequence, the local governments had to extend the license annually. This arrangement seems less effective and bear more cost to local government. It is suggested, that the National Secretariat provide the software from open source.

To increase the effectiveness of the training, it should develop the grand design of the training. The

grand design would explain the different kind models of training; training materials and training methods from the master of trainers until the operators.

It needs to provide the policy umbrella for P3BM. With this policy, in the local level, the socialization

and technical assistance for the local government could be more ensured. In the national level, the integration of the P3BM data into the other sector data can be undertaken.

The P3BM has a potential to be integrated into the implementation of Law No. 6 of 2014 on Villages.

These tools could play a role to monitor and evaluate the implementation of the Villages Law. The community based monitoring can be introduced to the village community to monitor implementation of the village plan and budget.

32

Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders N/A

33

Annex 9. List of Supporting Documents Standard Conditions for Grants made by the World Bank out of Various Funds. World Bank,

February 2012.

Project Information Document. World Bank. October 2012.

Project Paper for Small RETF Grant.

Country Partnership Strategy for Indonesia 2013-2015. World Bank. December 2012.

Implementation Status Reports (various). World Bank. 2013-2014.

Various Kemitraan and Seknas Reports.

34

MAP

Document of The World Bank - Documents & Reports -...

Documents

Transcript of Document of The World Bank - Documents & Reports -...