Document of The World Bankdocuments.worldbank.org/curated/en/109821468062649994/... ·...

Document of

The World Bank

Report No: ICR 2694

IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-78190, IDA-39180, TF-052794, TF-055474)

ON A CREDIT

IN THE AMOUNT OF SDR 4.9 MILLION (US$ 7.0 MILLION EQUIVALENT)

AND

ON A LOAN

IN THE AMOUNT OF EURO 5.1 MILLION (US$ 7.2 MILLION EQUIVALENT)

TO THE

REPUBLIC OF MONTENEGRO

FOR A

HEALTH CARE SYSTEM IMPROVEMENT PROJECT

AND ADDITIONAL FINANCING

June 17, 2013 Human Development Sector Unit (ECSHD) South East Europe Country Unit Europe and Central Asia Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

CURRENCY EQUIVALENTS

Exchange Rate Effective: May, 2013

Currency Unit = EURO 1 US$ = 0.77 EURO

1 SDR = 1.5 US$

FISCAL YEAR January 1 – December 31

ABBREVIATIONS AND ACRONYMS

AF Additional Financing MoH Ministry of Health ALOS Average Length of Stay MOU Memorandum of Understanding BBP Basic Benefit Package MMR Maternal Mortality Rate CIDA Canadian International Development

Agency MTR Mid-term Review

CME Continuous Medical Education NHA National Health Accounts CPG Clinical Pathways Guidelines NHS National Health Survey CPS Country Partnershp Strategy PAD Project Appraisal Document DCA Development Credit Agreement PCU Project Coordination Unit DRG Disease Related Group PDO Project Development Objective DZ Dom Zdravlja PEIR Public Expenditure Institutional Review EU European Union PMN Project Management Network EC European Commission PHC Primary Health Care HIF Health Insurance Fund PHRD Policy and Human Resources

Development Fund GP General Practitioner PRSP Poverty Reduction Strategy Paper HSIP Health System Improvement Project SAC Structural Adjustment Credit IHIS Integrated Health Information System SIL Specific Investment Loan IPH Institute of Public Health RF Results Framework ICR Implementation Completion Report OED Operations Evaluation Department ISR Implementation Status Report TOR Terms of Reference IT Information Technology TSS Transitional Support Strategy MDA Montenegro Drug Agency WHO World Health Organization M&E Monitoring and Evaluation

Vice President: Philippe H. Le Houerou

Country Director: Ellen A. Goldstein

Sector Manager: Daniel Dulitzky

Project Team Leader: Carlos Marcelo Bortman

ICR Team Leader: Lorena Kostallari

MONTENEGRO Health Care System Improvement Project

CONTENTS

Data Sheet

A. Basic Information ................................................................................................................... i B. Key Dates ............................................................................................................................... i C. Ratings Summary ................................................................................................................... i D. Sector and Theme Codes ....................................................................................................... ii E. Bank Staff .............................................................................................................................. ii F. Results Framework Analysis ................................................................................................ iii G. Ratings of Project Performance in ISRs .............................................................................. xv H. Restructuring (if any) ......................................................................................................... xvi I. Disbursement Profile ......................................................................................................... xvii

1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes ............................................... 6 3. Assessment of Outcomes ........................................................................................... 12 4. Assessment of Risk to Development Outcome ......................................................... 17 5. Assessment of Bank and Borrower Performance ...................................................... 18 6. Lessons Learned ........................................................................................................ 20 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners ........... 21 Annex 1. Project Costs and Financing .............................................................................. 22 Annex 2. Outputs by Component...................................................................................... 24 Annex 3. Economic and Financial Analysis ..................................................................... 28 Annex 4. Bank Lending and Implementation Support/Supervision Processes ................. 32 Annex 5. Beneficiary Survey Results ............................................................................... 34 Annex 6. Stakeholder Workshop Report and Results ....................................................... 35 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ......................... 36 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ........................... 47 Annex 9. List of Supporting Documents .......................................................................... 48 MAP IBRD 34825R1

i

A. Basic Information

Country: Montenegro Project Name: Healthcare System Improvement Project (Montenegro)

Project ID: P082223 L/C/TF Number(s): IBRD-78190,IDA-39180,TF-55474,TF-052794

ICR Date: 05/17/2013 ICR Type: Core ICR

Lending Instrument: SIL Borrower: MONTENEGRO

Original Total Commitment:

USD 7.00M Disbursed Amount: USD 13.83M

Revised Amount: USD 14.05M

Environmental Category: B Revised during Project Additional Financing to: C

Implementing Agencies: Ministry of Health Cofinanciers and Other External Partners: Canadian International Development Agency (CIDA) B. Key Dates

Process Date Process Original Date Revised / Actual

Date(s)

Concept Review: 09/22/2003 Effectiveness: 12/01/2004 12/01/2004

Appraisal: 02/02/2004 Restructuring(s): 07/12/2007

Approval: 06/08/2004 Mid-term Review: 02/12/2007 02/12/2007

Closing: 02/28/2009 12/31/2012 C. Ratings Summary C.1 Performance Rating by ICR

Outcomes: Moderately Satisfactory

Risk to Development Outcome: Moderate

Bank Performance: Moderately Satisfactory

Borrower Performance: Moderately Satisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings

Quality at Entry: Moderately Satisfactory Government: Satisfactory

Quality of Supervision: Satisfactory Implementing Agency/Agencies:

Moderately Satisfactory

ii

Overall Bank Performance:

Moderately SatisfactoryOverall Borrower Performance:

Moderately Satisfactory

C.3 Quality at Entry and Implementation Performance Indicators

Implementation Performance

Indicators QAG Assessments

(if any) Rating

Potential Problem Project at any time (Yes/No):

No Quality at Entry (QEA):

None

Problem Project at any time (Yes/No):

Yes Quality of Supervision (QSA):

None

DO rating before Closing/Inactive status:

Satisfactory

D. Sector and Theme Codes

Original Actual

Sector Code (as % of total Bank financing)

Central government administration 30 30

Compulsory health finance 10 10

Health 55 55

Non-compulsory health finance 5 5

Theme Code (as % of total Bank financing)

Administrative and civil service reform 33 33

Health system performance 33 33

Injuries and non-communicable diseases 17 17

Other accountability/anti-corruption 17 17 E. Bank Staff

Positions At ICR At Approval

Vice President: Philippe H. Le Houerou Shigeo Katsu

Country Director: Ellen A. Goldstein Orsalia Kalantzopoulos

Sector Manager: Daniel Dulitzky Armin H. Fidler

Project Team Leader: Carlos Marcelo Bortman Loraine Hawkins

ICR Team Leader: Lorena Kostallari

ICR Primary Author: Lorena Kostallari

iii

F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document)The objective of the proposed Health System Improvement Project is to put in place the first phase of steps towards reform of the health system in the Republic of Montenegro, giving priority to increasing capacity for policy, planning and regulation, stabilizing health financing, and improving primary health care service delivery. Revised Project Development Objectives (as approved by original approving authority) The project development objectives were not revised. (a) PDO Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1 : Reduced drug prices paid by the Health Insurance Fund (HIF) – for agreed list of high cost and high volume indicator drugs.

Value quantitative or Qualitative)

Share of HIF expenditures spent on drugs: 19.2% and on drugs and medical materials 28.4%.

Evidence of cost containment since baseline.

Share of HIF expenditure spent on drugs are below 18.0% and on drugs and medical materials at or below 27.5%.

Share of HIF expenditures spent on drugs are at 20% and expenditures spent on drugs and medical materials are at 28.0%

Date achieved 12/31/2004 12/31/2009 12/31/2012 12/31/2012

Comments (incl. % achievement)

Target partly achieved. The original RF did not specify a clear target value, which was defined during the AF (November, 2009). It is important to note that expanding the list of drugs may have increased costs even with the reduction in drug prices.

Indicator 2 : HIF annual deficit is reduced. Value quantitative or Qualitative)

Deficit of 2.2 million Euro

Zero deficit Zero deficit

Date achieved 12/31/2003 12/31/2009 12/31/2012 Comments (incl. % achievement)

Target achieved.

Indicator 3 : Waiting time (from arrival to consultation) in primary care is reduced in Podgorica and nationally.


57% of Podgorica patients report 21min. wait time. While 68% of patients of outside Podgorica report 21min.

Significant increases in proportion of patients waiting less than 30min.

30% of Podgorica patients reporting 21min. wait

29% of Podgorica patients reporting 21 min. wait time. Meanwhile 34% of patients outside

iv



approval documents)




of waiting. time. Patients outside Podgorica reporting 21min. of waiting time, improved by 15%.

Podgorica reporting 21 min. wait time.

Date achieved 12/31/2004 12/31/2009 12/31/2012 12/31/2012 Comments (incl. % achievement)

Target achieved. Target value was revised during project AF.

Indicator 4 : Utilization rates and satisfaction for primary health care in Podgorica by Roma and IDPs are increased.


11.4% of Pogdorica population report use of PHC; 21% of Roma report use of PHC. In Podgorica 62% of the population are totally satisfied and 20% partially satisfied.

40% of the population visit their chosen doctor. 70% of population in Podgorica totally satisfied and 27% partially satisfied. 66% of the population outside Podgorica totally satisfied and 31% partially satisfied. Satisfaction of Roma population 87.7%

44% of the general population and 82% of Roma use the services of a chosen doctor. In Podgorica 77% of population satisfied and 19% partially satisfied. Nationally, 70% satisfied and 24% partially satisfied. Roma satisfaction 82.1%.


The indicator is presented as per the PAD. The AF split the indicator into: (i) increase utilization rates: target achieved, and (ii) increase satisfaction: target partly achieved.

Indicator 5 : Gap between primary health care utilization of the poor and non-poor is reduced.Value quantitative or Qualitative)

v



approval documents)




Date achieved Comments (incl. % achievement)

During the Project Restructuring of July 2007, the indicator was dropped. Inclusion of this indicator in the original RF was considered inconsistent with project design and the development objectives.

Indicator 6 : Immunization rates for DPT and measles are maintained or improved (rates are already high) following reform.


DPT (initial and follow-up): 86%. MMR (initial and follow-up): 82%.

DPT (initial and follow-up):94%. MMR (initial and follow-up):92%.

DPT (initial and follow-up): 94.7%. MMR (initial and follow-up): 90.7%.


Target partly achieved.

Indicator 7 : Utilization of day care services by the elderly is increased. Value quantitative or Qualitative)


During the Project Restructuring of July 2007, the indicator was dropped. Inclusion of this indicator in the original RF was considered inconsistent with project design and development objectives.

Indicator 8 : Ministry of Health stewardship of the health system improves as measured by increases in health-related information and development of key health policies.


Secondary and tertiary reform strategy developed and approved, including legislative framework for PPP in health. Further refine PHC policies, new master plan adapted.

Secondary and tertiary reform strategy developed and approved, including legislative framework for PPP in health. Further refine PHC policies, new master plan adapted. Decision on network of health institutions adopted.

Date achieved 12/31/2012 12/31/2012 Comments (incl. % achievement)

Target achieved. This outcome indicator was added during the AF.

Indicator 9 : HIF increases its capacity to perform its functions of pooling health resources, better allocating health resources, and contracting providers toward improved

vi



approval documents)




performance.


HIF has developed the groundwork for developing DRGs, developing the information base, established the system of coding of diagnoses and training the hospital staff. Additional contracts with private providers signed.

Hospitals started reporting under DGR system. Number of trained hospital staff amounts to 293. Hospital information system introduced in seven general hospitals. System of coding and grouping diagnoses completed by the HIF. Additional contracts signed.

Date achieved 12/31/2012 12/31/2012 Comments (incl. % achievement)

Target achieved. Indicator added during the AF.

Indicator 10 : Duration of PHC Consultation in Podgorica and nationally is increased.


30% of Podgorica patients reporting 11min. or more.

70% of Podgorica patients reporting 11min. or more. While 55% of population outside Podgorica report 11min. or more

55% of Podgorica population report 11min.or more. While 55% of population outside Podgorica report 11min. or more.


Target partially achieved. Indicator added during the AF.

Indicator 11 : Referral rates from PHC to inpatient care are controlled in Podgorica and nationally.

Value quantitative or

Proportion of patients referred to specialists

10% improvement.

Proportions of visits referred to

vii



approval documents)




Qualitative) 24.43%. Proportion of patients referred to hospitals: 1.73%.

specialists: 25.71% for 2011 and 22.18% for the first 9 months of 2012. For hospitals was 1.82% for 2011, 1.52% for nine months of 2012.


Target achieved. Indicator added during the AF.

(b) Intermediate Outcome Indicator(s)



approval documents)

Formally Revised

Target Values



Indicator 1 : Adoption of bylaws and administrative documents under the new laws, currently under development covering health insurance, health care, and pharmaceuticals.

Value (quantitative or Qualitative)

Law on Health Insurance, Health Care and pharmaceuticals in draft.

Laws maintained including additional sub-regulation for pharmaceuticals.

11 new laws adapted and amended. 4 new bylaws adapted. 434 pharmaceutical drugs approved with 4 new sub regulations. New law on medicines approved. Laws on health care and health insurance amended.


Target achieved.

Indicator 2 : Annual analysis of growth of pharmaceutical expenditure by price, volume, and mix by HIF.


None Annual analysis conducted.

Annual analysis conducted until Mid-Term review.

Date achieved 12/31/2004 07/12/2007 07/12/2007 Comments Target achieved. No monitoring of the indicator from the mid-term review.

viii



approval documents)

Formally Revised

Target Values



(incl. % achievement)

Indicator 3 : Awareness and understanding of health reforms in opinion survey. The AF revised the language of the indicator: "PHC patients are aware of and supportive of health reform".


54% in Podgorica

70% of population (nationally) is aware.

89% of population (nationally) is aware and supportive of the reform.


Target achieved. Source: Household Survey, August 2012.

Indicator 4 : Revision of policy, benefits package, and standards for primary care.


Montenegro system of PHC inherited from former Yougoslavia

Revision and implementation of policy, benefits package, and standards for primary care completed.

Payment mechanisms fully operational. Benefit package developed, 11 clinical protocols and guidelines developed for most frequent diseases.


Target achieved. The indicator was added during project MTR/Restructuring (July 2007).

Indicator 5 : Revision of policy, benefits package, and standards for secondary and tertiary.


None

Master plan developed and adapted, benefits package of HIF-financed secondary and tertiary services defined, and information base for developing DRGs established.

Master plan 2010-213 adapted. Secondary and tertiary HC strategy adapted. Hospital information system introduced in 7 hospitals, benefit package defined and DRG reporting started in September 2012.

Date achieved 12/31/2012 12/31/2012

ix



approval documents)

Formally Revised

Target Values



Comments (incl. % achievement)

Target achieved. The indicator was added during project MTR/Restructuring (July 2007).

Indicator 6 : Drug agency developed to better perform its functions of regulating and insuring the quality of drugs in Montenegro.


None

Drug agency functional as regulatory agency. Drug agency has registered 10-15% of total drugs available in the market.

Drug agency established. The agency has registered 44.5% of total drugs available in the market.


Target achieved. The indicator was added during the AF.

Indicator 7 : Operational health information systems in HIF, primary health care providers and pharmaceutical supply chain, to provide timely, accurate data on key elements of performance.


HIF pharma and insuree registration system operational.

Information system between HIF, PHC provider and pharmaceutical supply chain remain operational. IT system for drugs includes all providers who prescribe drugs. IT system connects HIF and hospitals, provide info on DRG.

IT system between HIF, PHC providers and pharmaceutical chain is operational. HIF does electronic invoicing and has developed business information system with all hospitals. IT system for MoH, IPH, and Drug's Agency developed and operating.


Target achieved.

Indicator 8 : Increase duration of PHC consultation. Value (quantitative

30% of Podgorica patients reporting 11min.

70% of Podgorica

55% of Podgorica population report

x



approval documents)

Formally Revised

Target Values



or Qualitative) or more. patients reporting 11min. or more. While 55% of population outside Podgorica report 11min. or more

11min.or more. While 55% of population outside Podgorica report 11min. or more.


Target achieved. The indicator became a PDO indicator as part of the AF.

Indicator 9 : Increase percentage of Podgorica patients registered with chosen primary care doctor or group practice.


0% 85%

95.6% of patients (adults and children) in Podgorica are registered with a chosen PHC doctor.


Target achieved.

Indicator 10 : Increase share of primary care contacts among all health service contacts. Value (quantitative or Qualitative)


The indicator was included in the PAD but dropped at project restructuring.

Indicator 11 : Reduce proportion of laboratory results not picked up by the patient. Value (quantitative or Qualitative)


The indicator was included in the PAD but dropped at project restructuring.

Indicator 12 : Reduce proportion of patients receiving injectable drugs. Value (quantitative

32.124 referrals in Podgorica

12% decrease 12.8% decrease

xi



approval documents)

Formally Revised

Target Values



or Qualitative) Date achieved 12/31/2006 12/31/2009 12/31/2012 Comments (incl. % achievement)

Target achieved. Baseline data provided in 2006.

Indicator 13 : Percentage of patients in Montenegro (outside of Podgorica) registered with chose primary care doctor or group practice.


0%

85% of patients outside Podgorica registered with chosen doctor.

90.7% of patients outside Podgorica registered with chosen doctor.


Target achieved.

Indicator 14 : Number of trainees (PHC doctors, nurses) enrolled in specialization program for PHC.


0 101 staff 101 staff


Target achieved.

Indicator 15 : Number of PHC teams (1 chosen doctor + 1 nurse) provided retraining in family medicine nationally.


0

Retraining program completed with a total of 255 teams.

273 teams and additional 6 nurses retrained.


Target achieved. Indicator was added during the AF.

Indicator 16 : Family medicine specialization established and operational at Medical Faculty.


None

Family medicine specialization established and initial cohort of students enrolled

Family medicine specialization established and initial cohort of students enrolled in October 2012.

xii



approval documents)

Formally Revised

Target Values





Indicator 17 : Number of facilities renovated/constructed in Podgorica. Value (quantitative or Qualitative)

0 7 PHC centers renovated.

7 PHC centers renovated in Podgorica.



Indicator 18 : Number of facilities equipped with computers and medical equipment in Podgorica.


0

10 PHC centers equipped with computers and medical equipment.




Indicator 19 : Number of facilities renovated/constructed outside Podgorica. Value (quantitative or Qualitative)

0 1-the PHC center in Bijelo Polje.

The PHC center in Bijelo Polje is rehabilitated.



Indicator 20 : Number of facilities equipped with computers and medical equipment outside of Podgorica.


0





xiii



approval documents)

Formally Revised

Target Values



Indicator 21 : Adaption of PHC guidelines.


0 10 PHC guidelines adapted.

Total of 11 clinical protocol and guidelines developed (for primary and secondary health care level).



Indicator 22 : Proposal developed for the appropriate quality assurance system in Montenegro.


None

Proposal well developed and addresses the legal, institutional, financial, and technical aspects of a quality assurance system.

National strategy for improvement of health care quality and safety of patients adopted in February 2012.



Indicator 23 : Project progress monitoring reports produced every six months and action plans for the following six months developed.


None

Monitoring reports produced every 6 months, including action plan for the following six months.

Monitoring reports produced every 6 months, including action plan for the following six months.


Target achieved.

Indicator 24 : Financial management reports produced within 15 working days after each quarter.


None Every FM report produced within 15 working days

Every FM report produced within 15 working days after

xiv



approval documents)

Formally Revised

Target Values



after each quarter. each quarter. Date achieved 12/31/2004 12/31/2009 12/31/2012 Comments (incl. % achievement)

Target achieved.

Indicator 25 : Project outputs produced on time and on budget; PMN shows proactively in solving problems and seeking to achieve outcomes.



The indicator was introduced in PAD but dropped at project restructuring.

Indicator 26 : Minimize average number of days delay between planned and actual dates for preparation of TORs and technical specifications in the procurement plan.




Indicator 27 : Zero breaches of agreed business standards in the MOU for responding to the TSU.




Indicator 28 : Minimize average number of days delay between planned and actual dates for milestones in the procurement plan.




Indicator 29 : Minimize time between the request for invoice payment, and payment execution.Value (quantitative

xv



approval documents)

Formally Revised

Target Values



or Qualitative) Date achieved Comments (incl. % achievement)


Indicator 30 : Weekly meetings of project management committee; monthly meetings of project steering committee with full participation of agencies represented.




G. Ratings of Project Performance in ISRs

No. Date ISR Archived

DO IP Actual

Disbursements (USD millions)

1 06/24/2004 Satisfactory Satisfactory 0.00 2 12/22/2004 Satisfactory Satisfactory 0.23 3 06/06/2005 Satisfactory Satisfactory 0.23 4 10/19/2005 Moderately Satisfactory Satisfactory 0.60

5 04/12/2006 Moderately

Unsatisfactory Unsatisfactory 0.71

6 12/13/2006 Moderately

Unsatisfactory Moderately

Unsatisfactory 1.34

7 03/01/2007 Moderately Satisfactory Moderately Satisfactory 1.69 8 12/06/2007 Moderately Satisfactory Moderately Satisfactory 3.70 9 06/05/2008 Satisfactory Satisfactory 5.01

10 06/12/2009 Satisfactory Satisfactory 6.88 11 12/28/2009 Satisfactory Satisfactory 7.42 12 06/21/2010 Satisfactory Moderately Satisfactory 8.24 13 05/07/2011 Satisfactory Moderately Satisfactory 8.42 14 09/12/2011 Satisfactory Moderately Satisfactory 8.93 15 01/15/2012 Satisfactory Moderately Satisfactory 10.19 16 05/08/2012 Satisfactory Moderately Satisfactory 10.64 17 12/25/2012 Satisfactory Satisfactory 12.51

xvi

H. Restructuring (if any)

Restructuring Date(s)

Board Approved

PDO Change

ISR Ratings at Restructuring

Amount Disbursed at

Restructuring in USD millions

Reason for Restructuring & Key Changes Made

DO IP

07/16/2007 Y MS MS 2.89

Lack of implementation progress. Key changes included: (i) reallocation of credit proceeds, mainly to accommodate cost overruns in civil works for a number of PHC centers; (ii) revision of the RF, to promote a more effective monitoring, and (iii) revision of responsibilities for project components.

If PDO and/or Key Outcome Targets were formally revised (approved by the original approving body) enter ratings below:1 Outcome Ratings Against Original PDO/Targets Moderately Satisfactory Against Formally Revised PDO/Targets Moderately Satisfactory Overall (weighted) rating Moderately Satisfactory

1 Though the PDO was not revised, the original PDO indicators were partly revised at the time of the Additional Financing.

xvii

I. Disbursement Profile

1

1. Project Context, Development Objectives and Design 1. The Montenegro Health System Improvement project was approved on June 8, 2004. The credit was signed on June 11, 2004 and became effective on December 1, 2004. The US$15.2 million equivalent of donor financing included US$7.0 million equivalent credit from IDA, US$0.82 million grant from the Canadian International Development Agency (CIDA), US$0.45 million grant from the Policy and Human Resources Development Fund (PHRD), and an Additional Financing (AF) of US$7.2 million equivalent loan from the IBRD. 2. The Additional Financing (AF) to the Project, of Euro 5.10 Million (US$7.2 equivalent), was approved in November 2009, with a closing date of December 31, 2012. The additional funds were to scale-up project original activities to enhance its development impact. 1.1 Context at Appraisal

3. Country and Sector background. At project appraisal in 2003, the Republic of Montenegro was part of the constitutional union State of Serbia and Montenegro, established in February 2003, and later dissolved in 2006. The effects of poor economic management, compounded by international sanctions and conflict (between 1992-1996 and 1998-2000), which severely inhibited trade and investment in the country, led to a sharp decline in economic conditions. However, since late 2000, progress in structural reforms remains positive despite several domestic and external shocks. An extensive economic reform agenda was adopted in 2003, resulting in visible progress and sustained recovery. The GDP was estimated to have grown at 2 percent annually at the time of appraisal. Montenegro’s consolidated fiscal deficit was cut from about 8 percent of GDP in 2000 to an estimated 5.2 percent in 2003.

4. Main Health Sector Issues. Challenges faced by the health system in Montenegro were related to key areas, such as health service delivery, financing, and governance. The existing public health financing and delivery system was not financially sustainable due to high contribution rates on the payroll and the existence of an informal sector, which resulted in: (i) difficult collection; (ii) inadequate budget transfers for the uninsured; and (iii) a lack of the ability to adjust the benefit package to offset difficult economic circumstances. The government’s capacity for policy development, planning, managing and monitoring the health system were considerably weak, with deficient information systems. Major issues also existed in service delivery, including limited access to and fragmented organization of primary health care (PHC), as well as a lack of management and motivation of health sector staff.

5. In 2003, the Montenegro’s Parliament approved the health sector strategy, which was part of the government’s Poverty Reduction Strategy Paper (PRSP). In addition, draft framework laws for health protection and health insurance were prepared by the Ministry of Health (MoH) to underpin the health reform strategy. The project was conceptualized based on key pillars of the health sector strategy, and thus focused on supporting the government in developing priority areas of policy and regulation, building capacity, improving quality, efficiency and access in PHC, and taking measurable steps towards achieving financial sustainability in the health care system.

6. Country Partnership Strategy and Rationale for Bank involvement. The Bank was (and remains) the main source of support for policy and system reform in the Montenegro health sector. At appraisal, the Bank’s strategic program with Serbia and Montenegro was outlined in the 2001 (updated 2002) Transitional Support Strategy (TSS). The project was fully consistent with the goals made explicit by the TSS. These goals included the alleviation of poverty and the development of human capital through improved health status, by ensuring quality and effective health services at all levels. In addition,

2

the Bank conducted a Public Expenditure Institutional Review (PEIR) in 2002 and a Poverty Assessment in 2003, which provided analytical input to project preparation.

1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) 7. The objective of the Health System Improvement Project (HSIP) was to put in place the first phase of steps necessary to reform the health system in the Republic of Montenegro, giving priority to increasing capacity for policy, planning and regulation, stabilizing health financing, and improving PHC service delivery. Specifically, the project would: (i) support measures to achieve financial sustainability of the health care system by strengthening institutional capacity and information systems for health policy, planning, regulation, and management in the MoH, Health Insurance Fund (HIF), and the Institute of Public Health (IPH); (ii) improve quality, efficiency, and access to PHC services by investment in training of staff, facilities and equipment, organizational reform, and financing of primary care beginning in Podgorica and extending to outlying areas, and (iii) support a project management network (PMN) of the MoH and a central Technical Service Unit (TSU) that would provide procurement and financial management services.

8. The wording of the Project Development Objective (PDO) was consistent throughout the main text of the Project Appraisal Document (PAD) and of the Development Credit Agreement (DCA). However, there is a slight difference between the main text and Annex 3 of the PAD, concerning: “improving of services for elderly and people with long term mental illness and disabilities in primary care”, which is a specific objective only mentioned in Annex 3 of the PAD.

9. The PDO indicators were:

Drug prices are reduced and agreed indicators of rational drug use are improved HIF deficit is reduced Waiting time (from arrival to consultation) in primary care is reduced Utilization rates and satisfaction for primary health care in Podgorica by Roma and

Internally Displaced Persons (IDPs) are increased Gap between primary health care utilization of the poor and non-poor is reduced Immunization rates for DPT and measles are maintained or improved Utilization of day care services by the elderly is increased

1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification

10. The PDO was not revised. However, the project’s Results Framework was revised during the project formal restructuring (July 2007) and at the time of the Additional Financing (approved in November 2009). The project restructuring formally removed the following outcome indicators: (i) “utilization of day care services by the elderly is increased” and (ii) “the gap between PHC utilization of the poor and non-poor is reduced”, considering them as non-consistent with the PDO and the project activities, which were also revised as part of the restructuring. The AF added three new outcome indicators (as described in detail in the datasheet), to reflect the impact of new activities introduced. In addition, a number of intermediate outcome indicators were revised to better reflect the changes introduced during project restructuring and AF, as well as to promote a more effective progress monitoring.

3

1.4 Main Beneficiaries 11. Original project and AF investments were targeted to provide significant benefits to the whole population of Montenegro (Roma included), through improvements in quality and utilization of PHC services. In addition, the MoH and key health sector institutions were to benefit from project support in strengthening their capacity to develop a sustainable performance-oriented health care system. The HIF, specifically, was targeted to benefit from strengthened capacity in budget planning, expenditure and revenue forecasting, as well as analyses and resource allocation. Both the HIF and the Montenegro Drug Agency (MDA) were to benefit from the project through the receipt of quality control systems, and the establishment of a transparent drug registration and pricing system. In addition, PHC services were to benefit from interventions in infrastructure, IT systems, and substantial training of medical staff.

1.5 Original Components 12. The Project consisted of the following three components: 13. Component 1 - Support for health reform program (US$2.2 million): The objective of this component was to support the government to identify best practices in health policy, financing, and selected areas of service delivery. There were two subcomponents:

14. Subcomponent 1. Health policy development and capacity building for the MoH, HIF, and IPH, focusing on health finance, pharmaceutical regulation and expenditure management, planning and strategy development, communications and public information to support reform, and monitoring and evaluation. Priority would be given within these activities to aspects relevant to primary health care development, development of the benefits package and provider payment policies, and efficiency improvement and expenditure control. This sub-component would provide expert advice and train the MoH, HIF and IPH staff in health finance, planning, and monitoring and evaluation.

15. Subcomponent 1.2. Investments in health information systems to improve data collection, contracting, monitoring and management of primary health care and related services. These investments would be consistent with a longer term plan to develop integrated health information systems, and would build upon HIS development already carried out in the HIF. Priority would be given to the informational needs of the MoH, HIF and IPH, as well as to development of information systems for primary health care. 16. Component 2 - Phased implementation of primary health care development, beginning in Podgorica, and scaling up of aspects of the reform and development to other locations (US$6.6 million). The objective of this component was to support phased implementation of plans for the reform and development of primary health care (developed under Component 1). The component was structured under four subcomponents:

17. Subcomponent 2.1 First phase of implementation in Podgorica, which would be used to develop and refine the reform model before scaling up reforms to other parts of Montenegro. Key elements of the reform model included: (i) patient choice of and registration with a primary care doctor and reorganization of these "chosen doctors" into group practices; (ii) consolidation of specialized primary-care-based services into a Primary Care Support Center (including diagnostics, day services for the elderly and for people with chronic mental illness, specialized community-based clinics - including tuberculosis dispensary, youth health, and mental health - and teaching facilities); (iii) implementation of a new contract with these doctors to improve staff motivation based on a re-defined benefits package and a new payments model; and (iv) streamlining of the organization of services to reduce waiting and crowding and give medical staff more quality time with patients. These activities would be coordinated with information systems development and with training of staff in evidence-based protocols of care.

4

This subcomponent would also finance civil works to carry out internal remodeling of existing primary care facilities, to rationalize facilities, and improve the functional lay-out of buildings so as to support the development of group practice. 18. Subcomponent 2.2. Further development of primary health care policy, standards, and implementation capacity, through technical assistance and training. This would cover support for: (i) change management; (ii) development of measures to increase service quality through development of clinical standards and guidelines, licensing and accreditation, and professional chambers and associations; (iii) development of policies and strategies to improve patient relations and protect patients’ rights; and (iv) development of human resource policies for primary health care.

19. Subcomponent 2.3. Development of primary care specialization training and continuing professional development. This would include development of a training center in the Podgorica linked to the Medical Faculty; this is likely to develop a new professional specialization profile for primary health care doctors, and specialized training for nurses in primary health care, as well as upgrading of the skills of existing staff through continuing medical education.

20. Subcomponent 2.4. Support for phased scaling-up of implementation of the reforms piloted in Podgorica to the rest of the country. The key areas to be scaled up first throughout the rest of Montenegro would include: implementation of patient choice of and registration with a primary care doctor; implementation of a new contract with these doctors based on a re-defined benefits package and a new payments model.

21. Component 3 - Project management (US$ 1.0 million): The objective of this component was to ensure a proper and efficient implementation of project activities. The MoH was defined as lead implementation agency for the project. A Project Management Network (PMN) was established, led by a Project Coordinator in the MoH. A central Technical Services Unit (TSU) was established under the Office of the Deputy Prime Minister, responsible for providing core procurement and financial management services for all future Bank-financed projects. This component was comprised of subcomponent 3.1: the operation of the PMN under the MOH; and subcomponent 3.2: the operation of the central TSU serving the health project and other projects.

1.6 Revised Components

21. See section 1.7.

1.7 Other significant changes 22. Project Restructuring. The project went through a level II restructuring in July 2007. Project restructuring, agreed between the MoH, other project stakeholders and the Bank team during the mid-term review (MTR), had the same development objectives as the original project and was structured around the same original three components: (i) support for the health reform program; (ii) phased implementation of the PHC development; and (iii) project management and monitoring and evaluation.

23. The new structure of the project reflected the following decisions: (i) to implement the first phase of the PHC reform in Podgorica; (ii) the proposal to create a Primary Health Care Support Center in Podgorica was dropped since the same planned activities were integrated into the existing PHC network; (iii) reallocation of Credit proceeds, mainly to accommodate cost overruns in civil works for a number of PHC centers; and (iv) the responsibility of leading activities in each component and subcomponent was transferred from working groups to institutions in the Government of Montenegro.

5

24. The revised project subcomponents reflected a more logical grouping of the activities financed by the IDA Credit. Furthermore, all project documents, such as the Project Appraisal Document, the Procurement Plan, and the Results Framework became programmatically and operationally consistent.

25. The project’s Results Framework was revised during restructuring to promote a more effective monitoring of progress towards PDO achievement, without changing the intended outcomes. There were two outcome indicators, namely: (i) “utilization of day care services by the elderly is increased” and (ii) “the gap between PHC utilization of the poor and non-poor is reduced”, and a number of intermediate indicators which were dropped during the restructuring, since it was not expected to be impacted as a result of the project activities. In addition, baseline values for all indicators were obtained and recorded, and the parties responsible for the collection of monitoring data were identified. Table 1: Revised Allocation of the Credit Proceeds by Category of Expenditures Category

Original Amount of the Credit Allocated Expressed in SDR

Reallocated Amount of the Credit Expressed in SDR

% of Expenditures to be Financed

(1) Works

1,070,000

1,500,000

83%

(2) Goods, including technical services

990,000 1,000,000

80% of foreign expenditures; 83% of local expenditures

(3) Consultants’ services, including audit

1,580,000 1,620,000 75% for foreign individuals; 68% for local individuals; 81% for firms

(4) Training/ workshops and study tours

660,000 200,000 100%

(5) Incremental operating costs

40,000 80,000 80%

(6) Unallocated 560,000 500,000 TOTAL 4,900,000 4,900,000

26. Additional Financing (AF) to the project (Euro 5.10 Million) was approved in November 2009, with a closing date of December 31, 2012. The additional funds were to support the expansion of original activities -- deepening health reform through the development of institutional capacity and policies in the areas of health financing, pharmaceutical policy, health policy and planning, as well as providing further support to the phased implementation of the primary health care reform. In addition, the project’s rating with respect to Environmental Safeguards was upgraded from Category C to B (as a result of civil works for the rehabilitation of the Bijelo Polje health center. 27. The AF supported the following project components:

28. Component 1 - Support for the Health Reform Program. The AF supported three priority health policy reforms (scale up and expansion of subcomponent 1.1). First, it would finance technical assistance and training to support the development of a secondary and tertiary care health sector reform strategy. The elements of this reform included plans and a normative framework for the health network; defining the package of services financed by the state; defining an output-based (i.e. diagnostic related groups - DRGs) payment mechanism and contracts between the providers and insurer; developing the priority clinical guidelines to ensure effective and efficient delivery of state-financed services; and determining models of public and private cooperation in the delivery of health services, including defining appropriate models and proposing legislative solutions. Second, it would support further institutional capacity

6

building of the MDA, which is responsible for the regulation and oversight of the pharmaceutical sector, through provision of technical assistance, training, and information systems. Third, the AF would finance computer hardware and related equipment to establish the information system between hospitals and the insurance fund (expansion of subcomponent 1.2). The provision of equipment was linked to the development and operation of the software by the HIF through other funding sources. 29. Component 2 - Phased Implementation of Primary Health Care Development. The AF would provide technical assistance to support the MoH in monitoring and assessing the implementation of the primary health care reform (expansion of subcomponent 2.2). It would provide further technical assistance in establishing a Family Medicine specialization in the medical faculties (completion of activity under subcomponent 2.3 due to financing gap). Also, the AF would support the nationwide upgrade of primary health care services (scale up and expansion of subcomponent 2.4) through the provision of re-training of doctors and nurses; medical equipment for the offices of the “chosen doctor” and the diagnostic services in the Health Centers; and physical upgrade (rehabilitation and extension) of the health center in Bijelo Polje (with co-financing from the HIF and the Bijelo Polje municipality).

30. Component 3 - Project Management. The AF would continue to support the functioning of the Project Management Network through the provision of technical assistance, training, and the operating costs of the dedicated office (scale-up of Component 3.1), as well as the proportional cost of the staff and operating costs of the TSU (scale-up of Component 3.2). 2

2. Key Factors Affecting Implementation and Outcomes

2.1 Project Preparation, Design and Quality at Entry 31. The ICR team rates design and quality at entry as Moderately Satisfactory on the following basis: 32. Financing Instrument. The project was financed from an IDA Credit of SDR 4.9 million (US$7.0 million equivalent) and an IBRD Loan of Euro 5.1 (US$7.2 million equivalent), using a Specific Investment Loan (SIL) instrument. In addition, the project benefited from a co-financing from the Canadian International Development Agency (CIDA) of US$0.82 million3 (TF055474), a PHRD4 grant (TF052794) of US$0.45 million, and a government’s contribution of US$3.7 million. A SIL was considered the most appropriate instrument based on the following: (i) an investment operation was needed to complement the proposed Structural Adjustment Lending – SAC 2, which included health-related conditionalities; and (ii) certain efficiency gains and quality improvements in primary health care required re-design and equipping of facilities, as well as extensive staff training, which is uniquely and most effectively accomplished through investment lending instruments.

2 Total project costs as originally appraised, reallocated and proposed under the AF is presented in Annex 1.

3 At project appraisal phase, agreement was reached on a co-financing from CIDA of US$ 0.57 million. However, during project implementation, the Grant amount was increased to US$ 0.82 million. See section 7.b for more details.

4 Japan Policy and Human Resources Development (PHRD) fund

7

33. Soundness of background analyses. The project was prepared by a large team of experts with financial support from the PHRD fund. This was the first Bank-supported stand-alone project in the health sector in Montenegro. Project preparation benefited significantly from the 2003 Operations Evaluation Department OED study on a number of completed Bank-supported projects in the health sector in the ECA region. In addition, the team carefully considered key analyses and priorities extracted from the “Health Services Policy in the Republic of Montenegro up to Year 2020” (prepared by the government in 2001), and the “Strategy for Health Care Development in Montenegro” (2003).

34. Assessment of Project Design. A number of alternatives were considered during project preparation. The first alternative was not having a project at all. However, there were strong justifications for having an immediate investment in the health sector, including the fact that public spending on health was a significant drain on scarce public resources, the lack of efficiency of the existing public health provider network, as well as the strong demand from the government as a result of the high priority they placed on health reform. A second alternative was proceeding with a small technical assistance project that could support the implementation of the strategies under the second Structural Adjustment Credit (SAC 2). Yet, this alternative could not bring immediate visible results in the existing delivery system which was characterized by low capital investment and capacity, and as such, it was also dropped. Finally, the choice of an APL was also considered. However, the political climate of Montenegro was too uncertain (considering the country had just recently obtained independence, as well as the fact that they were considering the possibility of EU accession); hence the team did not proceed with the APL option. 35. Based on the existing government strategy for development of the health sector, a wide range of issues were identified and discussed by the team during the design phase. However, the project was designed to focus on the most critical issues, with a long-term outlook towards implementing the health sector strategy. Specifically, project design addressed: (i) health financing challenges facing the sector, based on the importance of financial sustainability and large inefficiencies in the existing health care system; (ii) capacity building needed for ensuring long term effects of the sector reform; as well as (iii) limited access to and fragmented organization of primary health care. Although consideration was given to also support long-term care services for the elderly and disabled, as per the MoH’s request, the Bank team concluded that this would add excessive complexity to the project, hence project support concentrated only on the development of primary health care.

36. Despite a careful assessment of health sector needs and a thorough preparation, project design was somewhat complex and with a low degree of realism. The design missed the opportunity to put in place a sound M&E system and overvalued the existing local capacity. For instance, lack of capacity, unclear division of roles and responsibilities between the MoH, the IPH, and the HIF as well as issues with functioning of the working groups established for each project component, proved to negatively impact the project implementation schedule and placed the achievement of project outcomes at risk.

2.2 Implementation 37. The ICR team rates overall implementation as Moderately Satisfactory. The project was successful in achieving most of its expected outcomes and delivering the outputs specified under the three components. Although project start up suffered from certain delays, the initial phase of project implementation was rated as Satisfactory for both Development Objective (DO) and Implementation Progress (IP). Despite delays at the beginning, project implementation progressed relatively well; hence, the rating was kept as Satisfactory or Moderately Satisfactory in terms of DO and IP until the end of 2005. 38. However, fourteen months after effectiveness the lack of implementation progress triggered the decision to downgrade the ratings for project DO and IP to Moderately Unsatisfactory. The

8

project had committed only 11.8 percent and disbursed only 8.3 percent of the total credit of US$7.0 million equivalent. At that point, it was believed that a project restructuring and revision of implementation arrangements would speed up the pace of implementation. 39. Key factors that negatively influenced implementation progress at this stage, included: (i) divisions between project stakeholders due to political tensions and the lack of clarity on redistribution of roles and responsibilities between the regulatory bodies (MoH, IPH), the purchaser (HIF) and the providers; (ii) existing arrangements adopted for project implementation (a network of working groups), which created additional confusion and limited communication and cooperation among stakeholders; and (iii) the diversity of activities and fragmentation of consultancies planned under Component 1 “Support for health reform program” created an additional burden on the project team and thus further contributed to the implementation lag.

40. The situation remained unchanged until the end of 2006, while waiting for the results of the legislative elections and government reshuffling.

41. The Mid-Term Review (MTR), carried out in February 2007, was considered a turning point for the project’s implementation progress. The MTR noted that the PDO of the original three components, as well as overall project design, remained valid. However, a number of decisions were made to improve implementation performance including: (i) revising some project subcomponents, in order to reflect a more logical grouping of activities, (ii) proceeding with a reallocation of Credit proceeds to reflect disconnects between the cost of civil works expected at appraisal and actual costs, (iii) focusing the implementation of the first phase of PHC reforms only in Podgorica, (iv) transferring the responsibility of implementing project activities from the working groups to government institutions; and (v) revising the results framework to promote a more effective monitoring of progress towards PDO attainment, including dropping two original outcome indicators – (i) “increased utilization of day care services by the elderly” and (ii) “the gap between PHC utilization of the poor and non-poor is reduced” - which were considered inconsistent with project design and its objectives. 42. In addition, the MoH decided that financing of activities for improved services for the elderly (per Part A of Schedule 2 of the DCA) should be dropped from the Credit and be initiated through the CIDA Grant, via development of options for palliative care. Also, the decision was made to drop the establishment of a primary care support center for Podgorica (as originally planned and referred to under Schedule 4 of the DCA).

43. These decisions led to a project level II restructuring in July 2007, through which the Bank endorsed all the agreements reached during the project’s MTR. Following the restructuring, project implementation showed good progress overall, leading to the upgrade of the DO and IP to Moderately Satisfactory and, at a later stage, to Satisfactory.

44. At the same time, the project was extended by 10 months from the original closing date until December 31, 2009. The extension allowed for the completion of the rehabilitation of two health centers in Podgorica, as well as a few technical assistance and training activities.

45. While the project continued to perform satisfactorily, an additional financing (IBRD Loan) was approved in November, 2009 in the amount of Euro 5.1 Million (US$7.2 Million equivalent). The AF funding was provided to support the extension of a few project activities (as described under section 1.7), and, to a lesser extent, cover cost over-runs of activities previously started under the original project.

9

46. AF implementation experienced certain delays due to the following factors: (i) lack of government certainty on how to best address hospital sector efficiency reforms (i.e. how to focus on financing tools, managerial autonomy, benchmarking and discrete restructuring of services rather than on historical planning norms and standards of services), which significantly delayed contracting of technical assistance in key related areas; and (ii) lack of clarity on requirements for the IT system for the Institute of Public Health. However, regardless of certain implementation issues and delays, the overall performance of the AF remained satisfactory.

47. Despite implementation issues and delays, the project successfully completed its planned activities. Key positive factors that influenced overall performance, included:

(i) Strong government commitment to reform the health sector. Despite relatively weak capacity, frequent staff changes and certain political tensions between different institutions involved in the project, the government has continuously displayed a long term strategic vision in the sector and commitment towards reforming the existing system.

(ii) Flexibility, within the original PDO, to address pressing implementation issues. Both the government and the Bank revealed flexibility during different project implementation stages, especially during the MTR (followed by first project restructuring in July 2007), and the AF (November, 2009).

(iii) Coordination with other international organizations. The project kept close coordination with other donors and international organizations. For example, during the AF implementation, the European Observatory for Health Systems was invited to support the TSU in procuring the IT system for the Institute of Public Health (IPH) through: (a) ensuring technical support for the development of the tender documentation; and (b) providing support to the IPH in defining its needs, which has proven crucial for the successful implementation of this activity. Also, the WHO provided significant support through their expertise, especially during the preparation of PHC standards and norms.

2.3 Monitoring and Evaluation (M&E) Design, Implementation, and Utilization 48. Design. The results framework in the PAD proved to be ambitious for a number of targets, in terms of monitoring and achieving them. No baseline data were defined at this stage, asking the related government institutions to carry out this task by project effectiveness. An M&E plan was developed and agreed with the counterparts, including specific instruments and responsible institutions for data collection. 49. Implementation. At the beginning of project implementation, M&E was relatively neglected. No baseline data were in place until project restructuring (July, 2007), resulting in a lack of ability to correctly monitor project achievements.

50. A number of significant decisions were taken during the project’s MTR, and formally approved at project restructuring. The results framework was intensely revised, to promote a more effective monitoring of progress towards the attainment of PDO, and outcome and output indicators were refined. Two of the project outcome indicators and a number of intermediate indicators were dropped (”increased utilization of day care services by elderly” and “the gap between PHC utilization of the poor and non-poor is reduced”) because their inclusion in the original results framework was considered inconsistent with project design and PDO. Although these changes were made, the intended outcomes of the project remained the same. In addition, an agreement was reached to hire a part-time M&E consultant to assist with data collection and close monitoring of project implementation. The work of the consultant started in November 2007, resulting in a significant improvement of the project’s M&E. In addition, a household

10

survey and public opinion survey was carried out in 2008 (supported by CIDA funding), to collect and provide appropriate information, before the original project closing date of December 31, 2009.

51. The AF (November, 2009) resulted in a second enhancement of the results framework, in order to reflect the newly introduced activities. Three new outcome indicators were added, to reflect the impact of newly introduced activities in the progress towards achievement of the PDO. Also, a few discrepancies found with the baseline data were corrected. A second household survey and public opinion survey was fielded (during the summer of 2011, with final results in 2012) to update the entire framework.

52. In addition to the fact that the M&E system was initially neglected, its ownership was weak and early reports were poor. However, the results framework and monitoring & reporting was significantly improved by the project’s MTR and deemed adequate for the remaining project implementation period.

53. Utilization. Besides monitoring internal project activities, the information collected after project restructuring included not only data related to the project’s results framework, but also other important data, which were used by the Bank and government (such as detailed data on quality, efficiency, and access to primary healthcare, and utilization of health care at all levels, etc). Specifically, this proved to be useful for the government while: (i) preparing the Master Plan of developing the health care system in the country for the period 2010-2013; and (ii) adopting the strategy for secondary and tertiary health care level reform, in July 2011.

2.4 Safeguard and Fiduciary Compliance 54. The environmental category of the project at appraisal was “C”, as the project was not designed to incur any major health or environmental impacts. The project supported physical rehabilitation of the existing primary health care facilities in Podgorica, which included only minor modeling and renovation activities. During the rehabilitation activities there were no hazardous wastes, and thus, there was no need for special plans for waste removal. There were only construction wastes, dumped on an authorized landfill, and managed by government waste management companies. In addition, reconstruction activities were carried out in full compliance with national environmental requirements and based on construction permits and urban planning documentation. During implementation of the original project, the scope of works on one site exceeded the initial design by including a small construction of a health center in Podgorica. However, an environmental safeguards assessment, carried out by the Bank team in collaboration with the government, concluded that no environmental or social harm occurred during construction and no subsequent mitigation measures were therefore found necessary. 55. The AF introduced the upgrade of the environmental category to “B”, as a result of the reconstruction of the Bijelo Polje health care center (added as part of the AF). The reconstruction included the expansion of the existing facility; an extension was added and an additional floor was placed on top of the existing facility. In addition, an agreement was reached to construct a small sewerage plant close to the health center, which was successfully completed by AF closure. An Environmental Assessment (EA) and a simple Environmental Management Plan (EMP) were prepared and implemented. The EMP included also a monitoring plan with environmental indicators, monitoring and reporting procedures, as well as institutional arrangements. Overall, no issues have been identified with the implementation of the safeguards requirements. All the works have been consistent with standards and environmental guidelines and safeguards compliance has been satisfactory, throughout the original project and the AF. This was also confirmed during the visit of the ICR team to the Bijelo Polje health center and a number of other rehabilitated primary care facilities in Podgorica. 56. Procurement. Considering the limited number of World Bank supported projects in Montenegro, the government established a central Technical Services Unit (TSU) initially under the General Secretariat (Prime Minister’s Department) and later on under the Ministry of Finance (MoF), responsible

11

for carrying out procurement functions for all Bank-financed projects. The Health System Improvement Project was the first one to work with these arrangements. While overall responsibility and decision-making authority was handled by the MoH, the TSU was accountable for providing prompt support in handling all procurement processes.

57. Project implementation suffered from a number of procurement-related delays, mainly due to:

(i) Lack of good communication between the TSU and MoH (including all project beneficiaries). While the required Terms of Reference or Technical Specifications were prepared by project beneficiaries, the procurement process was handled by the TSU. This arrangement required strong collaboration and close communication between all involved parties, which was lacking during periods of project implementation, resulting in delayed procurement.

(ii) Lack of clarity on decision-making authorities. A Memorandum of Understanding was signed between the MoH and MoF (TSU), outlining their rights and responsibilities for project implementation. However, its implementation proved to be difficult, resulting in delays in procurement processes for a number of activities (i.e: the physical rehabilitation of health care facilities, development of the information system, and purchasing of medical equipment).

(iii) Frequent staff changes in the MoH, TSU and all related institutions created a somewhat fluid environment for carrying out a timely procurement process.

58. However, overall the TSU had appropriate skills to successfully implement the project. Despite delays, the original project and AF activities were successfully completed, thus procurement has been rated as Moderately Satisfactory or Satisfactory throughout the project’s lifetime. 59. Financial Management. Similarly, the TSU carried the financial management responsibilities in a satisfactory way. Despite certain delays, caused by staff changes, overall the Unit had appropriate skills and ability to manage the project financial management and disbursement issues. No issues in the area of financial management have been identified. Quarterly financial reports were prepared and submitted in a timely manner, providing reliable financial information. Appropriate controls and procedures were instituted and described in the TSU operational manual since project inception. Financial statements were regularly audited by independent auditors, resulting in unqualified opinions. 2.5 Post-completion Operation/Next Phase 60. Transition Arrangements. The transitional arrangements put in place for future operations, seems to be adequate. The government has significantly improved financial performance of the health care system, by upgrading the legal framework and operations of the (HIF) and improving controls on pharmaceutical costs. In addition, a number of positive developments have taken place in PHC, including: (i) revising the PHC benefit package, (ii) introducing the concept of choosing the PHC doctor; and (iii) expanding the PHC infrastructure; leading to increased patient satisfaction and PHC utilization rates. While the government is strongly committed to sustaining achievements garnered to date, there is clear scope and need for further improvements in sector performance. In order to clearly assess the sustainability of the project’s achievements, it will be important that the government carry out an evaluation. This evaluation can also contribute to start discussions between the government and the Bank on future potential engagement. 61. Follow-up project. The government, during the ICR mission, expressed its strong interest in a follow-on project that will further support the on-going reforms. The Bank-Government of Montenegro Partnership Strategy (FY11 – FY14), has highlighted the following key priority areas in going forward:

12

(i) strengthening quality assurance programs at the central and local levels; and (ii) improving institutional capacity of the MoH, HIF and the MDA to enable their technical and managerial capacity to scale-up ongoing reforms.

3. Assessment of Outcomes

3.1 Relevance of Objectives, Design and Implementation 62. The relevance of objectives, design, and implementation is rated Substantial. Reforming the health system, with specific focus on improving the access and quality of primary care services and its financial sustainability, was a high priority for the government of Montenegro before the start of the project and remains so at completion. The PDO was fully relevant to the priorities of the national health sector strategy approved in 2003, as part of the overall Poverty Reduction Strategy Paper (PRSP). The explicit goals of the Transitional Support Strategy (TSS) of June 2000 and subsequent TSS Update covering FY04, included the development of human capital through improved health status by ensuring quality and cost-effective health services at all levels. The project has firmly supported these goals, improving primary health care service delivery, increasing capacity for policy planning and regulation, and stabilizing health financing. In addition, the project has served to address key challenges in the health sector reform, which clearly falls under one of the main pillars of the current Country Partnership Strategy (CPS) FY11-14. Achieving the full impact of these reforms will likely take a long time, hence the project’s impact would remain relevant even for the future development of the sector.

3.2 Achievement of Project Development Objectives 63. The achievement of the PDOs is rated as Substantial. This evaluation is based on the achievement of the outcome indicators (six out of ten outcome indicators are fully met, while the other four are only partially met), according to administrative data from the MoH, HIF and IPH, and the results from the household survey, conducted during July – August, 2012. It is also based on the analyses of the accomplishment of project intermediate indicators and the contribution of project activities to the achievement of the PDOs. 64. Achievement of PDO1: Increasing capacity for policy, planning, and regulation. With project support, capacity to develop a sustainable performance-oriented health care system has been strengthened. The project led to successful achievement of the two key outcome indicators, associated with this project objective. While these outcome indicators were introduced only as part of the AF, the ICR evaluation is also based on the analyses on how project activities contributed to the overall achievement of this objective. 65. Ministry of Health stewardship of the health system improved as measured by increases of health-related information and development of key health policies. The project was instrumental in supporting the MoH developing key strategic documents for health care reform and the related legal framework, providing the following impacts (over the original project life and AF):

- The Health Development Master Plan for 2010-2013 developed and adopted; - Secondary and Tertiary health care strategy adopted by the government and published on the

MoH website; - New strategy for quality assurance produced and adopted in February, 2012; - Decision on Network of Health institutions adopted; - National plan for development of human resources in health, for the period 2012-2022, developed

and adopted;

13

- Strategy for integrating private providers into the health system produced and approved, including a clear action plan and proposed revisions to the existing legal framework;

- Key laws, such as: (i) health care; (ii) health insurance; and (iii) record keeping were amended and adopted;

- The National Health Accounts produced (supported by CIDA funding). 66. With the support of the project, a total of eleven clinical protocols and guidelines were developed (ten of which were supported under the AF). The MoH has taken a very efficient approach to clinical guidelines development by selecting the most frequent diseases and those that require more financial resources. These two main groups account for around 70-80 percent of the variable costs of the health sector. The clinical pathways did not only include guidelines for diagnosis and treatment, but also at what level of the health network (primary, secondary or tertiary) patients should be receiving particular diagnosis and treatments. For example, the clinical guidelines for prevention of cervical cancer not only indicate the age group and frequency of screening, but also where the screening needs to be conducted (by primary care doctor or specialists in the hospital). These definitions have a critical impact on access and cost of services. 67. Strengthening the capacity of the MDA to improve regulation and oversight of the pharmaceutical sector (including the development of the Agency information system) led to the preparation and adoption of two new laws (law on medical devices and the law on pharmaceuticals) and 33 related sub-laws, in accordance with EU regulations. Furthermore, the Agency is able to proceed with the registration of new drugs, with help also provided by the Serbian Drug Agency.

68. Although the development of information systems for the MoH, HIF and IPH, experienced significant delays, it finally achieved its expected results on improving data collection, contracting, monitoring, and management of primary health care and related services.

69. The HIF increases its capacity to perform its functions of pooling health resources, better allocating health resources, and contracting providers toward improved performance. The project supported HIF capacity strengthening (through technical assistance, training, and the provision of the information system), that led to the establishment of a system based on output models of health care financing, consistent with the PDO. At project closure, all funds for health care services were pooled through the HIF, as the sole purchaser. A new model of contracting and financing of PHC was introduced: PHC operational based on capitation, while fee for services and adjustments for geographical location. The HIF established a comprehensive information system operational with pharmacists and PHC providers, which ensures monitoring and performance scheming.

70. With project assistance, the HIF introduced the DRG system for hospitals (as part of the AF), launching the necessary groundwork to shift the hospital payment system from historical budgets to the "products" that a hospital provides. The country was also helped to learn from better practices of its neighbors, particularly from Croatia, on the implementation of DRGs. All hospitals started DRG-based reporting before project closure. Intensive training of around 300 hospital staff (including seven general hospitals, two specialized hospitals, and the Clinical Center in Podgorica) and establishing an information system for hospitals is expected to ensure a smooth implementation of this reform. However, due to some delays, the project was not able to assist the HIF in completing the coding and grouping of diagnoses appropriate for the country. It has been agreed that this task would be developed by HIF using its own resources.

71. In addition, the Basic Benefit Package (BBP) was developed, with close coordination between the HIF and the IPH. Several working groups were organized (involving more than 50 specialists), which assessed the existing profile of services delivered, defined services to be provided (based on

14

epidemiological profile and costing) and the methodology to be used, taking into account cost effectiveness and cost benefit analyses. The BBP was developed based on approved clinical guidelines. 72. Achievement of PDO2: Stabilizing Health Financing. Although the results framework includes two outcome indicators linked to this PDO, there is no clear evidence to assess if health financing has been stabilized as a result of project activities. 73. HIF controls the increasing cost of nationally financed pharmaceuticals and medical materials. The indicator has been revised as part of the AF, especially its expected target value. The revision was made due to the effect of the economic crisis and frequent changes in the exchange rates for imported drugs.

74. By project closure, HIF’s share of expenditures on drugs was 20 percent (while the agreed target was 18 percent). Meanwhile, the share of expenditures on medical materials was 28 percent (with a target of 27.5 percent). Hence, the expected outcome on the HIF control over pharmaceuticals and medical materials is mostly achieved. It is also important to note that despite the HIF’s efforts for reducing drug prices, expanding the list of reimbursable drugs may have negatively affected the overall cost of drugs.

75. HIF annual deficit reduced. The HIF suffered small deficits during the first years of project implementation. However, by 2010 the institution became part of the State Treasury budget, hence resulting in zero annual deficit. Since then, the HIF budget is no longer directly linked with HIF revenues. There are a number of project-related activities that have positively influenced the stabilization of the health financing system. With project support, significant progress was achieved in developing the HIF’s managerial capacity on budget planning, expenditure and revenue forecasting as well as analyses and resource allocation. Several workshops and study tours were organized to help strengthening the HIF staff capacity. Technical assistance was provided to the HIF and the MDA, supporting them to successfully complete the following: (i) implementing a transparent pricing and procurement system to ensure cost containment; (ii) setting up a transparent drug registration system; (iii) implementing drug information system in compliance with overall health MIS, for transferring utilization and financial information from public and private pharmacies to the HIF and MDA; and (iv) setting up a quality control system.

76. Achievement of PDO3: Improving primary health care service delivery. With project support, overall primary health care services in Montenegro have significantly improved. The project has led to important achievements, in relation to the following key associated outcome indicators:

77. Reduced waiting time (from arrival to consultation) in primary health care. This indicator was successfully achieved. Relevant data are extracted from the HIF administrative data as well as two surveys organized in 2008 and 2012. Waiting time in primary care services in Podgorica met its target of having only 30 percent of population in the capital reporting a waiting time of 21 minutes. Meanwhile, reporting for the whole area outside Podgorica has even exceeded the agreed target: with 34 percent of the population (while the target was 53 percent) reporting a waiting time of 21 minutes.

78. Increased duration of PHC consultations. Based on survey data, the indicator was partially achieved. While only 50 percent of Podgorica patients (when the target was 70 percent) report a duration of 11 minutes or more; 55 percent of patients outside Podgorica (with a target of 55 percent) report a duration of 11 minutes. Hence the target has been fully met only for part of the PHC services, i.e. in areas outside Podgorica.

79. Increased utilization rates for primary health care for the general population (in Podgorica and outside Podgorica) and the Roma. The indicator is fully achieved. Survey data show that 44

15

percent of the general population (from a target of 40 percent) used the services of a general practitioner (GP). Meanwhile, majority of Roma adults (82 percent) have their own GP. Enrollment of Roma adults have significantly improved, considering that in 2008 (when the baseline data were collected), only 41 percent of Roma adults were enrolled with a GP. 80. Increased satisfaction for primary health care of the general population (in Podgorica and outside of Podgorica) and specifically for the Roma. The indicator is only partially achieved. Satisfaction with GP services has reached the targets for the population in Podgorica and at national level, with 77 percent satisfied and 19 percent partially satisfied in Podgorica and 70 percent satisfied and 24 percent partially satisfied at national level. However, in Roma settlements only 82.1 percent (while the target was 87.7 percent) declared that they were satisfied with the work of their GP doctor. Yet, this is a relatively large percentage of satisfaction. 81. Referral rates from PHC to hospitals is controlled. The indicator is fully achieved. Based on the last survey and the administrative data, the proportion of visits referred to specialists’ care was 25.71 percent for 2011 and 22.18 percent for the first 9 months of 2012. While the proportion of visits referred to hospitals was 1.82 percent for 2011 and 1.52 percent for the first 9 months of 2012. 82. Maintain or improve immunization rates (%) for DPT and measles, following reform. The indicator is only partially achieved. The main objective of this indicator was to maintain immunization rates high throughout the introduction of Chosen Doctor. IPH data shows that the immunization for the DPT (initial and follow-up) remained at 94.7 percent, while the target was 94 percent. Meanwhile, immunization for the Maternal Mortality Rate (MMR), initial and follow-up) is kept at 90.7 percent, when the target was 92 percent.

83. The project met or exceeded most of the targets outlined in the results framework, especially in reference to outputs and outcome indicators, thus it is deemed that the project was successful in improving delivery of primary health care services.

84. Primary health policy and standards. A number of documents defining PHC were produced and approved, including: (i) PHC organization model; (ii) PHC norms and standards on human resources, equipment, facilities, and services; and (iii) bylaws defining standards for teams of family physicians. These documents were crucial for the overall reform at the PHC level. Project support in this area was enhanced during the AF. A survey on cost and efficiency in primary health care centers was conducted (during 2010-2011), followed by a study that analyzed the productivity and efficiency of all 18 PHC centers in the country. The analyses provided by the study were taken over by the HIF (which continued the work based on its own administrative data), and used for further improvements of the PHC standards. 85. PHC Infrastructure Upgrade. The project supported the physical rehabilitation of eight out of eighteen PHC centers at the national level, including the center of Bijelo Polje, with the AF. Due to serious delays resulting from issues with cost overruns, three more centers (originally planned to be financed with project support) were taken over and completed by the government. Also, ten PHC centers were provided with required medical and non- medical equipment and with computer equipment. This upgraded infrastructure proved to be crucial for implementing the reform at the PHC level, bringing the delivery of primary health care up to standards and supporting PHC diagnosis and procedures, avoiding inappropriate referral to hospitals. 86. Capacity building for PHC staff. Project support was instrumental in introducing the chosen doctor concept at the PHC level and reorganizing the PHC around the primary care teams (primary physicians and nurses), which refers patients for more specialized care and diagnostic services. To enable the smooth implementation of this reform, wide training was provided, grouped in eleven rounds. A total

16

of 248 primary health care physicians and 278 nurses completed the entire training over the lifespan of the original project and additional financing. Several categories of expertise (international and local) were mobilized to carry out training needs assessments, prepare the training models, and organize all training rounds. To further improve capacity, a Family Medicine specialization was introduced at the University of Montenegro. Enabling legislation for the effective practice of this specialization as a career was fulfilled when the MoH adopted the Rulebook on the criteria of specialization. The first round of students enrolled in the regular specialization for the 2012 school year.

3.3 Efficiency

87. Efficiency is rated as Modest. The Montenegro Health System Improvement project had three components: (i) support for the health reform program to help government identify best practice in health policy financing and selected areas of service delivery; (ii) phased implementation of primary health care development, beginning with Podgorica, with scaling up reforms and development to other locations; and (iii) project management component to support the health sector organizations involved in the project activities led by the MoH. Although all of the above three project components generated benefits, most of them are from the second component.

88. The measures of economic efficiency under the base case scenario and two alternative scenarios are presented in Table 2 (see Annex 3 for a description of the base case and the two alternative scenarios). Under the base case scenario, the project yielded 21 percent Economic Rate of Return and a Benefit: Cost ratio of 1.82. The sensitivity analysis showed that even under conservative assumptions, the discounted project benefits covered costs resulting in modest returns to investment. The rates of return from the project are modest and are justified.

Table 2: Measures of economic efficiency under base case and alternative scenarios Base case Sensitivity analysis under two alternative scenarios Scenario 1 Scenario 2 Net present value (million $) 7.89 6.27 5.37 Benefit/Cost ratio 1.82 1.74 1.43 Economic rate of return 21 18 14

89. The project is expected to yield reasonable returns over the next few years. The operation and maintenance costs are not significantly high when compared with the public expenditures of health sector in the country. Therefore, the recurrent cost impacts of the project are negligible.

3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory 90. The project’s overall outcome rating is based on the achievement of project development objectives, as discussed under section 3.2 above, its relevance, and efficiency. The relevance of project objectives and its overall sound design is rated Substantial. Meanwhile, due to insufficient data, the evaluation of efficiency is carried out based on a number of assumptions. However, available data clearly show that the project, especially its second component, has generated considerable benefits and is expected to yield reasonable returns even during the coming years.

17

3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 91. The project contributed to improvements in the functioning of the health system and is also expected to have contributed to better affordability of primary health care services. Initially, the project’s results framework included two key outcome indicators related to the utilization of primary care by the poor (namely: “gap between PHC utilization of the poor and non-poor is reduced” and “utilization of day care services by the elderly is increased”). Since both indicators were not monitored and were dropped at project MTR, no clear qualitative evidence exists to prove the project’s impact on poverty. However, the project did have positive impacts on increasing the utilization rates of primary care for the Roma population (as described in section 3.2), even though it did not have a direct impact on gender aspects and social development. (b) Institutional Change/Strengthening 92. The project had a significant impact on institutional development and building long-term capacity. Interviews carried out during the ICR mission, which included officials from MoH, HIF, IPH, MDA and other project stakeholders, clearly highlighted the important impacts that project investments in capacity building and technology had on institutional development. The provision of technical assistance, training, and study tours significantly strengthened the HIF’s managerial capacity on budget planning, expenditure and revenue forecasting, as well as resource allocation. The project supported a large number of technical assistance and training to the HIF, MDA and IPH staff that were instrumental in enabling the establishment of needed structures for a performance-oriented health care system and quality assurance, such as: DRGs, BBP, clinical guidelines, and the new model of contracting and financing of PHC. Training for 248 primary health care physicians and 278 nurses was also provided, creating the required capacity for the smooth implementation of the above mentioned reform. In addition, the project enabled the adoption of laws and by-laws necessary for creating a legal framework to support health system reform. (c) Other Unintended Outcomes and Impacts (positive or negative) N/A

3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops N/A

4. Assessment of Risk to Development Outcome Rating: Moderate 93. The ICR team rates the risk that project development outcomes will not be maintained as Moderate. The government has demonstrated strong commitment, throughout the lifespan of the project towards achieving and maintaining its outcomes and, more broadly, supporting the overall reform of the health sector. Key reforms are currently preserved in a solid legal framework and a number of strategic documents, hence they are unlikely to be reversed. Although the country has been facing economic crises, financing of health sector reforms has continuously been among the government’s priorities. This was clearly revealed during project implementation, when the government took over financing some of the infrastructure in PHCs (including three PHC centers in Podgorica and the one in Bjelo Polje), as well as IT investments, which were key outputs for the reform in PHC and health financing. In addition, key

18

project-supported investments (i.e, reforming PHC services through registration with a chosen doctor, and introducing BBPs and DRGs) have been institutionalized. 94. However, caution should be exercised so as not to be too optimistic about the government’s capacity to remain committed to the reform, especially considering the high level of staff turnover in the MoH and other main related institutions. These changes may negatively influence the government’s commitment towards sector reform, as well as evolving priorities. In addition, without a follow-up support project by the Bank or other donors, the government may not have sufficient capacity to continue maintaining and deepening the system’s reforms (including the completion and utilization of DRGs, BBPs, clinical guidelines, and continuation of medical education among others).

5. Assessment of Bank and Borrower Performance

5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory 95. Project preparation was carried out with an adequate number of experts, providing the required technical skills to address sector issues and generally produce a proper design. The team considered a number of alternatives during the preparation process, as well as major risk factors and lessons learned from previous Bank involvement in the country. 96. The project’s consistency with the government’s strategy and priorities in the health sector was assured. The Bank team maintained good working relations with the Borrower throughout preparation.

97. Conversely, the project was prepared very rapidly so as to benefit from IDA eligibility. As a result, the team missed the opportunity to prepare detailed and sound implementation approaches. Also, a few ambiguities and duplications are noticed in the formulation of certain project activities. More importantly, the design could not put in place a proper M&E system for monitoring project implementation and assessing its achievements. Baseline data were lacking for most indicators and there was no forecasted activity related to a baseline survey. Although these shortcomings were addressed during project implementation (mainly during the MTR), the assessment of project progress and its achievements thus became quite challenging. (b) Quality of Supervision Rating: Satisfactory 98. The Bank’s performance during the implementation of the project was satisfactory. Task teams focused on the project’s development impact. Sufficient budget and staff resources were allocated and the project was adequately supervised and closely monitored. Bank supervision took place on a regular basis and provided appropriate and well-targeted advice and observations. The aide memoires provided evidence of regular supervision and professional advice given by the Bank’s experts throughout the project’s lifetime. In addition, interviews with stakeholders clearly show the government’s appreciation of the technical skills and advice provided by the Bank’s experts. 99. The Bank team was quite flexible in making adjustments, especially during the project’s MTR, which led to project restructuring in July, 2007. Following the restructuring, the team was commendably proactive in identifying implementation bottlenecks and measures to overcome project implementation delays. In addition, the task team’s flexibility and quick decision-making reflected the support received

19

from members of the Bank’s senior management team, especially during difficult periods of implementation. 100. In spite of frequent changes of TTLs (in total six), the smooth transition from one to the next, provided continuity and reduced any possible negative impact of such frequent change.

101. Fiduciary and safeguards policies were well managed and reported. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Satisfactory 102. While quality at entry was less than satisfactory, the Bank team’s performance significantly improved during project supervision, especially from the project MTR until closure. However, despite team’s efforts during supervision, certain shortcomings from design stage, such as M&E, could not be completely bridged. Hence, overall performance of the Bank is rated as Moderately Satisfactory.

5.2 Borrower Performance (a) Government Performance Rating: Satisfactory 103. The project was designed and mostly implemented by a government that has shown strong commitment towards reforming the health sector. In addition, the government demonstrated a long term strategic vision in the sector over the project’s lifetime. Government officials worked closely and fully cooperated with the Bank’s project team, and on a continual basis. The Steering Committee, headed by the Minister of Health and comprised of officials representing key project stakeholders, played a key role in strategic decision making and project monitoring. It was the initiative of this committee during project MTR, to carefully analyze the functioning of the working groups (responsible for implementing activities in each project component) and determine whether these groups created bottlenecks and therefore slowed down project implementation pace. This was resolved by the Committee’s decision to replace the working groups with institutions appropriately qualified to effectively implement project activities. 104. The project did not experience any counterpart funding problems. In contrast, the government took over funding of the physical rehabilitation of three PHC centers that could not be completed with project support due to significant cost overruns. Also, during the AF, the government took full responsibility for covering the entire counterpart funding for the physical rehabilitation of the Bijelo Polje health center, including the part originally planned to be covered by the municipality (due to a lack of local government financial resources). 105. Overall, government performance in ensuring political and institutional commitment was satisfactory. The first few years of project implementation was a period of political transition for the government; although implementation suffered significant delays at this time, the situation was quickly rectified. (b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory.

106. Although the MoH was the lead implementing agency for the project, daily project implementation was carried out by the Project Management Network (PMN), with all fiduciary functions managed by the Technical Services Unit (TSU) attached to the MoF. In general, both units had the required skills to implement the project. The “moderately satisfactory” rating is based on the

20

achievement of all project planned outputs and the flexibility these organizations demonstrated while working in a changing environment. 107. However, the manner in which project implementation was organized proved to be quite challenging. The separation of project management functions from the fiduciary ones resulted in coordination issues, which negatively affected the project implementation pace.

108. In addition, both units went through a number of staff changes during the project’s lifetime. Due to delays in replacing of staff, project implementation suffered more than once from a lack of a fully functional PMN and TSU. This created a somewhat fluid environment, especially during the first years of project implementation. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory 109. The satisfactory performance of the government and moderately satisfactory of the Implementing Agency averages to a rating of Moderately Satisfactory for overall borrower performance.

6. Lessons Learned 110. The need for quick project preparation must be balanced with the quality of project at entry. The project was prepared quite rapidly, with only nine months from concept review to Board approval. Additionally, the country situation at preparation stage was still fluid, coming out of a turbulent phase of its history. Although the speedy project preparation was related to benefiting from IDA financing, it negatively affected project implementation, especially during the first years. In addition, while designing the project, specific needs and situations of the country, such as human capacity, especially the capacity to undertake and implement major project activities, should be carefully considered.

111. Agreed institutional arrangements, government commitment and a viable single agency are basic prerequisites for a successful project implementation. The lead project implementing agency was the MoH, supported by a Project Management Unit (PMU), whereas all fiduciary functions were carried by the TSU under the MoF. While overall the government has shown strong commitment, project implementation suffered significant delays due to: (i) lack of coordination between the implementing agency/PMU/TSU and various project stakeholders; and (ii) frequent staff changes at all levels.

112. Both the Bank and the government need to involve proper expertise in different areas, to ensure a smooth implementation of project activities. While very good expertise in the areas of medical equipment and information systems was provided, project supervision lacked the expertise in the civil works area. Having no expert civil engineer to support the teams with reviewing required technical specifications and bid evaluations for all reconstruction works negatively influenced the implementation pace (including delays in completing most civil works and cost overrun issues in a number of PHC centers).

113. Lack of a proper M&E design raises questions on the overall quality of project design. Furthermore, project monitoring becomes challenging and it is quite difficult to prove project achievements. The lack of baseline data for most outcome and output indicators can negatively affect overall project relevance. The results framework for this project was somewhat neglected at design stage and continued to be so until project MTR.

21

114. Close coordination with other development partners is very important. Coordination with key donors involved in the sector, such as: WHO and EU Observatory, proved to be very important for keeping implementation of certain project activities on track.

115. Health sector reforms are long-term and quite complex. The project successfully supported key reforms at all levels of health care, including health financing efficiency, quality of services, and strengthening institutional capacity. In order to sustain, and, most importantly, further reform the health sector in the country, it is important that the Bank considers follow-up actions.

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies See Annex 7 for Borrower’s contribution to the ICR. Meanwhile, no comments were received to the letter sent by the Bank, dated May 28, 2013, requesting comments on the draft ICR from the Borrower. (b) Cofinanciers 116. The project was co-financed by a CIDA grant (TF055474), for an amount of US$0.82 million. Initially, at the time of project appraisal, agreement was reached on the amount of US$0.57 million grant from CIDA. However, as per the amendment to the Grant Agreement, dated October 31, 2008, the grant amount was increased to US$0.82 million. 117. The grant has contributed to policy development and capacity building for the MoH, HIF, the IPH, and to translation of improved knowledge and skills into concrete actions, while implementing PHC reforms. 118. Key activities supported by the grant (through provision of technical assistance) included:

(i) Training program in management provided in two modules to MoH, HIF PHC and general hospital representatives.

(ii) Household Survey conducted and data analyses provided to the MoH and other related health institutions.

(iii) National Health Accounts produced in 2008. (iv) Support provided to strengthen the M&E capacity in the PMN. (v) Training program for selected PHC staff working specifically with children with special

needs. The grant closed on February 17, 2009, fully disbursed. (c) Other partners and stakeholders N/A

22

Annex 1. Project Costs and Financing

(a) Project Cost by Component (in USD Million equivalent) Components Appraisal

Estimate (USD

millions)

Restructured Estimate IDA

(USD millions)

Actual/Latest Estimate

(USD millions)

Percentage of Restructured

Estimate

Appraisal Estimate AF (USD millions)

Actual/Latest Estimate AF

(USD millions)

Percentage of AF Appraisal

1.Support for health reform program

2.45 1.14 1.18 103.5% 3.99 2.62 66%5

2.Primary health care development

5.57 7.69 7.42 96.5% 6.49 5.00 77%6

3.Project management 0.99 1.06 0.94 89% 0.20 0.99 490%7

Total Baseline Cost 9.01 9.89 9.54 97% 10.68 8.61 81%Physical Contingencies 0.6 Price Contingencies 0.28 Total Project Costs Front-end fee 0.00 0.00 0.00 0.00 0.01 0.01Total Financing Required

9.89 9.89 9.54 97% 10.69 8.62 81%8

5 Due to cost savings in a number of project activities, including purchasing of IT systems, international and local consultancies and workshops. In addition, part of the funding for IT PHC software and coding for the DRG was taken over by the HIF budget.

6 Due to significant savings in purchasing medical and non-medical equipment, as result of price reduction.

7 Due to increase number of staff in the project management unit as well as strengthening the TSU

8 Overall, decrease in the actual project costs reflect exchange rate fluctuations in the Euro (AF) to $ US, during AF life.

23

(b) Financing

Source of Funds Appraisal Estimate (USD millions)

Actual/Latest Estimate (USD millions)

Percentage of Appraisal

Borrower 5.02 3.7 74% CANADA: Canadian International Development Agency (CIDA)

0.57 0.82 144%

International Bank for Reconstruction and Development

7.20 6.43 90%

International Development Association (IDA)

7.00 7.03 100.4%

Municipalities of Borrowing Country 0.79 0.00 0.00 Total Financing 20.58 18.16 91%

24

Annex 2. Outputs by Component

Component Planned outputs at Appraisal Actual outputs/outcomes at ICR 1. Component – Support for health reform program (US$ 3.8 million)

(1.1) health policy development and capacity building for the MOH, HIF, and IPH focused on health finance, pharmaceuticals regulation and expenditure management, planning and strategy development, communications and public information to support reform, and monitoring and evaluation.

All support centers were introduced into practice, apart from the Centre for education and centre for daily support and care; As a result of technical service for planning of health services norms and standards for health care service delivery, including norms and standards for human resources at the PHC level have been developed; The project supported development of an outline for human resources development in health sector; The project supported development and adoption of two laws (Law on pharmaceuticals and medical devices.), and 33 by-laws.

(1.2) investments in health information systems to improve data collection, contracting, monitoring and management of primary health care, and related services

As a result of provided technical assistance through the project, a new payment model for PHC was developed; Development, implementation and maintenance of the PHC temporary software has been fully financed by the HIF; International and national expertise was mobilized under the project to support activities on developing the contract template to be used between the Health insurance fund and PHC centers;

2 - Phased implementation of primary health care development, beginning in Podgorica, with scaling up of aspects of reform and development to other locations (US$ 12.42 million)

(2.1) First phase of implementation in Podgorica which would be used to develop the reform model and learn lessons before scaling up reforms to other parts of Montenegro. This sub-component would also finance civil works to carry out internal remodeling of existing primary care facilities, to rationalize facilities and improve the functional lay-out of building so as to support development of

The program was developed to elaborate clearly how the Model of organization of the PHC system will be supported from the functional point of view. The project supported implementation of civil works on all sites as per the Program on spatial organization except Nova Varos, Stara Varos, Studentska poliklinkia, Ubli I Vrela Ribnicka due to considerable cost

25

Component Planned outputs at Appraisal Actual outputs/outcomes at ICR group practice and to consolidate specialized services that support primary care from multiple sites into a single Primary Care Support Center.

over runs on certain construction sites. However, the government financed completion of civil works on the following sites: Nova, Varos and Stara Varos. Technical assistance was provided to develop the relevant detailed civil works designs, to conduct technical review of the detailed civil works designs and supervise conduct of civil works on all construction sites. The project supported procurement of medical equipment for diagnostics and therapeutic services provided by the chosen doctors and in Support centres, as well as the non-medical equipment and furniture.

(2.2) Further development of primary health care policy, standards, and implementation capacity, through technical assistance and training.

As a result of technical assistance mobilized under the project the following activities were implemented:

(i) IT support provided to the registration process

(ii) IT system concept was developed

(iii) The software application was introduced into practice

(iv) Hardware procurement (v) Capacity building

activities In order to support successful implementation of the software solution/application it was indispensable to upgrade the level of understanding, knowledge and skills in using the information system in all PHC centres at the national level. Consequently, numerous trainings were supported under the project. All chosen doctors attended the IT courses. Also, it was enabled for the first time to acquire the internationally recognized licence : ECDL (European Computer Driving License).

(2.3) Development of primary care The project supported development of

26

Component Planned outputs at Appraisal Actual outputs/outcomes at ICR specialization training and continuing professional development.

the Model of re-training for doctors and nurses that envisaged implementation of the

(i) main training course (ii) optional training course

According to the Model of re-training some training programs were attended jointly by doctors and nurse. Optional courses did not apply to nurses. Activities implemented under this sub-component resulted also in development of the Spatial plan for organization of the Training centre located at the PHC centre in Podgorica. During the project life 9 (nine) rounds of re-training courses were conducted, and trained 199 doctors and 243 nurses from the whole Montenegro.

(2.4) Support for phased scaling-up of implementation of the reforms piloted in Podgorica to the rest of the country. Mobilization of additional resources from local and donor sources will be necessary to fully implement all aspects of the primary care development strategy, extending beyond the life of the Project.

As a result of project activities implementation under this sub-component (i) PR campaign was designed

and implemented (ii) Three public opinion surveys

were conducted in 2004, 2006 and 2008

During the PR campaign numerous PR instruments were prepared and disseminated. The higher percentage of citizens and health workers with better understanding of the health care reform and related benefits and consequently higher percentage of supporters has been recorded.

3 - Project management (US$1.93 million).

(3.1) the operation of the PMN under the MOH

The PMN was established and comprised of the project coordinator and administrative assistant. During project life several component coordinators were hired:

PHC coordinator Coordinator for health policy Finance health coordinator

27

Component Planned outputs at Appraisal Actual outputs/outcomes at ICR IT coordinator PR coordinator Training coordinator Pharmaceuticals coordinator

(3.2) the operation of the central TSU

serving the health project and other projects.

Technical services unit was also established as it was envisaged by the project and it was fully financed by the project.

28

Annex 3. Economic and Financial Analysis

1. Introduction The health sector in Montenegro is characterized by high public health and total health expenditures; high expenditures for pharmaceuticals; weak capacity for policy development, planning, forecasting, and management; and an insignificant role played by primary health care in prevention, diagnosis, and treatment. The motivation for the project was the high public health expenditures in Montenegro and the need to reform the health sector to achieve sustainable levels of health care financing and health expenditures. The project helped the Government of Montenegro to reform the health sector and improve the efficiency and development of the primary health care system so as to attain financial sustainability. The project had three components:

a. Support for the health reform program to help the government identify best practices in health policy, financing, and selected areas of service delivery. This component had two sub-components. The first sub-component was intended to develop health policy and capacity building of the MoH, HIF, and IPH. The second sub-component was intended for investments in health information systems to improve data collection, contracting, monitoring, and management of primary health care and related services.

b. Phased implementation of primary health care development, beginning with Podgorica, with scaling up reforms and development to other locations. Under this component the project supported the development of primary care policy, standards and implementation capacity through technical assistance and training. The activities included the development primary care specialization training and continuing professional development. This component also supported the phased scaling up of implementation of the reforms piloted in Podgorica to the rest of the country. In initial phase of implementation the project facilitated patient choice and registration with a primary care doctor, consolidation of specialized primary care based services, streamlining organization of services to reduce patient waiting times and more efficient use of time with patients and motivating medical staff to provide better care.

c. Project management component to support the health sector organizations involved in the project activities led by the MoH.

The above mentioned three components improved the overall efficiency of the health sector by improving the delivery of primary health care and hospital services. The benefits and gains from improved efficiency were passed on to the patients in the form of higher quality of care through increased access to primary care, improved referral process and specialized treatments, lower treatment costs, lower out of pocket payments and less severe health problems among patients. As a result of improved access to quality care, better patient case management and use of preventive care people reported less fewer hospital referrals, faster recovery, less in-patient admissions, reduced mortality and fewer days lost through diseases and inactivity. Thus, the benefits from the project included a decrease in unnecessary hospital bed days due to improved quality of primary health care and the savings in hospital expenses, reductions in referrals to specialists and costs of specialist care, reductions in out-of-pocket expenditures, travel times and travel costs, losses in labor productivity averted and benefits from reduced mortality. 2. Methodology, data and assumptions All three project components complemented each other and generated both direct and indirect benefits, particularly with regards to Component 2 – “Phased implementation of primary health care development, beginning with Podgorica, with scaling up reforms and development to other locations”. The methods used to estimate those benefits are presented below.

29

2.1. Benefits The benefits from the project included:

a. Reduction in hospital outpatient care and hospital admissions as a result of the increase in quality and access to primary care in health houses and ambulatory consultations.

The project facilitated better primary care and consultations in health centers and ambulatory consultation facilities by increasing consultations from 52 to 57 percent in both settings. Hospital visits were also reduced annually by about 5 percent from 31 percent to 26 percent. Further, there were reductions in unnecessary hospital admissions which resulted in savings in total bed days and inpatient care expenses. Such savings in bed days were as high as about 20- 25 percent. Since we did not have data on actual cost of inpatient care avoided in Montenegro, we have estimated it from the national data on inpatient hospital expenses and the number of hospital days in Montenegro. It was then adjusted for inflation to represent the data over the project period. The cost per inpatient day is estimated at US$51 and the recurrent costs of the hospitals are assumed to be 20 percent of the total inpatient costs at US$9. The above costs are then adjusted using the inflation rates in Montenegro.

b. Savings from unnecessary specialist referrals due to improved care and case management in primary care centers and adherence to standard specialist care protocols and consolidation of primary care services.

In addition to reduction in hospital care, the development of primary health care program and the improved referral program resulted in a reduction in the number of specialist referrals and treatment. The beneficiary surveys and project progress reports showed a reduction in specialist visits and inpatient hospital care by as much as 25 percent in the project area.

c. Reduction in disease incidence in days and thus averted losses in productivity due to diseases and lost workdays.

Averted productivity and man days arise from fewer disease incidence leading to a reduction in inpatient admissions and therefore bed days. The reduced productivity losses are estimated to be 0.5 days per referral. Similarly, fewer admissions and fewer bed days results in significant savings in productivity. Reduction in number of bed days as a result of the project has been estimated at 30 percent. Furthermore, in order to estimate the value of averted productivity losses, productivity is valued at an average wage rate of US$3.2 per person per day at a labor participation rate of 54.5 percent.

d. Reduction in patient waiting times at primary health centers.

Streamlining the appointment process has reduced patient waiting time at the health care facilities. The average waiting time has been reduced by about 5 percent. The resulting savings in waiting time has been valued at the daily wage rate.

e. Savings in potential life years as a result of reduced mortality from diseases.

Better quality of treatment and preventive measures averted chronic diseases and mortality and the burden of diseases in Montenegro. According to the World Health Organization’s Atlas of Health in Europe, improved monitoring and treatment of respiratory and circulatory system diseases as well as early detection of diseases of digestive system reduces mortality rates from these diseases. Based on available evidence from surveys and evidence from other published works, early detection through higher utilization of public health services and better quality of care has reduced the mortality rate by about 3 percent over a ten-year period. In order to compute the gains from reduced mortality it is assumed that each death averted results in nine life years saved. The monetary value of life years saved was then estimated from the per capita incomes over nine years at a at 5 percent discount rate. The project

30

supervision documents and the beneficiary surveys showed that the benefits started to materialize from the second year of project implementation.

f. Reduction in expenditures for pharmaceuticals through regulation and expenditure control

The regulation and expenditure control supported by the project has reduced the price of pharmaceuticals sold in the country which reduced the out-of-pocket (OOP) expenditures for the population. These reductions in out of pocket expenditures for pharmaceuticals then become a benefit from the project.

g. Reduction in travel costs to hospitals and specialist clinics

OOP expenses and travel costs to the hospitals are in general greater than for visits to primary care centers. Hence a reduction in specialist consultation will result in savings in travel costs and OOP expenses. The difference in OOP and travel costs for the reduced number of visits to specialty centers then become project benefits.

h. Savings from reduction in space and use of fuel/energy

Reduction in space and fuel/energy savings through consolidation of primary-care based services into primary care reference center resulted in 30 percent reduction in space requirements. 3. Project costs The project had three components, of which Component 2 accounted for most of the project costs and benefits. The total project cost including additional financing was 88.2 percent of the appraisal estimate (Annex 1). The IBRD and IDA accounted for 35.4 percent and 38.7 percent respectively of the total project costs. Note that the borrower country shared a significant proportion (20.4%) of the total project costs, which indicates a higher country ownership of the project. The Canadian International Development Agency (CIDA) had a minor share in the total project costs are US$0.8 million accounting for 4.5 percent of the total project costs. In addition to the capital costs, the recurrent costs for the years 7 to 10 and the rest of the project period are estimated at US$38,100. The incremental staff salaries are assumed to be self-financing after the sixth year. The project costs include recurrent costs, excluding. The cost of outpatient visits vary significantly based on which particular health unit of the hospital is utilized. Based on data provided by the MoH, the average cost per visit at the primary care level is US$6.2 while it is around US$12.4 at the secondary level. 4. Project benefits and measures of economic efficiency As mentioned above, the project benefits arise from a reduction in hospital outpatient care and admissions, reduction in referrals to specialists, reduction in disease incidence and savings and averted losses in productivity, savings in potential life years as a result of reduced mortality, reduction in expenditures for pharmaceuticals and reduction in travel costs to hospitals and specialist care centers. The analysis considered a base case scenario and three alternative scenarios. Under the base case scenario, most project benefits started to accrue in the third year. The benefits from reduction in costs of pharmaceutical start to accrue in the second year though at a limited scale but continue to accrue until 20 years. As a result of the project, the outpatient treatments at hospitals and referrals to specialists were reduced on the average by 5 percent. The project resulted in a reduction in mortality by about 2.9 percent. The measures of economic efficiency of the project under the base case scenario are presented in the following table.

31

Table A3.1: Base case scenario: Measures of efficiency (5% discount rate) 5 percent discount rate Net present value (US$ million) 7.89 Benefit/Cost ratio 1.82 Economic rate of return 21 Most of the benefits from the project started to be realized from the third year of the project. Under the base case scenario, the project yielded modest returns on the investment as these returns compare closely with the rates of returns and measures of project as presented in the PAD with most of the benefits generated from Component ?. 5. Sensitivity analysis The total project benefits and measures of economic efficiency are sensitive to the different project benefits considered. Reduction in one or more of the benefits would affect the net present values and rates of return of the project. The sensitivities of the project’s results are considered under two scenarios.

i) Scenario 1: Reduction in hospital outpatient visits is only 3 percent (in the base case scenario, the reduction in hospital visits is 5 percent)

ii) Scenario 2: There is an increase in recurrent costs by about 50 percent.

Table A.3.2: Sensitivity analysis under alternative scenarios (5% discount rate) Scenario 1 Scenario 2 Net present value (Million $) 6.27 5.37 Benefit/Cost ratio 1.74 1.43

Economic rate of return 18 14 The sensitivity analysis showed that the returns from the project are positive even under the above two scenarios. The effect of an increase in recurrent costs seems to have a higher impact on the returns from the project. On the whole, these results under the base case scenario and the alternative cases are comparable to the ex-ante results presented in the PAD.

32

Annex 4. Bank Lending and Implementation Support/Supervision Processes

(a) Task Team members

Names Title Unit Responsibility/ Specialty

Lending Dominic S. Haazen Lead Health Policy Specialist AFTHW Loraine Hawkins Consultant EASHD Virginia H. Jackson Consultant MNSHD Richard James Senior Operations Officer AFTN2 Peyvand Khaleghian Country Sector Coordinator ECSH1 Cem Mete Lead Economist SASSP Marina Petrovic Consultant ECSHD Lindsay Norman Sales Consultant MNSHD

Supervision/ICR Ivana Aleksic Human Development Specialist ECSH2 Caryn Bredenkamp Senior Economist HDNHE Sarbani Chakraborty Senior Health Specialist EASHH Olav Rex Christensen Senior Public Finance Speciali HDNED Aleksandar Crnomarkovic Sr Financial Management Specia ECSO3 Francois Decaillet Lead Public Health Specialist ECSH1 Armin H. Fidler Advisor, Policy and Strategy HDNHE Michael Gascoyne Senior Resource Management Off WBICA Ana Holt Health Specialist ECSH1 Nikola Kerleta Procurement Specialist ECSO2 Carmen F. Laurente Senior Program Assistant ECSHD Zorica Lesic Operations Officer ECSH2 Sanela Ljuca Operations Officer ECCBM Silvia Mauri Consultant ECSAR Imelda Mueller Operations Analyst ECSH2 Jan-Peter Olters Country Manager ECCKO-DIV Marina Petrovic Consultant ECSHD Gennady Pilch Senior Counsel LEGOP Pia Helene Schneider Lead Evaluation Officer IEGPS Darius Stangu Financial Analyst CFPMI Sreypov Tep Program Assistant ECSHD Ethan Yeh Economist ECSH1 Kari Hurt Senior Operations Officer ECSH1 Carlos Marcelo Bortman Senior Public Health Specialist ECSH1

33

(b) Staff Time and Cost

Stage of Project Cycle Staff Time and Cost (Bank Budget Only)

No. of staff weeks USD Thousands (including travel and consultant costs)

Lending FY03 3.50 12,970.57FY04 42.14 284,930.56

Total: 45.64 297,901.10Supervision/ICR

FY05 27.64 106,557.15FY06 28.58 78,447.68FY07 30.38 103,297.81FY08 35.53 89,768.41FY09 25.20 72,727.52FY10 28.29 85,313.01FY11 31.48 105,968.27FY12 22.61 101,115.02FY13 19.56 65,069.35

Total: 249.27 808,264.22

34

Annex 5. Beneficiary Survey Results N/A

35

Annex 6. Stakeholder Workshop Report and Results N/A

36

Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR

1. Summary of Borrower’s ICR

MONTENEGRO HEALTH SYSTEM IMPROVEMENT PROJECT9 Summary Common to other transitional countries, Montenegro experienced threats to health status during the 1990s. Conflicts, disasters and considerable macroeconomic changes had an adverse effect on the health status of the population.

However, despite all difficult factors, the national average of some of the vital indicators has improved over the past years. It is obvious that Montenegrin population is getting older and its demographic structure resembles very much the European countries demographic structures. Also, Montenegro has faced a growing demand for access to quality and efficient health services and health expenditures.

Like many European countries Montenegro has experienced an epidemiological transition and it faces an increased burden of non-communicable diseases related to high-risk behaviors and inadequate nutrition. The health status of the poor and other vulnerable groups is particularly at risk. The Ministry of Health (MoH) plays a central role in providing stewardship function for the health system and has centralized approach to health system management. Since 2003 considerable reform efforts were invested in line with ‘Health Services Policy in the Republic of Montenegro Up to the Year 2020’ (2001) and “Strategy for Health Care Development in Montenegro” (2003) to strengthen priority areas of health policy, planning and regulation, build further institutional capacity, improve quality, efficiency and access in primary health care, take measurable steps towards ensuring financial sustainability of the health care system, improve governance of the health insurance and health care and develop health information system to support management and clinical care. In May 2010, the Government of Montenegro adopted the Master plan on development of healthcare system of Montenegro for the period 2010 – 2013 identifying direction and actions of health interventions aimed at hospital optimization (introduction of the new payment model based on outputs, redesign of the health institutions network reflecting the health needs, public private partnership) facilitating delivery of socially responsive, equitable, accessible and quality services to all the people of Montenegro. In July 2011 the government adopted the Strategy for secondary and tertiary health care level reform with the aim to ensure an adequate response to the population’s health needs and enable equal access to quality health services at the secondary and tertiary level of care to all social groups regardless of their socio-economic status and geographical distribution

9 The Borrower’s full contribution to this ICR is available upon request. This summary presents core information on the project and its performance.

37

Project Design, Project Objectives and Implementation The development objective of the Health System Improvement Project (PDO) was to put in place the first phase of steps towards reform of the health system in Montenegro, giving priority to increasing capacity for policy, planning and regulation; stabilizing health financing and improving primary health care service delivery. Specifically, the project was intended to: (i) support improvement in financial sustainability of the health care system by strengthening institutional capacity and information systems for health policy, planning, regulation and management in the MOH, HIF and IPH; (ii) improve quality, efficiency and access to primary health care services, by investment in training of staff, facilities and equipment and reform of organization and financing of primary care beginning in Podgorica, and extending reform to areas outside Podgorica; and (iii) support a project management network (PMN) of the Ministry of Health (MOH) and a central Technical Services Unit (TSU) that will provide procurement and financial management services for this and other future World Bank-financed projects. Components

The project was comprised of the following three components:

1. Support for health reform program of Ministry of Health (MOH) and Health Insurance Fund (HIF). This component was directed at helping the government to identify best practice in health policy, financing and selected areas of service delivery. The intention was that policy development and adoption would be built into the project and supported by it, rather than being fully determined ex-ante.

2. Phased implementation of primary health care development, beginning in Podgorica and additional locations in the rest of Montenegro. This component would support phased implementation of policy and plans to develop primary health care.

3. Project management, monitoring and evaluation. It was agreed that the Ministry of Health would be the lead implementation agency for the project. It established a project Management Network (PMN). Also the central Technical Services Unit (TSU), responsible for carrying out core procurement and financial management functions for all future Bank-financed projects, was established in the General Secretariat under the office of the Deputy Prime Minister.

a) Original Financing

The original Credit (Cr. 3918-YF) was defined in 2004 to include the three following components.

ORIGINAL PROJECT COMPONENTS

Total cost (US$)

1. SUPPORT FOR HEALTH REFORM PROGRAM

$2,200,000

1.1 Health Policy Development and Capacity Building 1.2 Health Information Systems 2. PHASED IMPLEMENTATION PF PHC DEVELOPMENT, beginning in Podgorica, with scaling up of aspects of reform and development to other locations

$6,600,000

2.1 First phase implementation in Podgorica 2.2 Primary health care policy, standards and implementation capacity 2.3 Primary care specialization training and continuing professional development 2.4 Phased scaling-up of implementation of the reforms 3. PROJECT MANAGEMENT, MONITORING AND EVALUATION $1,000,000 3.1 PMN 3.2 TSU GRAND TOTAL $9,800,000

38

Within the first two years of project implementation, some of the activities planned under HSIP (such as the renovation of a few facilities in the PHC Network, the training provided to the PHC doctors, etc.) were started and did contribute to the successful implementation of the reform in Podgorica. However, political tensions and unclear division of roles and responsibilities between the regulatory bodies (MOH, IPH), the purchaser (HIF) and the providers, as well as the arrangement of a network of working groups proved to be a bottle neck for a smooth and timely project implementation. These factors limited communication and cooperation among stakeholders and delayed the project implementation with respect to its original schedule. The delay put at risk the success of the project and, most importantly, and constituted a serious threat that the health system would be prevented from delivering the quality, equitable and efficient health services to the population of Montenegro. On February 12-17 2007 the Mid-Term Review (MTR) of the Montenegro Health System Improvement Project (HSIP) took place. During the MTR, the MOH and other project stakeholders and the Bank team agreed to restructure the project but to keep the same development objectives as the original project. Following MTR and restructuring, it was decided that the project activities would continue evolving around three original components. However, sub-components were revised to reflect a more logical grouping of the activities financed by the IDA Credit. The new restructured project reflected the following decisions10: (1) The implementation of the first phase of the PHC reform will be focused on Podgorica; (2) The proposal to create a Primary Health Care Support Center in Podgorica has been dropped since the same planned activities have been integrated into the existing PHC Network; and (3) the responsibility of leading activities in each component and subcomponent has been transferred from Working Groups to Institutions in the Government of Montenegro. During the MTR it was also agreed to reallocate the Credit Proceeds by Category of Expenditures.

Outputs

COMPONENT 1: SUPPORT FOR HEALTH REFORM PROGRAM 1.1 Health Policy Development This sub component was intended to support development and implementation of policy and plans with a particulars focus on the primary health care. Activities were implemented with the aim to increase quality, efficiency and access in primary healthcare, and to increase the ‘value-added’ of primary care in the health care system; to promote effective diagnosis and treatment of a larger number of patients and broader scope of conditions at the primary care level; and to strengthen the role of primary care in disease prevention. This sub component was intended to help:

- reorganization of the PHC service delivery model - define the basic benefit package for PHC and secondary and tertiary health care level

10 This refers to project restructuring in July 2007

39

- develop norms and standards for service delivery at the PHC and secondary level - develop human resources strategy - develop strategic and operational plan for the implementation of an efficient, effective, equitable

and sustainable health network of public and private facilities, including development of evidence based protocols/guidelines for common conditions and high cost conditions

- develop pharmaceutical policy and strengthen institutional capacities.

Actual outputs At the very beginning of the project implementation the Plan of reform implementation was developed. The primary health care service delivery was reorganized to include two categories: chosen doctors team and primary health care support centre under the organizational scheme of Dom Zdravlja (primary health care centre) as a legal entity. Chosen doctors for women, children and adults were introduced. The new PHC service delivery model has been elaborated in details in the “New PHC Service Organizational Model” that was adopted by the Steering Committee of the project. The registration process of for health insurers was designed and piloted in Podgorica on March 28, 2005. The registration process was the first visible project activity. To initiate the registration process it was necessary to secure many legal prerequisites, technical conditions and implement appropriate PR campaign. The Action plan for registration process with the chosen doctors envisaged all actions to be take, holders of responsibility and implementation dynamics. There were four working groups in charge of the registration process: PHC working group, finance working group, IT working group and PR working group. From the technical point of view necessary telecommunication conditions were provided, 22 working posts were equipped, software application was developed to support the registration process and hardware was procured. Prior to initiating the registration process, training for 70 nurses was organized to get familiar with the work on computers (basic course and course for the work with the application software) Within two years time that is in 2007 the registration model was replicated in Danilovgrad, Rozaje, Tivat where as of 2008 the registration process was conducted at the national level. Upon completion of the project the registration rate at the national level was over 83%. The basic benefit package for PHC was developed and translated into a Government Decree on scope of rights and standards of health care covered by the compulsory health insurance. The Decree was adopted by the Government of Montenegro in 2005. This government regulation defines only the basic benefit package for services provided at the PHC level. As a result of technical service for planning of health services norms and standards for health care service delivery, including norms and standards for human resources at the PHC level have been developed. In 2005 the Rulebook on conditions with respect to standards, norms and modes of delivering PHC services through the chosen doctor was adopted by the Ministry of Health and published in the Official Gazette.

The project supported development of an outline for human resources development in health sector. The key objective was to ensure a balanced supply, make more effective use of the workforce’s skills, and provide the ongoing support and education to promote high quality care and effective HHR planning, thus optimizing the Montenegrin health workforce. As this was an outline of human resources development

40

tested in consultation with the relevant stakeholders, it provided only strategic directions for developing the Strategy on Human Resources development in health sector.

International technical assistance was provided to: (i) support institutional and legal context to ensure compatibility with EU rules including formulation of by-laws and a pharmaceutical policy. The project supported development and adoption of two laws (Law on pharmaceuticals and medical devices), and 33 by-laws. (ii) support newly appointed Drugs agency director in developing Study on organization and structure of the MDA; (iii) set up and implement transparent pricing and procurement system to ensure cost containment and following EU rules; (iv) set up transparent drugs registration system in compliance with EU practice; (v) propose and implement drugs information system in compliance with overall HIS to transfer utilization and finances information from public and private pharmacies to the HIF and drug agency; (vi) set up quality control system. The project supported equipping (computer and telecommunication equipment) and furnishing of the MDA. Under the project, a local coordinator for pharmaceutical policy was hired to manage and support activities and involvement of the Pharmaceuticals working group in the reform process.

1.2 Health Financing This sub component was intended to provide expert advice on key areas of health finance policy , including evaluation and refinement of the primary health care payment model, budget planning, expenditure and revenue forecasting and analysis and resource allocation. Under the project the basic benefit package was supposed to be defined and coasted. It was also intended to provide advice on options for payment for secondary and tertiary health care. The project was designed to support capacity building of the key staff in the Ministry of health, Health insurance fund, Institute of public health and health care institutions about health finance and health finance reform. Actual outputs As a result of provided technical assistance through the project, a new payment model for PHC was developed. It envisaged combination of capitation and fee for service to depart from the traditional payment based on input factors. The new payment model was introduced into the practice at the country level as of 2009. Capitation/fee for service payment was simulated and supported by development of a temporary PHC software which was implemented in the HIF to keep records of services produced, generate automatically reports on services provided, generate automatically invoices and submit them to the HIF where they were reviewed and analyzed. Development, implementation and maintenance of the PHC temporary software has been fully financed by the HIF. COMPONENT 2: IMPLEMENTATION OF PHC REFORM IN PODGORICA 2.1 Renovation of PHC Network This sub component was to support conduct of civil works in the form of construction, reconstruction and adaptation with the view of securing sufficient space required for implementation of the new PHC service delivery organizational model in Podgorica. Also, equipping and furnishing of the PHC premises in Podgorica was envisaged. PHC medical and non-medical equipment was required to bring the PHC group

41

practices in Podgorica up to standard and support PHC diagnosis and procedures as well as to avoid inappropriate referral to hospitals. Actual outputs Following development and adoption of the Model of organization of the PHC system, technical expertise was mobilized to develop the Program on spatial organization of the PHC in Podgorica. The program was developed to elaborate clearly how the Model of organization of the PHC system will be supported from the functional point of view. The project supported implementation of civil works on all sites as per the Program on spatial organization. Due to cost over runs, the government financed completion of civil works on the following sites: Nova, Varos and Stara Varos. The project supported procurement of medical equipment for diagnostics and therapeutic services provided by the chosen doctors and in Support centers, as well as the non medical equipment and furniture. 2.2 Health and financial information system The project was intended to focus on HIS investments that were recognized as necessary to support the development objectives of the project, including introduction of primary health care, and the creation of an efficient, effective and integrated information system for the entire health system. The following activities were supposed to be supported under the project: (a) Identification of Information Flows and Needs - to examine information flows (both within and between institutions), as well as the information needs for management decision-making for the Ministry of Health, and the Health Insurance Fund, and within and between these institutions and other health service providers. (b) Information Systems Plan for the Institute of Public Health - to define the parameters to be monitored by the IPH, examine the best method of reporting and monitoring these indicators, define data sources, method of inclusion and processing; report users, method and schedule of distribution, and develop an implementation strategy to achieve these objectives. c) Information system for the Ministry of health (d) Primary Health Care Information Systems Actual outputs As a result of technical assistance mobilized under the project the following activities were implemented:

- IT support provided to the registration process. This included: development of a detailed Action plan for implementation, development of hardware specification, development of functional requirements for software application, development of technical specification for communication equipment, software development, training to use computers and software application, hardware procurement,

- IT system concept was developed. Assessment of needs for IT system and support on the part of different stakeholders was conducted. Strategy on IT harmonization was developed. Functional requirements for software development were produced. Also, the project supported development of technical specification for hardware and communication equipment.

- Software development – software application for the PHC; application software for the HIF. The software application was introduced into practice.

42

- Hardware procurement – hardware was procured as per the technical specifications for all primary health care facilities at the national level and Health insurance fund.

- Capacity building activities- several study tours were organized to acquire knowledge and exchange experience on the recent developments of the health information systems.

2.3 Training This sub component was to support retraining of PHC doctors and nurses to refresh their theoretical knowledge and skills and shift the health system to center around the family medicine. This intervention was indispensable to ensure that chosen doctors’ teams would be able to deliver under the new roles and responsibilities assigned as per the new Model of primary health care organization. Also, it was planned to support under the AF formal introduction of a department for Family medicine at the Medicine faculty and post graduate formal education in the field of primary health care. Actual outputs: The project supported development of the Model of re-training for doctors and nurses that envisaged implementation of the: (i) main training course covering 250 hours, and (ii) optional training course covering 50 hours. The main course was mandatory for all PHC doctors and nurses, and it was conducted twice a year during the project life. According to the Model of re-training some training programs were attended jointly by doctors and nurse. Optional courses did not apply to nurses. Prior to embarking into the re training of doctors and nurses, a comprehensive training needs assessment was conducted. Training needs assessment provided relevant inputs for development of the Re-training plan, program and methodology. 2.4 Communication (PR) This sub-component was to support promotion of health care reform and introduction of the new PHC organizational model (selection of the chosen doctors) into the health system in Montenegro. Actual outputs The PR campaign was designed and implemented. Also, public opinion surveys were focused on the degree of awareness of the reform, its benefits and novelties, as well as citizens’ perception of the problems in healthcare. Survey reports have demonstrated that health is an important segment of everyday life and preoccupies most citizens. But it also became evident that in time positive trends in terms of perception, support and confidence were recognized. COMPONENT 3: PROJECT MANAGEMENT, MONITORING AND EVALUATION Project management network and Technical services unit (TSU) According to the project, the Ministry of Health would be the lead implementation agency for the project, responsible for project management. It was necessary to establish a Project Management Network (PMN), consisting of a full-time project coordinator, an administrative assistant, including coordinators for the relevant project components.

43

The Government of Montenegro decided to establish a central Technical Services Unit (TSU), responsible for carrying out core procurement and financial management functions for all future Bank-financed projects, and potentially for other donor-supported projects. This TSU was established in the General Secretariat (under the Office of the Deputy Prime Minister). The TSU employed a full time procurement officer and finance officer. Actual output: The Project management network was established and comprised of the project coordinator and administrative assistant. During the project life several component coordinators were hired:

- PHC coordinator - Coordinator for health policy - Finance health coordinator - IT coordinator - PR coordinator - Training coordinator - Pharmaceuticals coordinator

The Government of Montenegro passed a decision to establish a project Steering committee providing strategic guidance for project implementation. The Steering committee consisted of the key stakeholders’ representatives (Ministry of health, Institute of public health, Health insurance fund, PHC, Medical faculty ) to ensure their full involvement in the decision making process on the key project implementation issues. The PMN office provided secretarial support. TSU was also established as it was envisaged by the project and it was fully financed by the project. b) Additional financing The original Project Development Objective remained the same. Under the additional financing the Project Development Objective is supported through the same three original components. COMPONENT 1: SUPPORT FOR HEALTH REFORM PROGRAM 1.1 Health policy development and capacity building Under this sub –component it was planned to provide technical assistance in development of a secondary and tertiary care health sector reform strategy. The elements of this reform would include, but not necessarily be limited to, the plans and normative framework for the health network; defining the package of services financed by the state (and those that are not financed); defining an output based (i.e. diagnostic related groups or DRGs) payment mechanism and contracts between the providers and insurer; developing the priority clinical guidelines to ensure effective and efficient delivery of the state financed services; executing a workforce study in the health care system with recommendations according to international norms and standards; determining models of public and private cooperation in the delivery of health services, including defining appropriate models and proposing legislative solutions; and using communications and information to ensure stakeholder involvement and buy-in to the reform strategy.

Another priority for intervention was further development of the MDA’s capacity, as an entity responsible for the regulation and oversight of the pharmaceutical sector. Establishment of a quality assurance system

44

was also recognized as an area requiring technical support under the project. It was planned to support development of the health service quality assurance strategy.

Actual outputs

As a result of mobilizing the international and local technical expertise the Network of public health institutions and Plan for development of human resources in health for 2012-2022 were

Under the project, the IPH produced the proposal on the Basic benefit package of services to be provided at the secondary and tertiary health care level, including Methodology for costing the basic benefit package. The HIF assisted with the international consortium and with strong involvement of the key policy makers in the health sector proposed a new mechanism of payment for services provided at the secondary and tertiary level. The project also supported the process of defining the National strategy for quality assurance in health sector. The Strategy was adopted by the Government of Montenegro in 2012.

Work on development of the Strategy on integration of private health care institutions into the network of health institutions was also successfully completed.

The project supported numerous activities with the view of strengthening MDA capacity.

1.2 Health Information System

The objective of this sub component was to ensure support for the health policy reform results and complement on-going investments of the Health insurance fund. It was decided to carry out procurement of computer and related equipment to establish the information system between hospitals and the insurance fund.

Actual outputs

The project supported development of functional requirements for software application, software design and software implementation in three institutions: Institute of public health, Ministry of health and Drugs agency. Required computer and related equipment was procured in order to support the software solutions and their installation in the Institute of public health, Ministry of health and Drugs agency. The project did not support development of the hospital information system, as that was supported by the Health insurance fund in its entirety.

COMPONENT 2: Phased Implementation of Primary Health Care Development. 2.1 Podgorica Phase I implementation

It was planned to provide technical assistance to support the Ministry of health in monitoring and assessing the implementation of the primary health care reform. In order to better inform the next stage of PHC reform under the additional financing, a primary health care (PHC) study was conducted to: (i) get a complete overview of the current PHC centres performance by analyzing the revenues, costs, and productivity of (PHC) facilities, (ii) examine the performance of Chosen Doctors Teams in delivering services, and evaluate the impact of new provider payment reforms on the efficiency, performance, and management of (PHC) facilities.

45

2.3 PHC Professional Training This sub component was to: (i) provide further technical assistance in establishing a Family Medicine specialization in the medical faculties, (ii) support the nation-wide upgrade of primary health care services through the provision of re-training of doctors and nurses; and (iii) procure medical equipment for the offices of the “Chosen Doctor” and the diagnostic services in the Health Centres outside of Podgorica. Actual outputs

The project supported realization of two re-training courses for doctors and nurses (round 10 and round 11). Technical assistance was provided to develop formal curriculum for Family medicine, which was introduced into the official undergraduate program at the Medical faculty. Additionally, the project supported capacity building and four persons were fully trained to be able to deliver the Family medicine curriculum. Project activities resulted in development of the family medicine specialization program. 2.4 Scaling up Reform Outside Podgorica Physical upgrade (rehabilitation and extension) of the Primary Health Care Center in Bijelo Polje (with co-financing from the Health Insurance Fund and Bijelo Polje municipality) was considered as a priority intervention area due to poor conditions for providing health services. Actual outputs

The project supported revitalization (reconstruction and extension) of the PHC Centre in Bijelo Polje and procurement of modern medical (diagnostic and therapeutic) equipment. The facility was opened and became fully functional in December 2012. COMPONENT 3: Project Management Due to the extension of the project, the Additional Financing continued to support the functioning of the Project Management Network through the provision of technical assistance, training and the operating costs of the dedicated office as well as proportional cost of the staff and operating costs of the TSU.

Actual outputs

The Project Management Network was functional throughout the project implementation. Due to the work load the PMN office was extended by hiring an administrative assistant, whereas the current administrative assistant was promoted to project coordinator assistant. The project co-shared costs of the services provided by the Technical services unit. CONCLUSIONS:

As a result of project activities implementation the following was achieved:

- Strengthened institutional capacities of the Ministry of health, Institute of public health and Health insurance fund for development, implementation effective and evidence based health policy and health finance policy, including monitoring and evaluation of the policy interventions,

46

- Establishment of Drugs agency and building its capacities to formulate and implement efficient and rational pharmaceutical policy

- Introduced regulatory framework and designed policy platform to support reform intervention measures, ensuring their sustainability and creating conditions for rational use of available resources

- Strategic framework for quality assurance and control, with quality indicators and clinical protocols for the most common and costly diseases was designed and introduced,

- Service delivery at the PHC level was standardized and created conditions for scaling up this process at the secondary and tertiary level

- Position of the primary healthcare was reaffirmed as a strong foundation of the national healthcare system,

- Family medicine services have been introduced to reverse the costly reliance on the expensive specialist care provided at the secondary and tertiary health care level

- Improved health care assess through reorganization of the service delivery model at the PHC level based on health needs of a citizen, with special emphasis on vulnerable categories (children with special needs, persons with mental disorders , etc…)

- New payment model results based at the primary and secondary health care level are to contribute to financial sustainability of the health system and efficient and rational use of resources,

- All processes at the PHC level and between the key reform stakeholders have been supported by the information system

- PHC infrastructure (facilities and equipment) was revitalized to ensure conditions for provision of equitable quality health care up to modern standards

- Reform interventions and measures have been supported by citizens.

47

Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders

No comments were received to the letter sent by the Bank, dated May 28, 2013, requesting comments on the draft ICR from the Canadian International Development Agency (CIDA).

48

Annex 9. List of Supporting Documents

Project Appraisal Document for Montenegro Health System Improvement Project, dated May 12,

2004 (Report No: 27981-YU) Aide Memoires, Back-to-office Reports, Implementation Status Reports and Project

Implementation Plan Country Partnership Strategy for Montenegro, for the period FY11-FY14 (Report No. 57149-ME) Borrower’s Evaluation Report dates December, 2012 Project progress reports, including Government Report during project MTR as well as key

Technical Assistance reports.

KapaKapaMorackaMoracka(2227 m)(2227 m)

KomoviKomovi(2656 m)(2656 m)

BiocBioc(2396 m)(2396 m)

LjubisnjaLjubisnja(2238 m)(2238 m)

SSaannddzzaakk

DDuurrmmii ttoorr

SSiinnjjaajjeevviinnaa

CCiijjeevvnnaa

ZZeettaa

MMoorr

aaccaa

TTaarraa

CCeehhoottiinnaa

SvetiSvetiStefanStefanBudvaBudva

RadovićiRadovići

Stari BarStari Bar

PlavnicaPlavnica

CetinjeCetinje

KotorKotorTivatTivat

PerastPerast

CrkviceCrkvice

GrahovoGrahovo

BajovoBajovoPoljePolje

RudiniceRudinice

GradacGradacPljevljaPljevlja

Bijelo PoljeBijelo Polje

GoranskoGoransko

VelimljeVelimlje

VilusiVilusi

RisanRisanHerceg-Herceg-NoviNovi

TuziTuzi

MedurijecjeMedurijecjevv

ZabljakZabljakvv

TomasevoTomasevovv

RozajeRozajevv

IvangradIvangrad

PlavPlav

GusinjeGusinje

AndrijevicaAndrijevica

MurinoMurino

MojkovacMojkovac

vvKolasinKolasin

vvMatesevoMatesevo

Lijeva RijekaLijeva Rijeka

PelevPelev

GvozdGvozd

MorakovoMorakovo

DanilovgradDanilovgrad

vvBioceBiocevvSpuzSpuz

SavnikSavnik˘

NiksicNiksicvv ‘‘PetroviPetroviccii

DurdevicaDurdevicaTaraTara

´

PODGORICAPODGORICA

PluPlužžineine

BeraneBerane

Shkodër

Ulcinj

Sveti Nikola

Bar

Petrovac

SvetiStefanBudva

Radovići

Stari Bar

Plavnica

Cetinje

KotorTivat

Perast

Crkvice

Grahovo

BajovoPolje

Rudinice Plužine

GradacPljevlja

Bijelo Polje

Goransko

Velimlje

Vilusi

RisanHerceg-Novi

Tuzi

Virpazar

Medurijecjev

Žabjlak

Tomasevov

Rozajev

Ivangrad

Plav

Gusinje

Andrijevica

Berane

Murino

Mojkovac

vKolasin

vMatesevo

Lijeva Rijeka

Pelev

Gvozd

Morakovo

Danilovgrad

vBiocevSpuz

Savnik˘

Niksicv ‘Petrovici

DurdevicaTara

´

PODGORICA

B O S N I A A N DH E R Z E G O V I N A

S E R B I A

KOSOVO

A L B A N I A

Cijevna

Zeta

Mor

aca

Tara

Zeta

Komarinca

Cehotina

Drin

a

Boja

na

- Buna

LakeScutariAdriat ic

Sea

To Foca

To Foca

To Mostar

To Dubrovnik

To Dubrovnik

To Tirane

To Kukes

To Dakovica

To Vucitrn

To Priboj

To Priboj

Sandzak

Durmitor

Sinjajevina

KapaMoracka(2227 m)

Komovi(2656 m)

Bioc(2396 m)

Ljubisnja(2238 m)

19°E

19°E

20°E

42°N

43°N

MONTENEGRO

This map was produced by the Map Design Unit of The World Bank. The boundaries, colors, denominations and any other informationshown on this map do not imply, on the part of The World BankGroup, any judgment on the legal status of any territory, or anyendorsement or acceptance of such boundaries.

0 105 15

0 105 15 20 Miles

20 Kilometers

IBRD 34825R1

DECEMBER 2012

MONTENEGROCITIES AND TOWNS

NATIONAL CAPITAL

RIVERS

MAIN ROADS

RAILROADS

OPSTINA (MUNICIPALITY) BOUNDARIES

INTERNATIONAL BOUNDARIES

Document of The World Bankdocuments.worldbank.org/curated/en/109821468062649994/... ·...

Documents

Transcript of Document of The World Bankdocuments.worldbank.org/curated/en/109821468062649994/... ·...