1

1

IMPACT EVALUATION OF PERFORMANCE BASE

FINANCING

A collaboration between the Rwanda Ministry of Health, CNLS, and SPH, the INSP in Mexico, UC

Berkeley and the World Bank

FOR GENERAL HEALTH AND HIV/AIDS SERVICES

In RWANDA

1

2

2

2

Presentation plan

1. Country profile2. Program description3. Impact evaluation design 4. Impact evaluation

implementation5. Lessons learned

3

04/10/23

RWANDA general & health sector

Total population : 9,038,001 (2005) 30 Administrative districts GDP per capita of $230 33 District Hosp. and 369 Health centers HDI: ranked 158th (2004) MMR: 750 per 100,000 (DHS 2005) IMR: 86 per 1,000 (DHS 2005) HIV: 3.1% (DHS 2005)

4

04/10/23

Relevance and Severity of the Health Issue Addressed Diagnostic

Shortage of human resources for health services Low levels of productivity and motivation High levels of absenteeism Low user satisfaction & poor quality of service lead

to low use. Increase morbidity and mortality

Goal Increase number of trained medical personnel Increase motivation Improve quality of services Increase personnel income

Policy Response Performance Based Contracting & Financing

5

5

RWANDA Performance Based Financing (PBF)

Raises the quantity and quality of health services provided

Increases health worker motivation through a system of incentives payments based on performance

Operates through contracts between the government and other partners (providing the financing) and health facilities (providing services)

6

6

History of PBF in Rwanda

Three pilot schemes: Cyangugu (since 2001) Butare (since 2002) BTC (since 2005) National model implemented in 2006

7

7

National PBF Model: 2005-2008

Creation of CAAC “Cellule d’Appui a l’Approche contractuelle” (MOH PBF department) : Dec 05 – April 06

All stakeholders involved through Technical Working Group: CAAC, Cordaid, HNI, MSH, BTCCTB, WB, SPH

Complex processes as stakeholders tried to advocate for the adoption of their model (indicators, quality measure, payment and supervision)

Finally, a common “national” model was chosen

8

National PBC model for Health Centers

16 Primary Health Care indicators, e.g.: New Curative Consultation = $0.27 Delivery at the HC = $3.63 Completely vaccinated child = $ 1.82

14 HIV/AIDS indicators, e.g.: One Pregnant woman tested (PMTCT) =

$1.10 HIV+ women treated with NVP = $1.10

Separation of Functions between stakeholders

9

04/10/23

Implementing organizations

Ministry of Health (CAAC) Ministry of Finance

Stakeholders Administrative districts Steering committee Purchasers District hospital

Purchase contract Provider: Health facilities

Head of the Health center ---- Management Committee Motivation contract Employee

10

04/10/23

Hypotheses

For both general health services and HIV/AIDS, we will test whether PBC:

Increases the quantity of contracted health services delivered

Improves the quality of contracted health services provided

Does not decrease the quantity or quality of non-contracted services provided,

Decreases average household out-of-pocket expenditures per service delivered

Improves the health status of the population

11

04/10/23

Evaluation Design

Make use of expansion of PBC schemes over time The rollout takes place at the District level Treatment and control facilities were allocated as

follows: Identify districts without PBC in health centers

in 2005 Group the districts based on characteristics:

rainfall population density livelihoods

Flip a coin to assign districts to treatment and control groups.

12

13

04/10/23

Roll-out plan

Phase 0 districts (white) are those districts in which PBF was piloted NOT part of the impact evaluation

Phase 1 districts (yellow) are districts in which PBF is being implemented in 2006, following the ‘roll-out plan’

Phase 2 districts (green) are districts in which PBF is phased in later; these are the so-called ‘Phase 2’ or ‘control districts’ following the roll-out plan. PBF is being introduced in these districts in 2008.

14

04/10/23

Program Implementation Timeline

Jan-06 Mar 06 Jun-06 – Sept 06 2007 Feb-08-Apr 08 Apr-08

Phase I Start of intervention

Program Implementation

Phase II

Start of intervention

Impact Evaluation

SURVEYS Baseline

(General Health) Baseline

(HIV/AIDS)

Follow-up

15

Quality assurance in comparisons

Law of large numbers does not apply here… Proposed solution:

Propensity scores matching of communities in treatment and comparison based on observable characteristics

Over-sample “similar” communities in Phase I & Phase II

It turned out Couldn’t find enough characteristics to predict

assignment to Phase I Took a leap of faith and did simple stratified

sampling

16

More money vs. More incentives

Incentive based payments increase the total amount of money available for health center, which can also affect services

Phase II area receive equivalent amounts of transfers average of what Phase I receives Not linked to production of services Money to be allocated by the health center Preliminary finding: most of it goes to

salaries

17

17

The baseline has 4 surveys

December 2005-March 2006: General Health facility survey (166 centers) General Health household survey (2,016 HH)

August – November 2006: HIV/AIDS facility survey (64 centers) HIV/AIDS household survey (1994 HH)

18

Design and sample (General health)

04/10/23

19

19

Follow-up surveys

February-September 2008 3 surveys:

Combined health facilities survey for General Health - HIV/Aids

Household survey for General Health (panel data)

Household survey for HIV/AIDS (panel data) Data cleaning ongoing. Preliminary

results will be available in early 2009

20

Lessons learned (1)

• This impact evaluation was designed prospectively and was started early enough for baseline data to be collected, before the scale-up of the intervention. Evaluating the impact of such an intervention is almost impossible without planning the evaluation before launching the intervention.

•

21

Lessons learned (2)

• Many impact evaluations focus on small-scale, localized pilot-projects who benefit from a lot of attention and supervision. The impact evaluation of PBC in Rwanda covers almost the entire country and will assess the impact of the intervention under "real" conditions.

• While it might be good to start with modest pilot-projects, it is also crucial to be able to assess whether an intervention is effective when scaled-up at country level, under realistic and sustainable conditions.

22

Lessons learned (3)

• Such a country-wide impact evaluation requires that the impact evaluation is strongly owned by the Government. In the case of Rwanda, the Government has been in the driving seat, by asking for the impact evaluation, making sure that the general health and the HIV/AIDS sector evaluation were well synchronized and requesting and enforcing a strong coordination among the donors.

• The result was a impact evaluation design and a geographical and chronological roll-out plan of PBC in Rwanda that was carefully adhered to by all stakeholders.

23

Lessons learned (4)

• An interesting feature of the evaluation in Rwanda is that it will try to measure impact not only on processes such as number of visits to the health centers, number of vaccinations, but also by directly measuring health outcomes such as malaria and anemia tests, anthropometrics among the target population.

• The ultimate objective of performance-based contracting is to improve the population's health status and the impact evaluation will explicitly measure it.