IMPACT EVALUATION OF PERFORMANCE BASED

CONTRACTING FOR

A collaboration between the Ministry of Health, CNLS, SPH, INSP-Mexico and World Bank

GENERAL HEALTH AND HIV/AIDS SERVICES IN

RWANDA

1

04/24/23

2

Presentation plan

1. Impact evaluation design 2. Results from the general

health baseline 3. Follow up studies4. Discussion

2

SECTION 1 :Impact Evaluation

Design

3

04/24/23

4

Study Rationale No examples of rigorously evaluated

bonus payment schemes to public sector health care providers in developing countries

No distinction between the incentive effect and the effect of an increase in resources for the health facilities

Link between worker motivation programs and quality of care

4

04/24/23

5

Hypotheses

For both general health services and HIV/AIDS services, we will test whether PBC:

Increases the quantity of contracted health services delivered

Improves the quality of contracted health services provided

Does not decrease the quantity or quality of non-contracted services provided,

Decreases average household out-of-pocket expenditures per service delivered

Improves the health status of the population

5

04/24/23

6

Evaluation Design Make use of expansion of PBC schemes over time The rollout takes place at the District level Treatment and control facilities were allocated as

follows: Identify districts without PBC in health centers in

2005 Group the districts in “similar sets” based on

characteristics: rainfall population density livelihoods

Flip a coin to assign districts within each “similar” to treatment and control groups.

6

7

04/24/23

8

Roll-out plan Phase 0 districts (white) are those districts in

which PBF was piloted Cyangugu = Nyamasheke + Rusizi districts Butare = Huye + Gisagara districts BTC = Rulindo + Muhanga + Ruhango + Bugesera + Kigali ville

Phase 1 districts (yellow) are districts in which PBF is being implemented in 2006, following the ‘roll-out plan’

Phase 2 districts (green) are districts in which PBF is not yet phased in; these are the so-called ‘Phase 2’ or ‘control districts’ following the roll-out plan. According to plan, PBF will be introduced in these districts by 2008.

8

Sets Phase I Phase II1 Kibungo all Kirehe all2 Nyanza remaining Kamonyi all3 Gakenke remaining

Rulindo remainingByumba remaining

4 Rwamagana remaining Kayonza west, allGatsibo west, all

5 Gatsibo east, all Nyagatare east,allKayonza east, all

6 Rutsiro all, south west Kibuye west, allNyamasheke remaining

7 Ngororero all Kibuye east, allGasiza

8 Rutsiro all, except south west Kabaya all9 Nyaruguru remaining Gikongoro all10 Burera all Ruhengeri all

Nyagatare west, all

Rollout plan for PBC in general health services

9

Immediately Phase I Phase IIThese places already have PBC in health centers for non-HIV/ AIDS services

HIV/ AIDS PBC to be introduced AFTER or SIMULATEOUSLY with PBC for general services (ie. NOT before)

HIV/ AIDS PBC to be introduced AFTER or SIMULATEOUSLY with PBC for general services (ie. NOT before)

Rulindo Kibungo KireheKamonyi Nyanza KamonyiGisozi Gakenke Gisagara Rulindo Butare ByumbaBugesera Rwamagana KayonzaGasabo Gatsibo NyagatareNyarugenge Nyamasheke KibuyeCyangugu Ngororero Gasiza

Rutsiro KabayaNyaruguru GikongoroBurera Ruhengeri

Rollout plan for PBC in HIV/AIDS services

10

04/24/23

11

Program Implementation Timeline

Jan-06 Mar 06 Jun-06 – Sept 06 2007 Feb-08-Apr 08 Apr-08

Phase I Start of intervention Program

Implementation Phase II

Start of intervention

Impact Evaluation SURVEYS Baseline

(General Health) Baseline

(HIV/AIDS) Follow-up

11

Quality assurance in comparisons Law of large numbers does not apply here… Proposed solution:

Propensity scores matching of communities in treatment and comparison based on observable characteristics

Over-sample “similar” communities in Phase I & Phase II

It turned out Couldn’t find enough characteristics to predict

assignment to Phase I Took a leap of faith and did simple stratified sampling

12

More money vs. More incentives Incentive based payments increase the

total amount of money available for health center, which can also affect services

Phase II area receive equivalent amounts of transfers average of what Phase I receives Not linked to production of services Money to be allocated by the health center Preliminary finding: most of it goes to salaries

13

The baseline has 4 surveys General Health facility survey (166

centers) General Health household survey (2,016

HH)

HIV/AIDS facility survey (64 centers) HIV/AIDS household survey (1994 HH)

14

04/24/23

15

Identified sectors and cells served by each of the 164 health facilities in the sample,

Randomly selected four cells from the catchment area,

For each cell, obtained number of zones (10-15 hh)

Randomly selected three zones in each cell Obtained household lists for each of the zones Randomly selected one household for each zone Produced random sample of 12 households per

health facility, with a final sample size of 2,016 households.

General Health Household Level: Sampling Method

15

04/24/23

16

Analysis PlanAll analyses will be clustered at the district level Compare the average outcomes of facilities and

individuals in the treatment group to those in the control group 24 months after the intervention began.

Use of multivariate regression (or non-parametric matching) : control confounding factors

Test for differential individual impacts by: Gender, poverty level Parental background (If infant : maternal

education, HH wealth)

16

04/24/23

17

Difference in differences models To test the robustness of the analysis Control sample (both observed and unobserved)

heterogeneity A two-way fixed effect linear regression:

Where:

is an outcome variable for facilitiy i in period t or for individual i who lives in the catchment area of facility i,

is an indicator of whether facility i

it it k itk i t itk

it

it

y PBC B X

y

PBC

is being paid by PBC in period t, are time varying control variables,

i is a facility fixed effect, is a year fixed effect, is an error term.

itk

t

it

X

17

SECTION 2 :Descriptive Results from General Health Baseline

18

Baseline Survey: Sampling Strategy19

GENERAL HEALTH FACILITIES

BASELINE SURVEY

20

21

General Health Centers Survey:Content

All 166 centers in General Health General characteristics Human resources module: Skills,

experience and motivations of the staff Services and pricing Equipment and resources Vignettes: Pre-natal care, child care, adult

care VCT, PMTCT, AIDS detection services Exit interviews: Pre-natal care, child care,

adult care, VCT, PMTCT

Classification of facility 22

LABORATORY TESTS: Availability

Number of Obs. cstd

Number of Obs. cstd

Anemia 80 37.50% 0.487 84 29.76% 0.460 -1.046Pregnancy 80 56.25% 0.499 84 34.52% 0.478 -2.846Blood Count 80 1.25% 0.112 84 3.57% 0.187 0.960Urine dip 80 50.00% 0.503 84 60.71% 0.491 1.380Stool Sample 80 8.75% 0.284 84 3.57% 0.187 -1.385Parasites 77 89.61% 0.307 84 94.05% 0.238 1.029Malaria 80 50.00% 0.503 84 53.57% 0.502 0.455Malaria dip 79 55.70% 0.500 84 72.62% 0.449 2.277TB 80 36.25% 0.484 84 40.48% 0.494 0.553HIV 80 62.50% 0.487 84 55.95% 0.499 -0.849Gonorrhea 80 18.75% 0.393 84 21.43% 0.413 0.425Syphilis 80 10.00% 0.302 84 8.33% 0.278 -0.368Pap smear 80 38.75% 0.490 84 36.90% 0.485 -0.242

Lab Tests

Phase I (Intervention) Phase II (Control)

T-Stat

23

Equipment: Availability24

Number of Obs. cstd

Number of Obs. cstd

scales_adults 80 62.50% 0.487 84 67.86% 0.470 0.717scales_child 80 70.00% 0.461 84 79.76% 0.404 1.443otoscope 80 72.50% 0.449 84 72.62% 0.449 0.017patient_beds 79 89.87% 0.304 83 89.16% 0.313 -0.148other_beds 79 64.56% 0.481 83 66.27% 0.476 0.227xray 80 1.25% 0.112 83 1.20% 0.110 -0.026forceps 80 2.50% 0.157 84 5.95% 0.238 1.091ventouse 80 21.25% 0.412 84 29.76% 0.460 1.247aspirators 79 48.10% 0.503 83 40.96% 0.495 -0.911speculum 80 83.75% 0.371 84 78.57% 0.413 -0.843oxygen 80 5.00% 0.219 83 6.02% 0.239 0.284echographs 79 1.27% 0.113 84 1.19% 0.109 -0.043ecg 79 1.27% 0.113 84 1.19% 0.109 -0.043incubators 79 1.27% 0.113 84 3.57% 0.187 0.948microscopes 80 95.00% 0.219 84 96.43% 0.187 0.450refridge 80 78.75% 0.412 83 75.90% 0.430 -0.431freezers 79 31.65% 0.468 84 35.71% 0.482 0.546ambulance 80 16.25% 0.371 84 11.90% 0.326 -0.798

Equipment

Phase I (Intervention) Phase II (Control)

T-Stat

Drugs: Availability

25 Number of Obs. cstd

Number of Obs. cstd

cotrimoxazole 80 100.00% 0.000 84 96.43% 0.187 -1.711ampicilline 80 63.75% 0.484 84 72.62% 0.449 1.218gentamycine 80 52.50% 0.503 84 45.24% 0.501 -0.927ceftrioxone 80 6.25% 0.244 84 4.76% 0.214 -0.416ciprofloxine 80 85.00% 0.359 84 67.86% 0.470 -2.615norfloxine 80 66.25% 0.476 84 63.10% 0.485 -0.420antibiotiques 80 95.00% 0.219 84 97.62% 0.153 0.890metronidazole 80 98.75% 0.112 84 97.62% 0.153 -0.537nevirapine 76 52.63% 0.503 83 39.76% 0.492 -1.630efavirenz 78 19.23% 0.397 82 7.32% 0.262 -2.252zidovudine 78 15.38% 0.363 82 4.88% 0.217 -2.235laminvudine 78 17.95% 0.386 82 7.32% 0.262 -2.046didanosine 77 3.90% 0.195 82 1.22% 0.110 -1.074stavudine 78 17.95% 0.386 82 7.32% 0.262 -2.046abacavir 78 6.41% 0.247 82 1.22% 0.110 -1.733

Drugs

Phase I (Intervention) Phase II (Control)

T-Stat

Prenatal Care: Provider Effort and Competence

92%

86%

80%

21%

18%

13%

64%

85%

85%

21%

17%

8%

Number of previous miscarriages

Number of previous pregnancies

Late menstrual dates

Lost/gain of weight, nausea, vomiting

Medicaments that is taking right now

Family history of genetic problems

Competence Effort

26

Prenatal care:Provider Effort and Competence

52%

58%

72%

81%

81%

86%

83%

19%

17%

97%

61%

93%

Weight measured

Pelvic examination

Height measured

Abdomen examination

Blood presure measured

Listen featal heatbeat

Competence Effort

27

Patient Satisfaction: Overall

4.08%

4.83%

14.66%

76.44%

0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00%

Very Unsatisfied

Not Satisfied

Very Satisfied

Satisfied

28

General Health Household

BASELINE SURVEY

29

30

Socio-economic information Anemia finger prick test: children 12-71 months

old Malaria dip stick test: children under age 6 Anthropometrics: <6 years old Mental health: mothers, pregnant women, adults

over age 20 Sexual history and preventative behavior

knowledge Pre-natal care utilization and results Parents or caretakers were asked for information

regarding child (<5 years) health status

General Health Household Survey: Content

30

Baseline Household Sample

2159 HH, 10,880 individuals Gender balance (49.3% male and 50.7%

female) Average HH size is 5.71 individuals, Age range of <12 months to 96 years

old. Half of the sample is below 12 years old 75% of the sample is under the age of 30

years old. (Sampling strategy)31

Household Data: AssetsVariable Nr. Obs Mean Std. Error Nr. Obs Mean Std. Error T-statComplete sofa set 921 2.18% 0.005 1147 3.08% 0.007 -1.073Radio 921 49.63% 0.023 1147 50.27% 0.017 -0.221Radio-cassette or music system 921 4.29% 0.009 1147 5.89% 0.008 -1.274Telephone Mobile 921 1.07% 0.005 1147 3.50% 0.007 -2.778 ***Mosquito nets 921 22.53% 0.023 1147 28.00% 0.027 -1.542Sewing machine 921 0.76% 0.003 1147 1.18% 0.004 -0.883A bed 921 62.82% 0.023 1147 58.71% 0.022 1.290Wardrobe 921 4.53% 0.009 1147 6.07% 0.010 -1.155Metalic library 921 0.57% 0.002 1147 1.28% 0.004 -1.645Table 921 63.47% 0.022 1147 63.40% 0.021 0.021Chair 921 85.48% 0.018 1147 84.82% 0.019 0.255A bicycle 921 16.37% 0.020 1147 17.26% 0.027 -0.262

PHASE I PHASE II

32

Household Data: Prenatal Care33

Variable Nr.Obs. Mean Std.Error Nr.Obs Mean Std.Error T-stat

Of all births since Jan. 2005

In facility birth 1238 32.05% 0.026 1462 34.87% 0.025 -0.777

Of most recent pregnancyTimes received PNC 723 2.781 0.046 900 2.687 0.052 1.368

Injection to prevent tetanus 728 74.52% 0.023 900 72.56% 0.019 0.654

Phase I Phase II

Household Data: Prenatal CareVariable Nr.Obs. Mean Std.Error Nr.Obs Mean Std.Error T-statSays she knows dangers 723 80.56% 0.018 897 84.69% 0.017 -1.655 *

Phase I Phase II

Danger Signs - Spontaneous Response

26.87%22.55%

7.57%

82.76%

31.49%

12.51%22.43%

77.88%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

Bleeding Fever Swelling Fatigue

Phase I

Phase II

34

Household Data: Child Immunization35

Variable Nr.Obs. Mean Std.Error Nr.Obs Mean Std.Error T-stat

12-23 months oldfully_immunized 262 75.36% 0.03 295 77.04% 0.04 -0.3312-71 months oldfully_immunized 1154 63.77% 0.03 1387 65.28% 0.02 -0.43

Phase I Phase II

Household Data: Child Recent Illness and Symptoms (<6 years)

30.79%

20.65%

16.37% 15.85%

11.71%

7.16%

2.77% 2.20%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

35.00%

Fever Respiratory Diarrhea Headache Malaria Eye Infection Ear Infection Accident

36

Household Data: Child Health Care Utilization (<6 years)

Variable Nr.Obs. Mean Std.Error Nr.Obs Mean Std.Error T-stat

days_sick 1438 1.96 0.12 1717 2.59 0.13 -3.54 ***receive_care 597 25.21% 0.02 803 25.58% 0.02 -0.13times_receive_care 143 1.58 0.10 218 1.44 0.08 1.12cost_fees 135 246.20 43.80 213 285.71 42.82 -0.65cost_supplies 132 93.05 52.04 208 101.43 28.25 -0.14cost_medicine 133 287.77 64.66 207 475.54 76.57 -1.87 *cost_medicine_nopres162 73.48 19.46 283 109.43 30.67 -0.99cost_lab 134 77.13 21.38 208 78.92 21.43 -0.06cost_other 134 21.35 8.28 204 45.29 19.51 -1.13

Phase I Phase II

37

Household Data:Child Biomarkers (<6 years)

18.67%

7.51%

22.16%

27.82%

19.53%

7.33%

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

30.00%

Anemic Malaria Fever and Malaria

Phase I

Phase II

38

Conclusion of the baseline studies

39 HH Results comparables to the recent DHS Validity of samples

Of 110 key characteristics and output variables of HF, the sample

is balanced on 104 of the indicators.

Of 80 key HH output variables, the sample is balanced on 73 of

the variables.

Majority of the indicators which differ between Phase I and Phase

II are results from patient exit interview, which is not a random sample.

Section 3:Follow-up surveys

and next steps

40

Follow-up surveys Early 2008 3 surveys:

Combined health facilities survey for General Health - HIV/Aids

Household survey for General Health – revisiting same households

Household survey for HIV/AIDS Timing was key… Ensure exposure to PBC for enough time

Avoid contamination (Training of phase 2 started after data collection for HF in May 2008)

41

QUESTIONS?

SUGGESTIONS?

42