Setting a Path for Improved Health Outcomes RBF

Setting a Path for Improved Health Outcomes Results-Based Financing: the Evidence thus Far

Early evidence on Results-Based Financing: Demand and Community Based incentives

Evidence from a preliminary analysis of financial incentives for health

3

Financial incentives have worked, but…– Demand- and supply-side incentives work on different

margins. Demand-side incentives encourage people to go to a facility, while supply-side incentives encourage health providers to deliver more and better care to people who have made it to the facility

– Demand- and supply-side incentives are complements, and are best combined;

– Community-based incentives, for example incentives to community health workers, could serve as “bridge” between supply and demand.

– But few evaluations so far have looked at the combination of supply and demand side incentives and at the role of community-based incentives.

– We need to learn more.

Conditional cash transfers and children health outcomes Some health outcomes and behaviors might be easier to

influence from the demand side (patients, population) rather than from the supply side (health care providers).

See example from Rwanda (Conditional) cash transfers have been widely used and

evaluated as a social protection mechanism. When they are conditional, the conditions are linked to

educational and/or health behaviors. They usually have impacts on reducing poverty, but also on

improving education and health outcomes.

4

5

Gender and Conditionality: A Randomized Evaluation of Alternative Cash Transfer Delivery

Mechanisms in Rural Burkina Faso

Cash Transfer Pilot Program Randomization Plan

6

75 villages (2775 households)

_________ |

_

____________ |

_

| ______|_____ |

_

____________ |

_

_______ |

15 villages (540 households) Randomized CCT to Father

15 villages (540 households) Randomized CCT to Mother

15 villages (540 households) Randomized UCT to Father

15 villages (540 households) Randomized UCT to Mother

15 villages (615 households) Randomized to Control Group

Cash Transfers Overview

Transfer amount:– Ages 0-6: 4000 FCFA/year– Ages 7-10 (Grades 1-4): 8000 FCFA/year– Ages 11-15 (Grades 5+): 16000 FCFA/year

$1 USD = 500 FCFA CCT:

– Ages 0-6: Quarterly visits to health clinic for preventive care (growth monitoring)

– Ages 7-15: School attendance rate>90% UCT:

– No requirements

7

Research Summary

Consider broad measure of welfare outcomes: education, health, livestock, agriculture, demographics, assets/infrastructure

For child education and health outcomes, conditional cash transfers outperform unconditional transfers

Giving cash to mothers does not lead to significantly better child education or health outcomes

Evidence that giving cash to fathers improves child health in bad rainfall years

Cash transfers to fathers yields more investment in livestock, cash crops, and improved housing

8

CCT and adolescent health outcomes including HIV prevention Traditionally CCTs target education outcomes as

well as mother/child health outcomes. More recently they have also been tested as a way

to influence adolescent/young adults health outcomes and behaviors, in particular for HIV prevention.

Those are behaviors and outcomes which are likely to be difficult to influence through a classic supply-side RBF program.

9

10

STIs?HIV?

$ →↓HIV?

Baird, Garfein, McIntosh and Özler, 2012

11

STDs?HIV?

+STIs?HIV?

$ →↓HIV?

12

Study population (N=1,328)

Control (N=827) Treatment (N=501)

Unconditional Cash Transfer

(N=265)

Conditional Cash Transfer

(N=236)

13

Study population (N=1,328)

Control (N=827) Treatment (N=501)

Unconditional Cash Transfer

(N=265)

Conditional Cash Transfer

(N=236)

Relative risk (compared to control, adjusted)

Pregnant now : 0.16 (p<0.05)Partner≥25 : 0.36

HIV : 0.47HSV-2: 0.08 (p<0.05)

Relative risk (compared to control, adjusted)

Pregnant now : 1.17Partner≥25 : 0.08 (p<0.05)

HIV : 0.29 (p<0.05)HSV-2: 0.37

NB UCT significantly different than CCT only for “pregnant now” outcome

http://www.thelancet.com/home

Impact Evaluation of the Rwanda Community Performance-Based

Financing Program

College of Medicine and Health Sciences School of Public Health

Background: Community PBF (Second Generation)

15

Since 2009, Community Health Workers (CHWs) were paid for reporting on health indicators in their communities

Additional components were added through the Community Performance-Based Financing Program in order to promote targeted services

This study evaluates the impact of 2 interventions that were added to the scheme:

1. Performance incentives for CHW cooperatives2. Demand-side in-kind incentives

Background: organization of CHWs in Rwanda

16

Each village has 3 volunteers serving as Community Health Workers (CHWs).

Multidisciplinary CHWs

CHW in Charge of Maternal and

Neonatal Health

Criteria• Can read and

write• Age 20-50• Lives in the

village• Elected by

the village residents


17

All the CHWs within the catchment area of a health center are organized in a CHW cooperative.

Cooperative


18

70% of payments received by a cooperative must be invested in income generating activities (IGAs).

30% of the payments and revenues from the IGAs are given to cooperative members. It is up to the cooperatives to determine distribution rules.

Intervention #1: Performance Incentives for CHW CooperativesCHW cooperatives received financial rewards for:1. Nutrition monitoring: # children 6-59 months monitored2. Timely Antenatal Care: # of women accompanied/referred

within first 4 months of pregnancy3. In-Facility Delivery: # of women accompanied/referred for

assisted delivery 4. New Family Planning users: # referred to health center5. Regular Family Planning Users: # regular users at health

center 4 indicators related to TB and HIV were added at a later

stage and not evaluated

19

Intervention #2: Demand-Side In-Kind Transfers

20

Women received gifts for seeking care for the following services:

* Women can only receive the gifts for one pregnancy every 3 years.

Eligibility* Value (Ceiling) Suggested Package

Initiation of Antenatal Care during first 4 months of pregnancy

5 USD Adult cloth and water treatment tablets OR baby cloth package and water treatment tablets

Delivery in health center 6.67 USD Baby soap, baby shawl and baby bed sheets

Initiation of Postnatal Care during the 10 days after delivery

3.33 USD An umbrella and water treatment tablets OR Adult cloths

Research Questions

1. Do the demand-side in-kind transfers and the performance incentives to CHW coops increase

– Initiation of prenatal care within first 4 months of pregnancies?

– Total prenatal care visit?– In-facility deliveries?– Rate of postnatal care within 10 days after delivery?

2. Is there a multiplicative effect when both interventions are implemented?

3. Do the performance incentives to CHW coops affect– Behavior and motivation of the CHWs?– Use of modern contraceptives?– Growth monitoring of children under 5

21

Study Design: RCT

22

198 sectors (sub-districts) were randomly allocated into 4 study arms:

* Coops paid for reporting received the average amount received by the coops paid for performance

Payments to CHW Coops

For Reporting* For Performance

Demand-Side Transfers

No C S

Yes D D+S

Timeline

23

2010

2011

2012

2013

2014

February-May 2010

• Baseline Survey

November 2013-June 2014

• Follow-up Survey

October 2010

•Interventions Introduced

February 2013

•Last transfer of funds for in-kind transfers

Results: Maternal Health Services

Indicators:– Timely ANC– In-facility deliveries– Timely PNC

Sample of women with most recent birth in their village– Pregnancies resulting in a live birth

24

Results: ANC visit within first 4 months of pregnancy

25

Control Demand Supply D + S50%

55%

60%

65%

70%

75%

80%

85%

Timely ANC

• A positive and significant (at the 1% level) impact of the demand-side in-kind incentives of about 10 percentage points

• The CHW incentives are not found to have a significant effect• No difference between the ‘Demand’ and the ‘Demand+Supply’ treatment arms

Results: at least 4 ANC visits

26


30%

35%

40%

45%

50%

Four or more ANC visits

• Not targeted by the program!• Higher in the intervention sectors, but not statistically significant at the 10%

level

Results: Skilled-attended in-facility delivery

27


75%

80%

85%

90%

95%

100%

In-Facility Delivery

• No statistically significant difference between the treatment arms• Rate has increased substantially in the duration of the study for other

reasons

Results: PNC within 10 days after delivery

28


5%

10%

15%

20%

25%

PNC within 10 days after delivery

• A positive and significant (at the 5% level) impact of the demand-side in-kind incentives of about 7 percentage points

• Not targeted by the CHW incentives intervention

Key Findings: Demand-Side In-Kind Incentives

• The demand-side in-kind incentives caused an increase in timely ANC and PNC services

• Although some challenges in procurement and frequent stock outs

• Although some health centers independently implemented their own demand-side incentives strategies to promote utilization

• Although funding ended before end-line data collection

• Consistent with findings in other countries that implemented demand-side cash transfers

29

Key Findings: Performance Incentives to CHW Coops• No impact of incentives to CHW cooperative on

targeted indicators, CHW behaviors and CHW motivation.

• Potential reasons for lack of impact– Incentives were too low– Collective reward but individual effort– Pay-for-reporting could have already oriented the

CHWs towards targeted indicators– Limited scope given the many supply-side

programs targeting the same indicators

30

Research Team

Ministry of Health – Fidel Ngabo – Cathy Mugeni

University of Rwanda– Ina R. Kalisa– James Humuza– Jeanine Condo– Vedaste Ndahindwa

The World Bank– Gil Shapira– Netsanet W. Workie– Jeanette Walldorf

31

The study was funded by the Health Results Innovation Trust Fund (HRITF)

The Case of Community RBF

What is cRBF?

Community RBF: a set of different practices:– Based on the idea of contracting (cRBF)– Separation of functions (purchaser, provider,

regulator and verifier)

RBF is: “a cash payment or non-monetary transfer made to a national or subnational government, manager, provider, payer or consumer of health services after predefined results have been attained and verified. Payment is conditional on measurable actions being undertaken” (Musgrove, 2010)

33

34

Rationale for cRBF?

What is the objective?Provide services at the most peripheral and decentralized level

Contracting of CHWOften attached to a health facility

Stimulate the demand sideAwareness meetingsContacts with the populationVouchers and incentives

Achieve health related behavioral changesPart of all cRBF, sometimes stated more clearly (The Gambia and Congo)

Health promotion / awareness [HP] Use of services [US] Health outcomes [HO]

Who is contracted in cRBF? Who are the community actors contracted in cRBF?

– Community Health Workers: in charge of providing specific services, often preventative care and awareness campaigns [in the spirit of the 1977 Alma-Ata conference]

– Health Facility Committee members: co-managers of the health facilities, intermediaries between population and service providers [in the spirit of the 1988 Bamako conference]

– Traditional healers –a large variety: traditional midwifes, herbalists, etc.

– Other community actors:▫ Village committee▫ Community-based organizations

What is not included under cRBF?– Individuals directly: then closer to Conditional Cash Transfers

(CCTs)

35

Lessons Learned from cRBF Operations

Country cRBF experiences Contracting of Community (Health) Workers:

– Benin– Cameroon– Republic of Congo– Rwanda

Contracting of Health Facility Cie.– DR Congo

Contracting community organizationsThe Gambia

Demand-side and voucher schemes (not discussed here)

– The Gambia– Rwanda– Congo

37

Lesson 1: cRBF programs should be designed taking into

consideration contextual factors

38

cRBF programs should be designed taking into consideration contextual factors (Cameroon)Example of Cameroon:

PBF Indicators started improving in HF but stagnated despite much efforts by health facilities

Reports of many drop outs concerning vaccinations, post natal consultations antenatal consultations and use of family planning among women.

Nutritional concerns of children were poorly addressed by program

Therefore something had to be done to re-stimulate demand for health services by the community

Reflection of the Government and partners led to identifying a Community PBF approach as a strategy worth trying

Experience of some health facilities sub-contracting with Health Committee Members had proven it’s worth in referrals and search for drop outs

Need therefore to contract Community Health Workers in a formal manner a cPBF pilot was then started in July 2015 with a Community Monitoring

component to strengthen the voice of the community in health care delivery

39

Lesson 1: Experience from RoC

Each Context is unique– Avoid Copy and paste

Context is essential to define the CPBF Model of RoC– Low coverage for some indicators– Absence of community networks

Objective: support households in the health seeking behaviors.

Interventions:– Put in place the community relays– Action plan signed with the household

40

Lesson 2: Existing community structures should be

assessed and, where possible, strengthened using cRBF

41

Contracting Community Committees: The Gambia

Most communities in The Gambia have:– Village Development Committees (VDC) responsible for all

development activities of the community; and ▫ Village Support Groups (VSG) comprising 4 women and 2 men who,

with the VHW and TBA, are trained to promote optimal maternal, infant and young child feeding practices. They are an arm of the VDC.

During the design stage of the Maternal and Child Nutrition and Health Results Project, anchored on PHC, it was unanimously agreed that the VDC be contracted to implement the Demand side of the Project

This was strengthened by the type of indicators which could not be contracted to individuals: the demand side (cRBF) indicators focused on knowledge and practice

The verification of these indicators is done using a survey (LQAS) – therefore the entire community is contracted through the VDC

20% of the quarterly subsidy payment is given to the VSG as an incentive while the balance goes into the implementation of a community development project identified through a PRA

42

Experience of Benin (Similar to Cameroon)

Preexistence of community health workers: sensitize the population on health, refer patients to the health center

But fragmentation of package of services depending on sources of funds

cRBF relies on existing CHWs and train them on the complete package

Then, sub-contract between individuals and HF The Health center: Coordination center to share

good practices, to declare results, group monitoring, supervision and payment

43

Lesson 3:Broad participation in the selection of indicators

is key…

44

Experience of Benin

Involve all actors in the process to prioritize indicators: – Central level MoH, Vertical programs, Donors,

district level, local levels,…… (with focal persons at all steps)

Build ownership :– Good understanding by all stakeholders– Appropriate indicators for the implementation of

the PIHI program– Coordinate and prioritize (All indicators cannot be

part of the package)

45

Lesson 4:But it’s important to limit the number of indicators

to ensure feasibility and quality

46

Variety and scope of indicators varies

Indicators can be at all levels The number of indicators matters:

– The Gambia (9): Health promotion– Benin (9): Referral system– DRC: Hybrid (functionality indicators, health

promotion) – Cameroun (20): Referral system, service

utilization– RoC: Health promotion

47

RoC: Advantages with few indicators

Better verification:– Good quality data: reliable

Better analysis of data collected:– Areas of weaknesses and strengths

Low cost of transactions for verification, high cost for individual indicators (Motivating for CHWs)

48

Lesson 5:Data collection tools should be simple

49

Make management tools simple

50

Current Challenges1. Tools for community health workers and other community

members are too complex2. Tools are not effectively used because they are time

consuming3. Tools and processes are designed for the purchaser or the

regulator rather than the users and community

Recommendations1. At the community level, tools should be simple and easy to use2. Tools should be validated by the relevant community actors3. Strengthen the community capacity for monitoring

Lesson 6: Systems are needed to monitor and maintain the

quality of training at all levels of the health pyramid

51

Systems are needed to monitor and maintain the quality of training at all levels of the health pyramid

52

Current Challenges:1. To decentralize, there is a need of training in cascade mode.2. But, the cascade mode doesn’t ensure the quality of training

at peripheral level A (100%)-- B (85%)--C ( 70%)-- D ( 45%)3. The content of the training is losing some key information4. During the implementation, new issues arise5. Differentiated adaptation

Recommendations:1. Ensure quality of training at lower levels2. More supervision and monitoring of the trained community actors3. Benchmark the good practices of those who succeed to support the

weak CHWs

Lesson 7 Payments to CHWs should be timely

53

Payments to CHWs should be timely (Cameroon)- During pilot period for cPBF payments from central level to health facilities

were often delayed. At first facilities waited for PBF subsidies to arrive before paying CHWs, this led to long delays in paying CHWs, leading to demotivation and frustration of CHWs

- To improve on the retention of the CHWs, the payment model was revised.

- Now the quarterly facility contracts stipulate that the health facility should pay the CHWs monthly as soon as their verification is done; using facility resources (mix of cost recovery, PBF subsidies, etc.).

- Difficult to convince all facilities to accept this approach, but by including it in the facility contract they, CVA was able to negotiate this payment mechanism.

- After several months facilities have noted that it is possible to ensure timely payment of CHWs

- CHW motivation and retention has improved. Model scaled up to other regions.

54

Lesson 8:ICT can be very useful but it should be built on

solid systems and carefully tested

55

Use of Mobile Devices for Data Reporting and Verification: The Gambia Experience The Gambia started with strengthening the already existing

HMIS and incorporating RBF indicators– Data collection and reporting tools were reviewed and

updated – The DHIS2 database updated to reflect the new information– PHC Circuits were re-demarcated to fit within health facility

catchment areas

The country team is now considering the gradual introduction of the use of mobile devices starting with verification using tablets

Also considering the use of mobile money for the payment of CCTs to pregnant women

56

Lesson 9:There’s still a lot to learn!

57

Learning Opportunities

How best can community level data be used to inform activities?

How to ensure that CHWs only provide the services they are meant to provide?

How to appropriately share data with communities and promote community ownership of activities?

What is the impact of sub-projects funded through community incentives?

Why was there high CHW drop-out after initial training?

How best to do verification of community data?

58

What are we learning?

Projects in the World Bank’s current portfolio of cRBF are in the process of answering some outstanding questions. RoC and DRC are evaluating a strategy of paying

health centers to conduct home visits jointly with community agents

Cameroon is assessing the impact on uptake of services of health centers subcontracting community health workers

In the Gambia, the impact on health behaviours and uptake of health services of performance payments to community organizations is being assessed

59

THANK YOU

Results-Based Financing & Quality of Care:Measuring and Paying for Quality Improvement

Session Outline: Measuring and Paying for Quality

I. Existing Instruments and MethodsII. Using Data for decision makingIII. Verifying Data AccuracyIV. Innovations in Measuring and Paying for

Quality

62

1. Existing Instruments and Methods

63

Measuring if the right inputs are in place

64

Liberia: Quality Assessment/ Monitoring Tools

1Complicated and assisted delivery (including C-section)

Any labor that is made more difficult or complex by a deviation from the normal procedure. Complicated delivery is defined as: assisted vaginal deliveries (vacuum extraction or forceps), C-section, episiotomy and other procedures.

17

2 Normal del iveries of at risk referralsHigh-risk pregnant women referred by health center to the hospital but delivered normally. A high-risk pregnancy is defined as: evidence of edema, mal presentation, increased BP, multi-parity, etc.

17

3Counter referral slips returned to health facilities

Hospital returns counter referrals letter with feedback on the referred patient to the referring health center. The counter referral letter is completed in triplicate, with one also given to the patient, and one retained by the hospital.

2.5

4Newborn referred for emergency neonatal care treatment and treated

Newborns referred for emergency neonatal care due to: perinatal complications, low birth weight, congenital malformation, asphyxia, etc.

5

6Referred infants and under-fives with fever Any surgical procedure that does not involve anesthesia or respiratory assistance. 2.5

7 Minor surgical interventionAny surgery in which the patient must be put under general spinal/anesthesia and given respiratory assistance. Major surgery in the case of this package of services is defined as any of the following: Herniarraphy, Appendectomy, Myomectomy, Sleenectomy, Salpingectomy, Hysterectomy, Thyrodectomy, Mastectomy.

5

8Major surgery (excluding CS, including major trauma)

Patients transferred from a lower-level facility (health center or health clinic) to the hospital for emergency treatment.

18

9 Patients transported by ambulance 2.5

10

Number of training sessions held by faculty for nurses, midwifes and PA according to in-service curriculum and defined protocols.

These indicators will incentivize the in-service training activities. 50

11Number of nurses, midwifes and PAs that received specialized in-service training, relevant to benchmarks

10

VerifiedTotal EarningsDefinition

Six Hospitals Total

Fee (USD)Indicators Claimed

(c) Quantity Checklist

Actual % Earned Points

1. Obstructed Labor 0.80 3.87 100% 33% 1.292. Hemorrhage 1.00 4.84 100% 71% 3.453. Maternal Sepsis 1.00 4.84 100% 50% 2.424. Eclampsia 0.70 3.39 100% 47% 1.595. Neonatal Asphyxia 1.00 4.84 100% 67% 3.236. Neonatal Sepsis 1.00 4.84 100% 54% 2.617. Prematurity 0.50 2.42 100% 47% 1.148. Maternal Newborn Best Practices 1.00 4.84 100% 54% 2.619. ETAT 1.00 4.84 100% 33% 1.6110. Malaria 1.00 4.84 100% 71% 3.4511. Pneumonia 1.00 4.84 100% 50% 2.4212. Acute Diarrhea 0.80 3.87 100% 47% 1.8213. Severe Acute Malnutrition 0.60 2.90 100% 67% 1.9414. Surgical Safety 1.00 4.84 100% 54% 2.61

100% 60.00 100% 53% 32.20Total/Average

Checklists Weight (by importance)

Point Allocation

Max %

(b) Process of Care Quality Checklists

Score 1.GENERAL MANAGEMENT (30pt)

2. HUMAN RESOURCES FOR HEALTH (16pt)3. HYGIENE AND MEDICAL WASTE DISPOSAL (27pt)

4. DRUGS MANAGEMENT (30 pt)5. EQUIPMENT AND SUPPLIES (84pt)

TOTAL %

Date of Verfication

TOTAL (187pt)

REPUBLIC OF LIBERIAMinistry of Health and Social Welfare (MOHSW)

Hospital Quarterly Quality AssessmentName of the Hospital

Name of Team Leader of Quality VerificationVerification Period

Quarterly Quality Verification Score

I. Management

II. Structural

(a) Management and Structural Checklist

IndicatorsMax Points

Actual Points Quarter I

1. General Management 30 2.62. Human Resources for Health 16 93. Hygiene and Medical Waste Disposal 27 04. Drugs Management 30 85. Equipment and Supplies 84 48

6. Aggregated Process of Care Score 60 32

Total 247 100Total Percentage 100% 40%

Total Quality Bonuses (USD) 159,678 64,517

PBF Bonus Calculation Tool

Business/Operation Plan

Health Worker Bonus AllocationLHSSP Indices Tool for Bonus Allocation to Individual Health Workers for Hospitals

1 200 50 30 300,000 0 6,944 2 200 70 30 420,000 0 9,722 3 150 80 30 360,000 0 8,333 4 - - - 5 - - - 6 - - - 7 - - - 8 - - - 9 - - - 10 - - - 11 - - - 12 - - -

Quarter:Total PBF Incentives Earned% for Individual Bonus

Attendance points [C]

Hospital Name

Total Individual Bonus

Redemption HospitalJuly-Sept 2013

No Name of staffStaff

categoryMonthly

salary [A]

Perfor-mance

points [B]

$50,00050%$25,000

Total points = [A] x [B] x [C]

Indices of the pe riod

PBF individual

bonus Signature of receipt

Min 50%

Max 50%

~60%

~20%

~20%

(1) C

ontin

uous

mon

itorin

g

(d) Impact Evaluation

Measuring processes and results

65

Liberia: Standards for Management Obstructed Labor: Illustrative Checklist Distilling Essential care Items

(admission, labor)Chart review elements (see chart review guide for specific criteria) ; each element if recorded = 1 point

Charts

1. Admission 1 2 3 4 5

1. Cervical dilation recorded at admission (# of cm)

2. Contraction frequency and duration charted at admission

3. Fetal presentation charted at admission

4. Partograph started when cervical dilation 4 cm or greater

Admission Score (x/4)2. Labor Monitoring (partograph)

1. Cervical dilation recorded at least every 4 hours

2. Frequency and duration contractions recorded at least every 30 minutes

3. Fetal HR recorded at least every 30 minutes

Labor Monitoring Score (x/3)

Each item has chart review guide that defines criteria

Five patient charts reviewed: average score (% adherence best practices) links with bonus

66

Record Reviews Simulations of routine labor

and delivery, postpartum hemorrhage and eclampsia using Mama Natalie

Simulation of newborn resuscitation using Neo Natalie

Simulation of surgical safety checklist use

Patient interviews by phone include basic quality tracers (access to sanitation facilities; recall health education messages; informal payments and general satisfaction using a Likert scale)

https://youtu.be/_miYvoWosS4

Kyrgyzstan: multiple approaches to measuring quality

67



2. Using Data for Decision Making

68

Nigeria: Institutional Deliveries increased from 20% to 44% during 2015(120% increase)

69

Jan-15

Feb-15

Mar-15

Apr-15

May-15

Jun-15Jul-1

5

Aug-15

Sep-15

Oct-15

Nov-15

Dec-15

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Population Coverage for Institutional de-livery – PBF districts

National (PBF) Adamawa NasarawaOndo

Jan-15

Feb-15

Mar-15

Apr-15

May-15

Jun-15Jul-1

5

Aug-15Se

p-15Oct-

15

Nov-15

Dec-15

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Population Coverage for Institutional De-liveries – DFF districts

National (DFF) Adamawa NasarawaOndo

70

Large variability in Institutional Deliveries across Health Centers Fufore District, Adamawa State Nigeria during 2013

December

January

February

March

AprilMay June

July

August

September

-

20

40

60

80

100

120 Pariya HC

Chigari HC

Dasin Hausa HC

Farang HC

Ribadu HC

Furore MCH HC

Choli HC

Gurin HC

Malabu HC

Karlahi HC

Wuro Bokki HC

Kabilo HC

Saint Mary's Clinic HC

Mayo-Ine HC

Burundi: Average total quality score for health centers, by province and time

71

Jun.2010

Sep.2010

Dec.2010

Mar.2011

Jun.2011

Sep.2011

Dec.2011

Mar.2012

Jun.20120.0

20.0

40.0

60.0

80.0

100.0 Mwaro

Muramvya

Kirundo

Cibitoke

Buja-Rural

Kayanza

Ngozi

Makamba

Rutana

Bubanza

Bururi

Gitega

Karuzi

Muyinga

Ruyigi

Cankuzo

Buja-Mairie

Quthing District: average quality in health centers is the same after 12 months piloting of PBF due to autonomy problems

72

General_Management

Child_Survival

Environmental Health

General_Consultations

Reproductive_Health

Essential_Drug_Management

Tracer_Drugs

Maternal_Health

STI_HIV_TB

Comm_Based_Services

0

50

100

2Q14 2Q15

3. Verifying Data Accuracy

73

NIGERIA: Quality of Care at PHCs: Raising the Bar

Dece

mbe

r

Mar

ch

June

Sept

embe

r

Dece

mbe

r

Mar

ch

June

Sept

embe

r

Dece

mbe

r

Mar

ch

June

Sept

embe

r

Dece

mbe

r

Mar

ch

2011 2012 2013 2014 2015

0

10

20

30

40

50

60

70

80

90

100

AdamawaNasarawaOndoNational

Perc

enta

ge Q

ualit

y Sc

ore

Quality of care also improved significantly with emphasis on structural and process of care indicators (higher emphasis on process end 2013 leads to drop)

Overall patient perceptions on quality of care is relatively satisfactory Counter-verification of the quality: relative large discrepancies

74

Concordance in 2015 and 2016

75

Fufore LGA Mayo Belwa LGA Wamba LGA Karu LGA Ile oluji / Okeigbo LGA

Ondo East LGA0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

90%

97%

81%

66%61%

66%

95% 96%

76% 76%

85%

92%2015 Average Concordance2016 Average Concordance0-5% Concordance

Ex-ante verification by district health team may be too gentle and not accurate: too close for comfort or still old fashioned ‘filling under the banana tree’?

Regular counter-verification with credible sanctions are an important requirement

Specifying incentives for district supervisors also seems a promising route (share of earnings; accreditation status; carrots and sticks)

Introduction of modern ICT such as tablet based checklists, which embed meta data (location; time; interviewer passcode) seem a promising approach too

76

Challenges to Measuring and Rewarding Quality Performance

4. Innovations in Measuring and Paying for Quality

77

Virtual Patient presents with symptoms

Provider cares for a variety of clinical cases

Provider goes through the different clinical domains as when they see a patient

Vignettes Provide a Standard Measure of Practice

78

Take History Conduct a Physical Exam Order Tests Make a provisional diagnosis Decide on treatment

Tablets for quantified quality checklists (‘balanced score cards’) with automated uploads to a cloud based database and public dashboard. Offline data entry possible

(as above) Tablet based solution for Vignettes (under development)

Smart phone for community client interviews. Off line data entry possible. Automated uploads to a cloud based database and public dashboard. Results impact on performance payments

Web-based public dashboard for performance benchmarking

79

Technology Aids for Quality Measurements in PBF

1. Quality is poor and varied 2. Much improvements in access and

structural elements of care 3. Improving clinical processes remains the

big immediate challenge 4. Innovations are happening in the space

of measuring clinical processes 5. Data from measurement needs to

translate to decisions

In Summary

81

Improving Quality of Care Using Measurement, Comparison, Validation

If we can measure:Target performanceKnowledge to performCapacity to performPerformance

Then the gap between performance and targeted performance can be broken down into:

The know gapThe know-can gapThe can-do gap

Target

Performance

Gap

Know Gap

Know-Can Gap

Can-Do Gap

Target

Knowledge to perform

Capacity to perform

Performance

Three Gap Model of Performance (Leonard et al., 2015)

83

The Three Gaps in Liberia from 2013 to 2015

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

30%39%

28%

11%

13%25%2%

2% 2%

57%46% 45%

performance can do gap know can gap know gap

2015 full

2015 par-tial

2013 par-tial

The three samples include 10 hospitals in 2015 (2015 Full) and 4 hospitals observed in both 2013 (2013 partial) and 2015 (2015 partial)

84

What do we learn from these gaps?

This is not a pure impact evaluation: the biggest driver of changes in this data is the Ebola crises, not the RBF.

The biggest change from 2013 to 2015 is an increase in the can-do gap, which suggests a drop in motivation consistent with the crises.

The biggest gap is clearly the knowledge gap, but does this mean improved knowledge leads to improved performance?

85

0.2

.4.6

.81

Per

form

ance

.2 .4 .6 .8 1Competence to Perform

bandwidth = .8

Examine the relationship between competence to perform and performance in the full sample. Does performance increase with competence?

When health workers work in teams, performance can be high even if competence is low, but we can see evidence that increasing the competence of health workers at the lower end can improve performance.

But at the upper end, improving competence does not improve performance, even though average performance is low.

86

How to measure?

Many tools are available to measure process quality. – Clinical Observations, Simulated Patients, Standardized

Patients, Paper-Based Vignettes, Tablet-Based Vignettes, Video Vignettes, Patient Chart Audit…

Identify the key bottlenecks.– Observing relatively rare events is difficult and costly.– Consider simulations and vignettes.

Know your sample size.– Larger countries will require larger banks of vignettes or

simulations.– These are costly to set up, but remember that rapid data

means investing in these high startup costs. Ken Leonard’s work in Tanzania shows that there are many

ways of increasing attention span.

87

The Kyrgyz Performance Based Payments (PBP) Project: work jointly done with Aneesa Arur, Arsen Askerov, Jed Friedman, and Asel Sargaldakova Kyrgyz Republic has had persistently high (for the region) maternal and

neonatal mortality rates – Near-universal institutional deliveries (over 95%) and coverage of

primary care services Hypothesis is that poor quality of care is limiting improvements in MMR and

NMR Project aims to improve quality of care for Maternal and Neonatal Health

(MNH)– 3 year pilot of Performance Based Payments (PBPs) focused on quality

of MNH services at district hospitals – Quality to be assessed by peer evaluators every quarter using a

Balanced Scorecard which includes structure, clinical care and process measures of quality (more on this later)

– PBPs will be a dimension of Diagnosis Related Group (DRG) payments for MNH services; Hospital Directors have autonomy over use

– In addition, hospitals expected to also receive performance feedback as part of the PBP intervention package

88

Measuring Quality of Maternal and Newborn Care

The study uses data from the baseline survey of the PBP Impact Evaluation

This survey was conducted in all 63 Rayon Territorial Hospitals and Centers of General Practice in the Kyrgyz Republic.

Instruments included:1.Health facility assessments: Hospital assessment and ANC

checklist2.Simulated patients for post partum hemorrhage and neonatal

asphyxia3.Direct observations of deliveries and antenatal care visits 4.Clinical record audits for normal deliveries, complicated

deliveries, stroke, AMI, neonatal asphyxia5.Patient exit interviews

All components used structured (quantitative) questionnaires or checklists to collect data, and all field workers were trained clinicians

89

Direct Observation: Labor and Delivery

90

Palpates uterus 15 minutes after delivery of placenta

Takes mother’s vital signs 15 minutes after birth

Tasks for second and third stage of labor [4]

Complications during previous pregnancies [3]

Danger signs [2]

General tasks for initial client assessment [1]

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

92%

78%

80%

45%

40%

87%

[1] Checks clients card or asks client her age, length of pregnancy, and parity, Takes temperature, Takes pulse, Asks/notes amount of urine output, Performs general examination (e.g. for anemia, edema), Performs abdominal examination: checks fundal height with measuring tape, Performs abdominal examination: checks fetal presentation by palpation of abdomen, Performs abdominal examination: checks fetal heart rate with fetoscope/ultrasound, Performs vaginal examination (cervical dilation, fetal descent, position, membranes, meconium)[2] Fever, Foul smelling discharge, Headaches or blurred vision, Swollen Face or Hands, Convulsions or loss of consciousness, Shortness of breath, Vaginal bleeding[3] High blood pressure, Convulsions, Heavy bleeding during or after delivery / hemorrhage, previous c-section, Prior stillbirth, Prolonged labor, Prior neonatal death, Abortion, Prior assisted delivery[4] Supports perineum as baby's head is delivered, Assesses completeness of the placenta and membranes, Assesses for perineal and vaginal lacerations

Pre-eclampsia/eclampsia Knowledge Test

91

Wrong: Actively Restrain

Wrong: Give Intravenous Diazepam

MeanActions To Take If Presented With Convulsion [2]

Action to take: stabilize with Anti-Hypertensives

Action to take: stabilize with Magnesium Sulfate

Proper Diagnosis: Severe Pre-Eclampsia

Mean Examination Actions [1]

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

51%

78%

62%

74%

92%

68%

58%

[1] Time Of Onset Of Present Symptoms, Level Of Consciousness, Any Convulsions, Check Vital Signs (Temp, Bp, Pulse, Respirations), Listen To / Assess Fetal Heart Tones, Fetal Movement, Check Urine Protein[2] Administer Oxygen At 4-6 L Per Minute If Available, Place In Side Lying Position, Protect From Injury, Give Magnesium Sulfate, Provide Anti-Hypertensives (Nifedipine Or Apresoline), Actions To Take If Presented With Convulsion: Mean

Comparing Patient Exit Interviews with Direct Observations

92

HIV status Blood pressure

Urine test Augment Episiotomy Timing of Meds

Dry Skin-to-skin Covered

Initial Client Assessment Intermittent Observation of First Stage

Labor

Continuous Observation of Second and Third

Stage

Immediate Care

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

0.800

0.900

1.000

Exit Interview: Unobserved Exit Interview: Observed Direct Observation

Measuring Quality of Maternal and Newborn Care

Administrative data from all 63 RTHs and CGPs on preventable maternal and neonatal complications that are targeted by Kyrgyz RBF pilot.

Extracted data on ICD-10 codes used for DRG payments on: Perineal lacerations Post-partum hemorrhage Other obstetric trauma Birth asphyxia

We calculate rates of delivery and neonatal complications for both types of hospitals and test the various measures of QoC from the survey data against these complications rates to see which measures are more predictive of complications rates.

93

Summary of Key Findings*

1.Instruments appear to be better suited to predicting complications rates for Territorial Hospitals rather than Centers of General Practice.

2.Criterion-based Clinical Audits do not appear to be predictive of hospital quality, particularly in Kyrgyzstan where meticulous documentation was not incentivized prior to the RBF pilot.

3.Direct Observations perform better in terms of having the expected sign on the correlation, but are often not significant predictors of QoC.

4.Simulations using the MamaNatalie anatomical model were more predictive of the administrative maternal and neonatal complications rates. This finding is important from a policy perspective because

training and evaluations of provider skill as well as IEs can use this relatively inexpensive tool.

94

Some Caveats

1.While we use data on the case mix treated by these hospitals, and consider preventable complications that are targeted by the Kyrgyz RBF pilot:

a)Our results may be driven by the fact that complicated cases are systematically referred to some of these hospitals.

b)However, the cadre of hospitals considered here is not the type that patients are referred to.

c)Further, we attempted to select complications that were less likely to be screened through antenatal care.

d)In addition, we account for hospital type in the analysis.2.Further, unobservable third factors may lead to certain areas having

less healthy populations3.Certain complications may also be beyond the control of the hospital

and may instead be a factor of the quality of ANC.

95

RBF and Quality of Care: What the impact evaluations are telling us

Evidence base for RBF and QoC is slim

Das et al. (2016) systematically review the published literature and find 8 studies that explore RBF impacts on QoC with methodological rigor

Wide variation in the studies– Burundi, DRC, Egypt, Philippines, and Rwanda– 3 RCTs, 4 dif-in-difs, 1 propensity matched case-control– 5 focused on PHCs, 2 on district hospitals, 1 on both– 3 directly incentivized limited set of quality indicators, 3 utilized

composite quality index (BSC)– 3 directly paid health workers, 4 paid facilities– Incentives ranged from 5% to 275% of base salary– Measurement of quality includes hosuehold interview, patient exit

interview, record review, direct observation, and vignette responses

97

Evidence base for RBF and QoC (II) Wide variation in the findings:

– Structural quality: very mixed findings

▫ Increase in number of qualified staff and drug availability in DRC 1

▫ Increase in clinical knowledge in Philippines

▫ However majority of cases find little change

– Process quality: some gains in ANC processes

▫ History taking, blood tests, urine tests increased in Egypt

▫ Summary process quality score improves by 0.2 SDs in Rwanda

▫ However no change in DRC, and no measurement in other studies

– Quality outcomes: again, mixed findings

▫ Improved patient knowledge in Egypt and DRC

▫ Improved client satisfaction in DRC 1 and Burundi but not DRC 2

▫ Little change in assessed health outcomes (nutritional status of U5s improves in Rwanda)

98

Evidence base for RBF and QoC (III)

Very difficult to generalize from current evidence base– Diversity of program design and involvement of QoC– Most evaluations not primarily concerned with QoC

Despite several programs granting autonomy and funds to enhance structural quality, evidence of improvement is minimal– Procurement and managerial bottlenecks?

Does increase in utilization negatively spillover onto QoC? These mixed findings call for deeper investigation into

– Design of RBF programs– Implementation of programs

99

Evidence base for RBF and QoC (IV)

RBF impact evaluation portfolio is expected to generate much more evidence (eventually over 30 country studies)

Let’s review in-depth results from two recently completed studies:– Zambia– Zimbabwe

100

Both Zambia and Zimbabwe saw gains in select targeted coverage measures Delivery

– In-facility deliveries increased 12.8 percentage points in Zambia– 13.4 pp increase in Zimbabwe

ANC and PNC– Concomitant gains in PNC in both countries– No gain in ANC coverage in either country

Family planning– No gains in Zambia– 12 pp increase in Zimbabwe, only among women with primary

education or below Child health

– No improvements in vaccination coverage in Zimbabwe– 6-7 pp increase in select vaccination measures in Zambia– 4 pp reduction in extreme stunting in Zimbabwe

101

Zambia: Structural Quality

• Little change in individual measures of structural quality, however an aggregate index suggests gains in RBF compared with pure control districts

• Gains in structural quality of care-specific indices

102

RBF vs. Control 1 RBF vs. Control 2Impact

estimate p-value Impact estimate

p-value

Facility experiences no power outage -0.019 0.881 0.194 0.159Facility experiences no water outage 0.041 0.688 0.051 0.476Infrastructure index 0.195 0.470 0.483* 0.099

RBF vs. Control 1 RBF vs. Control 2

Impact estimate p-value Impact estimate p-value

Curative Care 0.39 0.204 0.28** 0.042

Family planning 0.15 0.578 0.08 0.546Delivery Room 0.61** 0.010 0.57*** 0.000

Zambia: Quality of ANC

• Process measures of ANC quality for a few measures are improved in RBF as compared to C1 and C2, but little gain in overall index

• Household survey results suggest 3 percentage point increase in IPT coverage: a directly targeted process quality indicator

103


Impact estimate p-value Impact estimate

p-value

Weighed -0.02 0.632 0.06 0.251Blood pressure measured -0.03 0.809 0.08 0.452Abdomen measured 0.07 0.152 0.09* 0.063Abdomen palpated 0.00 0.987 0.12* 0.083Advice on diet 0.14*** 0.009 0.02 0.850Quality of ANC index 0.02 0.921 0.33 0.165

Zambia: Quality of child health care

• No apparent gain in process quality of child health visit

104


Impact estimate p-value Impact

estimate p-value

Asked age -0.01 0.880 0.02 0.776

Weighed child -0.07 0.378 0.06 0.498

Measured height -0.10 0.104 -0.02 0.577

Physically examined -0.09 0.327 -0.08 0.350

Quality of care index -0.09 0.669 0.14 0.565

Zambia: Satisfaction on ANC

• Higher levels of patient satisfaction in selected dimensions of ANC (but not all) in RBF as compared to the two controls

• Little apparent increase in overall satisfaction

105



estimate p-value

The health worker spent a sufficient amount of time with the patient 0.08* 0.067 0.08* 0.081You trust the health worker completely in this health facility 0.07* 0.066 0.03 0.569Satisfaction index 0.04 0.826 0.12 0.574

Zambia: Satisfaction on child health care

• Little apparent increase in overall satisfaction for child care

106



estimate p-value

The amount of time you spent waiting to be seen by a health provider was reasonable -0.02 0.823 -0.06 0.477You trust the health worker completely in this health facility 0.11* 0.057 0.04 0.504Satisfaction index 0.09 0.617 0.04 0.858

Zimbabwe: Structural Quality

Improvements in select measures of structural quality:

Higher incidence of biomedical waste disposal (16 % points; p = 0.027)

Increased availability of iron (16 pp), folic acid (21 pp), and urine dipsticks (42 pp)

Increased availability of select equipment electric autoclave (29 pp) and refrigerator (27 pp)

However no gains in majority of measures

107

Structural Quality - Mapping of Checklist

Elements from the quality checklist were extracted from the facility survey instrument and assigned the same weight to calculate the indices.

Process Quality – ANC (Household Survey)

Zimbabwe: Structural and Process Quality

108

Impact estimate p-value

Administration and planning 0.167 0.674Medicines and sundries stock management 0.017 0.969

Out Patient Department 0.468 0.213

Family and Child Health 0.837** 0.021Maternity Service 0.009 0.981Referral services 0.182 0.667

Community services 0.049 0.866Infection control and waste management 0.492 0.272

Impact estimate p-value

Blood pressure measured 0.025 0.570Urine sample taken 0.153** 0.027Blood sample taken 0.084 0.129Any tetanus injection 0.075* 0.056Number of tetanus injections 0.312* 0.063Any iron taken 0.003 0.951Number of days iron taken -1.161 0.868Anti-parasite drugs taken 0.031 0.117Malaria prophylaxis taken 0.033 0.654

Quality of service indicators recorded in the HMIS also show significant increases

Zimbabwe: Process quality in the HMIS

109

Even for indicators that show no significant increase from patient recall data.

Main takeaways and priority questions

Systematic review and two country studies suggest– RBF is effective to improve process quality of ante natal care– Very mixed results on structural quality and client satisfaction– Little evidence in either direction on (a) quality of other processes, (b) long

run health outcomes Challenges with QoC improvements suggest need to revisit how QoC is

measured and incentivized under RBF Scope to revisit efficiency of RBF spending: reallocate funds away from

coverage indicators where coverage is already high and towards quality indicators

Combine RBF with complementary investments in quality improvement (e.g. CQI) to amplify RBF impacts on quality?

Incentivize activities involved in the facility management of quality?

110

THANK YOU

Setting a Path for Improved Health Outcomes RBF

Healthcare

Transcript of Setting a Path for Improved Health Outcomes RBF