Using Operational and HMIS data for Program Monitoring and Impact Evaluation - Zambia

2014 Results & Impact Evaluation Workshop

Zambian Delegation

25th March 2014

Zambia RBF Model

One of the few examples of “contracting in” through the public health sector

Quasi Provider-Purchaser split by different levels of the Zambian Health Care Delivery System

Quantity and Quality data verification Steering Committees (SCs) as Independent Verifiers Periodic External Verification

Performance-Based Payments

“Fee-for-service” on a set of MNCH indicators at Health Centres

Performance Evaluation Framework for District Medical Offices

Managerial and financial autonomy of health facilities

2

Operational Data

3

Trends in Performance Indicators: Q2 2012-Q4 2013

0

2,000

4,000

6,000

8,000

10,000

12,000

14,000

16,000

Q2 2012 Q3 2012 Q4 2012 Q1 2013 Q2 2013 Q3 2013 Q4 2013

Nu

mb

er

Skilled Deliveries

ANC prenatal andfollow up visits

Postnatal visit

Fully vaccinatedchild

Third doseFansidar IPT

Preg. womengiven Niverapine

and AZT

4

Trends in Performance Indicators: Q2 2012-Q4 2013…

0

100,000

200,000

300,000

400,000

500,000

600,000

700,000

800,000

900,000

Q2 2012 Q3 2012 Q4 2012 Q1 2013 Q2 2013 Q3 2013 Q4 2013

Nu

mb

er

CurativeConsultation

FP users ofmodern methods

Preg. womencounseled andtested for HIV

5

Percentage Increase – Quantity Indicators

14%

-1%

23%

-49%

69%

44%

94% 70%

203%

14% 19% 21% 22% 22% 25%

63% 67%

88%

-100%

-50%

0%

50%

100%

150%

200%

250%

CurativeConsultation

HIV Preg.women

givenNiverapine

& AZT

PregnantWomen C&T

for HIV

ANCprenatal and

follow upvisits

SkilledDeliveries

Third doseFansider IPT

Postnatalvisit

FullyImmunized

Children

FP users ofmodern

methods

Per

cen

tag

e C

han

ge

Q2 2012 Vs Q4 2013 (Since Project Start) Q4 2012 Vs Q4 2013

6

Use of HMIS Data

7

Similarities and Differences: OP Vs HMIS Data

• OP and HMIS data are collected from the same health facilities, and same data records

• However, OP data is compiled much quicker than HMIS data given the requirements for verification, and linkages to payments

• OP data is only complied from patient registered as compared to HMIS which is complied from registers and tally sheets

• Consolidation of OP and HMIS data is done by different personnel at district level

• 100% of the OP data is verified on a monthly basis while the HMIS mainly relies on self-reported data which is occasionally verified

• OP and HMIS data MUST show the same trend in indicators, despite differences in magnitude

8

Using HMIS to compare across the 3 study arms of the Impact Evaluation

What can we say before endline?

Diff-in-diff analysis of trends before and after RBF

Analysis period – Jan 2011 to Dec 2012: RBF introduced April 2012

Not definitive analysis since HMIS is self-reported but:

Important check on RBF: Are results from the operational data consistent with HMIS?

What is the impact of RBF on non-incentivized indicators?

9

Impact Evaluation

Explores whether there is a causal link between the RBF project and the results

Baseline – Quantitative and Qualitative

Process Evaluation (interviews, Observations, operational and HMIS data review)

End line – Quantitative and Qualitative

Three (3) study arms:

10 RBF Intervention Districts (RBF)

10 Input-Based Financing Districts (C1)

10 Pure Control Districts (C2)

10

Impact on incentivized indicators

All measures: per service per facility per month

Gains in several targeted services, no change in total utilization, and declines in immunization

No gains from additional financing to districts

RBF vs. Additional financing

Coef 0.904 0.815 0.696 12.944 2.220 -9.316 -2.783

p-value 0.045 0.231 0.229 0.002 0.019 0.735 0.024

RBF vs. Control

Coef 1.174 1.954 1.586 7.850 2.243 -39.929 -2.761

p-value 0.005 0.002 0.011 0.055 0.031 0.158 0.011

Attendance

outpatient

total (calc)

Immunised

fully <1 year

new

Antenatal

1st visit

before 20

weeks

IPT 3rd

dose to

pregnant

woman

Postnatal

care within

6 days

Attendance

Family

Planning

total (Calc)

Delivery by

skilled

personnel

Additional financing vs. Control

Coef 0.035 1.117 0.882 -5.022 0.027 -31.420 0.035

p-value 0.979 0.107 0.114 0.194 0.976 0.247 0.979

11

Trend in Skilled delivery (Jan 2011-Dec 2012)

0

200

400

600

800

1000

1200

1400

1600

1800

2000

Input Based

Pure Control

Data Source: HMIS, MoH

RBF implemented

12

RBF

Impact on non-incentivized indicators

Little spill-over to the non-incentivized

Additional financing hired more staff (but no change in service measures)

RBF vs. Additional financing

Coef -0.945 3.046 0.136 -0.326 -3.578 -1.018

p-value 0.780 0.090 0.203 0.027 0.563 0.773

RBF vs. Control

Coef -3.847 2.147 0.046 0.181 8.057 0.298

p-value 0.306 0.191 0.799 0.289 0.032 0.759

Additional financing vs. Control

Coef -3.302 -0.840 -0.084 0.511 10.542 1.359

p-value 0.327 0.679 0.588 0.001 0.138 0.748

Supportive

supervision

visits this

month

Vitamin A

supplement

to 6-11

months

New TB

patient total

(calc)

TB patient

completed

treatment

Support

staff newly

recruited

Nurse

midwife

workdays

on duty

13

Use of OP and HMIS data has made it possible to: Triangulate OP data with HMIS data i.e. Check on

consistency of OP data with HMIS

Independently verify the OP data

Monitor trends in incentivized and non-incentivized indicators across the three (3) research arms

Monitor the utilization of funds by indicators, and the overall amount allocated to the RBF Project

Make adjustments to the RBF design, as well as to provide more capacity building and technical support

14

Examples of how the Emerging Information has been used

Application of the quality tool changed to reward for quality improvements instead of penalizing for quality deficits

Set investment component at a minimum of 40%, and staff performance incentives at a maximum of 60%

Increase assessment fees for hospitals doing quality audits

Revise TA package to draw on local capacities

Enhanced technical support to underperforming health facilities

Introduction of supervision fees for Provincial RBF Steering Committees

15

Challenges

• Late transmission of HMIS data

• Poor quality of HMIS data due to migration to a web-based DHIS-2

• Inadequate data entry clerks at health facilities particularly in the control districts

• Costly to conduct a process evaluation involving observations at health facilities, and interviews with service providers, patients, and community members

16

Thank You

17