Scalable and comprehensive M&E systems for the increasingly complex,

multi-site, multi-country, HIV prevention, care, and treatment programs

Denis Nash, PhD, MPH

Director, Monitoring, Evaluation, and Research Unit

Columbia University, ICAP

M&E challenges in the context of rapid scale-up

• Rapid scale-up of sites– More sites from which to collect indicators

• Expanding portfolio of services and funding sources– New (additional) indicators to collect from sites

• Still missing some key indicators– e.g., deaths and LTF among pre-ART patients

• Poor data quality– Often a lack of infrastructure for recordkeeping and data

management– Completeness and accuracy of paper-based systems

• Not always a priority at the site level

• Lack of simple yet robust data systems and skills to use them at sites– For aggregate information– For patient-level information

M&E challenges in the context of rapid scale-up

• Multiple reporting demands– Multiple entities require reports, often with different

indicator requirements, deadlines, and reporting periods• Some important aspects of programs are hard to

capture with patient-focused or patient-driven indicators– e.g., family focused care or comprehensiveness of

program • Data dissemination and use of data

– The data that are disseminated aren’t always being put to good use

• Why not???? Are the data actually useful? Are the data presented in a way that people can use them? Who is the right audience and are the data reaching the right audience?

Challenge: Rapid scale-up of sites; expanding portfolio of services and funding sources

Response:• Created a “Site Census”

activity whereby basic info about new and existing sites is routinely updated by ICAP staff online – Planned/funded activities– Targets– Broken out by donor

Number of ICAP/CU-supported sites by country

(n=310)

310

240

173 159138 122

92

43 390

50

100

150

200

250

300

350

Total sites Care &treatment

TBscreeningfor HIV pts

PMTCT Lab PICT/VCT HIVtesting for

TB pts

EID PMTCT+

Type of activity

Nu

mb

er

of

sit

es

Source: ICAP Site Census, September 2007

Number of ICAP-supported sites by activity, August 2007

Number of ICAP-required indicators by programmatic areas

0

50

100

150

200

250

Jun-Sep 06 Oct-Dec 06 Jan-Mar 07 Apr-Jun 07 Jul-Sept 07

Nu

mb

er

of

ind

ica

tors

re

po

rte

d p

er

sit

e

Care and treatment PMTCT TB/HIV

Care and treatment

(166 indicators)

PMTCT

(42 indicators)

TB/HIV(20 indicators)

Coming soon: 1. Testing and Counseling2. Laboratory

Challenge: Still missing some key indicators

Response:

• Planning to add a few key indicators to the existing list (mostly related to pre-ART patients)

• Collect and computerize good quality, patient-level data where possible

• SOPs for data collection

Challenge: Poor data quality

Response:• Developed a data quality SOP

– Compare medical records vs. registers– Compare medical records vs. database

• Data entry checks• Data quality queries

Issues: • Poor documentation in medical records

– Whose job is it to shore up documentation?• Not yet fully embraced by field staff

Care and treatment indicator collection and submission

M&E Officer M&E Officer

On-line reporting module (URS)

MER Data Analyst

Automated

data checks

Site B Site ASite A Site B

Automated

data checks

Manual data checks

MER Liaison MER LiaisonMER C&T Lead

CDC-AtlantaCDC-Atlanta

Data queries

Data queries

M&E & Clinical staff

Program Review

M&E & Clinical staff

USGUSG USGUSG

Data quality assessment: Bishoftu Hospital, Ethiopia

Variable % missing for ART patient visits (n=65)

% missing for pre-ART patient visits (n=77)

Visit date 0% 66%

Weight 0% 69%

Pregnancy 73% 93%

WHO stage 0% 68%

Opportunistic infection 60% 90%

TB screen 5% 70%

ART regimen 0% n/a

Cotrimoxazole 6% 81%

Date of next visit 0% 71%

Overall % missing 11% 75%

Challenge: Lack of simple/robust data systems as well as skills to use them at sites

Response:

• Recruiting/retaining local database developers

• Remote and on-site TA to develop and implement systems

• Training

Challenge: Multiple reporting demands

Response: Unified reporting system– Unifying factor: Site Census

GIS

Patient level data

Other indicators, as appropriateTesting &

counseling

indicators

Care &

treatment

indicators

TB/HIV

indicators

pMTCT

indicators

P-FACTS

Site Census

CIEISI

N

Donor/MOH

reports

Maps

Web

components

Data

processingOutputs

Datasets

Data ch

ecks & clean

ing

QA

Canned/

ad hoc

reports

Data entry

Data collection

Detailed

analyses

Dashboards for summary statistics

ICAP URS

Data entry modules for site census, care and treatment, pmtct, tb/hiv, p-facts

URS Data entry screen for OGAC care and treatment indicators

Challenge: Some important aspects of programs are hard to capture with patient-focused or patient-driven indicators

Response:

• Created a Program and Facility Characteristics Tracking System (PFaCTS)– Info on program and facility characteristics are

routinely and systematically captured using a standard instrument

0

20

40

60

80

100

120

Urban Rural

Primary Secondary Tertiary

Location and type of ICAP-supported care and treatment facilities (n=177)

Source: P-FaCTS, January 2007

2 services, 5%

3 services, 14%

4 services, 23%

5 services, 57%

1 service, 2%

On-site services at ICAP-supported care and treatment facilities (n=177)

Source: P-FaCTS, January 2007

0

20

40

60

80

100

120

140

160

180

PMTCT VCT TB Labor andDelivery

ARTpharmacy

On

-sit

e p

rog

ram

ava

ilab

ilit

y

n=152/177 n=165/177 n=158/177 n=142/177 n=141/177

None, 5%

1 service,

30%

2 services,

31%

3 services,

15%

4 services,

15%

5 services,

6%

Patient support services at ICAP-supported care and treatment facilities (n=177)

Source: P-FaCTS, January 2007

0

20

40

60

80

100

120

140

160

180

Adherencesupport

Foodsupport for

adults

Foodsupport for

infants

Peereducatorprogram

Outreach

On

-sit

e p

rog

ram

av

ail

ab

ilit

y

n=168/177 n=45/177 n=48/177 n=49/177 n=83/177

Challenge: Data dissemination and use of data

Response:

• Data Dissemination website

• Unified reporting system– Combine reported data across programmatic

areas to produce integrated analyses

• Data Dissemination meetings

Initiated ICAP data dissemination website

Automated summary reports

Target ART enrollment to February 2008 and actual enrollment by country at ICAP-supported facilities* as of June 2007

18,735

28,013

4,700

13,963

8,885

15,330

7,620

13,059

20,551

13,033

4,201

25,983

7,009

20,113

0

10,000

20,000

30,000

Ethiopia Kenya Mozambique Nigeria Rw anda South Africa Tanzania

Nu

mb

er o

f p

atie

nts

Target ART enrollment Actual ART enrollment

*Target data not available for Lesotho, Swaziland, or Zambia

Source: ICAP Site Census, August 2007

4.2

0.0

1.6

5.4

2.9

0.70.0

1.9

0.01.0 1.0 0.5 0.0

1.00.0

1.20.0

6.0

23.7 24.2

7.5

9.5

0.0 0.0

10.8

33.3

0.0

6.0

0.7

25.9

1.4

3.34.5

1.4

3.34.5

0

5

10

15

20

25

30

35

Ethiopia Kenya Lesotho Mozambique Nigeria Rwanda South Africa Tanzania All countries

per/

1000 p

ati

en

ts o

n A

RT

Physicians Health officers/medical assistantsNurses Outreach workers

Median number of staff per 1000 patients on ART at ICAP-supported care and treatment facilities by cadre (n=177)

Source: P-FaCTS, January 2007

ART patients in more comprehensive programs may have better follow-up care

Presence of CD4 Count and Support Activities

0

0.2

0.4

0.6

0.8

1

% o

f c

oh

ort

wit

h C

D4

co

un

t 6

mo

nth

s

po

st-

AR

T i

nit

iati

on

1 or 2 activities 3 or 4 activities

Median=44%

Median=74%

* Support activities include peer-educator program, outreach program nutritional support, adherence supportSource: Quarterly report data and PFaCTS, January 2007

*

p<0.01

Conclusions• Need to augment conventional M&E systems to keep up

with complexity of scale-up– Site census

• Important to capture indicators at multiple levels– Program and facility-level indicators (e.g., models of care)– Context

• Data quality is a major issue that is largely unaddressed• Combining data from multiple levels and contexts can

produce substantial synergy• Dissemination

– Need to be creative and provide data to those “who need to know” in a way that is useful and engaging