Task Force framework for assessment of surveillance data

using examples from Tanzania, Brazil and Russia

Ana Bierrenbach

Geneva, March 2010

March 26, 20102 |

Task Force frameworkTask Force framework

March 26, 20103 |

Standards or benchmarks for direct measure of TB incidence and mortality

Standards or benchmarks for direct measure of TB incidence and mortality

� To be attained by TB surveillance system

� To be attained by TB surveillance data

� To be attained for direct measurement of trends

� To be attained for surveillance systems to be

considered as missing negligible number of cases and

deaths

March 26, 20104 |

Progress in applying framework (2009)Progress in applying framework (2009)

4 regional workshops (red) with > 50 countries,

3 country missions (blue)

March 26, 20105 |

What do we do in the workshops

and country visits?

What do we do in the workshops

and country visits?

WORKBOOK

March 26, 20106 |

1. Data quality1. Data quality

March 26, 20107 |

Data qualityData quality

1. Completeness of notification data and other quality checks

• Are all reports complete and compiled?

• Are there duplications and misclassifications?

2. Internal consistency

• Is there more sub-national variability in notification rates than

expected?

• Is there more variability over time than expected?

• Is laboratory diagnosis of documented quality?

3. External consistency

• Are proportions and rates consistent with current knowledge on TB

epidemiology?

March 26, 20108 |

Removing duplicates in Brazil (2006)Removing duplicates in Brazil (2006)

+4.465.162.3-6.337.439.969,80274,492

afterbeforeafterbeforeafterbefore

Change(%)

Cured

(%)

Change(%)

Notification

rate

New cases

Source: Bierrenbach A et al. IJTLD 2010 (in press)

Duplicates/misclassifications inflate TB notification rates

March 26, 20109 |

Trends in TB notification rates

by case type in Tanzania

Trends in TB notification rates

by case type in Tanzania

05,000

10,00015,00020,00025,00030,00035,00040,00045,00050,00055,00060,00065,00070,000

Smear + Smear - Extra-P Relapse Return Failure Other

Smooth curve in line with HIV epidemic

Consistency in the distribution of case types over time

Reporting of retreatment case initiated 2001

March 26, 201010 |

Notification rates by region in Tanzania - 2007Notification rates by region in Tanzania - 2007

0

200

400

600

800

1,00 0

1,20 0

1,40 0

1,60 0

1,80 0

2,00 0

Ne

w c

ase

s pe

r 100

K p

op

Aru

sha

Dar

es s

ala

am

Dod

om

aIr

inga

Ka

ge

raK

aska

zin

i P

em

ba

Kaska

zin

i U

ng

uja

Kig

om

aK

ilim

an

jaro

Ku

sin

i P

em

ba

Ku

sin

i U

ng

uja

Lin

di

Ma

gh

ari

bi U

ng

uja

Man

yara

Mara

Mb

eya

Moro

go

roM

twara

Mw

an

za

Pw

an

i

Ru

kw

aR

uvum

aS

hin

ya

nga

Sin

gid

a

Ta

bo

raT

an

ga

Overdispersion of TB notification rates, only partially explained by HIV prevalence

March 26, 201011 |

Indicators for internal and external consistency

by region in Tanzania - 2007

Indicators for internal and external consistency

by region in Tanzania - 2007

0

20

40

60

80

100

% n

ew

/all

Aru

sha

Dar

es s

ala

am

Dodom

aIr

inga

Kagera

Kaskazin

i P

em

ba

Kaskazin

i U

nguja

Kig

om

aK

ilim

anja

roK

usin

i P

em

ba

Kusin

i U

nguja

Lin

di

Maghari

bi U

nguja

Manyara

Mara

Mbeya

Moro

goro

Mtw

ara

Mw

anza

Pw

ani

Rukw

aR

uvum

aS

hin

yanga

Sin

gid

aT

abora

Tanga

0

20

40

60

80

100

% p

ulm

/new

Aru

sha

Dar

es s

ala

am

Dodom

aIr

inga

Kagera

Kaskazin

i P

em

ba

Kaskazin

i U

nguja

Kig

om

aK

ilim

anja

roK

usin

i P

em

ba

Kusin

i U

nguja

Lin

di

Maghari

bi U

nguja

Manyara

Mara

Mbeya

Moro

goro

Mtw

ara

Mw

anza

Pw

ani

Rukw

aR

uvum

aS

hin

yanga

Sin

gid

aT

abora

Tanga

% new / all % pulm / new

% new / all : consistent, but above expected value

% pulm / new : consistent, variations in line HIV prev.

March 26, 201012 |

Standards/benchmarks forTB surveillance system

Standards/benchmarks forTB surveillance system

� NTPs routinely assess completeness of reporting of TB notification data

� Clear documentation of changes to case definitions and reporting of cases for at least the last 10 years

� Data on case finding efforts and main determinants of TB are available for several years

� NTPs have a case-based electronic TB database

� NTPs check and correct for duplications and misclassifications in notifications: linkage within the last 3 years covering a period of at least 5 years

� NTPs perform inventory studies using routinely collected data and through special studies where appropriate (large number of providers TB care not linked to NTP).

� NTPs analyse TB mortality data from vital registration systems

March 26, 201013 |

Standards/benchmarks forTB surveillance data

Standards/benchmarks forTB surveillance data

� Limited missing data for main variables, for at least 10

years

� Limited variability over time and space, for at least 10

years, or notification data show considerable variability,

which can be explained

� Data consistent with expected values based on TB

epidemiology knowledge

� Limited duplications and misclassifications in full audits

and spot checks, for at least 5 years

March 26, 201014 |

2. Trends2. Trends

March 26, 201015 |

Do surveillance data reflect trends in incidence and mortality?

Do surveillance data reflect trends in incidence and mortality?

1. Have notifications been increasing, decreasing or stable over

time?

2. Were there any changes in case-finding efforts and/or recording

and reporting that might have affected notifications over time?

3. How have factors that may influence TB incidence changed over

time, and have they had any impact on underlying TB

incidence?

March 26, 201016 |

0.00

0.02

0.04

0.06

0.08

1970 1980 1990 2000 2010

HIV

pre

vale

nce

Tanzania: Adults 15 years old or moreUNAIDS estimates

0

200

400

600

800

1000

Full-

tim

e T

B n

urs

es

1995 2000 2005 2010

Factors affecting

TB notifications in Tanzania

Factors affecting

TB notifications in Tanzania

0

200

400

600

800

Sm

ear

labs

1995 2000 2005 2010

↑ notification mirrors ↑ case finding and HIV, but difficult to disentangle given lack of data disaggregated by HIV and case type, for ≠ years

March 26, 201017 |

0

20000

40000

60000

80000

100000

120000

140000

1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006

Nu

mb

er

of

ne

w T

B c

as

es

0

20

40

60

80

100

120

Ne

w c

as

es

pe

r 1

00

,00

0 p

op

ula

tio

n

Reported new civilian cases Reported new non-civilian cases

TB notification rate (civilian) TB notification rate (non-civilian)

Civilian TB

Non-civilian TB cases

Source of referral

of TB notifications in Russia

Source of referral

of TB notifications in Russia

Impact of changes in notification policy

March 26, 201018 |

Standards/benchmarks for direct measurement of TB trends (examples)

Standards/benchmarks for direct measurement of TB trends (examples)

� Trends in TB notifications assessed to mirror trends in incidence:

– Trends in TB notifications mirror trends in incidence after corrections for changes in case definitions, policies in reporting and case finding

• Need notifications disaggregated on HIV status by case type, at least in high HIV burden countries

– Trends in TB notifications mirror trends in TB mortality, and no evidence in changes in case fatality rates

� Trends in TB deaths assessed to mirror trends in mortality:

– No evidence of changes in consistency in coding, policies in reporting and coverage of the VR system over 10 years

March 26, 201019 |

3. Are we capturing all cases and deaths?

3. Are we capturing all cases and deaths?

March 26, 201020 |

Are we capturing all TB cases and deaths?Are we capturing all TB cases and deaths?

1. What proportion of incident cases are missing from

routine notifications and why?

2. What proportion of TB deaths are missing from the

vital registration systems and why?

March 26, 201021 |

The Onion Model

All TB cases

Undiagnosed cases

Diagnosed but not notified cases

Notified casesRecorded in

notification data

Diagnosed by NTP or collaborating

providers

Diagnosed by public or private providers, but

not notified

Access to health facilities, but don't go

No access to health care

Presenting to health facilities, but undiagnosed

March 26, 201022 |

Estimates in Tanzania

before and after discussions

Estimates in Tanzania

before and after discussions

-7586.2Country's CDR (2007)

-2413.8Sum of % of missing cases:

layers 2 to 6

1.51.52. Diagnosed by NTP but not notified

- Exclusive distribution of TB

drugs by NTP

0.90.93. Diagnosed by public non-NTP

92.44. Presenting but not diagnosed

-↑ diagnosis TB following

introduction recent

interventions

-↑ diagnostic delay

52.75. Access but do not go

-93% pop within 10 km basic

health care unit6.36.36. No access to health care

Source of evidenceAfter

discussions

Before

discussions

Onion layers (% total new cases

missed in each layer)

March 26, 201023 |

Can we avoid having to

depend on expert opinion?

Can we avoid having to

depend on expert opinion?

� Inventory studies

� Vital registration data

� Capture-recapture studies

� Prevalence of TB disease surveys (health care seeking behaviour)

� Innovative operational research

March 26, 201024 |

Substantiating expert opinionSubstantiating expert opinion

� Access to health from demographic and health surveys data (Layer 6)

� Overall performance of health systems as measured by: (Layer 5, 6)– Infant mortality ratio

– Number of primary heath care units or doctors per population

– % of assisted births

� Performance of TB diagnostic systems (Layer 4, 5)– % people who died from TB (Vital registration data) and never accessed

TB diagnosis and treatment

– EQA of labs

– KAP studies (health seeking behaviour), delay studies

� Contribution of different TB care providers (Layer 3)– Health expenditure in the private or nongovernmental organization sectors,

out-of-pocket expenditure

� TB drug distribution (Layer 2)

March 26, 201025 |

Standards/benchmarks to be attained for surveillance systems to be considered as missing

negligible number of cases and deaths

Standards/benchmarks to be attained for surveillance systems to be considered as missing

negligible number of cases and deaths� VR systems meets minimum standards of quality and coverage

� Excellent geographical and financial access to health care facilities where TB diagnosis is available

� Staff have proven competency to diagnose TB

� Bacteriology laboratories are quality assured

� Limited or negligible undernotification from the private sector as evidenced by:

– Limited health care expenditures in the private sector (e.g. ≤ 5%)

� Limited or negligible undernotification from the public sector as evidenced by:

– All public providers only have access to anti-TB drugs following notification of a TB case

March 26, 201026 |

Framework for TanzaniaFramework for Tanzania

Relied on "Onion" model based on (mostly) expert opinion combined with some evidence about health system coverage/access and some TB-specific KAP study data

DATA QUALITY

TRENDS

Do surveillance data

reflect trends in TB incidence and

mortality?

ARE ALL TB CASES AND TB

DEATHS CAPTURED IN

SURVEILLANCE DATA?

IMPROVE

surveillance

Data do not yet provide direct measurement. Incidence estimates revised downwards (CDR up). Prevalence survey will provide important new data

UPDATE estimates of TB incidence and mortality

Completeness – appears to be verified, BUT analysis not available at national level Cannot assess duplications/ misclassifications since data are aggregated (not case-based); Data mostly consistent, within ranges expected, but extreme values in a few regions

EVALUATE

trends and impact of TB control

notifications ≈ TB incidence

VR TB mortality ≈ TB deaths

HIV and case-finding have affected trends in notification BUT difficult to disentangle effects - notifications disaggregated by HIV status not available and data on case-finding only available at national level

Need data disaggregated by HIV status at district level to be compiled at national level

4. Implement updated R&R recommendations

1. Roll-out case-based ERR

3. Strengthen M&E supervision

2. Routinely assess data quality esp. in "outliers"

March 26, 201027 |

Major lessons learnedMajor lessons learned

� Vital importance of case-based electronic R&R systems

� Data on notifications by case type disaggregated by HIV status need to be routinely collected, especially in high HIV-prevalence setting

� Need to strengthen analytical capacity

� Need for improvement for supervision of M&E

� Need to progressively increase reliance on data to assess cases missed by surveillance and progressively reduce use of expert opinion

� Need for standards and benchmarks

� Need for better availability and use of vital registration data, as well as more inventory studies and operational research

March 26, 201028 |

Major next stepsMajor next steps

� Further workshops and country missions with Global Fund

� Train group of consultants to participate in country missions

� Develop and reach agreement on standards / benchmarks

– Working group to further develop standards/benchmarks

� Help countries transition to TB electronic R&R

– Minimum standards for a TB electronic R&R system + software specifications

needed to achieve them

� Develop training material (SOPs, guidelines):

– Operational research to assess data quality and M&E systems

– NTP supervision of M&E activities

– Programme review missions

� Jointly review M&E GF grant applications

March 26, 201029 |

Questions to the Task ForceQuestions to the Task Force

1. What are your general comments on progress to date in applying the Task Force framework?

2. Do you agree with the proposed next steps? Is there

anything important that is missing or should be deleted?

3. Which of the standards listed in the annex do you agree or disagree with, and do you have any suggestions for

additional standards that could be used?

4. Would you be willing to be part of a small working groupresponsible for further development of standards and

benchmarks that would need to be met for surveillance data to be considered a direct measurement of disease burden?