11

Risks and Benefits of Risks and Benefits of Home-Use HIV Test KitsHome-Use HIV Test Kits

Richard Forshee, Ph.D.Richard Forshee, Ph.D. U.S. Food and Drug AdministrationU.S. Food and Drug AdministrationCenter for Biologics Evaluation and Center for Biologics Evaluation and ResearchResearchOffice of Biostatistics and EpidemiologyOffice of Biostatistics and Epidemiology

BLOOD PRODUCTS ADVISORY COMMITTEEBLOOD PRODUCTS ADVISORY COMMITTEE96th Meeting, November 16-17, 200996th Meeting, November 16-17, 2009

Bethesda Marriott HotelBethesda Marriott Hotel5151 Pooks Hill Rd., Bethesda, MD 208145151 Pooks Hill Rd., Bethesda, MD 20814

22

AcknowledgmentsAcknowledgments

Mark Walderhaug, Hong Yang, and Mark Walderhaug, Hong Yang, and Arianna SimonettiArianna SimonettiOffice of Biostatistics and EpidemiologyOffice of Biostatistics and Epidemiology

Elliot Cowan and Hilary HoffmanElliot Cowan and Hilary HoffmanOffice Of Blood Research & ReviewOffice Of Blood Research & Review

Bernard Branson, Arielle Lasry, and Bernard Branson, Arielle Lasry, and Stephanie SansomStephanie SansomCenters for Disease Control and Centers for Disease Control and PreventionPrevention

33

Purpose of Model 1Purpose of Model 1

Design a tool to estimate the public Design a tool to estimate the public health benefits and risks of a home-health benefits and risks of a home-use HIV test kit under different test use HIV test kit under different test characteristicscharacteristics

If benefits and risks could be If benefits and risks could be quantified, it would be possible to quantified, it would be possible to define a region of sensitivity and define a region of sensitivity and specificity where the benefits exceed specificity where the benefits exceed the risksthe risks

44

Purpose of the Model 2Purpose of the Model 2

Benefits and risks of different test Benefits and risks of different test outcomes are not comparable, so outcomes are not comparable, so professional judgment is requiredprofessional judgment is required

The tool will help decision-makers The tool will help decision-makers make informed judgments about make informed judgments about the likely tradeoffsthe likely tradeoffs

We seek the committee’s input on We seek the committee’s input on these issuesthese issues

55

Public Health Benefits Public Health Benefits and Risks Considered by and Risks Considered by ModelModel BenefitsBenefits

– True Positive resultsTrue Positive results– True Negative resultsTrue Negative results

RisksRisks– False Negative resultsFalse Negative results– False Positive resultsFalse Positive results– Failed Tests (invalid results or user error)Failed Tests (invalid results or user error)

Benefits and risks are in the context of Benefits and risks are in the context of persons persons who would not otherwise be who would not otherwise be testedtested

66

Qualitative Summary of Qualitative Summary of Public Health Benefits and Public Health Benefits and Risks of Different Test Risks of Different Test Outcomes for a Home-Use Outcomes for a Home-Use HIV Test KitHIV Test Kit

77

True PositiveTrue Positive

Allows earlier medical intervention and Allows earlier medical intervention and entry into careentry into care

Knowledge of individual HIV status allows Knowledge of individual HIV status allows for behavior modification to prevent HIV for behavior modification to prevent HIV transmission, and allows for partner transmission, and allows for partner notificationnotification

Knowledge of HIV prevalence can allow Knowledge of HIV prevalence can allow for better targeting of public health for better targeting of public health resources as cases come to medical resources as cases come to medical attentionattention

88

True NegativeTrue Negative

Peace of mindPeace of mind Assistance in appropriate Assistance in appropriate

targeting of public health targeting of public health resourcesresources

99

False PositiveFalse Positive

Unnecessary personal anxietyUnnecessary personal anxiety Additional testing requiredAdditional testing required Will people seek additional Will people seek additional

testing?testing?

1010

False NegativeFalse Negative

False reassurance False reassurance Unsuspected transmission of virus Unsuspected transmission of virus

and continued high risk behaviorand continued high risk behavior Delayed medical interventionDelayed medical intervention False negative results may False negative results may

include tests taken during window include tests taken during window periodperiod

1111

Failed Test, Person is Failed Test, Person is HIV+HIV+

Another test performedAnother test performed– Delay in diagnosisDelay in diagnosis– Additional testingAdditional testing

No further testingNo further testing– False reassuranceFalse reassurance– Unsuspected transmission of virus Unsuspected transmission of virus

and continued high risk behaviorand continued high risk behavior– Delayed medical interventionDelayed medical intervention

1212

Failed Test, Person is Failed Test, Person is HIV-HIV-

Another test performedAnother test performed– Possibility for personal anxietyPossibility for personal anxiety– Delay in determining statusDelay in determining status– Additional testingAdditional testing

No additional testingNo additional testing– Personal anxietyPersonal anxiety– Status remains unknownStatus remains unknown

1313

Subpopulations Subpopulations ModeledModeled1.1. Low Risk Heterosexuals (LRH):Low Risk Heterosexuals (LRH):

0 or 1 partner in previous year0 or 1 partner in previous year2.2. High Risk Heterosexuals (HRH): High Risk Heterosexuals (HRH):

previous yearprevious year3.3. Men Who Have Had Sex With Men Men Who Have Had Sex With Men

(MSM): (MSM): previous yearprevious year

4.4. Injectable Drug Users (IDU):Injectable Drug Users (IDU):previous yearprevious year

1414

Time Frame of the Time Frame of the ModelModel

Model estimates annual rates for 2nd Model estimates annual rates for 2nd and succeeding years after and succeeding years after introduction by considering currently introduction by considering currently active at-risk group (with risky active at-risk group (with risky behaviors in past 12 months) as behaviors in past 12 months) as regular users of the test kitregular users of the test kit

Use of test kit might be higher or lower Use of test kit might be higher or lower in the 1st year after introductionin the 1st year after introduction

1515

Global InputsGlobal Inputs

1.1. Percent of test kits that fail (5%)Percent of test kits that fail (5%)

2.2. Sensitivity of test kits that did NOT failSensitivity of test kits that did NOT fail

3.3. Specificity of test kits that did NOT failSpecificity of test kits that did NOT fail

These are treated as hypothetical and These are treated as hypothetical and will be estimated across a range of will be estimated across a range of plausible values. Sensitivity does not plausible values. Sensitivity does not include window period cases.include window period cases.

1616

Sensitivity Values Sensitivity Values ModeledModeled

MeanMean

99.5%99.5%97.0%97.0%95.0%95.0%92.0%92.0%90.0%90.0%

1717

Specificity Values Specificity Values ModeledModeled

MeanMean

99.5%99.5%99.0%99.0%98.5%98.5%98.0%98.0%97.5%97.5%

1818

Sub-population Sub-population Specific InputsSpecific Inputs

1.1. Size of sub-populationSize of sub-population2.2. Percent of sub-population that is Percent of sub-population that is

untested (previous year)untested (previous year)3.3. Percent of untested persons who Percent of untested persons who

are HIV+are HIV+4.4. Percent of untested persons who Percent of untested persons who

would use a home-use HIV test would use a home-use HIV test (highly uncertain)(highly uncertain)

1919

Sub-population Sub-population InputsInputs

Presenting means onlyPresenting means only

Uncertainty was included in Uncertainty was included in modelmodel

2020

Sub-population SizeSub-population Size

3,840,000

1,580,000

16,200,000

167,200,000

MSM

IDU

HRH

LRH

Sub

-pop

ulat

ion

0 50,000,000 100,000,000 150,000,000 200,000,000Size of sub-population

2121

Percent UntestedPercent Untested

30%

31%

65%

65%

MSM

IDU

HRH

LRH

Sub

-pop

ulat

ion

0 25% 50% 75% 100%Percent untested

2222

Percent Untested HIV+Percent Untested HIV+

8%

19%

2.5%

0.03%

MSM

IDU

HRH

LRH

Sub

-pop

ulat

ion

0 5% 10% 15% 20%Percent of untested who are HIV+

2323

Percent Untested Who Percent Untested Who Would Use a Home-Use Test Would Use a Home-Use Test KitKit

50%

40%

10%

5%

MSM

IDU

HRH

LRH

Sub

-pop

ulat

ion

0 10% 20% 30% 40% 50% 60%Percent of untested who would use a home use test

2424

Structure of the Model Structure of the Model 11

Size of the Sub-population

Number Untested

HIV + HIV -

% Untested

% HIV +

2525

Structure of the Model Structure of the Model 22

HIV +

Number Using Test

False Negative True Positive

Failed Test

% Failed

Sensitivity

% Using Test

2626

Model ResultsModel Results

2727

Absolute Numbers:Absolute Numbers:Model ResultsModel Results

Totals for all sub-populationsTotals for all sub-populations

Persons who would not otherwise be Persons who would not otherwise be testedtested

2828

Test SensitivityTest Sensitivity

105,279

100,520

5,565

527

5,286

True+, 99.5%

True+, 95.0%

Failed Tests

False-, 99.5%

False-, 95.0%

0 50,000 100,000 150,000Number of test results

Mean 5th to 95th %tile

Predicted Number of Test Results99.5% vs 95.0% Sensitivity

2929

Test SpecificityTest Specificity

6,774,414

6,706,370

358,120

33,997

102,040

True-, 99.5%

True-, 98.0%

Failed Tests

False+, 99.5%

False+, 98.0%

0 2,000,000 4,000,000 6,000,000 8,000,000 10,000,000Number of test results

Mean 5th to 95th %tile

Predicted Number of Test Results99.5% vs 98.0% Specificity

3030

Using Ratios to Explore Using Ratios to Explore TradeoffsTradeoffs

As a surrogate for quantitative As a surrogate for quantitative estimates of benefit and risk, ratios of estimates of benefit and risk, ratios of benefits to risks can provide some benefits to risks can provide some insight into public health tradeoffsinsight into public health tradeoffs

We will explore the number of We will explore the number of beneficial test outcomes to the number beneficial test outcomes to the number of adverse test outcomesof adverse test outcomes

There is no accepted threshold value There is no accepted threshold value for any of these ratiosfor any of these ratios

Professional judgment is required to Professional judgment is required to determine what ratio is acceptabledetermine what ratio is acceptable

3131

True Positive to False True Positive to False NegativeNegative

The ratio indicates how many newly The ratio indicates how many newly identified True Positive results are identified True Positive results are obtained for each False Negative resultobtained for each False Negative result

True Positive and False Negative results True Positive and False Negative results are related solely to the sensitivity of the are related solely to the sensitivity of the test and the number of HIV+ persons test and the number of HIV+ persons taking the testtaking the test

Failed tests for HIV+ persons are counted Failed tests for HIV+ persons are counted as False Negative results in this as False Negative results in this calculationcalculation

3232

Tradeoff between True Positive Test Tradeoff between True Positive Test Results and False Negative Test Results and False Negative Test ResultsResults

Sensitivity Ratio

99.5 20

97 13

95 10

92 7

90 6BPAC suggested that the lower bound

of the 95% CI should be no less than 95%010

2030

40R

atio

90 92 94 96 98 100

Sensitivity

5th Percentile/95th Percentile Mean Ratio

Ratio of True Positive to False Negativeand Failed HIV+ Results

3333

True Negative to False True Negative to False PositivePositive

The ratio indicates how many True The ratio indicates how many True Negative results are obtained for each Negative results are obtained for each False Positive False Positive

True Negative and False Positive results True Negative and False Positive results are related solely to the specificity of the are related solely to the specificity of the test and the number of HIV- persons test and the number of HIV- persons taking the testtaking the test

Failed tests for HIV- persons are counted Failed tests for HIV- persons are counted as False Positive results in this calculationas False Positive results in this calculation

3434

Tradeoff between True Negative Tradeoff between True Negative Test Results and False Positive Test Test Results and False Positive Test ResultsResults

1015

2025

3035

Rat

io

97.5 98 98.5 99 99.5

Specificity

5th Percentile/95th Percentile Mean Ratio

Ratio of True Negative to False Positiveand Failed HIV- Results

Specificity Ratio

99.5 20

99.0 18

98.5 16

98.0 15

97.5 13

3535

True Positive to False True Positive to False PositivePositive

True Positive and False Positive True Positive and False Positive results are related to both the results are related to both the sensitivity and the specificity of the sensitivity and the specificity of the test as well as the number of HIV+ test as well as the number of HIV+ and HIV- persons taking the testand HIV- persons taking the test

The ratio indicates how many True The ratio indicates how many True Positive results are obtained for Positive results are obtained for each False Positiveeach False Positive

3636

True Positive to False True Positive to False Positive RatioPositive Ratio

99.5 97.0 95.0 92.0 90.0

99.5 3.10 3.02 2.96 2.86 2.80

99.0 1.55 1.51 1.48 1.43 1.40

98.5 1.03 1.01 0.99 0.95 0.93

98.0 0.77 0.75 0.74 0.72 0.70

97.5 0.62 0.60 0.59 0.57 0.56

Specificity

Sensitivity

Under current assumptions, the ratio of true positive to false positive test results is more strongly related to specificity than to sensitivity.

3737

Current BPAC Recommended Current BPAC Recommended PerformancePerformance

The lower bound of the 95% The lower bound of the 95% confidence interval for sensitivity and confidence interval for sensitivity and specificity must be greater than or specificity must be greater than or equal to 95%equal to 95%

What is the mean sensitivity and What is the mean sensitivity and specificity needed to meet this?specificity needed to meet this?

Assume a 2% prevalence and a study Assume a 2% prevalence and a study population consisting of 100 positive population consisting of 100 positive subjects and 4,900 negative subjectssubjects and 4,900 negative subjects

3838

Test Test SensitivitySensitivity to Meet BPAC to Meet BPAC Recommended Lower Bound of the Recommended Lower Bound of the 95% CI95% CI

SuccessSuccess FailureFailure MeanMean Lower Lower BoundBound

9999 11 99%99% 97.0%97.0%9898 22 98%98% 95.2%95.2%9797 33 97%97% 93.6

%

3939

Test Test SensitivitySensitivity to Meet BPAC to Meet BPAC Recommended Lower Bound of the Recommended Lower Bound of the 95% CI95% CI

SuccessSuccess FailureFailure MeanMean Lower Lower BoundBound

46554655 245245 95.095.0 94.494.4

46804680 220220 95.595.5 94.994.946844684 216216 95.695.6 95.095.047044704 196196 96.096.0 95.495.4

40

Estimated ratio of true + to false + test results as a function of sensitivity and specificity of tests. Colors represent a temperature map. Assumes 111,000 HIV+ test users and 7,200,000 HIV- test users (mean values from the simulation). Specificity has a strong relationship with the ratio.

Ratio of True+ to False+ by Characteristics

Typical Currently Approved Rapid Tests

BPAC Minimum(see assumptions)

7.6

0.34

Low Sensitivity,Low Specificity

High Sensitivity,High Specificity

41

Ratio of True+ and False+ by Test Characteristics.

Estimated ratio of true + to false + test results as a function of sensitivity and specificity of tests. Colors represent a temperature map. Assumes 126,000 HIV+ test users and 6,000,000 HIV- test users (75 and 25 percentile values from the simulation). Specificity has a strong relationship with the ratio.

Typical Currently Approved Rapid Tests

BPAC Minimum(see assumptions)

More HIV+ and Fewer HIV- Users

10.4

0.47

4242

SummarySummary

We have presented a tool to explore the We have presented a tool to explore the public health benefits and risks of a home-public health benefits and risks of a home-use HIV test kit with different test use HIV test kit with different test characteristicscharacteristics

Tool provides estimates of the absolute Tool provides estimates of the absolute numbers of each type of test result and numbers of each type of test result and three ratios to assist in making informed three ratios to assist in making informed judgments about public health tradeoffsjudgments about public health tradeoffs

Each model result is important for Each model result is important for determining the overall public health determining the overall public health impactimpact

4343

Discussion 1Discussion 1

Several of the input parameters have Several of the input parameters have significant uncertainty because of a lack significant uncertainty because of a lack of dataof data

Our estimates that a large number of Our estimates that a large number of HIV- persons will use the test implies HIV- persons will use the test implies that high specificity will be required to that high specificity will be required to minimize False+ resultsminimize False+ results

Ratio of True + to False + is strongly Ratio of True + to False + is strongly related to specificity of the test under related to specificity of the test under current assumptionscurrent assumptions

Ratio of True+ to False- is related to Ratio of True+ to False- is related to sensitivity of the testsensitivity of the test

4444

Discussion 2Discussion 2

Assessing the public health impact of Assessing the public health impact of each test outcome requires a value each test outcome requires a value judgmentjudgment

We seek the committee’s input We seek the committee’s input regarding the appropriate balance regarding the appropriate balance between different test outcomes, e.g. between different test outcomes, e.g. how many False- test results would be how many False- test results would be acceptable for each newly identified acceptable for each newly identified True+ result?True+ result?

4545

Thank you!Thank you!

Richard Forshee, Ph.D.Richard Forshee, Ph.D.FDA/CBERFDA/CBER

Richard.Forshee@fda.hhs.govRichard.Forshee@fda.hhs.gov

4646

Reserve SlidesReserve Slides

4747

Sub-population SizeSub-population Size

3,840,000

1,580,000

16,200,000

167,200,000

MSM

IDU

HRH

LRH

Sub

-pop

ulat

ion

0 50,000,000 100,000,000 150,000,000 200,000,000Size of sub-population

Anderson et al. 2009

Brady 2008

National Survey of Family Growth, various criteria

National Survey of Family Growth

4848

Percent UntestedPercent Untested

30%

31%

65%

65%

MSM

IDU

HRH

LRH

Sub

-pop

ulat

ion

0 25% 50% 75% 100%Percent untested

Assumption

Heimer 2007

National Health Interview Survey

Jenness 2009

4949

Percent Untested HIV+Percent Untested HIV+

8%

19%

2.5%

0.03%

MSM

IDU

HRH

LRH

Sub

-pop

ulat

ion

0 5% 10% 15% 20%Percent of untested who are HIV+

Entering drug treatment, CDC website

Anderson et al. 2009

STD clinics, CDC website

Testing of military applicants, CDC website

5050

Percent Untested Who Percent Untested Who Would Use a Home-Use Test Would Use a Home-Use Test KitKit

50%

40%

10%

5%

MSM

IDU

HRH

LRH

Sub

-pop

ulat

ion

0 10% 20% 30% 40% 50% 60%Percent of untested who would use a home use test

Assumption

Assumption

Assumption

Assumption