Worldwide Safety Strategy

Implementation and Examination of Measures of Disproportionality as Screening Tools on Longitudinal Claims DataAndrew BateSenior Director, Analytics Team LeadEpidemiology33rd Annual Midwest Biopharmacuetical Statistics Workshop, Muncie, May 25, 2010

Worldwide Safety Strategy

Potential Conflicts of Interest statement

• Employee and Stock Holder of Pfizer Inc

Worldwide Safety Strategy

Acknowledgements• Thanks to co-authors on technical report/manuscript

on which this presentation is based:– Patrick Ryan, GSK– David Madigan, Columbia University– Ivan Zorych, Columbia University

• All results presented here have been produced as part of the OMOP project http://omop.fnih.org– Thanks to other OMOP team members that have

contributed to the production of these results• Result interpretation is that of the author and not

necessarily that of the individuals named above nor other OMOP representatives

Worldwide Safety Strategy

Overview• Measures of disproportionality• Challenges on implementation on claims data• Results

– Simulated claims data– Real world claims data

• General comments and conclusions

Worldwide Safety Strategy

Background• Measures of disproportionality are metrics

used for screening spontaneous reports• Detect early suggestions of unexpected

indications of possible adverse drug reactions– Standard quantitative approach

• Metrics routinely calculated prospectively for all possible drug and adverse event combinations in data sets on a periodic basis

• Explore use on longitudinal claims data

Worldwide Safety Strategy

Measures of disproportionalityDiagnosis of interest Other diagnoses

Drug of interest a bOther drugs c d

Proportional Reporting ratio (PRR) = a / (a+b) ÷ c/ (c+d)]Reporting Odds Ratio (ROR) = a / (b) ÷ c/ (d)]Observed to Expected ratio (RR) = a / (a+b) ÷ a+c/ (a+b+c+d)]

Information Component (IC) and Empirical Bayes Geometric Mean (EBGM) are Bayesian implementations of Observed to Expected ratio (IC is logarithmised)Shrinkage towards null, IC is fully Bayesian, whereas prior for EBGM is derived emprically

Support measures whether confidence intervals of metrics; or as separate metric such as Chi squared test, routinely used in combination with point estimates described above

Worldwide Safety Strategy

Data mining on spontaneous report example

-2

-1

0

1

2

3

4

5

6

79:1 81:1 83:1 85:1 87:1 89:1 91:1 93:1 95:1

Time(year)

Captopril - Coughing

IC

Ref Bate1998EJCP

Worldwide Safety Strategy

Quantitative Signal Detection for ‘all by all’ problem• Null value: The drug - ADR combination is present as

often as expected based on anticipated occurrence of drug and outcome under an independence model– Search for positive metric values

• On spontaneous reports– Subset crossing arbitrary quantitative threshold (e. g.

lower 95% confidence interval of IC) are then clinically reviewed

• Not perfect performance in isolation; used as a filter- As well as binary classifier can be used as ranking

classifier

Worldwide Safety Strategy

Implementation challenges of disproportionality metrics on claims data• Differences between what expect and what observe after

drug exposure• Rather than adverse event reports

– No predefined definition of ’co-occurrence’– Diagnoses and tests– Drug utilization

• Question is how to best collapse a longitudinal record with often many unique, related or repeating diagnoses and drugs into a two by two table for metric calculation– “distinct patients,” “SRS,” and “Modified SRS.”

Worldwide Safety Strategy

Patient 1

Patient 2

Patient 3

A

A

A

B

A A

A B B

B

X1

O1 X6

Consider Incident, on-drug events. Three different counting approaches:

1 1

1 0

A

¬A

Consider 3 patients; 3 drugs (A,B & C); 2 conditions (x and o)

¬XX

“Distinct patients”

If patient 2 had an X event on A it would not change the numbers

time

C C

X4

1 0

0 2

A

¬A

¬XX

On-drug events, “SRS”

1 2

2 4

A

¬A

¬XX

On drug events + “non-drug” events + “non-event” drug eras“Modified SRS”

Reports: 1: A+X12: A+*3: *+X44: A+*5: B+*6: C+*7,8: BC+O19: *+X6

Reports: 1: A+X12,3: BC+O1

C C

Counting options:

http://omop.fnih.org

Worldwide Safety Strategy

• Incident v Prevalent cases• Choice of metric for 2 by 2 contingency table• Counting Scenario

– Three options as shown on previous slides• Many other options possible…

• Stratification: Sex, Age and Year• Windows

– Surveillance window– Drug Era– Condition Era

• How much do Implementation choices impact results?

Disproportionality method has many parameters

Worldwide Safety Strategy

Simulated data• Software application to create simulated datasets

– OMOP Observational Medical Dataset Simulator• Creates hypothetical persons with fictitious drug

exposure and condition occurrence, with known characteristics– Scenarios expected in real observational sources– File containing “truth”

• Procedure used to create simulated datasets for testing measure of disproportionality (and other methods)

• Simulated data set designed to mimic a simplified version of a Claims database with known properties

Worldwide Safety Strategy

Simulated data• Number of unique patient records in data set: 10,000,000• Number of drug exposures: 92,803,110• Number of condition occurrences: 316,686,137• Number of distinct drug- health outcome combinations: 5000

drugs and 4519 conditions = 22,595,000 drug-condition pairs– of which 519 conditions are ‘indications’ i.e. no implied causal

relationship with drug• Number of ‘true’ injected signals:

– Number of ‘risk’ signals 359729, Drug preventative effect 40,395, Total 400,124

• 1.59% of drug-condition pairs ‘true safety issues’ and 1.77% have a true relationship (positive or negative).

http://omop.fnih.org

Worldwide Safety Strategy

OSIM scores by occurrence type and counting scenario

Preliminary resultshttp://omop.fnih.org

Some parameter choices make large differenceothers less so.General concordance

Worldwide Safety Strategy

Score distributions on simulated data

0%10%20%30%40%50%60%70%80%90%100%

0

500

1000

1500

2000

2500

3000

3500

Freq

uenc

y

Score

Histogram:PRR scores for 10,000 pairs in OSIM

Frequency

Cumulative %

0%10%20%30%40%50%60%70%80%90%100%

0

1000

2000

3000

4000

5000

6000

7000

Freq

uenc

y

Score

Histogram:EBGM scores for 10,000 pairs in OSIM

Frequency

Cumulative %

0%10%20%30%40%50%60%70%80%90%100%

0

500

1000

1500

2000

2500

3000

3500

4000

Freq

uenc

y

Score

Histogram:IC scores for 10,000 pairs in OSIM

Preliminary resultshttp://omop.fnih.orgMarkedly more shrinkage for EBGM

Worldwide Safety Strategy

0.1

1

10

100

1000

10000

-4 -2 0 2 4 6 8 10

PRR

IC

OSIM score scatterplot:DP prevalent, SRS, 10,000 pair sample

Negative controls

True effects

0.1

1

10

100

1000

10000

-4 -2 0 2 4 6 8 10

EBG

M

IC

OSIM score scatterplot:DP prevalent, SRS, 10,000 pair sample

Negative controls

True effects

Preliminary results

True effects highlighted quantitatively, not so with negative controls

Worldwide Safety Strategy

Evaluating the performance of methods:Mean Average Precision

Original Values

Sorted Values

Drug Condition zi yi z(i) y(i) P(K) C1 5 1 9 1 1/1=1 C2 0 1 8 1 2/2=1

D1

C3 9 1 7 0 C1 8 1 5 1 3/4=0.75 C2 4 1 4 1 4/5=0.8

D2

C3 3 0 3 0 C1 1 0 2 0 C2 2 0 1 0

D3

C3 7 0 0 1 5/9=0.55 Total Score (1+1+0.75+0

.8+0.55)/5 =0.82

http://omop.fnih.org

Worldwide Safety Strategy

Simulateddata results

Preliminaryresults

http://omop.fnih.org

Prevalent better than incident‘Distinct patients’ performs worstA lack of power?

Bayesian measures more effectiveLimited variability of MAP scoresOverall low MAP scores, emphasizes importance of further filtering/ other approaches

Worldwide Safety Strategy

Thomson MSLR• Claims database• 1.5M lives• Pharmacy dispensing• Diagnoses from inpatient and outpatient

services

Worldwide Safety Strategy

Method performance on MSLR data

General concordance between metrics

http://omop.fnih.org

Preliminary results

Worldwide Safety Strategy

MSLR data

Preliminary results

0.1

1

10

100

1000

-3 -2 -1 0 1 2 3

PRR

IC

HOI score scatterplot by metricDP, Prevalent events, SRS, 30d SW

negative controlsACE-AngioedemaAmphotericin-Renal FailureAntibiotic-Liver FailureAntiepileptic-Aplastic AnemiaBenzodiazepine-Hip FractureBisphosphonate-GI Ulcer HospTricyclic-AMITypical Antipsych-AMIWarfarin-Bleeding

http://omop.fnih.org

True associations stand out quantitatively

Worldwide Safety Strategy

Comments• A strength of the metrics are their routine use in

spontaneous reporting for exploratory analysis– Facilitates performance comparison between use on

observational data and spontaneous reports– Additionally the approaches are simple and results

interpretation can be made transparently• Existence of many false positive findings can (to an

extent) be explained by inability of measures of disproportionality to control for all confounding in observational data

• Generalizabilty of performance limited as simple simulation and limited real data experience to date

Worldwide Safety Strategy

Conclusions• Statistical approaches can be implemented on entire claims data

set and generate results for all drugs and all outcomes• Discrimination in performance between the different metrics and

implementation choices• Demonstrated in both real world data and simulated data that

measures of disproportionality would highlight findings of interest– Buried in large numbers of false positive findings

• Measures of disproportionality might have some promise in initial screening of claims data

• Results will help benchmark the performance of other metrics• Further work is needed to determine whether such metrics might

be capable of highlighting genuine emerging issues early• Extensive consideration is needed into if, and how, metrics might

be built into overall adverse reaction identification processes