Post on 13-Apr-2018
Worldwide Safety Strategy
Implementation and Examination of Measures of Disproportionality as Screening Tools on Longitudinal Claims DataAndrew BateSenior Director, Analytics Team LeadEpidemiology33rd Annual Midwest Biopharmacuetical Statistics Workshop, Muncie, May 25, 2010
Worldwide Safety Strategy
Potential Conflicts of Interest statement
• Employee and Stock Holder of Pfizer Inc
Worldwide Safety Strategy
Acknowledgements• Thanks to co-authors on technical report/manuscript
on which this presentation is based:– Patrick Ryan, GSK– David Madigan, Columbia University– Ivan Zorych, Columbia University
• All results presented here have been produced as part of the OMOP project http://omop.fnih.org– Thanks to other OMOP team members that have
contributed to the production of these results• Result interpretation is that of the author and not
necessarily that of the individuals named above nor other OMOP representatives
Worldwide Safety Strategy
Overview• Measures of disproportionality• Challenges on implementation on claims data• Results
– Simulated claims data– Real world claims data
• General comments and conclusions
Worldwide Safety Strategy
Background• Measures of disproportionality are metrics
used for screening spontaneous reports• Detect early suggestions of unexpected
indications of possible adverse drug reactions– Standard quantitative approach
• Metrics routinely calculated prospectively for all possible drug and adverse event combinations in data sets on a periodic basis
• Explore use on longitudinal claims data
Worldwide Safety Strategy
Measures of disproportionalityDiagnosis of interest Other diagnoses
Drug of interest a bOther drugs c d
Proportional Reporting ratio (PRR) = a / (a+b) ÷ c/ (c+d)]Reporting Odds Ratio (ROR) = a / (b) ÷ c/ (d)]Observed to Expected ratio (RR) = a / (a+b) ÷ a+c/ (a+b+c+d)]
Information Component (IC) and Empirical Bayes Geometric Mean (EBGM) are Bayesian implementations of Observed to Expected ratio (IC is logarithmised)Shrinkage towards null, IC is fully Bayesian, whereas prior for EBGM is derived emprically
Support measures whether confidence intervals of metrics; or as separate metric such as Chi squared test, routinely used in combination with point estimates described above
Worldwide Safety Strategy
Data mining on spontaneous report example
-2
-1
0
1
2
3
4
5
6
79:1 81:1 83:1 85:1 87:1 89:1 91:1 93:1 95:1
Time(year)
Captopril - Coughing
IC
Ref Bate1998EJCP
Worldwide Safety Strategy
Quantitative Signal Detection for ‘all by all’ problem• Null value: The drug - ADR combination is present as
often as expected based on anticipated occurrence of drug and outcome under an independence model– Search for positive metric values
• On spontaneous reports– Subset crossing arbitrary quantitative threshold (e. g.
lower 95% confidence interval of IC) are then clinically reviewed
• Not perfect performance in isolation; used as a filter- As well as binary classifier can be used as ranking
classifier
Worldwide Safety Strategy
Implementation challenges of disproportionality metrics on claims data• Differences between what expect and what observe after
drug exposure• Rather than adverse event reports
– No predefined definition of ’co-occurrence’– Diagnoses and tests– Drug utilization
• Question is how to best collapse a longitudinal record with often many unique, related or repeating diagnoses and drugs into a two by two table for metric calculation– “distinct patients,” “SRS,” and “Modified SRS.”
Worldwide Safety Strategy
Patient 1
Patient 2
Patient 3
A
A
A
B
A A
A B B
B
X1
O1 X6
Consider Incident, on-drug events. Three different counting approaches:
1 1
1 0
A
¬A
Consider 3 patients; 3 drugs (A,B & C); 2 conditions (x and o)
¬XX
“Distinct patients”
If patient 2 had an X event on A it would not change the numbers
time
C C
X4
1 0
0 2
A
¬A
¬XX
On-drug events, “SRS”
1 2
2 4
A
¬A
¬XX
On drug events + “non-drug” events + “non-event” drug eras“Modified SRS”
Reports: 1: A+X12: A+*3: *+X44: A+*5: B+*6: C+*7,8: BC+O19: *+X6
Reports: 1: A+X12,3: BC+O1
C C
Counting options:
http://omop.fnih.org
Worldwide Safety Strategy
• Incident v Prevalent cases• Choice of metric for 2 by 2 contingency table• Counting Scenario
– Three options as shown on previous slides• Many other options possible…
• Stratification: Sex, Age and Year• Windows
– Surveillance window– Drug Era– Condition Era
• How much do Implementation choices impact results?
Disproportionality method has many parameters
Worldwide Safety Strategy
Simulated data• Software application to create simulated datasets
– OMOP Observational Medical Dataset Simulator• Creates hypothetical persons with fictitious drug
exposure and condition occurrence, with known characteristics– Scenarios expected in real observational sources– File containing “truth”
• Procedure used to create simulated datasets for testing measure of disproportionality (and other methods)
• Simulated data set designed to mimic a simplified version of a Claims database with known properties
Worldwide Safety Strategy
Simulated data• Number of unique patient records in data set: 10,000,000• Number of drug exposures: 92,803,110• Number of condition occurrences: 316,686,137• Number of distinct drug- health outcome combinations: 5000
drugs and 4519 conditions = 22,595,000 drug-condition pairs– of which 519 conditions are ‘indications’ i.e. no implied causal
relationship with drug• Number of ‘true’ injected signals:
– Number of ‘risk’ signals 359729, Drug preventative effect 40,395, Total 400,124
• 1.59% of drug-condition pairs ‘true safety issues’ and 1.77% have a true relationship (positive or negative).
http://omop.fnih.org
Worldwide Safety Strategy
OSIM scores by occurrence type and counting scenario
Preliminary resultshttp://omop.fnih.org
Some parameter choices make large differenceothers less so.General concordance
Worldwide Safety Strategy
Score distributions on simulated data
0%10%20%30%40%50%60%70%80%90%100%
0
500
1000
1500
2000
2500
3000
3500
Freq
uenc
y
Score
Histogram:PRR scores for 10,000 pairs in OSIM
Frequency
Cumulative %
0%10%20%30%40%50%60%70%80%90%100%
0
1000
2000
3000
4000
5000
6000
7000
Freq
uenc
y
Score
Histogram:EBGM scores for 10,000 pairs in OSIM
Frequency
Cumulative %
0%10%20%30%40%50%60%70%80%90%100%
0
500
1000
1500
2000
2500
3000
3500
4000
Freq
uenc
y
Score
Histogram:IC scores for 10,000 pairs in OSIM
Preliminary resultshttp://omop.fnih.orgMarkedly more shrinkage for EBGM
Worldwide Safety Strategy
0.1
1
10
100
1000
10000
-4 -2 0 2 4 6 8 10
PRR
IC
OSIM score scatterplot:DP prevalent, SRS, 10,000 pair sample
Negative controls
True effects
0.1
1
10
100
1000
10000
-4 -2 0 2 4 6 8 10
EBG
M
IC
OSIM score scatterplot:DP prevalent, SRS, 10,000 pair sample
Negative controls
True effects
Preliminary results
True effects highlighted quantitatively, not so with negative controls
Worldwide Safety Strategy
Evaluating the performance of methods:Mean Average Precision
Original Values
Sorted Values
Drug Condition zi yi z(i) y(i) P(K) C1 5 1 9 1 1/1=1 C2 0 1 8 1 2/2=1
D1
C3 9 1 7 0 C1 8 1 5 1 3/4=0.75 C2 4 1 4 1 4/5=0.8
D2
C3 3 0 3 0 C1 1 0 2 0 C2 2 0 1 0
D3
C3 7 0 0 1 5/9=0.55 Total Score (1+1+0.75+0
.8+0.55)/5 =0.82
http://omop.fnih.org
Worldwide Safety Strategy
Simulateddata results
Preliminaryresults
http://omop.fnih.org
Prevalent better than incident‘Distinct patients’ performs worstA lack of power?
Bayesian measures more effectiveLimited variability of MAP scoresOverall low MAP scores, emphasizes importance of further filtering/ other approaches
Worldwide Safety Strategy
Thomson MSLR• Claims database• 1.5M lives• Pharmacy dispensing• Diagnoses from inpatient and outpatient
services
Worldwide Safety Strategy
Method performance on MSLR data
General concordance between metrics
http://omop.fnih.org
Preliminary results
Worldwide Safety Strategy
MSLR data
Preliminary results
0.1
1
10
100
1000
-3 -2 -1 0 1 2 3
PRR
IC
HOI score scatterplot by metricDP, Prevalent events, SRS, 30d SW
negative controlsACE-AngioedemaAmphotericin-Renal FailureAntibiotic-Liver FailureAntiepileptic-Aplastic AnemiaBenzodiazepine-Hip FractureBisphosphonate-GI Ulcer HospTricyclic-AMITypical Antipsych-AMIWarfarin-Bleeding
http://omop.fnih.org
True associations stand out quantitatively
Worldwide Safety Strategy
Comments• A strength of the metrics are their routine use in
spontaneous reporting for exploratory analysis– Facilitates performance comparison between use on
observational data and spontaneous reports– Additionally the approaches are simple and results
interpretation can be made transparently• Existence of many false positive findings can (to an
extent) be explained by inability of measures of disproportionality to control for all confounding in observational data
• Generalizabilty of performance limited as simple simulation and limited real data experience to date
Worldwide Safety Strategy
Conclusions• Statistical approaches can be implemented on entire claims data
set and generate results for all drugs and all outcomes• Discrimination in performance between the different metrics and
implementation choices• Demonstrated in both real world data and simulated data that
measures of disproportionality would highlight findings of interest– Buried in large numbers of false positive findings
• Measures of disproportionality might have some promise in initial screening of claims data
• Results will help benchmark the performance of other metrics• Further work is needed to determine whether such metrics might
be capable of highlighting genuine emerging issues early• Extensive consideration is needed into if, and how, metrics might
be built into overall adverse reaction identification processes