Using Metrics to improve study performance
-
Upload
bioclinica -
Category
Health & Medicine
-
view
318 -
download
2
description
Transcript of Using Metrics to improve study performance
© 2013 BioClinica, Inc. – Proprietary and Confidential
Global clinical trial solutions. Real-world results.
© 2013 BioClinica, Inc. – Proprietary and Confidential
Brian Bialkowski, PhD
Welcome to today’s BioClinica WebinarUsing Metrics to Improve Study Quality
See this content presented in an on-demand webinar @
bioclinica.com/resources/webinars
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 2
Defining Study Quality
A “quality” study is one that yields what the sponsor needs, when they need it.
• Enough Sites• Enough Subjects• Enough Study Drug• Enough Quality Data
What They Need
• Key Study Milestones• Budget• Shared Resources Schedules
When They Need It
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 3
Am I on Time?
Defining Study Quality
Do I Have What I Need?
Am I under Budget?
Measure Assess and Take Action
SiteABC357ABC579ABC123ABC456ABC789ABC246ABC468ABC135
???????
???????
???????
© 2013 BioClinica, Inc. – Proprietary and Confidential 4© 2013 BioClinica, Inc. – Proprietary and Confidential 4
• First Site• First Subject In• FPFV• Enroll Complete• LPLV• Data Lock
Milestones
• Site Statuses• Screening• Screen Failures• Enrollment• Dropouts
Enrollment
• Protocol Deviations• SAEs• Monitoring Visits• Out-of-Window
Visits
Site Monitoring
• Status• Expiration Dates
Documents
• Data Collection• SDV• Data Lock• Queries• Cycle Times
Data Management
• Shipments• Shipment Contents• Kit Statuses• Dosing Visits
Supply Chain
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 5
Selecting and Using Metrics
• Understand types of metrics and their uses• Know your audience and their key questions• Sequence metrics and action
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 6
What Is a Metric?
A metric is, simply put, a measurement.
What
WhyHow
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 7
How We Measure It
(Subject)
Detail and Context
Site
(Individual Records)
Aggregation
Country
Program
Study
RegionRates
Variance
KPI
Focus Metric Type
Summary and Judgment
2012 2014Trending and Forecasting
2013
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 8
Sample Exercise
• 10 Sites with enrollment data• Each site has an enrollment target.• All sites began enrollment in January 2012.• All were expected to complete enrollment in
March 2013.• Track # of subjects enrolled each month at the
sites.
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 9
Simple Aggregation
• Clear message.• Suitable for description.• Clear indicator of scale.• No built-in interpretation of the results.
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 10
Normalized Measurements (Rates and Variance
• Normalize measurements across time and scale.• Other normalized metrics: SAEs per visit; Protocol deviations per subject; Queries per 100
data items• Abstracting from raw data adds context for evaluatiion. • Base for trending (“When did things go wrong?”) and forecasting (“Where is my study
going?”)
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 11
KPI (Visual Indicator)
• Most easily digestible type of metric• No context on actual enrollment or target numbers; color values set using thresholds for
performance against targets.• Many options for indicators—traffic lights, arrows, gauges, etc.• Drives decision to take action, but not a tool for the action itself.
Green: >95%
Yellow: 80%-95%
Red: <80%
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 12
How We Measure It
(Subject)
Detail and Context
Site
(Individual Records)
Aggregation
Country
Program
Study
RegionRates
Variance
KPI
Focus Metric Type
Summary and Judgment
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 13
What Could You Measure?
• First Site• First Subject In• FPFV• Enroll Complete• LPLV• Data Lock
Milestones
• Site Statuses• Screening• Screen Failures• Enrollment• Dropouts
Enrollment
• Protocol Deviations• SAEs• Monitoring Visits• Out-of-Window
Visits
Site Monitoring
• Status• Expiration Dates
Documents
• Data Collection• SDV• Data Lock• Queries• Cycle Times
Data Management
• Shipments• Shipment Contents• Kit Statuses• Dosing Visits
Supply Chain
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 14
My data management team...
“Which forms are generating the most queries?”
Audience, Questions, and Outcome
My executive…“How many sites are below their enrollment target?”
My supply chain manager…
“Does my depot have enough study drug for the next 6 weeks of visits?”
And then…
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 15
There’s Never Just One Question…
• Am I enrolling enough subjects today...• And am I on pace to hit my enrollment targets (# and date) in
12 months…• And did I build enough screen fails and early terminations
into my budget…• And are some sites losing more subjects than others…• And do I have enough capacity to make up enrollment
shortfalls at other sites…• And are treatment groups divided proportionately across all
my sites?
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 16
Sequence Your Use of Metrics
• Which questions do you need to ask first?• What information do you need to answer them?• What questions do they lead to?
Question 1(Metric 1)
Question 2 (Metric 2)
Question 3 (Metric 3)
Question 5(Metric 5)
Question 4 (Metric 4)
Assess Analyze Action
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 17
Picking the Right Metrics for Your Study
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 18
Early Study
% Enroll vs. Target to Date
% Active Sites vs. Target to Date
CRF Entry Lag
Site Statuses
Screen Fail Rate Document Collection
Screen Fail ReasonsDiscontinued Rate
Discontinued Reasons
Site Milestone Targets vs Actuals
Study
Site
Assess Analyze Action
Query Rates Query Counts by Form
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 19
Early Study Dashboard
1001 1002 1003 1004 1005 1006-30%-25%-20%-15%-10%
-5%0%5%
10%15%
Enrollment vs Target
1001
1002
1003
1004
1005
1006
0% 5% 10% 15% 20% 25% 30% 35%
Screen Fail, Discontinue Rates
Discontinued Screen Fail
Target to DateActiveOpen
Start-UpEvaluationIdentified
0 1 2 3 4 5 6 7
Site Statuses
1001
1002
1003
1004
1005
1006
0 5 10 15 20 25 30
Queries/100 Data Items
Current Enrollment: 85 Target to Date: 95Current # Sites: 6 Target to Date: 5.8
1001
1002
1003
1004
1005
1006
0 2 4 6 8 10 12 14
CRF Entry Lag (Days)
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 20
Mid-Study
% Enroll vs. Target to Date
# Out-of-Window Subjects
Shipment History
Screen Fail Rate
Discontinued Reasons
Shipment Damaged/Lost
Rates
Discontinued Rate
Query Cycle Times
Expired/Destroyed Study Drug
Screen Fail Reasons Study
Site
Assess Analyze Action
% CRF SDV/Locked
Late Action Items
Inventory (Kit Statuses)
Query Cycle TimesQuery Rates
CRF Status
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 21
Mid-Study Dashboard
1001 1002 1003 1004 1005 1006-30%-25%-20%-15%-10%
-5%0%5%
10%15%
Enrollment vs Target
1001 1002 1003 1004 1005 10060
2
4
6
8
10
Out of Window Subjects
1001
1002
1003
1004
1005
1006
0% 10% 20% 30% 40% 50% 60% 70%
% Collected CRFs SDV and Locked
Locked Verified
1001
1002
1003
1004
1005
1006
0 5 10 15 20 25 30
Queries/100 Data Items
Current Enrollment: 85 Target to Date: 95 Late Action Items: 24
1001 1002 1003 1004 1005 100605
10152025303540
Kit Statuses
Available ExpiredQuarantined
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 22
Late Study
Study MilestoneTargets vs Actuals
% Data Locked
Queries Open/Answered >7
days
Open Action Items
CRF Statuses
Study
Site
Assess Analyze/Action
Query Statuses
Upcoming Site Milestones
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 23
Late Study Dashboard
1001
1002
1003
1004
1005
1006
0 20 40 60 80 100 120
Open/Answered Queries
Answered > 7 days Open > 7 Days
1500
1200
25000
Query Statuses
Open Answered Accepted
1001
1002
1003
1004
1005
1006
0 5000 10000 150002000025000 30000
CRF Statuses
Locked Verified ILB Entered
Milestone Baseline Revised Actual
Enrollment Complete 04-Feb-2014 08-Mar-2014 12-Mar-2014
LPLV 15-May-2014 21-May-2014
Data Lock 15-Jun-2014 1-Jul-2014
Final Report 1-Jul-2014 15-Jul-2014
Study Close 1-Jul-2014 15-Jul-2014
% Data Locked: 80%
Study Milestones Pending Site Milestones
Site Milestone Baseline Revised
1004 LPLV 24-Apr-2014 18-May-2014
1002 LPLV 15-May-2014 21-May-2014
1003 LPLV 1-May-2014 14-May-2014
1004 Last CRF 23-May-2014 30-May-2014
1005 Last CRF 24-May-2014 4-Jun-2014
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 24
Data Management Metrics
% Expected Pages Received
% Forms Locked
Cycle Times (Open, Answered, Accepted)
Data Entry Lag
Query Counts by Form
CRF Page Status
Queries Open/Answered >
7 days
Study
Site
Assess Analyze Action
Query Counts by Status
% SDV
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 25
Data Management Dashboard
Visit 1 Subject Questionnaire Visit InfoVisit 2 Pain Questionnaire
Visit 1 PAGI-Sym QuestionnaireVisit 3 Pain Questionnaire
Visit 3 McGill Pain QuestionairreVisit 4 Pain Questionnaire
Visit 4 McGill Pain Questionnaire
0 100 200 300 400 500 600 700
Queries by Form
% Expected CRFs received 85%
% SDV 55%
% Locked 25%
2400
1200
8000
Query Statuses
Open Answered Accepted
100110021003100410051006
0 2 4 6 8 10 12 14
CRF Entry Lag (Days)
100110021003100410051006
05000
1000015000
2000025000
30000
CRF Statuses
LockedVerifiedILBEntered
1001
1002
1003
1004
1005
1006
0 20 40 60 80 100 120
Open/Answered Queries
Answered > 7 daysOpen > 7 Days
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 26
Planning and Implementation
• Agree on most important questions.Define• Ensure your system(s) can supply the necessary
data.Validate• Build a model/system that will generate your
metrics.Automate• Combine metrics with action, and execute on
that plan.Value• Share metrics with employees; build them into
processes.Share
• Understand types of metrics• Know your key questions• Sequence metrics and actions they’ll
trigger
Other steps…• Work with the data you have• Range of technical options
© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential
Thank You!
Brian [email protected]
See this content presented in an on-demand webinar @
bioclinica.com/resources/webinars