David M. Dilts PhD, MBA - National-Academies.org/media/Files/Activity Files/Disease... ·...
-
Upload
nguyentuyen -
Category
Documents
-
view
220 -
download
5
Transcript of David M. Dilts PhD, MBA - National-Academies.org/media/Files/Activity Files/Disease... ·...
Activating & Opening
Oncology Clinical Trials:
Process & Timing Analysis
David M. Dilts, PhD, MBADirector & Professor, Engineering Management Program, School of Engineering
Professor, Owen Graduate School of Management
Co-Director, Center for Management Research in Healthcare (cMRHc.org)
Alan B. Sandler, MDAssociate Professor of Medicine, Division of Hematology/Oncology
Medical Director, Thoracic Oncology Program
Director, Vanderbilt-Ingram Cancer Center Affiliates NetworkCo-Director, Center for Management Research in Healthcare (cMRHc.org)
Thank you to the study sites
Method
Part I: Process Mapping– Extensive visits at each site to document processes, loops and
decisions:
l Say…..: What participants say is done
l Should: What policies and procedures say should be done
l Do…...: What trial chart reviews shows was done
– Creation of process map
Part II: Process Timing– Identify calendar time for total process and major steps, and
potential influencers of the time
Part III: Accrual Data– Investigate actual accrual results of trials
• Dilts DM and Sandler AB (2006) “The Invisible Barriers to Opening Clinical Trials, J Clinical Oncology, 24(28): 4545-52
• Dilts DM et. al (2006) “Processes to Activate Phase III Clinical Trials in a Cooperative Oncology Group: The Case of
Cancer and Leukemia Group B,” J Clinical Oncology, 24(28): 4553-57.
• Dilts DM et. al (2008) “Development of Clinical Trials in a Cooperative Group Setting: The Eastern Cooperative
Group,” Clinical Cancer Research, 14(11):3427-33
Investigator-Initiated Trials High Level Process Map
Set-up Steps Trial Steps
Open Trial
(n=37) (n=15)
Comparison of Median Development TimeDevelopment vs. Operational Time by Phase*
* Sample: All ECOG Phase II and III studies activated between 1/2000-7/2006 and closed to accrual (n=52)
43.9%
56.1%
54.1%
45.9%
CIRB Review
111 days
Activation
7 days
Concept Development
193 days
Concept
Review
126 days
Concept Review
16 days
Concept Voting
2 days
Concept Approval
7 days
Study Team Teleconference
16 days
Grant Development
222 days
FDA Review
100 days
Regulatory Affairs Development
350 days
Forms / Database Development
434 days
CDE Compliance Review
240 days
Protocol Develoment
477 days
Protocol Review
277 days
Major Processing Activities
35 ft x 5 ft in 8pt
font
Process Flows for a Phase III
Cooperative Group Trial Creation
50 ft x 5 ft in 8pt
font
45 ft x 5 ft in 8pt
font
Process Steps for Phase III
Cooperative Group Trial Activation*
CRM (old)Steering Comm.
(new)
Process Steps* >481 >370 >300 >303
Working Steps >420 >317 >248 >244
Decision Points 61 42 52 59
Processing Loops 26 29 20 22
Stopping Points 13 16 11 12
* The process steps reported only show one loop in the process. Actual development frequently includes multiple loops
Level-0 Process Maps - CCC
P ro to c o l F in a liz a t io n
L O I a n d P ro to c o l D e v e lo p m e n t ( in c lu d in g In d u s tr y S p o n s o r
re v ie w )
P re l im in a r y B u d g e t A s s e s s m e n t
In fo r m e d
C o n s e n t D e v e lo p m e n t
F o rm a l
B u d g e t D e v e lo p m e n t
C o n tra c ts
N e g o t ia t io n s
R e g u la to ry
R e q u ire m e n ts
IR B
R e v ie w
F o rm s
D e v e lo p m e n t
F in a l C o n tra c t
S ig n in g
S tu d y
A c t iv a t io n
P R C R e v ie w
P O D R e v ie w
F D A
R e v ie w
37 ft x 3.5 ft in 8pt font
10
Process CountsComprehensive Cancer Centers
CCC 1* CCC 2 CCC 3 CCC 4
Process Steps 117 374 345 316
…Working Steps 64 292 272 263
…Decision Points 53 61 62 53
Processing Loops - 31 27 31
Stopping Points 19 21 11 18
11
Opening a Phase III Cooperative
Group Trial at a CCC
50 ft x 5 ft in 8pt
font
37’1’ x 3’6” ft in 8pt
font
CCC
45 ft x 5 ft in 8pt
font
12
Process Steps for Opening a
Phase III Cooperative Group Trial1
1. Representative cooperative group and Comprehensive Cancer Center2. Process steps reported only show one loop in the process. Actual development frequently includes multiple loops
Cooperative
Group
CTEP
/CIRBCCC Total
Process Steps >458 >216 >136 >810
…Working Steps >399 >179 >74 >652
…Decision Points 59 37 62 158
Potential Loops2 26 15 27 68
No. of Groups Involved 11 14 13 38
13
Reviews Required to Develop a
Cooperative Group Phase III Trial
CTCG CTEP CCC CCOP/Affiliates Others
Scientific Review Disease Site Committee Steering / CRM Protocol Review Feasibility Review Industry Sponsor
Executive Committee PRC Site Surveys
Protocol Reviews (2-4) CTEP Final
Data Management CRF Reviews (2-4) CDE Review
Database Review
Saftey / Ethics Informed Consent Local IRB Informed Consent CIRB
Regulatory Regulatory Review PMB Review FDA
RAB Review
Contracts / Grants Budget Industry Sponsor
Language
Study Start-up Start-up Review Start-up Review Start-up Review
Table 2. Types of Reviews Required to Develop a CTCG Trial: Categorized by Stakeholders
Abbreviations: CCC, Comprehensive Cancer Centers; CCOP, Community Clinical Oncology Program; CDE, Common Data Element; CIRB, Central Institutional Board Review, CRF, Case Report Form; CRM, Concept Review Meeting; CTCG, Clinical Trials Cooperative Group; CTEP, Cancer Therapy Evaluation Program; FDA, Food and Drug Administration; PMB, Pharmaceutical Management Branch, PRC, Protocol Review Committee; RAB, Regulatory Affairs Branch
14
Conclusion:
w Redundancy and Scope Creep
– Scope creep: when one group or organization
expands the scope of its authority or power
w Example: how many different scientific
reviews does a study require?
– Theory: more reviews = better study
– Practice: more reviews = slower opening
trials, no evidence of improvement
l Inspect in quality
15
How many of these steps add
value?
50 ft x 5 ft in 8pt
font
37’1’ x 3’6” ft in 8pt
font
CCC
45 ft x 5 ft in 8pt
font
16
Non-Value Added Statistics
at one CCC
Participants ---------Steps---------- Outcomes
Level 0 (Macro Process
Level)
Primary Other Value
Added
Non-
Value
Added
%
Value
Added
Decision
Points Decline
Accept
Or N/A
CCC 11 16 15 5 75% 13 2 1
Affiliates Main Office 3 - 7 3 70% 3 2 1
Affiliate Member Sites
(range of 3 sites)
3 1-8 6-15 1-5 75% 4-13 4-6 1
Sub-Process Level
IRB 7 3 12 26 32% 23 9 3
IRB Amendments 6 - 14 13 52% 13 6 3
SRC 4 - 13 11 54% 10 3 1
Regulatory and Clinical
Research Center
6 - 13 15 46% 7 1 1
17
Conclusion:
w There are far too many non-value
added (NVA) steps
w Value-Added steps
– Those that impact the quality, safety or
effectiveness of the study
w Note: if a study does not achieve
minimum accrual goals, then all steps
are NVA
18
Activation & Opening TimePhase III Cooperative Groups Trials
Group/CCC n Median Min Max
CALGB 13 784 537 1130
ECOG 28 808 435 1604
CCC 1 58 120 27 657
CCC 2 3 252 139 315
CCC 3 4 122 81 179
CCC 4 178 116 21 836w Notes:
– From initial conception at CTCG to activation; from receipt by CCC to opening– Time is calendar days, not work days– These are lower bounds because we only investigated survivors – Total time to open a study is the addition of Group time + CCC time
19
Total Time to Open a Phase III
Cooperative Group Study
Cooperative Group & CTEP/CIRB Processes
Median: 116 to 252 days*
Range: 21-836 days
Median: 784 to 808 days*
Range: 435-1604 days
Comprehensive Cancer Center
Processes
* Depending upon site, based on the Phase III trials studied
Protocol Finalization
LOI and Protocol Development (including Industry Sponsor
review)
Preliminary Budget Assessment
Informed Consent
Development
Formal Budget
Development
Contracts Negotiations
Regulatory Requirements
IRB Review
Forms Development
Final Contract
Signing
Study Activation
PRC Review
POD Review
FDA Review
Total Median Time from idea to opening~920 days (2.5 years)Range: 456 – 2440 days (1.25 - 6.7 yrs)
20
Development time for a phase III trial
from CTEP concept receipt to Activation
Figure 4. Development time for a phase III trial from CTEP concept receipt to Activation. Box ranges (H-Spread) indicate lower 25th and upper 75th percentile of the sample.
T-bars indicate the 95.0% confidence intervals by year. Dots indicate trial development time outside the bounds of the CI. Year indicates the year that the trial was activated.
Development time does not include the time involved to develop the concept prior to CTEP submission.
21
Phase III Activation Time
Time
Pre-Executive Committee Submission Development Time
Post-Executive Committee Submission Development Time
~600 Days (Median)
~800 Days (Median)
Sample: 26 Phase III CALGB and ECOG trials investigated in-depth
22
Where is the hold up?Total Time-Formal
Submission
Concept Review
Protocol Dev.
Protocol Review CIRB
CTEP Final Approval
Pre-Activation
295 26% 16% 13% 23%
330 20% 6% 2% 34% 19% 2%
407 22% 19% 16% 23% 2% 5%
424 36% 14% 11% 23% 0% 14%
465 21% 7% 46% 8% 7% 5%
483 59% 8% 12% 12% 3% 5%
490 22% 27% 14% 8% 0% 3%
491 53% 10% 5% 25% 0% 2%
524 26% 18% 29% 18% 1% 7%
551 14% 18% 23% 33% 1% 7%
560 25% 2% 10% 19% 15% 22%
580 6% 31% 16% 35% 3% 1%
589 42% 12% 15% 18% 2% 7%
596 34% 4% 6% 26% 22% 0%
614 4% 21% 29% 15% 1% 9%
668 15% 26% 35% 13% 2% 5%
758 3% 37% 32% 15% 2% 9%
829 10% 27% 36% 2% 20%
840 13% 24% 35% 16% 2% 7%
877 8% 30% 35% 2% 3%
885 35% 9% 11% 9% 1% 23%
930 31% 26% 6% 17% 8% 8%
931 12% 19% 11% 14% 7% 32%
1063 19% 52% 19% 8% 0% 1%
1248 45% 23% 10% 7% 5% 4%
1549 6% 35% 35% 18%
23
Conclusion:
w There is no Silver Bullet with regard to
time
w There is no single, clearly identified
bottleneck process
l Particularly, not IRB or FDA
w There are floating bottlenecks, or rate
limiting factors
24
Example Flow: E1301
Concept Review Days
Protocol Review Days
Total Days
Study Chair 49 122 171
Co-Operative Group 59 340 399
CTEP 98 184 282
CIRB n/a 123 123
Total 206 769 975
25
Reviews Requiring ResponseCalendar Days of Reviews and Group response by review type* for Phase III
Cooperative Group Studies (n=28 studies) activated from 2000 - 2005
* Reviews listed are only are partial list of required reviews. Other reviews including RAB, PMB, and CTSU are required but
were not available at the time of data collection. ** Group response time to industry cooperation not available
*** Recorded time for amendments only include study amendments prior to study activation
Review Time Group Response Time Total Time
Reviewer n median min max median min max median min max
Concept
CRM CTEP 14 60.0 15 104 71.5 1 368 126 16 411
CEP CTEP 4 48.0 19 66 35.5 22 84 83.5 41 150
Concept Re-review CTEP 3 6.0 1 6 17.0 1 56 23 7 57
Industry ** Industry 14 32.5 1 168
Protocol
PRC CTEP 33 32.0 5 69 32.0 1 188 63 8 230
Protocol Re-review CTEP 22 7.5 1 84 8.5 1 266 17 2 315
CIRB
CIRB CIRB 43 29.0 5 55 21.0 2 83 51 10 112
Re-review after CIRB CTEP 19 12.0 1 32 17.0 1 140 34 2 144
Amendment ***
Protocol Re-Review CTEP 2 9.0 1 17 5.5 5 6 14.5 6 23
CIRB CIRB 10 12.0 2 34 29.5 3 67 40.5 11 101
Re-review after CIRB CTEP 1 1.0 1 1 22.0 22 22 1 23 23
26
Conclusion:
w Only by working together can major
improvements be made
27
Discrete Event Simulation Model
of CALGB
28
High Level Process Flow for
Phase III Studies
29
Current
Simulation
Cooperative
Group
Improvement
CTEP
Improvements
CTEP and
Cooperative
Group
Improvements
Description As-Is performanceImproved Selective Study
Crit eria
Improved Review
Performance
Simult aneous
Improvments
n* 99 103 108 121
mean 837.29 791.51 653.01 252.78
min 222 209.07 214.93 214.35
max 3857.89 3576.71 4292.98 311.96
st. dev 760.75 729.3 670.63 24.14
Timing
Distribution
* Simulation period defined over a period of 5 years (1825 Calendar Days)
* Note: Axes on the Timing Distribution Graphs are different
Simulation Results of Working Together
30
Actual Accrual Per Trial RangesComprehensive Cancer Centers1
Accrual Per Trial
CCC 1 CCC 2 CCC 3 CCC 4 Total
N 148 323 104 323 898
0 20.9% 26.9% 26.9% 34.4% 28.6%
1-4 33.0% 32.3% 30.3% 31.3% 30.8%
5-10 19.3% 16.1% 22.7% 18.0% 18.6%
11-15 11.0% 7.3% 8.4% 4.3% 7.7%
16-20 3.7% 3.7% 3.4% 5.3% 4.1%
>20 12.4% 15.0% 7.6% 6.8% 10.1%
1Excludes pediatric studies
31
Accrual at CCCs: Cooperative v. non-Cooperative Group Trials
Cooperative Group Trials CCC 1 CCC 2 CCC 3 CCC 4Total/
Average
N 37 166 61 130 394
0 27.0% 38.6% 29.5% 46.9% 38.8%
1 to 4 37.8% 38.6% 39.3% 38.5% 38.6%
5 or more 35.1% 22.9% 31.1% 14.6% 22.6%
Non-Cooperative Group Trials CCC 1 CCC 2 CCC 3 CCC 4
Total/ Average
N 111 157 43 193 504
0 18.9% 14.6% 23.3% 25.9% 20.6%
1 to 4 30.6% 22.9% 9.3% 26.4% 24.8%
5 or more 50.5% 62.4% 67.4% 47.7% 54.6%
ASCO Poster 2008
32
All therapeutic, non-pediatric, Phase I, I/II, II, and III oncology trials evaluated by
CTEP that with submitted protocol opened to patient accrual, and subsequently
closed to accruals between January 1, 2000 and December 31, 2007 in the United States. Starting date is initial LOI/concept receipt to CTEP, end date is trial open for
accrual.
33
Final Accrual Performance* of CTEP-
Sponsored Phase III Trials
Opened and Closed between 2000 – 2007 (n=61)
•Accrual Performance is calculated based on the percentage of final accruals of studies completely closed to accrual compared to the projected minimum accrual goals as stated at the time of activation
•Pediatric studies are excluded from the sample
63.9% do not achieve minimum projected accruals at closure
5,776 patients accrued to these studies
49.2% do not achieve 25% of minimum projected accruals at
closure
36.1% meet or exceed minimum
projected accruals
34
Impact of Development Time on Accrual
CTEP Therapeutic Studies
All therapeutic, non-pediatric, Phase I, I/II, II, and III oncology trials evaluated by CTEP that began trial development with concept
submission, opened to patient accrual, and subsequently closed to accruals between January 1, 2000 and December 31, 2007 in the
United States. Starting date is initial LOI/concept receipt to CTEP, end date is trial open for accrual. N=553
35
“What one postpones, one actually
abandons”--Peter Drucker “The Effective Executive”, p. 110
36
Recommendations
w Immediate (Quick-Fix)
w Mid-range
w Long term
37
Immediate Recommendations
wImmediately start collecting & analyzing data
wSay what you mean & mean what you say
w“Just Say No”– Eliminate “entitlement culture”
wSet expiry dates:– If a study has not opened in a year, why continue
it?
wStop tweaking– “Two strikes and you’re out”
38
Analyzing & Collecting Data
w Analyzing Existing Data
– What are the characteristics of the studies with low accruals
and/or long development times?
– How many studies have not achieved their original
timelines?
– Plot time-in & time-out for all studies
w Collecting & analyzing additional data
– Identify & count reasons for:
l loops (revisions), study closures, or any specific reasons
for rework (runs chart)
l What “stips” must be done the PI & which can be done
by others
39
Say what you mean &
mean what you say
w Use standard and consistent terminology
– Approval versus Final-Approval versus Final-Final Approval
w Use schedules that are schedules, priorities that are
priorities and “no’s” that are “no’s”– If 50% of studies are “Priority 1”, then all studies are
“Priority 1”
– If the only penalty for being late is getting more time, then
why do something on time?
– If a study can be rejected at concept, protocol, and group but
still be opened, does “no” mean “no”?l 14 disapproved concepts had protocols created, 11 resulted in active
protocols
l 17 of withdrawn concepts had protocols created, 8 of which resulted
in active protocols
40
Just say “No”Controlling the Inflow
w Study Period 2000-2005 Averages/year
w Inflows ~43.6 LOI’s/yr
– (70.7% eventually accepted)
w Work-In-Process ~75 concepts or
protocols
w Outflow 18 protocols /yr
w If no new studies are accepted, it would
take 4.16 years to clear the queue
41
Besides too much work,
what do more concepts do?
wAttention Deficit
“What information consumes is rather
obvious: it consumes the attention of its
recipients. Hence a wealth of information
creates a poverty of attention, and a need to
allocate that attention efficiently among the
overabundance of information sources that
might consume it.”
Herbert Simon,
Nobel Laureate and Turing Prize Winner
42
“A ‘No’ uttered from deepest conviction is
better and greater than a ‘Yes’ merely
uttered to please, or what is worse, to avoid
trouble.”
--- Mahatma Gandhi
43
Medium Term
Recommendations
w Eliminate redundant, non-value added steps in the
entire process
– Cooperative Groups, CTEP, CIRB, & Comprehensive
Cancer Centers
w Benchmark other NIH Institutes &
Pharmaceutical firms
w Develop and utilize standards
– i.e., vary in critical scientific issues, not in
administrative processes
w Triage concepts using scientific merit and
operational complexity
44
Triaging Concepts:A Technique for Determining Entrance
Scientific
Merit
High
Low
High Low
Operational
Complexity
Fill
45
“An analysis of executive contributions
comes up with an embarrassing richness of
important tasks; any analysis of executives’
time discloses an embarrassing scarcity of
time available for work that really
contributes.”--Peter Drucker “The Effective Executive”, p. 100
46
Long Term Recommendations
w Change culture:
– From one of a feeling of entitlement
– Build a Mass Customization Process instead
of an Engineered-To-Order Process
– Start with a clean sheet of paper
w Use Focused Phase III Teams
47
Use Focused Phase III teams
w Create a dedicate team to rapidly develop Phase III concepts
w Team should include:– Co-Operative Group members
l Local Study Champion
l Statistician
l Protocol Specialist
l Regulatory Affairs
l Grants & Contracts
l Forms Developer
– CTEP members
l Statistician
l CDE
l Former CIRB Member
– Pharmaceutical Representative
w Goal: Rapidly, cooperatively, and with high quality, activate a
Phase III protocol in 90 days
48
“No task is completed until it has become
part of organizational action and behavior”
--Peter Drucker “The Effective Executive”, p. 109
“Unless a decision has ‘degenerated into work’ it is
not a decision; it is at best a good intention.”-- Peter Drucker “The Effective Executive”, p. 114
49
Thank you
www.cMRHc.org
50
Additional Potential Rules for
Identifying Priorities
w Pick for the future, not the past
w Focus on opportunities, not problems
w Choose your own portfolio, not what
others do
w Aim high, not safe & easy
51
Development TimeNew-To-The-World Goods or Services
41.7
24
56.4
144
0
20
40
60
80
100
120
140
160
1995 2004
Mo
nth
s
New-to-the-world products or services Pharmaceuticals
▼42%
▲155%
• Adams M, Boike D. Product Development and Management Association Foundation's Comparative Performance
Assessment Study: PDMA Foundation; 2004;
• Slater EE. Today's FDA. N Engl J Med 2005;352:293-7.
52
CTEP Acceptance Success Rates
by Trial Phase
Phase I I/II II III Other Pilot Total
Concepts n 1421 490 2649 513 25 52 5150
Gone Forward 33.4% 29.8% 35.3% 55.0% 28.0% 48.1% 36.3%
In Review 2.4% 3.5% 1.4% 1.9% 8.0% 3.8% 2.0%
Withdrawn / Replaced 13.7% 6.3% 13.6% 21.2% 24.0% 11.5% 13.7%
Disapproved 50.5% 60.4% 49.6% 21.6% 40.0% 36.5% 47.9%
Protocols n 388 166 1,031 192 150 72 1,999
Gone Forward 88.9% 80.7% 86.9% 83.3% 74.7% 81.9% 85.3%
In Review 4.6% 6.0% 4.3% 8.3% 8.7% 6.9% 5.3%
Withdrawn / Replaced 5.9% 9.6% 6.8% 6.3% 10.0% 6.9% 7.1%
Disapproved 0.5% 3.6% 2.0% 2.1% 6.7% 4.2% 2.3%
For all protocols received by CTEP, 1/2000 to 12/2007
53
Time From Concept Receipt to ActivationPhase III Therapeutic Studies activated through CTEP 1/2000 – 6/2007†
by Cooperative Group
Group Number
Median
Days**
Min
Days
Max
Days
CALGB 21 532 229 1526
ECOG 27 635 274 1532
SWOG 27 597 342 1706
COG 34 719 203 1908
GOG 9 402 298 665
NSABP 13 691 317 1655
RTOG 16 432 264 989
Other* 18 583 372 1697
Total/ Overall 165 594 203 1908
*Other: ACOSOG, IBCSG, NCCTG, NCIC, NCIMB, TX035
† these dates do not include the days for concept development
& approval at the cooperative group
** Concept approval time represented 8% to 39% of the days,
depending upon Group
54
YearSample Size
(trials)
Median Development Time (d)
Minimum Development Time (d)
Maximum Development Time (d)
IQR
2000 27 533 203 1114 402 - 661
2001 15 563 312 1706 427 - 943
2002 18 541 370 1110 464 - 724
2003 21 587 321 1908 415 - 889
2004 24 655 229 1423 427 - 854
2005 18 496 264 1142 369 - 659
2006 22 678 329 1655 542 - 896
2007 22 832 495 1776 680 - 1039
Total 167 602 203 1908 454 - 861
Table 2: Phase III CTEP Sponsors Clinical Trials by Activation Year. Sample size includes all CTEP-sponsored phase III clinical trials
activated from January 2000 to December 2007. Development time is defined from the receipt of the concept by CTEP to the time the
trial is activated for patient accruals. refers to the 25th and 75th percentile of the trials within each year and total years.
Development time for a phase III trial
from CTEP concept receipt to Activation
55
One Critical Step:
Executive Committee Approval
“[Once you select a clinical trial to pursue] it is like
pushing a boulder off a cliff…it is impossible to stop”
56
Besides too much work,
what do more concepts do?
wReduction in Attractiveness of Choice
“In the last few years, there has been an increase in
literature that indicates that people often have real
difficulty making choices, especially when those choices
involve tradeoffs (Luce, 1998; Luce, Payne, & Bettman,
1999). Moreover, the difficulties increase as the number of
attractive options increase (Iyengar & Lepper, 2000;
Schwartz, 2004). Everything suffers from comparison
(Brenner, Rottenstreich, & Sood, 1999). Multiple
attractive options may even lead to paralysis.”
Keys & Schwartz, “’Leaky’ Rationality”, Persp. On Psy. Sci
2(2):162-180, 2007