Clinical Assessment – Part II William Sacks, PhD, MD
description
Transcript of Clinical Assessment – Part II William Sacks, PhD, MD
12/10/02 Sacks - Clinical Assessment 1
Clinical Assessment – Part IIWilliam Sacks, PhD, MD
COMPUTERIZED THERMAL IMAGING, INC.
Breast Cancer System BCS 2100
P010035
Radiological Devices Advisory PanelDecember 10, 2002
12/10/02 Sacks - Clinical Assessment 2
Safety of the BCS has two aspects:• Adverse events• Accuracy of BCS output
12/10/02 Sacks - Clinical Assessment 3
There were only 4 minor adverse events, two of which were thought not to be device related, during a clinical trial involving 2407 subjects.
12/10/02 Sacks - Clinical Assessment 4
From the point of view of the BCS output:
• Safety is more closely related to sensitivity (cancers), or FNR, but also FPR• Effectiveness is more closely related to specificity (benign masses)
12/10/02 Sacks - Clinical Assessment 5
POINTS TO BE COVERED
• What BCS 2100 (BCS) is and is not intended to do• How BCS does it• What the clinical trial demonstrated• Labeling issues
12/10/02 Sacks - Clinical Assessment 6
CLINICAL SUBMISSIONS• PMA • Amendment 4• Amendment 5• Amendment 7
12/10/02 Sacks - Clinical Assessment 7
After reviewing the PMA, the FDA sent a letter to CTI listing a number of deficiencies. CTI’s response was Amendment 4.
12/10/02 Sacks - Clinical Assessment 8
In Amendment 4, for their conclusions concerning BCS effectiveness (p. 11), CTI retrospectively selected from the PMA data one of two analytical indices (Se/Sp, not ROC) and two of three lesion types (masses, and arch. dist., not microcalcifications).
12/10/02 Sacks - Clinical Assessment 9
In Amendment 4 the revised labeling further deleted architectural distortion, and referred to masses alone.
12/10/02 Sacks - Clinical Assessment 10
The FDA sent another deficiency letter to CTI. The response was Amendment 5.
12/10/02 Sacks - Clinical Assessment 11
Amendment 5 was offered as a test of BCS in additional subjects, because Amendment 4 had contained retrospective selections. CTI called the additional dataset PPMA (Post-PMA).
12/10/02 Sacks - Clinical Assessment 12
Amendment 5 confined its analysis of the PPMA data to the newly chosen analytical index (Se/Sp) in the newly chosen subgroup (masses).
12/10/02 Sacks - Clinical Assessment 13
In addition to presenting data on a new set of subjects (PPMA), Amendment 5 also contained analysis of the combined datasets from Amendment 4 and the PPMA.
12/10/02 Sacks - Clinical Assessment 14
Because of the retrospective selections in Amendment 4, FDA asked CTI to justify combining that data with the PPMA. CTI’s response was Amendment 7.
12/10/02 Sacks - Clinical Assessment 15
In Amendment 7 CTI applied a Bonferroni correction in an attempt to compensate for• retrospective selection of data in Amendment 4 and • the smallness of the additional (PPMA) sample in Amendment 5.
12/10/02 Sacks - Clinical Assessment 16
Two overriding issues:• Adequacy of data• Interpretation of data, i.e., do the data demonstrate S&E of BCS?
12/10/02 Sacks - Clinical Assessment 17
Adequacy of data
Can data from Amendment 4 contribute to judgment of S&E, when it consists of retrospective selection of mammographic masses, because the Se was too low in microcalcifications and in the tested population as a whole?
12/10/02 Sacks - Clinical Assessment 18
Adequacy of data (cont.)
Is a Bonferroni correction applicable in this context?
12/10/02 Sacks - Clinical Assessment 19
Adequacy of data (cont.)
Are the data fromthe PPMA alone (inAmendment 5) adequate tojudge S&E?
12/10/02 Sacks - Clinical Assessment 20
Interpretation of the dataIt is noteworthy for the following discussion that no formal hypotheses were explicitly put forward for testing, either in the PMA or in the subsequent amendments.
12/10/02 Sacks - Clinical Assessment 21
To qualify as a testable hypothesis there must be a quantitative criterion, whereby either• a point estimate may imply rejection, or• a confidence interval may entail exclusion.
12/10/02 Sacks - Clinical Assessment 22
There were two implicit hypotheses:
• that the ROC area for BCS and mammography combined would exceed that of mammography alone (with statistical significance) [Protocol, Section 6.0 Statistical Analysis]
• that the point estimate for sensitivity would be at least 99.3% (in at least 75% of simulations with the data)
[PMA, Section 5.8.7.2, p. 489].
12/10/02 Sacks - Clinical Assessment 23
The protocol otherwise contained only non-quantitative statements of what CTI hoped to achieve in the clinical trial. For example,
The objective of the study is to determine if the (BCS 2100), when used in conjunction with clinical examination and/or diagnostic mammography, increases the ability of physicians to differentiate benign from malignant, or suspicious, breast abnormalities. [Protocol, Section 3.1 Study Objective]
12/10/02 Sacks - Clinical Assessment 24
In the original PMA submission, the comparison of ROC areas failed to achieve statistical significance, except as an artifact of too few points in the mammography-alone curve, so it was not pursued in any of the amendments.
12/10/02 Sacks - Clinical Assessment 25
In addition, the sensitivities failed to achieve a level of 99.3% with 75% confidence, in any of the datasets.
12/10/02 Sacks - Clinical Assessment 26
PPMA (n=15)
(Se 93.3%, Sp 25.4%)
Mammography alone
PMA (n=187)(Se 97.1%,Sp 14.3%)
Am. 4 (n=90)(Se 100%,Sp 18.0%)
Combined(n=105)
(Se 99.0%, Sp 19.2%)
72.1%
99.7%
94.1%
98.8%
95.6%96.7%
Reference Se 99.3%
SPECIFICITY
SENS
ITIV
ITY
100%
0%
Chance Line
12/10/02 Sacks - Clinical Assessment 27
Potential safety and effectiveness in U.S. population:
12/10/02 Sacks - Clinical Assessment 28
Percent of U.S. biopsies potentially obviated by the BCS if used on all eligible women: • 1.3 million U.S. women biopsied each year,• of which ~45% (585,000) have mammographic masses, • of which ~80% (468,000) are benign, • of which 15% to 20% (70,000 to 94,000) would be BCS(-) and saved a biopsy.• Thus ~5% to 7% of 1.3 million U.S. biopsies would be obviated,• and that’s if BCS were used on all 585,000 women with mammographic masses on their way to biopsy.• In addition to saving biopsies on these benign masses, approximately 1% to 6% of the malignant masses, and 0.5% to 3% of all breast cancers, might be delayed in diagnosis.
(
12/10/02 Sacks - Clinical Assessment 29
POINTS TO BE COVERED
• What BCS 2100 (BCS) is and is not intended to do• How BCS does it• What the clinical trial demonstrated• Labeling issues
12/10/02 Sacks - Clinical Assessment 30
Labeling issues
• Size of mass• Depth of mass
12/10/02 Sacks - Clinical Assessment 31
Size of mass
Effect of small lesion size on device sensitivity difficult to evaluate, since only 2/105 cancers < 5mm.
12/10/02 Sacks - Clinical Assessment 32
Size of mass (cont.)
Number of masses Malignant Benign<0.5 cm 2 12
0.5 cm - 1.0 cm
50 188
>1.0 cm 53 185
(Amendment 4, p.8, Table A4.4,
Amendment 5, p.23, Table A5.14)
12/10/02 Sacks - Clinical Assessment 33
Depth of mass
With the chosen IOS threshold (20.59) there was no definite effect of lesion depth on BCS result, but...
12/10/02 Sacks - Clinical Assessment 34
Depth of mass (cont.)
...the effect of lesion depth was difficult to evaluate, since depth is not easily gauged on mammography.
12/10/02 Sacks - Clinical Assessment 35
Conclusions
12/10/02 Sacks - Clinical Assessment 36
In summary• Only 4 out of 2407 subjects had an adverse event, all minor.• There were no explicit quantitative hypotheses.• There were two implicit quantitative hypotheses.• Neither hypothesis was fulfilled.• Most data were selected retrospectively.• Bonferroni correction is not applicable in this context.• Using the trial results, if the BCS were in general use in the U.S., it would obviate 5% to 7% of the 1.3 million biopsies per year. • Approximately 1% to 6% of these obviated biopsies would turn out to be malignant, and their diagnoses would thus be delayed.