Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show...
Transcript of Precision in Quantitative Imaging: Trial Development and ... PDFs... · • All ROI protocols show...
Precision in Quantitative Imaging:
Trial Development and
Quality Assurance
Susanna I Lee MD, PhD
Thanks to:
Mitchell Schnall, Mark Rosen. Dan Sullivan, Patrick Bossuyt
Imaging Chain: Patient Data
Raw data
Image
analysis123……………
2346…………..
65789…………
6578…………..
Data output
Image
reconstruction
Image
processing
Data analysis
Clinical Trials: Imaging is an “Assay”
♦ Disease Detection
• Screening
♦ Characterize Disease
• Diagnosis, eligibility or prognosis
• Anatomic distribution (e.g. tumor staging)
• Higher level features (e.g. heterogeneity, vascularity, etc.)
♦ Monitor Therapeutic Response
• Change in features with therapy
• Anatomic (e.g. RECIST) or functional (e.g. SUV)
REFERENCE STANDARD OF
PATIENT OUTCOME
• OVERALL SURVIVAL (OS)
• PROGRESSION FREE SURVIVAL
(PFS) OR DISEASE FREE
SURVIVAL (DFS)
• TREATMENT RESPONSE
REFERENCE STANDARD OF
DISEASE STATE
• PATHOLOGY
• CONFIRMATORY TEST
• FOLLOWUP
MANAGEMENT
• STANDARD OF CARE
• STUDY DEFINED
INDEX TEST
• IMAGING EXAM
• IMAGE-GUIDED
PROCEDURE
Schema
ENROLLED
PARTICIPANT
Diagnostic accuracy
Biomarker
Cancer Biomarkers
GeneticGenome
Expression (e.g.
RNA or protein)
Serum
(CA125,
PSA, AFP,
CA19-9)
Tissue(e.g. hormone
receptors,
cytokeratins)
PATIENT OUTCOME
1. Overall survival (OS)
2. Progression free
survival (PFS)
3. Clinical response
Imaging
What is a good biomarker?
♦Stable technology
♦Available widely
♦Standardized image acquisition
♦Reproducible
♦Range of normal defined
Sargent DJ, Rubinstein L, Schwartz L et al. Eur J Cancer 2009; 45: 290.
Balance “state of the art” with
“generalizability”
Pre-treatment
Post-treatment
Range of
test-retest
Lankester KJ, Taylor NJ, Stirling JJ et al. Br J Cancer. 2005.93:979.
Variability: Test – Retest
Same patient, day, scanning protocol but separate imaging sessions
Conclusion
Index test variability precludes detecting pre- vs. post-treatment change.
Signal Requires Data Quality
Precision vs. Bias
precise
accurate
not precise
not accurate
precise
not accurate
Multiple Sclerosis MRI
♦ Image acquisition
• T1
• T2
• Post-gadolinium T1
♦ Image analysis
• Number of new or enlarging lesions
• Number of enhancing lesions
• MRI endpoint in MS treatment studies
• 157 publications from 1995 to 2006
Imaging Chain: Patient Data
Raw data
Image
analysis123……………
2346…………..
65789…………
6578…………..
Data output
Image
reconstruction
Image
processing
Data analysis
Imaging Manual
♦ Hardware and software
♦ Scanner calibration
♦ Patient preparation
♦ Scanning protocol
♦ Post-processing
Image acquisition manual with a step by step description is part
of any prospective study design.
What determines resolution?
♦Physics of acquisition (i.e. modality)
♦Sampling (e.g. matrix, detector size)
♦Filtering and other contributions inherent in
the reconstruction
Point Spread Function
Patient Image
Image ““““edges”””” approximate anatomy
Structure
Real Edge
Image Edge
Resolution and Sampling
160 x 160 matrix 320 x 224 matrix
Filtering
4 2
222
2
2 2 2
1
1
1
1
1111
1
1
1
1 1 1 1 1
Filtering: MRI
Vendor 1 Vendor 2
Filtering: X-ray
Vendor 1 Vendor 2
Partial Volume Effects
Completely
in scan planePartially in
scan plane
Partial Volume Effects: CT
Completely in scan plane Partially in scan plane
HU = 0 HU = 30-60
Partial Volume Effects: PET
lesion
lesion
Blurred margins
Lower intensity
Quantitative Imaging
Biomarkers Alliance (QIBA)
♦ Started by RSNA 2007
♦ Mission: Improve the value and practicality of
quantitative imaging biomarkers by reducing
variability across devices, patients and time
♦ “Build imaging devices that are also measuring
devices”
♦ “Industrialize imaging biomarkers”
https://www.rsna.org/QIBA/
QIBA Approach
1. Identify the sources of error and variability
2. Specify potential solutions in the form of profiles
3. Test these solutions
4. Promulgate profiles to vendors and users
• Advise vendors what must be implemented in their product
• Communicate the necessary procedures to users
Purpose of profiles:
QIBA Profile Activity Diagram
Equipment
Assessment
Subject
Preparation
Image
Reconstruction
Image
AnalysisInterpretation
Image
Acquisition
Manufacturer specification (pre-delivery)
Installation specification
Maintenance Quality
Assurance
ACR Core Lab QA
♦Site qualification
• Instrument performance
• Training
♦Monitoring of image acquisition
• Scan header for protocol compliance
• On-line technologist qualitative review
• Periodic radiologist review
♦Centralized image analysis
• Post-processing
• Reader study
Require Site Protocol Compliance
Type T1 weighted GRE
Orientation Sagittal
Pulse Sequence Dynamic 3D
Field Of View (FOV) 16-18 cm
Slice Thickness 64 slices of thickness ≤ 2.5
mm
Skip
Matrix min. 256 x 192
Frequency A/P
NEX 2
Phase Wrap NO
Fat-Saturation YES
TR ≤ 20 ms
Effective TE 4.5 ms
Scan Duration Between 4.5 and 5 minutes
Flip angle <= 45 degrees
Correct
Submitted
ACRIN 6657 (I-SPY 1 trial), Nola Hylton, PI
Imaging Chain: Patient Data
Raw data
Image
analysis123……………
2346…………..
65789…………
6578…………..
Data output
Image
reconstruction
Image
processing
Data analysis
Image analysis: Turning image into data
♦User extracted features
♦Semi automated
♦Automated
Feature 1
Feature 2
Feature 3
.
.
.
Radiomics: Deep Learning
Untrained neural network
Radiomics: Deep Learning
Trained neural network
Radiomics: Automated Image Analysis
Improve diagnostic accuracy
Radiomics: Automated Image Analysis
Triage clinical workflow
Semi-automated: Manual Segmentation
Quantitative
feature
extraction
Large image
datasets
segmented for
tumor
Integrate with
genomic &
clinical data
for machine
learning
Model
predictive
indices
Aerts HJ et al. Nat Commun. 2014 ;5:4006.
Reader Extracted Features
•Density• Fluid, soft tissue, calcified
•Shape• Round, oval, irregular
•Size• Linear, volume
•Margin• Sharp, blurred, spiculated
•Intensityhigh/medium/low/minimal
•Summary assementBIRADS level
0.0 0.2 0.4 0.6 0.8 1.0
FP
0.0
0.2
0.4
0.6
0.8
1.0
TP
Beam, Layde, Sullivan Arch Intern Med 1996; 156:209-213
ROC operating points of 108 radiologists
reading same mammograms
Skill
Value
judgments
• Increases and decreases of <10% can be a result of
inherent variability.
Reader Variability: Size
Oxnard GR et al. J Clin Oncol. 2011;29:3114-9.
progression
SUV=4.0 SUV=5.6
Variability Introduced by ROI Selection
Slice 483 Slice 479 Slice 479
SUV=6.6
Variability Introduced by ROI Selection
• All ROI protocols show excellent inter-observer agreement (ICC 0.94)
• Different ROI protocols yield different ADC values
Priola AM et al. Eur Radiol. 2016 Aug 11.
Effect of Windowing
Soft tissue window Liver window
Effect of Windowing
• Significant measurement differences between window settings (p<0.001).
• No significant differences in measurement variability between the lung and
mediastinal window settings (p>0.05).
Lung window Mediastinal window
Kim H et al. PLoS One. 2016;11:e0148853.
Reader Study
♦ Reader blinded to reference standard
♦ Multiple readers
♦ Independent rather than consensus reads
♦ Rules for image interpretation
• Clinical information available to reader
• Image selection, windowing, order, etc.
• Choosing index lesions
• Selecting region of interest (ROI)
• Definition of positive vs. negative test
♦ Washout period between read sessions of paired imaging exams
♦ Digital data forms and screen to document reader study
A manual defining reader rules and training cases are part of any
prospective study design.
♦Overall trial framework
♦Hypothesis and specific aims
♦ Participants
♦ Index test
♦ Reference standard
♦ Data analysis plan (statistics)
♦ Conclusions and implications
♦ Funding and compliancehttp://www.equator-network.org/reporting-guidelines/stard/
Index Test (Imaging Exam)
♦ STARD 10 - Index test, in sufficient detail to allow replication
♦ STARD 12 - Definition of and rationale for test positivity cut-offs or
result categories of the index test, distinguishing pre-specified from
exploratory
♦ STARD 13 - Whether clinical information and reference standard
results were available to the performers/readers of the index test
♦ STARD 25 - Any adverse events from performing the index test
Steps toward precision:
♦ Define image acquisition
• Equipment, patient preparation, protocol
• Balance “state of the art” with “generalizability”
♦ Define image analysis
• Read rules, training and testing
♦ Validate the system
• Test-retest, reader agreement measurements
♦ Build in procedures for ongoing QA