What makes software dependable? Martyn Thomas CBE FREng independent consultant / expert witness .
Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert,...
Transcript of Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert,...
![Page 1: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/1.jpg)
Daniel Rueckert, FREng, FIEEE
Biomedical Image Analysis Group
Department of Computing, Imperial College London, UK
Learning clinically useful information from medical
images
![Page 2: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/2.jpg)
Disclosures
• Adviser – HeartFlow
• Co-founder – IXICO
![Page 3: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/3.jpg)
Artificial Intelligence and Machine Learning
AIMachine
LearningDeep
Learning
![Page 4: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/4.jpg)
The era of deep learning
Biologically inspired neural networks: Multi-layer perceptron, 1960s - 1980s
Convolutional neural networks: LeCun, 1990s
![Page 5: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/5.jpg)
The era of deep learning
Convolution + RELU
Max pooling
Fully connected layer
Softmax
Dog
Cat
![Page 6: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/6.jpg)
The era of deep learning
• Image-to-image networks (many different architectures)– Fully convolutional networks (Long et al., 2015)
– U-Net (Ronneberger et al., 2015)
– DeepMedic (Kamnitsas et al., 2016)
Convolution + RELU
Max pooling
Unpooling (transposed convolution)
Softmax Skip layers
![Page 7: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/7.jpg)
Valu
e p
ropositio
n
Le
ve
l o
f d
iag
no
stic s
up
po
rt
Deep learning for medical imaging:
Opportunities
Detection
Semantic Image Interpretation
Quantification of Imaging Biomarkers
Image Enhancement
Image Acquisition and Reconstruction
Diagnosis and Prediction
Perception
Reasoning
Computing
![Page 8: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/8.jpg)
Deep learning for image reconstruction
K-space Signal space
Full sampling
(slow)
25% sampling
(4-fold
acceleration)
learning-based
reconstruction
e.g. compressed
sensing
![Page 9: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/9.jpg)
Deep learning for image reconstruction
Convolution + RELU
Max pooling
Transposed convolution
Softmax Skip layers
![Page 10: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/10.jpg)
Magnitude reconstruction (6-fold)
Schlemper et al. IEEE TMI 2017
(a) 6x Undersampled (b) CNN reconstruction (c) Ground Truth
![Page 11: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/11.jpg)
Deep learning for image super-resolution
O. Oktay et al. IEEE TMI 2017
![Page 12: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/12.jpg)
Deep learning for image segmentation
Lavdas et al. 2017,
Medical Physics
![Page 13: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/13.jpg)
Deep learning for decision support
Prediction of survival in patients
with pulmonary hypertension
Bai et al., JCMR 2018
![Page 14: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/14.jpg)
Technical challenges
• Lack of data
![Page 15: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/15.jpg)
Technical challenges
• Lack of data Population studies
UK Biobank will provide large-scale imaging data from 100,000 subjects
![Page 16: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/16.jpg)
Technical challenges
• Lack of data Population studies
UK Biobank will provide large-scale imaging data from 100,000 subjects
+
Lifestyle
Genetics
Clinical
records
![Page 17: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/17.jpg)
Technical challenges
• Lack of data Population studies
• Domain shift
In the lab In the the real-world
![Page 18: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/18.jpg)
Technical challenges
• Lack of data Population studies
• Domain shift Transfer learning
“…the main problem was that
the data the project used to
establish standard of
attractiveness did not include
enough minorities.”
- The Guardian (url)
“You would think that Nikon, being a
Japanese company, would have
designed this with Asian eyes in mind.”
- Random commenter(url)
![Page 19: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/19.jpg)
Technical challenges
• Lack of data Population studies
• Domain shift Transfer learning
• Privacy Differential privacy and federated learning
𝑖=1
𝑘
Hospital 1 Hospital 2 Hospital k
![Page 20: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/20.jpg)
Technical challenges
• Lack of data Population studies
• Domain shift Transfer learning
• Privacy Differential privacy and federated learning
• Adversarial attacks Verification
![Page 21: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/21.jpg)
Traditional medical imaging
Reconstruction Analysis InterpretationAcquisition
✘ Serial process with no interaction between different components of imaging pipeline
✘ Limited ability for adjustment of upstream imaging pipeline based on downstream
requirements
✘ Stages of imaging pipeline not optimized for clinical endpoint
![Page 22: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/22.jpg)
AI-enabled medical imaging
Close coupling of acquisition, reconstruction, analysis and interpretation
Feedback and interaction between components of imaging pipeline
Optimization of whole imaging pipeline with respect to clinical endpoint
Acquisition Reconstruction Analysis Interpretation
![Page 23: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/23.jpg)
AI-enabled medical imaging
Acquisition Diagnosis
Do we need images at all?
![Page 24: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/24.jpg)
AI-enabled medical imaging: Example
Ground truth
20 lines of k-space 10 lines of k-space 1 line of k-spaceJ. Schlemper et al.
MICCAI 2018
![Page 25: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/25.jpg)
Acknowledgements
Ben Glocker
Bernhard Kainz
Wenjia Bai
Matthew Sinclair
Giacomo Tarroni
Ozan Oktay
Martin Rajchl
Aaron M. Lee
Nay Aung
Elena Lukaschuk
Mihir M. Sanghvi
Filip Zemrak
J. Duan
Kenneth Fung
Jose Miguel Paiva
Valentina Carapella
Young Jin Kim
Hideaki Suzuki
Paul M. Matthews
Steffen E. Petersen
Stefan K. Piechnik
Stefan Neubauer
Enzo Ferrante
Steven McDonagh
Ghislain Vaillant
G. Bello
This research has been conducted mainly using the UKBB Resource under
Application Number 2946. The initial stage of the research was conducted using the
UKBB Resource under Application Number 18545.
Jo Hajnal
Jose Caballero
Christian Ledig
Christian Baumgartner
Kostas Kamnitsas
Wenzhe Shi
Martin Rajchl
Jo Schlemper
Carlo Biffi
Nick Pawlowski
Matthew Lee
Andreas Rockall
Ioannis Lavdas
![Page 26: Learning clinically useful information from medical images · 2018. 11. 23. · Daniel Rueckert, FREng, FIEEE Biomedical Image Analysis Group Department of Computing, Imperial College](https://reader035.fdocuments.in/reader035/viewer/2022071100/5fd9a35cc65adf5de336a2fd/html5/thumbnails/26.jpg)
Acknowledgements