Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos...
Transcript of Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos...
![Page 1: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/1.jpg)
Deep Learning For Medical Image Interpretation
Pranav RajpurkarComputer Science DepartmentStanford University
![Page 2: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/2.jpg)
Today: How can we develop deep learning technologies that will be used routinely to improve clinical decision making?
![Page 3: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/3.jpg)
Diabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016)Melanomas from photos of skin lesions (Esteva et al., 2017)Lymph node metastases from H&E slides (Bejnordi et al., 2018)
Deep learning algorithms have driven successful application in medical imaging
![Page 4: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/4.jpg)
Hannun & Rajpurkar et al., Nature Medicine, 2019Rajpurkar & Irvin et al., PLOS Medicine, 2018Bien & Rajpurkar et al., PLOS Medicine, 2018Park & Chute & Rajpurkar et al., JAMA Network Open. 2019
CheXNeXt MRNet HeadXNetArrhythmiaFDA Cleared
Developing DL that matches/improves clinical experts
![Page 5: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/5.jpg)
We can develop AI technologies to improve clinical decision-making with:1. Clinically-informed ML techniques2. Dataset Design3. Human-AI Interaction
![Page 6: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/6.jpg)
1. How can clinically-informed ML techniques improve medical AI?
![Page 7: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/7.jpg)
Case study: chest x-ray interpretation
![Page 8: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/8.jpg)
Sep 2017100,000+ examples
![Page 9: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/9.jpg)
100,000+examples
100,000+ examples
Dec 2016 Jan 2017
Sep 2017
Diabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016)Melanomas from photos of skin lesions (Esteva et al., 2017)
100,000+ examples
![Page 10: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/10.jpg)
Chest x-ray input can map to multiple pathologies
![Page 11: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/11.jpg)
Search over:● CNN architectures● Class weighting strategies● Data augmentation
Developed a model to predict pathologies from chest x-ray
CheXNet
![Page 12: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/12.jpg)
CheXNet Achieved SOTA performance
![Page 13: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/13.jpg)
What can SOTA be attributed to?
● DenseNet, pretrained and all-layers finetuned● Weighted multi-label CE Loss on each category● Random horizontal flipping, and learning rate decay
- Early signal that DenseNet121 perform really well on medical imaging tasks.
- Combination of architecture and training strategy re-discovered in other datasets.
![Page 14: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/14.jpg)
The hidden cost of data-augmentation(Normal or Dextrocardia)
Normal Dextrocardia
![Page 15: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/15.jpg)
Test
Set
Diabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016)Melanomas from photos of skin lesions (Esteva et al., 2017)
X 9
Human Performance
How to set up performance vs radiologists?
![Page 16: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/16.jpg)
AU
C
Algorithm performs comparably on some, poorer on others. Why?
![Page 17: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/17.jpg)
AU
C
Having more prevalent categories helps
![Page 18: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/18.jpg)
AU
C
But there are exceptions. Why?
![Page 19: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/19.jpg)
0.4620.094 0.7150.1800.2240.237 Label Agreement with Ground Truth
AU
C
Labeling Agreement With the Ground truth Can Explain
![Page 20: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/20.jpg)
0.4620.094 0.7150.1800.2240.237
AU
C
Can clinical insights help achieve radiologist-level performance on pneumonia?
![Page 21: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/21.jpg)
Manual Expert labeled
Manual Expert labeled
Automatically labeled using NLP on radiology reports
![Page 22: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/22.jpg)
Low lung volumes. Mild pulmonary edema. Patchy bibasilar atelectasis with infection in the right lung base not excluded in the correct clinical setting.
Report labeling NLP System
Peng Y et al. NegBio. AMIA 2018 Informatics Summit. 2018.Wang X et al. ChestX-ray8. CVPR 2017.
Image is automatically labeled using NLP on corresponding radiology report
![Page 23: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/23.jpg)
Low lung volumes. Mild pulmonary edema. Patchy bibasilar atelectasis with infection in the right lung base not excluded in the correct clinical setting.
Report labeling NLP System
Peng Y et al. NegBio. AMIA 2018 Informatics Summit. 2018.Wang X et al. ChestX-ray8. CVPR 2017.
Is pneumonia present / absent / uncertain ?
NLP System uses rules on the dependency parse to label whether pneumonia present
Labeled 1 if present0 if absent / uncertain
![Page 24: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/24.jpg)
Peng Y et al. NegBio. AMIA 2018 Informatics Summit. 2018.Wang X et al. ChestX-ray8. CVPR 2017.
But pneumonia mostly expressed with uncertainty!
Labeled 0 Labeled 0
![Page 25: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/25.jpg)
Yarowsky et al. Unsupervised word sense disambiguation rivaling supervised methods. ACL. 1995.
Original Label
Ensemble of CNNs
Predicted Label
Train
Predict
Updated Label
Update if originally labeled 0
Key idea is to relabel the negatives using the vision algorithm
Original Labeled 0Now 1
![Page 26: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/26.jpg)
Yarowsky et al. Unsupervised word sense disambiguation rivaling supervised methods. ACL. 1995.
Original Label
Ensemble of CNNs
Predicted Label
Train
Predict
Updated Label
Update if originally labeled 0
Increases prevalence of pneumonia. But does relabeled dataset improve model?
1107 (1.1)
9838 (10.0)
Original Pneumonia # Examples (%)
Updated Pneumonia # Examples (%)
![Page 27: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/27.jpg)
Performance improves on 11 of 14 categories
![Page 28: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/28.jpg)
Now radiologist level performance on pneumonia achieved
The CheXNeXt algorithm performed as well as the radiologists for 10 of
14 pathologies and performed better than the radiologists on 1 pathology.
![Page 29: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/29.jpg)
Can pre-training on videos improve the performance of 3D
convolutional neural network on appendicitis?
Other examples of clinically informed ML
Rajpurkar & Irvin & Park et al. Scientific Reports. 2020
How does the collection of expert region of interest annotations affect
the performance of algorithm at different magnification levels?
![Page 30: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/30.jpg)
2. How can dataset design improve medical AI?
![Page 31: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/31.jpg)
NIH
TestSite
Zech & Badgeley et al., PLOS Medicine, 2018
TrainSite
CheXNet
NIH: 0.75MSH: 0.73
MSH
CheXNet
MSH: 0.80NIH: 0.72
Although reproducible training procedure, drop on external institution
Do we just need a bigger, cleaner dataset for cross institution generalization?
![Page 32: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/32.jpg)
Rajpurkar et al., EMNLP, 2016Rajpurkar & Jia et al., ACL, 2018
A large reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles
Lessons from building Stanford Question Answering Dataset (SQuAD)
![Page 33: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/33.jpg)
Drove advancements in deep learning models from top institutions and companies
![Page 34: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/34.jpg)
Rajpurkar et al., EMNLP, 2016, 1410 citationsRajpurkar & Jia et al., ACL, 2018, 299 citations
“Microsoft is already applying earlier versions of the models that were submitted for the SQuAD dataset leaderboard in its Bing search engine, and the company is working on applying it to more complex problems.” -- Microsoft AI Blog
“BERT will help Search better understand one in 10 searches in the U.S. in English, and we’ll bring this to more languages and locales over time.” -- Google AI Blog
Has been a driving force for advancements in language representations, network
architectures, and transfer to other tasks.
Methods it drove required scale and high label quality.
Advancements part of our search engine.Driven by scale and high label quality?
![Page 35: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/35.jpg)
200k Chest X-RaysRicher / Cleaner Labels
Strong Val+Test Ground TruthExpert Comparison
Irvin & Rajpurkar et al. 2019, AAAI
![Page 36: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/36.jpg)
Improve on NegBio’s NLP Tool to achieve SOTA with CheXpert labeler
- Comprehensive error analysis to identify gaps in rules.- Open-sourced, and now used to label some of largest
available datasets.
Johnson AE et al., Scientific Data. 2019
![Page 37: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/37.jpg)
Newer labeler uses BERT on existing feature-engineered systems with expert annotations
Development time lowered from months to days.
Smit & Jain & Rajpurkar et al., 2020
![Page 38: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/38.jpg)
Explicitly retain the uncertainty output
![Page 39: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/39.jpg)
uncertain
negative positive ignore
Current SOTA on Chexpert explicitly uses label smoothing regularization.
Using uncertainty output better than treating as negative for many classes
Best Label for Cardiomegaly Atelectasis, Edema, Effusion
Consolidation
![Page 40: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/40.jpg)
Released as open competition to the world
![Page 41: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/41.jpg)
Public leaderboard where teams submit on a hidden test set
Rajpurkar & Irvin et al. 2018, MIDLBien & Rajpurkar et al, PLOS Medicine
![Page 42: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/42.jpg)
3250 Users130 Teams Competing
Top models have significantly outperformed baseline. Are they truly more general?
![Page 43: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/43.jpg)
Rajpurkar & Joshi et al., preprint. 2020
We investigated generalization performance of top leaderboard models
![Page 44: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/44.jpg)
Could models detect TB, a label not included in model training?
- Global Health Relevance- Consolidation Proxy
![Page 45: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/45.jpg)
Yes! Performance competitive with when models are directly trained on those datasets.-- Avg AUC on 2 TB test
datasets: 0.82 and 0.89.
![Page 46: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/46.jpg)
Avg performance across tasks predicts performance
![Page 47: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/47.jpg)
Could models detect diseases on smartphone photos of chest x-rays?
- Appealing solution to scaled deployment.
- Performance drop not thoroughly investigated in medical imaging.
![Page 48: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/48.jpg)
Yes! < 0.01 drop in performance.
![Page 49: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/49.jpg)
Can models achieve radiologist level performance on external institution w/ zero fine-tuning?
If so, first demonstration of that effect!
![Page 50: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/50.jpg)
Yes! At radiologist specificity, higher model sensitivity for 4 of 5 tasks
![Page 51: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/51.jpg)
Link1 Link2 Link3
![Page 52: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/52.jpg)
3. How can Human-AI interaction be designed for clinical decision-making?
![Page 53: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/53.jpg)
Model
Clinician
Performance in clinical workflow
Performance in lab setting
Model
Would AI algorithms necessarily improve clinicians on diagnostic tasks?
![Page 54: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/54.jpg)
Rajpurkar & O’Connell & Schechter et al., in review
Can DL improve physician performance on TB detection in HIV+ patients?
![Page 55: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/55.jpg)
X-ray pretraining and clinical variables both improve performance
Strategy AUC (95% CI)
With CheXpert pre-training and inclusion of clinical variables
0.83 (0.75, 0.91)
No CheXpert pre-training 0.714 (0.618, 0.810)
No inclusion of Clinical Variables 0.57 (0.46, 0.68)
![Page 56: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/56.jpg)
Built out platform for deployment
![Page 57: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/57.jpg)
Investigated impacted of the AI assistant in a within-subjects design w/ 13 physicians
![Page 58: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/58.jpg)
AI Improved Physician Performance but...
Performance Accuracy (95% CI)
Physicians without assistance 0.60 (95% CI 0.57, 0.63)
Physicians with algorithm assistance 0.65 (95% CI 0.60, 0.70)
Algorithm by itself 0.79 (95% CI 0.77, 0.82)
![Page 59: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/59.jpg)
Improvement is not consistent
![Page 60: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/60.jpg)
Some studies across AI + Imaging
![Page 61: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/61.jpg)
Takeaway: expert-level AI -> improved clinician performance is misguided
Critical considerations include:
- experience levels- clinician interaction- case difficulty- automation bias.
![Page 62: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/62.jpg)
We can develop AI technologies to improve clinical decision-making with:1. Clinically-informed ML techniques2. Dataset Design3. Human-AI Interaction
How can I get involved?
![Page 63: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/63.jpg)
15 biomedical faculty collaborators
![Page 64: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/64.jpg)
Help researchers keep up with the cutting edge of AI + Healthcare research
http://doctorpenguin.com
2000+ Weekly ReadersWith Dr. Eric Topol
![Page 65: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/65.jpg)
Help equip interested individuals learn tools to contribute
https://www.coursera.org/specializations/ai-for-medicine
With Dr. Andrew Ng
![Page 66: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/66.jpg)
AI For Healthcare Bootcamp6-month training program for undergrad + masters students to do research in AI + healthcare.
9 quarters. 80 bootcampers.
15 biomedical faculty collaborators
https://stanfordmlgroup.github.io/programs/aihc-bootcamp/
![Page 67: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/67.jpg)
Applyhttps://stanfordmlgroup.github.io/programs/aihc-bootcamp/
![Page 68: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/68.jpg)
Lot of exciting opportunitiesTo build AI technologies that will be used routinely to improve clinical decision making.
![Page 69: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/69.jpg)
![Page 70: Deep Learning For Medical Image InterpretationDiabetic Retinopathy from retinal fundus photos (Gulshan et al., 2016) Melanomas from photos of skin lesions (Esteva et al., 2017) Lymph](https://reader035.fdocuments.in/reader035/viewer/2022071123/601f85a1c6afec308c3f0591/html5/thumbnails/70.jpg)
Thanks!