Radiation Oncology

Artificial Intelligence (AI) in Medicine:

Challenges and Opportunities

Steve Jiang, Ph.D.

Barbara Crittenden Professor in Cancer ResearchDirector, Medical Artificial Intelligence and

Automation (MAIA) LabVice Chair, Department of Radiation Oncology

Radiation Oncology

AI Is Changing The World

2 © Steve Jiang, Ph.D., MAIA Lab, 2020

Radiation Oncology

AI Is Going to Transform Healthcare

3 © Steve Jiang, Ph.D., MAIA Lab, 2020

http://time.com/5556339/artificial-intelligence-robots-medicine/

Radiation Oncology

AI Research at MAIA Lab

4

Radiation Oncology

Medical Artificial Intelligence and Automation LabMAIA Lab

5 © Steve Jiang, Ph.D., MAIA Lab, 2019

Since 07/2017

Radiation Oncology

§ To innovate, develop, and apply artificial intelligence technologies to empower clinicians, especially those with less experience or limited resources, for improved patient care, from three aspects:

§ Improving accuracy by retrieving hidden information from patient data/images and performing better than (most) physicians when there is ground truth– Precise diagnosis and better clinical decision making

§ Improving efficiency by automating clinical procedures and saving physicians’ time in front of computers– Re-humanize medicine

§ Transferring expertise to less experienced physicians by learning from experienced physicians when there is no ground truth– Reduce healthcare disparities

6 © Steve Jiang, Ph.D., MAIA Lab, 2019

Mission of MAIA Lab

Radiation Oncology

§ To empower clinicians with AI– Not trying to replace

clinicians with AI

§ To automate and augment clinical procedures with AI– Not focused on

informatics/data science

§ To reduce healthcare disparities with AI– The most impactful

application of AI in medicine

7 © Steve Jiang, Ph.D., MAIA Lab, 2020

Research Focuses of MAIA Lab

Radiation Oncology

Medical Image Segmentation§ Most organ segmentation problems are easy§ We focus on – Auto segmentation of challenging organs– Auto delineation of gross tumors and clinical targets

8 © Steve Jiang, Ph.D., MAIA Lab, 2017

Radiation Oncology

AI for Radiotherapy Treatment Planning§ Supervised learning to learn the relationship between patient

anatomy and optimal radiation dose§ Reinforcement learning to explore the large hyperparameter

space for treatment plan optimization§ Imitation learning to learn from experienced treatment planners– Similar to how to train AI playing video games or driving cars

9 © Steve Jiang, Ph.D., MAIA Lab, 2020

Nguyen, …, Jiang, arXiv:1805.10397, 2018; Phys Med Biol. 64(6):065020, 2019.

GT Pred

Radiation Oncology

Dose Prediction – Feasibility Study (9/2017)

© Dan Nguyen, Ph.D. and Steve Jiang, Ph.D., MAIA Lab, 201710

PTV

Bladder

L FemHeadR FemHead

Rectum

Body

Nguyen, …, Jiang, Sci Rep. 9(1):1076, 2019.

Radiation Oncology © Dan Nguyen, Ph.D. and Steve Jiang, Ph.D., MAIA Lab, 201811

Dose Prediction – H&N VMAT w/ HD U-NET

Nguyen, …, Jiang, Phys Med Biol. 64(6):065020, 2019.

Radiation Oncology © Chenyang Shen, Ph.D. and Xun Jia, Ph.D., MAIA Lab, 2018

HDR Planning w/ DRL Based Organ Weight Tuning

12

Shen,..., Jia, Phys Med Biol. 64(11):115013, 2019.

Radiation Oncology13

IMRT Planning w/ DRL Based Hyper-Parameter Tuning

© Chenyang Shen, Ph.D. and Xun Jia, Ph.D., MAIA Lab, 2020

Radiation Oncology

Dose Calculation using Deep Learning§ Deep learning can learn the difference between a simple and a sophisticated dose

calculation method– scatter dose and inhomogeneity effort

§ A completely different system is good for secondary dose check§ It can be used when real-time efficiency and reasonable accuracy are needed– Intermediate step dose calculation during plan optimization– Plan casting for online ART

14

HD Unet

© Penelope Xing, Ph.D. and Steve Jiang, Ph.D., MAIA Lab, 2018

Xing, …, Jiang, Med Phys 47(2)753-758, 2020

Radiation Oncology

– 120 lung cases in Eclipse (72 training/18 validation/30 testing)– Non-coplanar 3D CRT, 3D conformal arc, IMRT, and VMAT plans– Rx dose: 24 Gy to 60 Gy– Energy: 6 MV, 10 MV, 6xFFF, and 10xFFF

Dose Conversion: From AAA to Acuros XB

15 © Penelope Xing, Ph.D. and You Zhang Ph.D., MAIA Lab, 2018

Xing, Zhang, …, Jiang, JACMP DOI: 10.1002/acm2.12937, 2020

Radiation Oncology16

Convert PB Dose to MC Dose for Proton RT§ MGH Data: Pencil Beam (XiO) à Monte Carlo (TOPAS)

© Chao Wu and Steve Jiang, MAIA Lab, 2019

Predict doseComposite doseRotate BackRotate

Rotate Rotate Back

HD-UNET

Beam1

Beam2

HN Liver Prostate Lung

Number of patients 90 93 75 32

Number of beams 726 218 260 91

Pencil Beam vs MC (73.3±6.3) % (79.2±5.1) % (73.3±2.7) % (65.4±5.3) %

Predicted vs MC (92.8±2.9) % (92.7±2.9) % (99.6±0.3) % (89.7±3.8) %Gamma index (1%/1 mm)

Radiation Oncology

AI for Beam Orientation Optimization (BOO)§ Develop an AlphaGo type of DL algorithm – reinforcement learning (RL) policy network – Monte Carlo Tree Search (MCTS)

§ Go movements à CyberKnife robot sequence

© Dan Nguyen, Ph.D. and Steve Jiang, Ph.D., MAIA Lab, 2018

Ogunmolu, ..., Jiang, AAPM, 2018.

17

Radiation Oncology

DeepBOO V1: Policy Network Trained w/ Column Generation

© Dan Nguyen, Ph.D. and Steve Jiang, Ph.D., MAIA Lab, 201818

Sadeghnejad Barkousaraie, …, Nguyen, Med Phys 47(3)880-897, 2020.

Use column generation (CG) to train a supervised learning (SL) policy network

298⁰

174⁰

36⁰

242⁰

84⁰86⁰

212⁰240⁰

306⁰354⁰

Radiation Oncology19 © Azar Sadeghnejad Barkousaraie, Ph.D., and Dan Nguyen, Ph.D., MAIA Lab, 2019

DeepBoo V2: Guided Monte Carlo Tree Search§ Policy network quickly generates a sampling probability at each level§ Tree builds its own policy over time– Search policy is updated as a weighted sum of NN policy and tree policy

Radiation Oncology20 © Azar Sadeghnejad Barkousaraie, Ph.D., and Dan Nguyen, Ph.D., MAIA Lab, 2020

DeepBoo V3: w/o the need of pre-dose calculation

§ DNN is updating during the tree search after exploring each leaf§ No prior dose calculation is needed:– Using a pre-trained network to measure the quality of the treatment plan

Self Improving Tree

Back propagate

Deep BOO (v1)

Radiation Oncology

Convert CT Image to Ventilation Image§ Generating functional lung ventilation image from anatomical

4D CT images using CNN

21

Input Predicted Truth Predicted on CT

© Yuncheng Zhong, Ph.D. and Jing Wang, Ph.D., MAIA Lab, 2018

Zhong, …, Wang, Med Phys 2019 doi: 10.1002/mp.13421, arXiv:1808.06982

Radiation Oncology

AI for Cervical Lymph Node Malignancy Prediction

Chen,…, Wang, PMB, vol. 64, 075011 (13pp), 2019

q There is often uncertainty about the malignant potential of lymph nodes (LNs) in head and neck cancer

q INRT- AIR: A Prospective Phase II Study of Involved Nodal Radiation Therapy Using AI-Based Radiomics for Head and Neck Squamous Cell Carcinoma (NCT03953976 PI: David Sher)

q Predict the risk of malignancy in otherwise benign-appearing lymph nodes, using pathology as ground truth

q First clinical trial in rad onc to use AI to improve physician targetingq Already enrolled > 30 patients in 4 monthsq Expected phase III clinical trial comparing AI-based involved nodal

radiation with standard-of-care radiotherapy

© Jing Wang, Ph.D. And David Sher, M.D., MAIA Lab, 2018

Radiation Oncology

§ Infrastructure in EC building (BLE)§ Tracking patients, clinical staff, and device § Applications in clinic– A touchpad checklist before treatment to ensure correct steps are taken

and correct devices are used – Auto recording of chart round/seminar attendance – Exam room workflow management

© Steve Jiang, Ph.D., MAIA Lab, 2019

AI for Smart Clinic

23Iqbal, …, Jiang, arXiv:1711.08149, 2017Tang, …, Jiang, arXiv:1907.10554, 2019

Radiation Oncology

Implementing AI in Clinical Practice:Challenges and Solutions

24

Radiation Oncology

§ Dataset size– Dataset size is often limited for deep learning

§ Lateral data heterogeneity– Your AI model may not work for others

§ Lack of ground truth– Medicine is still an art in many cases

§ Model bias– AI model is as good as people/data learns from

§ Longitudinal data variation– Model performance will decline with time

§ Clinical data quality– Not all clinical data are created with high quality

§ Data curation and cleaning– Very expensive to curate clinical data

§ Model explainability– Model explainability is required for AI in medicine

§ Model robustness– Adversarial attacks are possible

25 © Steve Jiang, Ph.D., MAIA Lab, 2020

General Problems for AI in Medicine

Radiation Oncology

§ Distributions in latent space can be different for different institutions, machines, scanners, clinicians, etc

§ Model trained w/ Dataset A may perform poorly for Dataset B§ Not always the more data the better results

26 © Steve Jiang, Ph.D., MAIA Lab, 2019

Challenge: Data Heterogeneity (lateral/spatial variation)

Radiation Oncology27 © Steve Jiang, Ph.D., MAIA Lab, 2019

Breast and Skin Cancer Prediction (UK Biobank data)

Ghorbani and Zou, Data Shapley: Equitable Valuation of Data for Machine Learning, arXiv:1904.02868

Radiation Oncology

§ Distributions in latent space can be different for different institutions, machines, scanners, clinicians, etc

§ Model trained w/ Dataset A may perform poorly for Dataset B§ Not always the more data the better results

28 © Steve Jiang, Ph.D., MAIA Lab, 2020

Challenge: Data Heterogeneity (lateral/spatial variation)

§ What many people do– Train and test the model with your own data– No idea about model generalizability

§ What we are promoting– Train the model with your data and test it with others data– If works, show some generalizability. Enough?– How to demonstrate the model generalizability efficiently and sufficiently– Universal model (works anywhere anytime for anybody)?

Radiation Oncology

AI for Mammography Breast Cancer Screening

McKinney, Sieniek, Godbole, Godwin et al, Nature 577, 89–94(2020) (Google Health etc)

§ Trained the modelon UK datasets (25,856 women, AUC 0.889)

§ Tested on an USA dataset (3,097 women, AUC 0.810)

§ Outperformed 6 radiologists

§ “We provide evidence of the ability of the system to generalize from the UK to the USA.”– Most images were acquired with devices of a same vendor

© Steve Jiang, Ph.D., MAIA Lab, 202029

Radiation Oncology

§ Distributions in latent space can be different for different institutions, machines, scanners, clinicians, etc

§ Model trained w/ Dataset A may perform poorly for Dataset B§ Not always the more data the better results

30 © Steve Jiang, Ph.D., MAIA Lab, 2020

Challenge: Data Heterogeneity (lateral/spatial variation)

§ What many people do– Train and test the model with your own data– No idea about model generalizability

§ What we are promoting– Train the model with your data and test it with others data– If works, show some generalizability. Enough?– How to demonstrate the model generalizability efficiently and sufficiently– Universal model (works anywhere anytime for anybody)?

§ What is our solution– Train the model with your data. When deploy the model, automated

transfer learning with a small local dataset of end users– Model commissioning. Different models for different users.

Radiation Oncology

Dose Prediction - Model Generalizability

31 © Roya Kandalan and Steve Jiang,, MAIA Lab, 2019

Radiation Oncology

CBCT to CT Conversion – Model Generalizability

© Xiao Liang and Steve Jiang, Ph.D., MAIA Lab, 201832

Source dataset

Target dataset

§ Test dataset: Prostate 1§ Same vendor§ Different anatomical sites

§ Test dataset: Prostate 2§ Different vendor§ Different anatomical sites

Liang,..., Jiang, arXiv:1810.13350, 2018; PhysMed Biol, doi: 10.1088, 2019.

Radiation Oncology

§ Medicine is still an art in many cases§ Evidence based medicine and clinical guidelines only

give physicians the floor not the ceiling§ There is room for physicians to exercise their own

judgements§ Variation exists in physicians’ clinical practice§ There is often no ground truth to tell which one is

the best § How AI can help?

33 © Steve Jiang, Ph.D., MAIA Lab, 2019

Challenge: Lack of Ground Truth

à Personalized AI for Physicians

Radiation Oncology

§ Prostate gland has been surgically removed

§ Residual tumor cells are not visible in CT or MRI images

§ Not an object with defined boundary by image contrast

34 © Anjali Balagopal and Steve Jiang, Ph.D., MAIA Lab, 2019

Project: Post-op Prostate CTV Segmentation

§ Currently, manually contoured by physicians following clinical guidelines (such as RTOG) by considering – relationship with nearby organs– pathology report– pre-operative MRI – individual patient characteristics– variability in anatomy and co-morbidities

§ Large variation among physicians– Personal judgements for individual patients based on training

background, experience, and personal preference

Radiation Oncology35 © Anjali Balagopal and Steve Jiang, Ph.D., MAIA Lab, 2019

Anatomy-guided multi-task network (AG-MTN) for CTV segmentation

A deep learning framework

Project: Post-op Prostate CTV Segmentation

Radiation Oncology36

CTV Uncertainty Maps using Monte Carlo Dropout

© Anjali Balagopal, Dan Nguyen, and Steve Jiang, MAIA Lab, 2020

Radiation Oncology

COMPARISON WITH RESIDENT CONTOURS§ Residents contoured the CTV on test patients

§ Resident contours compared to the Physician contours using volume overlap

37

58%

63%

68%

73%

78%

83%

88%

93%

Physician vs Model Physician vs Resident

© Anjali Balagopal, Dan Nguyen, and Steve Jiang, MAIA Lab, 2020

Radiation Oncology38

CLINICAL EVALUATION BY PHYSICIANS§ 30 test patients contours were evaluated by experienced practicing physicians

§ Patients were completely anonymized

§ Physicians were asked to evaluate two contours - original physician contour and the AI contour and score each of them from 1-4

4 – Acceptable without changes3 -- Acceptable with minor changes2 -- Acceptable with major changes1 – Unacceptable

§ Physicians were also asked to guess which of the two contours was the physician contour.

© Anjali Balagopal, Dan Nguyen, and Steve Jiang, MAIA Lab, 2020

Radiation Oncology

CLINICAL EVALUATION RESULT§ For patients originally contoured and evaluated by the same physician – Intraobserver

evaluation, • Clinically accepted contours scored an average of 3.4 while AI contours scored 3.2• Able to identify 60% of the physician contours

§ For patients originally contoured by one physician and evaluated by a different physician – Interobserver evaluation, • Clinically accepted contours scored an average of 3.07 while AI contours scored

3.29• Able to correctly identify 25% of the physician contours

§ None of the contours were found to be clinically unacceptable

§ Statistical equivalence test showed that AI model contours and clinically accepted contours are equivalent.

39© Anjali Balagopal, Dan Nguyen, and Steve Jiang, MAIA Lab, 2020

Radiation Oncology40 © Anjali Balagopal and Steve Jiang, Ph.D., MAIA Lab, 2019

Personalized AI for Physicians

85.1

0%

82.8

0%

82.2

0%

82.2

0%

89.0

0%

84.8

0%

84.7

0% 85.8

0%

PHY1 PHY2 PHY3 PHY4

General segmentation model Physician Style adapted model

Radiation Oncology41 © Anjali Balagopal and Steve Jiang, Ph.D., MAIA Lab, 2019

Personalized AI for Physicians

340 post-op prostate cancer patients contoured by 4 physicians at UTSW

0 40 80 120

Phy 1

Phy 2

Phy 3

Phy 4

Style classification model (~90% accuracy for 4 styles)

§ Proposed workflow– Train a CTV segmentation model using whatever data you have– When deploying the model, find physicians’ contouring styles from the

local data – Automatically adapt the initial model to each style– When segmenting a new patient, choose the model for the desired style

Radiation Oncology

§ AI is only as good as the people and data it learns from § Biases exist in medicine (race, gender, income, etc), but AI can

make these biases automated and invisible§ Bias in AI is mainly from the biased training data– Patient cohort/patient geographical spread/genetic background/disease

incidence/data labeling/data-acquisition/data-curation method– Training data is a subset of a heterogeneous dataset (IMB Watson Oncology )

– Training data doesn’t include enough samples for a particular subset– Training data include a pre-existing bias (Low-income patients do poorly for X

treatment à AI suggests no X treatment for them)

§ How to design AI so that it is fair to all patients/clinicians– Identify sources of inequity/De-bias training data– Develop algorithms that are robust to skews in data– Concepts of local models and model commissioning may be helpful – Regularly monitor AI output and downstream consequences

42 © Steve Jiang, Ph.D., MAIA Lab, 2019

Challenge: Bias

Radiation Oncology

§ Typically pediatric radiotherapy involves 25-30 episodes of GA§ Pediatric Radiation Oncology with Movie Induced Sedation Effect§ Radiation and movie are paused if motion exceeds tolerance§ DL-based motion detection and prediction à bias issue

© Steve Jiang, Ph.D., MAIA Lab, 2020

PROMISE Project

43

Radiation Oncology

§ Due to the progress of the medical technologies, clinical data used for AI model training will become obsolete

§ Model trained today may not work well some time later§ Model performance will decay with time§ Each trained AI model has a half life§ Static models à Evolving models§ We need to continuously QA the deployed AI model§ We need to continuously update the deployed model– How to update the model automatically with new data– How to avoid/minimize human intervention for data curation– How to QA the updated model– How to deal with regulatory issues– … …

44 © Steve Jiang, Ph.D., MAIA Lab, 2019

Challenge: Data Evolution (longitudinal/temporal variation)

Radiation Oncology

§ Deep Learning is a black box§ Explainability is important for AI in medicine§ It is still controversial § For some people, we don't need to understand how it works– Complex systems: too many parameters and relationships– Beyond human's comprehension - humans can only handle 5 or 6 things

in their heads at once– End users don’t really care what's under the hood as long as its

performance is good and robust

45 © Steve Jiang, Ph.D., MAIA Lab, 2019

Challenge: Explainability (I)

Radiation Oncology

§ For many others, we need to understand how it works– It is not about knowing everything– It is about knowing enough for your downstream tasks– Correlations are often not sufficient to take action– Causality decreases the risk of the action

§ Methods for understanding how DL works– Attribution methods• E.g., which part of image leads to the diagnosis

– Local approaches• Local approximation and thus explainable

– Attention mechanism• Which part of the model is associated with a given class

– Distillation• Simpler and more interpretable models are used to approximate the behavior of complex models

– Causal methods46 © Steve Jiang, Ph.D., MAIA Lab, 2019

Challenge: Explainability (II)

Radiation Oncologyhttps://www.utsouthwestern.edu/labs/maia/