Convolutional Neural Networks For Modeling Temporal...

28
Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease Predictions Narges Razavian New York University Langone Medical Center GTC 2017 In collaboration with : David Sontag PhD , Saul Blecker MD , Ann-Marie Schmidt MD , Enrico Bertini PhD, Rahul Krishnan, YD Choi, Josua Krause, Somesh Nigam, Aaron Smith-McLallen, Ravi Chawla

Transcript of Convolutional Neural Networks For Modeling Temporal...

Page 1: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Convolutional Neural Networks For Modeling Temporal Biomarkers And

Disease Predictions

Narges Razavian New York University Langone Medical Center

GTC 2017

In collaboration with: David SontagPhD, Saul BleckerMD, Ann-Marie SchmidtMD, Enrico BertiniPhD, Rahul Krishnan, YD Choi, Josua Krause, Somesh Nigam, Aaron Smith-McLallen, Ravi Chawla

Page 2: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Deep learning progress Healthcare world getting digital

Parallel Developments

EHR adoption by healthcare centers in the US

Error rate on Image-Net object recognition challenge

Page 3: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

What is captured in the EHR?

Source: healthcare.gov

Page 4: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Healthcare has joined the data-rich world

Page 5: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Moving from Treatment to Prevention

Challenges: Each Individual has a different ‘healthy’ baseline.

- Temporal Patterns/Trends are predictive Each biomarker varies at a different speed in our bodies Measurements are sparse, asynchronous and correlated Many correlated outcomes are observed per patient

- Can we leverage this correlation?

Page 6: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Biomarkers and Outcomes

Biomarkers measurements

over time

Page 7: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Biomarkers and Outcomes

Biomarkers measurements

over time

Phenotype (diseases) over time

Page 8: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Biomarkers and Outcomes

Biomarkers measurements

over time

Phenotype (diseases) over time

Page 9: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Biomarkers and Outcomes

Biomarkers measurements

over time

Phenotype (diseases) over time

Page 10: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Step 1 Learn each biomarker from other biomarkers time-series

Page 11: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Kernel Regression

Observations

X

(Mea

sure

men

t Ti

me-

Ser

ies)

Time Not Observed Want to estimate

Page 12: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Kernel Regression

Observations

X

(Mea

sure

men

t Ti

me-

Ser

ies)

Time Not Observed Want to estimate

E[X(v)]= xP(x |∫ t = v,Xtrain )dx

E[X | t = v,Xtrain ]= x∫ P(x, t = v | Xtrain)P(t = v | Xtrain)

dx

E[X | t = v,Xtrain ]= x∫K(x − xi,v− ti )

xi ,ti

∑

K(v− ti )ti

∑dx

Page 13: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Kernel Regression

Observations

X

(Mea

sure

men

t Ti

me-

Ser

ies)

Time Not Observed Want to estimate

E[X(v)]= xP(x |∫ t = v,Xtrain )dx

E[X | t = v,Xtrain ]= x∫ P(x, t = v | Xtrain)P(t = v | Xtrain)

dx

E[X | t = v,Xtrain ]= x∫K(x − xi,v− ti )

xi ,ti

∑

K(v− ti )ti

∑dx

E[X | t = v,Xtrain ]=(K ⊗ Xtrain )(v)

(K ⊗ I(Xtrain :Observed))(v)

Page 14: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Use convolution framework to LEARN those kernels

Page 15: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

We can learn the kernel (No need for parametric forms and cross validations) Easily extendible to multivariate!

Unsupervised: All needed is (asynchronous) sequence of observations. Fast to train. Fast to apply.

Benefits

Page 16: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Data:30KIndividualsfromtheoriginaltrainingset.Datasetsplitequallybetweentrain,testandvalidateset.Loss:MSE.Trainandevaluateonlyon(lab,person)withmorethan1observaGon.

Mul$variateKernelslearnedforeachinputdimension(total18)

Page 17: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

More details in our ICLR paper

Narges Razavian, David Sontag Temporal Convolutional Neural Networks for Diagnosis from Lab Tests http://arxiv.org/abs/1511.07938 Open Source code available (torch/lua implementation): https://github.com/clinicalml/deepDiagnosis

Page 18: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Step 2 Predict 200+ correlated outcomes using multi-resolution convolutional neural networks and multi-task learning

Page 19: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Multi-Resolution Convolution Networks The Architecture - model (1)

Page 20: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Multi-Resolution Convolution Networks The Architecture - model (2)

Page 21: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Prediction AUCs on the held-out test set

Page 22: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

More details in our JMLR paper

Narges Razavian, Jake Marcus, David Sontag, Multi-task Prediction of Disease Onsets from Longitudinal Lab Tests JMLR, 2016 http://arxiv.org/abs/1608.00647 Open Source code available (torch/lua implementation): https://github.com/clinicalml/deepDiagnosis

Page 23: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Following up in clinical world •  Prediction models built and deployed for

–  Nurse calls and home visits for 250,000+ NYUMC patients at high risk for a number of these outcomes

–  Improved documentation in EHR •  Automation of mandatory visits/screening/follow-ups •  Best practice alerts •  Reimbursement for intense lifestyle management programs

•  Extending to broader outcomes and domains

Page 24: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

New York University (i2b2) Database

Page 25: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

New York University (i2b2) Database Nuclear Medicine Procedures

Magnetic Resonance Imaging

Page 26: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Conclusions •  Applications of deep learning in healthcare are unlimited

•  Unsupervised learning + back-propagation + deep learning can recover biomarker models from asynchronous high-dimensional time-series data

•  Multi-task learning benefits prediction tasks with smaller datasets.

Page 27: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Thanks!

Questions/comments: [email protected]

Open Source Package: https://github.com/clinicalml/deepDiagnosis

Page 28: Convolutional Neural Networks For Modeling Temporal ...on-demand.gputechconf.com/gtc/2017/presentation/s...Convolutional Neural Networks For Modeling Temporal Biomarkers And Disease

Convolutional Neural Networks (CNN)

Convolutional Neural Networks (CNN)

Understanding Convolutional Neural Networks

Understanding Convolutional Neural Networks

ImageNet Classification with Deep Deep Convolutional …rita/visual_recog_course/talks/ImageNet... · ImageNet Classification with Deep Deep Convolutional Convolutional Neural Neural

ImageNet Classification with Deep Deep Convolutional …rita/visual_recog_course/talks/ImageNet... · ImageNet Classification with Deep Deep Convolutional Convolutional Neural Neural

Convolutional Neural Networks Analyzed via Convolutional ...yromano/posters/ml_csc_poster.pdf · Convolutional Neural Networks ... Recently, convolutional sparse coding (CSC) ...

Convolutional Neural Networks Analyzed via Convolutional ...yromano/posters/ml_csc_poster.pdf · Convolutional Neural Networks ... Recently, convolutional sparse coding (CSC) ...

Convolutional Neural Networks

Convolutional Neural Networks

Convolutional Neural Networks - Virginia Techjbhuang/teaching/ece5554-4554/fa17/... · Convolutional Neural Networks Computer Vision ... ImageNet Classification with Deep Convolutional

Convolutional Neural Networks - Virginia Techjbhuang/teaching/ece5554-4554/fa17/... · Convolutional Neural Networks Computer Vision ... ImageNet Classification with Deep Convolutional

Convolutional Neural Networks for Object Recognition...Overview •What is Computer Vision ? •Convolutional Neural Networks •Convolutional Networks for Visual Object Recognition

Convolutional Neural Networks for Object Recognition...Overview •What is Computer Vision ? •Convolutional Neural Networks •Convolutional Networks for Visual Object Recognition

Temporal Convolutional Neural Networks for Radar Micro ...

Temporal Convolutional Neural Networks for Radar Micro ...

ImageNet Classification with Deep Convolutional Neural ... · Convolutional neural networks Output Hidden Data Here's a one-dimensional convolutional neural network Each hidden neuron

ImageNet Classification with Deep Convolutional Neural ... · Convolutional neural networks Output Hidden Data Here's a one-dimensional convolutional neural network Each hidden neuron

Convolutional Neural Networksimft.ftn.uns.ac.rs/ssip2017/wp-content/uploads/... · Convolutional Neural Networks for Semantic ... Shelhamer, Darrell, Fully Convolutional Networks

Convolutional Neural Networksimft.ftn.uns.ac.rs/ssip2017/wp-content/uploads/... · Convolutional Neural Networks for Semantic ... Shelhamer, Darrell, Fully Convolutional Networks

Interpretable Convolutional Neural Networksopenaccess.thecvf.com/content...Convolutional_Neural_CVPR_2018_… · Interpretable Convolutional Neural Networks Quanshi Zhang, Ying Nian

Interpretable Convolutional Neural Networksopenaccess.thecvf.com/content...Convolutional_Neural_CVPR_2018_… · Interpretable Convolutional Neural Networks Quanshi Zhang, Ying Nian

Convolutional Neural Networks - arXiv

Convolutional Neural Networks - arXiv

Using Support Vector Machines, Convolutional … Support Vector Machines, Convolutional Neural Networks and ... Using Support Vector Machines, Convolutional Neural ... orcircuit ,andmuchANNresearch

Using Support Vector Machines, Convolutional … Support Vector Machines, Convolutional Neural Networks and ... Using Support Vector Machines, Convolutional Neural ... orcircuit ,andmuchANNresearch

Probabilistic Forecasting with Temporal Convolutional ... · Probabilistic Forecasting with Temporal Convolutional Neural Network Yitian Chena, Yanfei Kangb,, Yixiong Chenc, Zizhuo

Probabilistic Forecasting with Temporal Convolutional ... · Probabilistic Forecasting with Temporal Convolutional Neural Network Yitian Chena, Yanfei Kangb,, Yixiong Chenc, Zizhuo

O-CNN: Octree-based Convolutional Neural Networks for … · O-CNN: Octree-based Convolutional Neural Networks for 3D ... object classification, ... Octree-based Convolutional Neural

O-CNN: Octree-based Convolutional Neural Networks for … · O-CNN: Octree-based Convolutional Neural Networks for 3D ... object classification, ... Octree-based Convolutional Neural

Convolutional Neural Networks (Part II)lvelho.impa.br/ip17/proj/slides/1110GoingDeeperConv.pdf · Convolutional Neural Networks 1 Convolutional Neural Networks (Part II) 08, 10 &

Convolutional Neural Networks (Part II)lvelho.impa.br/ip17/proj/slides/1110GoingDeeperConv.pdf · Convolutional Neural Networks 1 Convolutional Neural Networks (Part II) 08, 10 &

Convolutional Neural Networks Arise From Ising Models and ...web.stanford.edu/~sunilpai/convolutional-neural-networks.pdf · Convolutional neural networks are an attractive option

Convolutional Neural Networks Arise From Ising Models and ...web.stanford.edu/~sunilpai/convolutional-neural-networks.pdf · Convolutional neural networks are an attractive option

Hyperbolic Graph Convolutional Neural Networkspapers.nips.cc › paper › 8733-hyperbolic-graph-convolutional-neural-networks.pdfGraph Convolutional Neural Networks (GCNs) are state-of-the-art

Hyperbolic Graph Convolutional Neural Networkspapers.nips.cc › paper › 8733-hyperbolic-graph-convolutional-neural-networks.pdfGraph Convolutional Neural Networks (GCNs) are state-of-the-art

Neural Predictive Coding using Convolutional Neural ...Neural Predictive Coding using Convolutional Neural Networks towards Unsupervised Learning of Speaker Characteristics Arindam

Neural Predictive Coding using Convolutional Neural ...Neural Predictive Coding using Convolutional Neural Networks towards Unsupervised Learning of Speaker Characteristics Arindam

Convolutional Neural Network

Convolutional Neural Network