A novel deep learning framework for automated lesion ... · The final segmentation is cleaned using...
1
A novel deep learning framework for automated lesion segmentation of structural MRI data Ioannis Pappas 1 , Henrik Hector 2 , Kari Haws 2 , Brian Curran 2 , Andrew S Kayser 2,3 , Mark D’Esposito 1,2,4 Background “Neurologist-inspired” deep learning network §Lesion studies are important in terms of assessing the impact of brain damage on complex cognitive processes and behavior (Vaidya et al., 2019; Rorden and Karnath, 2004). Lesion segmentation is the most important step for neuroimaging data analysis in lesion studies. §In stroke studies lesion segmentation is not trivial due to the variability in lesion location, size, the associated structural changes (e.g. the size of the ventricles) and the dependence on time post stroke (e.g. chronic vs. acute). This makes lesion segmentation a tedious and time-consuming process. -Thus it is important to utilize automated methods that 1) can assist with lesion segmentation, 2) increase time efficiency and 3) assist with the observer-dependent difficulties. Training specifications and results 1. Helen Wills Neuroscience Institute, University of California, Berkeley, CA 2. VA Northern California Health Care System, Martinez, CA 3. Department of Neurology, University of California, San Francisco, San Francisco, CA 4. Department of Psychology, University of California, Berkeley, CA Contact: [email protected] §Previous methods for automated lesion identification include Gaussian naïve Bayes lesion detection (lesionGnb-Griffis et al., 2016), the ALI toolbox (ALI-Seghier et al., 2008), and neighborhood data analysis (LINDA) (Pustina et al., 2016). §At the same time, deep learning neural network techniques have emerged as a powerful alternative for supervised learning with great model capacity and the ability to utilize features that are relevant for classification. §Most importantly, Kamnitsas et al. (2017) showed an efficient implementation of 3D Convolutional Neural Networks (CNNs) with multiple pathways that can work on 3D segments of neuroimaging data. aa §Training dataset description 59 chronic stroke patients T1 and T2 structural images were acquired and used as input to the neural network. Image intensities were standardized. Ground truth lesion masks were manually segmented and reviewed by the neurologists Dr. Kayser and Dr. D’Esposito. §We utilize the power of 3D CNNs but include an approach that stems from how an experienced observer would manually demarcate the lesion. Specifically, we exploit the asymmetries between hemispheres in lesioned brains in order to enhance the neural network’s ability to identify the location of the lesion. Previous work i ii iii iv Lesion Non-lesion Neural network architecture. The neural network consists of 4 pathways. Each pathway takes different images as an input i) whole-brain ii) right hemisphere iii) left hemisphere iv) subsampled whole brain images. Each pathway consists of 9 layers. The input size and the size of the feature maps are shown only for the first pathway. The results of the 4 pathways are concatenated and, finally, fully connected layers classify the tissue into lesion or non-lesion. The final segmentation is cleaned using CRFs (Figure 2) Abbreviations FMs: Feature Maps. Figure 1 Training specifications We used 35 epochs (still training). §The network was trained with batch size =10. §Number of sub-epochs per epoch =50. §Number of cases loaded per sub-epoch =50. §Learning rate was stable between epochs. § We performed data augmentation with sagittal reflections. §The network output was compared against the Kamnitsas et al., (2017) network with one main pathway, 3 sub-sampled pathways and identical training specifications. Figure 3 Training statistics Accuracy: (TL+TNL)/(L+NL) Sensitivity: TL/L Specificity: TNL/NL DSC: Dice-Sørensen coefficient Figure 4 Examples of test segmentations. a) We show lesion ouput on two test cases (never seen by the neural network) b) We compare the output to the manually drawn masks and a recently proposed neural network with similar architecture. Predic*on Manually drawn mask Kamnitsas et al., 2017 3D CNN a b Subject 1 Subject 2 Accuracy 0 1 2 Epochs 0 1 2 Epochs 0 1 2 Epochs 0 1 2 Epochs Sensi*vity Specificity DSC §Deep convolutional neural networks can provide high accuracy in lesion segmentation (Figure 3). We propose a “neurologist-inspired” approach where the neural network exploits the lesion-induced asymmetries between brain hemispheres (Figure 1). §Segmentations in the test set show that there is a high overlap between the output of the network and the manually drawn mask (Figure 4) Compared to the state-of-the-art networks, our neural network does better at identifying the different parts of the lesions (e.g. around the ventricles and bilateral parts) (Figure 4). §By incorporating more data, accuracy will increase thus rendering our tool an easy, automated method that can assist with lesion demarcation. Summary 1) Vaidya AR, Pujara MS, Petrides M, Murray EA, Fellows LK (2019) Lesion Studies in Contemporary Neuroscience. 23:653-671 2) Rorden C, Karnath HO (2004) Using human brain lesions to infer function: a relic from a past era in the fMRI age? Nat Rev Neurosci 5:813-819 3) Griffis JC, Allendorfer JB, Szaflarski JP (2016) Voxel-based Gaussian naïve Bayes classification of ischemic stroke lesions in individual T1-weighted MRI scans. J Neurosci Methods 257:97-108 4) Seghier ML, Ramlackhansingh A, Crinion J, Leff AP, Price CJ (2008) Lesion identification using unified segmentation-normalisation models and fuzzy clustering. NeuroImage 41:1253-1266 5) Pustina et al. (2016) Automated segmentation of chronic stroke lesions using LINDA: Lesion identification with neighborhood data analysis. Hum Brain Mapp 37:1405-1421 6) Kamnitsas K et al. (2017) Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical Image Analysis 36:61-78 7) Krähenbühl P, Koltun V (2011). Efficient inference in fully connected CRFs with gaussian edge potentials. Adv. Neural Inf. Process. Syst. References Test segmentations Abbrevia(ons L=Lesion, NL=Non Lesion, TL= True Lesion, TNL=True Non Lesion Figure 2 Network output and post cleaning using CRFs. After obtaining the input we use CRFs to clean the segmentaion based onsmoothing and appearance kernels.
Transcript of A novel deep learning framework for automated lesion ... · The final segmentation is cleaned using...
A novel deep learning framework for automated lesion segmentation of structural MRI data Ioannis Pappas1, Henrik Hector2, Kari Haws2, Brian Curran2, Andrew S Kayser2,3, Mark D’Esposito1,2,4
Background
“Neurologist-inspired” deep learning network
§ Lesion studies are important in terms of assessing the impact of brain damage on complex cognitive processes and behavior (Vaidya et al., 2019; Rorden and Karnath, 2004). Lesion segmentation is the most important step for neuroimaging data analysis in lesion studies. § In stroke studies lesion segmentation is not trivial due to the variability in lesion location, size, the associated structural changes (e.g. the size of the ventricles) and the dependence on time post stroke (e.g. chronic vs. acute). This makes lesion segmentation a tedious and time-consuming process. -Thus it is important to utilize automated methods that 1) can assist with lesion segmentation, 2) increase time efficiency and 3) assist with the observer-dependent difficulties.
Training specifications and results
1. Helen Wills Neuroscience Institute, University of California, Berkeley, CA 2. VA Northern California Health Care System, Martinez, CA
3. Department of Neurology, University of California, San Francisco, San Francisco, CA 4. Department of Psychology, University of California, Berkeley, CA
Contact: [email protected]
§ Previous methods for automated lesion identification include Gaussian naïve Bayes lesion detection (lesionGnb-Griffis et al., 2016), the ALI toolbox (ALI-Seghier et al., 2008), and neighborhood data analysis (LINDA) (Pustina et al., 2016). § At the same time, deep learning neural network techniques have emerged as a powerful alternative for supervised learning with great model capacity and the ability to utilize features that are relevant for classification. § Most importantly, Kamnitsas et al. (2017) showed an efficient implementation of 3D Convolutional Neural Networks (CNNs) with multiple pathways that can work on 3D segments of neuroimaging data.
aa
§ Training dataset description 59 chronic stroke patients T1 and T2 structural images were acquired and used as input to the neural network. Image intensities were standardized. Ground truth lesion masks were manually segmented and reviewed by the neurologists Dr. Kayser and Dr. D’Esposito.
§ We utilize the power of 3D CNNs but include an approach that stems from how an experienced observer would manually demarcate the lesion. Specifically, we exploit the asymmetries between hemispheres in lesioned brains in order to enhance the neural network’s ability to identify the location of the lesion.
Previous work
i
ii
iii
iv
Lesion
Non-lesion
Neural network architecture. The neural network consists of 4 pathways. Each pathway takes different images as an input i) whole-brain ii) right hemisphere iii) left hemisphere iv) subsampled whole brain images. Each pathway consists of 9 layers. The input size and the size of the feature maps are shown only for the first pathway. The results of the 4 pathways are concatenated and, finally, fully connected layers classify the tissue into lesion or non-lesion. The final segmentation is cleaned using CRFs (Figure 2) Abbreviations FMs: Feature Maps.
Figure 1
Training specifications We used 35 epochs (still training). § The network was trained with batch size =10. § Number of sub-epochs per epoch =50. § Number of cases loaded per sub-epoch =50. § Learning rate was stable between epochs. § We performed data augmentation with sagittal reflections.
§ The network output was compared against the Kamnitsas et al., (2017) network with one main pathway, 3 sub-sampled pathways and identical training specifications.
Figure 3 Training statistics Accuracy: (TL+TNL)/(L+NL) Sensitivity: TL/L Specificity: TNL/NL DSC: Dice-Sørensen coefficient
Figure 4 Examples of test segmentations. a) We show lesion ouput on two test cases (never seen by the neural network) b) We compare the output to the manually drawn masks and a recently proposed neural network with similar architecture.
Predic*onManuallydrawnmaskKamnitsasetal.,20173DCNN
a
b
Subject1 Subject2
Accuracy
012Epochs
012Epochs
012Epochs
012Epochs
Sensi*vity
Specificity
DSC
§ Deep convolutional neural networks can provide high accuracy in lesion segmentation (Figure 3). We propose a “neurologist-inspired” approach where the neural network exploits the lesion-induced asymmetries between brain hemispheres (Figure 1). § Segmentations in the test set show that there is a high overlap between the output of the network and the manually drawn mask (Figure 4) Compared to the state-of-the-art networks, our neural network does better at identifying the different parts of the lesions (e.g. around the ventricles and bilateral parts) (Figure 4).
§ By incorporating more data, accuracy will increase thus rendering our tool an easy, automated method that can assist with lesion demarcation.
Summary
1) Vaidya AR, Pujara MS, Petrides M, Murray EA, Fellows LK (2019) Lesion Studies in Contemporary Neuroscience. 23:653-671 2) Rorden C, Karnath HO (2004) Using human brain lesions to infer function: a relic from a past era in the fMRI age? Nat Rev Neurosci 5:813-819 3) Griffis JC, Allendorfer JB, Szaflarski JP (2016) Voxel-based Gaussian naïve Bayes classification of ischemic stroke lesions in individual T1-weighted MRI scans. J Neurosci Methods 257:97-108 4) Seghier ML, Ramlackhansingh A, Crinion J, Leff AP, Price CJ (2008) Lesion identification using unified segmentation-normalisation models and fuzzy clustering. NeuroImage 41:1253-1266 5) Pustina et al. (2016) Automated segmentation of chronic stroke lesions using LINDA: Lesion identification with neighborhood data analysis. Hum Brain Mapp 37:1405-1421 6) Kamnitsas K et al. (2017) Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical Image Analysis 36:61-78 7) Krähenbühl P, Koltun V (2011). Efficient inference in fully connected CRFs with gaussian edge potentials. Adv. Neural Inf. Process. Syst.
References
Test segmentations
Abbrevia(onsL=Lesion,NL=NonLesion,TL=TrueLesion,TNL=TrueNonLesion
Figure 2 Network output and post cleaning using CRFs. After obtaining the input we use CRFs to clean the segmentaion based onsmoothing and appearance kernels.
