General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final...
Transcript of General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final...
![Page 1: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/1.jpg)
General Approaches of Histopathological Image
Classification using Convolutional Neural
Networks
Jerome Cheng M.D.
![Page 2: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/2.jpg)
Disclosures
none
![Page 3: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/3.jpg)
Outline
Background on Convolutional Neural Networksdescriptionapplicationslayers
Image classification strategiestrain from scratch transfer learning
ToolsOrangeKeras
![Page 4: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/4.jpg)
Convolutional Neural Networks
class of deep, feed forward artificial neural networks usually composed of convolution, pooling, fully connected layers
most commonly applied in pathology for image classification and segmentation tasks
Inspired by Dr. Hubel and Dr.Wiesel’s work on the cat visual system in the 50’s and 60’s that showed a hierarchical arrangement of neurons and some neurons were excited by lines in a particular orientation in a specific location
![Page 5: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/5.jpg)
Artificial Neural Network Example
a1
a3
a2 𝚺 Output
w1
w2
w3
Rel
u
Activation
Function
+Bia
s
a1w1+a2w2+a3w3 + bias → Relu → Output
![Page 6: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/6.jpg)
Convolutional
Layer
Convolutional
Layer
Convolutional
Layer
Pooling Layer
Pooling Layer
Fully Connected
Layer
Fully Connected Layer
![Page 7: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/7.jpg)
![Page 8: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/8.jpg)
Acceleration of CNN Progress and Adaptation
Faster PCs and GPUs
Publicly available image datasets
Open source CNN deep learning frameworks
![Page 9: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/9.jpg)
ImageNetOnline database containing more than 14 million images belonging to 20,000 categories
Large training dataset is important in deep learning
- Several pre-trained CNN models where trained on ImageNet
Accelerated the progress of CNN research
Hosted yearly competitions
Many publicly available pre-trained CNN models were trained on ImageNet
http://www.image-net.org/
![Page 10: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/10.jpg)
ImageNet - Images
![Page 11: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/11.jpg)
Layers
Convolutional
Pooling
Fully connected
Activation layers
Dropout layer
Regularization layers
Low level
features
High level
features
![Page 12: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/12.jpg)
Convolutional Neural Network Layers
![Page 13: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/13.jpg)
Training
Loss function - measure the difference between the predicted value and the expected value
Weights are adjusted in the convolution and fully connected (dense) layers to minimize the loss function
![Page 14: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/14.jpg)
Pre-trained CNN architectures
VGG-16
VGG-19
Inception
XCeption
MobileNet
![Page 15: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/15.jpg)
VGG-16 architecture
Simonyan, Karen, and Andrew Zisserman. "Very
deep convolutional networks for large-scale image
recognition." arXivpreprint
![Page 16: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/16.jpg)
Applications of CNN in Pathology
Image Classification
Image Segmentation
Object localization and classification
![Page 17: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/17.jpg)
Image Classification
Dataset Source: Levenson RM, Krupinski EA, Navarro VM, Wasserman EA. Pigeons (Columba livia) as trainable observers of
pathology and radiology breast cancer images. PLoSOne 2015;10:e0141357.
![Page 18: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/18.jpg)
Image Segmentation
Ground Truth Prediction
![Page 19: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/19.jpg)
Object Detection and Classification
Hung J, Carpenter A (2017) Applying Faster R-CNN for Object Detection on Malaria Images, IEEE
Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 808–813
![Page 20: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/20.jpg)
Classification Strategies (CNN)
Train from scratch
- Weights from all layers are updated
Transfer learning
- Freeze lower layers - Only weights from some layers are updated
- Combine pre-classification layer output with machine learning algorithms
- None of the weights are updated
![Page 21: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/21.jpg)
Transfer Learning - Freezing Layers
Weights are updated only on selected layers
Information learned from a previous dataset (ImageNet) is preserved
Speeds up training
![Page 22: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/22.jpg)
Transfer Learning - Combine Output with Another Machine Learning Algorithm
The final classification layer of the pre-trained network should not be included
Ex. an image processed through VGG 16 = 4096 features
4096 features + image label can be trained using various machine learning algorithms like Random Forest, logistic regression, and SVM
None of the weight parameters are updated during training
![Page 23: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/23.jpg)
Steps – Classifying Images with CNN
Scan slides
Create and label tiles - ex. 224x224 or 299x299 pixels
Training and Validation set
Augmentation
Choose or Create a Model
Model Training
Prediction
![Page 24: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/24.jpg)
Augmentation
Rotation
Shear
Channel Shift
Scaling
Height shift
Width shift
![Page 25: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/25.jpg)
DemsarJ, CurkT, ErjavecA, GorupC, HocevarT, MilutinovicM, MozinaM, PolajnarM, ToplakM, StaricA, StajdoharM, UmekL, ZagarL, ZbontarJ,
ZitnikM, ZupanB (2013) Orange: Data Mining Toolbox in Python. Journal of Machine Learning Research 14(Aug):2349−2353.
Chollet F. Keras [Internet] San Francisco (CA): GitHub, Inc.; 2017. [cited 2019 Apr 26]. Available from: https://github.com/keras-team/keras.
![Page 26: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/26.jpg)
Orange (Biolab) - https://orange.biolab.si
![Page 27: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/27.jpg)
Data preparation
Create image tiles from the digital slide or image (224 x 224 for VGG 16)
Create a folder with subfolders for each image Class
Ex. Training Set
Cancer
Benign
![Page 28: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/28.jpg)
![Page 29: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/29.jpg)
![Page 30: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/30.jpg)
![Page 31: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/31.jpg)
Orange - loading images
![Page 32: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/32.jpg)
Orange - Image Embedding
![Page 33: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/33.jpg)
Orange - Hardware (image embeddings)
Cluster of 8 machines
6 machines- 2 processors with 4 cores each (Intel Xenon CPU), 32 GB RAM
2 machines - 2 processors with 8 cores (Intel Xeon e5-2660), 157 GB RAM
Embedders are automatically assigned to available CPU cores
Soon: switch to an improved backend with a combination of CPUs and GPUs, with the addition of 3 TitanX GPUs (12 GB ram each)
![Page 34: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/34.jpg)
![Page 35: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/35.jpg)
Orange - Test and Score
![Page 36: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/36.jpg)
Orange - data visualization
Dimensional reduction + scatter plot
Hierarchical clustering
![Page 37: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/37.jpg)
Keras
High level neural network library that works on top of TensorFlow, CNTK, or Theano
Written in Python
CNN models can be built and trained with very little code
Works with CPU and GPUs
![Page 38: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/38.jpg)
![Page 39: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/39.jpg)
![Page 40: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/40.jpg)
![Page 41: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/41.jpg)
Summary
Convolutional Neural Networks can be used to classify and segment histopathological images
Transfer learning applies knowledge learned from a different dataset onto a new image subject matter
Orange (Biolab) is a GUI based machine learning tool that can be used for transfer learning
Keras is a high level framework written in Python for deep learning
![Page 42: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/42.jpg)
References
Chollet, François. Deep Learning with Python. Shelter Island, New York : Manning Publications Co., 2018.
Cheng J. Convolutional neural networks. PathologyOutlines.com website. http://www.pathologyoutlines.com/topic/informaticsconvnet.html. Accessed March 25th, 2019.
Levenson RM, Krupinski EA, Navarro VM, Wasserman EA. Pigeons (Columba livia) as trainable observers of pathology and radiology breast cancer images. PLoSOne 2015;10:e0141357.
Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXivpreprint
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo,Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis,Jeffrey Ronneberger O., Fischer P., Brox T. (2015) U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Navab N., Hornegger J., Wells W., Frangi A. (eds) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. MICCAI 2015. Lecture Notes in Computer Science, vol 9351. Springer, ChamDean,Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow,Andrew Harp, Geoffrey Irving, Michael Isard, Rafal Jozefowicz, Yangqing Jia,Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Mike Schuster,Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Jonathon Shlens,Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker,Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas,Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke,Yuan Yu, and Xiaoqiang Zheng.TensorFlow: Large-scalemachine learning on heterogeneous systems,2015. Software available from tensorflow.org.
![Page 43: General Approaches of Histopathological Image Classification using … · 2019-07-27 · The final classification layer of the pre-trained network should not be included Ex. an image](https://reader033.fdocuments.in/reader033/viewer/2022043004/5f88af3a9e6303570a686c5c/html5/thumbnails/43.jpg)
References
Wurtz RH. Recounting the impact of Hubel and Wiesel. J Physiol. 2009;587(Pt 12):2817–2823. doi:10.1113/jphysiol.2009.17020
HUBEL DH, WIESEL TN. Receptive fields of single neurones in the cat's striate cortex. J Physiol. 1959;148(3):574–591.
Olga Russakovsky*, Jia Deng*, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg and Li Fei-Fei. (* = equal contribution) ImageNet Large Scale Visual Recognition Challenge.
DemsarJ, CurkT, ErjavecA, GorupC, HocevarT, MilutinovicM, MozinaM, PolajnarM, ToplakM, StaricA, StajdoharM, UmekL, ZagarL, ZbontarJ, ZitnikM, ZupanB (2013) Orange: Data Mining Toolbox in Python. Journal of Machine Learning Research 14(Aug):2349−2353.
Chollet F. Keras [Internet] San Francisco (CA): GitHub, Inc.; 2017. [cited 2019 Apr 26]. Available from: https://github.com/keras-team/keras.