Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature...
Transcript of Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature...
![Page 1: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/1.jpg)
MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning
on Fashion-MNIST
Jason WU, Peng XU, Nayeon LEE
08.Mar.2018
![Page 2: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/2.jpg)
Introduction: Fashion-MNIST Dataset
Material: https://github.com/zalandoresearch/fashion-mnist 2
● 60,000 training examples and a 10,000 testing examples● Each example is a 28x28 grayscale image● 10 classes
● Zalando et al. intend Fashion-MNIST to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms.
![Page 3: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/3.jpg)
Why Fashion-MNIST?
Material: https://github.com/zalandoresearch/fashion-mnist 3
Quoted from their website:
● MNIST is too easy. Convolutional nets can achieve 99.7% on MNIST. Classic machine learning algorithms can also achieve 97% easily. Most pairs of MNIST digits can be distinguished pretty well by just one pixel.
● MNIST is overused. In this April 2017 Twitter thread, Google Brain research scientist and deep learning expert Ian Goodfellow calls for people to move away from MNIST.
● MNIST can not represent modern CV tasks, as noted in this April 2017 Twitter thread, deep learning expert/Keras author François Chollet.
![Page 4: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/4.jpg)
Introduction: Fashion-MNIST Dataset
Material: https://github.com/zalandoresearch/fashion-mnist 4
![Page 5: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/5.jpg)
How to import?
Material: https://github.com/zalandoresearch/fashion-mnist 5
● Loading data with Python (requires NumPy)○ Use utils/mnist_reader from https://github.com/zalandoresearch/fashion-mnist
● Loading data with Tensorflow○ Make sure you have downloaded the data and placed it in data/fashion.
Otherwise, Tensorflow will download and use the original MNIST.
![Page 6: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/6.jpg)
Feature Extraction
6
● We compared three different feature representation:○ Raw pixel features○ ScatNet features○ Pretrained ResNet18 last-layer features
![Page 7: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/7.jpg)
Feature Extraction(1): ScatNet
7
● The maximum scale of the transform: J=3● The maximum scattering order: M=2● The number of different orientations: L=1
The dimension of the final features is 176
https://arxiv.org/pdf/1203.1513.pdf
![Page 8: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/8.jpg)
Feature Extraction(2): ResNet
8
● Used pretrained 18 layers Residual Network from ImageNet ● We take the hidden representation right before the last fully-connected layer,
which has the dimension of 512
https://arxiv.org/abs/1512.03385
![Page 9: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/9.jpg)
Data Visualization
9
● Then, we visualized three different feature representation by the following 4 different dimension reduction methods:○ Principal Component Analysis (PCA)○ Locally Linear Embedding (LLE)○ t-Distributed Stochastic Neighbor Embedding (t-SNE)○ Uniform Manifold Approximation and Projection (UMAP)
![Page 10: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/10.jpg)
Data Visualization
10
● Then, we visualized three different feature representation by the following 4 different dimension reduction methods:○ Principal Component Analysis (PCA)○ Locally Linear Embedding (LLE)○ t-Distributed Stochastic Neighbor Embedding (t-SNE)○ Uniform Manifold Approximation and Projection (UMAP)
![Page 11: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/11.jpg)
Data Visualization (1): PCA
11
Raw Features ScatNet Features ResNet Features
● Normalization, Covariance Matrix, SVD, Project to top K eigen-vectors● Linear dimension reduction methods:
○ not that obviously difference between labels
![Page 12: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/12.jpg)
Data Visualization (2): LLE
12http://www.robots.ox.ac.uk/~az/lectures/ml/lle.pdf
![Page 13: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/13.jpg)
Data Visualization (2): LLE
13https://pdfs.semanticscholar.org/6adc/19cf4404b9f1224a1a027022e40ac77218f5.pdf
![Page 14: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/14.jpg)
Data Visualization (2): LLE
14
Raw Features ScatNet Features ResNet Features
● Non-linear dimension reduction that is good at capture “streamline” structure
![Page 15: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/15.jpg)
Data Visualization (3): t-SNE
15
● Use Gaussian pdf to approximate the high dimension distribution● Use t distribution for low dimension distribution● Use KL Divergence as cost function for gradient descent
http://www.jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf
![Page 16: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/16.jpg)
Data Visualization (3): t-SNE
16
Raw Features ScatNet Features ResNet Features
● Block-like visualization due to the gaussian approximation
![Page 17: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/17.jpg)
Data Visualization (4): UMAP
17
https://arxiv.org/pdf/1802.03426.pdf
● The algorithm is founded on three assumptions about the data○ The Riemannian metric is locally constant (or can be approximated);○ The data is uniformly distributed on Riemannian manifold;○ The manifold is locally connected.
![Page 18: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/18.jpg)
Data Visualization (4): UMAP
18
Raw Features ScatNet Features ResNet Features
● Much more Faster in training process, which implies it can handle large datasets and high dimensional data
https://github.com/lmcinnes/umap
![Page 19: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/19.jpg)
Any News from Visualization?
19
● Is there different patterns between different visualization methods?● Is there clear separation of different classes?● Is there any groups that tend to cluster together?
● Let’s look closer!
![Page 20: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/20.jpg)
20
PCA
LLE
![Page 21: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/21.jpg)
21
t-SNE
UMAP
![Page 22: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/22.jpg)
22
Sneaker, Sandal, Ankle boot
![Page 23: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/23.jpg)
23
PCA
LLE
![Page 24: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/24.jpg)
24
t-SNE
UMAP
![Page 25: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/25.jpg)
25
Trouser
![Page 26: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/26.jpg)
26
PCA
LLE
![Page 27: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/27.jpg)
27
t-SNE
UMAP
![Page 28: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/28.jpg)
28
Bag
![Page 29: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/29.jpg)
29
PCA
LLE
![Page 30: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/30.jpg)
30
t-SNE
UMAP
![Page 31: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/31.jpg)
31
T-Shirt,Pullover,Dress,Coat,Shirt
![Page 32: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/32.jpg)
Simple Classification Models
32
● Logistic Regression● Linear Discriminant Analysis ● Support Vector Machine● Random Forest● ...
![Page 33: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/33.jpg)
Simple Classification Models
33
● Logistic Regression
![Page 34: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/34.jpg)
Simple Classification Models
34
● Linear Discriminant Analysis○ maximize between class covariance○ minimize within class covariance
![Page 35: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/35.jpg)
Simple Classification Models
35
● Linear Support Vector Machine○ Hard-margin
○ Soft-margin
![Page 36: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/36.jpg)
Simple Classification Models
36
● Random Forest● An ensemble learning method that construct multiple decision trees● Bagging (Bootstrap aggregating)
![Page 37: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/37.jpg)
Simple Classification Results
37
![Page 38: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/38.jpg)
Simple Classification Results
38
![Page 39: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/39.jpg)
Simple Classification Results
39http://fashion-mnist.s3-website.eu-central-1.amazonaws.com
![Page 40: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/40.jpg)
Fine-Tuning the ResNet
40
● The best accuracy now is 93.42% ● Seems like transfer learning in our case is not that promising.
![Page 41: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/41.jpg)
Other Existing Models...
41
![Page 42: Feature Extraction and Transfer Learning on Fashion-MNIST · MATH6380o Mini-Project 1 Feature Extraction and Transfer Learning on Fashion-MNIST Jason WU, Peng XU, Nayeon LEE 08.Mar.2018](https://reader030.fdocuments.in/reader030/viewer/2022040606/5eb63d6fad537c34be529be1/html5/thumbnails/42.jpg)
Q/AHong Kong University of Science and Technology
Electronic & Computer EngineeringHuman Language Technology Center (HLTC)
Jason WU, Peng XU, Nayeon LEE