Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study:...
Transcript of Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study:...
![Page 1: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/1.jpg)
Convolutional Neural Networks II
CS194: Image Manipulation, Comp. Vision, and Comp. PhotoAlexei Efros, UC Berkeley, Spring 2020
![Page 2: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/2.jpg)
Lecture 7 - 27 Jan 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 20162
Case Study: LeNet-5[LeCun et al., 1998]
Conv filters were 5x5, applied at stride 1Subsampling (Pooling) layers were 2x2 applied at stride 2i.e. architecture is [CONV-POOL-CONV-POOL-CONV-FC]
![Page 3: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/3.jpg)
Andrew NgImageNet Challenge (1000 object classes), Fei-Fei et al.
![Page 4: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/4.jpg)
Lecture 7 - 27 Jan 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 20164
Case Study: AlexNet[Krizhevsky et al. 2012]
Input: 227x227x3 images
First layer (CONV1): 96 11x11 filters applied at stride 4=>Q: what is the output volume size? Hint: (227-11)/4+1 = 55
![Page 5: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/5.jpg)
Lecture 7 - 27 Jan 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 20165
Case Study: AlexNet[Krizhevsky et al. 2012]
Input: 227x227x3 images
First layer (CONV1): 96 11x11 filters applied at stride 4=>Output volume [55x55x96]
Q: What is the total number of parameters in this layer?
![Page 6: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/6.jpg)
Lecture 7 - 27 Jan 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 20166
Case Study: AlexNet[Krizhevsky et al. 2012]
Input: 227x227x3 images
First layer (CONV1): 96 11x11 filters applied at stride 4=>Output volume [55x55x96]Parameters: (11*11*3)*96 = 35K
![Page 7: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/7.jpg)
Lecture 7 - 27 Jan 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 20167
Case Study: AlexNet[Krizhevsky et al. 2012]
Input: 227x227x3 imagesAfter CONV1: 55x55x96
Second layer (POOL1): 3x3 filters applied at stride 2
Q: what is the output volume size? Hint: (55-3)/2+1 = 27
![Page 8: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/8.jpg)
Lecture 7 - 27 Jan 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 20168
Case Study: AlexNet[Krizhevsky et al. 2012]
Input: 227x227x3 imagesAfter CONV1: 55x55x96
Second layer (POOL1): 3x3 filters applied at stride 2Output volume: 27x27x96
Q: what is the number of parameters in this layer?
![Page 9: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/9.jpg)
Lecture 7 - 27 Jan 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 20169
Case Study: AlexNet[Krizhevsky et al. 2012]
Input: 227x227x3 imagesAfter CONV1: 55x55x96
Second layer (POOL1): 3x3 filters applied at stride 2Output volume: 27x27x96Parameters: 0!
![Page 10: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/10.jpg)
Lecture 7 - 27 Jan 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 201610
Case Study: AlexNet[Krizhevsky et al. 2012]
Input: 227x227x3 imagesAfter CONV1: 55x55x96After POOL1: 27x27x96...
![Page 11: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/11.jpg)
Lecture 7 - 27 Jan 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 201611
Case Study: AlexNet[Krizhevsky et al. 2012]
Full (simplified) AlexNet architecture:[227x227x3] INPUT[55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0[27x27x96] MAX POOL1: 3x3 filters at stride 2[27x27x96] NORM1: Normalization layer[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2[13x13x256] MAX POOL2: 3x3 filters at stride 2[13x13x256] NORM2: Normalization layer[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1[13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1[6x6x256] MAX POOL3: 3x3 filters at stride 2[4096] FC6: 4096 neurons[4096] FC7: 4096 neurons[1000] FC8: 1000 neurons (class scores)
![Page 12: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/12.jpg)
Lecture 7 - 27 Jan 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 201612
Case Study: AlexNet[Krizhevsky et al. 2012]
Full (simplified) AlexNet architecture:[227x227x3] INPUT[55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0[27x27x96] MAX POOL1: 3x3 filters at stride 2[27x27x96] NORM1: Normalization layer[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2[13x13x256] MAX POOL2: 3x3 filters at stride 2[13x13x256] NORM2: Normalization layer[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1[13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1[6x6x256] MAX POOL3: 3x3 filters at stride 2[4096] FC6: 4096 neurons[4096] FC7: 4096 neurons[1000] FC8: 1000 neurons (class scores)
Details/Retrospectives: - first use of ReLU- used Norm layers (not common anymore)- heavy data augmentation- dropout 0.5- batch size 128- SGD Momentum 0.9- Learning rate 1e-2, reduced by 10manually when val accuracy plateaus- L2 weight decay 5e-4- 7 CNN ensemble: 18.2% -> 15.4%
![Page 13: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/13.jpg)
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 201613
(slide from Kaiming He’s recent presentation)
![Page 14: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/14.jpg)
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 201614
Case Study: ResNet[He et al., 2015]
224x224x3
spatial dimension only 56x56!
![Page 15: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/15.jpg)
Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 201615
Case Study: ResNet [He et al., 2015]
![Page 16: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/16.jpg)
Lecture 7 - 27 Jan 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 201616
“You need a lot of a data if you want to train/use CNNs”
![Page 17: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/17.jpg)
Lecture 7 - 27 Jan 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 201617
Transfer Learning
“You need a lot of a data if you want to train/use CNNs”
![Page 18: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/18.jpg)
Deep Features & their Embeddings
![Page 19: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/19.jpg)
The Unreasonable Effectiveness of Deep Features
Classes separate in the deep representations and transfer to many tasks.[DeCAF] [Zeiler-Fergus]
![Page 20: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/20.jpg)
Can be used as a generic feature (“CNN code” = 4096-D vector before classifier)
query image nearest neighbors in the “code” space
![Page 21: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/21.jpg)
ImageNet + Deep Learning
Beagle
- Image Retrieval- Detection (RCNN)- Segmentation (FCN)- Depth Estimation- …
![Page 22: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/22.jpg)
ImageNet + Deep Learning
Beagle
Pose?
Boundaries?Geometry?Parts?
Materials?
![Page 23: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/23.jpg)
Lecture 7 - 27 Jan 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 201623
Transfer Learning with CNNs
1. Train on Imagenet
![Page 24: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/24.jpg)
Lecture 7 - 27 Jan 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 201624
Transfer Learning with CNNs
1. Train on Imagenet
2. If small dataset: fix all weights (treat CNN as fixed feature extractor), retrain only the classifier
i.e. swap the Softmax layer at the end
![Page 25: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/25.jpg)
Lecture 7 - 27 Jan 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 201625
Transfer Learning with CNNs
1. Train on Imagenet
2. If small dataset: fix all weights (treat CNN as fixed feature extractor), retrain only the classifier
i.e. swap the Softmax layer at the end
3. If you have medium sized dataset, “finetune”instead: use the old weights as initialization, train the full network or only some of the higher layers
retrain bigger portion of the network, or even all of it.
![Page 26: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/26.jpg)
Lecture 7 - 27 Jan 2016Fei-Fei Li & Andrej Karpathy & Justin JohnsonFei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 27 Jan 201626
Transfer Learning with CNNs
1. Train on Imagenet
2. If small dataset: fix all weights (treat CNN as fixed feature extractor), retrain only the classifier
i.e. swap the Softmax layer at the end
3. If you have medium sized dataset, “finetune”instead: use the old weights as initialization, train the full network or only some of the higher layers
retrain bigger portion of the network, or even all of it.
tip: use only ~1/10th of the original learning rate in finetuning to player, and ~1/100th on intermediate layers
![Page 27: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/27.jpg)
27
Learning an Embedding
CNN
Embeddingshared representation
CNNShared W
Image 1 Image 2
![Page 28: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/28.jpg)
28
CNN
Matchingshared representation
CNNShared W
Image 1 Image 2
Learning an Embedding
![Page 29: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/29.jpg)
Siamese Network w/ Contrastive Loss
Siamese Architecture[Chopra 2005, Hadsell 2006]
![Page 30: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/30.jpg)
L E A R N I N G V I S U A L S I M I L A R I T YF O R P R O D U C T D E S I G N W I T HC O N V O L U T I O N A L N E U R A L N E T W O R K SS E A N B E L L A N D K A V I T A B A L AC O R N E L L U N I V E R S I T Y
![Page 31: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/31.jpg)
T H E P R O B L E M
Name: ”Great Bowl O’Fire Sculptural Fire Bowl”
(1) “What is this?” (2) “Where is it used?”
Category: Fire pitSold by: John T. Unger, LLC
![Page 32: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/32.jpg)
T H E P R O B L E M
(1) “What is this?” (2) “Where is it used?”
Challenge: determine whether these are the same product(different resolution, viewpoint, color, lighting, occlusions)
Name: ”Great Bowl O’Fire Sculptural Fire Bowl”Category: Fire pitSold by: John T. Unger, LLC
![Page 33: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/33.jpg)
T W O K I N D S O F I M A G E SIconic In context
(From a product website) (Cropped from a scene photo)
![Page 34: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/34.jpg)
P R O J E C T I N G I N T O A J O I N T E M B E D D I N G
Embedding
Iconic In context
![Page 35: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/35.jpg)
S E A R C H U S I N G T H E E M B E D D I N G
Embedding
“What is it?”
Name: Hemel RingCategory: Hanging lightSold by: Holly Hunt
![Page 36: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/36.jpg)
S E A R C H U S I N G T H E E M B E D D I N G
Embedding
“Where is it used?”
![Page 37: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/37.jpg)
Embedding
C O N T R A S T I V E L O S S : P O S I T I V E E X A M P L E
Parameters θ
Iconic (same)
In context
CNN
CNN
Loss Lp
xp
xq
![Page 38: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/38.jpg)
Embedding
C O N T R A S T I V E L O S S : N E G A T I V E E X A M P L E
In context
Loss Ln
Margin m
Iconic (different)
CNN
CNN
Parameters θxq
xn
![Page 39: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/39.jpg)
C O N T R A S T I V E L O S S : A L L T O G E T H E R
[Chopra 2005, Hadsell 2006]
Minimize L(θ) with stochastic gradient descent and momentum
Margin
![Page 40: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/40.jpg)
T R A I N I N G P I P E L I N E
StochasticGradientDescent
CNN Embedding
Image pairsCNN Parameters
θ
Image database
![Page 41: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/41.jpg)
R E S U L T S : “ W H A T I S I T ? ”
In context
![Page 42: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/42.jpg)
R E S U L T S : “ W H A T I S I T ? ”
In context Iconic Top 4 results:
![Page 43: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/43.jpg)
R E S U L T S : “ W H A T I S I T ? ”
In context
![Page 44: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/44.jpg)
R E S U L T S : “ W H A T I S I T ? ”
In context Iconic Top 4 results:
![Page 45: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/45.jpg)
R E S U L T S : “ W H A T I S I T ? ”
In context
![Page 46: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/46.jpg)
R E S U L T S : “ W H A T I S I T ? ”
In context Iconic Top 4 results:
![Page 47: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/47.jpg)
C O M P A R I S O N : T R A I N E D O N L Y O N C A T E G O R I E S
IconicIn context Top 4 results:
![Page 48: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/48.jpg)
IconicIn context Top 4 results:
C O M P A R I S O N : T R A I N E D O N L Y O N I M A G E N E T
![Page 49: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/49.jpg)
R E S U L T S : F A I L U R E C A S E
In context
![Page 50: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/50.jpg)
R E S U L T S : F A I L U R E C A S E
In context Iconic Top 4 results:
![Page 51: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/51.jpg)
“Maskros Pendant Lamp”
R E S U L T S : “ W H E R E I S I T U S E D ? ”
![Page 52: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/52.jpg)
R E S U L T S : “ W H E R E I S I T U S E D ? ”
"LEM Piston Stool | Design Within Reach”
![Page 53: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/53.jpg)
S E A R C H I N G A C R O S S C A T E G O R I E S
![Page 54: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/54.jpg)
Color distribution cross-entropy loss with colorfulness enhancing term.
Zhang et al. 2016
[Zhang, Isola, Efros, ECCV 2016]
Designing loss functionsInput Ground truth
![Page 55: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/55.jpg)
![Page 56: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/56.jpg)
Image colorization
Cross entropy loss, with colorfulness term
“semantic feature loss” (VGG feature covariance matching objective)
[Johnson et al. 2016]
Super-resolution[Zhang et al. 2016]
Designing loss functions
![Page 57: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/57.jpg)
Universal loss?
… …
![Page 58: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/58.jpg)
…
Generated vs Real(classifier)
[Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville, Bengio 2014]
Generative Adversarial Network(GANs)
Real photos
Generated images
…
…
![Page 59: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/59.jpg)
Generator
[Goodfellow et al., 2014]
![Page 60: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/60.jpg)
G tries to synthesize fake images that fool D
D tries to identify the fakes
Generator Discriminator
real or fake?
[Goodfellow et al., 2014]
![Page 61: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/61.jpg)
fake (0.9)
real (0.1)
[Goodfellow et al., 2014]
![Page 62: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/62.jpg)
G tries to synthesize fake images that foolD:
real or fake?
[Goodfellow et al., 2014]
![Page 63: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/63.jpg)
G tries to synthesize fake images that fool the bestD:
real or fake?
[Goodfellow et al., 2014]
![Page 64: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/64.jpg)
Loss Function
G’s perspective: D is a loss function.
Rather than being hand-designed, it is learned.
[Isola et al., 2017][Goodfellow et al., 2014]
+ L1
![Page 65: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/65.jpg)
real or fake?
[Goodfellow et al., 2014]
![Page 66: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/66.jpg)
real!(“Aquarius”)
[Goodfellow et al., 2014]
![Page 67: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/67.jpg)
real or fake pair ?
[Goodfellow et al., 2014][Isola et al., 2017]
![Page 68: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/68.jpg)
real or fake pair ?
[Goodfellow et al., 2014][Isola et al., 2017]
![Page 69: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/69.jpg)
fake pair
[Goodfellow et al., 2014][Isola et al., 2017]
![Page 70: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/70.jpg)
real pair
[Goodfellow et al., 2014][Isola et al., 2017]
![Page 71: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/71.jpg)
real or fake pair ?
[Goodfellow et al., 2014][Isola et al., 2017]
![Page 72: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/72.jpg)
BW → Color
Data from [Russakovsky et al. 2015]
![Page 73: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/73.jpg)
BW → Color
Data from [Russakovsky et al. 2015]
![Page 76: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/76.jpg)
Labels → FacadesInput Output
Data from [Tylecek, 2013]
![Page 77: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/77.jpg)
Labels → FacadesInput Output Input Output
Data from [Tylecek, 2013]
![Page 78: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/78.jpg)
Day → NightInput Output Input Output Input Output
Data from [Laffont et al., 2014]
![Page 79: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/79.jpg)
Thermal → RGB
![Page 80: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/80.jpg)
Edges → ImagesInput Output Input Output Input Output
Edges from [Xie & Tu, 2015]
![Page 81: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/81.jpg)
Sketches → ImagesInput Output Input Output Input Output
Trained on Edges → ImagesData from [Eitz, Hays, Alexa, 2012]
![Page 82: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/82.jpg)
![Page 83: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/83.jpg)
#edges2cats [Christopher Hesse]
Ivy Tasi @ivymyt
Vitaly Vidmirov @vvid
@gods_tail
@ka92
![Page 84: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/84.jpg)
Twitter-driven research: #pix2pix
Bertrand Gondouin @bgondouin
Brannon Dorsey @brannondorsey
Mario Klingemann @quasimondo
![Page 85: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/85.jpg)
© Memo Akten, “Learning to See: Gloomy Sunday”
![Page 87: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/87.jpg)
![Page 88: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/88.jpg)
“Do as I Do”
OpenPose
pix2pix
![Page 89: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/89.jpg)
Everybody Dance NowCaroline Chan, Shiry Ginosar, Tinghui Zhou, Alexei A. EfrosUC Berkeley
Source Subject Target Subject
![Page 90: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/90.jpg)
Results
https://www.youtube.com/watch?v=PCBTZh41Ris&feature=youtu.be
![Page 91: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/91.jpg)
CEO: our own Dr. Tinghui Zhou
![Page 92: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/92.jpg)
…
Paired training examples Unpaired training examples
… …
![Page 93: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/93.jpg)
…
CycleGAN, or “there and back aGAN”
[Zhu*, Park*, Isola, Efros. ICCV 2017]
……
![Page 94: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/94.jpg)
Cycle-Consistency LossG(x) F(G x )x
F G x − x 1
![Page 95: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/95.jpg)
G(x) F(G x )x F(y) G(F x )𝑦𝑦
Cycle-Consistency Loss
F G x − x 1 G F y − 𝑦𝑦 1
![Page 96: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/96.jpg)
![Page 97: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/97.jpg)
Video
![Page 98: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/98.jpg)
![Page 99: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/99.jpg)
![Page 100: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/100.jpg)
![Page 101: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/101.jpg)
Collection Style Transfer
Van Gogh
Cezanne
Monet
Ukiyo-e
Photograph© Alexei Efros
![Page 102: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/102.jpg)
![Page 103: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/103.jpg)
CG to RealGrand Theft Auto
![Page 104: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/104.jpg)
Real to CG
![Page 105: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/105.jpg)
Shallower depth of field
![Page 106: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/106.jpg)
Failure case
![Page 107: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/107.jpg)
A Neural Algorithmof Artistic Style
Gatys, Ecker, Bethge (arXiv 2015)
![Page 108: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/108.jpg)
![Page 109: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/109.jpg)
Van Gogh (1889)
![Page 110: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/110.jpg)
Picasso (1910)
![Page 111: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/111.jpg)
Munch (1893)
![Page 112: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/112.jpg)
Turner (1805)
![Page 113: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/113.jpg)
Kandinsky (1913)
![Page 114: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/114.jpg)
Early Vision Texture Models
2.37.13.8
Heeger & Bergen (1995) Portilla & Simoncelli (2000)
Linear filter bank
![Page 115: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/115.jpg)
Heeger & Bergen, SIGGRAPH‘95Start with a noise image as output Main loop:
• Match pixel histogram of output image to input
• Decompose input and output images using multi-scale filter bank (Steerable Pyramid)
• Match subband histograms of input and output pyramids
• Reconstruct input and output images (collapse the pyramids)
Heeger, Bergen, Pyramid-based texture analysis/synthesis, SIGGRAPH 1995
![Page 116: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/116.jpg)
Multi-scale filter decomposition
Filter bank
Input image
![Page 117: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/117.jpg)
Filter response histograms
![Page 118: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/118.jpg)
Simoncelli & Portilla ’98+
Match joint histograms of pairs of filter responses at adjacent spatial locations, orientations, and scales.
Optimize using repeated projections onto statistical constraint sufraces
![Page 119: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/119.jpg)
Texture SynthesisImage Space Model Space
Images with equalmodel response
Portilla & Simoncelli (2000)
![Page 120: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/120.jpg)
Convolutional Neural Network Texture Model
2.37.13.8
Convolutional Neural Network
Gatys et al. (NIPS 2015)
![Page 121: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/121.jpg)
CNN - Multiscale Filter Bank
conv1_1
pool4pool3
pool2
pool1
64
# features
64
128
256
512
![Page 122: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/122.jpg)
CNN - Texture Features
![Page 123: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/123.jpg)
CNN - Texture Features
64
# features
64
128
256
512
Gram Matrices
![Page 124: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/124.jpg)
Texture Synthesis
![Page 125: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/125.jpg)
Texture Synthesis
![Page 126: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/126.jpg)
Texture Synthesis
![Page 127: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/127.jpg)
Texture Synthesis
![Page 128: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/128.jpg)
Texture Synthesis
![Page 129: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/129.jpg)
Texture Synthesis
![Page 130: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/130.jpg)
Texture Synthesis
![Page 131: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/131.jpg)
Test Julesz’ Conjecture
![Page 132: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/132.jpg)
Test Julesz’ Conjecture
![Page 133: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/133.jpg)
CNN - Texture Synthesis
Gatys et al. (NIPS 2015)
![Page 134: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/134.jpg)
Artistic Style Transfer
![Page 135: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/135.jpg)
Artistic Style Transfer
![Page 136: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/136.jpg)
Artistic Style Transfer
![Page 137: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/137.jpg)
Artistic Style Transfer
![Page 138: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/138.jpg)
Artistic Style Transfer
![Page 139: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/139.jpg)
Artistic Style Transfer
![Page 140: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/140.jpg)
Artistic Style Transfer
![Page 141: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/141.jpg)
Artistic Style Transfer
![Page 142: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/142.jpg)
Artistic Style Transfer
![Page 143: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/143.jpg)
Artistic Style Transfer
![Page 144: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/144.jpg)
Artistic Style Transfer
![Page 145: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/145.jpg)
Artistic Style Transfer
![Page 146: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/146.jpg)
Relative Weighting of Content and Style1e-4
1e-2 1e-1
1e-3
![Page 147: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/147.jpg)
Different Reconstruction Layers
Conv2_2 Conv4_2
![Page 148: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/148.jpg)
Different Reconstruction Layers
Conv2_2 Conv4_2
![Page 149: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/149.jpg)
Different Reconstruction Layers
Conv2_2Original Conv4_2
![Page 150: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/150.jpg)
General Style Transfer
![Page 151: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/151.jpg)
General Style Transfer
![Page 152: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/152.jpg)
![Page 153: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/153.jpg)
![Page 154: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/154.jpg)
![Page 155: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/155.jpg)
![Page 156: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/156.jpg)
![Page 157: Convolutional Neural Networks IIcs194-26/sp20/Lectures/...Lecture 7 - 6 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Input: 227x227x3 images First layer (CONV1): 96 11x11](https://reader033.fdocuments.in/reader033/viewer/2022060907/60a17c269c285f3f6c1eeecf/html5/thumbnails/157.jpg)