Feature-Based Aggregation and Deep Reinforcement Learning ...
Deep Feature Learning for Contour Detection
Transcript of Deep Feature Learning for Contour Detection
![Page 1: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/1.jpg)
Deep Feature Learning for Contour Detection
Wei Shen (沈为) http://wei-shen.weebly.com/
Shanghai University
Shanghai University 1
Vision And Learning SEminar (VALSE)
![Page 2: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/2.jpg)
Outline 2
Contour Detection Overview
Milestones
Our Work
Discussions
![Page 3: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/3.jpg)
Introduction of Contour Detection 3
Reduce dimensionality of data
Preserve content information
Useful in applications such as
Object Detection Image Segmentation Sketch-based Image Search
![Page 4: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/4.jpg)
Introduction of Contour Detection 4
Edge Detection
the characteristic changes in brightness, color, and texture
Contour Detection
The changes in pixel ownership from one object or surface to another
easy difficult
![Page 5: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/5.jpg)
Challenge 5
To differentiate contour and non-contour is difficult, even for human beings!
![Page 6: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/6.jpg)
Challenge 6
Confusion between contour and non-contour
changes caused by cluttered textures
changes correspond to object boundaries
![Page 7: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/7.jpg)
Key Problems 7
Contour detection is usually formulated as a per-pixel classification problem
How to extract discriminative contour features?
How to learn a efficient contour classifier?
![Page 8: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/8.jpg)
Perspective 8
Year 1990 2000 2004 2008
Learning Based
Contour Detection
Learning Based
Contour Detection
Edge Detection Edge Detection
Deep Learning
Based Contour
Detection
Deep Learning
Based Contour
Detection
8
2014
![Page 9: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/9.jpg)
Outline 9
Contour Detection
Overview
Milestones Our Work
Discussions
![Page 10: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/10.jpg)
Edge Detection 10
Use local differential template to compute gradient
121
000
121
FX
101
202
101
FY
22YXG
The original Sobel edge
![Page 11: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/11.jpg)
Related Works 11
[1] J. Canny, A Computational Approach To Edge Detection, IEEE Trans. Pattern Analysis and Machine Intelligence, 8(6):679–698, 1986 [2] R. Kimmel and A.M. Bruckstein (2003) "On regularized Laplacian zero crossings and other optimal edge integrators", International Journal of Computer Vision, 53(3) pages 225-243. [3] T. Lindeberg (1993) "Discrete derivative approximations with scale-space properties: A basis for low-level feature extraction", J. of Mathematical Imaging and Vision, 3(4), pages 349--376. [4] D. Ziou and S. Tabbone (1998) "Edge detection techniques: An overview", International Journal of Pattern Recognition and Image Analysis, 8(4):537–559, 1998 [5] H. Farid and E. P. Simoncelli, Differentiation of discrete multi-dimensional signals, IEEE Trans Image Processing, vol.13(4), pp. 496--508, Apr 2004
[1] J. Canny, A Computational Approach To Edge Detection, IEEE Trans. Pattern Analysis and Machine Intelligence, 8(6):679–698, 1986 [2] R. Kimmel and A.M. Bruckstein (2003) "On regularized Laplacian zero crossings and other optimal edge integrators", International Journal of Computer Vision, 53(3) pages 225-243. [3] T. Lindeberg (1993) "Discrete derivative approximations with scale-space properties: A basis for low-level feature extraction", J. of Mathematical Imaging and Vision, 3(4), pages 349--376. [4] D. Ziou and S. Tabbone (1998) "Edge detection techniques: An overview", International Journal of Pattern Recognition and Image Analysis, 8(4):537–559, 1998 [5] H. Farid and E. P. Simoncelli, Differentiation of discrete multi-dimensional signals, IEEE Trans Image Processing, vol.13(4), pp. 496--508, Apr 2004
![Page 12: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/12.jpg)
Learning Based Contour Detection 12
Learn a classifier to combine different features
Image Boundary Cues
Model
Pb
Brightness
Color
Texture
Cue Combination
![Page 13: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/13.jpg)
Learning Based Contour Detection 13
Discriminative features
Spectral component (gPb - Arbeláez et al. PAMI 11)
Sparse code gradients (SCG – Ren&Bo, NIPS 12)
![Page 14: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/14.jpg)
Learning Based Contour Detection 14
Efficient Detector
Structured Forest (SE – Dollar&Zitnick ICCV 13, PAMI 15)
![Page 15: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/15.jpg)
Related Works 15
[1] D. Martin, C. Fowlkes, J. Malik. Learning to Detect Natural Image Boundaries Using Local Brightness, Color, and Texture Cues. IEEE Trans. Pattern Anal. Mach. Intell. 26(5): 530-549 (2004) [2] P. Arbelaez, M. Maire, C. Fowlkes, J. Malik. Contour Detection and Hierarchical Image Segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5): 898-916 (2011) [3] X. Ren. Multi-scale Improves Boundary Detection in Natural Images. ECCV (3) 2008: 533-545 [4] X. Ren, L. Bo. Discriminatively Trained Sparse Code Gradients for Contour Detection. NIPS 2012: 593-601 [5] J. Lim, C. Zitnick, P. Dollár. Sketch Tokens: A Learned Mid-level Representation for Contour and Object Detection. CVPR 2013: 3158-3165 [6] P. Dollár, C. Zitnick. Structured Forests for Fast Edge Detection. ICCV 2013: 1841-1848 [7] P. Dollár, Z. Tu, S. Belongie. Supervised Learning of Edges and Object Boundaries. CVPR (2) 2006: 1964-1971 [8] P. Arbeláez, J. Pont-Tuset, J. Barron, F. Marqués, J. Malik. Multiscale Combinatorial Grouping. CVPR 2014: 328-335
[1] D. Martin, C. Fowlkes, J. Malik. Learning to Detect Natural Image Boundaries Using Local Brightness, Color, and Texture Cues. IEEE Trans. Pattern Anal. Mach. Intell. 26(5): 530-549 (2004) [2] P. Arbelaez, M. Maire, C. Fowlkes, J. Malik. Contour Detection and Hierarchical Image Segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5): 898-916 (2011) [3] X. Ren. Multi-scale Improves Boundary Detection in Natural Images. ECCV (3) 2008: 533-545 [4] X. Ren, L. Bo. Discriminatively Trained Sparse Code Gradients for Contour Detection. NIPS 2012: 593-601 [5] J. Lim, C. Zitnick, P. Dollár. Sketch Tokens: A Learned Mid-level Representation for Contour and Object Detection. CVPR 2013: 3158-3165 [6] P. Dollár, C. Zitnick. Structured Forests for Fast Edge Detection. ICCV 2013: 1841-1848 [7] P. Dollár, Z. Tu, S. Belongie. Supervised Learning of Edges and Object Boundaries. CVPR (2) 2006: 1964-1971 [8] P. Arbeláez, J. Pont-Tuset, J. Barron, F. Marqués, J. Malik. Multiscale Combinatorial Grouping. CVPR 2014: 328-335
![Page 16: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/16.jpg)
Deep Learning Based Contour Detection
16
Use deep networks to extract contour features and predict contour map
Contour Patch Contour Patch Background
Patch Background
Patch
Feature map Feature map
![Page 17: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/17.jpg)
Related Works 17
[1] J. J. Kivinen, C. K. I.Williams, and N. Heess. Visual boundary prediction: A deep neural prediction network and quality dissection. In Proc. AISTATS, pages 512–521, 2014. [2] Y. Ganin and V. S. Lempitsky. N4-fields: Neural network nearest neighbor fields for image transforms. In Proc. ACCV, 2014. [3] W. Shen, X. Wang, Y. Wang, X. Bai, Z. Zhang. DeepContour: A Deep Convolutional Feature Learned by Positive-sharing Loss for Contour Detection. In Proc. CVPR, 2015. [4] G. Bertasius, J. Shi, L. Torresani. DeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection. In Proc. CVPR, 2015. [5] J.-J. Hwang, T.-L. Liu. Pixel-wise Deep Learning for Contour Detection. CoRR abs/1504.01989 (2015) [6] J.-J. Hwang, T.-L. Liu: Contour Detection Using Cost-Sensitive Convolutional Neural Networks. CoRR abs/1412.6857 (2014)
[1] J. J. Kivinen, C. K. I.Williams, and N. Heess. Visual boundary prediction: A deep neural prediction network and quality dissection. In Proc. AISTATS, pages 512–521, 2014. [2] Y. Ganin and V. S. Lempitsky. N4-fields: Neural network nearest neighbor fields for image transforms. In Proc. ACCV, 2014. [3] W. Shen, X. Wang, Y. Wang, X. Bai, Z. Zhang. DeepContour: A Deep Convolutional Feature Learned by Positive-sharing Loss for Contour Detection. In Proc. CVPR, 2015. [4] G. Bertasius, J. Shi, L. Torresani. DeepEdge: A Multi-Scale Bifurcated Deep Network for Top-Down Contour Detection. In Proc. CVPR, 2015. [5] J.-J. Hwang, T.-L. Liu. Pixel-wise Deep Learning for Contour Detection. CoRR abs/1504.01989 (2015) [6] J.-J. Hwang, T.-L. Liu: Contour Detection Using Cost-Sensitive Convolutional Neural Networks. CoRR abs/1412.6857 (2014)
![Page 18: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/18.jpg)
Outline 18
Contour Detection
Overview
Milestones
Our Work Discussions
![Page 19: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/19.jpg)
Our Work 19
DeepContour: A Deep Convolutional Feature Learned by Positive-sharing
Loss for Contour Detection
Wei Shen, Xinggang Wang, Yan Wang, Xiang Bai, Zhijiang Zhang
IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2015
![Page 20: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/20.jpg)
Motivation 20
Why to apply CNN for contour detection?
Contour is hard to define
Contour data are sufficient for CNN training (millions
of local contour patches)
![Page 21: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/21.jpg)
Problem Formulation 21
Given a color image patch 𝑥 ∈ ℝ𝑛×𝑛×3, our goal is to determine whether its center pixel is passed through by contours or not.
𝑥 ∈ ℝ𝑛×𝑛×3 → 𝑧 ∈ 0,1
Q: Is a good idea to directly use CNN as a blackbox to address this binary classification problem?
Non-contour Contour
![Page 22: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/22.jpg)
Obstacle 22
The large variations in the contour shapes
Solution: Partitioning contour patches into compact clusters to convert the binary classification problem to a multi- class classification problem
Ref: J. J. Lim, C. L. Zitnick, and P. Dollár. Sketch tokens: A learned mid-level representation for contour and object detection. CVPR, 2013.
![Page 23: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/23.jpg)
Obstacle 23
How to define the loss function?
Q: Is softmax a good choice?
Softmax function penalizes the loss of each class equally
The losses for contour versus non-contour should be emphasized
Solution: Adding a regularized term to focus on the end goal of binary classification
![Page 24: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/24.jpg)
24
Data Preparation
Pre-cluster contour patches according to their contour shapes.
Assign a label 𝑦 to each contour patch 𝑥 according to the pre-cluster index 1,…𝐾 .
𝑥 ∈ ℝ𝑛×𝑛×3 → 𝑦 ∈ 0,1,…𝐾
Negative Positive
![Page 25: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/25.jpg)
25
Method
CNN Architecture
Loss Function
Let (𝑎𝑗𝑖; 𝑗 = 1,…𝐾) be the output of unit 𝑗 in FC2 for a
image patch 𝑥𝑗(𝑖)
, the probability that the label is 𝑗 is
𝑝𝑗(𝑖)=exp(𝑎𝑙
(𝑖))
exp(𝑎𝑙(𝑖))𝐾
𝑙=0
![Page 26: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/26.jpg)
26
Method
𝐽 = −1
𝑚 𝟏 𝑦 𝑖 = 𝑗 log 𝑝𝑗
𝑖
𝐾
𝑗=0
𝑚
𝑖=1
−1
𝑚 𝜆 𝟏 𝑦 𝑖 = 0 log 𝑝0
𝑖+ 𝟏 𝑦 𝑖 = 𝑗 log(1 − 𝑝0
𝑖)
𝐾
𝑗=1
𝑚
𝑖=1
Softmax loss
Positive-sharing loss, the loss for positive class is shared among each pre-clustered contour classes
![Page 27: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/27.jpg)
27
Method
To apply standard back-propagation to optimize the parameters of the network
𝜕𝐽
𝜕𝑎0(𝑖)=1
𝑚𝜆 + 1 𝟏 𝑦 𝑖 = 0 𝑝0
𝑖− 1 + (𝜆 + 1) 𝟏 𝑦 𝑖 = 𝑗
𝐾
𝑗=1
𝑝0𝑖
𝜕𝐽
𝜕𝑎𝑙(𝑖)=1
𝑚𝜆𝟏 𝑦 𝑖 = 0 + 1 𝑝𝑙
(𝑖)− 𝟏 𝑦 𝑖 = 𝑙 − 𝜆 𝟏 𝑦 𝑖 = 𝑗
𝐾
𝑗=1
𝑝0𝑖𝑝𝑙𝑖
1 − 𝑝0𝑖
![Page 28: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/28.jpg)
28
Method
CNN model validation
𝛾 ∈ [−1,1], measuring the discrimination of the learned model between positive and negative samples.
The learned features of FC1 will be fed into structured forest to perform contour detection.
𝛾 =1
𝑚 𝟏 𝑦 𝑖 = 0 − 𝟏 𝑦 𝑖 > 0 (𝑝0
𝑖− (1 − 𝑝0
𝑖))
𝑚
𝑖=1
![Page 29: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/29.jpg)
29
Experimental results
Deep Feature Visualization
Positive-sharing loss
Local gradient
![Page 30: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/30.jpg)
Experim
ental resu
lts
30
Results on BSDS500
![Page 31: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/31.jpg)
31
Experimental results
Results on BSDS500
![Page 32: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/32.jpg)
32
Experimental results
Results on NYUD
![Page 33: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/33.jpg)
33
Experimental results
Cross Dataset Generalization
DeepContour ODS OIS AP
BSDS/BSDS .76 .78 .80
NYU/BSDS .72 .74 .77
BSDS/NYU .59 .60 .53
NYU/NYU .62 .63 .57
SE [11] ODS OIS AP
BSDS/BSDS .74 .76 .78
NYU/BSDS .72 .73 .76
BSDS/NYU .55 .57 .46
NYU/NYU .60 .61 .56
SCG [39] ODS OIS AP
NYU/NYU .55 .57 .46
gPb [2] ODS OIS AP
NYU/NYU .51 .52 .37
![Page 34: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/34.jpg)
34
Experimental results
Parameter Discussion
![Page 35: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/35.jpg)
Outline 35
Contour Detection
Overview
Milestones
Our Work
Discussions
![Page 36: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/36.jpg)
Discussions 36
Speed
Caffe – per-patch mean subtraction is the bottleneck
Accuracy
Limitation may caused by the confusing labels
![Page 37: Deep Feature Learning for Contour Detection](https://reader031.fdocuments.in/reader031/viewer/2022012018/61686b96d394e9041f6f77bd/html5/thumbnails/37.jpg)
Discussions 37
Thank you! Q&A