Weakly Supervised Semantic Segmentation with Image-level...
Transcript of Weakly Supervised Semantic Segmentation with Image-level...
Yunchao Wei
Weakly Supervised Semantic Segmentation with Image-level Annotation
https://weiyc.github.io
The trend of weakly-supervised learning
2014 2015 2016
CVPR ICCV/ECCV
2013 2014
5
3
8
6
8 8
12 12
Why do we need WSL?
•The success of DCNN-based object recognition approaches rely on alarge number of labeled images
•Labeling a large amount of images is very costly in terms of bothfinance and human effort.
•Object detection
•Semantic Segmentation
Semantic Segmentation
… …
Fully-convolutional Segmentation Network
Loss
Segmentation
Task
•Fully supervised scheme
Semantic Segmentation
•Weakly-supervised scheme with image-level annotation
person
horsetable
images
annotations
…
…
Weakly-supervised
Semantic Segmentation
Test Image
Proposal-based Solution
Learning to segment with image-level annotations. PR 2016
Proposal-based Solution
Hypotheses-CNN-Pooling
HCP: A flexible CNN framework for multi-label image classification Yunchao Wei, etc. TPAMI 2016
Localization Map Generation
•Exhaustedly examine each proposal togenerate localization
•Time consuming
•Introducing false negative pixels(background)
STC: Simple to Complex
Simple Images Complex Images
•Motivation
STC: A simple to complex framework for weakly-supervised semantic segmentation TPAMI 2017
STC: Simple to Complex
•Simple images with the corresponding saliency maps
STC: A simple to complex framework for weakly-supervised semantic segmentation TPAMI 2017
STC: Simple to Complex
•Framework•Initial-DCNN
•Enhanced-DCNN
•Powerful-DCNN
STC: A simple to complex framework for weakly-supervised semantic segmentation TPAMI 2017
STC: Simple to Complex
•Flickr-Clean(40K)
STC: Simple to Complex
Networks Training Set mIoU
I-DCNN Flickr-Clean 44.1
E-DCNN Flickr-Clean 46.8
P-DCNN Flickr-Clean+VOC 49.8
Ablation Analysis on Pascal VOC12 val
Comparisons on Pascal VOC12 test
Methods mIoU
MIL-FCN (ICLR 2015) 24.9
CCNN (ICCV 2015) 35.5
EM-Adapt (ICCV 2015) 39.6
MIL-ILP-Seg (CVPR 2015) 40.6
STC (ours) 51.2
STC: A simple to complex framework for weakly-supervised semantic segmentation TPAMI 2017
STC: Simple to Complex
•Testing Results
•Shortcomings•Depend on a large number of simple images for training.
Image Result GT Image Result GT
Object Region Mining with AE
•Top-down attention• Class Activation Mapping [1]; Excitation Backpropagation [2]
[1] Learning Deep Features for Discriminative Localization. Bolei Zhou etc. CVPR 2016
Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)
[2] Top-down Neural Attention by Excitation Backprop. Jianmin Zhang etc. ECCV 2016
CVPR 2016 ECCV 2016
Object Region Mining with AE
•Motivation•Classification networks are only responsive to small and sparse discriminative regions from object of interest
•How to obtain dense and integral object-related regions for learning to semantic segmentation?
Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)
Object Region Mining with AE
•Solution: Adversarial erasing (AE)
Some visualized samples
Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)
Object Region Mining with AE
•Framework of AE
Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)
Object Region Mining with AE
•Examples of mined object regions produced by the AE approach
Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)
Object Region Mining with AE
•Online prohibitive segmentation learning (PSL) for Semantic Segmentation
PSL
Producing Segmentation Mask
Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)
Object Region Mining with AE
•Ablation Analysis on Pascal VOC12 val
0 150
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Epoch
Loss
AE-step4
AE-step3
AE-step2
AE-step1
AE-Steps mIoU
AE-step1 44.9
AE-step2 49.5
AE-step3 50.9
AE-step4 48.8
Training Schemes mIoU
w/o PSL 50.9
w/ PSL 54.1
w/ PSL+ 55.0
Adversarial erasing
Prohibitive Segmentation Learning
Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)
Object Region Mining with AE
•Comparisons on Pascal VOC12 testMethods mIoU
MIL-FCN (ICLR 2015) 24.9
CCNN (ICCV 2015) 35.5
EM-Adapt (ICCV 2015) 39.6
MIL-ILP-Seg (CVPR 2015) 40.6
STC (PAMI 2016) 51.2
DCSM (ECCV 2016) 45.1
BFBP (ECCV 2016) 48.0
SEC (ECCV 2016) 51.7
AF-SS(ECCV 2016) 52.7
AE-PSL (ours) 55.7
images predictions GT
Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)
Future work
22
•Simultaneous weakly-supervised object detection and semanticsegmentation
•Semi-supervised object detection and semantic segmentation
23