for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow...
Transcript of for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow...
![Page 1: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/1.jpg)
Direction-aware Spatial Context Featuresfor Shadow Detection
Jing QinXiaowei Hu* Lei Zhu* Chi-Wing Fu Pheng-Ann Heng
The Chinese University of Hong Kong
The Hong Kong Polytechnic University
Shenzhen Institutes of Advanced Technology
![Page 2: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/2.jpg)
Problem
2
![Page 3: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/3.jpg)
3
2017 ICCV V. Nguyen, et al. Shadow detection with conditional generative adversarial networks.
2016 ECCV T. F. Y. Vicente, et al. Large-scale training of shadow detectors with noisily-annotated …
2016 Pattern Rec. J. Tian, et al. New spectrum ratio properties and features for shadow detection.
2015 CVPR L. Shen, et al. Shadow optimization from structured deep edge detection.
2015 ICCV Y. Vicente, et al. Leave-one-out kernel optimization for shadow detection.
2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection
2011 CVPR R. Guo, et al. Single-image shadow detection and removal using paired regions.
2011 ICCV X. Huang, et al. What characterizes a shadow boundary under the sun and sky?
2011 CVPR A. Panagopoulos, et al. Illumination estimation and cast shadow detection through …
2010 CVPR J. Zhu, et al. Learning to recognize shadows in monochromatic natural images.
2010 ECCV J.-F. Lalonde, et al. Detecting ground shadows in outdoor consumer photographs.
… … … …
1999 ICCV T. Horprasert, et al. A statistical approach for real-time robust background subtraction …
1995 BMVC P.L. Rosin, et al. Image difference threshold strategies and shadow detection.
1990 ICASSP J.M. Scanlan, et al. A shadow detection and removal algorithm for 2-D images
Related Works
![Page 4: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/4.jpg)
Related Works
▪ Data-driven approaches by learning the features using deep neural networks.
4
inputs scGAN stacked-CNN
scGAN: V. Nguyen, et al., “Shadow detection with conditional generative adversarial networks,” In ICCV, 2017.stacked-CNN: T. F. Y. Vicente, et al., “Large-scale training of shadow detectors with noisily-annotated shadow examples,” in ECCV, 2016.
![Page 5: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/5.jpg)
Motivation
5
![Page 6: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/6.jpg)
Motivation
6
➢ Shadow detection requires
an understanding of the
global image context
Motivation #1: Global Context
![Page 7: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/7.jpg)
Motivation
7
➢Analyze the global image
context in a direction-
aware manner
A
B
C
Motivation #2: Direction-aware Context
![Page 8: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/8.jpg)
Motivation #2: Direction-aware Context
8
inputs scGAN stacked-CNN
ours
![Page 9: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/9.jpg)
Spatial Recurrent Neural Network (RNN)
(a) input feature map(after 1*1 conv)
1st round in spatial RNN
S. Bell, et al., “Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks,” in CVPR, 2016.9
![Page 10: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/10.jpg)
Spatial Recurrent Neural Network (RNN)
(a) input feature map(after 1*1 conv)
(b) intermediatefeature map
(c) output map
1st round in spatial RNN 2nd round in spatial RNN
S. Bell, et al., “Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks,” in CVPR, 2016.10
![Page 11: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/11.jpg)
Spatial Recurrent Neural Network (RNN)
11
![Page 12: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/12.jpg)
Direction-aware Spatial Context (DSC) Module
12
![Page 13: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/13.jpg)
Direction-aware Spatial Context (DSC) Module
13
![Page 14: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/14.jpg)
Direction-aware Spatial Context (DSC) Module
14
![Page 15: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/15.jpg)
Method Overview
input
Feature Extraction Network
DSC module
DSC module
DSC module
DSC module
DSC module
DSC module
Direction-awareSpatial Context Module
concat 1×1 conv & up-sampling
15
concat & 1×1 conv
MLIF
![Page 16: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/16.jpg)
Method Overview
16
![Page 17: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/17.jpg)
Training and Testing
Loss Function:
➢ Weighted cross entropy loss:
y: ground truth value p: prediction label 𝑁𝑛: the number of non-shadow pixels
𝑁𝑝: the number of shadow pixels TP: true positive TN: true negative
17
Datasets:
➢ Training: SBU training set (4089 images)
➢ Testing: SBU testing set (638 images) and UCF testing set (76 images)
SBU: T. F. Y. Vicente, et al., “Large-scale training of shadow detectors with noisily-annotated shadow examples,” in ECCV, 2016. UCF: J. Zhu, et al., “Learning to recognize shadows in monochromatic natural images,” in CVPR, 2010.
![Page 18: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/18.jpg)
Training and Testing
Testing:
➢ Shadow map: the mean of the score maps over the MLIF layer and the fusion layer
➢ Post-processing: conditional random field (CRF)
18
![Page 19: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/19.jpg)
Results - Balance Error Rate (%)
19
scGAN[ICCV 17’]
scGAN[ICCV 17’]
DSC (ours)
5.59
9.10
stacked-CNN
[ECCV 16’]
11.00
SBU testing set UCF testing set
DSC (ours)
8.10
11.50
stacked-CNN
[ECCV 16’]
13.00
![Page 20: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/20.jpg)
Visual Comparison Results
20
input ground truth scGAN 17’ stacked-CNN 16’DSC (ours)
![Page 21: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/21.jpg)
• Ablation Analysis:Network Design Evaluation
21
DSC (ours)basic+contextbasicground truthinput
![Page 22: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/22.jpg)
Failure Cases
22
input ground truth DSC (ours)
![Page 23: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/23.jpg)
Xiaowei Hu, Chi-Wing Fu, Lei Zhu, Jing Qin, and Pheng-Ann Heng. Direction-aware Spatial Context Features for Shadow Detection and Removal.
arXiv preprint arXiv:1805.04635, 2018.23
Our Recent Extension - Shadow Removal
![Page 24: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/24.jpg)
Conclusion
➢ Direction-aware spatial context features for shadow detection and removal.
➢ Achieve the state-of-the-art performance on two benchmark datasets for
shadow detection and another two benchmark datasets for shadow removal.
24
Code & Results:
https://github.com/xw-hu/DSC
Poster: D12
![Page 25: for Shadow Detection · 2014 CVPR S. H. Khan, et al. Automatic feature learning for robust shadow detection 2011 CVPR R. Guo, et al. Single-image shadow detection and removal using](https://reader034.fdocuments.in/reader034/viewer/2022050416/5f8c0d218d27975f227b9268/html5/thumbnails/25.jpg)
Q&A
Thank you!
25The Chinese University of Hong Kong
Code & Results:
https://github.com/xw-hu/DSC
Poster: D12