UPC SECTION SECTION 1-101. SUBJECT UPC Statute (with Maine ...
Recurrent Instance Segmentation (UPC Reading Group)
-
Upload
xavier-giro -
Category
Data & Analytics
-
view
362 -
download
0
Transcript of Recurrent Instance Segmentation (UPC Reading Group)
![Page 1: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/1.jpg)
Recurrent Instance Segmentation
Slides by Manel BaradadComputer Vision Reading Group, UPC
9th September, 2016
Bernardino Romera-Paredes, Philip H. S. Torr[arxiv] (25 Nov 2015) - ECCV 2016
![Page 2: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/2.jpg)
Contents
2
● Introduction● Structure
○ FCN○ ConvLSTM○ Spatial Inhibition module○ Post processing
● Loss function● Experiments
○ Multiple Person Segmentation○ Plants Leaf Segmentation
● Conclusions
![Page 3: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/3.jpg)
Introduction● Detecting and delineating each distinct object of a
specific class appearing in an image● Contributions:
○ End-to-end approach for semantic instance segmentation
○ Derivation of a loss function for this problem
● Two particular classes tested:○ Multiple Person Segmentation○ Plants Leaf Segmentation and Counting
● It is not an attention based model, though the goal is attention on regions
3
![Page 4: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/4.jpg)
Structure
h1
h2
Ŷ1
Ŷ2
s2
4
s1
ht
![Page 5: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/5.jpg)
Structure
h1
h2
Ŷ1
Ŷ2
5
s1
s2
ht
![Page 6: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/6.jpg)
Structure
Stopping condition:
h1
h2
Ŷ1
Ŷ2
s2
s1
6
Ŷn + 1^
sn + 1^
ht
hn + 1 ^^
![Page 7: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/7.jpg)
Fully convolutional network
h1
h2
Ŷ1
Ŷ2
s2
s1
7
ht
Ŷn + 1^
sn + 1^
hn + 1 ^^
![Page 8: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/8.jpg)
Fully convolutional network● Objective: obtain features that serve as the input of the ConvLSTM● The article builds upon other good FCN’s for the semantic segmentation task● Specific for each of the two experiments performed (explained later)
Example: For the Multiple Person Segmentation FCN-8
8
![Page 9: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/9.jpg)
ConvLSTM
h1
h2
Ŷ1
Ŷ2
s2
s1
9
ht
Ŷn + 1^
sn + 1^
hn + 1 ^^
![Page 10: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/10.jpg)
LSTM: Recurrent structure
● Ability to produce sequential output● Provides memory
○ Implicitly model occlusion, segmenting non-occluded instances first, and keeping in its state the regions of the image that have already been segmented
○ Consider potential relationships from different instances in the image (i.e. all the instances of are always or never found together)
ConvLSTM
10
![Page 11: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/11.jpg)
ConvLSTMConvLSTM: “Standard” LSTM replacing the Fully connected layers ( ) for Convolutional layers
ct
ht
ht-1
ct-1
ht
ftit ot
11Extracted from: http://colah.github.io/posts/2015-08-Understanding-LSTMs/
![Page 12: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/12.jpg)
ConvLSTMWhy Conv instead of FC Layer for LSTM?
● Similar advantages of Conv Layers with respect to FC Layers○ Suitable for learning filters○ Useful for spatially invariant inputs such as images○ Require less memory for the parameters
12
![Page 13: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/13.jpg)
ConvLSTM
13
k(ht): sum of the absolute values across channels of ht Ŷt: predicted mask
![Page 14: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/14.jpg)
Spatial inhibition module
h1
h2
Ŷ1
Ŷ2
s2
s1
14
ht
Ŷn + 1^
sn + 1^
hn + 1 ^^
![Page 15: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/15.jpg)
Spatial inhibition module
15
Region proposals
Scores
![Page 16: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/16.jpg)
Region proposals
Region proposals
Scores
16
![Page 17: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/17.jpg)
Region proposalsRegion proposals
Scores
17
Value ranges:
Discriminate only one instance: Convolution + log-softmax
Adapt output to binary mask
At inference time, a pixel is assigned to an instance if the predicted value is higher than 0.5 (though values are usually saturated, very close to 0 or 1)
![Page 18: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/18.jpg)
Scores
Region proposals
Scores
18
![Page 19: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/19.jpg)
19
Region proposals
Scores
Scores
Really simple: The “intelligence” of the scoring must be learned in the previous states
Look at the values of the hidden state and apply a linear function
![Page 20: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/20.jpg)
Post ProcessingResults are further improved using a Conditional Random Field:
● Refine regions, as the ConvLSTM operates on a low resolution representation of the image
● Outside of the trainable modules
20
RIS (Recurrent Instance Segmentation)
RIS + CRF post processing
![Page 21: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/21.jpg)
Structure
Stopping condition:
h1
h2
Ŷ1
Ŷ2
s2
s1
21
Ŷn + 1^
sn + 1^
ht
hn + 1 ^^
![Page 22: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/22.jpg)
Loss Function● End-to-end training
22
![Page 23: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/23.jpg)
Loss Function1-Compute the intersection over the union for all pairs of Predicted/GT masks
23
Ŷt Yt
![Page 24: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/24.jpg)
Loss Function1-Compute the Intersection over the union for all pairs of Predicted/GT masks
0.9
0
0
0.1
0.8
0.1
...
...
...
...
24
![Page 25: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/25.jpg)
0.9
0
0
0.1
0.8
0.1
...
...
...
...
Loss Function2-Find best matching:
25
![Page 26: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/26.jpg)
0.9
0
0
0.1
0.8
0.1
...
...
...
...
2-Find best matching:
Simply find maximum weight bipartite matching (being the weights )
Easily solved using the Hungarian Algorithm (polynomial time )
Loss Function
26
![Page 27: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/27.jpg)
Loss Function2-Find best matching:
Loss: - Sum of the Intersections over the union for the best matching
27
![Page 28: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/28.jpg)
Loss Function3-Also take into account the scores
28
s1
s2
s3
s4
s5
Where:
is the binary cross entropy:
is the Iverson bracket which:
Is 1 if the condition is true and 0 else
![Page 29: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/29.jpg)
Loss Function3-Also take into account the similarities
29
s1
s2
s3
s4
s5
Simply:
For matched instances
For unmatched instances
> 0, and we want it small
1 - s5
![Page 30: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/30.jpg)
Loss Function4-Add everything together
30
> 0, and we want it big > 0, and we want it small
![Page 31: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/31.jpg)
● For each iteration:○ Forward propagate○ Find optimal matching
○ Once we have the matching, backpropagate the gradients of the loss function, with the values previously found
● The minimization of the loss function is ignored when backpropagating
Loss Function4-Add everything together
31
![Page 32: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/32.jpg)
Integrates the model on the FCN-8s network developed in Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation
Experiments: Multiple Person Segmentation
32
ConvLSTM introduced before the upsampling layer
![Page 33: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/33.jpg)
Multiple Person SegmentationTrained using the MSCOCO dataset and the training images of the Pascal VOC 2012 dataset
1. Fix the weights of the FCN-8s except for the last layer, and learn the parameters of that last layer, together with the ConvLSTM and the spatial inhibition module
2. Fine-tune the whole network
33
![Page 34: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/34.jpg)
Multiple Person SegmentationThe FCN features show a “mix” between semantic and instance segmentation
34
![Page 35: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/35.jpg)
Experiments: Plants Leaf SegmentationLearn the fully convolutional network from scratch: 5 convolutional layers +ReLU.
Computer Vision Problems in Plant Phenotyping (CVPPP) dataset: 161 images
Low SBD because of low resolution (though Difference in count is good)
35SBD is a measure about the accuracy of the segmentation of the instances
![Page 36: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/36.jpg)
Plants Leaf SegmentationThere are systems that perform better at the moment (better resolution…)
36
Mengye Ren, Richard S. Zemel: End-to-End Instance Segmentation and Counting with Recurrent Attention (30th May 2016). The article studied in this presentation was published the 25th Nov 2015
RIS+CRF Ren & ZemelRen & Zemel
![Page 37: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/37.jpg)
ConclusionsThe model integrates in a single pipeline all the required functions to segment instances, and their parameters are jointly learned end-to-end
The model uses a recurrent structure that is able to track visited areas in the image as well as to handle occlusion among instances, similarly to humans
The defined loss function accurately represents the instance segmentation objective
The experiments show promising performance
37
![Page 38: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/38.jpg)
Appendix
38Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation
![Page 39: Recurrent Instance Segmentation (UPC Reading Group)](https://reader031.fdocuments.in/reader031/viewer/2022030310/58f9a92a760da3da068b6bb6/html5/thumbnails/39.jpg)
Appendix
39Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation