Visualizing Networks Caffe overview Slides are now...
Transcript of Visualizing Networks Caffe overview Slides are now...
![Page 1: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/1.jpg)
Recap from Monday
• Visualizing Networks
• Caffe overview
• Slides are now online
![Page 2: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/2.jpg)
Today
• Edges and Regions, GPB
• Fast Edge Detection Using Structured Forests
– Zhihao Li
• Holistically-Nested Edge Detection
– Yuxin Wu
• Selective Search for Object Recognition
– Chun-Liang Li
![Page 3: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/3.jpg)
Logistics
• Please read:
– Region-based Convolutional Networks for Accurate Object Detection and Semantic Segmentation
• If you’re up next, please meet us
• Project Proposals Due in < 1 week
– If you have questions, ask to meet
![Page 4: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/4.jpg)
Edges and Regions
David Fouhey
![Page 5: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/5.jpg)
Task
"I stand at the window and see a house, trees, sky. Theoretically I might say there were 327 brightnesses and nuances of colour. Do I have "327"? No. I have sky, house, and trees.” -Max Wertheimer
Quote from Jitendra Malik’s page
![Page 6: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/6.jpg)
Approaching the Task – Regions
…
Decomposing image into K connected regions (Clustering task)
![Page 7: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/7.jpg)
Approaching the Task – Edges
HxWx {0,1} classification problem
![Page 8: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/8.jpg)
Are the Tasks Equivalent?
Segmentation Boundaries
?
![Page 9: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/9.jpg)
Are the Tasks Equivalent?
Segmentation Boundaries
?
![Page 10: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/10.jpg)
Are the Tasks Equivalent?
Segmentation Boundaries
?
Contours have to be closed!
![Page 11: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/11.jpg)
Does This Matter in the CNN Era?
HED – State of the Art
![Page 12: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/12.jpg)
Are These Well-Defined Tasks?
Should blue and yellow go in the same segment?
Image credit: NYU depth dataset
![Page 13: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/13.jpg)
Successes – Superpixels
Problem: >10^5 pixels intractable for reasoning
Solution: use bigger/super pixels that don’t ruin any boundaries
First from Ren et al. 2003, Fish image from Achanta et al. 2012
![Page 14: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/14.jpg)
Successes – Multiple Segmentations
• Problem: No one segmentation is good
• Solution: Use many, figure it out later
Hoiem et al. 2005
![Page 15: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/15.jpg)
Contributions of Paper
• Merges the (edges + regions) approaches
• Introduces machinery used throughout vision
• Landmark paper in segmentation/boundary detection
• Note: the questions are often as important as the answers
![Page 16: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/16.jpg)
Questions from Piazza
• Where’s the learning?!
– Great idea! Two papers next
• What’s this useful for?
– Great question! Last paper today, paper for Monday.
![Page 17: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/17.jpg)
Dataset – BSDS 500
Images
– 500 Total
– 300 Training, 200 Testing
Annotation
– 5 annotators (CV students) per image
– Annotators annotate segment
![Page 18: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/18.jpg)
Dataset – Instructions
Divide each image into pieces, where each piece represents a distinguished thing in the image. It is important that all of the pieces have approximately equal importance. The number of things in each image is up to you. Something between 2 and 20 should be reasonable for any of our images
Martin et al. “A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics.” ICCV 2001
![Page 19: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/19.jpg)
Dataset – Image and Annotations
![Page 20: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/20.jpg)
Evaluation Criteria – Boundaries
Precision = TP / (TP+FP) (fraction of predicted + results that are +)
Recall = TP / (TP+FN) (fraction of + results that are predicted +)
![Page 21: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/21.jpg)
Evaluation Criteria – Segments
• In words: Average the intersection/union of the best predicted region for all GT regions, weighted by GT region size
• Previous evaluation criteria don’t clearly distinguish dumb baselines from algorithm outputs
![Page 22: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/22.jpg)
GPB-OWT-UCM
Boundary Detection
Local Discontinuity
Segmentation Machinery
Spectral Embedding
Boundary Detection
Spectral Discontinuity
Segmentation Machinery
OWT+UCM
![Page 23: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/23.jpg)
Local Terms
• Core Idea: can compute histogram distances
![Page 24: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/24.jpg)
Local Terms
Orientation 1 Orientation 2 Max over
Orientations Luminance
Image
![Page 25: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/25.jpg)
Local Terms
Max over Orientations Luminance Image
![Page 26: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/26.jpg)
Local Terms – Multiple Cues
Accumulate evidence per-orientation
Weighted Sum of
Predictions
![Page 27: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/27.jpg)
Learning
• Simple linear combinations = few parameters
• Gradient ascent in the reading
• Logistic regression in past
weights
Contour strength in feature + scale
![Page 28: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/28.jpg)
GPB-OWT-UCM
Boundary Detection
Local Discontinuity
Segmentation Machinery
Spectral Embedding
Boundary Detection
Spectral Discontinuity
Segmentation Machinery
OWT+UCM
Probability of contour at location x,y, orientation t
![Page 29: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/29.jpg)
Globalization – Motivation
Local Globalized
![Page 30: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/30.jpg)
Globalization
𝑊 ∈ 𝑅 𝐻𝑊 𝑥 𝐻𝑊
Normal Spectral Clustering 1. Use W to produce embedding/space
defined by eigenvectors of a system of equations. See links on Piazza for why
2. Cluster in induced space
This Paper 1. Use W to produce embedding/space
defined by eigenvectors of a system of equations
2. Treat eigenvectors as images, compute gradient
![Page 31: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/31.jpg)
Globalization
Input Eigenvectors of Spectral System Weighted
Sum of Gradients
![Page 32: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/32.jpg)
Combining Global + Local
• Linear weighting; weights learned with gradient ascent
Orientations processed separately throughout Why is this important?
![Page 33: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/33.jpg)
GPB-OWT-UCM
Boundary Detection
Local Discontinuity
Segmentation Machinery
Spectral Embedding
Boundary Detection
Spectral Discontinuity
Segmentation Machinery
OWT+UCM
Could cluster in this space
Probability of contour at location x,y, orientation t taking into consideration soft segmentations
![Page 34: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/34.jpg)
Watershed Transform – 1D Version
• Black region: probability of boundary
• Black lines: watershed boundaries
![Page 35: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/35.jpg)
Problem: probability of boundary is orientation-dependent
Solution: get probability of boundary in direction
Orientation
![Page 36: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/36.jpg)
Output of Watershed Transform
“Oversegmentation” of image with boundary strengths
![Page 37: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/37.jpg)
UCM
• Hierarchical merging; guarantees closed contours
![Page 38: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/38.jpg)
GPB-OWT-UCM
Boundary Detection
Local Discontinuity
Segmentation Machinery
Spectral Embedding
Boundary Detection
Spectral Discontinuity
Segmentation Machinery
OWT+UCM
Contour that can be cut at any point to yield closed regions
![Page 39: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/39.jpg)
Results – State of the Art
This: 72.6
Current SOA: 78.2
![Page 40: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/40.jpg)
Results – Ablative Analysis
• Combining Local + Global helps
• Why does local help in high-recall regime?
![Page 41: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/41.jpg)
Results – Ablative Analysis
OWT/UCM:
• Ensures closed boundaries
• Helps a little
![Page 42: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/42.jpg)
Next Up
• Fast Edge Detection Using Structured Forests
– Zhihao Li
• Holistically-Nested Edge Detection
– Yuxin Wu
• Selective Search for Object Recognition
– Chun-Liang Li
![Page 43: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/43.jpg)
![Page 44: Visualizing Networks Caffe overview Slides are now onlinegraphics.cs.cmu.edu/courses/16-824/2016_spring/slides/edges.pdf · –Great question! Last paper today, paper for Monday.](https://reader034.fdocuments.in/reader034/viewer/2022042105/5e841a874a2b18282f1b80a8/html5/thumbnails/44.jpg)
Stash