Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill...
-
Upload
kelley-horton -
Category
Documents
-
view
212 -
download
0
Transcript of Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill...
![Page 1: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/1.jpg)
Context-based vision system for place and object recognition
Antonio TorralbaKevin MurphyBill FreemanMark Rubin
Presented by David LeeSome slides borrowed from Kevin Murphy
![Page 2: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/2.jpg)
Object out of context
![Page 3: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/3.jpg)
Object in context
![Page 4: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/4.jpg)
Wearable test-bed
![Page 5: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/5.jpg)
System diagram
![Page 6: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/6.jpg)
Computing the features
![Page 7: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/7.jpg)
24 filteredImages
Downs
ampl
e
to 4
x4
4x4x24=384 dim 80 dim
![Page 8: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/8.jpg)
Visualizing the filter bank output
Images
80-dimensional representation
![Page 9: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/9.jpg)
Place recognition system
![Page 10: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/10.jpg)
Hidden Markov Model
Hidden states = location (63 values) Observations = vG
t ∈ R80
Transition model encodes topology of environment
Observation model is a mixture of Gaussians (100 views per place)
![Page 11: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/11.jpg)
Hidden Markov Model
Observation Likelihood
Prediction Prior
Transition Matrix
Mixture of Gaussians MLE (counting)
![Page 12: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/12.jpg)
Scene Categorization
17 Categories (Office, Corridor, Street, etc)
Train a separate HMM on category labels
![Page 13: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/13.jpg)
Place recognition demo
![Page 14: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/14.jpg)
Specific location
Location category
Indoor/outdoor
Ground truthSystem estimate
Performance on known env.
![Page 15: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/15.jpg)
Performance on new env.
![Page 16: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/16.jpg)
Comparison of features
Recognition Categorization
![Page 17: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/17.jpg)
Effect of HMM on recognition
With Without(But with temporal smoothing)
![Page 18: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/18.jpg)
From place to object recognition
![Page 19: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/19.jpg)
Object priming Predict object properties based on
context (top-down signals): Visual gist, vt
G
Specific Location, Qt
Kind of location, Ct
![Page 20: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/20.jpg)
Object Priming
Again…MLE
Probability of object i
Probability of object i in image vi given entire video
sequence
Probability of object i Given current
observation & place
Estimate of current place
(Output of HMM)
Mixture of Gaussians
Observation Likelihood
Prior probability of object i being
in place q
![Page 21: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/21.jpg)
Predicting object presence
![Page 22: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/22.jpg)
ROC curves for object detection
![Page 23: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/23.jpg)
Predicting object position and scale
![Page 24: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/24.jpg)
Predicting object position and scaleEstimate of
mask
Probability of an object i being present and location being q(Output of previous system)
Estimate of mask given current gist, place, and object
delta Gaussian
![Page 25: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/25.jpg)
Predicted segmentation
![Page 26: Context-based vision system for place and object recognition Antonio Torralba Kevin Murphy Bill Freeman Mark Rubin Presented by David Lee Some slides borrowed.](https://reader031.fdocuments.in/reader031/viewer/2022020417/5697bfc11a28abf838ca4249/html5/thumbnails/26.jpg)
Conclusion
Real world problem (and it works!)
Uses only global feature (context)
How much did {HMM / place prior} affect{place recognition / object detection}?Can we really say “context” did the job?