MIT6870_ORSU_lecture11 Hierarchies
Transcript of MIT6870_ORSU_lecture11 Hierarchies
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 1/36
Lecture 11Hierarchies
6.870 Object Recognition and Scene Understandinghttp://people.csail.mit.edu/torralba/courses/6.870/6.870.recognition.htm
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 2/36
Next weekAlec Rivers
Scene Understanding Based on Object RelationshipsGokberk Cinbis
Category Level 3D Object Detection Using View-Invariant Representations
Hueihan Jhuang and Sharat Chikkerur
Video shot boundary detection using GIST representation
Jenny Yuen
Semiautomatic alignment of text and images
Nathaniel R Twarog
A Filtering Approach to Image Segmentation: Perceptual Grouping in Feature Space
Nicolas Pinto
Evaluating dense feature descriptor and multi-kernel learning for face detection/recognition
Tilke Judd and Vladimir Bychkovsky
Identify the same people in different photographs from the same event
Tom Kollar
Context-based object priors for scene understanding
Tom Ouyang
Hand-Drawn Sketch Recognition, A Vision-Based Approach
Papers due this Friday (5pm): send PDF by email
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 3/36
Hierarchies vs. holistic features
Although we have
seen some ³successful´
holistic methods.
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 4/36
Hierarchies, compositionality and
reusable parts
Compositionality refers to our evident ability to
construct hierarchical representations, whereby
constituents are used and reused in an
essentially infinite variety of relationalcompositions.
Assumption (Bienenstock, Geman): what islearnable is what is representable as a hierarchy
of more-or-less simple composition rules.
Bienenstock, Geman. Compositionality in neural systems.
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 5/36
Hierarchies vs. holistic features
Feature hierarchies are often inspired by the structure of the primate visual system,
which has been shown to use a hierarchy of features of increasing complexity, fromsimple local features in the primary visual cortex, to complex shapes and object
views in higher cortical areas.
S. Ullman et al.
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 6/36
Diagram of the visual system
Felleman and Van Essen, 1991
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 7/36
Modified by T. Serre from Ungerleider and Haxby, and then shamelessly copied by me.
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 8/36
Modified by T. Serre from Ungerleider and Haxby, and then copied by me.
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 9/36
Modified by T. Serre from Ungerleider and Haxby, and then copied by me.
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 10/36
Modified by T. Serre from Ungerleider and Haxby, and then copied by me.
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 11/36
Modified by T. Serre from Ungerleider and Haxby, and then copied by me.
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 12/36
IT readout
Slide by Serre
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 13/36
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 14/36
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 15/36
Voxel Activity ModelGoal: to predict the image seen by the observer out of a large collection of
possible images. And to do this for new images: this requires predicting f MRIactivity for unseen images.
Kay, K.N., Naselaris, T., Prenger, R.J., & Gallant, J.L. (2008). Identifying natural imagesfrom human brain activity. Nature, 452, 352-355.
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 16/36
Kay, K.N., Naselaris, T., Prenger, R.J., & Gallant, J.L. (2008). Identifying natural imagesfrom human brain activity. Nature, 452, 352-355.
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 17/36
Performance
Kay, K.N., Naselaris, T., Prenger, R.J., & Gallant, J.L. (2008). Identifying natural imagesfrom human brain activity. Nature, 452, 352-355.
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 18/36
D.Marr
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 19/36
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 20/36
Neocognitron
Learning is done greedily for each layer
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 21/36
Convolutional Neural Network
The output neurons share all the intermediate levels
Le Cun et al, 98
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 22/36
Hierarchical models of object recognition in cortex
Hierarchical extension of the classical paradigm of building complex cells from simple cells.
Uses same notation than Fukushima: ³S´ units performing template matching, solid lines and
³C´ units performing non-linear operations ( ³M AX´ operation, dashed lines)
Riesenhuber, M. and Poggio, T. 99
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 23/36
Slide by T. Serre
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 24/36
Slide by T. Serre
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 25/36
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 26/36
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 27/36
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 28/36
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 29/36
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 30/36
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 31/36
Learning a Compositional Hierarchy of Object Structure
Fidler & Leonardis, CVPR¶07; Fidler, Boben & Leonardis, CVPR 2008Fidler & Leonardis, CVPR¶07; Fidler, Boben & Leonardis, CVPR 2008
The architecture
Parts model
Learned parts
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 32/36
Learning a Compositional Hierarchy of Object Structure
Fidler & Leonardis, CVPR¶07; Fidler, Boben & Leonardis, CVPR 2008Fidler & Leonardis, CVPR¶07; Fidler, Boben & Leonardis, CVPR 2008
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 33/36
Learning a Compositional Hierarchy of Object Structure
Fidler & Leonardis, CVPR¶07Fidler & Leonardis, CVPR¶07
Fidler, Boben & Leonardis, CVPR 2008Fidler, Boben & Leonardis, CVPR 2008
Layer 2
Layer 3
Layer 4
Layer 1
LEARNLEARNhierarchical libraryhierarchical library
car motorcycle dog person
Hierarchical compositional architectureHierarchical compositional architecture
Features are shared at each layer Features are shared at each layer
Learning is done on natural imagesLearning is done on natural images
Indexing and matching detection schemeIndexing and matching detection scheme
Learned L1Learned L1 ± ± L3L3
Learned hierarchicalLearned hierarchical
vocabularyvocabulary DetectionsDetections
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 34/36
Learning a Compositional Hierarchy of Object Structure
Fidler & Leonardis, CVPR¶07Fidler & Leonardis, CVPR¶07
Fidler, Boben & Leonardis, CVPR 2008Fidler, Boben & Leonardis, CVPR 2008
Layer 2
Layer 3
Layer 4
Layer 1
LEARNLEARNhierarchical libraryhierarchical library
car motorcycle dog person
Learned hierarchicalLearned hierarchical
vocabularyvocabulary DetectionsDetections
Hierarchical compositional architectureHierarchical compositional architecture
Features are shared at each layer Features are shared at each layer
Learning is done on natural imagesLearning is done on natural images
Biologically plausible?Biologically plausible?
Learns TLearns T-- and Land L-- junctions, different junctions, differentcurvatures, and features that graduallycurvatures, and features that graduallyincrease in complexityincrease in complexity
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 35/36
8/3/2019 MIT6870_ORSU_lecture11 Hierarchies
http://slidepdf.com/reader/full/mit6870orsulecture11-hierarchies 36/36
HDP Object Model
We learn the
number of parts. Each object
uses a different
number of parts.
The model
assumes aknown number
of object
categories.
Parts are distributions
over appearances andlocations
Sudderth et al. IJCV 2008