CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard...
Transcript of CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard...
![Page 1: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/1.jpg)
CEE598 - Visual Sensing for
Civil Infrastructure Eng. & Mgmt.
Session 18 – Object Recognition I
Department of Civil and Environmental Engineering, University of Illinois at Urbana-Champaign
Mani Golparvar-Fard Department of Civil and Environmental Engineering
3129D, Newmark Civil Engineering Lab
e-mail: [email protected]
![Page 2: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/2.jpg)
Object Recognition
Bill Gates demoing visual recognition gadget
@ CES 2008
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
2
http://www.youtube.com/watch?v=LwRsvKhWSB0&feature=related
![Page 3: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/3.jpg)
Outline
Object Recognition
• Introduction
• Recognition of single 3D objects
• Bag of world models
• Part based models
• Models for 3D objects categorization
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
3 Some slides of this lectures are courtesy of prof S. Savarese, prof S.
Lazebnik & S. Seitz, R. Szeliski, M. Pollefeys, K. Hassan-Shafique. S. Thrun
![Page 4: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/4.jpg)
Outline
Object Recognition
• Introduction
• Recognition of single 3D objects
• Bag of world models
• Part based models
• Models for 3D objects categorization
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
4
Part of this segment is based on the tutorial “Recognizing and Learning
Object Categories: Year 2007”, by Prof A. Torralba, R. Fergus and F. Li
![Page 5: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/5.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
5
5
http://www.dubaichronicle.com/wp-content/uploads/2008/10/dubai-fountain-under-
construction-at-downtown-burj-dubai.jpg
DUBAI FOUNTAIN UNDER CONSTRUCTION AT DOWNTOWN BURJ KHALIFA (DUBAI)
![Page 6: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/6.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
6
Bruegel, 1564
6
![Page 7: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/7.jpg)
What is this abstract form?
Plato said…
• Ordinary objects are classified together if they `participate' in the same abstract Form, such as the Form of a Human or the Form of Quartz.
• Forms are proper subjects of philosophical investigation, for they have the highest degree of reality.
• Ordinary objects, such as humans, trees, and stones, have a lower degree of reality than the Forms.
• Fictions, shadows, and the like have a still lower degree of reality than ordinary objects and so are not proper subjects of philosophical enquiry.
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
7
![Page 8: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/8.jpg)
Example: Excavators
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
9
![Page 9: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/9.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
10
10
![Page 10: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/10.jpg)
Challenges: Viewpoint Variation
11
Michelangelo 1475-1564
slide credit: Fei-Fei, Fergus & Torralba CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
![Page 11: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/11.jpg)
Challenges: illumination
12
(b) 01/04/2005; 4:02:00 PM (a) 01/02/2005; 4:02:00 PM (c) 01/13/2005; 4:00:00 PM
(a) 01/16/2005; 3:03:00 PM (b) 01/16/2005; 4:03:00 PM (c) 01/16/2005; 5:03:00 PM
Project: Institute of Genomic Biology, Courtesy of College of ACES, UIUC
Sources: Golparvar-Fard et al. (2009) CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
![Page 12: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/12.jpg)
Challenges: scale
13
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
![Page 13: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/13.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
14
Challenges: scale
slide credit: Fei-Fei, Fergus & Torralba
14
![Page 14: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/14.jpg)
Challenges: deformation
15
http://entertainment.webshots.com/photo/2351907180017385169rbxbPP
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
![Page 15: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/15.jpg)
Challenges: occlusion
16
Superintendent
Visible Changed
Occluded Changed
Occluded Unchanged
Visible Unchanged
Shadow
Static Occlusion
Dynamic Occlusion
Student Dining Hall Construction Project, Champaign, IL - 8/27/2008 CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
![Page 16: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/16.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
17
Challenges:
occlusion
Magritte, 1957 slide credit: Fei-Fei, Fergus & Torralba
17
![Page 17: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/17.jpg)
Challenges: background clutter
18
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
![Page 18: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/18.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
19
Challenges: background clutter
Kilmeny Niland. 1995
19
![Page 19: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/19.jpg)
Challenges: object intra-class variation
20
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
![Page 20: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/20.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
21
Challenges: intra-class variation 21
![Page 21: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/21.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
History: single object recognition No intra-class variation: single object recognition
22
![Page 22: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/22.jpg)
So what does object recognition involve?
23
http://www.dubaichronicle.com/wp-content/uploads/2008/10/dubai-fountain-under-construction-at-downtown-burj-dubai.jpg
DUBAI FOUNTAIN UNDER CONSTRUCTION AT DOWNTOWN BURJ KHALIFA (DUBAI) http://www.dubaichronicle.com/wp-content/uploads/2008/10/dubai-fountain-under-construction-at-downtown-burj-dubai.jpg
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
![Page 23: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/23.jpg)
Identification: does this region contain the Dubai Fountain?
24
http://www.dubaichronicle.com/wp-content/uploads/2008/10/dubai-fountain-under-construction-at-downtown-burj-dubai.jpg
DUBAI FOUNTAIN UNDER CONSTRUCTION AT DOWNTOWN BURJ KHALIFA (DUBAI) http://www.dubaichronicle.com/wp-content/uploads/2008/10/dubai-fountain-under-construction-at-downtown-burj-dubai.jpg
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
![Page 24: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/24.jpg)
Object Labeling and Categorization
25
http://www.dubaichronicle.com/wp-content/uploads/2008/10/dubai-fountain-under-construction-at-downtown-burj-dubai.jpg
DUBAI FOUNTAIN UNDER CONSTRUCTION AT DOWNTOWN BURJ KHALIFA (DUBAI) http://www.dubaichronicle.com/wp-content/uploads/2008/10/dubai-fountain-under-construction-at-downtown-burj-dubai.jpg
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
![Page 25: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/25.jpg)
Object Labeling and Categorization
26
http://www.dubaichronicle.com/wp-content/uploads/2008/10/dubai-fountain-under-construction-at-downtown-burj-dubai.jpg
DUBAI FOUNTAIN UNDER CONSTRUCTION AT DOWNTOWN BURJ KHALIFA (DUBAI)
Mobile Crane
Concrete Truck
Mobile Crane
Excavator Dump truck
Concrete Pump
Tower Crane
http://www.dubaichronicle.com/wp-content/uploads/2008/10/dubai-fountain-under-construction-at-downtown-burj-dubai.jpg
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
![Page 26: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/26.jpg)
Why Object Recognition?
Tracking
Action Recognition
Events
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
27
![Page 27: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/27.jpg)
• Turk and Pentland, 1991
• Belhumeur, Hespanha, & Kriegman, 1997
• Schneiderman & Kanade 2004
• Viola and Jones, 2000
• Amit and Geman, 1999
• LeCun et al. 1998
• Belongie and Malik, 2002
• Schneiderman & Kanade, 2004
• Argawal and Roth, 2002
• Poggio et al. 1993
Some early works on object
categorization
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
28
![Page 28: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/28.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
29
Applications: Assisted driving
meters
me
ters
Ped
Ped
Car
Lane detection
Pedestrian and car detection
• Collision warning systems with
adaptive cruise control
• Lane departure warning systems
• Rear object detection systems
29
![Page 29: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/29.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
30
Computational photography 30
![Page 30: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/30.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
31
Improving online search
Query:
STREET
Organizing photo collections
31
![Page 31: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/31.jpg)
Applications of computer vision
Factory inspection Surveillance
Autonomous driving,
robot navigation
Assistive technologies
Security
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
32
![Page 32: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/32.jpg)
Three main issues
Representation
• How to represent an object category; which
classification scheme?
Learning
• How to learn the classifier, given training data
Recognition
• How the classifier is to be used on novel data
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
33
![Page 33: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/33.jpg)
Generative / discriminative / hybrid
Representation
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
34
![Page 34: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/34.jpg)
)|( imageexcavatorp
)( imageexcavator|nopvs.
• Bayes rule:
Object categorization: the statistical viewpoint
)|(
)|(
imageexcavatornop
imageexcavatorp
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
35
![Page 35: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/35.jpg)
)|( imageexcavatorp
)( imageexcavator|nopvs.
• Bayes rule:
Object categorization: the statistical viewpoint
)(
)(
)|(
)|(
)|(
)|(
excavatornop
excavatorp
excavatornoimagep
excavatorimagep
imageexcavatornop
imageexcavatorp
posterior ratio likelihood ratio prior ratio
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
36
![Page 36: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/36.jpg)
• Bayes rule:
Object categorization: the statistical viewpoint
Discriminative methods model posterior
Generative methods model likelihood and
prior
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
37
)(
)(
)|(
)|(
)|(
)|(
excavatornop
excavatorp
excavatornoimagep
excavatorimagep
imageexcavatornop
imageexcavatorp
posterior ratio likelihood ratio prior ratio
![Page 37: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/37.jpg)
Representation
• Generative / discriminative / hybrid
• Appearance only or location and appearance
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
38
![Page 38: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/38.jpg)
Representation
– Generative / discriminative
/ hybrid
– Appearance only or
location and appearance
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
39
![Page 39: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/39.jpg)
Representation
Generative / discriminative / hybrid
Appearance only or location and appearance
Invariances
• View point
• Illumination
• Occlusion
• Scale
• Deformation
• Clutter
• etc.
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
40
![Page 40: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/40.jpg)
Learning
Learning model parameters
– Degree of supervision
– Batch-vs-online
– etc…
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
41
![Page 41: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/41.jpg)
Recognition
• Scale / orientation range to search over
• Speed
• Context
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
42
![Page 42: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/42.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
43
Hoie
m, E
fros,
Herb
ert
, 2006
43
![Page 43: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/43.jpg)
Outline
Object Recognition
• Introduction
• Recognition of single 3D objects
• Bag of world models
• Part based models
• Models for 3D objects categorization
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
44
![Page 44: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/44.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
•Various level of
supervision
…
Car: front-right
Iron: top-rear-left
train test
45
![Page 45: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/45.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
46
![Page 46: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/46.jpg)
•Zhang et al ’95
•Schmid & Mohr, ‘96
•Schiele & Crowley, ’96
•Lowe, ‘99
•Jacob & Barsi, ’99
• Mahamud and Herbert, 00
•Edelman et al. ’91
•Ullman & Barsi, ’91
• Rothwell ‘92
•Linderberg, ’94
•Murase & Nayar ‘94
•Rothganger et al., ‘04
•Ferrari et al, ’05
•Moreels and Perona, 05
•Brown & Lowe ’05
•Snavely et al ’06
•Yin & Collins, ‘07
•Ballard, ‘81
•Grimson & L.-Perez, ‘87
•Lowe, ’87
Single 3D object recognition
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
47
![Page 47: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/47.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
48
![Page 48: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/48.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
+ GPS
49
![Page 49: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/49.jpg)
Usual Challenges
Variability due to:
• View point
• Illumination
• Occlusions
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
50
![Page 50: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/50.jpg)
Basic Scheme
- Representation - Features
- 2D/3D Geometrical constraints
- Model learning
- Recognition - Hypothesis generation
- Validation
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
51
1. Rothganger et al. ’04, ’06
2. Brown et al, ‘05
3. Lowe ‘99, ‘04 4. Ferrari et al. ’04, ‘06
5. Lazebnick et al ’04
![Page 51: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/51.jpg)
Representation
Interest points -- or Regions (group of interest points)
• Detection
• Difference of Gaussian (DOG) [Lowe ’99]; Harris-Laplacian [Mikolajczyk & Schmid ’01]
• Kadir-Brady [Kadir et al. ‘01] ; Laplacian [Gårding & Lindeberg, ‘96]
• Adaptation [invariants] • Scale, rotation
• Affine
• Description • SIFT
• Color histograms
Geometrical constraints • 2D spatial layout of keypoints
• Tracks of keypoints (regions) across views
• 3D locations and/or surface normal
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
52
![Page 52: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/52.jpg)
Cou
rtesy o
f D. L
ow
e
Courte
sy o
f Roth
ganger e
t al
Difference of Gaussian (DOG): used in Lowe 99, Brown et al ‘05
Harris-Laplace: used in Rothganger et al. ‘06 Laplacian: used in Lazebnik et al. ‘04
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
53
![Page 53: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/53.jpg)
Representation
Interest points -- or Regions (group of interest points)
• Detection
• Difference of Gaussian (DOG) [Lowe ’99]; Harris-Laplacian [Mikolajczyk & Schmid ’01]
• Kadir-Brady [Kadir et al. ‘01] ; Laplacian [Gårding & Lindeberg, ‘96]
• Adaptation [invariants] • Scale, rotation
• Affine
• Description • SIFT
• Color histograms
Geometrical constraints • 2D spatial layout of keypoints
• Tracks of keypoints (regions) across views
• 3D locations and/or surface normal
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
54
• x,y
•Scale
•Orientation
•Affine structure
![Page 54: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/54.jpg)
Adaptation
• keypoints are transformed in order to be invariant to
translation, rotation, scale, and other geometrical parameters
Co
urte
sy o
f D. L
ow
e
Change of scale, pose, illumination…
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
55
![Page 55: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/55.jpg)
Scale & orientation adaptation
SIFT: Create histogram of local
gradient directions computed
at selected scale
Assign canonical orientation at
peak of smoothed histogram
0 2
Co
urte
sy o
f D. L
ow
e
[used in Lowe ’99]
• x,y
•Scale
•Orientation
detector
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
56
![Page 56: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/56.jpg)
Representation
Interest points -- or Regions (group of interest points)
• Detection
• Difference of Gaussian (DOG) [Lowe ’99]; Harris-Laplacian [Mikolajczyk & Schmid ’01]
• Kadir-Brady [Kadir et al. ‘01] ; Laplacian [Gårding & Lindeberg, ‘96]
• Adaptation [invariants] • Scale, rotation
• Affine
• Description • SIFT
• Color histograms
Geometrical constraints • 2D spatial layout of keypoints
• Tracks of keypoints (regions) across views
• 3D locations and/or surface normal
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
58
• x,y
•Scale
•Orientation
•Affine structure
![Page 57: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/57.jpg)
• Thresholded image gradients are sampled over 16x16 array of locations in
scale space
• Create array of orientation histograms
• 8 orientations x 4x4 histogram array = 128 dimensions
Co
urte
sy o
f D. L
ow
e
[Lowe ’99]
Keypoint description
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
59
![Page 58: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/58.jpg)
Representation
Interest points -- or Regions (group of interest points)
• Detection
• Difference of Gaussian (DOG) [Lowe ’99]; Harris-Laplacian [Mikolajczyk & Schmid ’01]
• Kadir-Brady [Kadir et al. ‘01] ; Laplacian [Gårding & Lindeberg, ‘96]
• Adaptation [invariants] • Scale, rotation
• Affine
• Description • SIFT
• Color histograms
Geometrical constraints • 2D spatial layout of keypoints
• Tracks of keypoints (regions) across views
• 3D locations and/or surface normal
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
60
• x,y
•Scale
•Orientation
•Affine structure
![Page 59: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/59.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
Object representation:
3D location of key points
Brown & Lowe ‘05 Rothganger et al. ’06
x,y,z +
descriptor
x,y,z +
h,v +
descriptor
Cou
rtesy o
f Roth
ga
ng
er e
t al
Co
urte
sy o
f Bro
wn
& L
ow
e
61
![Page 60: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/60.jpg)
Courte
sy o
f D. L
ow
e
[Lowe ’99]
Object representation
2D layout of key points
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
62
![Page 61: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/61.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
Object representation: Collections of semi-local affine parts
[Lazebnick et al ’04]
Parts = group of key points that share
‘consistent’ affine configuration across views
The learning technique establishes
the meaning of consistency
Cou
rtesy o
f La
ze
bn
ick
et a
l
63
![Page 62: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/62.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
Object representation: Collections of GAMs and tracks
[Ferrari et al ’04]
GAM (Group of Aggregated Matches): “A set of key point matches between two
images, which are distributed over a smooth connected surface of the object”
Cou
rtesy o
f Fe
rrari e
t al
See also: Tuytelaars & Van Gool, 2004
64
![Page 63: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/63.jpg)
Basic Scheme
- Representation - Features
- 2D/3D Geometrical constraints
- Model learning
- Recognition - Hypothesis generation
- Validation
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
65
![Page 64: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/64.jpg)
x,y,z +
h,v +
descriptor C
ou
rtesy o
f Roth
ga
ng
er e
t al
Rothganger et al. ‘03 ’06 Model learning
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
66
![Page 65: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/65.jpg)
Rothganger et al. ‘03 ’06 Model learning
• N images of object from N
different view points
• Match key points between
consecutive views
[ create sample set]
• Use affine structure from motion to
compute 3D location and orientation +
camera locations
Build a 3D model:
• Affine factorization Tomasi & Kanade ’92
• RANSAC
• 2 matches are needed rather than 4 thanks
to affine invariant patches
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
67
![Page 66: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/66.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
RANSAC
Algorithm:
Sample set = set of matches between views
1. Select a random sample of minimum required size [2 matches]
2. Compute a putative model from these
3. Compute the set of inliers to this model from whole sample space
4. Continue until model with the most inliers over all samples is found
Rothganger et al. ‘03 ’06
Cou
rtesy o
f Roth
ga
ng
er e
t al
[Affine factorization
Tomasi & Kanade ’92]
68
![Page 67: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/67.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
x,y,z +
descriptor
Co
urte
sy o
f Bro
wn
& L
ow
e
[Brown & Lowe ’05]
70
Model learning
![Page 68: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/68.jpg)
[Brown & Lowe ’05]
• N images of an object from N different view
points
• Match key points between view pairs [ create
sample set]
•Computer fundamental matrix F between view
pair [RANSAC]
• Find connected components across views
• Bundle adjustment & Metric upgrade
Model learning
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
71
Photosynth [Snavely et al. ‘06]
Courte
sy o
f Bro
wn &
Low
e
Full perspective model! (not affine)
![Page 69: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/69.jpg)
The model is a template of 2D layout of key points
[Lowe ’99]
Model learning
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
72
![Page 70: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/70.jpg)
Collections of semi-local affine parts
Parts = group of key points that share
‘consistent’ affine configuration across views
Cou
rtesy o
f La
ze
bn
ick
et a
l
[Lazebnick et al ’04]
73
Model learning
![Page 71: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/71.jpg)
Model learning
Collections of semi-local affine parts Goal: to find collections of local affine regions that can be mapped onto each other using a single affine transformation
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
74
A
[Lazebnick et al ’04]
Cou
rtesy o
f La
ze
bn
ick
et a
l
•Implementation: greedy search based on geometric and photometric consistency constraints
–Returns multiple correspondence hypotheses
–Automatically determines number of regions in correspondence
![Page 72: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/72.jpg)
Model learning
Collections of semi-local affine parts
Initialization:
• Identify triples of neighboring regions (i, j, k ) in first image
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
75
i
j
k
i'
j'
k'
Courtesy of Lazebnick et al
–Find all triples (i', j', k') in the second image such that i' (resp. j', k‘ ) is a
potential match of i (resp. j, k), and j', k' are neighbors of i'
![Page 73: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/73.jpg)
Collections of semi-local affine parts
Beginning with each seed triple, iterate:
• Estimate the affine transformation between centers of
corresponding regions in current group of matches
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
76
Model learning
A
Courtesy of Lazebnick et al
![Page 74: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/74.jpg)
Collections of semi-local affine parts
Beginning with each seed triple, iterate:
• Estimate the affine transformation between centers of
corresponding regions in current group of matches
• Determine geometric consistency of current group
of matches
• Search for additional matches in the neighborhood of the current
group
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
77
Model learning
• Geometric consistency
criteria: – Distance between ellipse
centers (residual)
– Difference of major and minor
axis lengths
– Difference of ellipse
orientations
![Page 75: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/75.jpg)
Collections of semi-local affine parts
Beginning with each seed triple, iterate:
• Estimate the affine transformation between centers of
corresponding regions in current group of matches
• Determine geometric consistency of current group
of matches
• Search for additional matches in the neighborhood of the current
group
- Stop when residual error is bigger than threshold CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
78
Model learning
![Page 76: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/76.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
Rothganger et al. ‘03 ’06
Co
urte
sy o
f Ro
thg
an
ge
r et a
l
81
Learnt models
![Page 77: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/77.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
Brown & Lowe ‘05
Cou
rtesy o
f Bro
wn
& L
ow
e
82
Learnt models
![Page 78: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/78.jpg)
[Lowe ’99]
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
83
Learnt models
![Page 79: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/79.jpg)
[Ferrari et al ’06]
[Lazebnick et al ’04]
Learnt models
84
![Page 80: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/80.jpg)
Basic Scheme
- Representation - Features
- 2D/3D Geometrical constraints
- Model learning
- Recognition - Hypothesis generation
- Validation
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
85
![Page 81: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/81.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
Verification: The hypothesis generates high
fitting error 86
• Hypothesis generation & model verification
• Basic idea
Recognition
![Page 82: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/82.jpg)
Hypothesis generation & model verification
Basic idea
Verification: The hypothesis generates low
fitting error
Recognition
87
![Page 83: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/83.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
RANSAC
Algorithm:
1. Select random sample of minimum required size to fit model [?]
2. Compute a putative model from sample set
3. Compute the set of inliers to this model from whole data set
Repeat 1-3 until model with the most inliers over all samples is found
Sample set = set of points in 2D
=[2]
88
![Page 84: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/84.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
RANSAC
Algorithm:
1. Select random sample of minimum required size to fit model [?]
2. Compute a putative model from sample set
3. Compute the set of inliers to this model from whole data set
Repeat 1-3 until model with the most inliers over all samples is found
O = 14
Sample set = set of points in 2D
=[2]
89
![Page 85: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/85.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
),(Pf
O
min OPI ,:
such that:
TTPPPPf
1),(
Model parameters
Line fitting
with outliers
90
![Page 86: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/86.jpg)
Hypothesis generation & model verification
• Find (appearance based) matches
between model keypoints and
test image
• Use RANSAC to find a set of
matches consistent with a
candidate camera pose: • For every 2 pairs of matches
• Compute camera
• Use camera to project other
matched 3D model patches
into test image
• Verification test
[Rothganger et al. ‘03 ’06]
Cou
rtesy o
f Roth
ga
ng
er e
t al
Recognition
91
![Page 87: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/87.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
[Rothganger et al. ‘03 ’06]
1. Find matches between model and test image features
92
Recognition
![Page 88: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/88.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
1. Find matches between model and test image features 2. Generate hypothesis:
•Compute transformation M from N matches (N=2; affine camera; affine key points)
3. Model verification • Use M to project other matched 3D model features into test image
• Compute residual = D(projections, measurements)
93
[Rothganger et al. ‘03 ’06]
Recognition
![Page 89: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/89.jpg)
[Rothganger et al. ‘03 ’06]
Goal: Estimate (fit) the best M in presence of outliers
Recognition
94
![Page 90: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/90.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
Initial matches based
on appearance Object to recognize
Matches verified with
geometrical constraints Recovered pose
Cou
rtesy o
f Roth
ga
ng
er e
t al
95
![Page 91: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/91.jpg)
[Lowe ’99, ’01, ’04]
Courte
sy o
f D. L
ow
e
Model Test image
Recognition
3 matches generate an hypothesis
(6 parameters model 2D affine transformation)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
96
![Page 92: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/92.jpg)
If the inlier/outlier ratio is too small RANSAC
doesn’t work…
[Lowe ’99, ’01, ’04]
Courte
sy o
f D. L
ow
e
• SOLUTION: Hough transform
–Vote for each potential match according to model ID
and pose
Recognition - generating hypothesis
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
97
![Page 93: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/93.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
x
y
n
m
x
y m 3 5 3 3 2 2
3 7 11 10 4 3
2 3 1 4 5 2
2 1 0 1 3 3
n 98
Hough Transform
![Page 94: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/94.jpg)
Each matched keypoint
Entry in Hough transform space
theta
Scale x
y
Hough transform
99
![Page 95: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/95.jpg)
Hough transform
100
![Page 96: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/96.jpg)
Recognition - model verification
1. Examine all clusters with at least 3 features
2. Perform least-squares affine fit to model.
3. Discard outliers and perform top-down check for
additional features.
4. Evaluate probability that match is correct
• Use Bayesian model, with probability that features
would arise by chance if object was not present (Lowe,
CVPR 01)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
101
[Lowe ’99, ’01, ’04]
Consistent configuration
![Page 97: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/97.jpg)
Basic Scheme
- Representation - Features
- 2D/3D Geometrical constraints
- Model learning
- Recognition - Hypothesis generation
- Validation
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
103
Let’s see some
results!
![Page 98: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/98.jpg)
Co
urte
sy o
f Ro
thg
an
ge
r et a
l
Rothganger et al. ‘03 ’06
• Handle severe clutter
3D Object Recognition results
104
![Page 99: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/99.jpg)
Brown & Lowe ‘05
Cou
rtesy o
f Bro
wn
& L
ow
e
Recognition
105
![Page 100: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/100.jpg)
Lowe. ’99, ’04
•Handle severe occlusions
•Fast!
Courte
sy o
f D. L
ow
e
3D Object Recognition results
106
![Page 101: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/101.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
107
Cou
rtesy o
f La
ze
bn
ick e
t al
107
[Lazebnick et al ’04]
3D Object Recognition results
![Page 102: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/102.jpg)
Training images
Test images (blue: occluded regions)
[Lazebnick et al ’04] C
ou
rtesy o
f La
ze
bn
ick e
t al
All regions found in the test images
3D Object Recognition results
108
![Page 103: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/103.jpg)
[Ferrari et al ’04]
Cou
rtesy o
f Fe
rrari e
t al
3D Object Recognition results
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
109
![Page 104: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/104.jpg)
A comparative experiment
110
![Page 105: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/105.jpg)
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
111
![Page 106: CEE598 - Visual Sensing for Civil Infrastructure Eng. & Mgmt.€¦ · Mani Golparvar-Fard Department of Civil and Environmental Engineering 3129D, Newmark Civil Engineering Lab ...](https://reader034.fdocuments.in/reader034/viewer/2022042910/5f40924645654a767a4676fb/html5/thumbnails/106.jpg)
Outline
Object Recognition
• Introduction
• Recognition of single 3D objects
• Bag of word models
• Part based models
• Models for 3D objects categorization
CEE598 Visual Sensing for Civil Infrastructure Eng. & Mgmt. © Mani Golparvar-Fard, 2013
112