SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding...
-
Upload
payton-hornbrook -
Category
Documents
-
view
212 -
download
0
Transcript of SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding...
![Page 1: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/1.jpg)
SPONSORED BYSA2014.SIGGRAPH.ORG
MCGraph:
Multi-criterion representation for scene understanding
Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado
University College London
![Page 2: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/2.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Motivation
Indoor scene analysisAcquisition easier
Less constrained scenes, growing complexity
Target more complex: object counting segmentation, labelling
![Page 3: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/3.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Motivation
Indoor scene analysisAcquisition easier
Less constrained scenes, growing complexity
Target more complex: object counting segmentation, labelling
Typical processing:
Vision (SLAM) 3D2D images, temporal information, depth maps, pointclouds, 3D models
pointclouds, shapes, 3D meshes
localization, mapping, reconstruction, model recognition and fitting
local geometric or multi-scale features, abstraction by primitives, shapes from collections, inference of categorical knowledge, functional information (interaction)
![Page 4: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/4.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Motivation
Few types of information at a time
Complex scene understanding different information domains at the same time
Stacked, disjoint abstraction layers
Joint representation with mutual refinement
Concurrent information processing > iterative
![Page 5: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/5.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
MCGraph
MCGraph – a unified Multi-Criterion data representation
Structure and meaning modelled separately,connected fully
For understanding and processing of large-scale 3D scenes
A standardized structure to format the data created by our community
Prior knowledge
Discovered knowledge
Abstraction graph
![Page 6: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/6.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Related work - viewpoints
2D 3D
Arrangements of … features [Felzenszwalb10] , … recurring parts in shape collections [Zheng14] , …
![Page 7: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/7.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Related work - viewpoints
2D 3D
Arrangements of … features [Felzenszwalb10] , … recurring parts in shape collections [Zheng14] , …
Abstraction by … super-pixels [Zitnick04], …regions [Gould09, Kumar10] , …features[Hoiem08] , …
planes [Gallup07] , …primitives [Schnabel07] , …cuboids [Fidler12,Shao14] , …
![Page 8: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/8.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Related work - viewpoints
2D 3D
Arrangements of … features [Felzenszwalb10] , … recurring parts in shape collections [Zheng14] , …
Abstraction by … super-pixels [Zitnick04], …regions [Gould09, Kumar10] , …features[Hoiem08] , …
planes [Gallup07] , …primitives [Schnabel07] , …cuboids [Fidler12,Shao14] , …
Graph bases representation of …
RGB segmentation [FelzenszwalbHuttenlocher04], [Rother04], …
RGBD segmentation [Zheng13], …inter-shape analysis [Mitra14], …joint shape segmentation [Huang11], ...shape collections [Fish*14], …
![Page 9: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/9.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Related work - criteria
Few criteriaAppearance co-occurrence, relative spatial layout [Hedau10]
Primitive abstraction, physics [Gupta10]
Semantic labelling [Huang13]
![Page 10: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/10.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Related work - criteria
Few criteriaAppearance co-occurrence, relative spatial layout [Hedau10]
Primitive abstraction, physics [Gupta10]
Semantic labelling [Huang13]
Multi-criteria 2DAppearance, shape, context [Shotton09]
Verbal descriptions of actors, actions + object properties, relations [Zitnick13]
Material, function, spatial envelope [Patterson14]
![Page 11: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/11.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Related work - criteria
Few criteriaAppearance co-occurrence, relative spatial layout [Hedau10]
Primitive abstraction, physics [Gupta10]
Semantic labelling [Huang13]
Multi-criteria 2DAppearance, shape, context [Shotton09]
Verbal descriptions of actors, actions + object properties, relations [Zitnick13]
Material, function, spatial envelope [Patterson14]
Multi-criteria 3DRegularities of shape collections, function [Laga13]
Intra and inter-object symmetries, physics, function [Mitra10]
![Page 12: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/12.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Related work – formalisation
Information fusionMulti-Criteria Decision Analysis [Doumpos13]
Hypergraphs [Zhang14]Low-level processing
Hmida et al. [2013]Separates knowledge from abstraction
Processing and representation coupled [Hmida et al.
2013]
![Page 13: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/13.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Classic model
Prior knowledge: Graph nodes (object types)
Graph edges (relationship types)
Discovered knowledge:Graph layout
Coupled representation of a priori and discovered
![Page 14: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/14.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Proposed model
Prior knowledge, discovered knowledge and abstraction separate
Inspired by graph databasesLabelling = edge between object and label
Supports multi-criteria processing
![Page 15: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/15.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
1. Abstraction graph
Represents objects and relations
Abstractions may be connected to segments of data
Segments can be overlapping
Object hierarchies
![Page 16: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/16.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
2. Knowledge graph
Encodes prior knowledge -> “knowledge units”
Hierarchical
Multi-criterion by separate sub-graphs
Is defined a priori - portable
Can have internal edges
![Page 17: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/17.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
3. Relation set
Represents discovered knowledge, “labellings”
Abstractions labels, or relation nodes labels
Edge can store parameters to represent an instantiation (i.e. primitive size)
![Page 18: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/18.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Example
Segmentation knowledge sub-graph
Super-pixelsRegions
![Page 19: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/19.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Example
Bounding box
Bounding box
Segmentation knowledge sub-graph
![Page 20: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/20.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Example
Abstraction knowledge sub-graph
![Page 21: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/21.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Example
Relations knowledge sub-graph
![Page 22: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/22.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Example
Legs same size, and parallel
![Page 23: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/23.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Common knowledge sub-graphs
Primitive proxiesI.e. primitive – polyhedron – cuboid, pyramid
Hierarchy induces high inference power
SemanticHierarchical object labels
Classification, function, etc.
RelationshipsCoaxial, co-planar, equiangular [Li11]
Covers, supports, occluded by, belong together [Gupta10]
Can be searched by sub-graph matching [Schnabel08]
![Page 24: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/24.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Common knowledge sub-graphs
Primitive proxiesI.e. primitive – polyhedron – cuboid, pyramid
Hierarchy induces high inference power
SemanticHierarchical object labels
Classification, function, etc.
RelationshipsCoaxial, co-planar, equiangular [Li11]
Covers, supports, occluded by, belong together [Gupta10]
Can be searched by sub-graph matching [Schnabel08]
Large freedom, high representational power
Use cases: Primitive abstraction, RGBD annotationSee paper
![Page 25: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/25.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Extension 1. Scene collections for assisted living
Robot assistant needs to be able to reach the subject at all times
Pro-active discovery of dynamic environmentNeeds to identify danger sources
Vision, scene-collections, intra-domain inference
MCGraph: “Objects that easily tip over, and incur danger, when in contact with water”
Movable
Electric, movable, unstable
Emits water
![Page 26: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/26.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Extension 2. - Multi-criteria multi-scale
Multi-scale analysis of scene understanding [Mitra14]HKS [Sun09], GLS [Mellado12]
Multi-scale similarity queries [Hou12]
Only spatial domain, controlled environments
Open-world problem
Multi-criteria multi-scale look-ups
Scan
?
Scale Time Location
Classification
![Page 27: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/27.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Extension 3. Prior knowledge for object registration
Geometric priors, priors on part-relations, function
If you discover, what you are scanning, you can use the extra information to enhance quality
looks like engine generic engine model prior
will have axes and wheels
looks like cargeneric car model prior
function
lookup working CAD modelwill have shiny material
decrease RGB weight in registration
Co-occurrence
![Page 28: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/28.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Conclusion
LimitationsPortability -> standardization
Diversity -> efficiency
Data online
Flexible and extendable representation
A data structure to standardize storage of annotated data
Can start a discussion, debate and movement how to harness the powers of multi-criteria problem representations, focused on 3D scene understanding
![Page 29: SPONSORED BY SA2014.SIGGRAPH.ORG MCGraph: Multi-criterion representation for scene understanding Moos Hueting ∗ Aron Monszpart ∗ Nicolas Mellado University.](https://reader031.fdocuments.in/reader031/viewer/2022032517/56649cc05503460f949876a9/html5/thumbnails/29.jpg)
SA2014.SIGGRAPH.ORG SPONSORED BY
Thank you!
http://geometry.cs.ucl.ac.uk/projects/2014/mcgraph/