Challenges to image parsing researchers
11
Challenges to image parsing researchers Lana Lazebnik UNC Chapel Hill sky sidewalk building road car person car mountain
description
Challenges to image parsing researchers. Lana Lazebnik UNC Chapel Hill. sky. mountain. building. person. car. car. sidewalk. road. The past: “closed universe ” datasets Tens of classes, hundreds of images, offline learning. Figure from Shotton et al. (2009). - PowerPoint PPT Presentation
Transcript of Challenges to image parsing researchers
Challenges to image parsing researchers
Lana LazebnikUNC Chapel Hill
sky
sidewalk
building
road
carpersoncar
mountain
The past: “closed universe” datasetsTens of classes, hundreds of images, offline learning
He et al. (2004), Hoiem et al. (2005), Shotton et al. (2006, 2008, 2009), Verbeek and Triggs (2007), Rabinovich et al. (2007), Galleguillos et al. (2008), Gould et al. (2009), etc.
Figure from Shotton et al. (2009)
Evolving images, annotations
http://labelme.csail.mit.edu/
The future: “open universe” datasets
build
ing floor seawater
sand
perso
nsky
scrap
er sign
mirror
pillow
founta
inflo
wersho
pcou
nter to
ppa
per
furnit
urecrane pot
arcad
ebri
dge
windshi
eld brick
clock
drawer fan
dishw
asher
vase
closet
hand
lebo
ttleou
tlet
bag
tail li
ght
lights
witch
02000000400000060000008000000
1000000012000000
Non-uniform class frequencies
The future: “open universe” datasets
Combination of local cues? Multiple segmentations/grouping
hypotheses? Context? Graphical models (MRFs, CRFs, etc.)? Offline learning and inference?
Which “closed universe” techniques can survive in the “open universe” setting?
Learning from all of LabelMe50K images, 232 labels
sky
tree
road
car
sky
sea
sun
building
window
door
road
sky
building
mountain
skybuilding
sidewalk
carroad
car
ceiling
wall
floor
Tighe & Lazebnik, work in progress
0%25%50%75%
100%SiftFlow Barcelona LM + Sun
0%25%50%75%
100%
Learning from all of LabelMe50K images, 232 labels
Per-class classification rates
Tighe & Lazebnik, work in progress
Challenge: Parsing high-res images
Challenge: Dynamic image interpretation
Image parsing algorithms should become autonomous decision-making agents
Visual “detective task”: Where was this photo taken?
Challenge: Dynamic image interpretation
Image parsing algorithms should become autonomous decision-making agents
Input
Summary Challenges to image parsing
researchers: Learn to parse images from “open
universe” evolving datasets Try parsing gigapixel images! Develop active, sequential image
interpretation strategies
