Institute of Electrical Measurement and Measurement Signal ... · Representation and Processing of...
Transcript of Institute of Electrical Measurement and Measurement Signal ... · Representation and Processing of...
Institute of Electrical Measurement and Measurement Signal Processing
1
Axel Pinz WS 2017/18 Image and Video Understanding 3
To Conclude: Vision in terms of …
• …Neurophysiology Receptive fields
Left/right hemisphere
Visual pathway, packing problem, columns, complementary features
• …Cognitive psychology– Perceptual grouping
– Bottom-up vs. top-down processes
– Optical illusions
– Hemispheres, motion perception, …
• …Information processing
The “Marr Paradigm”
Institute of Electrical Measurement and Measurement Signal Processing
2
Axel Pinz WS 2017/18 Image and Video Understanding 3
David Marr [1945-1980] – Vision [1982]
VISION
A Computational Investigation into the Human
Representation and Processing of Visual Information
Freeman Co., 1982NEUROBIOLOGY
Institute of Electrical Measurement and Measurement Signal Processing
3
Axel Pinz WS 2017/18 Image and Video Understanding 3
David Marr – Vision
Institute of Electrical Measurement and Measurement Signal Processing
4
Axel Pinz WS 2017/18 Image and Video Understanding 3
David Marr – Vision
Institute of Electrical Measurement and Measurement Signal Processing
5
Axel Pinz WS 2017/18 Image and Video Understanding 3
David Marr – Vision
“What does it mean, to see? The plain man’s answer
(and Aristotle’s, too) would be, to know what is where by
looking. In other words, vision is the process of
discovering from images what is present in the world, and
where it is.” (p.3, 1st paragraph of General Introduction)
Vision Image Understanding: To know what is where.
D. Marr
Video Understanding: What is where and when?
(borrowed from D. Marr) in space and time
3D 2D, Reconstruction vs. recognition
4D 3D
Institute of Electrical Measurement and Measurement Signal Processing
6
Axel Pinz WS 2017/18 Image and Video Understanding 3
Marr – Vision: Emphasis on Reconstruction
Institute of Electrical Measurement and Measurement Signal Processing
7
Axel Pinz WS 2017/18 Image and Video Understanding 3
Marr – Vision:
Emphasis on
Reconstruction
Institute of Electrical Measurement and Measurement Signal Processing
8
Axel Pinz WS 2017/18 Image and Video Understanding 3
The “Marr Paradigm” – “Computational Framework”
Stone (Vision and Brain, 2012): “computational framework” suggests:
“Vision works like a computer”. Better: “informational framework” …“… because Marr was keen to emphasize the nature of information being processed
without necessarily referring to the particular machinery (e.g., neurons or chips)…”
Institute of Electrical Measurement and Measurement Signal Processing
9
Axel Pinz WS 2017/18 Image and Video Understanding 3
The “Marr Paradigm” – Analogy with Flying
Marr (p.27): “Importance of Computational Theory”
“… an algorithm is likely to be understood more readily by understanding
the nature of the problem being solved than by examining the mechanism
(and the hardware) in which it is embodied.”
“… trying to understand perception by studying only neurons is like trying to
understand bird flight by studying only feathers: It just cannot be done.”
First understand aerodynamics, then think about structures of feathers,
shape of wings etc.
Wright brothers 1902
(from [Stone, 2012])
Institute of Electrical Measurement and Measurement Signal Processing
10
Axel Pinz WS 2017/18 Image and Video Understanding 3
The “Marr Paradigm” – “Computational Framework”
3D surface shape by
finding surface normals
from shading information
SfS
0 90 180
Greylevels
Surface
normals
Neurons
A single CPU
Multicore CPUs
GPUs
“…frogs passing cupcakes.” [Stone]
Institute of Electrical Measurement and Measurement Signal Processing
11
Axel Pinz WS 2017/18 Image and Video Understanding 3
David Marr – Vision
Representational Framework
Primal sketch
2-1/2D sketch
3D model
Institute of Electrical Measurement and Measurement Signal Processing
12
Axel Pinz WS 2017/18 Image and Video Understanding 3
David Marr – Vision
Representational Framework
Primal sketch
2-1/2D sketch
3D model
“viewer centered”
“object centered”
Institute of Electrical Measurement and Measurement Signal Processing
13
Axel Pinz WS 2017/18 Image and Video Understanding 3
Marr – Primal Sketch
Compare today’s interest point, line, edge detection, etc.
“Raw primal sketch” “full primal sketch” (includes grouping)
“saliency” !
Institute of Electrical Measurement and Measurement Signal Processing
14
Axel Pinz WS 2017/18 Image and Video Understanding 3
Marr – 2-1/2D Sketch
Surface patches (surface normals), depth discontinuitites
Institute of Electrical Measurement and Measurement Signal Processing
15
Axel Pinz WS 2017/18 Image and Video Understanding 3
Marr – 3D Model Representation
Generalized cylinder, generalized cone
3D hierarchical models
Institute of Electrical Measurement and Measurement Signal Processing
16
Axel Pinz WS 2017/18 Image and Video Understanding 3
David Marr – Vision
Representational Framework
Primal sketch
2-1/2D sketch
3D model
Institute of Electrical Measurement and Measurement Signal Processing
17
Axel Pinz WS 2017/18 Image and Video Understanding 3
Defining the Terms – Image Understanding
+ Video Understanding
Processing
Image Understanding
Image Scene description
Computer Graphics
Institute of Electrical Measurement and Measurement Signal Processing
18
Axel Pinz WS 2017/18 Image and Video Understanding 3
Scene Description
Please describe this scene:
Many possible (+correct!) descriptions
Correct/best description may depend on the particular goal(s)
“purposive, qualitative, active vision” [Aloimonos, 1992]
Institute of Electrical Measurement and Measurement Signal Processing
19
Axel Pinz WS 2017/18 Image and Video Understanding 3
My Model of Image Understanding [Pinz, 1994]
… Repräsentationen
… Prozesse
… Datenfluss
… Kontrollfluss
Institute of Electrical Measurement and Measurement Signal Processing
20
Axel Pinz WS 2017/18 Image and Video Understanding 3
My Model of Image Understanding
Institute of Electrical Measurement and Measurement Signal Processing
21
Axel Pinz WS 2017/18 Image and Video Understanding 3
My Model of Image Understanding
KU: 2D image
and scene description
Up to WS 2014/15:
- Mostly 2D
- Image understanding
This course:
- Can this be extended towards video understanding?
Institute of Electrical Measurement and Measurement Signal Processing
22
Axel Pinz WS 2017/18 Image and Video Understanding 3
2D Scene Description
“houses” [Matsuyama’90] “face” [Brunelli’92] “pedestrians” [Suzuki’90]
2D “image objects” “tokens”
Institute of Electrical Measurement and Measurement Signal Processing
23
Axel Pinz WS 2017/18 Image and Video Understanding 3
2D (+time!) Video Description
Fast object segmentation in unconstrained video
[Papazoglou&Ferrari, ICCV’13]
http://groups.inf.ed.ac.uk/calvin/FastVideoSegmentation/
Institute of Electrical Measurement and Measurement Signal Processing
24
Axel Pinz WS 2017/18 Image and Video Understanding 3
2D (+time!) Video DescriptionFast object segmentation in unconstrained video [Papazoglou&Ferrari, ICCV’13]
Institute of Electrical Measurement and Measurement Signal Processing
25
Axel Pinz WS 2017/18 Image and Video Understanding 3
3D Scene Description
Scene-
coordinate
system S
Object 2
Object 1
Institute of Electrical Measurement and Measurement Signal Processing
26
Axel Pinz WS 2017/18 Image and Video Understanding 3
3D (+time!) Video Description
Institute of Electrical Measurement and Measurement Signal Processing
27
Axel Pinz WS 2017/18 Image and Video Understanding 3
More Definition: Visual Recognition [Perona’09]
“The holy grail of Computer Vision”
Five tasks of “visual recognition”:
– Verification (is a “car” in the image?)
– Detection and localization (what is there? where?)
– Classification (n “beach” images, m “city” images)
– Naming (name and locate all objects in an image)
– Description: objects, actions, relations, etc.
(example “kissing” “scene understanding”)
Increasing complexity from top bottom
Image and Video Understanding: mostly 2D (+time) recognition
Image-based Measurement: 3D (+time) reconstruction
Co
mp
lexity
Institute of Electrical Measurement and Measurement Signal Processing
28
Axel Pinz WS 2017/18 Image and Video Understanding 3
2D Scene Representation and Description
You can get very far in 2D !
2D “image object”
“token”
“tokenset”
2D scene description
image
image
description
segmentation
2D grouping