Vision, Video

and Virtual Reality SensorsSensors

Lecture 4Sensors

CSC 59866CDFall 2004

Zhigang Zhu, NAC 8/203A

http://www-cs.engr.ccny.cuny.edu/~zhu/Capstone2004/Capstone_Sequence2004.html

Vision, Video

and Virtual Reality AcknowledgementsAcknowledgements

The slides in this lecture were adopted and modified from lectures by

Professor Allen HansonUniversity of Massachusetts at Amherst

Vision, Video

and Virtual Reality SensorsSensors

Static monocular reflectance data (monochromic or color) Films Video cameras (with tapes) Digital cameras (with memory)

Motion sequences (camcorders) Stereo (2 cameras) Range data (Range finder) Non-visual sensory data

infrared (IR) ultraviolet (UV) microwaves

Many more

Vision, Video

and Virtual Reality The Electromagnetic SpectrumThe Electromagnetic Spectrum

Visible Spectrum

700 nm 400 nm

C = f

f

Vision, Video

and Virtual Reality The Human EyeThe Human Eye

Vision, Video

and Virtual Reality The EyeThe Eye

The Retina: rods (low-level light, night vision) cones (color-vision) synapses optic nerve fibers

Sensing and low-level processing layer 125 millions rods and cones feed into 1 million nerve fibers Cell arrangement that respond to horizontal and vertical lines

Retina

RodsCones

Vision, Video

and Virtual Reality Film, Video, Digital CamerasFilm, Video, Digital Cameras

Black and White (Reflectance data only) Color (Reflectance data in three bands - red, green, blue)

Vision, Video

and Virtual Reality Color ImagesColor Images

Blue Green Red

‘Dimensions’ of an Image

Spatial (x,y)Depth (no. of components)Number of bits/channelTemporal (t)

Pixel

Spatial Resolution

Spectra Resolution

Radiometric Resolution

Temporal Resolution

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

Crab Nebula

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

Cargo inspection using Gamma Rays

Mobile Vehicle and Cargo Inspection System (VACIS®)

Gamma rays are typically waves of frequencies greater than 1019 Hz

Gamma rays can penetrate nearly all materials and are therefore difficult to detect

Courtesy:Science Applications International Corporation (SAIC),

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

Cargo inspection using Gamma Rays

Mobile Vehicle and Cargo Inspection System (VACIS®)

Gamma rays are typically waves of frequencies greater than 1019 Hz

Gamma rays can penetrate nearly all materials and are therefore difficult to detect

Courtesy:Science Applications International Corporation (SAIC),

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

Cargo inspection using Gamma Rays

Mobile Vehicle and Cargo Inspection System (VACIS®)

Gamma rays are typically waves of frequencies greater than 1019 Hz

Gamma rays can penetrate nearly all materials and are therefore difficult to detect

Courtesy:Science Applications International Corporation (SAIC),

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

Medical X-Rays

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

Chandra X-Ray Satellite

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

From X-Ray images to 3D Models: CT Scans

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

Flower Patterns in Ultraviolet

Dandelion - UV

Po

ten

till

a

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

Messier 101 in Ultraviolet

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

Traditional images

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

Non-traditional Use of Visible Light: Range

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

Scanning Laser Rangefinder

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

IR: Near, Medium, Far (~heat)

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

IR: Near, Medium, Far (~heat)

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

IR: Finding chlorophyll -the green coloring matter of plants that functions in photosynthesis

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

(Un)Common uses of Microwaves

CD MovieExploding Water

Movie

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

Microwave Imaging: Synthetic Aperture Radar (SAR)

San Fernando Valley Tibet: Lhasa River

Thailand: Phang Hoei RangeAthens, Greece

Red: L-band (24cm)

Green: C-band (6 cm)

Blue:C/L

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

Radar in Depth: Interferometric Synthetic Aperture Radar - IFSAR(elevation)

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

Low Altitude IFSAR

IFSAR elevation, automatic, in minutes

Elevation from aerial stereo, manually, several days

Vision, Video

and Virtual Reality Across the EM SpectrumAcross the EM Spectrum

Radio Waves (images of cosmos from radio telescopes)

Vision, Video

and Virtual Reality Stereo GeometryStereo Geometry

Single Camera (no stereo)

Vision, Video

and Virtual Reality Stereo GeometryStereo Geometry

P(X,Y,Z)

f = focal length

Optical Center

pr(x,y)

Film plane

pl(x,y)

Optical Center

f = focal length

Film plane

LEFT CAMERA RIGHT CAMERA

B = Baseline

Vision, Video

and Virtual Reality Stereo GeometryStereo Geometry

LEFT IMAGE RIGHT IMAGE

Disparity = xr - xl

P

Pr(xr,yr)Pl(xl,yl)

≈ depth

Vision, Video

and Virtual Reality Stereo ImagesStereo Images

A Short Digression

StereoscopesStereoscopes

Vision, Video

and Virtual Reality Stereo ImagesStereo Images

Darjeeling Suspension

Bridge

Vision, Video

and Virtual Reality Picture of you?Picture of you?

Vision, Video

and Virtual Reality StereoStereo

Stereograms

Vision, Video

and Virtual Reality Stereo X-RayStereo X-Ray

Vision, Video

and Virtual Reality Range SensorsRange Sensors

Light Striping

David B. Cox, Robyn Owens and Peter HartmannDepartment of BiochemistryUniversity of Western Australia http://mammary.nih.gov/reviews/lactation/Hartmann001/

Vision, Video

and Virtual Reality MosaicsMosaics

A mosaic is created from several images

Vision, Video

and Virtual Reality MosaicsMosaics

Stabilized Video

Vision, Video

and Virtual Reality MosaicsMosaics

Depth from a Video Sequence (single camera)

P(X,Y,Z)

Height H from Laser Profiler

GPS

Vision, Video

and Virtual Reality MosaicsMosaics

Brazilian forest…..made at UMass CVL

Vision, Video

and Virtual Reality Why is Vision Difficult?Why is Vision Difficult?

Natural Variation in Object Classes: Color, texture, size, shape, parts, and relations

Variations in the Imaging Process Lighting (highlights, shadows, brightness, contrast) Projective distortion, point of view, occlusion Noise, sensor and optical characteristics

Massive Amounts of Data 1 minute of 1024x768 color video = 4.2 gigabytes

(Uncompressed)

Vision, Video

and Virtual Reality The Need for KnowledgeThe Need for Knowledge

Knowledge

Function

Context

Shape

SpecificObjects

GenericObjects

Structure Size

Shape

Motion

Purpose

Variation

Vision, Video

and Virtual Reality The Figure Revealed The Figure Revealed

Light Source

Vision, Video

and Virtual Reality The Effect of ContextThe Effect of Context

Vision, Video

and Virtual Reality The Effect of Context - 2The Effect of Context - 2

Vision, Video

and Virtual Reality Context, cont.Context, cont.

….a collection of objects:

Vision, Video

and Virtual Reality ContextContext

The objects as hats:

Vision, Video

and Virtual Reality

And as something else…..

‘To interpret something is to give it meaning in context.’

ContextContext

Vision, Video

and Virtual Reality Vision System ComponentsVision System Components

…..at the low (image) level, we need Ways of generating initial descriptions of the image data Method for extracting features of these descriptions Ways of representing these descriptions and features Usually, cannot initially make use of general world

knowledge

IMAGE(numbers)

DESCRIPTION(symbols)

Vision, Video

and Virtual Reality

….at the intermediate level, we need Symbolic representations of the initial descriptions Ways of generating more abstract descriptions from the initial ones (grouping) Ways of accessing relevant portions of the knowledge base Ways of controlling the processing

Intermediate level processes should be capable of being used top-down (knowledge-directed) or bottom-up (data-directed)

IMAGEIINTERMEDIATEDESCRIPTIONS

KNOWLEDGE

Vision System ComponentsVision System Components

Vision, Video

and Virtual Reality Vision System ComponentsVision System Components

….at the high (interpretation) level, we need Ways of representing world knowledge

Objects Object parts Expected scenarios (relations) Specializations

Mechanisms for Interferencing Beliefs Partial matches

Control Information Representations of

Partial interpretations Competing interpretations Relationship to the image descriptions

Vision, Video

and Virtual Reality NextNext

Anyone who isn't confused really doesn't understand the situation.

--Edward R. Murrow

Next:Image Formation

Reading: Ch 1, Ch 2- Section 2.1, 2.2, 2.3, 2.5

Questions: 2.1. 2.2, 2.3, 2.5

Exercises: 2.1, 2.3, 2.4