1

Image Basics

2

Outline

Image formation: the pinhole camera modelImages as functionsDigital imagesColor, light and shading

Reading: textbook: 2.1, 2.2, 2.4

3

Image formation

Images are acquired through some device from some real-world scene, anatomy, etc

4

Images acquired with cameras

Basic abstraction: pinhole cameraAbstract camera model - box with a small hole in itPinhole cameras work in practiceThe pinhole perspective projection equations were discovered by Brunelleschi, in the 15th century.First pinhole camera: 16th century.still represent the most used theoretical camera model

5

Camera obscura : each point on the image plane sees light from only one direction, the one that passes through the pinhole. The pinhole is the center of projection through which all light passes.

! Perspective projection creates inverted images.

! It is sometimes convenient to consider a virtual image in a plane lying in front of a pinhole and symmetric to the image plane with respect to the pinhole point.

6

Pinhole optics

Using ray-tracing, we see that only a narrow light beam passes through a pinhole

A) In a wide pinhole, light from the source spreads across the image, making it blurry.

B) In a narrow pinhole, only a small amount of light is let in. The image is sharper.

Small apertures require longer exposure times.

The sharpness is limited by diffraction.

7

Pinhole too big -many directions areaveraged, blurring theimage

Pinhole too small-diffraction effects blurthe image.

Generally, images from pinhole cameras are dark, because a very small set of rays from a particular point hits the screen.

8

Pinhole optics summary

Pinhole optics focuses images:- without lenses- with an infinite depth of field- Depth of field = The distance between the nearest

and farthest objects that appear in acceptably sharp focus in the image.

Small pinhole:- Better focus- Less light energy available from any scene point - The sharpness is limited by difraction

9

The perspective projection equation

10

Perspective effect: Far objects appear smaller than close ones

! The image plane is behind the pinhole (inverted images).

11

Pinhole optics: Horizon and vanishing points

(virtual image plane).

H is the horizon line. Considering all the possible sets of parallel lines in plane Π, their intersection (vanishing points) lie on the horizon line.

12Parallel lines meet at vanishing points

13

Image functions

The image can be modeled by a function of two or three variables;

f(x,y)f(x,y,z)f(x,y,t)

Values in an image can be of many types:Scalars: monochromatic images;Physical significance: X-Ray, MRI, Range imagesVectors:

color images (R,G,B); LANDSAT images ( 7 distinct channels)

14

15

Digital images

Sampling=spacing of discrete values in the domain of an image

sampling rate–how many samples are taken per unit of each dimension. “dots per inch”, etc.

Quantization= spacing of discrete values in the range of an image

number of bits per pixel. “black and white images”(1 bit per pixel), “24-bit color images”, etc.

Sampling and quantization are independentShannon’s sampling theorem: must sample at atleast twice the highest spatial frequency in the image.Resolution: ability to discern fine detail in the image

16

Sensing illuminated objects

Light reaches surfaces in 3DSurfaces reflectSensor element receives light energyWhat counts:

Light intensityAnglesSurface material

17

The perception of an object’s brightness and colour depends on:

The amount of energy and its spectral distribution (various wavelengths) illuminating the object surfaceThe spectral reflectance of the object surface (i.e. how the surface changes the received spectrum into the radiated spectrum)The spectral sensitivity of the sensor (human eye, CCD array etc) irradiated by the light energy from the object’s surface

18

Non-white light sources

(Left) a ‘warm’ light source: it enhances reds and oranges while dulling blues and greens; (Middle) a neutral light source; (Right) a ‘cool’ source: it enhances blues and greens while dulling reds and oranges.

19

Light SourcesGeneral light sources are difficult to work with. We must integrate light coming from all points on the source.

20

Light source models

Ambient lightDirectional lightPoint light Etc.

Important for computer vision but also for computer graphics;Computer Vision: known light sources allows for shadow detection and removalComputer Graphics: source light for realistic scene rendering (shadow and specularitymodeling) and for facilitating 3D data visualization

21

Ambient light

Objects not directly lit are typically still visible

ceilings, undersides of desks etc.

This is the result of indirect illuminationfrom emitters, undergoing multiple reflections from intermediate surfacesThe ambient light source model

illuminates all surfaces equallyamount reflected depends only on surface propertiesis the preferred model in Computer Vision

22

Ambient light (cont’d)

The ambient light reflected from a surface depends on:

-The surface properties, kambient

-The intensity of the ambient light source (constant for all points on all surfaces)

Ireflected = kambient Iambient

The ambient light is not necessarily white!

All object points have the same intensity in the image!

Adapted from David Luebke,

Lecture notes, Introduction to Computer Graphics

23

Directional Light Sources

Simplifying assumption: all rays of light from the source are parallel

As if the source were infinitely far away from the surfaces in the sceneA good approximation to sunlight

The direction from a surface to the light source is important in lighting the surface

Adapted from David Luebke,

Lecture notes, Introduction to Computer Graphics

24

Directional Light Sources

The same scene lit with a directional and an ambient light source

Adapted from David Luebke,

Lecture notes, Introduction to Computer Graphics

25

Point Light SourcesA point light source emits light equally in all directions from a single point The direction to the light from a point on a surface thus differs for different points:

we need to calculate a normalized vector to the light source for every point we light:

pAdapted from David Luebke,

Lecture notes, Introduction to Computer Graphics

26

What happens to light when it arrives at surfaces ?

Consider reflection from the object surfaceThe fraction of the incident radiation that a surface element reflects is called its albedo. Usually, the albedo is considered an intrinsic property.Albedo is low for dark surfaces and high for light surfaces

27

An ideal diffuse reflector, at the microscopic level, is a very rough surface (example: chalk)

Because of these microscopic variations, an incoming ray of light is equally likely to be reflected in any direction over the hemisphere:

Ideal diffuse (Lambertian) reflection

28

Lambert’s Law

The reflected intensity depends on the angle between the normal of the surface and the direction of the incident ray of light.

29

Diffuse reflection (cont’d)Diffuse or Lambertian reflection distributes energy uniformly in all directions of the hemisphere centered at a surface element. Thus, it reflects light uniformly in all directions.A planar patch will have uniform brightness for all viewpoints from which that patch is visible.A surface element is not visible when n • v < 0 where v is the direction towards the viewpoint

From Shapiro and Stockman

30

Diffuse Lighting Example

Lambertian objects:

- Vase and egg

A plot of intensities along one row

Shape of the plot shows that the shape of the object surface is closely related to the amount of reflected light.

From Shapiro and Stockman

31

Another Diffuse Lighting Example

If we assume a diffuse reflection model, what can we say about the position of the lighting source?

32

Specular reflection

Specular reflecting surfaces are very smooth at a microscopic level. Ex: polished metal, glossy car finish.They behave like a mirror, reflecting almost all the received illumination along the ray of reflection. The reflected energy will have the same spectral composition as the received light, regardless of the surface colour.Specular highlights are white when a white light source is used.They are also view dependent, since reflection obeys Snell’s law.

33

Example of specular reflection

Veggie Vision, IBM

34

Attenuation (darkening) with distance

Conservation of energy: the total energy radiating from a point source of constant energy flux per unit of time through any enclosing spherical surface is the same.

24 rAS π=2

1r

≈The energy per unit area

Similarly, light energy reflecting off a surface element decreases in intensity with the square of the distance from which the surface is viewed.

35

Shading

Image intensities are non-uniform due to the non-uniform distances along the imaging rays.

Cast shadows are caused by the fact that the objects block a certain amount of light energy (opaque assumption)

Computer Vision: how to accurately detect shaded objects based on colour information

Computer Graphics: how to model shading in order to create a realistic image of the object

36

Colour perception

The perception of an object’s color depends on 3 factors:

- the spectrum of energy in various wavelengths illuminating the object surface

- the spectral reflectance of the object surface

- the spectral sensitivity of the sensor irradiated by the light energy from the object’s surface.

37

Colour perception: example

We usually think of colour as an intrinsic property of objects.However:

A ‘blue’ object has a surface material that appears blue when illuminated with white light. The same object will appear violet if illuminated only with red light.A blue car under intense sunlight (white) will become hot and radiate energy in the IR spectrum (which can be imaged with an IR camera)

38

Spectral sensitivity of the sensor

A pair of ’visible’ (left) and IR (right) images from the same scene. The IR and visible cameras share approximately the same field ofview

39

Sensors for colour perception in the human eye

Three types of cones having the ability to sense three different (but overlapping) spectral regions.

40

Sensitivity curves for cones

- ‘blue’ cone is mildly sensitive to blue light 400-500nm (peak at 430 nm)

- ‘green’ cone is very sensitive to green light, but also sensitive to blue and red (peak at 560)

- ‘red’ cone (peak at 610)

41

The three-cone theorysuppose that light in the yellow range of wavelengths strikes the retina. it activates both the green and the red cones of the retina. electrical messages are sent by both the red and the green cones to the brain. The brain recognizes that the light has activated both the red and the green cones and somehow interprets this to mean that the object is yellow.In this sense, the yellow appearance of objects is simply the result of yellow light that stimulating the red and the green cones simultaneously.

42

The three cone theory (cont’d)

http://www.glenbrook.k12.il.us/gbssci/phys/Class/light/u12l2b.html

43

Colour representation in the RGB basis

The primary colors of light (red, green and blue) are those that stimulate one type of cone most predominantly. An arbitrary colour is created by mixing primary colours in appropriate amounts. The RGB system is additive because it involves simultaneous light emission. a yellow sensation can be obtained by combining red light and green light

44

Colour representation in RGB basis

Normalized (r,g,b) coordinates:

BGRBbbluenormalized

BGRGggreennormalized

BGRRrrednormalized

BGRI

++=

++=

++=

++=

;3

Useful when object surface is not uniformly illuminated (shadows)

45

Colour representation in CMY basis

Printers produce color by reflective light The process describes what kind of inks need to be applied so that the light reflected from the white substrate (paper) and passing through the inks produces a given colour. This is a subtractive process and uses a model based on the colors: Cyan, Magenta, Yellow.cyan = green + blue, so light reflected from a cyan pigment has no red component, i.e., the red is absorbed by cyan. Similarly magenta subtracts green and yellow subtracts blue. The conversion between the RGB and CMY is easily computed as below:

C=1-RM=1-GY=1-B

46

The HSV colour space

HSV=[Hue Saturation Value]Same thing as HSI=[Hue Saturation Intensity]This representation allows for an efficient separation of brightness (V) and chromaticity (H,S) information.It is useful for shadow removal

47

The HSV colour space

http://www.siggraph.org/education/materials/HyperGraph/color/colorhs.htm

We obtain the hexagon by looking at the RGB cube along the gray diagonal

48

The HSV colour space

HSV color model is more intuitive than RGB The user specifies a color (hue) and then adds white or black. There are 3 color parameters: Hue, Saturation, and Value. Changing the S parameter corresponds to adding or subtracting white (creating tints)Changing the V parameter corresponds to adding or subtracting black (creating shades)

For example: pure blue H = 240°, S = V = 1dark blue H = 240°, S = 1, V = 0.40light blue H = 240°, S = .3, V = 1.0

Adapted from http://www.siggraph.org/education/materials/HyperGraph/color

49

Shadow removal

Cast shadows are most difficult to deal with.Motion information has little relevance: shadow motion is consistent with object motion.Colour information is useful.The shadow cast by a moving object exhibits similar chromaticity but lower brightness than the corresponding background region.

50

Comparing color codes

51

Exploiting colour information in computer vision

52

Colors can be used for image segmentation into regions

Can cluster on color values and pixel locations

Can use connected components and an approximate color criteria to find regions

Can train an algorithm to look for certain colored regions – for example, skin color

53

Finding a face in video frame

(left) input video frame(center) pixels classified according to RGB space(right) largest connected component with aspect similar to a face (all work contributed by Vera Bakic)

54

Example

A computer vision-based system that deals with image formation issues and exploits colourinformation

55

Veggie vision: system outline

Task description: the system recognizes fruits and vegetables in a supermarket

Main steps:Imaging the produceForeground segmentationFeature selection (colour, shape, texture, size)Classification and recognitionThe system is strongly task-oriented, thus customized solutions are preferred over generic paradigms of Computer Vision.

56

Veggie Vision publication and patents

VeggieVision: A Produce Recognition System R. M. Bolle, J. Connell, N. Haas, R. Mohan, G. TaubinProc. of the Third IEEE Workshop on Applications of Computer Vision (WACV-96), December 1996.Related patents:US05546475 Veggie Vision Concept US05631976 Veggie Segmentation Box US05649070 Veggie Learning US06005959 Veggie Size Measure US06310964 Veggie Size Measure 2

57

Veggie vision : imaging setup

Upward looking imaging system with a transparent top surface (glass window) integrated with a scale.Quasi-uniform illumination: two circular fluorescent bulbsLinear polarizing filter covering the internal light sources; second polarizer on the camera, orthogonal to the first.

reason: if polarized light is normally incident on a surface- specular reflections will have the same polarization- diffuse reflection will be unpolarized- the orthogonal polarizer eliminates

the specular reflection

58

Veggie vision: imaging the produce

Issue

Specular reflection introduces errors in colour-based segmentation - it may be due to the wrapping bagSolution

- filtering out highlights

59

Parallel configuration of polarizers

Orthogonal configuration of polarizers

60

Veggie vision: figure-ground segmentationScene illumination=ambient illumination + illumination of the system light sourceIssue: ambient illumination is not controlled. Solution: data fusion between two imagesImage 1: lights off. Image 2: lights on.

Foreground detection through image differencing and thresholding

61

Veggie vision: feature selection

Colour

Texture

Shape

Size

Descriptors must be invariant to:

-Translation

-Rotation

-Number of produce items

62

Veggie vision: feature selectionAll descriptors will be extracted from histograms of colour

a histogram = a global, compact representation of an object (see section 2.3.2)Example: the brightness histogram of an image provides information about the frequency of all brightness levels in the imageIf the image has 256 brightness levels, then its brightness histogram is a one dimensional array with 256 elements.Useful function in Matlab: imhist

63

Veggie vision: feature selection –colour histograms

a) apples b) oranges

hue

saturation

intensity

64

Veggie vision media coverage

BBC News Online about the "Future Store" in Rheinberg, Germany, part of the METRO retail group

‘The intelligent scale doesn't like our bunch of bananas. When we had put tomatoes on the scale, its digital camera took just a split second to recognize the produce, weigh it and print a bar-coded price tag. [..] No need to find "tomatoes" on a 50-button display. But now the scale is baffled, and offers four choices: Are we weighing bananas, chicory salad, long beans or avocados? Touching the banana logo on the screen solves the slip-up.’

Story from BBC NEWS:Published: 2004/05/18 © BBC MMV