Image Analysis Manipulate an image to extract information to help solve a problem. –Preprocessing...

Image AnalysisImage Analysis• Manipulate an image to extract information

to help solve a problem.– Preprocessing - get rid of unnecessary

information– Data reduction - transform the image to a

useable form– Feature analysis - make inferences about the

image.

Image AnalysisImage Analysis

• Preprocessing– Noise reduction– Gray level quantization – Spatial quantization– Finding regions of interest–

Reducing the number of bits

• Data reduction process, in which the image(s) are transformed into a more convenient form.– RGB HSL– Image subtraction– Histogram– Feature extraction


• Feature analysis - Specific results– Blood cell counting– Tumor size and location– 3D model–


• Geometric– Resizing– Rotating

• Noise Reduction– Image Smoothing– Median Filtering

• Edge Detection– Searching for discontinuities

• Histogram slicing• Blob detection


Geometric TransformationGeometric Transformation

InterpolationInterpolation

• Interpolation to determine image values at integer pixel locations of transformed image. – Numerous interpolation schemes exist

• 2 simple methods are

– Nearest neighbor: assign value of nearest known point

– Bilinear interpolation: example on next slide

Interpolation used to simplify data for further processing

Nearest Neighbor InterpolationNearest Neighbor Interpolation

Interpolate this point To

this value

Linear InterpolationLinear Interpolation

Bi-linear InterpolationBi-linear Interpolation

I(x+1,y)

I(x+1,y+1)

I(x,y+1)

I(x,y)

0 (a,b) 1

I(x’,y) = (1-a)I(x,y) + (a)I(x+1,y)

I(x’,y’) = (1-b)I(x’,y) + (b)I(x’,y+1)

I(x’,y+1) = (1-a)I(x,y+1) + (a)I(x+1,y+1)

I(x’,y’) = (1-b)(1-a)I(x,y) + (1-b)(a)I(x+1,y) + (b)(1-a)I(x,y+1) + (a)(b)I(x,y)

I(x’,y’)

I(x’,y’)

Bi-linear InterpolationBi-linear Interpolation

Neighborhood Neighborhood OperationsOperations

ObjectivesObjectives• Why are neighborhoods important?• What is linear convolution?

– discrete– templates, masks or filters– algorithm mechanics– graphical interpretation

• Describe non-linear operators– maximum– minimum– median

• What is tiling?

Why are neighbourhoods important?Why are neighbourhoods important?

pixel

Because…Because…

• Provide context for individual pixels.

• Relationships between neighbors determine image features.

• Neighborhood operations:– noise reduction– edge enhancement– zooming

Noise reduction Edge Enhancement

Zooming

Neighbourhood OperationsNeighbourhood Operations

• Linear convolution (*)– A*B*C*D = B*C*D*A = ….

• Non-linear operators– median, max, min, ...

Convolution versus Spectral• We learnt two methods of processing images:

– Convolution– Spectral

• We analyzed and demonstrated how to build a processor (systolic, pipelined, parallel, cellular automaton) for 1D convolution.– 1D convolution is used in speech processing and in polynomial

multiplication.

• We will use visualized animations now to show in more detail how 2D convolution works for images.

• This should convince you how important it is to do convolution quickly in modern Spectral Architectures, especially for 3D etc.

2D Convolution2D Convolution

• Consists of filtering an image A using a filter (mask, template) B.

• Mask is a small image whose pixel values are called weights.

• Weights modify relationships between pixels.

We will show more examples of convolution now, especially for 2D data

A1,1 A1,2 A1,3 A1,4

A2,1 A2,2 A2,3 A2,4

A3,1 A3,2 A3,3 A3,4

A4,1 A4,2 A4,3 A4,4

B1,1 B1,2

B2,1 B2,2

BA C

C1,1 C1,2 C1,3

C3,1

C2,1 C2,2 C2,3

C3,2 C3,3

=

Filter,mask or template

Input image

ConvolvedImage

2 2

4 43 3

A1,1 A1,2

A2,1 A2,2

A1,3 A1,4

A2,3 A2,4

A3,1 A3,2 A3,3 A3,4

A4,1 A4,2 A4,3 A4,4

B1,1 B1,2

B2,1 B2,2 A1,1B1,1

A2,1B2,1

A1,2B1,

2

A2,2B2,

2

A1,1B1,1C1,1= A1,2B1,

2

A2,1B2,1 A2,2B2,

2

A1,1 A1,2

A2,1 A2,2

A1,3 A1,4

A2,3 A2,4

A3,1 A3,2 A3,3 A3,4

A4,1 A4,2 A4,3 A4,4

B1,1 B1,2

B2,1 B2,2A1,2B1,

1

A2,2B2,1

A1,3B1,

2

A2,3B2,

2

A1,2B1,1C1,2= A1,3B1,

2

A2,2B2,1 A2,3B2,

2

A1,1 A1,2

A2,1 A2,2

A1,3 A1,4

A2,3 A2,4

A3,1 A3,2 A3,3 A3,4

A4,1 A4,2 A4,3 A4,4

B1,1 B1,2

B2,1 B2,2A1,3B1,

1

A2,3B2,1

A1,4B1,

2

A2,4B2,

2

A1,3B1,1C1,3= A1,4B1,

2

A2,3B2,1 A2,4B2,

2

A1,1 A1,2

A2,1 A2,2

A1,3 A1,4

A2,3 A2,4

A3,1 A3,2 A3,3 A3,4

A4,1 A4,2 A4,3 A4,4

B1,1 B1,2

B2,1 B2,2

A2,1B1,

1

A3,1B2,1

A2,2B1,

2

A3,2B2,

2

A2,1B1,1C2,1= A2,2B1,

2

A3,1B2,1 A3,2B2,

2

A1,1 A1,2

A2,1 A2,2

A1,3 A1,4

A2,3 A2,4

A3,1 A3,2 A3,3 A3,4

A4,1 A4,2 A4,3 A4,4

B1,1 B1,2

B2,1 B2,2B1,1 B1,2

B2,1 B2,2B1,1 B1,2

B2,1 B2,2

Mathematical NotationMathematical Notation

1 1

1,1,,

Mk

ki

Nl

ljljkijilk BAC

NMB

A1,1B1,1C1,1= A1,2B1,

2

A2,1B2,1 A2,2B2,

2

ConvolutionConvolution

4 4 7 9

4 3 8 9

3 5 9 9

3 6 10 9

-1 2

-1 2

BA C

=

Filter,mask or template

Input image

ConvolvedImage

2 2

4 43 3

6

9

16

23

26

27

21

19

17

Convolution sizeConvolution sizeImage size = M1 N1

Mask size = M2 N2

Convolution size = M1- M2 +1 N1-N2+1

N1

N2

N1-N2+1

Typical Mask sizes= 33, 5 5, 77,9 9, 1111

What is the convolved image size for a 128 128 image and 7 7 mask?

*1 1 11 1 1 1 1 1

1 1 11 1 1 1 1 1

1 1 11 1 1 1 1 1

1 1 11 1 1 1 1 1

1 1 11 1 1 1 1 1

1 1 11 1 1 1 1 1

1 1 11 1 1 1 1 1

1 1 11 1 1 1 1 1

1 1 11 1 1 1 1 1

=

We convolve with 9*9 averaging filter

Nonlinear Neighbourhood Nonlinear Neighbourhood OperationsOperations

• Maximum

• Minimum

• Median

1,1,

1:,1:,

1,1,1:,1:

,

1,1,1:,1:

,

median

min

max

ljkijiNlljMkki

lk

ljkijiNlljMkki

lk

ljkijiNlljMkki

lk

BAC

BAC

BAC

We discussed already sorter architecture (three variants – pipelined, butterfly combinational and sequential controller). It can be used for all these operations, and also for other non-linear operators

61 62

57 60

59 65

63 56

59 55 58 57

49 53 55 45

1 1

1 1C1,2=

62

60

59

636359

Max and Min OperationsMax and Min Operations

63=max, 59=min

61 62

57 60

59 65

63 56

59 55 58 57

49 53 55 45

C1,2=

Median OperationMedian Operation

1 1

1 1

1

1

1 1 1

62

60

59

63

65

56

55 58 57

62

59

65

60

63

56

57

58

55

987654321

59

rank

9x9 Median

Edge Detection

• What do we mean by edge detection?

• What is an edge?

What is Edge Detection?What is Edge Detection?

• Detects large intensity transitions between pixels

• Redraws the image with only the edges showing

0 0 0 33

0 0 45 78

0 45 23 33

0 0 42 76

0 0 0 38

What is an Edge?

Edge easy to findEdge easy to find

What is an Edge?

Where is edge? Single pixel wide or multiple pixels?Where is edge? Single pixel wide or multiple pixels?

What is an Edge?

Noise: have to distinguish noise from actual edgeNoise: have to distinguish noise from actual edge

Noise Noise is hereis here

What is an Edge?

Is this one edge or two?Is this one edge or two?

What is an Edge?

Texture discontinuityTexture discontinuity

Edge Detection – so what is an Edge Detection – so what is an edge to be detected?edge to be detected?

• What is an edge– A large change in image brightness of a short

spatial distance – Edge strength = (I(x,y)-I(x+dx,y))/dx

But this general definition still allows for many theories, software implementation and hardware architectures.

Now we will Now we will discuss and discuss and illustrate various illustrate various kinds of filter kinds of filter operatorsoperators

Edge Detection FiltersEdge Detection Filters

•High - Pass Filtering Eliminates Uniform Regions (Low Frequencies)

•edge “detection” or “enhancement”

Edge Detection ContinuedEdge Detection Continued

•Sum of Kernel Coefficients = 0

•differences in signs emphasize differences in pixel values

•reduces average image intensity

•Negative pixel values in output?


vertical

horizontal diagonal

Edge DirectionEdge Direction

Directional High Directional High Pass FiltersPass Filters

Convolution Edge Convolution Edge Detection using Detection using

Sobel and similar Sobel and similar operatorsoperators

Example of Sobel Operator

Sobel OperatorSobel Operator

Sobel Edge DetectionSobel Edge Detection

Convolution Application ExamplesConvolution Application Examples

--Edge DetectionWe apply Sobel Operator

-1 -2 -10 0 01 2 1

-1 0 1

-2 0 2

-1 0 1

Column Mask

Row Mask

as mask to a sub-field of a picture

-1 2 -1

0 0 0

1 2 1

p0, p1, p2

p3, p4, p5

p6, p7, p8 = (p6-p0)+2(p7-p1)+(p8-p2)

We can learn from the result obviously •The result of the above calculation for column mask is horizontal difference•With Row Mask we will get vertical difference

*

The final step of the convolution equation, dividing by the weight , must be ignored

--Edge Detection with Sobel Operator

The weight of a mask determines the grey level of the image after convolution.

Like the weight of Sobel Mask WW= (-1) + (-2) + (-1) + 0 + 0 + 0 +1 + 2 +1= 0

The resulting image lost its “lightness” to be dark.dark.

Convolution Application ExamplesConvolution Application Examples


-1 -2 -1 0 0 0 1 2 1

-1 0 1-2 0 2 -1 0 1

S1= S2 =

Edge Magnitude =

Edge Direction =

22

21 SS

2

11tanS

S

SobelPrewitt

Ticbetts

Canny

Comparison of Edge Detection Comparison of Edge Detection AlgorithmsAlgorithms

Edge DirectionEdge DirectionAssymetric kernels detect edges from Assymetric kernels detect edges from specific directionsspecific directions

EastEast

1 1 -1

1 -2 -1

1 1 -1

NorthEaNorthEastst

1 -1 -1

1 -2 -1

1 1 -1

NorthNorth

-1 -1 -1

1 -2 1

1 1 1

Robinson Robinson OperatorOperator

Robinson Compass MasksRobinson Compass Masks

-1 0 1-2 0 2 -1 0 1

0 1 2-1 0 1 -2 -1 0

1 2 1 0 0 0 -1 -2 -1

2 1 0 1 0 -1 0 -1 -2

1 0 -1 2 0 -2 1 1 -1

0 -1 -2-1 0 -1 2 1 0

-1 -2 -1 0 0 0 1 2 1

-2 -1 0-1 0 1 0 1 2

Arrows show edge directions

Roberts Roberts OperatorOperator

Roberts OperatorRoberts Operator

• Does not return any information about the orientation of the edge

22 ),1()1,()1,1(),( yxIyxIyxIyxI

),1()1,()1,1(),( yxIyxIyxIyxI

or

1 00 -1

0 1-1 0+

Prewitt OperatorPrewitt Operator-1 -1 -1 0 0 0 1 1 1

-1 0 1-1 0 1 -1 0 1

P1= P2 =

Edge Magnitude =

Edge Direction =

22

21 PP

2

11tanP

P

Prewitt Row


0 1 2-1 0 1 -2 -1 0

1 2 1 0 0 0 -1 -2 -1Original

and filtered cow

Edge Detection Edge Detection (continued)(continued)

• First Order (Gradient) Kernels

• Prewitt RowPrewitt Row

1 0 -11 0 -1

1 0 -11 0 -1

1 0 -11 0 -1

• Sobel RowSobel Row

1 0 -11 0 -1

2 0 -22 0 -2

1 0 -11 0 -1

• Combine Row and Column Operators

Edge Detection Filters: compare Edge Detection Filters: compare Prewitt Prewitt and Sobeland Sobel

1D Laplacian Operator

)(xf

x

xf

)(

2

22 )(

x

xfxf

first derivative

second derivative

2D Laplacian Operator2D Laplacian Operator

2

2

2

22 ,,

),(y

yxf

x

yxfyxf

0 -1 0-1 4 -1 0 -1 0

1 -2 1-2 4 -2 1 -2 1

-1 -1 -1-1 8 -1-1 -1 -1

Convolution masks approximating a Laplacian

This is just one example of Laplacian, we can use much larger window

0 -1 0-1 4 -1 0 -1 0

Input Mask Output

Image Processing Operations for Early Vision:

Edge Edge DetectionDetection

Reminder: Effect of FiltersReminder: Effect of Filters

low

high

EdgesEdges… are the important part of images

intensity

color

edges

textures

contours

condensation...

simplest, least robust

most difficult, most robust There are many letters B occluded

by black shape here. How to find them?

Edge Detection

• Edges are curves in the image plane across which there is a “significant” change in image brightness.

• The goal of edge detection is the construction of an idealized line drawing

Image Processing OperationsImage Processing Operations

Pixels on edgesPixels on edges

Edges foundEdges found

Edge effects: rarely ideal edgesEdge effects: rarely ideal edges

Not all information is created equal...

Causes of edgesCauses of edges

• Depth discontinuity– One surface occludes another

• Surface orientation discontinuity– the edge of a block

• reflectance discontinuity– texture or color changes

• illumination discontinuity– shadows

Edges: causesEdges: causesWhat are they? Why?

four physical events that cause image edges...

What are they? Why?

discontinuities in

• surface color/intensity• surface normal• depth• lighting (specularities)

four physical events that cause image edges...

Edges: causesEdges: causes

Edges are image locations with a local maximum in image gradient in the direction of that gradient

(steepness)

Edges: causesEdges: causes

Edge DetectionEdge Detection

• Finding simple descriptions of objects in complex images– find edges

– interrelate edges

Examples of edgesExamples of edges

•One idea to detect edges is to differentiate the image and look for places where the brightness undergoes a sharp change•Consider a 1-D example. Below is an intensity profile for a 1-D image.

Edges are not ideal... Edges are not ideal...

•Below we have the derivative of the previous graph.

•Here we have a peak at x=18, x=50 and x=75.•These errors are due to the presence of noise in the image.


Finding EdgesFinding EdgesImage Intensity along a line

First derivative of intensity

Smoothed via convolving with gaussian

• This problem is countered by convolving a smoothing function along with the differentiation operation.

• The mathematical concept of convolution allows us to perform many useful image-processing operations.


• One standard form of smoothing is to use a Gaussian function.

• Now using the idea of convolving with the Gaussian functionwe can revisit the 1-D example.

Image Processing OperationsImage Processing Operations

• With the convolving applied we can more easily see the edge at x=50.

Using convolving we are able to discover where edges are located and this allows us to make an accurate line drawing.

Convolving to find edgesConvolving to find edges

•Here is an example of using convolving in an 2-D picture of Mona Lisa

Edge Detection by ConvolutionEdge Detection by Convolution

edge

Second derivative

derivative

Zero CrossingZero Crossing

EdgeEdge

Edge ParametersEdge Parameters

Remainder: How the Point Remainder: How the Point Detection Mask operates on one Detection Mask operates on one

color imagecolor image

Goal: to find regions of an image with locally maximal gradient magnitude.(1) Smooth the image to reduce the effects of noise

Replace each pixel by a weighted sum of its neighbors.

I(x,y) = wij I(x+i,y+j) i,j = -1

1

old Imageweight “mask” new

Image

oldnewweights

4 22

2 11

2 11

x =

(scaled)

34 37 137 148

29 46 141 140

34 130 149 131

41 142 152 144

35 210 210 210

43 210 210 210

60 210 210 210

72 210 210 210

Edge Detection uses ConvolutionEdge Detection uses Convolution


It’s possible to do this one dimension at a time...



1

old Imageweight “mask” new

Image

oldnewweights

4 22

2 11

2 11

x =

(scaled)

34 37 137 148

29 46 141 140

34 130 149 131

41 142 152 144

35 62 116 143

43 76 122 141

60 102 136 140

72 112 145 143

smoothed vertically

smoothed horizontally

original image



(2) Estimate the gradient at each pixel


1

oldnewweights


Use another mask of weights -- this time to approximate taking derivatives.

In the x direction

In the y direction

-1 1

-1

1

with new weights

same procedure

(convolution)

= Iy= Ixddx

I ddy

I




(3) Find the gradient magnitude and thin the resulting lines.


1

oldnewweights



-1 1-1

1

same procedure with new weights (convolution)

x

y

I(x,y) = sqrt(Ix + Iy ) 22

Seek out maxima in the gradient direction .

= atan2(Ix, Iy)




(3) Find the gradient magnitude and thin the resulting information.

(4) Choose a threshold -- any gradients above it classify as edges.


1

oldnewweights



I(x,y) = sqrt(Ix + Iy ) 22

-1 1-1

1

same procedure with new weights (convolution)

x

y

In the gradient direction, look for local maxima: = atan2(Ix, Iy)


Theory of Theory of Gradient Gradient

Based Edge Based Edge DetectionDetection

Formal Model of Edge Formal Model of Edge

Formal Model of Edge (cont)

Formal Model of Edge Formal Model of Edge

Formal Model of Edge (cont) Formal Model of Edge Formal Model of Edge

Formal Model of Edge (cont) Formal Model of Edge: Roberts Formal Model of Edge: Roberts

Formal Model of Edge (cont) Formal Model of Edge: Laplacian and Marr-Formal Model of Edge: Laplacian and Marr-Hildreth Hildreth

Formal Model of Edge (cont) Formal Model of Edge Formal Model of Edge

ThresholdsThresholds

very high thresholdoriginal image

Thresholds are important, done before or during edge detection.

Thresholds



very high threshold

reasonable

original image


very high threshold

too low !reasonablethis all takes time...

original image


Image Analysis Manipulate an image to extract information to help solve a problem. –Preprocessing...

Documents

Transcript of Image Analysis Manipulate an image to extract information to help solve a problem. –Preprocessing...