ComputerVision

Linear filters and edges

Marc PollefeysCOMP 256

ComputerVision

2

Last class

ComputerVision

3R

G

B

Illuminant color

Diffuse component

T

S

HSV

ComputerVision

4

Jan 16/18 - Introduction

Jan 23/25 Cameras Radiometry

Jan 30/Feb1 Sources & Shadows Color

Feb 6/8 Linear filters & edges Texture

Feb 13/15 Multi-View Geometry Stereo

Feb 20/22 Optical flow Project proposals

Feb27/Mar1 Affine SfM Projective SfM

Mar 6/8 Camera Calibration Silhouettes and Photoconsistency

Mar 13/15 Springbreak Springbreak

Mar 20/22 Segmentation Fitting

Mar 27/29 Prob. Segmentation Project Update

Apr 3/5 Tracking Tracking

Apr 10/12 Object Recognition Object Recognition

Apr 17/19 Range data Range data

Apr 24/26 Final project Final project

Tentative class schedule

ComputerVision

5

Project

• Project proposals – Feb 22 4 minute presentation and 1-2 page proposal

• Project update – Mar 294 minute presentation

• Final project presentation – Apr 24/26presentation/demo and short paper-style report

• One or more students per project– need identifiable contribution (modules,

alternative approaches, …)

ComputerVision

6

Project proposals

• Some ideas– Vision-based UI (gesture recognition,…)

– Tracking people with PTZ camera(s)(single camera, multi camera collaborative)

– 3D reconstruction using Shape-from-?– Camera network calibration– Computer vision on GPU– Environment reconstruction with UAVs– …

Come talk to me!

ComputerVision

7

Linear Filters

• General process:– Form new image

whose pixels are a weighted sum of original pixel values, using the same set of weights at each point.

• Properties– Output is a linear

function of the input– Output is a shift-

invariant function of the input (i.e. shift the input image two pixels to the left, the output is shifted two pixels to the left)

• Example: smoothing by averaging– form the average of

pixels in a neighbourhood

• Example: smoothing with a Gaussian– form a weighted

average of pixels in a neighbourhood

• Example: finding a derivative– form a weighted

average of pixels in a neighbourhood

ComputerVision

8

Convolution

• Represent these weights as an image, H

• H is usually called the kernel

• Operation is called convolution– it’s associative

• Result is:

• Notice wierd order of indices– all examples can be put in

this form– it’s a result of the

derivation expressing any shift-invariant linear operator as a convolution.

Rij H i u, j vFuvu,v

ComputerVision

9

c22

+ c22 f(i,j)

f(.)

o (i,j) =

f(.)

c12

+ c12 f(i-1,j)

f(.)

c13

+ c13 f(i-1,j+1) +

f(.)

c21

c21 f(i,j-1)

f(.)

c23

+ c23 f(i,j+1) +

f(.)

c31

c31 f(i+1,j-1)

f(.)

c32

+ c32 f(i+1,j)

f(.)

c33

+ c33 f(i+1,j+1)

f(.)

c11

c11 f(i-1,j-1)

Convolution

ComputerVision

10

Example: Smoothing by Averaging

ComputerVision

11

Smoothing with a Gaussian

• Smoothing with an average actually doesn’t compare at all well with a defocussed lens– Most obvious

difference is that a single point of light viewed in a defocussed lens looks like a fuzzy blob; but the averaging process would give a little square.

• A Gaussian gives a good model of a fuzzy blob

ComputerVision

12

exp x2 y2

22

An Isotropic Gaussian

• The picture shows a smoothing kernel proportional to

(which is a reasonable model of a circularly symmetric fuzzy blob)

ComputerVision

13

Smoothing with a Gaussian

ComputerVision

14

Differentiation and convolution

• Recall

• Now this is linear and shift invariant, so must be the result of a convolution.

• We could approximate this as

(which is obviously a convolution; it’s not a very good way to do things, as we shall see)

fx

lim 0

f x , y

f x,y

fx

f xn1,y f xn , y

x

ComputerVision

15

Finite differences

ComputerVision

16

Noise

• Simplest noise model– independent stationary

additive Gaussian noise

– the noise value at each pixel is given by an independent draw from the same normal probability distribution

• Issues– this model allows noise

values that could be greater than maximum camera output or less than zero

– for small standard deviations, this isn’t too much of a problem - it’s a fairly good model

– independence may not be justified (e.g. damage to lens)

– may not be stationary (e.g. thermal gradients in the ccd)

ComputerVision

17

sigma=1

ComputerVision

18

sigma=16

ComputerVision

19

Finite differences and noise

• Finite difference filters respond strongly to noise– obvious reason: image

noise results in pixels that look very different from their neighbours

• Generally, the larger the noise the stronger the response

• What is to be done?– intuitively, most pixels

in images look quite a lot like their neighbours

– this is true even at an edge; along the edge they’re similar, across the edge they’re not

– suggests that smoothing the image should help, by forcing pixels different to their neighbours (=noise pixels?) to look more like neighbours

ComputerVision

20

Finite differences responding to noise

Increasing noise ->(this is zero mean additive gaussian noise)

ComputerVision

21

The response of a linear filter to noise

• Do only stationary independent additive Gaussian noise with zero mean (non-zero mean is easily dealt with)

• Mean:– output is a weighted

sum of inputs– so we want mean of

a weighted sum of zero mean normal random variables

– must be zero

• Variance:– recall

• variance of a sum of random variables is sum of their variances

• variance of constant times random variable is constant^2 times variance

– then if is noise variance and kernel is K, variance of response is

2 Ku,v2

u,v

ComputerVision

22

Filter responses are correlated

• over scales similar to the scale of the filter

• Filtered noise is sometimes useful– looks like some natural textures, can be

used to simulate fire, etc.

ComputerVision

23

ComputerVision

24

ComputerVision

25

ComputerVision

26

Smoothing reduces noise

• Generally expect pixels to “be like” their neighbours– surfaces turn slowly– relatively few

reflectance changes

• Generally expect noise processes to be independent from pixel to pixel

• Implies that smoothing suppresses noise, for appropriate noise models

• Scale– the parameter in the

symmetric Gaussian– as this parameter goes

up, more pixels are involved in the average

– and the image gets more blurred

– and noise is more effectively suppressed

ComputerVision

27

The effects of smoothing Each row shows smoothingwith gaussians of differentwidth; each column showsdifferent realisations of an image of gaussian noise.

ComputerVision

28

Some other useful filtering techniques

• Median filter• Anisotropic diffusion

ComputerVision

29

Median filters : principle

non-linear filter

method :

1. rank-order neighbourhood intensities 2. take middle value

no new grey levels emerge...

ComputerVision

30

Median filters : odd-man-out

advantage of this type of filter is its “odd-man-out” effect

e.g.

1,1,1,7,1,1,1,1

?,1,1,1.1,1,1,?

ComputerVision

31

Median filters : example

filters have width 5 :

ComputerVision

32

Median filters : analysis

median completely discards the spike,linear filter always responds to all aspects

median filter preserves discontinuities,linear filter produces rounding-off effects

DON’T become all too optimistic

ComputerVision

33

Median filter : images

3 x 3 median filter :

sharpens edges, destroys edge cusps and protrusions

ComputerVision

34

Median filters : Gauss revisitedComparison with Gaussian :

e.g. upper lip smoother, eye better preserved

ComputerVision

35

Example of median

10 times 3 X 3 median

patchy effectimportant details lost (e.g. ear-ring)

ComputerVision

36

Gradients and edges

• Points of sharp change in an image are interesting:– change in

reflectance– change in object– change in

illumination– noise

• Sometimes called edge points

• General strategy– determine image

gradient

– now mark points where gradient magnitude is particularly large wrt neighbours (ideally, curves of such points).

ComputerVision

37

There are three major issues: 1) The gradient magnitude at different scales is different; which should we choose? 2) The gradient magnitude is large along thick trail; how do we identify the significant points? 3) How do we link the relevant points up into curves?

ComputerVision

38

Smoothing and Differentiation

• Issue: noise– smooth before differentiation– two convolutions to smooth, then

differentiate?– actually, no - we can use a derivative of

Gaussian filter• because differentiation is convolution, and

convolution is associative

ComputerVision

39

The scale of the smoothing filter affects derivative estimates, and alsothe semantics of the edges recovered.

1 pixel 3 pixels 7 pixels

ComputerVision

40

We wish to mark points along the curve where the magnitude is biggest.We can do this by looking for a maximum along a slice normal to the curve(non-maximum suppression). These points should form a curve. There arethen two algorithmic issues: at which point is the maximum, and where is thenext one?

ComputerVision

41

Non-maximumsuppression

At q, we have a maximum if the value is larger than those at both p and at r. Interpolate to get these values.

ComputerVision

42

Predictingthe nextedge point

Assume the marked point is an edge point. Then we construct the tangent to the edge curve (which is normal to the gradient at that point) and use this to predict the next points (here either r or s).

ComputerVision

43

Remaining issues

• Check that maximum value of gradient value is sufficiently large– drop-outs? use hysteresis

• use a high threshold to start edge curves and a low threshold to continue them.

ComputerVision

44

Notice

• Something nasty is happening at corners• Scale affects contrast• Edges aren’t bounding contours

ComputerVision

45

ComputerVision

46

fine scalehigh threshold

ComputerVision

47

coarse scale,high threshold

ComputerVision

48

coarsescalelowthreshold

ComputerVision

49

Filters are templates

• Applying a filter at some point can be seen as taking a dot-product between the image and some vector

• Filtering the image is a set of dot products

• Insight – filters look like the

effects they are intended to find

– filters find effects they look like

ComputerVision

50

Normalized correlation

• Think of filters of a dot product– now measure the

angle– i.e normalised

correlation output is filter output, divided by root sum of squares of values over which filter lies

• Tricks:– ensure that filter has a

zero response to a constant region (helps reduce response to irrelevant background)

– subtract image average when computing the normalizing constant (i.e. subtract the image mean in the neighbourhood)

– absolute value deals with contrast reversal

ComputerVision

51

Positive responses

Zero mean image, -1:1 scale Zero mean image, -max:max scale

ComputerVision

52

Positive responses

Zero mean image, -1:1 scale Zero mean image, -max:max scale

ComputerVision

53

Figure from “Computer Vision for Interactive Computer Graphics,” W.Freeman et al, IEEE Computer Graphics and Applications, 1998 copyright 1998, IEEE

ComputerVision

54

The Laplacian of Gaussian

• Another way to detect an extremal first derivative is to look for a zero second derivative

• Appropriate 2D analogy is rotation invariant– the Laplacian

• Bad idea to apply a Laplacian without smoothing– smooth with

Gaussian, apply Laplacian

– this is the same as filtering with a Laplacian of Gaussian filter

• Now mark the zero points where there is a sufficiently large derivative, and enough contrast

ComputerVision

55

sigma=2

sigma=4

contrast=1 contrast=4LOG zero crossings

ComputerVision

56

We still have unfortunate behaviourat corners

ComputerVision

57

Orientation representations

• The gradient magnitude is affected by illumination changes– but it’s direction

isn’t

• We can describe image patches by the swing of the gradient orientation

• Important types:– constant window

• small gradient mags

– edge window• few large gradient

mags in one direction

– flow window• many large

gradient mags in one direction

– corner window• large gradient

mags that swing

ComputerVision

58

Representing Windows

• Types– constant

• small eigenvalues

– Edge • one medium, one

small

– Flow• one large, one

small

– corner• two large

eigenvalues

H I I Twindow

ComputerVision

59

ComputerVision

60

ComputerVision

61

ComputerVision

62

ComputerVision

63

Next class:Pyramids and Texture

F&PChapter 9