Morphological Operations

1

Outline

• Basic concepts:• Erode and dilate• Open and close.

G l t• Granulometry• Hit and miss transform• Thinning and thickening• Skeletonization and the medial axis transformSkeletonization and the medial axis transform• Introduction to gray level morphology.

2

Pixel connectivity

• We need to define which pixels are neighbors. Warning:

i l lg

• Are the dark pi els in this arra

Pixels are samples, not squarespixels in this array

connected? not squares.

4

Pixel connectivity

• We need to define which pixels are neighbors.g

• Are the dark pixels in this arraypixels in this array connected?

5

Pixel Neighborhoods

4-neighborhood 8-neighborhood

6

Pixel paths

• A 4-connected path between pixels p1 and pn is a set of pixels {p1, p2, …, pn} such that pi is a 4-neighbor of pi+1, i=1,…,n-1.pi g pi+1, , ,

d h i i hb• In an 8-connected path, pi is an 8-neighbor of pi+1.i 1

7

Connected regions

• A region is 4-connected if it contains a 4-connected path between any two of its pixels.p

i i d if i i• A region is 8-connected if it contains an 8-connected path between any two of its pixels.

8

Connected regions

• Now what can we say about the dark pixels in this parray?

• What about the light pixels?

9

Connected componentsConnected components labellinglabelling

• Labels each connected component of a binary image with a separate number.

1 1 1 1 1 1 1 1 11 1 1 1 1 1 1 1 11 1 2 2 1 3 1 1 11 1 1 2 2 1 1 1 11 1 1 1 1 1 1 1 11 4 1 1 5 5 1 1 11 1 1 1 5 5 5 1 16 6 1 1 1 1 5 5 17 6 1 1 8 8 1 1 17 6 1 1 8 8 1 1 1

10

Foreground labelling

• Only extract the connected components of the foreground

1 1 1 1 1 1 1 1 11 1 1 1 1 1 1 1 11 1 0 0 1 0 1 1 11 1 1 0 0 1 1 1 11 1 1 1 1 1 1 1 11 0 1 1 0 0 1 1 11 1 1 1 0 0 0 1 10 0 1 1 1 1 0 0 12 0 1 1 0 0 1 1 12 0 1 1 0 0 1 1 1

11

What Are MorphologicalWhat Are Morphological Operators?Operators?

• Local pixel transformations for processing region shapes

• Most often used on binary imagesMost often used on binary images

• Logical transformations based on comparison of pixel neighbourhoods with acomparison of pixel neighbourhoods with a pattern.

18

Simple Operations - Examples

• Eight-neighbour erode– a.k.a. Minkowski subtraction

• Erase any foreground pixel that has one eight-connected neighbour that is background. g

19

8-neighbour erode

Threshold

Erode ×1 Erode ×2 Erode ×5

20

8-neighbour dilate

• Eight-neighbour dilate• a.k.a. Minkowski addition

• Paint any background pixel that has one eight-connected neighbour that is foreground.

21

8-neighbour dilate

Dilate ×1 Dilate ×2 Dilate ×5

22

Why?

• Smooth region boundaries for shape analysis.

• Remove noise and artefacts from anRemove noise and artefacts from an imperfect segmentation.

• Match particular pixel configurations in anMatch particular pixel configurations in an image for simple object recognition.

23

Structuring Elements

• Morphological operations take two arguments:

• A binary imagey g• A structuring element.

• Compare the structuring element to theCompare the structuring element to the neighbourhood of each pixel.

• This determines the output of the morphological operation.

24

p g p

Structuring elements

• The structuring element is also a binary array.

• A structuring element has an originA structuring element has an origin.

1 1 11 1 1

0 1 01 1 1

0 1 01 0 1

11

1 1 1 0 1 0 0 1 0 0

25

Binary images as sets

• We can think of the binary image and the structuring element as sets containing the pixels with value 1.p

1 1 1 0 0

0 1 1

0 0 0

0

0

0

0

0 0 0 1 0

I = {(1,1), (2,1), (3,1), (2,2), (3,2), (4,4)}

0 0 0 1 0

0 0 0 0 0 (Matlab)S = SPARSE(X) converts a sparse or

26full matrix to sparse form bysqueezing out any zero elements

Some sets notation

• Union and i i

• Differenceintersection:

{ }2121 or : IxIxxII ∈∈=∪{ }2121 and:\ IxIxxII ∉∈=

• We use φ for the

{ }2121 and : IxIxxII ∈∈=∩

• Complement• We use φ for the

empty set .{ }IxxI C ∉= :

27

More sets notation

• The symmetrical set of S with respect to point o is

{ }SS :(

0 1 1

1 1 1

1

1

0

1{ }SxxoS ∈−= :

1 1 1 1

1 1 1 1 1

1 1 1 1 1

0 1 1 1 0

0 1 1 1 01 1 1

0 1 1

0 0 0

0 0 0

1 1 0

1 1 1

0 1 1

1 1 1

1 1 1

1

1

1

0

1

1

28

0 0 0 1 1 11 1 1 1 1

0 1 1 1 0

Fitting, Hitting and Missing

• S fits I at x if

• S hits I at x ifISssxyy ⊂∈+= },:{

• S hits I at x ifφ≠∩∈+= ISssxyy },:{

• S misses I at x ifφ=∩∈+= ISssxyy },:{ φyy },{

29

Fitting, Hitting and Missing

0 1 0 1 1 1 1 1

Image Structuring element

1 0 10 1 1

11

10

10

10

10

0 1 01 1 10 1 1 1 0 0 0 0

0 1 1 1 1 0 0 00 1 1 1 0 0 0 0 0 1 0

0 1 0 0 0 0 1 00 1 0 1 0 1 0 00 1 1 1 1 0 0 0

0 1 0 0 0 0 1 00 1 0 0 0 0 0 0

30

Erosion

• The image E = I S is the erosion of image I by structuring element S.

⎩⎨⎧

=otherwise 0

at fits if 1)(

xISxE

}every for :{ SsIsxyyE ∈∈+== }y{ yy

31

Implementation (naïve)Implementation (naïve)% I is the input image, S is a structuring element

% with origin (ox, oy). E is the output image.

function E = erode(I, S, ox, oy)

[X,Y] = size(I);

[SX, SY] = size(S);

E = ones(X, Y);

for x=1:X; for y=1:Y

f i 1 f j 1for i=1:SX; for j=1:SY

if(S(i,j))

E( ) E( ) & ImageEntr (I +i +j )E(x,y) = E(x,y) & ImageEntry(I, x+i-ox, y+j-oy);

end

end; end

32

end; end

end; end

Example

1 1 1 1 1

Structuring element

1 1 1

1 1 1

1

1

1

1

1 1 1 1 1

33

1 1 1 1 1

1 1 1 1 1

Example

0 1 1 1 0

Structuring element

1 1 1

1 1 1

1

1

1

1

1 1 1 1 1

34

1 1 1 1 1

0 1 1 1 0

Example

1 0 0 0 0

Structuring element

0 1 0

0 0 1

0

0

0

0

0 0 0 1 0

35

0 0 0 1 0

0 0 0 0 1

Dilation

• The image D = I ⊕ S is the dilation of image I by structuring element S.

⎩⎨⎧

=th i0

at hits if 1)(

yISxD

⎩ otherwise 0

}anyfor:{ SsIsxyyD ∈∈+== }any for :{ SsIsxyyD ∈∈+==

36

Alternative Implementation

• For another implementation of erosion and dilation, note that:

( ))(min)( sxIxE += ( )( ))(max)(

)(min)(

sxIxD

sxIxESs

+=

+=∈

( ))(max)( sxIxDSs

+∈

37

Example

1 1 1 1 1

Structuring element

1 1 1

1 1 1

1

1

1

1

1 1 1 1 1

38

1 1 1 1 1

1 1 1 1 1

Example

0 1 1 1 0

Structuring element

1 1 1

1 1 1

1

1

1

1

1 1 1 1 1

39

1 1 1 1 1

0 1 1 1 0

Example

1 0 0 0 0

Structuring element

0 1 0

0 0 1

0

0

0

0

0 0 0 1 0

40

0 0 0 1 0

0 0 0 0 1

Erosion and dilation

• Erosion and dilation are dual operations:SISI CC (

⊕=) (

• Commutativity and associativity

ISSIISSI

≠⊕=⊕

)()()()(

TSITSITSITSI⊕=⊕⊕=⊕⊕

ISSI ≠ )( ) ( TSITSI ⊕=

41

42

Input OutputInput Output

Structuring Element:

43

Opening and Closing

• The opening of I by S isSSISI ⊕= ) (o

• The closing of I by S isSSISI )( ⊕=•

44

Opening and Closing

• Opening and closing are dual transformations:

SISI CC (o=• )(

• Opening and closing are idempotenti

SISI o=• )(

operations:SSISI = )( ooo

SSISI ••=• )()(

45

ExampleExample

close

1 1 1 1 1

Structuring element

1 1 1

1 1 1

1

1

1

1

1 1 1 1 1

46

1 1 1 1 1

1 1 1 1 1open

ExampleExample

close

0 1 1 1 0

Structuring element

1 1 1

1 1 1

1

1

1

1

1 1 1 1 1

47

1 1 1 1 1

0 1 1 1 0open

ExampleExample

close

1 0 0 0 0

Structuring element

0 1 0

0 0 1

0

0

0

0

0 0 0 1 0

48

0 0 0 1 0

0 0 0 0 1open

Morphological filtering

• To remove holes in the foreground and islands in the background do both opening and closingthe background, do both opening and closing.

Th i d h f h i l• The size and shape of the structuring element determine which features survive.

• In the absence of knowledge about the shape of f i l ifeatures to remove, use a circular structuring element.

49

ExampleExample

Close then open

1 1 1 1 1

Structuring element

1 1 1

1 1 1

1

1

1

1

1 1 1 1 1

50

1 1 1 1 1

1 1 1 1 1Open then close

ExampleExample

Close then open

0 1 1 1 0

Structuring element

1 1 1

1 1 1

1

1

1

1

1 1 1 1 1

51

1 1 1 1 1

0 1 1 1 0Open then close

ExampleExample

Close then open

1 0 0 0 0

Structuring element

0 1 0

0 0 1

0

0

0

0

0 0 0 1 0

52

0 0 0 1 0

0 0 0 0 1Open then close

Count the Red Blood Cells

53

Granulometry

• Provides a size distribution of distinct regions or “granules” in the image.

• We open the image with increasingWe open the image with increasing structuring element size and count the number of regions after each operationnumber of regions after each operation.

• Creates a “morphological sieve”.

54

Granulometryfunction gSpec = granulo(I, T, maxRad)

% Segment the image I

B = (I>T);

% Open the image at each structuring element size up to a

% i i i i% maximum and count the remaining regions.

for x=1:maxRad

O i (B t l(‘di k’ ))O = imopen(B,strel(‘disk’,x));

numRegions(x) = max(max(connectedComponents(O)));

endend

gSpec = diff(numRegions);

55

Count the Red Blood Cells

56

Threshold and Label

57

Disc(11)

58

Disc(19)

59

Disc(59)

60

Number of Regionss s

freg

ions

n ra

dius

mbe

r of

ller t

han

Num

sma

61Struct. Elt. radius

GranulometricGranulometric Pattern SpectrumPattern Spectrum

sfr

egio

nsm

ber o

f N

um

62Struct. Elt. radius

Hit-and-miss transform

• Searches for an exact match of the structuring element.

• H = I ⊗ S is the hit-and-miss transform of i b i limage I by structuring element S.

• Simple form of template matching.

63

Hit-and-miss transform1 0 1

1 1 0

1

1

1

0

0 1 0

0 0 1

0

0

0

0

1 0 0 1 1

1 1 0 1 1

1 0 1 0 1

1 0 1⊗ = 0 0 0 0 0

0 0 1 0 0

0 1 0 1 01 0 1 0 1 0 1 0 1 0

1 0 1 1 1 0 1 0 0 0

1 1 0

1 0 0

1

1

0

1* 1

⊗ =0 0 0

0 1 0

0

0

1

01 0

1 1 0 1 1

1 0 1 0 1

10 0 1 0 0

0 0 0 0 0

64

Upper-Right Corner Detector

65bwhitmiss(BW1,SE1,SE2) ==

imerode(BW,SE1) & imerode(~BW,SE2)

Thinning and Thickening

• Defined in terms of the hit-and-miss transform:

• The thinning of I by S isThe thinning of I by S is)(\ SIISI ⊗=

• The thickening of I by S is)( SIISI ⊗∪=

• Dual operations:)( SIISI ⊗∪=

SISI CC ) ( =

66{ }2121 and :\ IxIxxII ∉∈=(Note) Difference:

SequentialSequential Thinning/ThickeningThinning/Thickening

• These operations are often performed in sequence with a selection of structuring elements S1, S2, …, Sn.1, 2, , n

• Sequential thinning:{ } )))(((1 SSSIiSI

• Sequential thickening:

{ } ) )... ) (((,...,1: 21 ni SSSIniSI ==

Sequential thickening:{ } ) )... ) (((,...,1: 21 ni SSSIniSI ==

67

SequentialSequential Thinning/ThickeningThinning/Thickening

• Several sequences of structuring elements are useful in practice

• These are usually the set of rotations of aThese are usually the set of rotations of a single structuring element.

• Sometimes called the Golay alphabet.Sometimes called the Golay alphabet.

68

Golay element L

* 1 *1 * 0* 0 00 0 0 * 1 *1 1 0* 0 0

L4

1 * 01 1 01 * 0

L3

* 0 01 1 0* 1 *

L2

0 0 0* 1 *1 1 1

L1

* 0 01 * 0* 1 *1 1 1

0 0 *0 1 1L

0 * 10 1 1L

* 1 *0 1 1L

1 1 1* 1 *L 0 1 1

* 1 *L80 1 1

0 * 1L70 1 1

0 0 *L6* 1 *

0 0 0L5

69

Sequential Thinning

• See bwmorph in matlab.

0 iterations 1 iteration 2 iterations 5 iterations Inf iterations

70

S ti l Thi k iSequential Thickening

1 iteration 5 iterations

2 iterations Inf iterations

71

iterations

Skeletonization and the MedialSkeletonization and the Medial Axis TransformAxis Transform

• The skeleton and medial axis transform(MAT) ti k fi t ti f(MAT) are stick-figure representations of a region X ⊂ ℜ2.

• Start a grassfire at the boundary of the region.Start a grassfire at the boundary of the region.

Th k l i h f i hi h• The skeleton is the set of points at which two fire fronts meet.

72

Skeletons

73

Matlab ExampleI = imread('im1.bmp');image(bwdist(I)); colormap(gray)image(bwdist(I)); colormap(gray)

Idist = bwdist(I);IdistNormed = 255* (Idist / max(max(Idist)));IdistNormedThreshed = IdistNormed;

i iIdistNormedThreshed(IdistNormed<1) = NaN;surf(IdistNormedThreshed)

74

Medial axis transform

• Alternative skeleton definition:• The skeleton is the union of centres of maximal

discs within X.• A maximal disc is a circular subset of X that touches

the boundary in at least two places.

• The MAT is the skeleton with the maximalThe MAT is the skeleton with the maximal disc radius retained at each point.

75

Medial axis transform

76

Skeletonization usingSkeletonization using morphologymorphology

• Use structuring element 0 1 0

1 1 1B =

• The n th skeleton subset is

1 1 1

0 1 0

• The n-th skeleton subset is[ ]BBXBXXS nnn o) (\) ()( =

n denotes nsuccessive erosions

• The skeleton is the union of all the skeleton subsets:

erosions.

all the skeleton subsets:

U∞

= )()( n XSXS

77

U=1n

MAT Implementation

• How can we extend the skeletonization algorithm to obtain the MAT?

78

Reconstruction

• We can reconstruct region X from its skeleton subsets:

U∞

⊕)( BXSX U=

⊕=0

)(n

nn BXSX

• We can reconstruct X from the MAT.• We cannot reconstruct X from S(X)• We cannot reconstruct X from S(X).

79

DiFi: Fast 3D Distance Field Computation Using Graphics HardwareSud, Otaduy, Manocha, Eurographics 2004

80

, y, , g p

MAT in 3D

from Transcendata Europe Medial Object

81

from Transcendata Europe Medial Object Price, Stops, Butlin Transcendata Europe Ltd

Applications and Problems

• The skeleton/MAT provides a stick figure i h i hrepresenting the region shape

• Used in object recognition, in particular, character recognition.

• Problems:• Definition of a maximal disc is poorly defined on a digital grid.• Sensitive to noise on the boundary.

• Sequential thinning output sometimes preferred to skeleton/MAT.

82

Example

k lSkeletons:

Thinned:

83

Gray-level Morphology

• Erosion, dilation• Opening and closing• The image and the structuring element areThe image and the structuring element are

gray level arrays.

• Used to remove speckle noise.• Also for smoothing, edge detection and

segmentation.84

g

Umbra{ })(:),()( xfyyxfU ≤=

f U ( f )85

f U ( f )

Top surface{ }RzxzyyxRT ∈≥= ),(allfor :),()(

T(R)R86

T(R)R

Gray level erode and dilate

• Erosion:))( )(( kUfUTkf =

• Dilation:))()(( kUfUTkf ⊕=⊕

• Open and close defined as before.87

p

f U ( f )f U ( f )

k

)(kU

88))()(( kUfUT ⊕)()( kUfU ⊕

Max and min operators

• K is the domain of k.• Erosion:

{ })()(min)( zkzxfxkf −+= { })()(min)( zkzxfxkfKz

+∈

• Dilation:{ })()(max)( zkzxfxkf ++=⊕ { })()(max)( zkzxfxkf

Kz++=⊕

∈

89

Erosion

S=ones(3,3) S=ones(5,5) S=ones(7,7)

90

Dilation

S=ones(3,3) S=ones(5,5) S=ones(7,7)

91

Speckle removalErode

Open

Salt and pepper noise

Close

92Dilate

Favorites

• E = medfilt2()• [D,L] = bwdist()

93

Summary

• Simple morphological operations• Erode and dilate• Open and close

• Applications:• GranulometryGranulometry• Thinning and thickening• Skeletons and the medial axis transform• Skeletons and the medial axis transform

• Gray level morphology

94

95

Find the Letter ‘e’

96

Find the Letter ‘e’

97

Perception: ChangePerception: Change Detection?Detection?

• Examples from Daniel Simons’s many projectsprojects

• UBC examples of change blindness

Background Subtraction

• Chromakeying is often impracticalChromakeying is often impractical

• Camera pose is sometimes locked – opportunity!

Distance Measures

• Chromakey distance – remember?T>−= gIIα

T 20T = ~20g = ( 0 255 0 ) (for example)

• Plain Background-subtraction metric:TIII >= TIII >−= bgα

T = [ 20 20 10] (for example)[ ] ( p )Ibg = Background Image

Kylie Minogue, “Come Into My World”Di Mi h l G d

101

Director: Michel Gondry, Effects: Olivier “Twist” Gondry

Making of “Come Into My World”