Image Segmentation Using Gm Ms

ALLAN [email protected]

HTTP: / /WWW.CS.UWATERLOO.CA/~ADCAINE/

MAY 28 , 2009

Chroma Keying using Gaussian Mixture Models (GMMs)

mailto:[email protected]

http://www.cs.uwaterloo.ca/~adcaine

Allan Caine, http://www.cs.uwaterloo.ca/~adcaine/

2

Expectations

You should take notes in case this material appears on an exam or assignment, possibly in a different form

It is expected that you will use the Matlab software found at http://www.cs.uwaterloo.ca/~adcaine/ to generate a toy problem and solve it

These lecture notes are on-line at the above address

http://www.cs.uwaterloo.ca/~adcaine


3

Inspiration

Have you ever wondered how is this done? Can we emulate this process on a computer?


4

Co-ordination

By viewing an off-camera monitor, theweather announcer can co-ordinate herhand movements with the background.

The result is a seemingly realistic televisionbroadcast, in spite of the fact that theannouncer is not actually in front of anyweather map.


5

Teaching Objectives

1. Describe how chroma keying (blue screening) works

2. Theorize how classification might solve the chroma keying problem on a computer

3. Enumerate the axioms of probability 4. Explain the concept behind digital images5. Express the intuition behind the Expectation-

Maximization algorithm in terms of a Gaussian Mixture Model (GMM), using the definition of a Gaussian.

6. Generate a toy GMM problem, and solve the toy problem, recovering the hidden variables


6

Outline

Chroma Keying and our AssumptionsRBG Color ModelAxioms of ProbabilityGaussian DistributionsGaussian Mixture Model (GMM)Bayes’ Rule and the Expectation StepUpdate Rules and the Maximization StepSolving a Toy ProblemSegmentation using a GMM and MasksFurther Work


7

Chroma Keying and our Assumptions



8

Basic Idea behind Chroma Keying

If the background is of one color, saygreen, and the foreground person is notof that color, then the person can besegmented from the background.

The person can then be superimposed on abackground of our choosing.

Normally, the chroma key is assumed to beknown. To make this problem moreinteresting, we will assume that the chromakey is unknown, and must be learned. So,it’s an AI problem.


9

Assumptions for the Segmentation Problem

1. We want to segment the foreground from the background

2. There is a background of a single color3. However, the background color is unknown;

the background color must be learned4. No part of the person or object in the

foreground has the background color5. The foreground person or object can be in

any position in the image; we cannot deduce the background color simply by examining a pixel in a certain fixed position


10

RBG Color Model



11

Explanation of RGB Color Model

Every color is expressed as an ordered triple

Each value can be between 0 and 255; i.e. 256 possibilities 8 bits can store 256 possibilities

The numbers represent the amounts of red, green, and blue in that order (R,G,B)

If R = G = B, then the pixel is a shade of gray


12

200 222 32 12

88 22 45 23

30 31 56 22

11 98 99 100

89 120 56 23

89 32 90 111

45 67 23 29

49 111 121 122

RGB as a Data Structure in Matlab

45 188 64 90

111 255 0 11

88 44 21 211

45 32 45 88

)12,23,90()32,56,64(

)222,120,188()200,89,45(

43

21

xx

xx


13

Scatter Diagrams

If only we could determine which class a pixel belonged to: foreground orbackground; the purple sphere or the green background, respectively.


14

Make your own Scatter Plot


15

Actual Scatter Plot


16

Axioms of Probability



17

(Some) Axioms of Probability

)(1)(

1)(

0)(

1)(

1)(0

aPaP

dDP

falseP

trueP

aP

ii


18

Bayesian Philosophy of Choosing

Suppose two events, A and B, occur with probability P(A) and P(B) such that P(A) + P(B) = 1.

If you had to place a bet on which event would occur and if P(A) ≠ P(B), what event would you bet on?

Suppose P(A) = P(B). What event would you bet on?


19

Gaussian Distributions



20

Gaussian Distributions

)()(2

1 1

)2(

1)(

μxμxx

T

ePn


21

One-D Gaussian


22

Variations in μ


23

Variations in σ


24

2D Gaussian


25

Comparison of two 2D Gaussians


26

2D Gaussian with σx ≠ σy


27

3D Gaussian


28

Gaussian Mixture Model (GMM)



29

GMM Model

The number of classes is knownEach class has a particular mean and

varianceEach class has a mixing (or membership)

weight. All mixing weights add up to one


30

Toy Problem Parameters

ClassMixingWeights μ Σ

1 .2 (0.0 -0.1) 1.0 0.20.2 0.8

2 .4 (10.1 10.1)

2.6 1.81.8 1.9

3 .4 (9.9 -2.1) 3.3 1.6 1.6 2.1


31

Generating the Toy Problem’s DataSet

Start

8.02.0

2.00.1)1.0,0.0(

9.18.1

8.16.2)1.0,0.10(

1.26.1

6.13.3)1.2,9.9(

20%

40%

40%


32

The GMM Problem

Start

????

??????)(??,

??%

????

??????)(??,

????

??????)(??,

??%

??%


33

Bayes’ Rule and the Expectation Step



34

Bayes’ Rule

)(

)()|()|(

AP

BPBAPABP

The likelihood of A given B

A normalization constantThe likelihood of B given A

A prior probability on B


35

The Main of Idea of GMMs: E-Step

)(

)()|()|(

j

jj P

iCPiCPiCP

x

xx

The probability that position j is generatedby class 1 or 2: foreground or background

The probability of position j havinga pixel with values x given class i. Basically, evaluate a Gaussian at x using μi and Σi as parameters

The prior probability ofclass i; i.e. the mixingweights

Essentially, a normalization constantto ensure that P(C=1|xj) +P(C=2|xj) = 1for j fixed.

)(

)()|()|(

AP

BPBAPABP


36

Update Rules and the Maximization Step



37

Update Rules: M-Step

records ofnumber

:

)(

)|(

:

ii

i

Tijjij

ji

i

jiji

i

ijj

i

jij

pw

p

p

p

p

Update

iCPw

pp

iCPp

Define

μxμxΣ

xμ

x

iWhat do you think is the intuitionbehind these three update rules?

In words, define what each ofthese mathematical statementsmean.


38

Solving a Toy Problem



39

Toy Problem Visually


1 .2 (0.0 -0.1) 1.0 0.20.2 0.8

2 .4 (10.1 10.1)

2.6 1.81.8 1.9

3 .4 (9.9 -2.1) 3.3 1.6 1.6 2.1


40

Typical Output

Class 1Class 3

Class 2


41

Solution to the Toy Problem


42

Comparison of Statistics

ClassToy Problem GMM Solver

Mix μ Σ Mix μ Σ

1 .2 (0.0 -0.1) 1.0 0.20.2 0.8 .2 (0.0 -0.1) 1.0 0.2

0.2 0.8

2 .4 (10.1 10.1)

2.6 1.81.8 1.9 .4 (10.1

10.1)2.6 1.81.8 1.9

3 .4 (9.9 -2.1) 3.3 1.6 1.6 2.1 .4 (9.9 -2.1) 3.0 1.6

1.6 2.1


43

Recovery of the Model


44

3D GMM Problem


45

Segmentation using a GMM and Masks



46

Segmentation Example


47

Segmentation

The upshot of all of this is that the foreground has a parameterization of:


Background 0.9695 (62 195

52)

137 42 159

42 149 49159 49

188

Foreground 0.0305 (122 35

140)

167 26 191

26 134 30191 30

220

The novelty of this result is that we have a complete and numericalcharacterization of the image without knowing beforehand thecolor of the background. These parameters have been learned.


48

Producing the Mask

)2()2|()1()1|(

)1()1|()|1(

CPCPCPCP

CPCPCP

jj

jj xx

xx

The mixing weight 0.9695

The mixing weight 0.0305

A Gaussian computed using themean and variance of the foregroundclass.

A Gaussian computed using themean and variance of the backgroundclass.

How would we use a Bayesian philosophy to decide ifa pixel represents the foreground or background?


49

Masks

This mask permits the sphereto show through while excludingthe background. It’s used tosegment the sphere; that is,segment out the foreground.

This mask is the logical complementof the first mask. It indicates wherethe background pixels can be found.This mask is used to prepare the newbackground so that it may becombined with the foreground.


50

Using the Masks for the Foreground and Background


51

Adding the Images Together


52

Before and After

Before After


53

Further Work



54

Further Work

Complex foregrounds and backgroundsTracking objectsSpeeding up processingSamplingSmoothing the transition from background to

foreground to avoid ragged edgesUsing alternative color spaces such as Hue-

Saturation-Value (HSV) for better classification

Image Segmentation Using Gm Ms

Documents

Transcript of Image Segmentation Using Gm Ms