Computer vision: models, learning and inference

Chapter 18 Models for style and identity

Please send errata to s.prince@cs.ucl.ac.uk

2

Identity and Style

2Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Identity differs, but images similar

Identity same, but images quite

different

3

Structure

3Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Factor analysis review• Subspace identity model• Linear discriminant analysis• Non-linear models• Asymmetric bilinear model• Symmetric bilinear model• Applications

4

Factor analysis review

4Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Generative equation:

Probabilistic form:

Marginal density:

5

Factor analysis

5Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

6

Factor analysis review

6Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

E-Step:

M-Step:

7

Factor analysis vs. Identity model

7Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Each color is a different identity• multiple images lie in similar part of subspace

8

Subspace identity model

8Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Generative equation:

Probabilistic form:

Marginal density:

9

Subspace identity model

9Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

10

Factor analysis vs. subspace identity

10Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Factor analysis Subspace identity model

11

Learning subspace identity model

11Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

E-Step:

Extract moments:

12

Learning subspace identity model

12Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

M-Step:

E-Step:

13

Subspace identity model

13Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

14

Subspace identity model

14Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

15

Inference by comparing models

15Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Model 1 – Faces match (identity shared):

Model 2 – Faces dont match (identities differ):

Both models have standard form of factor analyzer

16

Inference by comparing models

16Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Compute likelihood (e.g. for model zero)

where

Compute posterior probability using Bayes rule

Face Recognition TasksPROBE

…

GALLERY

? 1. CLOSED SET FACE IDENTIFICATION

…

GALLERY PROBE

?NO MATCH

2. OPEN SET FACE IDENTIFICATION

PROBE

?NO MATCH

3. FACE VERIFICATION

4. FACE CLUSTERING?

17Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

18

Inference by comparing models

18Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

19

Relation between models

19Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

20

Structure

20Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Factor analysis review• Subspace identity model• Linear discriminant analysis• Non-linear models• Asymmetric bilinear model• Symmetric bilinear model• Applications

21

Probabilistic linear discriminant analysis

21Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Generative equation:

Probabilistic form:

22

Probabilistic linear discriminant analysis

22Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

23

Learning

23Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

E-Step – write out all images of same person as system of equations– Has standard form of factor analyzer– Use standard EM equation

M-Step – write equation for each individual data point– Has standard form of factor analyzer– Use standard EM equation

24

Probabilistic linear discriminant analyis

24Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

25

Inference

25Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Model 1 – Faces match (identity shared):

Model 2 – Faces dont match (identities differ):

Both models have standard form of factor analyzer

Compute likelihood in standard way

26

Example results (XM2VTS database)

26Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

27

Structure

27Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Factor analysis review• Subspace identity model• Linear discriminant analysis• Non-linear models• Asymmetric bilinear model• Symmetric bilinear model• Applications

28

Non-linear models (mixture)

28Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Mixture model can describe non-linear manifold.

Introduce variable ci which represents which cluster

To be the same identity, must also belong to the same cluster

29

Non-linear models (kernel)

29Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Pass hidden variable through non-linear function f[ ].• Leads to kernelized algorithm• Identity equivalent of GPLVM

30

Structure

30Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Factor analysis review• Subspace identity model• Linear discriminant analysis• Non-linear models• Asymmetric bilinear model• Symmetric bilinear model• Applications

31

Asymmetric bilinear model

31Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Introduce style variable sij

• indicates conditions in which data was observed • Example: lighting, pose, expression face recognition

Asymmetric bilinear model

• Introduce style variable sij

• indicates conditions in which data was observed • Example: lighting, pose, expression face recognition

32

Asymmetric bilinear model

32Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

33

Asymmetric bilinear model

33Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Generative equation:

Probabilistic form:

Marginal density:

34

Learning

34Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

E-Step:

M-Step:

35

Asymmetric bilinear model

35Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

36

Inference – inferring style

36Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Likelihood of style

Prior over style

Compute posterior over style using Bayes’ rule

37

Inference – inferring identity

37Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Likelihood of identity

Prior over identity

Compute posterior over identity using Bayes’ rule

38

Inference – comparing identities

38Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Model 1 – Faces match (identity shared):

Model 2 – Faces dont match (identities differ):

Both models have standard form of factor analyzer

Compute likelihood in standard way, combine with prior in Bayes rule

39

Inference – Style translation

39Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Compute distribution over identity

• Generate in new style

40

Structure

40Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Factor analysis review• Subspace identity model• Linear discriminant analysis• Non-linear models• Asymmetric bilinear model• Symmetric bilinear model• Applications

41

Symmetric bilinear model

41Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Generative equation:

Probabilistic form:

Mean can also depend on style...

42

Symmetric bilinear model

42Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

43

Inference – translating style or identity

43Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

44

Multilinear models

44Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

Extension of symmetric bilinear model to more than two factors

e.g.,

45

Structure

45Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Factor analysis review• Subspace identity model• Linear discriminant analysis• Non-linear models• Asymmetric bilinear model• Symmetric bilinear model• Applications

46

Face recognition

46Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

47

Tensortextures

47Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

48

Synthesizing animation

48Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

49

Discussion

49Computer vision: models, learning and inference. ©2011 Simon J.D. Prince

• Generative models• Explain data as combination of identity and

style factors • In identity recognition, we build models where

identity was same or different• Other forms of inference such as style

translation also possible