Active shape appearance model-presentation 1st

We value our relationshipWe value our relationships.

9 December 2012

© IRDC India 2012 www.irdcindia.comWe value our relationship

Chandrashekhar Padole

Title for Presentation Active Shape/Appearance Model ( ASM & AAM)by

IRDC India

IRDC India

www.irdcindia.com

Presenter: IRDC [email protected]

9 December 2012 2

Objectives

• To understand Active Shape Model

• To investigate the computations involved in ASM

• MATLAB Modules

• To understand Active Appearance Model

• To investigate the computations involved in AAM

• Application in Pose Estimation and Pose Compensation

IRDC India

www.irdcindia.com


9 December 2012 3

Roadmap

•Paper I

•Paper II

•Paper III

• Paper IV

• Paper V

IRDC India

www.irdcindia.com


9 December 2012 4

Paper I

Title: Training Models of shape from set of examples

Authors:T.E. Cootes, C.J.Taylor et.al (BMVC 1992)

•Shape of resistors

–different shapes in

some extent

•Labelling training set

IRDC India

www.irdcindia.com


9 December 2012 5

Contd..

•Constraints over labelling point across the set of training samples

• Point 0 & 31 always represent end of wires

• Point 3 , 4 ,5 represent one end of the body of the resistor and so on

• Manual process is ok for simple shapes but for some biological complex

shapes �automated tools

•Alligning the training set

• By scaling, rotating and translating

• To make them as close as possible – minimise a weighted sum of suqares of

distances between equivalent points of diffrent shapes

• This is form of Genralized Procrustes Analysis

IRDC India

www.irdcindia.com


9 December 2012 6

Contd..

•Genralized Procrustes Analysis ( Ref- Generalized Procrustes

analysis by Gower , 1975)

• Let xi be a vector describing the n points of the ith shape in the set

– xi=(xi0,yi0, xi1,yi1 ,...... xik,yik ........ xi(n-1),yi(n-1)

• Let Mj[xj] be a raotaion by ϴj and a scaling by sj

• Given two similar shapes ,xi and xj, we can choose ϴj , sj and translation

(tx,ty)j mapping xi onto Mj[xj] so as to minimize weighted sum

where

and W is diagonal matrix of weights for each point and used

for giving importance more or less to corresponding lable

points and it can be choosen practially as described ahead

])[.(.])[( jji

T

jjij xMxWxMxE −−=

+−

+−=

jyjkjjjkjj

jxjkjjjkjj

jk

jk

jtysxs

tysxs

y

xM

)cos()sin(

)sin()cos(

θθ

θθ

IRDC India

www.irdcindia.com


9 December 2012 7

Contd..

•Choosing Wk �

• let Rkl be the distance between points k and l in a shape

• VRkl be the varaince in this distances over the set of shapes, then Wk for the kth

point ,

• Thus ,a point tends to be remain fixed with respect to the otehr , if sum of variances

will be small , a large weight will be given and matching such points indiffrent

shapes will be a priority

11

0

−−

=

= ∑

n

l

Rk klVW

IRDC India

www.irdcindia.com


• Alignment algorithm

1. Rotate ,scale and trasnlate each of the shapes in the set to align the first

shape

2. Repeat1. Calcualte the mean of the transformed shapes

2. Either

1. Adjust the eman to a default scale ,orientation and origin

2. Rotate scale translate the mean to align the first shape

3. Rotate ,scale and translate each of the shapes again to amtch the adjusted mean

3. Untill Convegence

•Inside the iteration loop , it is requried to renormalise the

mean,without whcih this algorithm is ill-conditioned

Practical Implementation:

9 December 2012 8

Contd..

IRDC India

www.irdcindia.com


9 December 2012 9

Contd..

•Statistics of set of aligned shapes

• Mean shape

• Apply PCA to deviations from

mean to find the modes of variations

∑=

=sN

i

i

s

xN

x1

1

IRDC India

www.irdcindia.com


9 December 2012 10

Paper II

Title: Use of Active Shape Models for locating structures in Medical

Images

Authors:T.E. Cootes, C.J.Taylor (Image & Vision Computing 1994)

•Modelling object shape-point distribution model ( PDM)

• We have set of images containing examples of variable structure

• E.g. Left Ventricle –shape of this can vary bioth with time( as heart beats)

and across individuals

• These shape variations is to be modeled ( ASM)

• Choose points around the left ventricle boundary and also around the nearby

edge of the right ventricle and top of the left atrium

IRDC India

www.irdcindia.com


9 December 2012 11

Contd..

•Addition of points other than left ventricle gives more specific

model whcih peroform better iamge search for left ventricle

•Each smaple from set of examples , represented by 96 points

•Used 66 images ( examples/samples)

•11 key positions were marked on each boundary

•96 points points gnerated from the key points along the boundaries

between key positionss

•In order to be able to compare equivalent points from diffrent

shapes, they were alligned by scaling, rotating and translating the

training shapes so that they correspond as closely as possible

IRDC India

www.irdcindia.com


9 December 2012 12

Contd..

•Capturing the statistics of a set of aligned shapes

• N aligned shapes, then ith shape is given by

– xi=(xi0,yi0, xi1,yi1 ,...... xik,yik ........ xi(n-1),yi(n-1)

• Mean shape and deviation

• 2n x 2n covaraince matrix S ,

• Eigen Analysis of S will give modes of variations by Pk(k=1....2n)

Where is kth eigen value of S and ,

• t modes of variations – select t columns of Pk corresponding values

of

∑=

=N

i

ixN

x1

1xxdx ii −=

∑=

=N

i

T

iidxdxN

S1

1

kkp PSk

λ=

kλ 1+≥ kk λλ 1=k

T

k PP

kλ

IRDC India

www.irdcindia.com


9 December 2012 13

Contd..

•Shape reconstruction:Any shape in training set can be apporximated

using mean shape and weighted sum of deveiations from first t

modes

where P=(P1,P2,...Pt) is the matrix of first t eigen vectors

and b = (b1,b2,...bt)T is the vector of weights

•Since, eigen vectors are orthogoanl , so

•The above equation allow us to generate new examples of the shapes

by varying the parameters (b)within suitable limits, so the new

shapes will be similar to those in the training set.

•The limits for each each element of b , bk, are derived by examining

the distribution of the parameters values required to generate the

training set.

Pbxx +=

1=PPT )( xxPb

T −=

IRDC India

www.irdcindia.com


9 December 2012 14

Contd..

•If gaussian distribution are assumed , one can choose sets of

parameters {b1,b2,...bt }such that the Malhalanobis distance (Dm)

from teh mean is less than sutiable value Dmax

•Effects of variations of model parameters,{b}

2

max

1

22

Db

Dt

k k

km ≤

=∑

= λ

IRDC India

www.irdcindia.com


9 December 2012 15

Contd..

•Modelling Grey Level Appearance

• Grey level patterns about the key point of model in images of diffrent

examples will be often similar

• For every model point i in each image j, we can extract a profile ,gij, of

length np pixels, centered at the point

• Author choose to smaple the derivative of the grey levels along the profile in

the image and normalize it.

• Profile runs from Pistartto Piend

and is of length np pixels,

• kth element of the derivative profile is

where yik is the kth point along the ith profile and is given by

and is the grey level in image j at that point

)()( )1()1( −+ −= kijkijijk yIyIg

)(1

1startendstart ii

p

iik ppn

kpy −

−

−+=

)( ikj yI

IRDC India

www.irdcindia.com


9 December 2012 16

Contd..

•We then normalise this profile vector,

•For each point ,i, we can calcualte a mean normalised derivative

profile

•Sgi is the npxnp covraince matrix of

•Calcualting the eigen vector and values , we get model parameter

associated with grey level or appearance.

∑=

=′pn

k

ijk

ij

ij

g

gg

1

∑=

′=sN

j

ij

s

i gN

g1

1

ig

IRDC India

www.irdcindia.com


9 December 2012 17

Contd..

Practical Implementation:

IRDC India

www.irdcindia.com


9 December 2012 18

Paper III

Title: Modelling Object Appearance using the Grey-level Surface

Authors: T.E. Cootes, C.J.Taylor (BMVC 1994)

•Based on Landmark ( LM) points and triangulation

• Labeled point-particular part of stucture, a corner, a point of high intesity etc

IRDC India

www.irdcindia.com


9 December 2012 19

Contd..

• First traingulate origianl LM points . Then Add traingualtion by

adding additioanl points at the mid-point of each connecting arc

•Instead of applying traingualtion to each example(sample), it is

applied to the mean configuration of landmarks. This mean is

generated by alligning the set of examples so that they overlap as

much as possible and then calculating the mean of each co-ordinate

for the LM point

IRDC India

www.irdcindia.com


9 December 2012 20

Contd..

•To add the appearance infaotaion to the shape each point is

decribed as triplet. ( xi,yi,Ii) to have

x=(x0,y0,λI0 , x1,y1, λ I1 ...... Xn-1,yn-1, λ In-1 )

where λ� proportionality constant to allow for x & y being

measured in diffrent units to the grey-level intensity.

•Apply PCA to the set of example vectors ,

�the mean set of points

P� 3n x t matrix, the columns of which are the t orthonramal

unit eigen vectors of the covaraince matrix corresponding to the

largest eighen value ( each column describes a mode of shape

varaition in the data, the first being the msot significant

b� set of t model parameters

PbxX += ˆ

x̂

IRDC India

www.irdcindia.com


9 December 2012 21

Contd..

•Varying the model parameters , b=(b1,b2,...bt) with certain limits

whcih can be learnt from the stastics of the training set, we can

generate new examples of the 3D shape,simialar the those in training

set.

•Varying each parameter causes changes to both the position of each

landmark and the intesity values at that landmark.

•Example of models:• Eye model

• Banana model

IRDC India

www.irdcindia.com


9 December 2012 22

Contd..

•Eye Model-

• 10 example (left eye from 10 persons)

• 9 base landmark points

• Traingulation algorithm with two iterations

of interpolations to get 345 points

• the relative importance of intensity

to x,y values , λ=0.25,

IRDC India

www.irdcindia.com


9 December 2012 23

Contd..

•Banana model-• Bananas were

illuminated from

diffrent directions

• 33 points � 369 points

• λ=0.25

•Application: Image

Search

IRDC India

www.irdcindia.com


9 December 2012 24

Paper IV

Title: Active Shape Models : Evaluation of a Multi-resolution

Method for Improving Image Search

Authors: T.E.

Cootes, A. Lanitis, C.J.Taylor (BMVA 1994)

•Image Search using an Active Shape Model

Where

M(s,ϴ)[.] performs a rotation by ϴ and a scaling by s

xc,yc is the position of the centre of the model in the

image frame

• Image search problem� searching for s, ϴ, (xc,yc)

T

ccccc

c

yxyxX

XxsMX

),.......,(

])[,(

=

+= θ

IRDC India

www.irdcindia.com


9 December 2012 25

Contd..

•An iterative approach to improving the fit of the instance, X, to an

image proceeds as follows :

1. Examine a region of the image around each point to calculate the

displacement of the point required to move it to a better location.

2. From these displacements calculate adjustments to the pose and

the shape parameters.

3. Update the model parameters; by enforcing limits on the shape

parameters, global shape constraints can be applied ensuring the

shape of the model instance remains similar to those of the

training set.

IRDC India

www.irdcindia.com


9 December 2012 26

Contd..

•To find a better location for each model point we sample a profile

perpendicular to the boundary at the point and run the grey-level

model along it to find best match

•Use least square appraoch to find the best change in pos

(dxc,dyc,ds,dϴ)

IRDC India

www.irdcindia.com


9 December 2012 27

Contd..

• Problem- choosing the length of profile along which to search with

grey-level model

• If the search profile is too short , the model landmark points must be close

to their targets in teh iamge before they can ‘latch on ‘ and pull the shape

model into place.

• If they are too long the search becomes computationally expensive and the

grey-level models are more likely to latch on to distracting stractures in the

image away from the target object,preventing ASM from converging to the

correct shape.

•Solution: far from target,make large jumps and as model approaches

target structure, search should be restricted to immediate locality

• Multi resolution approach- Model to be applied first at coarse mean

to low resolution image , then refined oh higher resolution images.

• Low resolution images( Level 1 ,2...) can be obtained from Level 0,

i.e. original image by smoothing the image and subsampling every

other pixel.

IRDC India

www.irdcindia.com


9 December 2012 28

Contd..

•As smoothing may modify the image structure, grey level model is

not restrinceted to be obtained only from Level 0 image but from

other levels of images.

•Thus, each landmark point will have set of grey level models.

•Start first from highest level image ( level N)and run number of

iterations of the ASM using the models trained at that level. Then

move to next level towards level 0.

•Note: higher level�coarse and level 0� finest

•Examples of Multi resolution search

• Face Model

• Vertebra Model

IRDC India

www.irdcindia.com


9 December 2012 29

Contd..

•Face Model- 169 points, 11 images of single person’s face,

greylevel models 7 pixel long

•Each iteration at

any level takes less

than 150ms on

Sun Sparc10

workstation

•Search completed

in 7 sec

IRDC India

www.irdcindia.com


9 December 2012 30

Paper IV

Title: A Unified Approach to Coding and Interpreting Face Images

Authors: T.E. Cootes, G.J. Edwards, C.J.Taylor (ICCV 1995)

•Shape model-152 points on each of 160 training examples. 16

parameters

IRDC India

www.irdcindia.com


9 December 2012 31

Contd..

•Shape-free Appearance Model

• Apply warping algortihm to deform all training images to the mean shape,

in such a way that changes in grey-level intensities are kept to a minimum

• Training images were deformed to the mean shape and grey level intensities

within the face area were

extracted.

• Each training example was

represented a vector

containing the nomalised

grey-level at each pixel in

the patch

• A flexible grey-level model

was generated for dataase;

only 79 parameters were

needed to explain 95% of

the variation

IRDC India

www.irdcindia.com


9 December 2012 32

Contd..

•Local Appearance Model- 4 model parameters to explain 95% of

the variation

IRDC India

www.irdcindia.com


9 December 2012 33

Contd..

IRDC India

www.irdcindia.com


9 December 2012 34

Contd..

IRDC India

www.irdcindia.com


9 December 2012 35

Contd..

IRDC India

www.irdcindia.com


9 December 2012 36

Contd..

IRDC India

www.irdcindia.com


9 December 2012 37

Paper V

Title: Statistical Models of Face Images –Improving Specificity

Authors: T.E. Cootes, C.J.Taylor et.al (IVC 1998)

•Previous work, Shape model and grey level model

•When new image is

presented to this

system, facial features

are located autoamtically

using active shape model

( ASM).

•The resulting

auomatically located

model points are

transformed into shape model parameters. The face is deforemd to the mean face

shape and grrey level appearance is transformed into the parameters of the shape-

free grey level model.

IRDC India

www.irdcindia.com


9 December 2012 38

Contd..

•However, shape and grey-level varaintions may be correlated

• E.g. The shape mode of variation responsible for opening and clising the

mouth is correlated with the grey level mode responsible for appearance of

teeth

•Also , certain combinations of shape and grey-level modes may

correspond to illigal facial reconstruction

•Solution: Combine ( Shape+Grey level) = Appearance Model

• Train indvidually shape and shape free grey-level models

• Represent each training example by a vector containing both sape and

grey-level parameters.

• PCA is applied to these new training vectors in order to extract the

combined shape and grey level modes of variation.

• Before applying the final PCA , we scale the shape parameters so that their

varaince wihtin the training set is equal to the variance of the grey level

parameters.

IRDC India

www.irdcindia.com


9 December 2012 39

Contd..

•Figure 3 below shows the first few modes of the combined

shape/grey-level model trained using image from Home Office DB.

•Figure 4 shows the parametric reconstructions of original images

using a compbined shape and grey-level model ( included hair)

IRDC India

www.irdcindia.com


9 December 2012 40

Contd..

•Isolating sources of Variation

• To be done later

•Using Non-linear Shape Models

• To be done later

IRDC India

www.irdcindia.com


9 December 2012 41

Paper VI

Title: Active Appearance Models

Authors: T.E. Cootes, G.J. Edwards, C.J.Taylor (PAMI June 2001)

•Statistical appearacne models are generated by combining a model

shape varaition with model of teture variation

• By “texture” means the pattern of inteisities or color across an image patch

• To build a face model , we require face iamges marked with points defining

the main features

IRDC India

www.irdcindia.com


9 December 2012 42

Contd..• Apply a Procrustes analysis to align the sets of points ,x, and build a shape

model

• Warp each training s other points match those of the mean shape,obtaining a

“shape-free patch”

• This is a raster scanned into a texture vector, g, whcih is normalized by

applying a linear trasnformation where 1 is vector of

ones , µs and σs2 are the mean and varaince of elements of g.

• After normalization, gT 1=0 and |g|=1 and calculate texture model by taking

eigen analysis of g.

• Finally the correaltions between shape and texture are learned to generate a

combined model ( as seen in previous paper)

ssgg σµ /)1( −→

IRDC India

www.irdcindia.com


9 December 2012 43

Contd..

•To be continued...

IRDC India

www.irdcindia.com


9 December 2012 44

Thank you

Queries?

IRDC India

[email protected]

visit us at www.irdcindia.com

Active shape appearance model-presentation 1st

Engineering

Transcript of Active shape appearance model-presentation 1st