Appearance Models
• Shape models represent shape variation• Eigen-models can represent texture
variation• Combined appearance models represent
both
Appearance Models
• Statistical model of shape and texture
• Generative model– general
– specific
– compact
Building Appearance Models
• For each example extract shape vector
• Build statistical shape model,
Shape, x = (x1,y1, … , xn, yn)T
ssbPxx
Building Appearance Models
• For each example, extract texture vector
Shape, x = (x1,y1, … , xn, yn)T
Texture, gWarp tomeanshape
Warping texture
• Problem:– Given corresponding points in two images,
how do we warp one into the other?
• Two common solutions1. Piece-wise linear using triangle mesh2. Thin-plate spline interpolation
Interpolation using Triangles
Region of interest enclosed by triangles.
Moving nodes changes each triangle
Just need to map regions between two triangles
),( :points Control ii yx )','( :points Warped ii yx
Barycentric Co-ordinates
cbax
a b
c
x
''' cbax '
'a
'b
'c
'x
1
10 and 10if triangle theinside is
βαx
Barycentric Co-ordinates
a b
c
x)( ab
)( ac cba
cbaacabax
)1(
)()(
cbax
1
Three linear equations in 3 unknowns
1111yyy
xxx
cbacba
yx
Interpolation using Triangles
• To find out where each pixel in new image comes from in old image
• Determine which triangle it is in• Compute its barycentric co-ordinates• Find equivalent point in equivalent triangle in
original image• Only well defined in region of `convex
hull’ of control points
Thin-Plate Spline Interpolation
• Define a smooth mapping function (x’,y’)=f(x,y) such that
– It maps each point (x,y) onto (x’,y’) and does something smooth in between.
– Defined everywhere, even outside convex hull of control points
niyxyxf iiii ..1 allfor )','(),(
Thin-Plate Spline Interpolation
• Function has form )),(),,((),( yxfyxfyxf yx
ii
ni
ixixxx rrwxbayxf log),( 2
1
ii
ni
iyiyyy rrwybayxf log),( 2
1
222 )()( where iii yyxxr
)','(),(bygiven equationslinear thesolvingby found are
),,,,,( parameters The
iiii
yiyyxixx
yxyxf
wbawba
Building Texture Models
• For each example, extract texture vector
• Normalise vectors (as for eigenfaces)• Build eigen-model
Texture, g
Warp tomeanshape
ggbPgg
Face Texture Model
1b12 12
2b22 22
3b32 32
Textured Shape Modes
Shape variation (texture fixed)
Generate position of control points
Warp mean texture image
(Mean points go to new points, X)
)( ssT bPxX
Textured Shape Model
1b12 12
2b22 22
3b32 32
Combined Models
• Shape and texture often correllated– When smile, shadows change (texture) and
shape changes• Learning this correlation leads to more
compact (and specific) model
Learning Correlations
sb
gbModel assuming shape and texture independent
Model accounting for correlations between shape and texture
Learning Correlations
• For each image in training set we have best fitting shape and texture param.s
• Construct new vector,
• Apply PCA (mean + eigenvec.s of covar.)
gs bb ,
g
sc b
Wbb
cQQ
Qcb
g
sc
too.does thus
set, trainingover themean zero have and Both
c
gs
b
bb
Combined Appearance Models
cQ
WQQc
bWb
b
g
s
g
sc
ggbPgg ssbPxx
cQggcQxx
g
s
Varying c changes both
shape and texture
Combined Appearance Model
• Generate shape, X, and texture, g• Warp texture so mean control points lie on
new X
1b12 12
Face Appearance Model
1b12 12
2b22 22
Face Appearance Model
3b32 32
4b42 42
Sub-cortical structures
• 72 examples• 123 points• 5000 pixel model
Ventricles
Lentiform Nucleus
Caudate Nucleus
Shape and Texture Modes
Shape variation (texture fixed)
Texture variation (shape fixed)
Combined Appearance Model
• Shape and texture correlated
Full brain slice
Shape:
Texture:
Full brain slice
CombinedMode 1
CombinedMode 2
Problems with viewpoint
• Models require all points visible– Sometimes a problem for 2D images of 3D objects
• Small rotations (+/-30o) of face modelled well• Large rotations cause occlusions
– Eg eye hidden behind nose etc• Solutions
1. Use multiple `view based’ 2D models2. Use a full 3D model
View-Based Models
• Build 3 distinct models– Exploit symmetry
Profile Profile (Reflected)
Frontal
Half-Profile Half-Profile (Reflected)
Face Profile Model
Mode 1:
Mode 2:
Half-Profile Model
Mode 1:
Mode 2:
3D Models
• Use 3D shape model (3n-D vectors)
• Points control a polyhedral mesh
• Texture mapped onto mesh and modelled
• Reconstruct by generating new texture and mapping onto 3D mesh described by shape model
3D Models
=+
Mesh
Texture
Interpreting Images (1)
Place model in image
Measure Difference
Update Model
Iterate
Top Related