Representing, Learning, and Recognizing Non-Rigid Textures and Texture Categories

Svetlana Lazebnik Cordelia Schmid Jean PonceBeckman Institute Gravir Laboratory Beckman InstituteUIUC, USA INRIA, France UIUC, USA

Supported in part by the UIUC Campus Research Board, the UIUC/CNRS Collaborative Research Agreement, and the National Science Foundation under grant IRI-990709.

• 3D objects are never planar in the large,but they are always planar in the small.

• Representation: Local invariants andtheir spatial layout.

• Affine-invariant patches.

LeCun’03

(Lindeberg & Garding’97)(Mikolcajczyk & Schmid’02)

• Spatial selection • Shape selection• Affine adaption

Schaffalitzky & Zisserman (2001); Tuytelaars & Van Gool (2003)

(0,0)

(0,1)

(1,0)

Image 1 Image 2

Affine adaptation/Rectification process

Lindeberg & Garding (1997)Mikolcajczyk & Schmid (2002)Rectified patch

[Range spin images: Johnson & Hebert (1998)]

Intensity-Domain Spin Images

System architecture (Lazebnik, Schmid, & Ponce, CVPR’03)

[Signatures and EMD for image retrieval: Rubner, Tomasi, & Guibas (1998)]

• Signature: SS = { ( m1 , w1 ) , … , ( mk , wk ) }• Earth Mover’s Distance: D( SS , SS’’ ) = [i,j fij d( mi , m’j)] / [i,j fij ]

Texture retrieval/classification experiments

Schmid (2001); Varma & Zisserman (2002)

10 texture classes, with 20 samples per class.

NN classification

Ave

rage

reco

gniti

on ra

te

Ave

rage

reco

gniti

on ra

te

More retrieval/classification experiments: Brodatz database

• Picard et al. (1993, 1996)• Xu et al. (2000)

111 images divided into 9 windows

111 classes with 9 samples per class

Ave

rage

reco

gniti

on ra

te

Ave

rage

reco

gniti

on ra

te

T1 (brick) T2 (carpet) T3 (chair) T4 (floor 1) T5 (floor 2) T6 (marble) T7 (wood)

Multi-texture Samples

Texture Classes [NOTE: we do NOT use color information.]

A Two-Layer Architecture(Lazebnik, Schmid, & Ponce, ICCV’03)

Modeling:1. Use EM to learn a mixture-of-Gaussians model of

each texture class.2. Compute co-occurrence statistics of sub-class labels

over affinely adapted neighborhoods.

Recognition:1. Use the generative model to obtain initial class

membership probabilities.2. Use relaxation (Rosenfeld et al., 1976) to refine these

probabilities.

Malik, Belongie, Leung, & Shi (2001); Schmid (2001); Kumar & Hebert (2003)

Neighborhood Statistics

Estimate:• probability p(c,c’),• correlation r(c,c’).

Relaxation (Rosenfeld et al., 1976)

Iterate, for all regions i:

where

and wij=0 is region j is not in the neighborhood of i, with j wij=1.

Classification rates for single-texture images

10 training images per class, 10 test images per class.

Weakly-Supervised Modeling

Idea: Replace L mixture models with M components by a single mixture model with L x M components.

• Annotate each image with the set C of labels associated with classes occurring in it.

• Run EM:• E step: update class membership probabilities:

p (clm | x, C ) / p ( x | clm ) p ( clm | C ).• M step: update model parameters.

Nigam, McCallum, Thrun & Mitchell (2000)

T1 (brick) T2 (carpet) T3 (chair) T4 (floor 1) T5 (floor 2) T6 (marble) T7 (wood)

T1 (brick) T2 (carpet) T3 (chair) T4 (floor 1) T5 (floor 2) T6 (marble) T7 (wood)

Single-texture training images only

Single- and multi-texture training images

ROC Curves

10 single-texture images per class, 13 two-texture training images, 45 multi-texturetest images.

Effect of relaxation on labelingOriginal image

Top: before relaxation, bottom: after relaxation

Successful Segmentation Examples

Unsuccessful Segmentation Examples

Animal Dataset

• 10 training images for each animal + background, 20 test images per class.

Bradshaw, Scholkopf, & Platt (2001); Schmid (2001); Kumar & Hebert (2003)

• No manual segmentation.

Oh well..

• 3D Objects without distinctive texture

• Category-level recognition of 3D objects

• Please join us in trying to solve the 3D object recognition problem..