The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv...

34
e Quotient Image: Class-based Recognit and Synthesis Under Varying Illuminati T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew University

Transcript of The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv...

Page 1: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination

T. Riklin-Raviv and A. Shashua

Institute of Computer Science Hebrew University

Page 2: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

Example: 3 basis images

The movie is created by simplylinearly combining the 3 basis images.

Page 3: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

Given a single image y =

And a database of other images of the same “class”

Image Synthesis:

We would like to generate new images from y simulatingchange of illumination.

BootstrapSetlight

ID

Page 4: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

Recognition

Given a database of images of the same “class”, under varying illumination conditions

and a novel image

Match between images of the same object.

Page 5: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

Definition: Ideal Class of Objects

The images produced by an ideal class of objects are

jT

i syxnyxyxI ),(),(),(

Where ),( yxi is the albedo (surface texture) of object i of the class

),( yxn is the normal direction (shape) of the surface, i.e., all objects have the same shape.

js is the point light source direction.

yx,

Page 6: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

Related WorkBasic result :Shashua 91,97

Application and related systems:Hallinan 94,

Kriegman et. al 96,98

Work on “class-based” synthesis and recognition of images-mostly with varying viewing positions:Basri 96, Bymer & Poggio 95, Freeman & Tenenbaum 97,Vetter & Blanz 98, Vetter, Jones & Poggio 97, 98Edelman 95, Atick, Griffin & Redlich 97.

Rendering under more general assumption: Dorsey et al. 93,94

Linear class : Vetter & Poggio 97,92

Additive error term : Sali & Ulman 98

Reflectance Ratio: Nayar & Bolle 96

Page 7: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

The Quotient image: Definition

Given images ss ay ,

of objects y and a respectively, under illumination S

a

yyQ

=

sNy Tys

sNa Tas

Thus yQ depends only on relative surface information

and is independent of illumination.

(pixel by pixel division)

Page 8: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

The Quotient image Method: Proposition

Let 321 ,, aaa 3 images of object a .

Let sy Image of object y illuminated by light s .

Moreover, the image space of y is spanned by

varying the coefficients.

321 ,, xxx that satisfy:

j

yjjs Qaxy )(

Then, there exist

Page 9: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

The Quotient image Method: Proof

321 ,, aaa Illuminated by: 321 ,, sss

j

jjsxssy Illuminated by:

j

yjjs Qaxy )(

snTa

a

y

Page 10: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

a1a2

a3

Pa

a1a2

a3

Q-Image

* =?N=1

j

jjsxs

s

s1

s2

s3

The Quotient image Method: N=1sy

s332211 axaxaxas

sa

sa

Page 11: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

Pa

a1a2

a3

* =

Q-ImageN=1

The Quotient image Method: N=1sa Synthesized Image

s

332211 azazazas

sa s

Page 12: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

The Quotient image Method: Conclusions

yQ one can generate sy and all other images of

the image space of y .

Given

Given sy and the coefficients jx that satisfies

j jjsxs then yQ readily follows

j jj

sy ax

yQ

In order to obtain the correct coefficients jx

a bootstrap set of more than one object is needed .

Page 13: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

The Quotient image Method: N>1

Original image

sy

s

xA1

1AP

xA2

2AP

xA3

3AP

syxA 11 syxA 22

syxA 33

N

isii yxAxf

1

2

3misAi

Page 14: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

The Quotient image Method: Theorem-1

The energy function

N

isii yxAxf

1

2

has a (global) minimum xx ˆ , if the albedo y

of object y is rationally spanned by the bootstrap set.

i.e if there exist coefficients N ,,1 such that

NN

Ny

11

221 ...

Page 15: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

N

isii yxAxf

1

2

iiis

i

Tii

ii

Ti vyAAAx

1

ˆ

sTi

N

rr

Tri yAAAv

1

1

The Quotient Image Method: Solving For X and i

Min,x

We also have: sTi

Ts

Tsi

i

yAxyyf

ˆ0

Page 16: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

for Ni 1 written explicitly

0

0

0

11

2211

1111

sTss

TN

TNNs

TN

T

sTT

NNsTT

sTT

NNsTss

TT

yyyAvyAv

yAvyAv

yAvyyyAv

The Quotient image Method: Solving For X and i

Page 17: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

The Algorithm

Given : NAA ,,1 a bootstrap set and a novel image sy

homogenous system of linear equations in N ,1

Scale such that i i N

Compute i iivx

Ax

yQ s

y Where A is the average of NAA ,,1

ynew QAzZy )( For all choices of z

Use the minimization function:

N

isii yxAxf

1

2

1ˆ Min

,xto generate

Page 18: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

Samples of few faces out of 9*200 faces images taken from T. Vetter databasewhich was mainly used as a bootstrap set and as a source for novel images inthe further demonstration.

Frontal faces : Collection of objects all have the same shape but differ in their surface texture (albedo)...

light

ID

Page 19: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

Synthesis from Single PictureAnd 10 faces from the bootstrap setunder 3 different light conditions

Synthesis from 3 pictures

Linear CombinationQuotient Method

Page 20: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

10 other faces

from the database, each under 3light conditions

Synthesis from 3 picturesSynthesis from Single imageand the bootstrap set

Page 21: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

Synthesis from 3 picturesSynthesis from Single imageand the bootstrap set

Bootstrap Set

Original image Quotient Image

Page 22: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

(0,+20) (+50,0)(-50,0)(0,-35)(0,0)

Original Images Compared to Q-Image Synthesized Images

1st Row: Original Images

2nd Row: Q-Image Synthesized Images

3rd Row: Exact Values of Light Direction: center, down, up, right, left

Page 23: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

Light Coefficient Comparison Ground Truth Vs. Q-Image Coefficients

1st Coefficient

2nd Coefficient

3rd Coefficient

Page 24: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

Using Different Database

Animation Using the database

3x3 images’ Database

The Quotient Image The original image

Page 25: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

The Original images

Q images , 1 object bootstrap set

Q images , 10 object bootstrap set

Page 26: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

Handling Color Images

RGB HSV Transformation

Original color image R G B

H S V

Page 27: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

Original Image Quotient Image

Synthesized Sequence

Page 28: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

Original Images Quotient Images

Synthesized Sequences

Monica and Bill Under a New Light

Page 29: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

Original Image Quotient Image

Synthesized Sequence

Page 30: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

Recognition under varying illumination

Each object in the database is represented by its quotient image only.The quotients can be made of images with varying illuminations.

The quotient images was generated out of N*3 (N=20) base images.

A=

B=

C=

. . .

. . .

N

Database generation

Page 31: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

Identification

Given a new image of an object appears in the data base under any lightcondition, it’s quotient is computed from A,B,C … (as was done in the database generation) . Then It is compared to the quotientsin the data base.

Other methods used for comparison

1. CorrelationDatabase: Each object is represented in the database by it’s image under

any/certain lightening condition. Identification: Correlation between the test image to the images stored in the

database.2. PCA Database: Applying PCA on the objects’ images + 3*20 additional images of 20 objects under 3 illumination (to compare conditions to the quotient method). Having eigen vectors,

each object is represented by it’s eigen vectors’ coefficients.Identification: Comparison (LSE) between the test image coefficients (generated the same way as the database) and the database.

Page 32: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

Recognition Results

0%

5%

10%

15%

20%

25%

30%

35%

sameill.

varyingill

Q methodCorrelation

Quotient method comparing to correlation

Page 33: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

Recognition Results - cont

0%

1%

2%

3%

4%

5%

6%

7%

same ill. varyingill

Q methodPCA 30 evPCA 50 ev

Quotient Method Vs. PCA

Page 34: The Quotient Image: Class-based Recognition and Synthesis Under Varying Illumination T. Riklin-Raviv and A. Shashua Institute of Computer Science Hebrew.

The End