MACHINE VISION GROUP

Face image mapping from NIR to VIS

Jie ChenMachine Vision Group

http://www.ee.oulu.fi/mvg

MACHINE VISION GROUP

Outline

• Problem

• Methods

• Preliminary results

• Plans for next period

MACHINE VISION GROUP

Face image mapping from NIR to VIS

• Problem– NIR: Near infrared imaging– VIS: Visual light imaging

MACHINE VISION GROUP

Face image mapping from NIR to VIS

• Problem– NIR: Near infrared imaging– VIS: Visual light imaging

MACHINE VISION GROUP

Algorithm: Patches mapping Training

• Training

wf

hf

wp

hp

wo

ho

MACHINE VISION GROUP

Algorithm: Patches mapping Training

• Mapping

φi,j

1,,k

i j

MACHINE VISION GROUP

Look up the KNN

Looking up

Ddictionary of face patches and their LBP

histograms

1,1,i j

4,i j

3,i j

1,0,i j

2,1,i j

2,0,i j

0

1

1K

1, 1,K

i j

2, 1,K

i j

1,,ki j

3,i j

k

2,,k

i j

4,i j

1,,ki j

A patch of an input sample in S3

k-th nearest patch in S1

Weight of k-th nearest neighbor

A patch of an input sample in S4

Corresponding patch of in S2

MACHINE VISION GROUP

Weight computing

1 2 1, 2,1

( , ) min( , )L

i ii

H H H H

1

0

kk K

pp

3 2,, ,

ki j k i j

Looking up

Ddictionary of face patches and their LBP

histograms

MACHINE VISION GROUP

Experiments

• Setup– both S1 and S2 is composed of 300

samples.• 50 subjects, • each subject has 6 images but in

different expression (anger, disgust, fear, happiness, sadness, and surprise).

– wf =64, hf =80, wp=16, hp=16, wo =12, ho =12

– Testing:using leave-one-out and K=15.

wf

hf

wp

hp

wo

ho

MACHINE VISION GROUP

Reconstructed images

(a) Input images in NIR

24.88 23.76 27.55 26.29 26.48 27.43 21.54 21.18

(b) Reconstructed images in VIS using LBP(8,1) and the PSNR

31.89 32.11 32.11 34.41 32.11 31.68 32.18 31.08

(c) Reconstructed images in VIS using the combined Multi-resolution LBP and their PSNR

(d) Ground truth in VIS

MACHINE VISION GROUP

Multi-resolution LBP (MLBP)

(P=4,R=1) (P=8,R=1) (P=12,R=1.5) (P=16,R=2) (P=24,R=3)

29 9 4

42 29 2

55 15 6

1 0 0

1 0

1 0 0

1 0 0

8 0

32 0 0

1 2 4

8 16

32 64 128

LBP=1+8+32=41

MACHINE VISION GROUP

PNSR

2

10 1010 log 20 logI IMAX MAXPSNR

MSE MSE

1 12

0 0

1( , ) ( , )

w h

i j

MSE I i j I i jwh

2

1 21 ( , )MSE H H

2

10 10

110 log 20 logIMAX

PSNRMSE MSE

Pixel wise

LBP

MACHINE VISION GROUP

Multi-resolution LBP

, , ,( )p s p s cf

1 2 1, 2,1

( , ) min( , )L

i ii

H H H H

1

0

kk K

pp

3 2,, ,

ki j k i j

Looking up

Ddictionary of face patches and their LBP

histograms

1

, , , , ,0

( )C

p s p s c p s cc

f

MACHINE VISION GROUP

PSNR on MLBP

0 2 4 6 8 100

5

10

15

20

25

30

35

40

24. 4526. 52

29. 77

35. 13

26. 45

29. 2329. 2929. 7729. 6427. 92

PS

NR

CS

LBP

PSNR for combining multi-resolution LBP by different methods

Sum

Produ

ctM

axM

in

Med

ianM

ean

1-LBP-C

S

1-LBP

Conca

tenati

onPCA

MACHINE VISION GROUP

Plans for next period

• Training data:– Use more samples (192*10 from CASIA, a group in Beijing,

China)

• Methods:– Combine the methods proposed in the paper (A.

Hertzmann, SIGGRAPH, 2001) for better performance