Reflectance Modeling for Vision and Graphics

1

Reflectance Modelingfor Vision and Graphics

Todd ZicklerHarvard School of Engineering and Applied Sciences

Reflectance Modeling

Acknowledgements

Satya Mallick, UCSD

Sebastian Enrique, EA Sports

Ravi Ramamoorthi, Columbia University

Peter Belhumeur, Columbia University

David Kriegman, UCSD

Jeffrey Ho, UFL

Jean Ponce, UIUC, ENS

Funding: NSF

2


Appearance

f -1( I ) = ?

I = f (shape, reflectance)illumination,


Reflectance: BRDF

n(θi,φi)

fr(θi,φi; θo,φo)

(θo,φo)

Bi-directional Reflectance Distribution Function

3


Typical Assumption: Lambertian Reflectance

LAMBERTIAN:IDEALLY DIFFUSE


Handling Complex Reflectance

1. Ignore (treat as noise)2. Detect and remove3. Model parametrically

Proposed Approach:

Exploit common reflectance phenomena (reciprocity, isotropy, separability,…)

4


♦ Reciprocity:Helmholtz stereopsis– reconstruction

Outline

u

v

θh

2φd

♦ Separability:Color subspaces– reconstruction, recognition, motion

estimation, segmentation

♦ Spatial coherence, compressibility:Reflectance sharing– modeling, appearance capture


i e

n

Helmholtz Reciprocity

ie

n

[Helmholtz 1925; Minnaert 1941; Nicodemus et al. 1977]

)i,e()e,i( rr ff =

5


Stereo vs. Helmholtz Stereo

STEREO HELMHOLTZ STEREO


Stereo vs. Helmholtz Stereo

STEREO HELMHOLTZ STEREO

6


In Practice

HELMHOLTZ STEREO


Reciprocal Images

Specularities “fixed” to surface

el er

Relation between el and er independent of BRDF

7


Reciprocity Constraint

n

vl^ vr

^

x

ol or

=

vl^ vr

^

x

ol or

n

er(x) = fr(vl(x), vr(x))slvl(x) · n(x)|ol − x|2el(x) = fr(vr(x), vl(x))

srvr(x) · n(x)|or − x|2

sr sl



n

vl^ vr

^

x

ol or

vl^ vr

^

x

ol or

n

sr sl

µel(x)

slvl(x)

|ol − x|2 − er(x)srvr(x)

|or − x|2¶· n(x) = 0.

8



µel(x)

slvl(x)

|ol − x|2 − er(x)srvr(x)

|or − x|2¶· n(x) = 0.

Near-field reciprocity constraint:

Far-field reciprocity constraint:

(el(x)slvl − er(x)srvr) · n(x) = 0

[Zickler et al., ECCV 2002]


Example

[Zickler et al., ECCV 2002]

9


Reciprocal Images: Typical Dataset

SOURCEVIEW



SOURCE

VIEW

Conventional Stereo• Constant brightness (Lambertian)• No structure in textureless regions

10



SOURCEVIEW


Photometric Stereo• Needs reflectance model• No direct depth estimates



SOURCE

VIEW


Photometric Stereo• Needs reflectance model• No direct depth estimates

Helmholtz Stereo• No assumed reflectance• Gives depth and surface normals

12


Auto-calibration Strategy

INPUT SPARSE FEATURES

EPIPOLAR GEOMETRY/SPARSE RECONSTRUCTION

DENSE, METRICRECONSTRUCTION

[Zickler, CVPR 2006]


Specular Highlights as Features

INPUT

13


Auto-calibration Example

♦ Five ‘far-field’ cameras/sources♦ Dense reconstruction: [Zickler et al., 2002]

{si} = {0.74, 0.89, 0.89, 0.92, 1}

{si} = {1, 1, 1, 1, 1}

[Zickler, CVPR 2006]


♦ Reciprocity (& isotropy):Helmholtz stereopsis– reconstruction

Outline

u

v

θh

2φd




14


Separability

DIFFUSE

= +

SPECULAR

[Shafer, 1985]

Reflectance Modeling[Shafer, 1985]

λ

E

λ

R

λ

kCDk =

ZE(λ)R(λ)Ck(λ)dλ

IRGB = (n · i)D+ fs(i, e)(n · i)S

Separability: Dichromatic Model

15

Reflectance Modeling[Shafer, 1985]

λ

E

λ

R

λ

kCDk =






[Shafer, 1985]

λ

E

λ

R

λ

kC

Sk =

ZE(λ)Ck(λ)dλ

Dk =



16


Dichromatic Materials

[Tominga and Wandell, 1989; Healey, 1989; Lee et al., 1990]


Explicit Separation

DIFFUSE

= +

SPECULAR

= +σd(u)D(u) σs(u)SIRGB(u)

17


Explicit Separation

DIFFUSE

= +

SPECULAR

[Klinker et al., 1988; Bajscy et al., 1996; Criminisi et al., 2005; Lee and Bajscy, 1992; Lin et al., 2002; Lin and Shum, 2001; Miyazaki et al., 2003; Nayar et al., 1997; Ragheb and Hancock, 2001; Sato and Ikeutchi, 1994; Tan and Ikeutchi, 2005; Wolfe and Boult, 1991;…]

= +σd(u)D(u) σs(u)SIRGB(u)


Observation:Explicit Separation not Required

Gr2r1

S

IRGB

B

R J

IRGB = σdD+ σsS

Jl =< IRGB , rl >= σdr>l D

1. INVARIANT TOSPECULAR REFLECTIONS

2. BEHAVES ‘LAMBERTIAN’

18


Observation:Explicit Decomposition not Required

Gr2r1

S

IRGB

B

RJ

IRGB = σdD+ σsS


IRGB || J ||

[Mallick, Zickler et al., CVPR 2005; Zickler et al., CVPR 2006]


Generalization: Mixed Illumination

Gr2r1

S

IRGB

B

R Jr1

S1

IRGB

B

G

R

S2

J

SINGLE ILLUMINANT MIXED ILLUMINATION

[Mallick, Zickler et al., CVPR 2005; Zickler et al., CVPR 2006]

19


Generalization: Mixed Illumination


Example: Optical Flow

[Algorithm: Black and Anandan, 1993]

Conv

entio

nal

Gra

ysca

le(R

-+G

+B)/3

Spec

ular

In

varia

nt,

||J||

(blu

e ill

umin

ant)

Spec

ular

Inv

aria

nt,

||J||

(blu

e &

yel

low

ill

umin

ants

)

Ground truth flow

20


Example: Binocular Stereo

[Algorithm: Boykov, Veksler and Zabih, CVPR 1998]

Conventional Grayscale(R+G+B)/3

Specular Invariant, ||J||(blue illuminant)

Specular Invariant, ||J||(blue & yellow illuminants)

One image from input stereo pair

Reco

vere

d de

pth


Generalized Hue

Gr2r1

S

IRGB

B

R Jψ

ψ = tan−1(J1/J2) = tan−1(r>1 D/r>2 D)


[Zickler et al., CVPR 2006]

21


Example: Material-based Segmentation

Input image

Conventional Grayscale Specular Invariant ||J||

Conventional Hue Generalized Hue ψ

[Zickler et al., CVPR 2006]


Example: Photometric Stereo

[Mallick, Zickler et al., CVPR 2005]

J behaves ‘Lambertian’→ Linear function of surface normal

Jl =< IRGB , rl >= σdr>l D = (n · i)r>l D

22











23






Outline

u

v

θh

2φd




24


Modeling; Appearance Capture

1. SHAPE

+

2. REFLECTANCE


5º sampling: 1,000,000 images >106 MB1º sampling: 625,000,000 images >109 MB

n

ˆ ˆ( , )xf i er

Spatially-varying BRDF (SBRDF)From Images and Shape

~x

25


ˆ ˆ( ; , )x xf i eαr rr

ˆ ˆ( , )xf i er

1. Parametric BRDF models[Sato, Wheeler, Ikeuchi, 1997][Yu et al., 1999][Boivin, Gagalowicz, 2001][Lensch, et al., 2001] [McAllister, Lastra, Heidrich, 2002][Georghiades, 2003]…

Existing Approaches


ˆ ˆ( , )xf i er


2. Data-Driven (Non-parametric)[Debevec et al., 2000][Wood et al., 2000][Matusik et al., 2002]

Existing Approaches

26




ˆ ˆ( , )xf i er

Subset of reflectance

12,0002000 600# IMAGES:

Existing Approaches




PRO: Sparse Images

Efficient Rendering

CON: Limited Generality

PRO: General BRDFs

CON: Expensive ( , $ )

Cumbersome

Existing Approaches

27




PRO: Sparse ImagesEfficient Rendering

CON: Limited Generality

PRO: General BRDFs

CON: Expensive

Cumbersome

Existing Approaches


Reflectance Sharing: Three BRDF properties

n(θi,φi)

(θo,φo)

fr(θi,φi; θo,φo) −→ fr(θi, θo,φi − φo)

ISOTROPY

28


Spatial coherence Compressibility

Reflectance Sharing: Three BRDF properties

[Zickler et al., T-PAMI 2006]

fr(x, y, θi, θo,φi − φo)


Evaluation: No spatial variation

uv

w

SLO

W

RAPID

(θi, θo,φi − φo) −→ (u, v, w) = ~q

29



uv

w

( )1

( ) ( ) -N

i ii

f q p q q qλψ=

= +∑r r r r%

(θi, θo,φi − φo) −→ (u, v, w) = ~q


uv

w

( )1

( ) ( ) -N

i ii

f q p q q qλψ=

= +∑r r r r%

LAFORTUNE LAMBERTIAN LAFORTUNE WARDRBFACTUAL RBFACTUAL

[Oren, Nayar 2001] [Matusik et al., 2003]


30


Example: Human Face

DIFFUSE SPECULAR

θr

xr

( , , , , )q x y u v w=r

aRGB(x, y)


hθ

2 dφ

Example: Human Face

31


Results: Lighting


Results: Viewpoint

32


Results



Summary

u

v

θh

2φd




33


Some Future Work

♦Ubiquitous appearance capture– Spatial reflectance discontinuities– Interreflections; sub-surface scattering

[Nayar et al., 2004]

www.eecs.harvard.edu/[email protected]

Reflectance Modeling for Vision and Graphics

Documents

Transcript of Reflectance Modeling for Vision and Graphics