Radiometric Compensation through Inverse ... - MIT Media...

Radiometric Compensation through Inverse Light Transport 1|30Wetzstein and Bimber

Radiometric Compensation through Inverse Light Transport

Gordon Wetzstein and Oliver Bimber

Pacific Graphics 2007

contact: [email protected], [email protected]


original image observed projection

SmartProjector - no Screens required!


live-stage performances

SmartProjector - Applications

museums

cultural heritage sitesarchitectural visualization

outdoor advertisement car interiorair plane cabin


SmartProjector - Limitations

no direct mapping: refractions no direct mapping: inter-reflections


Related Work

[Yang et al. 2005]

unconventional projections[no screens, HDR, high-speed, super-resolution]

state-of-the-art report: Bimber et al. „The Visual Computing of Projector-Camera Systems“, EG 2007

tiled screen calibration [geometric correction and luminance matching]

book: Majumder and Brown „Practical Multi-Projector Display Design“, AK Peters 2007

image-based relighting, environment matting and dual photography [forward light transport acquisition and relighting]

[Debevec et al. 2000], [Masselus et al. 2003], [Sen et al. 2005], [Zonker et al. 1999]

inverse illumination [indirect light removal for photography and projection]

[Seitz et al. 2005], [Bimber et al. 2006]

focus related projector-camera techniques [image sharpening for defocused projections]

[Bimber and Emmerling 2006], [Zhang and Nayar 2006], [Brown et al. 2006]


The 8D Reflectance Field

?

),,,( φϕvufLF =


Forward Light Transport

( ) ( ) ( ) ( ) '',',,,,~

wdwxLwwxTwxLwxL ieo ∫Ω+=

( ) ( ) ( ) ( )∑+=j jijiieio wLwxTxLxL ,

0 ( 1)0 0 0 0 0

0 ( 1)( 1) ( 1) ( 1) ( 1) ( 1)

pq

pqmn mn mn pq mn

c T p e

c t t p e

c t t p e

λ λ λ λ

λ λ λ λ λ

λ λ λ λ λ

−

−− − − − −

= +

⎡ ⎤⎡ ⎤ ⎡ ⎤ ⎡ ⎤⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥= +⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎢ ⎥⎢ ⎥ ⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦ ⎣ ⎦⎣ ⎦

LM M O M M M

L

eoiL ,, incoming, outgoing, emissivelight field

~T transport function

T discrete transport function

wx , points in space / discretesamples

', ww directions

cλ camera image eλ environment light in camera space

pλ projected lightλT light transport matrix

λ color channelmn camera resolution pq projector resolution

radiometric compensation?


Light Transport Acquisition

m

n

q

p

mn x 1

pq x 1

C

P

pq

mnc = Tp



m

n

q

p

mn x 1

pq x 1

C

P

pq

mnc = Tp



projected patterns camera image

video clip


Dual Photography

m

n

q

p

mn x 1

pq x 1

P’

C’

pq

mn

c’ = TTp’

c = Tp pq

mn

T

’

’

[Sen et al. 2005]

interchange camera and projector by transposing the light transport matrix


dual

light transport matrix T

Dual Photography

composition

illumination pattern

illuminated composition

illuminated dual

Tpc =

'' cTp T=


Generalized Radiometric Compensation

R G BR R R R G R B R

R G BG G R G G G B G

R G BB B R B G B B B

c T p T p T p e

c T p T p T p e

c T p T p T p e

= + + +

= + + +

= + + +

R G BR R R R R R

R G BG G G G G G

R G BB B B B B B

c e T T T pc e T T T pc e T T T p

⎡ ⎤−⎡ ⎤ ⎡ ⎤⎢ ⎥⎢ ⎥ ⎢ ⎥− = ⎢ ⎥⎢ ⎥ ⎢ ⎥⎢ ⎥⎢ ⎥ ⎢ ⎥−⎣ ⎦ ⎣ ⎦⎣ ⎦

c T p eλ λ λ λ= +

0 0 0 ( 1) 00 0 0 0 0 0

0 0 0 ( 1) 00 0 0 0 0 0

0 0 0 ( 1) 00 0 0 0 0 0

0 0 0 ( 1)( 1) ( 1) ( 1) ( 1) ( 1) ( 1)

R G B k BR R R R R R R

R G B k BG G G G G G G

R G B k BB B B B B B B

R G B k Br B r B r B r B r B r B

c e T T T T pc e T T T T pc e T T T T p

c e T T T T

−

−

−

−− − − − − −

− ⎡ ⎤⎡ ⎤⎢ ⎥⎢ ⎥− ⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥− =⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥−⎣ ⎦ ⎣ ⎦

LLL

M M M M O ML ( 1)k

Bp−

⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦

M

single camera, single projector

general setup with r cameras and k projectors

solve with iterative non-negative least squares


Diffuse Scattering and Inter-Reflections


Diffuse Scattering and Inter-Reflections

[“The

Chu

bbC

hubb

s“, P

ixar

]shadows cannot be compensated with single projector


Defocus Compensation

original uncompensated

compensation compensated


Multi-projector compensation

left rightleft + right


Interactive Compensation on the GPU

reformulate problem for GPU optimized implementation

pre-processing: compute inverse light transporton-line matrix-vector multiplicationSVD:

λλλλ epTc +=

( )

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−

−

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=−

−−−

+

−−

−

−

+

)1(

0

)1()1(

00

)1()1(

)1(0

0)1(

00

pqmnmnpqmn

pq

mn

p

p

ec

ec

tt

ttpecT

λ

λ

λλ

λλ

λλ

λλ

λλλλ

ΜΜΛ

ΜΟΜΛ

,T TT U V T V U+ += Σ = Σ


Sample Light Transport

composition

dual


Cluster Decomposition


Compensation Results

[“9“,

Focu

s Fe

atur

es a

nd 9

, LLC

]


Projecting on Refractive Material



light transport matrixpseudo-inverse



[“Mike‘s New Car“, Pixar]


Interactive Compensation on GPU

[“Mike‘s New Car“, Pixar]30 fps, GeForce 7900 GTX

video clip


Summarygeneralized theory of radiometric compensation using inverse light transport

proof-of-concept:– diffuse scattering and inter-reflections– reflecting statuette– refracting glass– defocus compensation– multiple overlapping projector

interactive compensation on the GPU


Limitations

projection hardware

physical setup

computational resources

– resolution– black level– brightness | contrast– depth of focus

– environment light– projection surface

– matrix sparsity


Outlookview-dependent compensation [Bimber et al. 2005]

incremental inverse light transport acquisition (possibly direct-indirect separation) [Nayar et al. 2006]

novel transport acquisition|storage|processing schemes [Garg et al. 2006]


Thank you!Questions?

www.cs.ubc.ca/~wetzste1www.uni-weimar.de/medien/AR


Related WorkSeamless Multi-Projections

Inverse Illumination

Forward Light Transport, BRDF Acquisition and Relighting

Focus Related Projector-Camera Techniques

Seitz, S., Matsushita, Y., Kutulakos, K. A Theory of Inverse Light Transport. ICCV, 2005

Ashdown, M., Okabe, T., Sato, I., Sato, Y. Robust Content-Dependent Photometric Projector Compensation, ProCams 2006

Bimber, O., Grundhöfer, A., Zeidler, T., Danch, D., Kapakos, P. Compensating Indirect Scattering for Immersive and Semi-ImmersiveProjection Displays. IEEE VR, 2006

Bimber, O., Emmerling, A. Multi-Focal Projection: A Multi-Projector Technique for Increased Focal Depth. IEEE TVCG, 2006

Levoy, M., Chen, B., Vaish, V., Horowitz, M., McDowall, I., Bolas, M. Synthetic Aperture Confocal Imaging. SIGGRAPH 04

Zhang, L., Nayar, S. Projection Defocus Analysis for Scene Capture and Image Display. SIGGRAPH 06

Masselus, V., Peers, P., Dutré, P, Willems, Y. Relighting with 4D incident Light Fields. ACM TOGS 2003

Goesele, M., Lensch. H., Lang, J., Fuchs, C., Seidel, H. DISCO: Acquisition of Translucent Objects. SIGGRAPH 04

Peers, P., vom Berge, K., Matusik, W., Ramamoorthi, R., Lawrence, J., Rusinkiewicz, S., Dutré, P. A Compact Factored Representation of Heterogeneous Subsurface Scattering. SIGGRAPH 2006Sen, P., Chen, B., Garg, G, Marschner, S, Horowitz, M., Levoy, M., Lensch, H. Dual Photography. SIGGRAPH 05

Debevec, P., Hawkins, T., Tchou, C., Duiker, H., Sarokin, W., Sagar, M. Acquiring the Reflectance Field of a Human Face. SIGGRAPH 00

Yang, R., Majumder, A, Brown, M. Camera Based Calibration Techniques for Seamless Multi-Projector Displays. ACM TOGS. 2005

Nayar, S. Peri, H., Grossberg, M., Belhumeur, P. A Projection System with Radiometric Compensation for Screen Imperfections. ProCams 2003Bimber, O., Emmerling, A., Klemmer, T. Embedded Entertainment with Smart Projectors. IEEE Computer, 2005Bimber, O., Iwai, D., Wetzstein, G., Grundhöfer, A. The Visual Computing of Projector-Camera Systems. EuroGraphics (STAR) 2007Fuji, K., Grossberg, M., Nayar, S. A Projector-Camera System with Real-Time Photometric Adaptation for Dynamic Environments. IEEE CVPR 2005

Grossberg, M., Peri, H., Nayar, S. Making one Object Look Like Another: Controlling Appearance using a Projector-Camera System. IEEE CVPR 2004

Bimber, O., Wetzstein, G., Emmerling, A., Nitschke, C. Enabling View-Dependent Stereoscopic Projection in Real Environments. ISMAR 05

Grundhöfer, A., Bimber, O. Real-Time Adaptive Radiometric Compensation. IEEE Transactions on Visualization and Computer Graphics, to appear

Brown, M., Song, P., Cham, T. Image Pre-Conditioning for Out-of-Focus Projector Blur. CVPR 06

Bimber, O., Iwai, D., Wetzstein, G., Grundhöfer, A., The Visual Computing of Projector-Camera Systems. EuroGraphics state-of-the-art report 2007

Majumder, A, Brown, M. Practical Multi-Projector Display Design. AK Peters 2007


Comparison CPU - GPU

pseudo-inverse is less stable than solving explicitly

however, no visible difference


Error Analysis

Radiometric Compensation through Inverse ... - MIT Media...

Documents

Transcript of Radiometric Compensation through Inverse ... - MIT Media...