Exploiting Temporal Coherence for Incremental All-Frequency Relighting

Ryan Overbeck Ravi Ramamoorthi

Aner Ben-Artzi Eitan Grinspun

Columbia University

Ng et al. 2003 Our Method

30 Wavelet lights per frame

CG Lighting Design

Doom 3 2004 (

www.doom3.com)

Unreal Championship 2 2004 (

www.unrealchampionship2.com)

The Lord of the Rings: The Two Towers 2002 (www.lordoftherings.net)

Star Wars Episode I 1999 (

thecia.com.au/reviews/s/star-wars-1.shtml)

Video Games Movies

CG Lighting Design: Why is it hard?

Complex Lighting

Complex Materials

Takes Hours to Render

Need Interactivity

PRT

Sloan et al. 2002

Ng et al. 2003

Hard Shadows

Soft Shadows

Specularities

/ Reflections

Caustics

PRT Relighting: Matrix-Vector Multiply

1

2

3

N

P

P

P

P

11 12 11

21 22 22

31 32 3

1 2

M

M

M

NN N NM

T T TL

T T TL

T T T

LT T T

Slides from Ng et al. SIGGRAPH 2003

PRT Relighting: Matrix-Vector Multiply

1

2

3

N

P

P

P

P

11 12 11

21 22 22

31 32 3

1 2

M

M

M

NN N NM

T T TL

T T TL

T T T

LT T T

Input Lighting(Cubemap Vector)

Output Image(Pixel Vector)

TransportMatrix

Slides from Ng et al. SIGGRAPH 2003

Light-Transport Matrix Columns

11 12 1

21 22 2

31 32 3

1 2

M

M

M

N N NM

T T T

T T T

T T T

T T T

Slides from Ng et al. SIGGRAPH 2003

Light-Transport Matrix Columns

11 12 1

21 22 2

31 32 3

1 2

M

M

M

N N NM

T T T

T T T

T T T

T T T

11 12 1

21 22 2

31 32 3

1 2

M

M

M

N N NM

T T T

T T T

T T T

T T T

11 12 1

21 22 2

31 32 3

1 2

M

M

M

N N NM

T T T

T T T

T T T

T T T

Slides from Ng et al. SIGGRAPH 2003

Matrix Multiplication is Enormous

Dimension

512 x 512 pixel images ( )

6 x 64 x 64 cubemap ( )

Full matrix-vector multiplication is intractable

On the order of 1010 operations per frame

PRT exploits coherence to enable real-time rendering

B T L

LB

Slides from Ng et al. SIGGRAPH 2003

L~

Signal / Spatial Coherence[Sloan et al. 2003]

[Liu et al. 2004]

PRT: Exploiting Coherence

Image / Vertex Colors

Transport Matrix Lighting Vector

Angular Coherence[Ng et al. 2003]

30 – 100 Wavelet Lights

Temporal Coherence[Our Contribution]B T L

Previous Work: PRT

Dorsey, J., Arvo, J., and Greenberg, D. 1995. Interactive Design of Complex Time-Dependent Lighting. In IEEE Computer Graphics and Applications, 15(2): 26-36.

Ng R., Ramamoorthi R., Hanrahan P. All-frequency shadows using non-linear wavelet lighting approximation. ACM TOG(SIGGRAPH 03) 22, 3 (2003), 376-381.

Sloan, P., Kautz, J., Snyder, J. Precomputed Radiance Transfer for Real-Time Rendering in Dynamic, Low-Frequency Environments. In Proceedings of SIGGRAPH 2002

Ng R., Ramamoorthi R., Hanrahan P. Triple product wavelet integrals for all-frequency relighting. ACM TOG (SIGGRAPH 04) 23, 3 (2004), 475-485.

Sloan P., Hall J., Hart. J, Snyder J. Clustered principal components for precomputed radiance transfer. ACM TOG (SIGGRAPH 03) 22, 3 (2003), 382-391.

Sloan P., Luna B., Snyder J. Local, deformable precomputed radiance transfer. ACM TOG (SIGGRAPH 05) 24, 4 (2005), 1216-1224.

Wang R., Tran J., Luebke D. All-frequency relighting of non-diffuse objects using separable BRDF approximation. In EGSR (2004), pp.345-354.

Wang R., Tran J., Luebke D. All-frequency interactive relighting of translucent objects with single and multiple scattering. ACM TOG (SIGGRAPH 05) 24, 3 (2005), 1202-1207.

Zhou K., Hu Y., Lin S., Guo B., Shum H. Precomputed shadow fields for dynamic scenes. ACM TOG (SIGGRAPH 05) 25, 3 (2005).

Ben-Artzi A., Overbeck R., Ramamoorthi R. Real-time BRDF editing in complex lighting. ACM TOG (SIGGRAPH 06) (2006).

Wang R., Luebke D., Humphreys G., Ng R. Efficient wavelet rotation for environment map rendering. In EGSR (2006).

Kontkanen J., Turquin E., Holzschuch N., Sillion F. Wavelet radiance transport for real-time indirect lighting. In EGSR (2006)

Einarsson P., Chabert C., Jones A., Lamond B., Ma A., Hawkins T., Sylwan S., Debevec P. Relighting human locomotion with flowed

reflectance fields. In EGSR (2006)

This Session

Previous Work: PRT

Dorsey, J., Sillion, F., and Greenberg, D. 1991. Design and simulation of opera lighting and projection effects. In Computer Graphics (Proceedings of SIGGRAPH 91), vol. 25, 41-50.

Ng R., Ramamoorthi R., Hanrahan P. All-frequency shadows using non-linear wavelet lighting approximation. ACM TOG(SIGGRAPH 03) 22, 3 (2003), 376-381.

Sloan, P., Kautz, J., Snyder, J. Precomputed Radiance Transfer for Real-Time Rendering in Dynamic, Low-Frequency Environments. In Proceedings of SIGGRAPH 2002

Ng R., Ramamoorthi R., Hanrahan P. Triple product wavelet integrals for all-frequency relighting. ACM TOG (SIGGRAPH 04) 23, 3 (2004), 475-485.

Sloan P., Hall J., Hart. J, Snyder J. Clustered principal components for precomputed radiance transfer. ACM TOG (SIGGRAPH 03) 22, 3 (2003), 382-391.

Sloan P., Luna B., Snyder J. Local, deformable precomputed radiance transfer. ACM TOG (SIGGRAPH 05) 24, 4 (2005), 1216-1224.

Wang R., Tran J., Luebke D. All-frequency relighting of non-diffuse objects using separable BRDF approximation. In EGSR (2004), pp.345-354.

Wang R., Tran J., Luebke D. All-frequency interactive relighting of translucent objects with single and multiple scattering. ACM TOG (SIGGRAPH 05) 24, 3 (2005), 1202-1207.

Zhou K., Hu Y., Lin S., Guo B., Shum H. Precomputed shadow fields for dynamic scenes. ACM TOG (SIGGRAPH 05) 25, 3 (2005).

Ben-Artzi A., Overbeck R., Ramamoorthi R. Real-time BRDF editing in complex lighting. ACM TOG (SIGGRAPH 06) (2006).

Wang R., Luebke D., Humphreys G., Ng R. Efficient wavelet rotation for environment map rendering. In EGSR (2006).

Kontkanen J., Turquin E., Holzschuch N., Sillion F. Wavelet radiance transport for real-time indirect lighting. In EGSR (2006)

Einarsson P., Chabert C., Jones A., Lamond B., Ma A., Hawkins T., Sylwan S., Debevec P. Relighting human locomotion with flowed

reflectance fields. In EGSR (2006)

This Session

L~ Temporal Coherence

[Our Contribution]B T

Our Method

Outline

Motivation / Previous Work

Basic Approach to Incremental Relighting

Analysis of Temporal Coherence in Lighting

Per-Band Incremental

Results

ΔB ΔL

Basic Incremental Algorithm

B T LtB T tL

1tB T 1tL

B

Basic Incremental Algorithm

ΔB ΔL TT L

B

Basic Incremental Algorithm

ΔLTΔL tL 1tL~More

Compressible

(Approx

B

Basic Incremental Algorithm

ΔLTΔL tL 1tL~ )~

~

Basic Incremental

B T ΔL~

Basic Incremental: Problems

Reference Incremental

Frame 0

Basic Incremental: Problems

Reference Incremental

Frame 30

Basic Incremental: Problems

Reference Incremental

Frame 75

Ghost Shadows

Basic Incremental: Problems

Reference Incremental

Frame 125

Basic Incremental: Problems

Reference Incremental

Frame 400

Outline

Motivation / Previous Work

Basic Approach for Incremental Relighting

Analysis of Temporal Coherence in Lighting

Per-Band Incremental

Results

Medium FrequencyLow Frequency

2tL

Frequency Analysis of Temporal Coherence

1tL

3tL

High Frequency

Medium FrequencyLow Frequency

2tL

Frequency Analysis of Temporal Coherence

1tL

3tL

High Frequency

Medium FrequencyLow Frequency

Frequency Analysis of Temporal Coherence

High Frequency

1t

2t

3t

Low Frequency Medium Frequency

Frequency Analysis of Temporal Coherence

High Frequency

1t

2t

3t

2t 1t

Low Frequency Medium Frequency

Frequency Analysis of Temporal Coherence

High Frequency

1t

2t

3t

2t 1t

Frequency Analysis of Temporal Coherence

Medium FrequencyLow Frequency High Frequency

Tem

pora

l W

avele

t Tra

nsfo

rm

Frequency Analysis of Temporal Coherence

Medium FrequencyLow Frequency High Frequency

Tem

pora

l W

avele

t Tra

nsfo

rm

Frequency Analysis of Temporal Coherence

Angular Frequency

Tem

pora

l Fre

qu

en

cy

100 %

0 %

ENERGY

Outline

Motivation / Previous Work

Basic Approach for Incremental Relighting

Analysis of Temporal Coherence in Lighting

Per-Band Incremental

Results

B = + +

Non-Incremental

B = T L1 1

B = T L22

B = T L3 3

B = T LIncremental+

Incremental+

1B 2B 3B

Per-Band Incremental (PBI)

. . ....

Exhaustive

Try all combinations over all wavelet bands.

Very Slow.

Simple

Compare L1 Error in each band individually.

Oracle (Incremental or Not)

Exhaustive

Try all combinations over all wavelet bands.

Very Slow.

Simple

Compare L1 Error in each band individually.

Oracle (Incremental or Not)

L1 DistanceL1 Distance

IncrementalNon-Incremental

Oracle (Incremental or Not)

L1 DistanceL1 Distance

< or >

IncrementalNon-Incremental

Exhaustive

Try all combinations over all wavelet bands.

Very Slow.

Simple

Compare L1 Error in each band individually.

Almost zero overhead.

Comparable results to Exhaustive.

PBI vs. Basic Incremental

Outline

Motivation / Previous Work

Basic Approach for Incremental Relighting

Analysis of Temporal Coherence in Lighting

Per-Band Incremental

Results

Results

Results

Results

Per-Band Incremental 30 Wavelets16

14

13

12

15

55 70 85Frame

Perc

en

tag

e L

1 E

rror

Simple

Exhaustive

Results

Results

Results

Results

Per-Band Incremental

3x – 4x Speed / Quality Improvement

Progressively Convergent

Minimal Overhead

Minimal Overhead

~ 100 Lines of Code (Pseudo-code in paper).

< 10 % Memory Overheads

Speed:

Average case (30 wavelets): 5 % Overhead

Applies to All (?) Wavelet PRT Frameworks

Old PRT

Standard All-Frequency PRT [Ng et al. 2003]

Current PRT

Clustered PCA [Liu et al. 2004]

Changing View with Separable BRDF Approximation [Wang et al. 2004]

Future PRT

Real-time BRDF Editing [Ben-Artzi et al. SIGGRAPH 2006]

PBI: CPCA and Complex BRDFs

Future

Temporal Coherence

PRT Animation.

Beyond PRT.

B T L

Acknowledgments

Ren Ng for wavelet relighting framework code and slides from SIGGRAPH 2003.

Microsoft / Sloan et al. for Spherical Harmonics relighting framework in DirectX SDK.

This research was funded in part by a Sloan Research Fellowship and NSF grants #0305322 and #0446916.

Londa Fiorella for getting me here.

Thank you!