sequential importance sampling for estimating expectations ...
Importance sampling Framework at MPC
description
Transcript of Importance sampling Framework at MPC
![Page 1: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/1.jpg)
Importance Sampling
Framework at MPC Guillaume Francois
Kraken character from Clash of the Titans. © Warner Bros
![Page 2: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/2.jpg)
Importance Sampling
Framework at MPC
• Lighting solution for movies such as
– Clash of the Titans
– Harry Potter and the Deathly Hallows
– The Chronicles of Narnia: The voyage of the Dawn Treader
• Image Based Lighting using Importance Sampling
became a standard solution in a majority of shots of
Clash of the Titans
![Page 3: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/3.jpg)
![Page 4: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/4.jpg)
Image Based Lighting Constraints
• Energy Conserving BRDF and Albedo Pump-up
• Adjusting With Number of Samples
• Raytracing heavy scenes such as the the ones
in Clash of the Titans
• Extras
![Page 5: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/5.jpg)
Energy Conserving BRDF
• Why?
– Avoid or reduce light leakage / Edge darkening
– Ensure that independently lookdeved objects work
together
![Page 6: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/6.jpg)
Energy conserving BRDF
• Different BRDF tests:
– Ward Brdf [War92][Wal05]
• Anisotropic gaussian shaped Brdf, visible edge darkening
– Ashikhmin-Shirley Brdf [AS00]
• Anisotropic phong-like Brdf, small yet visible edge darkening
– Halfway Vector Disk Brdf [EBJ06]
• Designed to be energy conserving, anisotropic, no edge
darkening, straightforward retroreflection
![Page 7: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/7.jpg)
Energy conserving BRDF
• Comparison
Ward Ashikmin-Shirley Halfway Vector Disk
![Page 8: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/8.jpg)
Energy Conserving BRDF
• Ashikhmin-Shirley Brdf
– Well known Phong-like behaviour
– Anisotropic
– No mirror reflections at grazing angles like the
Halfway Vector Disk Brdf
![Page 9: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/9.jpg)
Albedo Pump up
• How to make Ashikhmin-Shirley Brdf “more” energy conservative?
– Analytical Albedo pump up [Neu99]
– One simple method: multiply by a precomputed factor
Factor(ωo,roughness) = 1 / ∫Ω f(ωo, ωi, roughness) cos(θi)dωi
![Page 10: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/10.jpg)
Albedo Pump up
• This pump up reduced the quality of the Filtered
Importance Sampling technique, overestimating
the filtering kernel for low pdf directions
• Replace the pdf in the mipmap equation by:
pow(pdf, α) with α<1
![Page 11: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/11.jpg)
Importance Sampling = Number of
Samples
• Not very intuitive
• Very important to adjust these numbers to save
memory and time (or even just to get the renders
to finish in some cases)
• Needs of tools or automatic setup methods to
simplify the life of the technical directors
![Page 12: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/12.jpg)
Adjusting with Number of Samples
• Low number of samples
needed to estimate the
reflection of glossy
objects
• Large number of samples
needed for rough
materials
![Page 13: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/13.jpg)
Adjusting with Number of Samples
• Non linear influence of
the number of samples
on the picture quality
• Automatic solution:
remapping of the number
of samples
![Page 14: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/14.jpg)
Adjusting with Number of Samples
• Let the TDs find the best adjustement?
• If the specular roughness varies over the surface
it becomes impossible to manually control and
optimize the number of samples
• Solution: Experimental mapping of the number
of samples
![Page 15: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/15.jpg)
• Depends on chosen brdf / level of quality needed
– No trivial derivation of the number of samples while
looking at the brdf equations
– Render references spheres with varying roughness for
varying environments and pick the lower number of
samples that gives us an acceptable result
Use tabulated number of samples for each type of brdf
(overall quite similar number of samples)
Experimental mapping of the
Number of Samples
![Page 16: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/16.jpg)
Experimental mapping of the
Number of Samples
Images from debevec.org
![Page 17: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/17.jpg)
• Automatic remapping of the number of samples
depending on the specular roughness (2-64 samples)
Kraken character from Clash of the Titans. © The Moving Picture Company
Experimental mapping of the
Number of Samples
![Page 18: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/18.jpg)
Kraken character from Clash of the Titans. © The Moving Picture Company
Experimental mapping of the
Number of Samples
![Page 19: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/19.jpg)
• Limitations:
– Depends on the IBL used to define the automatic number of
samples
– Not accounting occlusion or reflections from objects
• Possible solutions:
– Get some occluders in the remapping experimentations?
– Have a dedicated UI to give extra control to the TDs
Experimental mapping of the
Number of Samples
![Page 20: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/20.jpg)
• Raytracing is memory and computationally prohibitive
• Solution: trace less rays!
Camera
Adjusting the Number of Traced Rays
![Page 21: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/21.jpg)
• Project the raytraced visibility into an appropriate basis
• Read the visibility function for new rays
Camera
Adjusting the Number of Traced Rays
![Page 22: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/22.jpg)
• Interpolating in between raytraced rays
– Spherical or Zonal Harmonics
• Works well but needs fast projection and reconstruction
• Memory needs to be well managed
• SH in production: Nvidia/Weta Pantaray [PFHA10]
– Our trick
• Very simple and intuitive basis: Sort the raytraced rays by
quadrants
Adjusting the Number of Traced Rays
![Page 23: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/23.jpg)
Adjusting the Number of Traced Rays
• Sort the raytraced rays by quadrants
– Quadrants around the direction of the mirror reflection
ωr
– Build T and B so that (B,T, ωr) is an orthonormal basis
![Page 24: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/24.jpg)
Adjusting the Number of Traced Rays
• Simple Basis:
B1(ωi)=(α-1)(β-1)/4
B2(ωi)=-(α+1)(β-1)/4
B3(ωi)=(α+1)(β+1)/4
B4(ωi)=-(α-1)(β+1)/4
with
α=sign(ωi.T)
β=sign(ωi.B)
B
T
ωr
![Page 25: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/25.jpg)
Adjusting the Number of Traced Rays
• Occlusion of a quadrant K:
Occk = ∑N Occ(ωi)Bk(ωi)
• For a non traced direction ωj:
Occ(ωj) = Occ1B1(ωj) + Occ2B2(ωj) + Occ3B3(ωj) + Occ4B4(ωj)
![Page 26: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/26.jpg)
Adjusting the Number of Traced Rays
Algorithm: vector randomVector = normalize(vector(random(),random(),random()));
vector T = randomVector^wr ;
vector B = wr^T;
color visibilityQuadrant[4] = {0,0,0,0};
int nSampleQuadrant[4] = {0,0,0,0};
int inQuadrant[4] = {0,0,0,0};
//Raytraced samples
for( int i=0; i<nRaytracedSamples; i+=1 ){
alpha = sign(wi[i].T); beta = sign(wi[i].B);
visibility = rayTraceVisibility(wi[i]);
inQuadrant[0] = 0.25*(alpha-1)*(beta-1);
inQuadrant[1] = -0.25*(alpha+1)*(beta-1);
inQuadrant[2] = 0.25*(alpha+1)*(beta+1);
inQuadrant[3] = -0.25*(alpha-1)*(beta+1);
for( j=0; j<4; j+=1 ){
nSampleQuadrant[j] += inQuadrant[j]; //save the number of raytraced rays for normalization
visibilityQuadrant[j] += inQuadrant[j]*visibility;
}
//do the other computations…
}
//Normalize values within each quadrants
for(int j = 0; j<4; j+=1 ){
if(nSampleQuadrant[j]>0){
visibilityQuadrant[j] /= nSampleQuadrant[j];
}
else{
…
}
}
//Non raytraced samples
for( int i=nRaytracedSamples; i<nSamples; i+=1 ){
alpha = sign(wi[i].T);
beta = sign(wi[i].B);
color visibility = visibilityQuadrant[0]*0.25*(alpha-1)*(beta-1)
+ visibilityQuadrant[1]*-0.25*(alpha+1)*(beta-1)
+ visibilityQuadrant[2]*0.25*(alpha+1)*(beta+1)
+ visibilityQuadrant[3]*-0.25*(alpha-1)*(beta+1);
//do the other computations...
}
![Page 27: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/27.jpg)
Adjusting the Number of Traced Rays
• Images rendered with 64 samples for the environment lookup
100% of traced rays for
occlusion computation
60% of traced rays for
occlusion computation
Difference
x5
![Page 28: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/28.jpg)
Raytracing Strategies in Clash of the
Titans
• Shading and lighting scenario:
– Large city
– Large Creature
– Plenty of reflections
• No miraculous recipe: combination of multiple
techniques to reduce memory, optimize
raytracing, avoid recomputations when possible
![Page 29: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/29.jpg)
Raytracing Strategies in Clash of the
Titans
• Cache Occlusion and IS intermediaries
• Use of multiple IBL
– Give the ability to the TDs to add or separate their IBL into
multiple ones for better control of their lighting without massive
additional cost
– ~6 IBL used in Clash of the Titans + Reflection cards
• Solution: Store an array of directions at the surface
shader level and enhance it when needed
![Page 30: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/30.jpg)
Occlusion Caching
Camera
• Sample the Brdf
![Page 31: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/31.jpg)
Occlusion Caching
Camera
• Randomly choose traceable samples
![Page 32: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/32.jpg)
• Estimate 1st IBL
• Trace if needed
Occlusion Caching
Camera
? ? ?
![Page 33: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/33.jpg)
• Trace if needed
• Estimate 2nd IBL
Occlusion Caching
Camera
? ? ?
![Page 34: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/34.jpg)
• Estimate Reflection cards
Occlusion Caching
Camera
![Page 35: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/35.jpg)
Raytracing Strategies in Clash of the
Titans
• Diffuse estimation with Importance Sampling is
expensive
• Irradiance Caching [SRKG07][Chr09]
– Image Based Lighting of the city of Argos
• Many samples for accurate result
– Bake the diffuse contribution for key frames in the
sequence and merge the computed point clouds
• Renderman based ptc color bleeding
![Page 36: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/36.jpg)
Irradiance Caching
![Page 37: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/37.jpg)
Irradiance Caching
• Ptc used for the city illumination itself but also for the
reflections of the city onto the Kraken and Water
![Page 38: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/38.jpg)
Raytracing Strategies in Clash of the
Titans
• Trace Distance and Trace Sets
Infinite Trace Distance – 3 hours 60 meters – 32 minutes
City Rendering with diffuse computed by cosine weighted
importance sampling of a Lambertian - 512 samples per pixel
![Page 39: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/39.jpg)
Raytracing Strategies in Clash of the
Titans
• Trace Distance and Trace Sets
– Second / Third Trace Sets …
2
1
![Page 40: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/40.jpg)
Kraken and Argos city from Clash of the Titans. © Warner Bros
Raytracing Strategies in Clash of the
Titans
![Page 41: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/41.jpg)
Raytracing
• Raytracing within Pixar RenderMan
• Stay at the primitive level as much as possible
• Occlusion just need the hit information
• “Raytrace reflection” vs “Raytrace reflection from bake”
• Raytrace reflection execute the shader at each hit points
(Even with degraded number of samples for secondary rays it costs a lot)
• Raytrace reflection from bake
Usually these ptc are already computed for GI
Gather the hit point and normal to lookup within point clouds
Drawback: Specular contribution of reflected objects either not taken into
account or incorrect (but believable [SRKG07])
![Page 42: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/42.jpg)
Raytracing Strategies in Clash of the
Titans
• Raytracing Proxy Objects
– Shooting from/against polycages
• High res: Kraken ~ 8M polygons
• Low res ~ 2M polygons
• No raytracing of displaced surfaces!
• Careful if you compute anisotropic Brdf based on the surface
derivatives!
![Page 43: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/43.jpg)
Raytracing Strategies in Clash of the
Titans
• Raytracing Proxy Objects
High Resolution Geometry
Displaced Geometry
Poly Cage of low Resolution Geometry
![Page 44: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/44.jpg)
Raytracing Strategies in Clash of the
Titans
• Effective ray origin [TL04]
– If the ray hits before a
given offset : new origin
P
![Page 45: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/45.jpg)
• Primitive Variable Pcage
Raytracing Strategies in Clash of the
Titans
Pcage
P
![Page 46: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/46.jpg)
Clash of the Titans - Kraken
Kraken character from Clash of the Titans. © The Moving Picture Company
![Page 47: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/47.jpg)
Clash of the Titans - Kraken
Kraken character from Clash of the Titans. © The Moving Picture Company
![Page 48: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/48.jpg)
Clash of the Titans - Kraken
Kraken character from Clash of the Titans. © The Moving Picture Company
![Page 49: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/49.jpg)
![Page 50: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/50.jpg)
A few more things
• Banding artifacts reduction
• Iridescence and IS
![Page 51: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/51.jpg)
Sampling Strategies
• Many different sampling techniques [KK02]
– Jittered
– N-rooks
– Hammersley sequence
– …
• FIS uses Hammersley
– How to reduce the banding artifacts?
![Page 52: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/52.jpg)
Sampling Strategies
Camera
Camera
![Page 53: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/53.jpg)
Sampling Strategies
FIS and Hammersley
sampling with no
occlusion
IS and N-rook sampling
with ray-traced
occlusion
FIS and Hammersley
sampling with ray-
traced occlusion
FIS and Hammersley
shifted sampling with
ray-traced occlusion
![Page 54: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/54.jpg)
Quick Implementation of Iridescence
and Color Shifts with FIS
![Page 55: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/55.jpg)
Quick Implementation of Iridescence
and Color Shifts with FIS
θi
• Straightforward filtering of ramps
![Page 56: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/56.jpg)
Aknowledgements
• Anders Langlands, Pascal Gautron, Jean Colas
Prunier, Peter Seager, Douglas Mc Hale,
Sheldon Stopsack, Adrien Saint Girons, all my
former colleagues from MPC
![Page 57: Importance sampling Framework at MPC](https://reader033.fdocuments.in/reader033/viewer/2022052704/577c7ffe1a28abe054a6dc5a/html5/thumbnails/57.jpg)
Bibliography
• [AS00] - Michael Ashikhmin and Peter Shirley - An Anisotropic Phong BRDF Model
• [Wal05] - Bruce Walter - Notes on the Ward BRDF
• [EBJ06] - Dave Edwards, Solomon Boulos, Jared Johnson, Peter Shirley, Michael Ashikhmin,
Michael Stark and Chris Wyman - The halfway vector disk for BRDF modeling
• [Neu99] - László Neumann, Attila Neumann and László Szirmay-Kalos -Reflectance Models by
Pumping up the Albedo Function
• [PFHA10] - Jacopo Pantaleoni, Luca Fascione, Martin Hill and Timo Aila – Pantaray: Fast Ray-
traced Occlusion Caching of Massive Scenes
• [SRKG07] - Apurva Shah, Justin Ritter, Chris King and Stefan Gronsky - Fast, soft reflections
using radiance caches
• [Chr09] - Per Christensen - Point Based Approximate Color Bleeding
• [TL04] - Eric Tabellion and Arnauld Lamorlette - An approximate global illumination system for
computer generated films
• [KK02] - Thomas Kollig and Alexander Keller - Efficient Multidimensional Sampling