Advanced Lighting Techniques Dan Baker (Meltdown 2005)
90
Transcript of Advanced Lighting Techniques Dan Baker (Meltdown 2005)
Advanced Lighting TechniquesDan BakerSoftware Design EngineerMicrosoft Corporation
Review The BRDF• Bidirectional Reflectance
Distribution Function• Ratio of irradiance to
reflected radiance
ωedωi
ωi
The BRDF• Incident, excitant directions defined
in the surface’s local frame (4D function) Nωe
ωi
T
θiθe
φiφe
The BRDF• Isotropic BRDFs are 3D functions
Nωe ωiθi
θe
φ
The Reflection Equation
The Reflection Equation
The BRDF• The laws of physics impose certain constraints
on a BRDF• To be physically plausible, it must be:
• Reciprocal• Energy-Conserving
• You are already using a BRDF, even if you don’t know it• Phong• Blinn-Phong• Diffuse
How Do We Evaluate a Lighting Model?• Real time graphics technology is
based on rasterization• Currently, our pipeline consists of
triangles, which get processed by a vertex shader, and then rasterized
• The rasterization step produces pixels (aka fragments), which are then processed and placed into a rendertarget
A Modern Real Time Graphics System
VertexBuffer
InputAssembler
VertexShader
SetupRasterizer
OutputMerger
PixelShader
IndexBuffer Texture Render
TargetDepthStencilTexture
Memory
memory programmable fixed
Sampler Sampler
Constant Constant
Tomorrows Graphic’s System
VertexBuffer
InputAssembler
VertexShader
SetupRasterizer
OutputMerger
PixelShader
GeometryShader
IndexBuffer Texture Texture Render
TargetDepthStencilTexture Stream
Buffer
Stream out
Memory
memory programmable fixed
Sampler Sampler Sampler
Constant Constant ConstantMemory
Where We Have Control• Shader controls each process point• With some restrictions, can perform
any computation we want• A reflectance model can be evaluated
at any of these stages, or any combination
• Historically, computations were done on vertex levels, but have steadily moved into pixels
• Soon, will be able to shade on a triangle
Frequency of Evaluation• Generally, vertex evaluation is lowest
frequency, while pixel is at a higher frequency
• Not the case for low resolution• Small triangles (sliver) not rendered
by realtime hardware• Geometry aliasing avoided by not
drawing sliver triangles
An Example BRDF• In HLSL, the Blinn-Phong model looks like:
float3 BlinnPhong(float3 L, float3 V){
float3 H = normalize(V + L);float3 C = Ks*pow(dot(H, float3(0,0,1)), Power) + Kd*dot(N,L);
}
Gamma Space• BRDFs operate in linear space, not
gamma space• Most albedo textures are authored in
gamma space, so must convert into linear space in the shader (and convert them back into gamma space)
• Can use SRGB to convert gamma albedo textures into linear space, gives more precision where needed and acts as a compression scheme
Gamma Correcting• If we render straight to the screen, and
backbuffer isn’t linear (usual case), need to go into gamma
• sqrt is a close approximation
float3 BlinnPhong(float3 L, float3 V){
float3 H = normalize(V + L);float3 C = Ks*pow(dot(H, float3(0,0,1)), Power) + Kd*dot(N,L);
C = sqrt(C);}
Dynamic Range• BRDFs can have a wide range of color
intensities• 8 bit per channel backbuffer does poor
job of capturing range• Easy for information to be lost – e.g.
clamped to max value, or not enough precision
Blooming/Tone Mapping• Bright pixels bleed to neighbors• Exposure control • If these are used – BRDF will store
color value stored in rendertarget, and an image space filter applied
• Likely de-facto standard in future
How Do We Evaluate a BRDF?• Direct Evaluation
• Make an ALU program in the GPU• Texture Evaluation
• Perform a texture lookup• A combination of the 2.
• Factor some things into textures• Do others with ALU
How Do We Represent Material Variation?
• Microvariation is important• Want more then just a single
BRDF for a surface, this BRDF must change across the surface
• Bumpmapping is a coarse variation
Which Reflectance Model to Use• Different reflectance models have
different costs• Cost – Data• Cost – ALU• Cost – Time to make content• Cost – Accuracy• Consider: Strong trend toward more
ALU, less data
BRDF Costs, Minimal Surface Variation
Model Texture costs ALU costsBlinn-Phong Direct 0 7Blinn-Phong factored 1 2Banks Direct 0 12Banks Factored 1 5Ashikhmin/Shirley 0 40Ashikhmin/Shirley factored 4 10Lafortune Direct 0 10 + 5 *nCook-Torrance 0 35
BRDF Costs With Surface Variation
Model Texture costs ALU costsBlinn-Phong Direct 1 15Blinn-Phong factored 2 10Banks Direct 1 25Banks Factored 2 18Ashikhmin/Shirley 2 50 (60)*Ashikhmin/Shirley factored 6 30 Lafortune Direct 2 30 +
5*LobesCook-Torrance 1 40
chart courtesy of Addy Ngan, Frédo Durand, and Wojciech Matusik
Accuracy of Blinn-Phong
chart courtesy of Addy Ngan, Frédo Durand, and Wojciech Matusik
Accuracy of Ashikhmin Shirley
Getting Variation• Could sample real materials to
generate a BRDF map [McAllister SBRDF].
• But, variation usually done by an artist.
• Sometimes we get variation by using an understanding of mesogeometry
• Sometimes it is done by making changes in the assumption of the microgeometry
Bump Mapping
Pixel Level Evaluation
Pixel Level Evaluation, Shift
Shifting Samples• The sample points change drastically
as the triangle moves and changes size
• If using micro variation, e.g. loading data elements from a texture, the evaluation can be significantly different
• Texture filters – solves this problem for linear data elements
• Linear for temporal, MIP mapping for resolution changes
But…• Texture filters are linear• BRDFs are usually non linear• Get some image stability, but not
great results• Must at least prevent radical shifts in
image • Often, variation must be mitigated• Would be nice to texel shade instead
What Does a Sample Point Mean?
What happens What happens when this pixel when this pixel is evaluated? is evaluated? Given a set of Given a set of parameters parameters interpolated interpolated from the from the triangles 3 triangles 3 vertices, how vertices, how do we go about do we go about generating a generating a color?color?
Texture Filtering Review
A (1-A)
B
(1-B)
Sample Color =
A*B * +A*(1-B) * +(1-A)*B * +(1-A)*(1-B)* +
Texture swimming
What about lower resolutions?
A common hack: Level of Detail
MIP
Lev
el
By lowering the By lowering the amplitudeamplitudeOf the Of the displacement, displacement, effectively effectively decreasing to a decreasing to a less complex less complex model as the model as the model moves model moves further away. This further away. This will prevent will prevent aliasing, but isn’t aliasing, but isn’t accurateaccurate
Scale independent lighting• Ideally, the size in pixels of an
object on the screen should not affect its overall appearance
• High frequency detail should disappear
• Global effects should stay the same
• Reasoning behind MIP mapping• Shouldn’t see drastic changes in
image as sample points change
A low resolution rendering of an object should look like a scaled down version of the high resolution one
Scale independent lighting
MIP Mappingr
t
Mip mapping for diffuse
• For a simple diffuse case, the lighting calculating can be approximately refactored
• The normal integration can also be substituted by a MIP map aware texture filter
rr
rr
rrrrr
r WAWNLWANL *)()(
),(2*)),(2,()( tcolormapDtextnormalmapDtexLdotWTNL rrr
r
Non Linear Lighting models
prr
r
prr NWHHNW )*()(
p
r
prr tnormalmapDtexHHNW )),(2()(
Blinn-Phong isn’t Linear!Blinn-Phong isn’t Linear!
Texture Space Lighting• Rasterize using texture coordinates
as a position• Equivalent to explicitly evaluating
the summation• An object needs a texture atlas• The values in the texture space are
now colors – and are linear• Create a MIP chain explicitly or
through the auto MIP gen option• Render the object with this MIP chain
Texture Space Lighting
Rasterize triangles using texture coordinates as positions, the left image is the normal sampled at each point, and the right image is the computed lighting
Texture Space lighting
We paste the texture onto the object
TSL: A case study• Texture Space Lighting (TSL) can
allows us to have high frequency lighting on fast moving objects
• Will only have aliasing problems associated with standard filtering techniques
• A roadway is a perfect candidate, subtle specular reflections with a high degree of motion
A roadway, snapshots
We can see Moiré patterns on any filtered non linear functions. Additionally, temporal aliasing becomes problematic.
Using TSL
The artifacts are largely mitigated if we render the non linear function and MIP reduce. We can also see more high frequency detail.
How this demo works• Each road segment is rendered into a
1024x1024 rendertarget in texture space – alpha is set to 0
• AUTOMIPGEN is then used to do a simple mip filter of the highest level rendering
• This model is then light with the prelit, mip filtered texture and rendered with full aniso
• Art content is simple – just an albedo texture, a normal map with a glossy term
• Everything is done in linear space
The Shadervoid lightPositional( …){ // norm, light and half are in tangent space // power, Kd, Ks and norm come from textures … //normalized blinn phong float fS = pow(saturate(dot(norm, half)), power)) * power; float fD = saturate(dot(norm, light)) * diffuseIntensity; vColorOut.xyz = fS * Ks + fD * Kd; vColorOut.a = 1; //put us in gamma space for the direct rendering only if (bTextureView) vColorOut = sqrt(vColorOut); }
Pasting the texture on the Scene void light_from_texture( float2 vTexCoord: TEXCOORD0,
out float4 vColorOut: COLOR0 ){
// this is done with full anisotropy turned on// with SRGBTexture set to truevColorOut = tex2D(LightMapSampler, vTexCoord);
//for atlased objects, we do not want to blend in // unrendered pixels
if( vColorOut.a > 1e-5 )vColorOut.xyz /= vColorOut.a;
//put us in gamma spacevColorOut = sqrt(vColorOut);
}
Drawing polygons on the backside[emittype [triangle MyVType ]][maxvertexcount [3]]void ClipGeometryShader(triangle MyVType TextureTri[3]){ float2 coord1 = project(TextureTri[0]); float2 coord2 = project(TextureTri[1]); float2 coord3 = project(TextureTri[2]); float3 Vec1 = float3(coord3 – coord1, 0); float3 Vec2 = float3(coord3 – coord2, 0); float Sign = cross(Vec1, Vec2).z; if (Sign > -EPSILON) { emit(TextureTri[0]); emit(TextureTri[1]); emit(TextureTri[2]); cut; }}
Problems with TSL• Main problem is performance• We can’t render each object in the
screen at a fixed resolution and scale down
• We need a way to render the object at a reduced resolution which looks close to the higher detail one when zoomed out
• Basically, we need to know how to create non linear MIP chains
MIP-Mapping Reflectance Models
MIP-Mapping Reflectance Models• Commonly thought of as the normal
map filtering problem • Some common techniques involve
first MIP mapping the height map, and then creating a normal map from them, or fading the normal to (0,0,1)
• All of these techniques are grossly inaccurate– in fact, the normal is the least important aspect of a correct MIP filter
• Only NDM (Olano and North) filters linearly
A little bit of microfacet Theory• Surface modeled as flat microfacets
• Each microfacet is a Fresnel mirror• most BRDFs are microfacet models
• Normal Distribution Function (NDF) p(ω)• p(ω)dω is fraction of facets which have
normals in the solid angle dω around ω• ω is defined in the surfaces’ local
frame• There are various different options for p(ω)
Microfacet Theory• For given ωi and ωe, only facets
oriented to reflect ωi into ωe are active
• These facets have normals at ωhωi
ωe
ωiωiωi
ωiωiωi
ωeωe ωe ωe ωe
ωe
ωhωhωh
ωh ωh ωh
ωh
Microfacet Theory• Fraction of active microfacets is
p(ωh)dω• Active facets have reflectance RF(αh)• αh is the angle between ωi (or ωe) and
ωh ωiωi ωi
ωe
ωiωiωiωi
ωeωe ωe ωe ωe
ωe
ωhωhωh
ωh ωh ωh
ωh
So what does a MIP map mean anyway?• Non-linear lighting functions
fundamentally represent microfacets • Specifically, represents a distribution
of micofacets• A mip map is a lower resolution image
which best represents the higher resolution one
• But – how does this work if we have a bunch of little mirrors?
A simple non linear function
The Blinn-Phong lighting model is one of the most common half angle based lighting functions. The above is a normalized version, that is the total emitted energy is close to constant for any power. This model is parameterized by N, Kd, Ks, and p. Changing these values changes the properties of the underlying surface
psd VLNkpLNkVLf )(*)(),(
A common approach
The image on the left is lit with MIP maps fading to flat, while the image on the right is what the image would look like if rendered at a higher resolution and scaled down. The objects do not physically look the same.
A more correct MIP map
The images in the middle were rendered directly to the screen using a BRDF approximating MIP map. The more we zoom out, the larger the difference between the correct and incorrect approaches
Examining a mip level – simple
32 32
32 32
32
Next MIP Level – same thing
Examining a mip level8 45
16 25
25
Next MIP Level, Average Power?
Examining a MIP level
32 32
32 32
32
Next MIP Level, Average power, normal?
What about a larger patch and a lower mip?
5
Setting up the problem• Forgetting for a second about
texture filtering, each MIP level should approximate the signal in the higher level texture
• For a BRDF model, this means that the lower MIP levels should approximate the more detailed version
• In other words, we want the BRDF at a different scale
A simple BRDF• To start, consider the specular part
of the normalized Blinn-Phong BRDF
• For each texel, there is a function F(V,L), which given a View direction and a Light direction, returns a color
• Each texel has a different N, Kd, and Power to get surface variation
• N, Ks, and Power are the parameters of a Blinn-Phong lighting model
Reflectance at a single texel
Ks = .02, P = 30 Ks = .02, P = 30
Ks = .02, P = 24 Ks = .01, P =30 Ks = ???, P = ???
(?,?,?)
For a given light and view For a given light and view direction, how do we pick a Ks, direction, how do we pick a Ks, a Power and Normal so that a Power and Normal so that we get the same final color?we get the same final color?
MIP Level 0 MIP Level 1
Solving for a single patch of texels
),(4
),(),(),(),( 22211211 VLFVLFVLFVLFVLF
For a given patch of texels w wide and h tall:For a given patch of texels w wide and h tall:
),(),(10 0
VLFVLFwj
w
i
h
jij
ijpijijijij VLNkpVLF )(),(
Where,Where,
We care about all pairs of L and V
L V
w
i
h
jij VLFVLF
wjpkNE
2
0 0
),(),(1),,(
2
0 0
),(),(1),,(
VLFVLFwj
pkNew
i
h
jij
Creating an error function for a single pair of L and V:Creating an error function for a single pair of L and V:
Need to find the minimum error across all possible V and L Need to find the minimum error across all possible V and L
Creating a MIP filter• We want to find a value of N, k, and p
such that E(N, k, p) is the smallest value possible
• This is a non-linear optimization problem
• There are many ways to fit non-linear problems, for this problem we chose to use BFGS which is a quasi-Newton algorithm
Non Linear fitting• BFGS requires the partial derivatives of the
error function with respect to each parameter we want to fit
• The derivatives must be continuous over the space we are interested in
• Double integral cannot be solved analytically, so it is numerically evaluated
• A good way to express the normal is as an X, Y with an implied Z. But error function must take care to keep the normal normalized
Giving a good first guess• Non-linear algorithms work well when
we don’t have many local minima and we have a reasonable starting guess
• For the Blinn-Phong, seed of the averaged Normal, the averaged Ks, and a low power – less then half of the previous level works well
• But, power should not be near or less then 1!
Non linear with Blinn PhongWith the Blinn-With the Blinn-Phong BRDF, the Phong BRDF, the error plot shows error plot shows that the function that the function is mostly smooth. is mostly smooth. As long as we take As long as we take a guess in the a guess in the dark region, we dark region, we should be fine. should be fine. Graph is log(1 + Graph is log(1 + error)error)
power
Ks
Speeding it up• Running a BFGS with that error function
took a few hours on a 512x512 texture• Gives a good fit, but still too slow• However, this function can be
refactored in terms of the half angle• This reduces dimensionality from 4 to 2!• Additionally, summation in middle of
integral is constant for a given H, so this can be precomputed and reused, assuming enough memory
Speeding it up
H
w
i
h
jij HFHF
wjpkNE
2
0 0
)()(1),,(
But, we must use Half Angle But, we must use Half Angle Distribution Distribution
Results• Resulting MIP level looks closer to
original detail • Turns out that the optimal normal is
usually pretty close to the average normal. You can get away with minimizing just P and Ks
• The power decreases rapidly for a rough surface. Each MIP has about 50-70% of the exponent of its previous level!
• An object with a shiny rough surface will appear dull as the object is further away
Dealing with Anisotropy• Using a simple Blinn-Phong BRDF doesn’t allow
anisotropic effects to be captured at lower MIP levels
• A simplified, half angle version of the Ashikhmin-Shirley BRDF can do this
• We now have E(N, T, Ks, Pu, Pv), that is, we have 2 powers for the tangent frame – and the Tangent frame might vary per pixel
• Anisotropic objects, e.g. a record with grooves on it should stay anisotropic
Why Ashikhmin-Shirley ?• Convergence is generally better,
compared to other BRDFS • Is half angle based – so MIP
evaluation is still fast• Is data light, computation heavy• Can exactly represent Blinn-
Phong
Anisotropy• For more complex BRDFs, one should take
care to ensure function well behaved at limit points
222
22
)()(21)(
)()(
0)()(lim HTPHTPHN
HTPHTP
HN
vvuu
vvuu
eHN
2
22
)()()(
)()( HNHTPHTP
vus
vvuu
HNPPKHF
A side note:• Artists can create extremely high
resolution height maps, modeling the micro geometry aspects of a surface
• Brush strokes, grooves, scratches etc would just appear on the high-res height map
• A fitting algorithm could then capture the various aspects – roughness, anisotropy, etc.
Solution for high quality rendering1) Determine objects maximum MIP level2) Render from that MIP level downward,
culling off unseen polygons3) Compute lower MIP levels4) Draw object with the lit texture, using best
filter possible*5) At lower MIP levels, once texel variation is
minimal, disable TSL *Optionally blend MIP transitions to eliminate any
popping
System considerations• Don’t need a texture for each objects
–just a set of textures which we recycle
• Can reduce render resolution to adjust framerate
Final thoughts on fitting• More research to be done on
BRDF fitting • Untackled issues : more
anisotropic fitting, need to look at more data
• Optimizing for bilinear reconstruction would also help direct rasterization at the expense of removing some detail
More thoughts• Even if you don’t use Texture space
lighting, fitting non linear functions to create MIP maps should be done
• It’s free at runtime! • Texture Space Lighting is an easily
scalable feature. It can be enabled on newer hardware
• Because TSL gets computed in point sampling mode and pasted once, we can invest a resources in a much better reconstruction filter – e.g. bicubic with better anisotropy
• Introduction To Modern Optics, 2nd ed., Fowles (Dover)• Principles of Digital Image Synthesis, Glassner (MK)• The Physics and Chemistry of Color, 2nd ed., Nassau (Wiley)• Geometric considerations and nomenclature for reflectance, Nicodemus et. al.
(NBS 1977)• A practitioners' assessment of light reflection models, Shirley et. al. (Pacific
Graphics 1997)• A survey of shading and reflectance models for computer graphics, Schlick
(Computer Graphics Forum)• Illumination for computer generated pictures, Phong (Communications of the ACM)• Models of light reflection for computer synthesized pictures, Blinn (SIGGRAPH
1977)• A reflectance model for computer graphics, Cook and Torrance (ACM ToG)• Measuring and modeling anisotropic reflection, Ward (SIGGRAPH 1992)• An anisotropic phong BRDF model, Ashikhmin and Shirley (JGT)• Non-linear approximation of reflectance functions, Lafortune et. al. (SIGGRAPH
1977)• Generalization of Lambert's reflectance model, Oren and Nayar (SIGGRAPH 1994)• Predicting reflectance functions from complex surfaces, Westin et. al. (SIGGRAPH
1992)• A microfacet-based BRDF generator, Ashikmin et. al. (SIGGRAPH 2000)• Illumination in diverse codimensions, Banks (SIGGRAPH 1994)• An accurate illumination model for objects coated with multilayer films, Hirayama
et. al. (Eurographics 2000)• Diffraction shaders, Stam (SIGGRAPH 1999)• Simulating diffraction, Stam, in GPU Gems (AW)
References - Background
References - Techniques• Experimental validation of analytical BRDF models, Ngan et. al. (SIGGRAPH 2004
Sketch)• Normal distribution functions and multiple surfaces, Fournier (Graphics Interface
1992)• From structure to reflectance, Fournier and Lalonde, in “Cloth Modeling and
Animation” (AK Peters)• Efficient rendering of spatial bi-directional reflectance distribution functions, McAllister
et. al. (Graphics Hardware 2002)• An efficient representation for irradiance environment maps, Ramamoorthi and
Hanrahan (SIGGRAPH 2001)• Precomputed radiance transfer for real-time rendering in dynamic, low-frequency
lighting environments, Sloan et. al. (SIGGRAPH 2002)• The plenoptic function and the elements of early vision, Adelson and Bergen, in
Computational Models of Visual Processing (MIT)• Steerable illumination textures, Ashikhmin and Shirley (ToG)• Polynomial Texture Maps, Malzbender et. al. (SIGGRAPH 2001)• Interactive subsurface scattering for translucent meshes, Hao et. al. (Si3D 2003)• Clustered principal components for precomputed radiance transfer, Sloan et. al.
(SIGGRAPH 2003)• All-frequency precomputed radiance transfer for glossy objects, Liu et. al. (EGSR 2004)• Triple product wavelet integrals for all-frequency relighting, Ng et. al. (SIGGRAPH
2004)• All-frequency shadows using non-linear wavelet lighting approximation, Ng et. al.
(SIGGRAPH 2003)• The irradiance volume, Greger et. al. (IEEE CG&A Mar.98)• Spherical harmonic gradients for mid-range illumination, Annen et. al. (EGSR 2004)• Ambient occlusion fields, Kontkanen and Laine (Si3D 2005)• Normal Distribution Mapping, Olano and North (Tech Report, 1996)
Acknowledgements• Shanon Drone, John Rapp, Jason
Sandlin and John Steed for demo help• Michael Ashikhmin, Henrik Wann
Jensen, Kazufumi Kaneda, Eric Lafortune, David McAllister, Addy Ngan, Michael Oren, Kenneth Torrance, Gregory Ward, and Stephen Westin for permission to use images
• Naty Hoffman for use of his background slides
© 2005 Microsoft Corporation. All rights reserved. Microsoft, is a registered trademark of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
© 2005 Microsoft Corporation. All rights reserved. Microsoft, is a registered trademark of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
