Spherical Convolution in Computer Graphics and Vision Ravi Ramamoorthi Columbia Vision and Graphics...
-
Upload
ericka-dempster -
Category
Documents
-
view
216 -
download
0
Transcript of Spherical Convolution in Computer Graphics and Vision Ravi Ramamoorthi Columbia Vision and Graphics...
Spherical Convolution in Spherical Convolution in Computer Graphics and VisionComputer Graphics and Vision
Ravi Ramamoorthi
Columbia Vision and Graphics CenterColumbia University
SIAM Imaging Science Conference: May 17, 2006
OutlineOutline
Motivation and Practical Problems
Spherical Convolution and Applications
A Theory of Spherical Harmonic Identities
Signal Processing for Visual Appearance
Real-Time RenderingReal-Time Rendering
Motivation: Interactive rendering with natural illumination and realistic, measured materials
Inverse RenderingInverse Rendering
Geometric model
ForwardRenderingAlgorithm
BRDF
Novel lighting
Rendering
Photographs
Checking Image ConsistencyChecking Image Consistency
Easy to tamper / splice images Image processing software widely available
In news reporting and other applications Need to detect tampering or photomontage Verify image consistency
Try to check consistency of lighting, shading
OutlineOutline
Motivation and Practical Problems
Spherical Convolution and Applications
A Theory of Spherical Harmonic Identities
Signal Processing for Visual Appearance
Irradiance Environment MapsIrradiance Environment Maps
Incident Radiance(Illumination Environment Map)
Irradiance Environment Map
R N
Computing IrradianceComputing Irradiance
Classically, hemispherical integral for each pixel
Lambertian surface is like a low pass filter
Frequency-space analysis (spherical harmonics)
IncidentRadiance
Irradiance
Analytic Irradiance FormulaAnalytic Irradiance Formula
Lambertian surface is low-pass filter
lm l lmE A LlA
2 / 3
/ 4
0
2 1
2
2
( 1) !2
( 2)( 1) 2 !
l
l l l
lA l even
l l
l0 1 2
Basri & Jacobs 01Ramamoorthi & Hanrahan 01a
9 Parameter Approximation9 Parameter Approximation
-1-2 0 1 2
0
1
2
( , )lmY
xy z
xy yz 23 1z zx 2 2x y
l
m
Exact imageOrder 29 terms
RMS Error = 1%
For any illumination, average error < 2% [Basri, Jacobs 01]
Ramamoorthi and Hanrahan 01b
Real-Time RenderingReal-Time Rendering
Simple procedural rendering method (no textures) Requires only matrix-vector multiply and dot-product In software or NVIDIA vertex programming hardware
Widely used in Games (AMPED for Microsoft Xbox), Movies (Pixar, Framestore CFC, …)
( ) tE n n Mn
surface float1 irradmat (matrix4 M, float3 v) {
float4 n = {v , 1} ;
return dot(n , M*n) ;
}
Computer Vision Complex IlluminationComputer Vision Complex Illumination
Low Dimensional Subspace Lighting Insensitive Recognition (Basri and
Jacobs 01, Lee et al. 01, Ramamoorthi 02, …)
Photometric stereo, shape acquisition
Convolution for General MaterialsConvolution for General Materials
Ramamoorthi and Hanrahan 01
( , ) ( ) ,B N V L R N l l V dl
B L
Frequency: product
Spatial: integral
ij i ijB L
Spherical Harmonics
Related Theoretical WorkRelated Theoretical Work
Qualitative observation of reflection as convolution: Miller & Hoffman 84, Greene 86, Cabral et al. 87,99
Reflection as frequency-space operator: D’Zmura 91
Lambertian reflection is convolution: Basri Jacobs 01
Our Contributions Explicitly derive frequency-space convolution formula Formal quantitative analysis in general 3D case Apply to real-time, inverse rendering, computer vision
Natural Lighting, Realistic MaterialsNatural Lighting, Realistic Materials
Ramamoorthi and Hanrahan 02
Inverse RenderingInverse Rendering
3 photographs of cat sculpture• Complex unknown illumination• Geometry known• Estimate microfacet BRDF and distant lighting
OutlineOutline
Motivation and Practical Problems
Spherical Convolution and Applications
A Theory of Spherical Harmonic Identities
Signal Processing for Visual Appearance
Mahajan, Ramamoorthi, Curless ECCV 06
Ligh
ting
1Li
ghtin
g 2
Material 1 Material 2
Two Objects – Two LightingsTwo Objects – Two Lightings
1111lmllm LAB 1212
lmllm LAB
2121lmllm LAB 2222
lmllm LAB
Ligh
ting
1Li
ghtin
g 2
Material 1 Material 2
Two Objects – Two LightingsTwo Objects – Two Lightings
1111lmllm LAB 1212
lmllm LAB
2121lmllm LAB 2222
lmllm LAB
21212211lmlmlllmlm LLAABB
Ligh
ting
1Li
ghtin
g 2
Material 1 Material 2
Two Objects – Two LightingsTwo Objects – Two Lightings
1111lmllm LAB 1212
lmllm LAB
2121lmllm LAB 2222
lmllm LAB
21212211lmlmlllmlm LLAABB
21212112lmlmlllmlm LLAABB
Ligh
ting
1Li
ghtin
g 2
Material 1 Material 2
Two Objects – Two LightingsTwo Objects – Two Lightings
1111lmllm LAB 1212
lmllm LAB
2121lmllm LAB 2222
lmllm LAB
21212211lmlmlllmlm LLAABB
21212112lmlmlllmlm LLAABB
2211lmlmBB 2112
lmlmBB
Independent of Lighting and BRDF
Ligh
ting
1Li
ghtin
g 2
Material 1 Material 2
Image Estimation FrameworkImage Estimation Framework
?
11lmB
21lmB
12lmB
22lmB
21122211lmlmlmlm BBBB
11
211222
lm
lmlmlm B
BBB
Image EstimationImage Estimation
Object 1 Object 2
Ligh
ting
1
Our Method Actual
Ligh
ting
2
?
No BRDF and lighting known or estimated
Image Consistency CheckingImage Consistency Checking
Single Image Identitydiffuse + specular case
Two Lightings – Same Reflectance Identity
Two Materials – Two Lightings identity
Tampered Cat
Tampered Cat Untampered Cat
Signal Processing for AppearanceSignal Processing for Appearance
Signal Processing widely applicable visual appearance
Convolution relation for cast shadows [Soler and Sillion 98, Ramamoorthi et al. 04]
Convolution with glows for participating media (mist, fog, haze) [Sun et al. 05]
Signal-Processing analysis of light field and reflectance [Chai et al. 00, Zickler et al. 06]
Triple Product Integrals [Ng et al. 04]
First Order Analysis [Ramamoorthi et al. 06]
AcknowledgementsAcknowledgements
Collaborators Dhruv Mahajan Brian Curless Pat Hanrahan Sameer Agarwal (helpful discussions)
Funding: NSF and Sloan Foundation
http://www.cs.columbia.edu/~ravir