Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination...
-
Upload
harry-sullivan -
Category
Documents
-
view
217 -
download
0
Transcript of Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination...
![Page 1: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/1.jpg)
Mitsubishi Electric Research Labs
Progressively Refined Reflectance Fields from Natural Illumination
Wojciech Matusik
Matt Loper
Hanspeter Pfister
![Page 2: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/2.jpg)
MotivationMotivation
• Complex natural scenes are difficult to acquire• Acquisition needs to be easy and robust• Image-based lighting offers high realism • We would like to relight image-based models at
any scale (from small objects to cities)
![Page 3: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/3.jpg)
MotivationMotivation
• Image-based Relighting– no scene geometry – just images– no assumptions about scene reflectance properties
![Page 4: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/4.jpg)
Previous WorkPrevious Work
• Forward Approaches– Georghiades 99, Debevec 2000, Malzbender 01,
Masselus 02, Peers 03
• Inverse Approaches– Zongker 99, Chuang 00, Wexler 02
• Pre-computed Light Transport– Sloan 02, Ng 03
![Page 5: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/5.jpg)
Reflectance FieldReflectance Field
• 8D function:
[Debevec 2000]
),;,;,;,( rrrriiiir vuvuf
(θr,φr)
(ur,vr)
(θi,φi)
(ui,vi)
![Page 6: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/6.jpg)
Reflectance (Weighting) FunctionReflectance (Weighting) Function
• Assumes incident illumination originates at infinity
• x,y are image space coordinates
θi
φi
),;,( iiw yxf
![Page 7: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/7.jpg)
Light Transport ModelLight Transport Model
• A light flow in the scene can be modeled as a multiple-input / multiple-output linear system:
TLB
Scene
light transport
matrix
T
Incident Light
L
Observed Image
B
Unroll to a vector Unroll to a vector
![Page 8: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/8.jpg)
Light Transport ModelLight Transport Model
• Solve independently for each output pixel multiple-input / single-output linear system :
LTb ii
Scene
light transport
vector
Ti
Incident Light
L
Observed Pixel
bi
![Page 9: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/9.jpg)
RepresentationRepresentation
• Approximate Ti as a sum of 2D rectangular kernels Rk,i, each with weight wk,i.
k
ikiki RwT ,,
θi
φi
![Page 10: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/10.jpg)
Inverse EstimationInverse Estimation
• Given input images Lj we record observed pixel values bij:
• Given matrix L and vector bi the goal is to estimate Ti
– Positions and sizes of the rectangular kernels Rk,i
– Weights wk, i
jiij LTb
,...],[,...],[ 2121 LLTbb iii
![Page 11: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/11.jpg)
Estimating Kernel WeightsEstimating Kernel Weights
• Assume that we know sizes and positions of the kernels Rk,i and would like to compute their weights
0minarg2 iiiiw wtosubjectbwA
LTb ii
02
1minarg
i
iTi
Tiii
Ti
Tiw
wtosubject
bAwwAAw
ii wLRLRw ikk
ik
,, iiwA
•Efficient solution using quadratic programming
![Page 12: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/12.jpg)
Estimating Kernel Positions & SizesEstimating Kernel Positions & Sizes
• Hierarchical kd-tree subdivision of the kernels input image domain
• At each level choose subdivision that reduces error the most
• Kernels are non-overlapping
2
iii bwA
![Page 13: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/13.jpg)
Kernel SubdivisionsKernel Subdivisions
specular
refractive
1 2 3 4 5 10 20 24
subsurface
scattering
glossy
hard
shadow
Subdivisions
![Page 14: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/14.jpg)
Spatial CorrectionSpatial Correction
• The kernels search strategy does not always work• Solution: For each output pixel:
– try kernel positions and sizes of the neighboring output pixels
– try shifted versions of the current kernels– solve for new weights– keep new kernels if the error decreases
![Page 15: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/15.jpg)
Integration with Incident IlluminationIntegration with Incident Illumination
• Is very efficient• For each output pixel i
• The incident illumination is stored as a summed-area table to evaluate
kiikiki
kikikiii LRwLRwLTb ,,,,
iik LR ,
![Page 16: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/16.jpg)
Data AcquisitionData Acquisition
• We have built two acquisition systems– Indoor scenes / small objects
– Outdoor scenes (city)
![Page 17: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/17.jpg)
Acquisition System IAcquisition System I
![Page 18: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/18.jpg)
Example Input ImagesExample Input Images
![Page 19: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/19.jpg)
Results Results
• Refractive and specular elements
Prediction Actual
![Page 20: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/20.jpg)
Results – New IlluminationResults – New Illumination
![Page 21: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/21.jpg)
Results - White Vertical BarResults - White Vertical Bar
Prediction Actual
![Page 22: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/22.jpg)
ResultsResults
Estimate Actual
• Diffuse elements, shadows
![Page 23: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/23.jpg)
Results - White Vertical BarResults - White Vertical Bar
![Page 24: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/24.jpg)
ResultsResults
Estimate Actual
• Subsurface Scattering
![Page 25: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/25.jpg)
Results - White Vertical BarResults - White Vertical Bar
![Page 26: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/26.jpg)
ResultsResults
• Glossy elements and interreflections
Estimate
Actual
![Page 27: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/27.jpg)
Results - White Vertical BarResults - White Vertical Bar
![Page 28: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/28.jpg)
ResultsResults
• One shifted version of the same image used as input illumination
![Page 29: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/29.jpg)
Acquisition System IIAcquisition System II
• Two Synchronized CamerasCamera #1 Camera #2
![Page 30: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/30.jpg)
Example Observed ImagesExample Observed Images
![Page 31: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/31.jpg)
Results – Relighting The CityResults – Relighting The City
• White vertical bar
![Page 32: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/32.jpg)
LessonsLessons
• Inverse approaches benefit from good kernel search strategies & more computation power
• Inverse approaches are more efficient than forward approaches
• Challenges:– Scene needs to be static– Varied set of input illumination– Illumination is not at infinity
![Page 33: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/33.jpg)
ConclusionsConclusions
• Advantages of our algorithm:– Natural Illumination Input– All-frequency Robustness– Compact Representation– Progressive Refinement– Fast Evaluation– Simplicity
![Page 34: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/34.jpg)
Future WorkFuture Work
• New acquisition systems– object and camera are fixed w.r.t. each other and they
rotate in a single, natural environment
• Combining representations from different viewpoints and proxy geometry
• Coarse-to-fine estimation in the observed image space– start with low resolution observed images & search
exhaustively for the best kernels– propagate the kernels to higher resolution images
![Page 35: Mitsubishi Electric Research Labs Progressively Refined Reflectance Fields from Natural Illumination Wojciech Matusik Matt Loper Hanspeter Pfister.](https://reader030.fdocuments.in/reader030/viewer/2022012922/5697bf9c1a28abf838c933d2/html5/thumbnails/35.jpg)
AcknowledgementsAcknowledgements
• Jan Kautz• Barb Cutler• Jennifer Roderick Pfister• EGSR Reviewers