Learning-Based Rendering - KAIST
Transcript of Learning-Based Rendering - KAIST
![Page 1: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/1.jpg)
Learning-Based Rendering
2019.05.16.
20193138 Jaeyoon Kim
1
![Page 2: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/2.jpg)
Review for previous presentation
• “Microfacet Model and Material Appearance” by Hakyeong Kim
2
![Page 3: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/3.jpg)
Machine Learning in Rendering
From CheolMin’s slides From Saehun’s slides3
![Page 4: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/4.jpg)
Deep Scattering:Rendering Atmospheric Clouds withRadiance-Predicting Neural Networks
by S. Kallweit, T. Müller, B. McWilliams, M. Gross, J. Novák
Slides and images from “Deep Learning for Light Transport” by Jan Novák, Disney Research 4
![Page 5: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/5.jpg)
Remind previous presentation
From Hyunsoo’s slides 5
![Page 6: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/6.jpg)
6
![Page 7: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/7.jpg)
7
![Page 8: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/8.jpg)
8
![Page 9: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/9.jpg)
9
![Page 10: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/10.jpg)
Rendering Clouds
𝐿𝐿 𝐱𝐱,𝑤𝑤 = �0
∞𝑝𝑝 𝐲𝐲|𝐱𝐱 𝛼𝛼�
𝑆𝑆𝑝𝑝 �𝑤𝑤|𝑤𝑤 𝐿𝐿 𝐲𝐲, �𝑤𝑤 d�𝑤𝑤d𝑦𝑦
Predict using a network
10
![Page 11: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/11.jpg)
Rendering Clouds
• Scattered direct illumination• Estimated using MC
11
![Page 12: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/12.jpg)
Rendering Clouds
• Scattered direct illumination• Estimated using MC
• Scattered indirect illumination• Predicted using NN
12
![Page 13: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/13.jpg)
Radiance Predicting MLP
• Uses hierarchical stencil for sampling.• for both local details and overall shape
• Feeds into the network.
13
![Page 14: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/14.jpg)
Radiance Predicting MLP
• Uses hierarchical stencil for sampling.• for both local details and overall shape
• Feeds into the network.• One block with two layers for each stencil.• Some FC layers at the end of the network.
14
![Page 15: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/15.jpg)
15
![Page 16: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/16.jpg)
16
![Page 17: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/17.jpg)
Slides and images from “Bayesian online regression for adaptive direct illumination
sampling” by Petr at al. 17
![Page 18: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/18.jpg)
Noise in MC Rendering
18
![Page 19: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/19.jpg)
Noise in MC Rendering
Non-adaptive sampling[Wang et al. 2009]
Reference
19
![Page 20: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/20.jpg)
Noise in MC Rendering
Non-adaptive sampling[Wang et al. 2009]
Adaptive sampling[Donikian et al. 2006]
20
![Page 21: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/21.jpg)
Direct Illumination
Less important
Occluded
21
![Page 22: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/22.jpg)
Light Clustering
22
![Page 23: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/23.jpg)
Direct Illumination
• 𝐿𝐿 𝐱𝐱,𝑤𝑤 = ∫𝐴𝐴 𝐹𝐹 𝐲𝐲 → 𝐱𝐱 → 𝑤𝑤 d𝐲𝐲
• 𝐹𝐹 𝐲𝐲 → 𝐱𝐱 → 𝑤𝑤 = 𝐿𝐿𝑒𝑒 𝐲𝐲 → 𝐱𝐱 𝐵𝐵 𝐲𝐲 → 𝐱𝐱 → 𝑤𝑤 𝑉𝑉 𝐲𝐲 ↔ 𝐱𝐱 𝐺𝐺 𝐲𝐲 ↔ 𝐱𝐱
• 𝐿𝐿 𝐱𝐱,𝑤𝑤 = 𝐹𝐹 𝐲𝐲→𝐱𝐱→𝑤𝑤𝑝𝑝 𝐲𝐲|𝐱𝐱,𝑤𝑤
𝑤𝑤
𝐲𝐲 𝐱𝐱𝐴𝐴
23
![Page 24: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/24.jpg)
Probability Density Function
• 𝑝𝑝 𝐲𝐲|𝐱𝐱,𝑤𝑤 = 𝑃𝑃 𝑐𝑐|𝐱𝐱 𝑃𝑃 𝑙𝑙|𝑐𝑐 𝑝𝑝 𝐲𝐲|𝑙𝑙,𝑤𝑤• 𝑃𝑃 𝑐𝑐|𝐱𝐱 : selecting a light cluster c ∈ C• 𝑃𝑃 𝑙𝑙|𝑐𝑐 : selecting a light 𝑙𝑙 ∈ 𝑐𝑐, 𝑃𝑃 𝑙𝑙|𝑐𝑐 = 𝛷𝛷𝑙𝑙/∑𝑙𝑙’∈𝑐𝑐 𝛷𝛷𝑙𝑙’
• 𝑝𝑝 𝐲𝐲|𝑙𝑙,𝑤𝑤 : selecting a point 𝑦𝑦 ∈ 𝑙𝑙, using standard techniques in [Pharr et al. 2016; Shirley et al. 1996].
• Only need to predict 𝑃𝑃 𝑐𝑐|𝐱𝐱 .
24
![Page 25: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/25.jpg)
Optimal Cluster Selection
• 𝑉𝑉𝑉𝑉𝑉𝑉 𝐿𝐿 𝐱𝐱 = −𝐿𝐿 𝐱𝐱 2 + ∑𝑐𝑐∈𝐶𝐶1
𝑃𝑃 𝑐𝑐|𝐱𝐱 ∫𝐴𝐴𝑐𝑐𝐹𝐹 𝐲𝐲→𝐱𝐱 2
𝑃𝑃 𝑙𝑙|𝑐𝑐 𝑝𝑝 𝐲𝐲|𝑙𝑙d𝐲𝐲
• 𝑚𝑚2,𝑐𝑐 = ∫𝐴𝐴𝑐𝑐𝐹𝐹 𝐲𝐲→𝐱𝐱 2
𝑃𝑃 𝑙𝑙|𝑐𝑐 𝑝𝑝 𝐲𝐲|𝑙𝑙d𝐲𝐲: second moment of 𝐿𝐿𝑐𝑐 𝐱𝐱 = 𝐹𝐹 𝐲𝐲→𝐱𝐱
𝑃𝑃 𝑙𝑙|𝑐𝑐 𝑝𝑝 𝐲𝐲|𝑙𝑙
• 𝑃𝑃𝑜𝑜𝑝𝑝𝑜𝑜 𝑐𝑐|𝐱𝐱 ∝ 𝐿𝐿𝑐𝑐2 𝐱𝐱 + 𝑉𝑉𝑉𝑉𝑉𝑉 𝐿𝐿𝑐𝑐 𝐱𝐱
25
![Page 26: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/26.jpg)
Bayesian Online Regression
• 𝐃𝐃: training data• 𝜃𝜃: parameters• 𝑝𝑝 𝐃𝐃|𝜃𝜃 : a model describing the likelihood of 𝐃𝐃 given 𝜃𝜃
• Maximum likelihood (ML) estimate• Finding 𝜃𝜃 maximizing 𝑝𝑝 𝐃𝐃|𝜃𝜃• Prone to overfit when data is scarce• Gives poor approximations in early stages of rendering
• Maximum a posteriori (MAP) estimate• Finding 𝜃𝜃 maximizing 𝑝𝑝 𝜃𝜃|𝐃𝐃 ∝ 𝑝𝑝 𝐃𝐃|𝜃𝜃 𝑝𝑝 𝜃𝜃• More robust than ML estimate
26
![Page 27: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/27.jpg)
Bayesian Online Regression
• �̂�𝑒 = 𝐿𝐿𝑒𝑒 𝐲𝐲→𝐱𝐱 𝑉𝑉 𝐲𝐲↔𝐱𝐱 cos𝜃𝜃𝐲𝐲𝑑𝑑2 𝐱𝐱,𝐲𝐲 𝑃𝑃 𝑙𝑙|𝑐𝑐 𝑝𝑝 𝐲𝐲|𝑙𝑙
, �̂�𝑒𝐱𝐱 = �̂�𝑒cos𝜃𝜃𝐱𝐱
• Data 𝐃𝐃 consists of tuples �̂�𝑒𝐱𝐱,𝑖𝑖 , �̂�𝑑𝑖𝑖 .
• 𝑝𝑝(𝐃𝐃|𝜃𝜃) = ∏𝑖𝑖𝑁𝑁 𝑝𝑝 �̂�𝑒𝐱𝐱,𝑖𝑖|�̂�𝑑𝑖𝑖 ,𝜃𝜃
• 𝑝𝑝 �̂�𝑒𝐱𝐱|�̂�𝑑,𝜃𝜃 = 𝛿𝛿 �̂�𝑒𝐱𝐱 𝑝𝑝0 + 1 − 𝑝𝑝0 𝑁𝑁 �̂�𝑒𝐱𝐱| 𝑘𝑘�𝑑𝑑2
, ℎ�𝑑𝑑4• 𝜃𝜃 = 𝑝𝑝0, 𝑘𝑘, ℎ• 𝑝𝑝 𝜃𝜃 = 𝐵𝐵 𝑝𝑝0| �𝑁𝑁𝑜𝑜, �𝑁𝑁𝑣𝑣 𝑁𝑁 − 𝛤𝛤−1 𝑘𝑘, ℎ|𝜇𝜇0, �𝑁𝑁, �𝑁𝑁𝛼𝛼 ,𝛽𝛽
27
![Page 28: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/28.jpg)
Bayesian Online Regression
• 𝐿𝐿𝑐𝑐 𝐱𝐱 ≈ 1−𝑝𝑝0 𝑘𝑘�𝑑𝑑2
• 𝑉𝑉𝑉𝑉𝑉𝑉 𝐿𝐿𝑐𝑐 𝐱𝐱 ≈ 1−𝑝𝑝0 𝑝𝑝0𝑘𝑘2+ℎ�𝑑𝑑4
• 𝑃𝑃∗ 𝑐𝑐|𝐱𝐱 ∝ 1�𝑑𝑑2
1 − 𝑝𝑝0 2𝑘𝑘2 + 1 − 𝑝𝑝0 𝑝𝑝0𝑘𝑘2 + ℎ
28
![Page 29: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/29.jpg)
Summary
• Light preprocess (clustering)• During each Next event estimation:
• Obtain cached clustering.• Compute distributions of estimates for each cluster. (mean, variance)• Build distribution over clusters.• Sample direct illumination.• Record new data for sampled cluster.
29
![Page 30: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/30.jpg)
Results
30
![Page 31: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/31.jpg)
Results
31
![Page 32: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/32.jpg)
Reference
• Simon Kallweit, Thomas Muller, Brian McWilliams, Markus Gross, and Jan Novak. 2017. Deep Scattering: Rendering Atmospheric Clouds with Radiance-Predicting Neural Networks. ACM Trans. Graph. 36, 6, Article 231 (November 2017), 11 pages.
• J. Novák at al. 2018. Deep Learning for Light Transport. Disney Research.
• Petr Vevoda, Ivo Kondapaneni, and Jaroslav Křivánek. 2018. Bayesian online regression for adaptive direct illumination sampling. ACM Trans. Graph. 37, 4, Article 125 (August 2018), 12 pages.
32
![Page 33: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/33.jpg)
Thank you!Any questions?
33
![Page 34: Learning-Based Rendering - KAIST](https://reader034.fdocuments.in/reader034/viewer/2022052013/62860cc9f762602dab16f128/html5/thumbnails/34.jpg)
Quiz
Fill in the blanks with words in the box.
1. In “Deep Scattering …“ paper, scattered ( ) illumination is predicted using a neural network.
2. In “Bayesian online regression …” paper, authors aim to reduce noise in ( ) illumination with Bayesian online learning.
(a) Direct(b) Indirect(c) Diffuse(d) Specular
34