Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 ·...
Transcript of Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 ·...
![Page 1: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/1.jpg)
DenoisingMonte Carlo Images with Machine Learning
Team 5: Cheolmin Lee, Minki Jo, Nick Heppert
![Page 2: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/2.jpg)
INDEX
2
Approach
3
Discussion
1
Introduction
![Page 3: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/3.jpg)
Introduction
![Page 4: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/4.jpg)
* =
Noisy Image Clear ImageKernel* =
KPCN
KPCN
Review
![Page 5: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/5.jpg)
Changing Subject
• Previous Subject - Channel Pruning
• Reduce the number of channels
• Current Subject - Modify Network Structure
• Improve performance of denoising
![Page 6: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/6.jpg)
Review – Channel Pruning
Learning Algorithm
KPCN
Input(28 + 28 channel)
Output(3 channel)
KPCN
Input(28 + 28 channel)
Output(3 channel)Trained Network
Fine-tuned Network
W(Weight)
![Page 7: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/7.jpg)
Problem of Channel Pruning
1 loop = 1day
• Learning iteration takes too long time
• 1 day per 1 loop
• Almost 20 days are required to complete one experiment
Prunning은학습을 20회정도반복해야하는데, 1회학습이안정적으로수
렴하기까지 1일이상이소요되어현실적으로불가능하다고판단함.
![Page 8: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/8.jpg)
Modify Network Structure
cConv(28, 100)
cConv(100, 100)cConv(100, 100)cConv(100, 100)cConv(100, 100)cConv(100, 100)cConv(100, 100)cConv(100, 100)
cConv(100, 21 * 21)
7 layers
KPCN structure
SENet structure
[Zhang et. al. 18] Image Super-Resolution Using Very Deep Residual Channel Attention Networks, ECCV2018
[He et. al. 16] Deep Residual Learning for Image Recognition, CVPR2016
ResNet structure
![Page 9: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/9.jpg)
Approach &Experiments
![Page 10: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/10.jpg)
Road to Baseline
1. In-Official Re-Implementation
2. No dataset
3. Patch storage
4. Large-scale training
5. Instability of lossa. Skip batch with NaN kernelb. Patches with infinity
6. Finally Baseline
![Page 11: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/11.jpg)
Diffuse Network (s)
Prediction
Ground Truth
Baseline Results
Specular Network (s)
Prediction
Ground Truth
![Page 12: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/12.jpg)
Baseline Results
Prediction
Ground-Truth
![Page 13: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/13.jpg)
Experiments on Baseline
• L1-Loss on a 100-patch test-set• Layer increase: 11.40• Batch-normalization: 11.04• Layer increase + dropout: 9.170• Dropout: 8.137• Baseline: 7.916• Combined loss: 7.901• Learning rate reduction: 7.824
![Page 14: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/14.jpg)
Comparison: Baseline vs. Combined Loss
![Page 15: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/15.jpg)
Minor idea
Recent novel network architectures
• Channel attention This method can break the high correlation between channels and improve the performance of the model.
[Zhang et. al. 18] Image Super-Resolution Using Very Deep Residual Channel Attention Networks, ECCV2018
![Page 16: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/16.jpg)
KPRCN
Residual learning
• KPCN follows the design of the VGG net (2014)
• Small receptive field
• Deep layers
• Therefore, we applied the residual learning(2016) technique to the KPCN.
• No bottle neck, no batch normalization layer
[He et. al. 16] Deep Residual Learning for Image Recognition, CVPR2016
![Page 17: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/17.jpg)
KPRCAN
Channel Attention
• Squeeze-Excitation network(2017) uses the channel attention over the convolutional layer.
• The KPRCAN uses the channel attention block.
[Zhang et. al. 18] Image Super-Resolution Using Very Deep Residual Channel Attention Networks, ECCV2018
![Page 18: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/18.jpg)
Direct Prediction model
Channel Attention
• For the new models, we also implemented the direct prediction model by employing the recent image restoration technique.
• Recent denoising models does not use Kernel prediction method, but use Direct prediction.
• Divide the model into head, body, tail block, and add the skip connection that cross the body block
[Zhang et. al. 18] Image Super-Resolution Using Very Deep Residual Channel Attention Networks, ECCV2018
![Page 19: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/19.jpg)
Direct Prediction model
Direct prediction
• All of the Direct prediction models does not trained well.
![Page 20: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/20.jpg)
Direct Prediction model
Direct prediction
• All of the Direct prediction models does not trained well.
• The model easily exploded.
![Page 21: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/21.jpg)
Kernel Prediction model
Kernel prediction
• For the patches, the performance of the KPCN is the best.
• KPCN• KPRCN• KPRCAN
Test accuracy for patches
Input
Output
GT
![Page 22: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/22.jpg)
Kernel Prediction model
Kernel prediction
• For the test images, the performance of the KPRCAN is the best.
KPCN
KPRCN
KPRCAN
![Page 23: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/23.jpg)
Kernel Prediction model
Kernel prediction
• For the test images, the performance of the KPRCAN is the best.
• But if we set the same layer number for the KPRCAN, it achieve the best performance.
• However more careful modification is required.(Explained later)
KPCN
KPRCN
KPRCAN
KPRCAN4 Block
![Page 24: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/24.jpg)
Final result - Input
![Page 25: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/25.jpg)
Final result – Output(KPRCAN)
![Page 26: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/26.jpg)
Final result - GT
![Page 27: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/27.jpg)
Discussion
![Page 28: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/28.jpg)
Discussion
Data format
• The original data is stored in HDR format. Which has the range [0, inf]. Therefore, the inference range is also [0, inf] while most of the points have the value in [0,1].
• However when we evaluate the data, we convert the HDR data into RGB data by clamping the tensor value. Which means, we don’t need to exactly infer the value over 1.
Original GT data
GT data in RGB데이터학습(최적화)는 HDR에서하는데, 성능평가는 RGB에서한다.
서로다른특성으로인해성능이나빠질수있다.
![Page 29: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/29.jpg)
Discussion
Data format
• In short, we take loss from the HDR but evaluate in RGB where those format have different data range.
• This cause numerical extrapolation problem. (the value used for optimization and evaluation has different data range, different characteristic)
Original GT data
GT data in RGB데이터학습(최적화)는 HDR에서하는데, 성능평가는 RGB에서한다.
서로다른특성으로인해성능이나빠질수있다.
![Page 30: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/30.jpg)
Discussion
Data format
• When we have a invalid value for the GT, we can detach the loss from those invalid data points in order to avoid irrelevant training.
• What if we applied this method? Or clamp the data at the training session.
Invalid point에서로스를받지않도록 detach 할수있다. 이런기법
혹은데이터전처리를통해성능을개선할수있을지도모른다.
![Page 31: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/31.jpg)
Discussion
Kernel prediction
• For the test images, the performance of the KPRCAN is the best.
• But if we set the same layer number for the KPRCAN, it achieve the best performance.
• However more careful modification is required.(Explained later)
KPCN
KPRCN
KPRCAN
KPRCAN4 Block
![Page 32: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/32.jpg)
Discussion
Network structure
• In terms of the receptive field, stacking two 3x3 layer has same size of receptive field with one 5x5 layer.
• In addition, ResBlock require at least two block between the skip connection
3x3 conv
3x3 conv
5x5 conv
3x3 conv
3x3 conv
5x5 conv
5x5 conv
5x5 conv3x3 conv
3x3 conv
3x3 conv
3x3 conv
Same receptive field, double number of layers
3x3 conv 2개와 5x5 conv 1개는다른레이어수를갖지만
같은크기의영역을커버한다.
![Page 33: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/33.jpg)
Network structure
• The KPRCAN 4 Block has same size of receptive field and number of layers with the KPCN.
• Since this network does not use the Batch normalization, it cannot use large number of layers.
• Furthermore, this model is extremely sensitive so we need to modify this network carefully.
Kernel Prediction model
8 of 5x5 conv layer body(Total 10 layer)
KPCN
KPRCN
KPRCAN
KPRCAN4 Block
8 of 5x5 ResBlock body(Total 18 layer)
8 of 5x5 SEBlock body(Total 18 layer)
4 of 3x3 SEBlock body(Total 10 layer)
앞의슬라이드처럼 KPCN과 KPRCAN이같은영역을커버하
도록층수와커널수를조절하니확연히다른결과를얻었다.
BN등을사용할수없어깊은층수를갖게되면학습이되지않고 모델
자체가매우민감한특성등으로인해세심한조정이필요하다.
![Page 34: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/34.jpg)
Team Contribution
• Cheolmin: Baseline Code
• Nick: Experiments on Baseline
• Minki: Extended Models
![Page 35: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/35.jpg)
Thank You!
![Page 36: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/36.jpg)
Reference
[Liu et. al. 17] Learning Efficient Convolutional Networks through Network Slimming, ICCV2017
[Zhang et. al. 18] Image Super-Resolution Using Very Deep Residual Channel Attention Networks, ECCV2018
[Lehtinen et. al. 18] Noise2Noise: Learning Image Restoration without Clean Data, ICML2018
[Bako Et al. 17] “Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings.” ACM Transactions on Graphics 36, no. 4 (July 20, 2017)
![Page 37: Denoising Monte Carlo Images with Machine Learningsungeui/GCG/Student... · 2019-06-04 · Denoising Monte Carlo Images with Machine Learning Team 5: Cheolmin Lee, Minki Jo, Nick](https://reader033.fdocuments.in/reader033/viewer/2022050407/5f8499a4b69aac14742ab34b/html5/thumbnails/37.jpg)
Back up