Temporal structure of ENSO in 20 th Century Climate simulations
DeepRemaster: Temporal Source-Reference Attention...
Transcript of DeepRemaster: Temporal Source-Reference Attention...
![Page 1: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/1.jpg)
sa2019.siggraph.org
DeepRemaster: Temporal Source-Reference Attention Networks for Comprehensive Video EnhancementSatoshi Iizuka Edgar Simo-Serra
![Page 2: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/2.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Background
• Vintage film is deteriorated• Noise, blur, and low contrast
• Black and white or low quality colors
• Digital remastering is a challenging task• Conducted manually by experts
• Requires a significant amount of both time and money
“Oliver Twist”(1933)
“A-Bomb Blast Effects”(1952)
Seven Samurai (1954)
![Page 3: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/3.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Our Goal
• Semi-automatically remastering of vintage films• This includes restoration, enhancement, and colorization
![Page 4: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/4.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Related Work
• Image/video restoration• Gaussian noise [Dabov+ ’07, Maggioni+ ’12 ’14, Lefkimmiatis ’18]
• JPEG noise [Zhang+ ’17]
• Blur [Shi+ ’16]
• Image Colorization• Scribble-based [Levin+ 2004; Yatziv+ ’04;
An+ ’09; Xu+ ’13; Endo+ ’16; Zhang+ ’17]
• Reference-based [Chia+ ’11;
Gupta+ ’12; He+ ’18]
• Automatic [Iizuka+ ’16; Larsson+ ’16; Zhang+ ’16][Zhang+ ’17][Levin+ ’04]
Gaussian Noise
![Page 5: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/5.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Related Work
• Reference-based video colorization• Recurrent neural networks [Liu+ ’18; Vondrick+ ’18]
• Processes a video by propagating color frame-by-frame
• Cannot propagate between scene changes
• Continues amplifying errors
![Page 6: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/6.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Our Method• Model based on spatial and temporal convolutions
• Automatic noise removal, super-resolution, and contrast adjustment
• Semi-automatic colorization source-reference attention• Can colorize a video using an arbitrary number of reference images
RemasteringNet
References
![Page 7: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/7.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Our Network
Pre-processing Network
Source-reference Network
Input
Output
Temporal source-reference attention
Reference images
![Page 8: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/8.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Our Network
Pre-processing Network
Source-reference Network
Input
Output
Temporal source-reference attention
Reference images
![Page 9: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/9.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Pre-processing Network
• Removes artifacts and noise from the input grayscale video
• Formed exclusively by temporal convolutions
Pre-processing Network
Input
Temporal conv.
![Page 10: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/10.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Our Network
Pre-processing Network
Source-reference Network
Input
Output
Temporal source-reference attention
Reference images
![Page 11: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/11.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Source-reference Network• Takes the output of the pre-processing network along with an arbitrary number
of reference color images
• Colorizes the frames based on the reference imagesby using source-reference attention
Reference images
Spatial conv.
Temporal conv.
Source-reference attention
![Page 12: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/12.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Temporal Source-reference Attention
• Compute similarity between the source images and reference images• Actually computed on feature maps
Input frame
References Feature map
source-reference attention
![Page 13: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/13.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Advantages of Source-Reference Attention
Recursion-based network
Our temporal source-reference attention
• Cannot form long-term dependencies• Temporal consistency is lost when a new reference is used• Require precise scene segmentation
Can use all the color reference information for colorization
![Page 14: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/14.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Optimization
• Combination of two L1 losses• Fully supervised learning
• Uses ADADELTA[Zeiler ’12] for optimization
• Objective function:
Output ofpre-processing network
Ground truthluminance
Output ofsource-reference network
Ground truthchrominance
![Page 15: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/15.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Training Data Generation
• Example-based and algorithm-based deterioration• Example-based: scratch noise, fractal noise, dust noise, …
• Algorithm-based: Gaussian noise, blur, low contrast
• 1200 videos from Youtube8M[Abu-El-Haji+ ’16] for training
Examples of noise data Original Deteriorated
![Page 16: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/16.jpg)
Results
![Page 17: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/17.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Comparisons
[Zhang+ ’17b]&[Vondrick+ ’18][Yu+ ’18]&[Zhang+ ’17a]Input Ours
![Page 18: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/18.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Quantitative ResultRemastering results
![Page 19: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/19.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Quantitative Results
Restoration results Colorization results
![Page 20: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/20.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Comparisons
Input [Yu+ ’18]&[Zhang+ ’17a]
[Zhang+ ’17b]&[Vondrick+ ’18] Ours
Reference images(manually created)
![Page 21: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/21.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Results
Attention
Reference images
Input Output
![Page 22: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/22.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Results
Reference images
Attention
Input Output
![Page 23: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/23.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Results
“Isewan typhoon” (1959), the original film is provided by CBC Television Co.
![Page 24: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/24.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Restoration Results
• Large noise removal
Input [Zhang et al. 2017b] Ours
![Page 25: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/25.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Limitations
• Severely deteriorated film is difficult to remaster• Cannot fill large missing regions
• Scene with intense motion
Input Output Input Output
![Page 26: DeepRemaster: Temporal Source-Reference Attention …iizuka.cs.tsukuba.ac.jp/projects/remastering/data/remastering_siggraphasia2019...CONFERENCE 17-20 November 2019 - EXHIBITION18-20](https://reader034.fdocuments.in/reader034/viewer/2022042220/5ec681d575f0810deb1c4090/html5/thumbnails/26.jpg)
CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC, Brisbane, AUSTRALIASA2019.SIGGRAPH.ORG
Conclusion• Novel single framework to tackle entire remastering task
• Automatic noise removal, super-resolution, and contrast adjustment
• Reference-based colorization via temporal source-reference attention
• Significant improvement with respect to existing methods
• Applicable to other reference-based image/video processing
• GitHub: https://github.com/satoshiiizuka/siggraphasia2019_remastering