Post on 12-Apr-2017
Deep Unsupervised Learning using Nonequlibrium Thermodynamics
Tran Quoc Hoan
@k09ht haduonght.wordpress.com/
14 December 2015, Paper Alert, Hasegawa lab., Tokyo
The University of Tokyo
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, Surya Ganguli Proceedings of the 32nd International Conference on Machine Learning, 2015
Abstract
Deep Unsupervised Learning using Nonequilibrium Thermodynamics 2
“…The essential idea, inspired by non-equilibrium statistical
physics, is to systematically and slowly destroy structure in
a data distribution through an iterative forward diffusion
process. We then learn a reverse diffusion process
that restores structure in data, yielding a highly flexible
and tractable generative model of the data…”
Outline
3
- The promise of deep unsupervised learning
• Motivation
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
- Diffusion processes and time reversal
• Physical intuition
- Derivation and experimental results
• Diffusion probabilistic model
Deep Unsupervised Learning
4
- Novel modalities
• Unknown features/labels
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
- Ex. disease part in medical image
• Expensive labels
• Unpredictable tasks / one shot learning
- Exploratory data analysis
https://www.ceessentials.net/article40.html
Physical Intuition
5
- Destroy structure in data
• Diffusion processes and time reversal
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
- Carefully characterize the destruction
- Learn how to reverse time
Observation 1: Diffusion Destroy Structure
6
Data distribution
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Uniform distribution
Uniform distributionData distribution
(Observation)Diffusion destroys structure
(Recover structure)Recover data distribution by starting from uniform
distribution and running dynamics backwards
Observation 2: Microscopic Diffusion
7
• Time reversible
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
https://www.youtube.com/watch?v=cDcprgWiQEY
• Brownian motion
• Position updates are small Gaussians (both forwards and backwards in time)
Diffusion-based Probabilistic Models
8
• Destroy all structure in data distribution using diffusion process
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Learn reversal of diffusion process
- Estimate function for mean and covariance of each step in the reverse diffusion process (Ex. binomial rate for binary data)
• Reverse diffusion process is the model of the data
Diffusion-based Probabilistic Models
9
• Algorithm
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Deep convolutional network: universal function approximatior
• Multiplying distributions: inputation, denoising, computing posteriors
Destroy by Diffusion Process
10
Datadistribution
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Forwarddiffusion
Noisedistribution
Temporal diffusion rate
Destroy by Gaussian Process
11
Datadistribution
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Forwarddiffusion
Noisedistribution
Decay towards origin
Add small noise
Reversal Gaussian Diffusion Process
12
Datadistribution
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Reversediffusion
Noisedistribution
Learned drift and covariance functions
Case Study: Swiss Roll
13Deep Unsupervised Learning using Nonequilibrium Thermodynamics
True model
Inference model
Training the reverse diffusion
14Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Model probability
Annealed importance sampling
Training the reverse diffusion
15Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Log likelihood
Jensen’s inequality
Training the reverse diffusion
16Deep Unsupervised Learning using Nonequilibrium Thermodynamics
…do some algebra…
Training the reverse diffusion
17Deep Unsupervised Learning using Nonequilibrium Thermodynamics
…for Gaussian diffusion process…
Training
unsupervised learning becomes regression problem
Training the reverse diffusion
18Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Setting the diffusion rate
• For Binomial diffusion (erase constant fraction of stimulus variance each step)
• For Gaussian diffusion
�t
�1
�t = (T � t+ 1)�1
= small constant (prevent over-fitting)
Training �t
Multiplying Distributions
19Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Required to compute posterior distribution - Missing data (inpainting)
- Corrupted data (denoising)
• Difficult and expensive using competing techniques
- Ex. VAE, GSNs, NADEs, most graphical models
Interested in
Acts as small perturbation to diffusion process
Multiplying Distributions
20Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Modified marginal distributions
Interested in
Acts as small perturbation to diffusion process
Multiplying Distributions
21Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Modified diffusion steps
Equilibrium condition
Normalized
Multiplying Distributions
22Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Reversal gaussian Diffusion Process
Interested in
Acts as small perturbation to diffusion process
Small perturbation affects only mean
Deep Network as Approximator for Images
23Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Multi-scale convolution
24Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Downsample
Convolve
Upsample
Sum
Applied to CIFAR-10
25Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Training data Samples from Generative Adversarial [Goodfellow
et al, 2014]
Samples from diffusion model
Applied to CIFAR-10
26Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Samples from DRAW
[Gregor et al, 2015]
Samples from Generative Adversarial [Goodfellow
et al, 2014]
Samples from diffusion model
Applied to Dead Leaves
27Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Training dataSamples from
[Theis et al, 2012]Log likelihood 1.24
bits/pixel
Samples from diffusion model
Log likelihood 1.49 bits/pixel
Applied to Inpainting
28Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Table App.1
29Deep Unsupervised Learning using Nonequilibrium Thermodynamics
References
30Deep Unsupervised Learning using Nonequilibrium Thermodynamics
h"p://jmlr.org/proceedings/papers/v37/sohl-dickstein15.html
h"p://videolectures.net/icml2015_sohl_dickstein_deep_unsupervised_learning/
h"p://www.inference.vc/icml-paper-unsupervised-learning-by-inverEng-diffusion-processes/