![arXiv:2006.01943v1 [cs.CV] 2 Jun 2020 · 2020-06-04 · arXiv:2006.01943v1 [cs.CV] 2 Jun 2020. Figure 1: The generative Ear2Face network learns a mapping between ear and face images.](https://static.fdocuments.in/doc/165x107/5f633b76855124286e0a8fa3/arxiv200601943v1-cscv-2-jun-2020-2020-06-04-arxiv200601943v1-cscv-2.jpg)
Deep Generative Models · BOWMAN, Samuel R., et al. Generating sentences from a continuous space....
88
Deep Generative Models Jakub M. Tomczak AMLAB, UvA 2017/11/29
Transcript of Deep Generative Models · BOWMAN, Samuel R., et al. Generating sentences from a continuous space....
Deep Generative ModelsJakub M. TomczakAMLAB, UvA2017/11/29
Do we need generative modeling?
Do we need generative modeling?
Do we need generative modeling?
Do we need generative modeling?
Do we need generative modeling?
Do we need generative modeling?
Do we need generative modeling?new data
Do we need generative modeling?High probabilityof the blue label.=Highly probabledecision!
new data
Do we need generative modeling?High probabilityof the blue label.=Highly probabledecision!
High probabilityof the blue label.xLow probabilityof the object.=Uncertaindecision!
new data
Generative Modeling● Providing decision is not enough. How to evaluate uncertainty? Distribution of y is only a part of the story.● Generalization problem. Without knowing the distribution of x how we can generalize to new data?● Understanding the problem is crucial (“What I cannot create, I do not understand”, Richard P. Feynman). Properly modeling data is essential to make better decisions.
Generative Modeling● Semi-supervised learning.Use unlabeled data to train a better classifier.
Generative Modeling● Handling missing or distorted data.Reconstruct and/or denoise data.
Generative ModelingImage generation
Real GeneratedCHEN, Xi, et al. Variational lossy autoencoder. arXiv preprint arXiv:1611.02731, 2016.
Generative ModelingSequence generation
Generated
BOWMAN, Samuel R., et al. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349, 2015.
How to formulate a generative model?Modeling in a high-dimensional space is difficult.
How to formulate a generative model?Modeling in a high-dimensional space is difficult.
How to formulate a generative model?Modeling in a high-dimensional space is difficult. → modeling all dependencies among pixels.
How to formulate a generative model?Modeling in a high-dimensional space is difficult. → modeling all dependencies among pixels.very inefficient!
How to formulate a generative model?Modeling in a high-dimensional space is difficult. → modeling all dependencies among pixels.
A possible solution? → Models with latent variablesvery inefficient!
Boltzmann distributionA joint distribution of random variables:
Boltzmann distributionA joint distribution of random variables:
Energy function
Boltzmann distributionA joint distribution of random variables:
Energy functionPartition function
Boltzmann distributionA joint distribution of random variables:
It is called the Boltzmann (or Gibbs) distribution.
Boltzmann distributionA joint distribution of random variables:
Advantages:– Joint distribution is described by E(x).– Well-studied distribution in statistical physics.
Boltzmann distributionA joint distribution of random variables:
Advantages:– Joint distribution is described by E(x).– Well-studied distribution in statistical physics.Drawbacks:– Calculation of partition function requires enumeration of all x, e.g., if x’s are binary → combinations.
Boltzmann MachinesLet us consider the following energy function:where x are binary.
Boltzmann MachinesLet us consider the following energy function:where x are binary.
first-order term(models “prior”)
Boltzmann MachinesLet us consider the following energy function:where x are binary.
second-order term(models correlations) first-order term(models “prior”)
Boltzmann MachinesLet us consider the following energy function:where x are binary.
second-order term(models correlations) first-order term(models “prior”)Very complicated…Can we do better?
Boltzmann MachinesLet us consider the following energy function:where x are binary.
second-order term(models correlations) first-order term(models “prior”)Very complicated…Can we do better? Latent variables!
Restricted Boltzmann MachinesLet us consider the following energy function:where x and h are binary.
Restricted Boltzmann MachinesLet us consider the following energy function:where x and h are binary.
imagehidden unitsweights (receptive field)
Restricted Boltzmann MachinesConditional dependencies:
where
RBM: TrainingWe train RBMs using the log-likelihood function:where
RBM: TrainingWe train RBMs using the log-likelihood function:where Free energy
softplus
RBM: TrainingLet us calculate gradient of the LL wrt parameters:
RBM: TrainingLet us calculate gradient of the LL wrt parameters:
easy difficultIt requires application of an approximate inference (e.g., MCMC).
RBM: Contrastive DivergenceApply k steps of Gibbs sampler and approximate the gradient as follows:
Hinton, G. E. (2002), Training products of experts by minimizing contrastive divergence, Neural Computation, 14(8), 1771-1800
RBM with Gaussian inputs
Subspace RBM
Tomczak, J. M., & Gonczarek, A. (2017). Learning invariant features using Subspace Restricted Boltzmann Machine. Neural Processing Letters, 45(1), 173-182.
Deep Belief Network
Deep Boltzmann Machines
Training DBM
Latent Variable Models● Latent variable model:
Latent Variable Models● Latent variable model:
First sample z.Second, sample x for given z.
Latent Variable Models● Latent variable model:
First sample z.Second, sample x for given z.
Latent Variable Models● Latent variable model:● If and , then → Factor Analysis. ● What if we take a non-linear transformation of z? → an infinite mixture of Gaussians.
Latent Variable Models● Latent variable model:● If and , then → Factor Analysis. ● What if we take a non-linear transformation of z? → an infinite mixture of Gaussians.Convenient but limiting!
Latent Variable Models● Latent variable model:● If and , then → Factor Analysis. ● What if we take a non-linear transformation of z? → an infinite mixture of Gaussians.
Latent Variable Models● Latent variable model:● If and , then → Factor Analysis. ● What if we take a non-linear transformation of z? → an infinite mixture of Gaussians. Neural network
Deep Generative Models (DGM):Density Network
MacKay, D. J., & Gibbs, M. N. (1999). Density networks. Statistics and neural networks: advances at the interface.Oxford University Press, Oxford, 129-144.
DGM: Density NetworkNeural Network
MacKay, D. J., & Gibbs, M. N. (1999). Density networks. Statistics and neural networks: advances at the interface.Oxford University Press, Oxford, 129-144.
DGM: Density NetworkNeural NetworkHow to train this model?!
MacKay, D. J., & Gibbs, M. N. (1999). Density networks. Statistics and neural networks: advances at the interface.Oxford University Press, Oxford, 129-144.
DGM: Density Network● MC approximation:
where:MacKay, D. J., & Gibbs, M. N. (1999). Density networks. Statistics and neural networks: advances at the interface.Oxford University Press, Oxford, 129-144.
DGM: Density Network● MC approximation:
where:MacKay, D. J., & Gibbs, M. N. (1999). Density networks. Statistics and neural networks: advances at the interface.Oxford University Press, Oxford, 129-144.
Sample z many times,apply log-sum-exp trick and maximize log-likelihood.
DGM: Density Network● MC approximation:
where:MacKay, D. J., & Gibbs, M. N. (1999). Density networks. Statistics and neural networks: advances at the interface.Oxford University Press, Oxford, 129-144.
Sample z many times,apply log-sum-exp trick and maximize log-likelihood.It scales badly in high dimensional cases!
DGM: Density Network
MacKay, D. J., & Gibbs, M. N. (1999). Density networks. Statistics and neural networks: advances at the interface.Oxford University Press, Oxford, 129-144.
PROSLog-likelihood approachEasy samplingTraining using gradient-basedmethodsCONSRequires explicit modelsFails in high dim. cases
DGM: Density Network
MacKay, D. J., & Gibbs, M. N. (1999). Density networks. Statistics and neural networks: advances at the interface.Oxford University Press, Oxford, 129-144.
PROSLog-likelihood approachEasy samplingTraining using gradient-basedmethodsCONSRequires explicit modelsFails in high dim. cases
Can we do better?
DGM: so far we haveDensity NetworkDensity Network
Generative Adversarial Net
DGM: so far we haveWorks only for low dim. cases...Inefficient training...Density NetworkDensity Network
Generative Adversarial Net
DGM: so far we haveWorks only for low dim. cases...Works for high dim. cases!Inefficient training...Density NetworkDensity Network
Generative Adversarial Net
DGM: so far we haveWorks only for low dim. cases...Works for high dim. cases!Doesn’t train a distribution...Inefficient training...
Unstable training...Density NetworkDensity Network
Generative Adversarial Net
DGM: so far we haveQUESTIONCan we stick to the log-likelihood approachbut with a simple training procedure?Density NetworkDensity Network
Generative Adversarial Net
DGM: so far we have
Generative Adversarial Net
Density Network
DGM: Variational Auto-EncoderDensity Network
Generative Adversarial NetVariational Auto-Encoder
Kingma, D. P., & Welling, M. (2013). Auto-encoding Variational Bayes. arXiv preprint arXiv:1312.6114.
DGM: Variational Auto-EncoderDensity Network
Generative Adversarial NetVariational Auto-Encoder
Encoder Decoder
Kingma, D. P., & Welling, M. (2013). Auto-encoding Variational Bayes. arXiv preprint arXiv:1312.6114.
DGM: Variational Auto-Encoder
DGM: Variational Auto-EncoderVariational posterior
DGM: Variational Auto-Encoder
Reconstruction error Regularization
DGM: Variational Auto-EncoderOur objective it the evidence lower bound.We can approximate it using MC sample.
DGM: Variational Auto-EncoderOur objective it the evidence lower bound.We can approximate it using MC sample.How to properly calculate gradients (i.e., train the model)?
DGM: Variational Auto-EncoderPROBLEM: calculating gradient wrt parametersof the variational posterior (i.e., sampling process).
Kingma, D. P., & Welling, M. (2013). Auto-encoding Variational Bayes. arXiv preprint arXiv:1312.6114.
DGM: Variational Auto-EncoderPROBLEM: calculating gradient wrt parametersof the variational posterior (i.e., sampling process).SOLUTION: use a non-centered parameterization(a.k.a. reparameterization trick).
Kingma, D. P., & Welling, M. (2013). Auto-encoding Variational Bayes. arXiv preprint arXiv:1312.6114.
DGM: Variational Auto-EncoderPROBLEM: calculating gradient wrt parametersof the variational posterior (i.e., sampling process).SOLUTION: use a non-centered parameterization(a.k.a. reparameterization trick).
Output of a neural networkKingma, D. P., & Welling, M. (2013). Auto-encoding Variational Bayes. arXiv preprint arXiv:1312.6114.
DGM: Variational Auto-Encoder
DGM: Variational Auto-EncoderA deep neural net that outputs parametersof the variational posterior (encoder):
DGM: Variational Auto-EncoderA deep neural net that outputs parametersof the generator (decoder), e.g., a normal distribution or Bernoulli distribution.
DGM: Variational Auto-EncoderA prior that regularizes the encoder andtakes part in the generative process.
DGM: Variational Auto-Encoder
DGM: Variational Auto-EncoderFeedforward netsConvolutional netsPixelCNNGated PixelCNN
DGM: Variational Auto-EncoderNormalizing flowsVolume-preserving flowsGaussian processesStein Particle DescentOperator VI
Feedforward netsConvolutional netsPixelCNNGated PixelCNN
DGM: Variational Auto-EncoderNormalizing flowsVolume-preserving flowsGaussian processesStein Particle DescentOperator VI
Feedforward netsConvolutional netsPixelCNNGated PixelCNNAuto-regressive PriorObjective PriorStick-Breaking PriorVampPrior
DGM: Variational Auto-EncoderNormalizing flowsVolume-preserving flowsGaussian processesStein Particle DescentOperator VI
Feedforward netsConvolutional netsPixelCNNGated PixelCNNAuto-regressive PriorObjective PriorStick-Breaking PriorVampPriorImportance Weighted AERenyi DivergenceStein Divergence
DGM: VAEPROSLog-likelihood frameworkEasy samplingTraining using gradient-basedmethodsStable trainingDiscovers latent representationCould be easily combined withother probabilistic frameworks
CONSOnly explicit modelsProduces blurry images(?)
In order to make betterdecisions, we need abetter understandingof reality.=generative modeling
Web-page:https://jmtomczak.github.ioCode on github:https://github.com/jmtomczakContact:[email protected]@gmail.com
Part of the presented research was funded by the European Commission within the Marie Skłodowska-Curie Individual Fellowship (Grant No. 702666, ''Deep learning and Bayesian inference for medical imaging'').
![PixelSNAIL: An Improved Autoregressive Generative Model · pixelcnn with discretized logistic mixture likelihood and other modications. arXiv preprint arXiv:1701.05517 , 2017. [12]](https://static.fdocuments.in/doc/165x107/5f6203b3e488741798165d6a/pixelsnail-an-improved-autoregressive-generative-model-pixelcnn-with-discretized.jpg){kind=tracking}
![of Information Science and Technology. arXiv:1612.08169v1 ... · the updated appearances, we formulate a spatio-temporal graphical model ... [cs.CV] 24 Dec 2016. Keywords: unsupervised](https://static.fdocuments.in/doc/165x107/603316626f4066547d79da0f/of-information-science-and-technology-arxiv161208169v1-the-updated-appearances.jpg)
![Coupled Generative Adversarial Networks29th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain. arXiv:1606.07536v2 [cs.CV] 20 Sep 2016 We apply CoGAN](https://static.fdocuments.in/doc/165x107/5fbccb9b51085d578115367f/coupled-generative-adversarial-networks-29th-conference-on-neural-information-processing.jpg)
![arXiv:1703.06412v2 [cs.CV] 26 Mar 2017 · marcus.liwicki@unifr.ch, afzal@iupr.com Abstract In this work, we present the Text Conditioned Auxiliary Classifier Generative Adversarial](https://static.fdocuments.in/doc/165x107/5f56142c0e75ae771c4052f6/arxiv170306412v2-cscv-26-mar-2017-marcusliwickiunifrch-afzaliuprcom-abstract.jpg)
![arXiv:1901.05103v1 [cs.CV] 16 Jan 2019groups.csail.mit.edu/robotics-center/public_papers/Park19.pdf · Generative Adversial Networks. GANs [21] and their variants [13, 41] learn deep](https://static.fdocuments.in/doc/165x107/5ec604865638540e6d6ee443/arxiv190105103v1-cscv-16-jan-generative-adversial-networks-gans-21-and-their.jpg)
![ABSTRACT arXiv:1710.10772v1 [cs.LG] 30 Oct 2017 · INTRODUCTION Generative Adversarial Networks [1] demonstrate high perfor-mance within the class of generative models. ... state-of-the-art](https://static.fdocuments.in/doc/165x107/5b5c03df7f8b9ad2198b85a4/abstract-arxiv171010772v1-cslg-30-oct-2017-introduction-generative-adversarial.jpg)
