Wasserstein Training of Deep Boltzmann Machines

Chaoyang Wang 1 Tian Tong 2 Yang Zou 2

AbstractTraining various probabilistic graphical models,such as restricted Boltzmann machines (RBMs),deep Boltzmann machines (DBMs), is usuallydone by maximum likelihood estimation, orequivalently minimizing the Kullback-Leibler(KL) divergence between the data distributionand the model distribution. On the other hand,Wasserstein distance, also known as optimaltransport distance and earth mover’s distance, isa fundamental distance to quantify the differ-ence between probability distributions. Due toits good properties like smoothness and symme-try, Wasserstein distance aroused numerous re-searchers’ interests in machine learning and com-puter vision. In this project, we proposed a gen-eral Wasserstein training method for graphicalmodels by replacing the standard KL-divergencewith Wasserstein distance as novel loss func-tions. And we developed Wasserstein training ofRBMs and DBMs as specific examples. Finally,we experimentally explored Wasserstein trainingof RBMs and DBMs for digit generation withMNIST dataset, and showed the superiority ofWasserstein training compared to traditional KL-divergence training.

1. IntroductionBoltzmann machine is a family of stochastic recurrentneural networks and Markov Random Fields. It has ap-plications in dimensionality reduction, classification, col-laborative filtering, feature learning and topic modelling(Hinton & Salakhutdinov, 2006; Larochelle & Bengio,2008; Salakhutdinov et al., 2007; Coates et al., 2010; Hin-ton & Salakhutdinov, 2009). Boltzmann machines havemany variants, such as Deep Boltzmann Machines (DBMs)(Salakhutdinov & Hinton, 2009), Restricted BoltzmannMachines (RBMs) (Salakhutdinov et al., 2007), Condi-

*Equal contribution 1Robotics Institute 2Department of Elec-trical & Computer Engineering. Correspondence to: Tian Tong<ttong1>, Chaoyang Wang <chaoyanw>, Yang Zou <yzou2>.

tional Restricted Boltzmann Machines (CRBMs) (Taylor &Hinton, 2009), etc.

Learning in Boltzmann machines is usually done by max-imum likelihood estimation, which is equivalent to mini-mizing the Kullback-Leibler Divergence (KL-divergence)between the empirical distribution and the model distribu-tion.

In the world of high dimensions, KL-divergence will fail towork when the two distributions do not have non-negligiblecommon supports, which happens commonly when dealingwith distributions supported by lower dimensional man-ifolds. In such cases, another kind of probability dis-tance, Wasserstein distance, has gained lots of attentionin machine learning and computer vision (Cuturi, 2013),(Doucet, 2014) (Montavon et al., 2016), (Arjovsky et al.,2017) developed a training method for a restricted Boltz-mann machine with the Wasserstein distance as the lossfunction, and showed the power of Wasserstein training inRBM through its application in digit generation, data com-pletion and denoising. The Wasserstein training of RBMleads to better generative properties compared to traditionalRBMs. (Arjovsky et al., 2017) introduced the Wassersteindistance for Generative Adversarial Networks (GAN) toimprove learning stability. (Frogner et al., 2015) definedWassserstein distance as the loss function for supervisedlearning in the label space. Theoretically, the Wassersteindistance is a well-defined distance metric even for non-overlapping distributions, and has the advantages of conti-nuity with respect to the parameters (Arjovsky et al., 2017).

In this project, we propose a general training method tominimize the Wasserstein distance, and discuss specific ex-amples on training the RBMs and DBMs. In experiments,we evaluated the validity of Wasserstein training of theRBMs and DBMs in digit generation with MNIST dataset.The organization of the report is as follows. Section 2gives a brief introduction to the definition of Wassersteindistance, its duality and comparison with KL-divergence.Section 3 presents the theory of Wasserstein training. Insection 4 we focus on Wasserstein RBM while section 5 fo-cuses on Wasserstein DBM. Section 6 considers the prac-tical implementation of Wasserstein training with the KLregularization. In section 7, we experimentally demon-strate Wasserstein Boltzmann machines’ validity in digit

Wasserstein Training of DBM

generation with MNIST dataset. Section 8 gives our finalconclusion.

2. Background2.1. Wasserstein Distance

Given a complete separable metric space (M,d), let p andq be absolutely continuous Borel probability measures onM . Consider π as a joint probability measure on M ×Mwith marginals p and q, also called a transport plan:

π(A×M) = p(A) for any Borel subset A ⊆M ;π(M × C) = q(B) for any Borel subset B ⊆M.

(1)

For d(x, x′) as a distance metric on M ×M , the Wasser-stein distance between p and q is defined as

W (p, q) = minπ∈Π(p,q)

Eπd(x, x′), (2)

where Π(p, q) denotes the marginal polytope of p and q. Atransport plan π is optimal if (2) achieves its infimum. Op-timal transport plans on the Euclidean spaces are charac-terized by the push forward measures. Practically, a linearprogram is usually adopted to solve the Wasserstein dis-tance. However, when the dimension of probability distri-bution domain is high, solving the linear programming be-comes intractable. Recently, (Cuturi, 2013) and (Doucet,2014) proposed an efficient approximation of Wassersteindistances along with their derivatives. The approximationplays an important role in the practical implementation ofthese computations.

2.2. Kantorovich Duality

Under the case that p and q are discrete distributions, theWasserstein distance can be written as a linear program-ming as

W (p, q) = minπ

∑x,x′

π(x, x′)d(x, x′)

s.t.∑x′

π(x, x′) = p(x),∑x

π(x, x′) = q(x′),

π(x, x′) ≥ 0. (3)

Its dual form can be written as

W (p, q) = maxα,β

∑x

α(x)p(x) +∑x′

β(x′)q(x′)

s.t. α(x) + β(x′) ≤ d(x, x′). (4)

The primal and dual forms have an interesting intuitive in-terpretation (Villani, 2008). Consider p(x) as the amount ofbread produced in bakery located at x, q(x′) as the amount

of bread sold in cafe located x′, d(x, x′) as the cost to trans-port unit bread from x to x′, and π(x, x′) as the trans-port plan, i.e., the amount of bread to transport from xto x′. The primal problem can be interpreted as the min-imum cost to transport all bread from bakeries to cafes.Furthermore, assume that there is an agent who wants tocompete for business with the transportation department.He buys the bread from bakery located at x with unit price−α(x), and sells the bread to cafe located at x′ with unitprice β(x′). The agent hopes to keep competitive, mean-ing that his net price is lower than the transportation cost,i.e., α(x) + β(x′) ≤ d(x, x′). Therefore, the dual prob-lem can be interpreted as the maximum profit of the agentwhile keeping competitive. If the agent chooses the optimalpricing scheme, he will gain the same as the transportationdepartment, i.e., the dual achieves the primal.

2.3. Comparison between Wasserstein Distance andKL-divergence

The KL-divergence is defined as

KL(p, q) :=

∫M

log

(p(x)

q(x)

)p(x)dx. (5)

KL-divergence is asymmetric and thus not a valid distance.From the definition of KL-divergence, we know that if thesupports of p and q do not have non-negligible intersec-tions, which means that the probabilities in the two distri-butions cannot be both non-zero except for some negligibleregions, the KL-divergence will become undefined or infi-nite. The following example could illustrates the idea.

Let Y ∼ U [0, 1] be the uniform distribution on the unit in-terval. Let p0 be the distribution of (0, Y ) ∈ R2, uniformon a straight horizontal line segment passing through theorigin; pθ be the distribution of (θ, Y ) with θ as a parame-ter. Fig. 1 illustrates both distributions.

Figure 1. Example for Wasserstein distance vs. KL-divergence.

Wasserstein Training of DBM

The Wasserstein distance between p0 and pθ is

W (p0, pθ) = |θ|, (6)

while the KL-divergence is

KL(p0, pθ) =

{∞ if θ 6= 00 if θ = 0

. (7)

As we could see, the Wasserstein distance is continuouswith respect to θ, while KL-divergence is not.

The above simple example illustrates the idea that if twodistributions are non-overlapping, the Wasserstein dis-tance could provide discriminant information, while KL-divergence fails to do so.

3. Theory of Wasserstein TrainingIn this section, we introduce the theory of Wassersteintraining. First, we tell about computing the Wassersteindistance. Since solving the vanilla Wasserstein distanceinvolves a linear programming and is quite time consum-ing, we turn to a γ-smoothed Wasserstein distance (Cuturi,2013). Given the marginal distributions p(x) and q(x′), theγ-smoothed Wasserstein distance is defined as:

Wγ(p, q) = minπ∈Π(p,q)

Eπd(x, x′)− γH(π), (8)

where Π(p, q) denotes the set of joint distributions withmarginals as p(x) and q(x′), H(π) denotes the Shan-non entropy of π. This is equivalent to considering themaximum-entropy principle. The γ-smoothed Wassersteindistance is differentiable with respect to p and q, whichconsiderably facilitates computations. We will show thatit can be computed for a much lower complexity than thevanilla Wasserstein distance.

For simplicity, assume that p and q take discrete values.The γ-smoothed Wasserstein distance can be written as anoptimization problem:

Wγ(p, q) = minπ

∑x,x′

π(x, x′)(d(x, x′) + γ log π(x, x′))

s.t.∑x′

π(x, x′) = p(x),∑x

π(x, x′) = q(x′).

To solve the problem, the Lagrangian is introduced as

L =∑x,x′

π(x, x′)(d(x, x′) + γ log π(x, x′))

+∑x

α(x)(p(x)−∑x′

π(x, x′))

+∑x′

β(x′)(q(x′)−∑x

π(x, x′)),

where α(x), β(x′) are Lagrange multipliers, also as dualvariables. Set the derivative of L with respect to π(x, x′)as 0:

∂L

∂π(x, x′)= d(x, x′) + γ(1 + log π(x, x′))

−α(x)− β(x′) = 0.

The solution is

π(x, x′) = exp

(1

γ(α(x) + β(x′)− d(x, x′))− 1

).

(9)

Define vectors u(x) = exp(α(x)γ ), v(x′) = exp(β(x′)

γ ),

matrix K(x, x′) = exp(−d(x,x′)γ − 1). The constraints

require that:

u(x)∑x′

K(x, x′)v(x′) = p(x)∑x

u(x)K(x, x′)v(x′) = q(x′). (10)

Under the case where x, x′ take finite values, (10) becomesa matrix equation:

Kv = p./u,KTu = q./v, (11)

which can be solved by iteratively updating u, v until fixedpoints. After that, we can recover α?(x), β?(x′) as

α?(x) = γ log u(x), β?(x′) = γ log v(x′), (12)

and the final result for γ-smoothed Wasserstein distance is

Wγ(p, q) =∑x

α?(x)p(x) +∑x′

β?(x′)q(x′)− γ∑x,x′

exp

(1

γ(α?(x) + β?(x′)− d(x, x′))− 1

).

(13)

Notice that given (α?(x), β?(x′)) as optimal dual vari-ables, for any constant c, (α?(x) − c, β?(x′) + c) are alsooptimal. The optimal dual variables have one extra degreeof freedom, due to the existence of one redundant con-straint. Therefore, we can set c =

∑x α

?(x)p(x), i.e.,require that α?(x) as the centered optimal dual variable,satisfying

∑x α

?(x)p(x) = 0.

After that, we consider the parameter estimation using theγ-smoothed Wasserstein distance. Here we choose p as themodel distribution pθ, while q as the data distribution p =∑Ni=1

1N δxi , where {xi}ni=1 are data points. The sensitive

analysis theorem relates the gradient with respect to pθ(x)to the optimal dual variable α?(x), as

∂Wγ(pθ, p)

∂pθ(x)= α?(x),

Wasserstein Training of DBM

Algorithm 1 Compute Wasserstein distance and optimaldual variables

Input: data xi, samples xj , distance metric d, smoothingparameter γ.Output: Wasserstein distance W (pθ, p), optimal dualvariables α?(xj), β?(xi).D = [d(xj , xi)]; K = exp(D/γ − 1); u = 1/Nwhile u changes or other relevant stopping criterion dov = 1/N./(KTu)u = 1/N./(Kv)

end whileW (pθ, p) = dot(u, (D. ∗K)v)α? = γ log uβ? = γ log v + mean(α?)α? = α? −mean(α?) % centered

therefore, by chain rule, the gradient with respect to θ is:

∂Wγ(pθ, p)

∂θ=∑x

α?(x)∂pθ(x)

∂θ

=∑x

α?(x)∂ log pθ(x)

∂θpθ(x)

= Ex∼pθ(x)

[α?(x)

∂ log pθ(x)

∂θ

]. (14)

Since α?(x) are centered, pθ(x) in (14) actually can be anunnormalized distribution.

Next, we point out that solving α?(x) corresponding toWγ(pθ, p), and computing the expectation in (14), are usu-ally intractable, since x has exponentially many configura-tions under high dimensional cases. In practice, we adoptthe sampling approximation, i.e., draw samples {xj}Nj=1

from the distribution pθ, and replace pθ by an empiricaldistribution pθ =

∑Nj=1

1Nδxj . The optimal dual variable

α?(xj), corresponding toWγ(pθ, p), can be computed witha complexity of order NN through iteratively updates in(11), whose details are described in Algorithm 1. The gra-dient is estimated as:

∂Wγ(pθ, p)

∂θ≈ 1

N

N∑j=1

α?(xj)∂ log pθ(xj)

∂θ. (15)

Notice that in the gradient, the real data involves only indi-rectly through optimal dual variables α?(xj).

Finally, in some probabilistic graphical models like DBMs,it is simpler to consider a joint distribution including hid-den variables, where the marginal distribution is pθ(x) =∑h pθ(x, h). In such cases, the gradient with respect to θ

is:

∂Wγ(pθ, p)

∂θ=∑x

α?(x)∂pθ(x)

∂θ

=∑x

α?(x)∑h

∂pθ(x, h)

∂θ

=∑x

∑h

α?(x)∂ log pθ(x, h)

∂θpθ(x, h)

= E(x,h)∼pθ(x,h)

[α?(x)

∂ log pθ(x, h)

∂θ

].

(16)

By adopting the sampling approximation, we can drawsamples (xj , hj) from the joint distribution pθ(x, h), andestimate the gradient in (16) as

∂Wγ(pθ, p)

∂θ≈ 1

N

N∑j=1

α?(xj)∂ log pθ(xj , hj)

∂θ. (17)

4. Wasserstein Training of RBMIn this section, we implement the Wasserstein training al-gorithm for a restricted Boltzmann machine (RBM). In aRBM, the joint distribution of the observable variable x andthe hidden variables h is

pθ(x, h) =1

Zθexp(cTx+ hTWx+ bTh).

The gradients of log pθ(x) with respect to parameters θ =(W, b, c) are:

∂ log pθ(x)

∂W= σ(Wx+ b)xT − Ex∼pθ(x)[σ(Wx+ b)xT ]

∂ log pθ(x)

∂b= σ(Wx+ b)− Ex∼pθ(x)[σ(Wx+ b)]

∂ log pθ(x)

∂c= x− Ex∼pθ(x)[x].

Plug these results into (15). The second terms vanish dueto the requirement that α?(xi) are centered. The gradientsare estimated as:

∂Wγ(pθ, p)

∂W≈ 1

N

N∑j=1

α?(xj)σ(Wxj + b)xTj (18)

∂Wγ(pθ, p)

∂b≈ 1

N

N∑j=1

α?(xj)σ(Wxj + b) (19)

∂Wγ(pθ, p)

∂c≈ 1

N

N∑j=1

α?(xj)xj . (20)

The samples {xj}Nj=1 are updated using persistent con-trastive divergence (PCD).

Wasserstein Training of DBM

5. Wasserstein Training of DBMIn this section, we implement the Wasserstein training al-gorithm for a deep Boltzmann machine (DBM). Take a twolayer DBM as example. The joint distribution is

pθ(x, h) =1

Zθexp(cTx+ h(1)TW (1)x+ b(1)Th(1)

+ h(2)TW (2)h(1) + b(2)Th(2)).

The gradients of log pθ(x, h) with respect to parametersθ = (W (1),W (2), b(1), b(2), c) are:

∂ log pθ(x, h)

∂W (1)= h(1)xT − E(x,h)∼pθ(x,h)[h

(1)xT ]

∂ log pθ(x, h)

∂W (2)= h(2)h(1)T − E(x,h)∼pθ(x,h)[h

(2)h(1)T ]

∂ log pθ(x, h)

∂b(1)= h(1) − E(x,h)∼pθ(x,h)[h

(1)]

∂ log pθ(x, h)

∂b(2)= h(2) − E(x,h)∼pθ(x,h)[h

(2)]

∂ log pθ(x, h)

∂c= x− E(x,h)∼pθ(x,h)[x].

Plug the results into (17). The second terms vanish. Thegradients are estimated as:

∂Wγ(pθ, p)

∂W (1)≈ 1

N

N∑j=1

α?(xj)h(1)j xTj (21)

∂Wγ(pθ, p)

∂W (2)≈ 1

N

N∑j=1

α?(xj)h(2)j h

(1)Tj (22)

∂Wγ(pθ, p)

∂b(1)≈ 1

N

N∑j=1

α?(xj)h(1)j (23)

∂Wγ(pθ, p)

∂b(2)≈ 1

N

N∑j=1

α?(xj)h(2)j (24)

∂Wγ(pθ, p)

∂c≈ 1

N

N∑j=1

α?(xj)xj . (25)

The samples {xj , h(1)j , h

(2)j }Nj=1 are updated using PCD.

Compared to that the gradients of KL-divergence are com-posed of model expectation terms minus data expectationterms, the gradients of Wasserstein distance does not havethe data expectation terms. We should know in mind thatreal data only influences through the optimal dual variableα?(xj).

6. Stabilization with KL regularizationFor the training based on the pure Wasserstein distance,we notice that it is very likely to be entrapped into local

epoch0 200 400 600 800 1000 1200 1400

cro

ss-e

ntr

opy

300

350

400

450

500

550

trainingvalidation

Figure 2. loss v.s. epoch numbers for pure Wasserstein RBM

epoch0 500 1000 1500 2000 2500 3000

cro

ss-e

ntr

opy

0

500

1000

1500

trainingvalidation

Figure 3. loss v.s. epoch numbers for KL regularized WassersteinRBM

minimum. Based on the observed data x, we generate x′

using one Gibbs step, and calculate the cross-entropy lossbetween x and x′ as a monitor of the training. As shown inFig.2, the loss stops decreasing at a level around 300.

We hypothesize the poor training is due to the fact that theWasserstein gradient mainly depends on the generated sam-ples from the model distribution pθ, only indirectly on thedata distribution p. Suppose the samples generated by themodel strongly differs from data, this becomes a problembecause there is no weighting α?(xj) of the generated sam-ples that can represent the desired direction to a better min-imum. In that case, the Wasserstein gradient will lead to abad local minimum.

To alleviate this issue, we use a hybrid learning of Wasser-stein distance and KL-divergence, which is similar to thatproposed in (Montavon et al., 2016). The learning objectivenow becomes minimizing:

(1− η)W (pθ, p) + ηKL(p, pθ), (26)

where η is the mixing ratio.

Fig.3 shows that by using the proposed KL regularized

Wasserstein Training of DBM

(a) Samples generated from standard RBM (b) Samples generated from Wasserstein RBM

(c) Illustration of W for standard RBM (d) Illustration of W for Wasserstein RBM

Figure 4. Visualization of generated samples and model weights for Wasserstein RBM(η = 0.1) and standard RBM.

Wasserstein learning objective with η = 0.1, the validationcross-entropy loss now decreases to < 200, which is a bigimprovement over the pure Wasserstein distance training.

7. ExperimentsWe evaluate the proposed Wasserstein training for RBMand DBM on MNIST dataset. We strictly follow theMNIST protocol which divides the dataset into a trainingset with 60,000 digits, and a testing set with 10,000 digits.Throughout the experiments, we set the smoothing param-eter γ = 0.1 for the γ-smoothed Wasserstein distance.

7.1. RBM

The results of training with standard KL divergence andwith the Wasserstein distance are compared as following.

In the training of RBM, we use 100 hidden nodes, adopt

the gradient descent using batch size 100, PCD chain size100, start with learning rate 0.01 with adaptive adjustments(RMSprop), and train for at most 5000 epochs with earlystopping.

The learned feature W is shown in Fig. 4(c,d). It revealssome structures such as edges.

Samples drawn from the learned RBMs are shown inFig. 4(a,b). We draw these samples by first randomly ini-tialize the nodes and then perform 1000 Gibbs samplingsteps.

7.2. DBM

The DBM model we used in this experiment consists of twohidden layers: the first hidden layer consists of 1000 hiddennodes; and the second layer has 500 nodes. In the trainingof DBM, we adopt the gradient descent using batch size100, PCD chain size 100, RMSprop to adaptively adjust

Wasserstein Training of DBM

(a) Samples generated from standard DBM (b) Samples generated from Wasserstein DBM

(c) Illustration of W (1) for standard DBM (d) Illustration of W (1) for Wasserstein DBM

Figure 5. Visualization of generated samples and model weights for W-DBM(η = 0.3) and standard DBM.

learning rate, and train for 2000 epochs.

To investigate the effect of mixing ratio parameter η, wetrain DBMs with different η ranging from 0 to 1, and reporttheir performance in terms of Wasserstein distance between10,000 digits sampled form the model and 10,000 digitsfrom the test set. Moreover, to faithfully reflect the perfor-mance of Wasserstein training, we do not perform discrim-inative fine-tuning which is usually done in DBM training.The final values of Wasserstein distances after 2000 epochsof training are shown in Fig. 6.

From Fig. 6, we can see that a mixture of Wassertein-distance and KL-divergence outperforms either pureWasserstein-distance (η = 0) or KL-divergence (η =1). This result empirically shows that though usingWassertein-distance alone is problematic, it is complemen-tary to KL-divergence and a combination of those twoachieves better result for DBM training.

To give a more subjective comparison between the pro-posed Wasserstein-trained DBM (W-DBM, η = 0.3) toDBM trained with KL-divergence, we show the learnedmodel weightsW (1) and digits randomly sampled from W-DBM (Fig. 5, right) and standard DBM (Fig. 5, left).

Finally, in Fig. 7, we visualize the learned W-DBM modeldistribution by projecting 10,000 randomly drawn samplesfrom the model (red dots) into a 2D plane. The projectionis acquired by performing PCA over the data points of thetraining set. As a reference, we also plot the samples fromtest set (blue dots). From the figure, we can see that thoughnot perfectly assembles the whole empirical data distribu-tion, W-DBM does loosely align with the data distributionin many local areas.

Wasserstein Training of DBM

Figure 6. Wasserstein training using different mixing ratio η.When η = 1, it’s equivalent to standard DBM (green star) trainedby KL divergence; When η = 0.3 (orange star), it achieves thelowest W-distance on test set.

-0.06 -0.04 -0.02 0 0.02-0.03

-0.02

-0.01

0

0.01

0.02

0.03

data

W-DBM

Figure 7. 2-D visualization of samples from W-DBM (red dots)and real data (blue dots).

8. ConclusionIn this report, we introduced Wasserstein distance as anovel loss function instead of KL-divergence, and proposeda general training method to minimize the Wasserstein dis-tance between the sample distribution and the model distri-bution for various probabilistic models. By adding a regu-larized entropy term, we found an efficient method to ap-proximate the gradient of γ-smoothed Wasserstein distancewith respect to parameters. We also discussed how to trainfor models containing hidden variables. After that, we de-veloped the Wasserstein training of RBMs and DBMs asspecific examples. Experiments and analysis have shownthe superiority of W-RBMs/DBMs compared to their tradi-tional counterparts.

ReferencesArjovsky, Martin, Chintala, Soumith, and Bottou, Leon.

Wasserstein gan. arXiv preprint arXiv:1701.07875,2017.

Coates, Adam, Lee, Honglak, and Ng, Andrew Y. Ananalysis of single-layer networks in unsupervised featurelearning. Ann Arbor, 1001(48109):2, 2010.

Cuturi, Marco. Sinkhorn distances: Lightspeed computa-tion of optimal transport. In Advances in Neural Infor-mation Processing Systems, pp. 2292–2300, 2013.

Doucet, Arnaud. Fast computation of wasserstein barycen-ters. 2014.

Frogner, Charlie, Zhang, Chiyuan, Mobahi, Hossein,Araya, Mauricio, and Poggio, Tomaso A. Learning witha wasserstein loss. In Advances in Neural InformationProcessing Systems, pp. 2053–2061, 2015.

Hinton, Geoffrey E and Salakhutdinov, Ruslan R. Reduc-ing the dimensionality of data with neural networks. sci-ence, 313(5786):504–507, 2006.

Hinton, Geoffrey E and Salakhutdinov, Ruslan R. Repli-cated softmax: an undirected topic model. In Advancesin neural information processing systems, pp. 1607–1614, 2009.

Larochelle, Hugo and Bengio, Yoshua. Classification us-ing discriminative restricted boltzmann machines. InProceedings of the 25th international conference on Ma-chine learning, pp. 536–543. ACM, 2008.

Montavon, Griegoire, Muller, Klaus-Robert, and Cuturi,Marco. Waterstone training of restricted boltzmann ma-chines. In Advances in Neural Information ProcessingSystems, pp. 3711–3719, 2016.

Salakhutdinov, Ruslan and Hinton, Geoffrey. Deep boltz-mann machines. In Artificial Intelligence and Statistics,pp. 448–455, 2009.

Salakhutdinov, Ruslan, Mnih, Andriy, and Hinton, Geof-frey. Restricted boltzmann machines for collaborativefiltering. In Proceedings of the 24th international con-ference on Machine learning, pp. 791–798. ACM, 2007.

Taylor, Graham W and Hinton, Geoffrey E. Factored condi-tional restricted boltzmann machines for modeling mo-tion style. In Proceedings of the 26th annual interna-tional conference on machine learning, pp. 1025–1032.ACM, 2009.

Villani, Cedric. Optimal transport: old and new, volume338. Springer Science & Business Media, 2008.