Wasserstein Adversarial Examples via Projected Sinkhorn Iterations

Eric Wong 1 Frank R. Schmidt 2 J. Zico Kolter 3 4

AbstractA rapidly growing area of work has studied the ex-istence of adversarial examples, datapoints whichhave been perturbed to fool a classifier, but thevast majority of these works have focused primar-ily on threat models defined by `p norm-boundedperturbations. In this paper, we propose a newthreat model for adversarial attacks based on theWasserstein distance. In the image classificationsetting, such distances measure the cost of mov-ing pixel mass, which naturally cover “standard”image manipulations such as scaling, rotation,translation, and distortion (and can potentiallybe applied to other settings as well). To generateWasserstein adversarial examples, we develop aprocedure for projecting onto the Wasserstein ball,based upon a modified version of the Sinkhorn it-eration. The resulting algorithm can successfullyattack image classification models, bringing tra-ditional CIFAR10 models down to 3% accuracywithin a Wasserstein ball with radius 0.1 (i.e.,moving 10% of the image mass 1 pixel), and wedemonstrate that PGD-based adversarial trainingcan improve this adversarial accuracy to 76%. Intotal, this work opens up a new direction of studyin adversarial robustness, more formally consid-ering convex metrics that accurately capture theinvariances that we typically believe should ex-ist in classifiers. Code for all experiments in thepaper is available at https://github.com/locuslab/projected sinkhorn.

1. IntroductionA substantial effort in machine learning research has gone to-wards studying adversarial examples (Szegedy et al., 2014),commonly described as datapoints that are indistinguishable

1Machine Learning Department, Carnegie Mellon Univer-sity, Pittsburgh, Pennsylvania, USA 2Bosch Center for ArtificialIntelligence, Renningen, Germany 3Computer Science Depart-ment, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA4Bosch Center for Artificial Intelligence, Pittsburgh, Pennsylvania,USA. Correspondence to: Eric Wong <ericwong@cs.cmu.edu>.

Preprint.

+ ∆W=

+ ∆∞=

Figure 1. A minimal example exemplifying the difference betweenWasserstein perturbations and `∞ perturbations on an image withsix pixels. The top example utilizes a perturbation ∆W to shiftthe image one pixel to the right, which is small with respect toWasserstein distance since each pixel moved a minimal amount,but large with respect to `∞ distance since each pixel changeda maximal amount. In contrast, the bottom example utilizes aperturbation ∆∞ which changes all pixels to be grayer. This issmall with respect to `∞ distance, since each pixel changes by asmall amount, but large with respect to Wasserstein distance, sincethe mass on each pixel on the left had to move halfway across theimage to the right.

from “normal” examples, but are specifically perturbed tobe misclassified by machine learning systems. This notionof indistinguishability, later described as the threat modelfor attackers, was originally taken to be `∞ bounded pertur-bations, which model a small amount of noise injected toeach pixel (Goodfellow et al., 2015). Since then, subsequentwork on understanding, attacking, and defending againstadversarial examples has largely focused on this `∞ threatmodel and its corresponding `p generalization. While the `pball is a convenient source of adversarial perturbations, it isby no means a comprehensive description of all possible ad-versarial perturbations. Other work (Engstrom et al., 2017)has looked at perturbations such as rotations and transla-tions, but beyond these specific transforms, there has beenlittle work considering broad classes of attacks beyond the`p ball.

In this paper, we propose a new type of adversarial pertur-bation that encodes a general class of attacks that is funda-mentally different from the `p ball. Specifically, we proposean attack model where the perturbed examples are boundedin Wasserstein distance from the original example. Thisdistance can be intuitively understood for images as thecost of moving around pixel mass to move from one imageto another. Note that the Wasserstein ball and the `p ballcan be quite different in their allowable perturbations: ex-amples that are close in Wasserstein distance can be quitefar in `p distance, and vice versa (a pedagogical exampledemonstrating this is in Figure 1).

arX

iv:1

902.

0790

6v2

[cs

.LG

] 1

8 Ja

n 20

20

Wasserstein Adversarial Examples via Projected Sinkhorn Iterations

We develop this idea of Wasserstein adversarial examplesin two main ways. Since adversarial examples are typi-cally best generated using variants of projected gradientdescent, we first derive an algorithm that projects onto theWasserstein ball. However, performing an exact projectionis computationally expensive, so our main contribution hereis to derive a fast method for approximate projection. Theprocedure can be viewed as a modified Sinkhorn iteration,but with a more complex set of update equations. Second,we develop efficient methods for adversarial training un-der this threat method. Because this involves repeatedlyrunning this projection within an inner optimization loop,speedups that use a local transport plan are particularly cru-cial (i.e. only moving pixel mass to nearby pixels), makingthe projection complexity linear in the image size.

We evaluate the attack quality on standard models, showingfor example that we can reduce the adversarial accuracy ofa standard CIFAR10 classifier from 94.7% to 3% using aWasserstein ball of radius 0.1 (equivalent to moving 10%of the mass of the image by one pixel), whereas the sameattack reduces the adversarial accuracy of a model certi-fiably trained against `∞ perturbations from 66% to 61%.In contrast, we show that with adversarial training, we areable to improve the adversarial accuracy of this classifierto 76% while retaining a nominal accuracy of 80.7%. Weadditionally show, however, that existing certified defensescannot be easily extended to this setting; building modelsprovably robust to Wasserstein attacks will require funda-mentally new techniques. In total, we believe this workhighlights a new direction in adversarial examples: convexperturbation regions which capture a much more intuitiveform of structure in their threat model, and which movetowards a more “natural” notion of adversarial attacks.

2. Background and Related WorkMuch of the work in adversarial examples has focused onthe original `∞ threat model presented by Goodfellow et al.(2015), some of which also extends naturally to `p pertur-bations. Since then, there has been a plethora of papersstudying this threat model, ranging from improved attacks,heuristic and certified defenses, and verifiers. As there arefar too many to discuss here, we highlight a few which arethe most relevant to this work.

The most commonly used method for generating adversarialexamples is to use a form of projected gradient descent overthe region of allowable perturbations, originally referred toas the Basic Iterative Method (Kurakin et al., 2017). Sincethen, there has been a back-and-forth of new heuristic de-fenses followed by more sophisticated attacks. To name afew, distillation was proposed as a defense but was defeated(Papernot et al., 2016; Carlini & Wagner, 2017), realistictransformations seen by vehicles were thought to be safe un-

til more robust adversarial examples were created (Lu et al.,2017; Athalye et al., 2018b), and many defenses submittedto ICLR 2018 were broken before the review period evenfinished (Athalye et al., 2018a). One undefeated heuristicdefense is to use the adversarial examples in adversarialtraining, which has so far worked well in practice (Madryet al., 2018). While this method has traditionally been usedfor `∞ and `2 balls (and has a natural `p generalization), inprinciple, the method can be used to project onto any kindof perturbation region.

Another set of related papers are verifiers and provabledefenses, which aim to produce (or train on) certificatesthat are provable guarantees of robustness against adver-sarial attacks. Verification methods are now applicableto multi-layer neural networks using techniques rangingfrom semi-definite programming relaxations (Raghunathanet al., 2018), mixed integer linear programming (Tjenget al., 2019), and duality (Dvijotham et al., 2018). Prov-able defenses are able to tie verification into training non-trivial deep networks by backpropagating through certifi-cates, which are generated with duality-based bounds (Wong& Kolter, 2018; Wong et al., 2018), abstract interpreta-tions (Mirman et al., 2018), and interval bound propagation(Gowal et al., 2018). These methods have subsequentlyinspired new heuristic training defenses, where the resultingmodels can be independently verified as robust (Croce et al.,2018; Xiao et al., 2019). Notably, some of these approachesare not overly reliant on specific types of perturbations (e.g.duality-based bounds). Despite their generality, these cer-tificates have only been trained and evaluated in the contextof `∞ and `2 balls, and we believe this is due in large partto a lack of alternatives.

Highly relevant to this work are attacks that lie outside thetraditional `p ball of imperceptible noise. For example, sim-ple rotations and translations form a fairly limited set ofperturbations that can be quite large in `p norm, but aresometimes sufficient in order to fool classifiers (Engstromet al., 2017). On the other hand, adversarial examples thatwork in the real world do not necessarily conform to the no-tion of being “imperceptible”, and need to utilize a strongeradversary that is visible to real world systems. Some exam-ples include wearing adversarial 3D printed glasses to foolfacial recognition (Sharif et al., 2017), the use of adversarialgraffiti to attack traffic sign classification (Eykholt et al.,2018), and printing adversarial textures on objects to attackimage classifiers (Athalye et al., 2018b). While Sharif et al.(2017) allows perturbations that are physical glasses, theothers use an `p threat model with a larger radius, when adifferent threat model could be a more natural descriptionof adversarial examples that are perceptible on camera.

Last but not least, our paper relies heavily on the Wassersteindistance, which has seen applications throughout machine

Wasserstein Adversarial Examples via Projected Sinkhorn Iterations

learning. The traditional notion of Wasserstein distance hasthe drawback of being computationally expensive: com-puting a single distance involves solving an optimal trans-port problem (a linear program) with a number of variablesquadratic in the dimension of the inputs. However, it wasshown that by subtracting an entropy regularization term,one can compute approximate Wasserstein distances ex-tremely quickly using the Sinkhorn iteration (Cuturi, 2013),which was later shown to run in near-linear time (Altschuleret al., 2017). Relevant but orthogonal to our work, is thatof Sinha et al. (2018) on achieving distributional robust-ness using the Wasserstein distance. While we both use theWasserstein distance in the context of adversarial training,the approach is quite different: Sinha et al. (2018) use theWasserstein distance to perturb the underlying data distribu-tion, whereas we use the Wasserstein distance as an attackmodel for perturbing each example.

Contributions This paper takes a step back from using `pas a perturbation metric, and proposes using the Wassersteindistance instead as an equivalently general but qualitativelydifferent way of generating adversarial examples. To tacklethe computational complexity of projecting onto a Wasser-stein ball, we use ideas from the Sinkhorn iteration (Cuturi,2013) to derive a fast method for an approximate projection.Specifically, we show that subtracting a similar entropy-regularization term to the projection problem results in aSinkhorn-like algorithm, and using local transport plansmakes the procedure tractable for generating adversarialimages. In contrast to `∞ and `2 perturbations, we findthat the Wasserstein metric generates adversarial exampleswhose perturbations have inherent structure reflecting theactual image itself (see Figure 2 for a comparison). Wedemonstrate the efficacy of this attack on standard models,models trained against this attack, and provably robust mod-els (against `∞ attacks) on MNIST and CIFAR10 datasets.While the last of these models are not trained to be robustspecifically against this attack, we observe that that some(but not all) robustness empirically transfers over to pro-tection against the Wasserstein attack. More importantly,we show that while the Wasserstein ball does fit naturallyinto duality based frameworks for generating and trainingagainst certificates, there is a fundamental roadblock pre-venting these methods from generating non-vacuous boundson Wasserstein balls.

3. PreliminariesPGD-based adversarial attacks The most commonmethod of creating adversarial examples is to use a vari-ation of projected gradient descent. Specifically, let (x, y)be a datapoint and its label, and let B(x, ε) be some ballaround x with radius ε, which represents the threat modelfor the adversary. We first define the projection operator

+ =

+ =

0 1 −1 1 0 1

Figure 2. A comparison of a Wasserstein (top) vs an `∞ (bottom)adversarial example for an MNIST classifier (for ε = 0.4 and0.3 respectively), showing the original image (left), the addedperturbation (middle), and the final perturbed image (right). Wefind that the Wasserstein perturbation has a structure reflectingthe actual content of the image, whereas the `∞ perturbation alsoattacks the background pixels.

onto B(x, ε) to be

projB(x,ε)

(w) = arg minz∈B(x,ε)

‖w − z‖22 (1)

which finds the point closest (in Euclidean space) to theinput w that lies within the ball B(x, ε). Then, for somestep size α and some loss ` (e.g. cross-entropy loss), thealgorithm consists of the following iteration:

x(t+1) = projB(x,ε)

(x(t) + arg max

‖v‖≤αvT∇`(x(t), y)

)(2)

where x(0) = x or any randomly initialized point withinB(x, ε). This is sometimes referred to as projected steepestdescent, which is used to generated adversarial examplessince the standard gradient steps are typically too small. Ifwe consider the `∞ ball B∞(x, ε) = {x+ ∆ : ‖∆‖∞ ≤ ε}and use steepest descent with respect to the `∞ norm, thenwe recover the Basic Iterative Method originally presentedby Kurakin et al. (2017).

Adversarial training One of the heuristic defenses thatworks well in practice is to use adversarial training witha PGD adversary. Specifically, instead of minimizing theloss evaluated at a example x, we minimize the loss onan adversarially perturbed example xadv, where xadv isobtained by running the projected gradient descent attackfor the ball B(x, ε) for some number of iterations, as shownin Algorithm 1. Taking B(x, ε) to be an `∞ ball recoversthe procedure used by Madry et al. (2018).

Wasserstein distance Finally, we define the most crucialcomponent of this work, an alternative metric from `p dis-tances. The Wasserstein distance (also referred to as theEarth mover’s distance) is an optimal transport problem thatcan be intuitively understood in the context of distributionsas the minimum cost of moving probability mass to changeone distribution into another. When applied to images, this

Wasserstein Adversarial Examples via Projected Sinkhorn Iterations

Algorithm 1 An epoch of adversarial training for a lossfunction `, classifier fθ with parameters θ, and step sizeparameter α for some ball B.

input: Training data (xi, yi), i = 1 . . . nfor i = 1 . . . n do

// Run PGD adversaryxadv := xifor t = 1 . . . T doδ := arg max‖v‖≤α v

T∇`(xadv, yi)xadv := projB(xi,ε) (xadv + δ)

end for// Backpropagate with xadv , e.g. with SGDUpdate θ with∇`(fθ(xadv), yi)

end for

can be interpreted as the cost of moving pixel mass fromone pixel to another another, where the cost increases withdistance.

More specifically, let x, y ∈ Rn+ be two non-negative datapoints such that

∑i xi =

∑j yj = 1, so images and other

inputs need to be normalized, and let C ∈ Rn×n+ be somenon-negative cost matrix where Cij encodes the cost ofmoving mass from xi to yj . Then, the Wasserstein distancedW between x and y is defined to be

dW(x, y) = minΠ∈Rn×n

+

〈Π, C〉

subject to Π1 = x, ΠT 1 = y

(3)

where the minimization over transport plans Π, whose en-tries Πij encode how the mass moves from xi to yj . Then,we can define the Wasserstein ball with radius ε as

BW(x, ε) = {x+ ∆ : dW(x, x+ ∆) ≤ ε} (4)

4. Wasserstein Adversarial ExamplesThe crux of this work relies on offering a fundamentallydifferent type of adversarial example from typical, `p per-turbations: the Wasserstein adversarial example.

4.1. Projection onto the Wasserstein Ball

In order to generate Wasserstein adversarial examples, wecan run the projected gradient descent attack from Equation(2), dropping in the Wasserstein ball BW from Equation(4) in place of B. However, while projections onto regionssuch as `∞ and `2 balls are straightforward and have closedform computations, simply computing the Wasserstein dis-tance itself requires solving an optimization problem. Thus,the first natural requirement to generating Wasserstein ad-versarial examples is to derive an efficient way to projectexamples onto a Wasserstein ball of radius ε. Specifically,projecting w onto the Wasserstein ball around x with radius

ε and transport cost matrix C can be written as solving thefollowing optimization problem:

minimizez∈Rn

+,Π∈Rn×n+

1

2‖w − z‖22

subject to Π1 = x, ΠT 1 = z

〈Π, C〉 ≤ ε

(5)

While we could directly solve this optimization problem(using an off-the-shelf quadratic programming solver), thisis prohibitively expensive to do for every iteration of pro-jected gradient descent, especially since there is a quadraticnumber of variables. However, Cuturi (2013) showed thatthe standard Wasserstein distance problem from Equation(3) can be approximately solved efficiently by subtractingan entropy regularization term on the transport plan W , andusing the Sinkhorn-Knopp matrix scaling algorithm. Mo-tivated by these results, instead of solving the projectionproblem in Equation (5) exactly, the key contribution thatallows us to do the projection efficiently is to instead solvethe following entropy-regularized projection problem:

minimizez∈Rn

+,Π∈Rn×n+

1

2‖w − z‖22 +

1

λ

∑ij

Πij log(Πij)

subject to Π1 = x, ΠT 1 = z

〈Π, C〉 ≤ ε.

(6)

Although this is an approximate projection onto the Wasser-stein ball, importantly, the looseness in the approximation isonly in finding the projection z which is closest (in `2 norm)to the original example x. All feasible points, including theoptimal solution, are still within the actual ε-Wassersteinball, so examples generated using the approximate projec-tion are still within the Wasserstein threat model.

Using the method of Lagrange multipliers, we can introducedual variables (α, β, ψ) and derive an equivalent dual prob-lem in Lemma 3 (the proof is deferred to Appendix A.1).

Lemma 1. The dual of the entropy-regularized Wassersteinprojection problem in Equation (6) is

maximizeα,β∈Rn,ψ∈R+

g(α, β, ψ) (7)

where

g(α, β, ψ) =− 1

2λ‖β‖22 − ψε+ αTx+ βTw

−∑ij

exp(αi) exp(−ψCij − 1) exp(βj)(8)

Note that the dual problem here differs from the traditionaldual problem for Sinkhorn iterates by having an additional

Wasserstein Adversarial Examples via Projected Sinkhorn Iterations

quadratic term on β and an additional dual variable ψ.Nonetheless, we can still derive a Sinkhorn-like algorithmby performing block coordinate ascent over the dual vari-ables (the full derivation can be found in Appendix A.3).Specifically, maximizing g with respect to α results in

arg maxαi

g(α, β, ψ) =

log (xi)− log

∑j

exp(−ψCij − 1) exp(βj)

,(9)

which is identical (up to a log transformation of variables)to the original Sinkhorn iterate proposed in Cuturi (2013).The maximization step for β can also be done analyticallywith

arg maxβj

g(α, β, ψ) =

λwj −W

(λ exp(λwj)

∑i

exp(αi) exp(−ψCij − 1)

)(10)

whereW is the LambertW function, which is defined as theinverse of f(x) = xex. Finally, since ψ cannot be solvedfor analytically, we can perform the following Newton step

ψ′ = ψ − t · ∂g/∂ψ

∂2g/∂ψ2(11)

where

∂g/∂ψ = −ε+∑ij

exp(αi)Cij exp(−ψCij) exp(βj)

∂2g/∂ψ2 = −∑ij

exp(αi)C2ij exp(−ψCij) exp(βj)

(12)

and where t is small enough such that ψ′ ≥ 0. Once wehave solved the dual problem, we can recover the primalsolution (to get the actual projection), which is described inLemma 4 and proved in Appendix A.2.

Lemma 2. Suppose α∗, β∗, ψ∗ maximize the dual problemg in Equation (16). Then,

z∗i = wi − βi/λΠ∗ij = exp(α∗i ) exp(−ψ∗Cij − 1) exp(β∗j )

(13)

are the corresponding solutions that minimize the primalproblem in Equation (6).

The whole algorithm can then be vectorized and imple-mented as Algorithm 2, which we call projected Sinkhorniterates. The algorithm uses a simple line search to ensurethat the constraint ψ ≥ 0 is not violated. Each iterationhas 8 O(n2) operations (matrix-vector product or matrix-matrix element-wise product), in comparison to the originalSinkhorn iteration which has 2 matrix-vector products.

Algorithm 2 Projected Sinkhorn iteration to project x ontothe ε Wasserstein ball around y. We use · to denote element-wise multiplication. The log and exp operators also applyelement-wise.

input: x,w ∈ Rn, C ∈ Cn×n, λ ∈ RInitialize αi, βi := log(1/n) for i = 1, . . . , n and ψ := 1u, v := exp(α), exp(β)while α, β, ψ not converged do

// update KKψ := exp(−ψC − 1)

// block coordinate descent iteratesα := log(x)− log(Kψv)u := exp(α)β := λw −W

(uTKψ · λ exp(λw)

)v := exp(β)

// Newton stepg := −ε+ uT (C ·Kψ)vh := −uT (C · C ·Kψ)v

// ensure ψ ≥ 0α := 1while ψ − αg/h < 0 doα := α/2

end whileψ := ψ − αg/h

end whilereturn: w − β/λ

Matrix scaling interpretation The original Sinkhorn it-eration has a natural interpretation as a matrix scaling al-gorithm, iteratively rescaling the rows and columns of amatrix to achieve the target distributions. The ProjectedSinkhorn iteration has a similar interpretation: while theα step rescales the rows of exp(−ψC − 1) to sum to x,the β step rescales the columns of exp(−ψC − 1) to sumto −β/λ + w, which is the primal transformation of theprojected variable z at optimality as described in Lemma4. Lastly, the ψ step can be interpreted as correcting forthe transport cost of the current scaling: the numerator ofthe Newton step is simply the difference between the trans-port cost of the current matrix scaling and the maximumconstraint ε. A full derivation of the algorithm and a moredetailed explanation on this interpretation can be found inAppendix A.3.

4.2. Local Transport Plans

The quadratic runtime dependence on input dimension cangrow quickly, and this is especially true for images. Ratherthan allowing transport plans to move mass to and fromany pair of pixels, we instead restrict the transport plan to

Wasserstein Adversarial Examples via Projected Sinkhorn Iterations

Table 1. Classification accuracies for models used in the experi-ments.

DATA SET MODEL NOMINAL ACCURACY

MNIST STANDARD 98.90%BINARIZE 98.73%ROBUST 98.20%ADV. TRAINING 96.95%

CIFAR10 STANDARD 94.70%ROBUST 66.33%ADV. TRAINING 80.69%

move mass only within a k × k region of the originatingpixel, similar in spirit to a convolutional filter. As a result,the cost matrix C only needs to define the cost within ak×k region, and we can utilize tools used for convolutionalfilters to efficiently apply the cost to each k × k region.This reduces the computational complexity of each iterationto O(nk2). For images with more than one channel, wecan use the same transport plan for each channel and onlyallow transport within a channel, so the cost matrix remainsk × k. For 5 × 5 local transport plans on CIFAR10, theprojected Sinkhorn iterates typically converge in around30-40 iterations, taking about 0.02 seconds per iterationon a Titan X for minibatches of size 100. Note that if weuse a cost matrix C that reflects the 1-Wasserstein distance,then this problem could be solved even more efficientlyusing Kantrovich duality, however we use this formulationto enable more general p-Wasserstein distances, or evennon-standard cost matrices.

Projected gradient descent on the Wasserstein ballWith local transport plans, the method is fast enough to beused within a projected gradient descent routine to generateadversarial examples on images, and further used for ad-versarial training as in Algorithm 1 (using steepest descentwith respect to `∞ norm), except that we do an approximateprojection onto the Wasserstein ball using Algorithm 2.

5. ResultsIn this section, we run the Wasserstein examples througha range of typical experiments in the literature of adver-sarial examples. Table 1 summarizes the nominal er-ror rates obtained by all considered models. All exper-iments can be run on a single GPU, and all code forthe experiments is available at https://github.com/locuslab/projected sinkhorn.

Architectures For MNIST we used the convolutionalReLU architecture used in Wong & Kolter (2018), withtwo convolutional layers with 16 and 32 4× 4 filters each,followed by a fully connected layer with 100 units, whichachieves a nominal accuracy of 98.89%. For CIFAR10 we

+ =

standard, ε = 0.53

binary, ε = 0.44

`∞ robust, ε = 0.78

adv. training, ε = 0.86

Figure 3. Wasserstein adversarial examples on the MNIST datasetfor the four different models. Note that the `∞ robust and theadversarially trained models require a much larger ε radius forthe Wasserstein ball in order to generate an adversarial example.Each model classifies the corresponding perturbed example as an 8instead of a 5, except for the first one which classifies the perturbedexample as a 6.

focused on the standard ResNet18 architecture (He et al.,2016), which achieves a nominal accuracy of 94.76%.

Hyperparameters For all experiments in this section, wefocused on using 5× 5 local transport plans for the Wasser-stein ball, and used an entropy regularization constant of1000 for MNIST and 3000 for CIFAR10. The cost matrixused for transporting between pixels is taken to be the 2-norm of the distance in pixel space (e.g. the cost of goingfrom pixel (i, j) to (k, l) is

√|i− j|2 + |k − l|2), which

makes the optimal transport cost a metric more formallyknown as the 1-Wasserstein distance. For more extensiveexperiments on using different sizes of transport plans, dif-ferent regularization constants, and different cost matrices,we direct the reader to Appendix C.

Evaluation at test time We use the follow evaluation pro-cedure to attack models with projected gradient descent onthe Wasserstein ball. For each MNIST example, we startwith ε = 0.3 and increase it by a factor of 1.1 every 10iterations until either an adversarial example is found oruntil 200 iterations have passed, allowing for a maximumperturbation radius of ε = 2. For CIFAR10, we start withε = 0.001 and increase it by a factor of 1.17 until eitherand adversarial example is found or until 400 iterationshave passed, allowing for a maximum perturbation radiusof ε = 0.53.

5.1. MNIST

For MNIST, we consider a standard model, a model withbinarization, a model provably robust to `∞ perturbationsof at most ε = 0.1, and an adversarially trained model. Weprovide a visual comparison of the Wasserstein adversarialexamples generated on each of the four models in Figure3. The susceptibility of all four models to the Wassersteinattack is plotted in Figure 4.

Wasserstein Adversarial Examples via Projected Sinkhorn Iterations

0.0 0.5 1.0 1.5ε radius for Wasserstein ball

0.0

0.2

0.4

0.6

0.8

1.0Ad

vers

aria

l acc

urac

y

standardℓ∞ robustadv. trainingbinarize

Figure 4. Adversarial accuracy of various models on MNIST whenattacked by a Wasserstein adversary over varying sizes of ε-Wasserstein balls. We find that all models not trained with adver-sarial training against this attack eventually achieve 0% accuracy,however we do observe that models trained to be provably ro-bust against `∞ perturbations are still somewhat more robust thanstandard models, or models utilizing binarization as a defense.

Standard model and binarization For MNIST, despiterestricting the transport plan to local 5× 5 regions, a stan-dard model is easily attacked by Wasserstein adversarialexamples. In Figure 4, we see that Wasserstein attacks withε = 0.5 can successfully attack a typical MNIST classifier50% of the time, which goes up to 94% for ε = 1. A Wasser-stein radius of ε = 0.5 can be intuitively understood as mov-ing 50% of the pixel mass over by 1 pixel, or alternativelymoving less than 50% of the pixel mass more than 1 pixel.Furthermore, while preprocessing images with binarizationis often seen as a way to trivialize adversarial examples onMNIST, we find that it performs only marginally better thanthe standard model against Wasserstein perturbations.

`∞ robust model We also run the attack on the modeltrained by Wong et al. (2018), which is guaranteed to beprovably robust against `∞ perturbations with ε ≤ 0.1.While not specifically trained against Wasserstein perturba-tions, in Figure 4 we find that it is substantially more robustthan either the standard or the binarized model, requiring asignificantly larger ε to have the same attack success rate.

Adversarial training Finally, we apply this attack as aninner procedure within an adversarial training frameworkfor MNIST. To save on computation, during training weadopt a weaker adversary and use only 50 iterations of pro-jected gradient descent. We also let ε grow within a rangeand train on the first adversarial example found (essentiallya budget version of the attack used at test time). Specificdetails regarding this ε schedule and also the learning pa-rameters used can be found in Appendix B.1. We find thatthe adversarially trained model is empirically the most welldefended against this attack of all four models, and cannotbe attacked down to 0% accuracy (Figure 4).

10−3 10−2 10−1

ε radius for Wasserstein ball

0.0

0.2

0.4

0.6

0.8

Adve

rsar

ial a

ccur

acy

standardℓ∞ robustadv. training

Figure 5. Adversarial accuracy of various models on the CIFAR10dataset when attacked by a Wasserstein adversary. We find that themodel trained to be provably robust against `∞ perturbations is notas robust as adversarial training against a Wasserstein adversary.

plan

e car bird

deer

bird

cat deer

dog

deer

dog + =

hors

e cat

frog

horse bird

dog

ship

truck plan

e car

Figure 6. Wasserstein adversarial examples for CIFAR10 on a typ-ical ResNet18 for all 10 classes. The perturbations here representsthe total change across all three channels, where total change isplotted within the range ±0.165 (the maximum total change ob-served in a single pixel) for images scaled to [0,1].

5.2. CIFAR10

For CIFAR10, we consider a standard model, a model prov-ably robust to `∞ perturbations of at most ε = 2/255, andan adversarially trained model. We plot the susceptibility ofeach model to the Wasserstein attack in Figure 5.

Standard model We find that for a standard ResNet18CIFAR10 classifier, a perturbation radius of as little as 0.01is enough to misclassify 25% of the examples, while a radiusof 0.1 is enough to fool the classifier 97% of the time (Figure5). Despite being such a small ε, we see in Figure 6 that thestructure of the perturbations still reflect the actual content ofthe images, though certain classes require larger magnitudesof change than others.

`∞ robust model We further empirically evaluate the at-tack on a model that was trained to be provably robustagainst `∞ perturbations. We use the models weights fromWong et al. (2018), which are trained to be provably robust

Wasserstein Adversarial Examples via Projected Sinkhorn Iterations

against `∞ perturbations of at most ε = 2/255. Furthernote that this CIFAR10 model actually is a smaller ResNetthan the ResNet18 architecture considered in this paper, andconsists of 4 residual blocks with 16, 16, 32, and 64 fil-ters. Nonetheless, we find that while the model suffers frompoor nominal accuracy (achieving only 66% accuracy onunperturbed examples as noted in Table 1), the robustnessagainst `∞ attacks remarkably seems to transfer quite wellto robustness against Wasserstein attacks in the CIFAR10setting, achieving 61% adversarial accuracy for ε = 0.1 incomparison to 3% for the standard model.

Adversarial training To perform adversarial training forCIFAR10, we use a similar scheme to that used for MNIST:we adopt a weaker adversary that uses only 50 iterationsof projected gradient descent during training and allow εto grow within a range (specific details can be found inAppendix B.2). We find that adversarial training here is alsoable to defend against this attack, and at the same thresholdof ε = 0.1, we find that the adversarial accuracy has beenimproved from 3% to 76%.

5.3. Provable Defenses against WassersteinPerturbations

Lastly, we present some analysis on how this attack fits intothe context of provable defenses, along with a negative resultdemonstrating a fundamental gap that needs to be solved.The Wasserstein attack can be naturally incorporated intoduality based defenses: Wong et al. (2018) show that to usetheir certificates to defend against other inputs, one onlyneeds to solve the following optimization problem:

maxx∈B(x,ε)

−xT y (14)

for some constant y and for some perturbation regionB(x, ε) (a similar approach can be taken to adapt the dualverification from Dvijotham et al. (2018)). For the Wasser-stein ball, this is highly similar to the problem of projectingonto the Wasserstein ball from Equation (6), with a linearobjective instead of a quadratic objective and fewer vari-ables. In fact, a Sinkhorn-like algorithm can be derived tosolve this problem, which ends up being a simplified versionof Algorithm 2 (this is shown in Appendix D).

However, there is a fundamental obstacle towards generat-ing provable certificates against Wasserstein attacks: thesedefenses (and many other, non-duality based approaches)depend heavily on propagating interval bounds from the in-put space through the network, in order to efficiently boundthe output of ReLU units. This concept is inherently at oddswith the notion of Wasserstein distance: a “small” Wasser-stein ball can use a low-cost transport plan to move all themass at a single pixel to its neighbors, or vice versa. As aresult, when converting a Wasserstein ball to interval con-

straints, the interval bounds immediately become vacuous:each individual pixel can attain their minimum or maximumvalue under some ε cost transport plan. In order to guaranteerobustness against Wasserstein adversarial attacks, signifi-cant progress must be made to overcome this limitation.

6. ConclusionIn this paper, we have presented a new, general threat modelfor adversarial examples based on the Wasserstein distance,a metric that captures a kind of perturbation that is fun-damentally different from traditional `p perturbations. Togenerate these examples, we derived an algorithm for fast,approximate projection onto the Wasserstein ball that canuse local transport plans for even more speedup on im-ages. We successfully attacked standard networks, showingthat these adversarial examples are structurally perturbedaccording to the content of the image, and demonstratedthe empirical effectiveness of adversarial training. Finally,we observed that networks trained to be provably robustagainst `∞ attacks are more robust than the standard net-works against Wasserstein attacks, however we show thatthe current state of provable defenses is insufficient to di-rectly apply to the Wasserstein ball due to their reliance oninterval bounds.

We believe overcoming this roadblock is crucial to the de-velopment of verifiers or provable defenses against not justthe Wasserstein attack, but also to improve the robustness ofclassifiers against other attacks that do not naturally convertto interval bounds (e.g. `0 or `1 attacks). Whether we candevelop efficient verification or provable training methodsthat do not rely on interval bounds remains an open question.

Perhaps the most natural future direction for this work is tobegin to understand the properties of Wasserstein adversarialexamples and what we can do to mitigate them, even ifonly at a heuristic level. However, at the end of the day,the Wasserstein threat model defines just one example of aconvex region capturing structure that is different from `pballs. By no means have we characterized all reasonableadversarial perturbations, and so a significant gap remainsin determining how to rigorously define general classesof adversarial examples that can characterize phenomenadifferent from the `p and Wasserstein balls.

Finally, although we focused primarily on adversarial exam-ples in this work, the method of projecting onto Wassersteinballs may be applicable outside of deep learning. Projec-tion operators play a major role in optimization algorithmsbeyond projected gradient descent (e.g. ADMM and alter-nating projections). Perhaps even more generally, the tech-niques in this paper could be used to derive Sinkhorn-likealgorithms for classes of problems that consider Wassersteinconstrained variables.

Wasserstein Adversarial Examples via Projected Sinkhorn Iterations

ReferencesAltschuler, J., Weed, J., and Rigollet, P. Near-linear time ap-

proximation algorithms for optimal transport via sinkhorniteration. In Advances in Neural Information ProcessingSystems, pp. 1964–1974, 2017.

Athalye, A., Carlini, N., and Wagner, D. Obfuscated gra-dients give a false sense of security: Circumventing de-fenses to adversarial examples. In Proceedings of the 35thInternational Conference on Machine Learning, ICML2018, July 2018a. URL https://arxiv.org/abs/1802.00420.

Athalye, A., Engstrom, L., Ilyas, A., and Kwok, K. Syn-thesizing robust adversarial examples. In Dy, J. andKrause, A. (eds.), Proceedings of the 35th InternationalConference on Machine Learning, volume 80 of Pro-ceedings of Machine Learning Research, pp. 284–293,Stockholmsmssan, Stockholm Sweden, 10–15 Jul 2018b.PMLR. URL http://proceedings.mlr.press/v80/athalye18b.html.

Carlini, N. and Wagner, D. Towards evaluating the robust-ness of neural networks. In Security and Privacy (SP),2017 IEEE Symposium on, pp. 39–57. IEEE, 2017.

Croce, F., Andriushchenko, M., and Hein, M. Provablerobustness of relu networks via maximization of linearregions. CoRR, abs/1810.07481, 2018. URL http://arxiv.org/abs/1810.07481.

Cuturi, M. Sinkhorn distances: Lightspeed computationof optimal transport. In Burges, C. J. C., Bottou, L.,Welling, M., Ghahramani, Z., and Weinberger, K. Q.(eds.), Advances in Neural Information ProcessingSystems 26, pp. 2292–2300. Curran Associates, Inc.,2013. URL http://papers.nips.cc/paper/4927-sinkhorn-distances-lightspeed-computation-of-optimal-transport.pdf.

Dvijotham, K., Stanforth, R., Gowal, S., Mann, T., andKohli, P. A dual approach to scalable verification ofdeep networks. In Proceedings of the Thirty-Fourth Con-ference Annual Conference on Uncertainty in ArtificialIntelligence (UAI-18), pp. 162–171, Corvallis, Oregon,2018. AUAI Press.

Engstrom, L., Tran, B., Tsipras, D., Schmidt, L., and Madry,A. A rotation and a translation suffice: Fooling cnns withsimple transformations. arXiv preprint arXiv:1712.02779,2017.

Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A.,Xiao, C., Prakash, A., Kohno, T., and Song, D. Robustphysical-world attacks on deep learning visual classifica-tion. In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition, pp. 1625–1634, 2018.

Goodfellow, I., Shlens, J., and Szegedy, C. Explainingand harnessing adversarial examples. In InternationalConference on Learning Representations, 2015. URLhttp://arxiv.org/abs/1412.6572.

Gowal, S., Dvijotham, K., Stanforth, R., Bunel, R., Qin,C., Uesato, J., Arandjelovic, R., Mann, T. A., andKohli, P. On the effectiveness of interval bound prop-agation for training verifiably robust models. CoRR,abs/1810.12715, 2018. URL http://arxiv.org/abs/1810.12715.

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn-ing for image recognition. In Proceedings of the IEEEconference on computer vision and pattern recognition,pp. 770–778, 2016.

Kurakin, A., Goodfellow, I., and Bengio, S. Adversarialexamples in the physical world. ICLR Workshop, 2017.URL https://arxiv.org/abs/1607.02533.

Lu, J., Sibai, H., Fabry, E., and Forsyth, D. No need toworry about adversarial examples in object detection inautonomous vehicles. arXiv preprint arXiv:1707.03501,2017.

Madry, A., Makelov, A., Schmidt, L., Tsipras, D., andVladu, A. Towards deep learning models resistantto adversarial attacks. In International Conferenceon Learning Representations, 2018. URL https://openreview.net/forum?id=rJzIBfZAb.

Mirman, M., Gehr, T., and Vechev, M. Differ-entiable abstract interpretation for provably ro-bust neural networks. In International Confer-ence on Machine Learning (ICML), 2018. URLhttps://www.icml.cc/Conferences/2018/Schedule?showEvent=2477.

Papernot, N., McDaniel, P., Wu, X., Jha, S., and Swami,A. Distillation as a defense to adversarial perturbationsagainst deep neural networks. In Security and Privacy(SP), 2016 IEEE Symposium on, pp. 582–597. IEEE,2016.

Raghunathan, A., Steinhardt, J., and Liang, P. S. Semidefi-nite relaxations for certifying robustness to adversarialexamples. In Bengio, S., Wallach, H., Larochelle,H., Grauman, K., Cesa-Bianchi, N., and Garnett, R.(eds.), Advances in Neural Information ProcessingSystems 31, pp. 10900–10910. Curran Associates, Inc.,2018. URL http://papers.nips.cc/paper/8285-semidefinite-relaxations-for-certifying-robustness-to-adversarial-examples.pdf.

Wasserstein Adversarial Examples via Projected Sinkhorn Iterations

Sharif, M., Bhagavatula, S., Bauer, L., and Reiter, M. K.Adversarial generative nets: Neural network attackson state-of-the-art face recognition. arXiv preprintarXiv:1801.00349, 2017.

Sinha, A., Namkoong, H., and Duchi, J. Certifying some dis-tributional robustness with principled adversarial training.2018.

Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Er-han, D., Goodfellow, I., and Fergus, R. Intriguingproperties of neural networks. In International Confer-ence on Learning Representations, 2014. URL http://arxiv.org/abs/1312.6199.

Tjeng, V., Xiao, K. Y., and Tedrake, R. Evaluating ro-bustness of neural networks with mixed integer program-ming. In International Conference on Learning Repre-sentations, 2019. URL https://openreview.net/forum?id=HyGIdiRqtm.

Wong, E. and Kolter, Z. Provable defenses against adver-sarial examples via the convex outer adversarial polytope.In International Conference on Machine Learning, pp.5283–5292, 2018.

Wong, E., Schmidt, F., Metzen, J. H., and Kolter,J. Z. Scaling provable adversarial defenses. In Ben-gio, S., Wallach, H., Larochelle, H., Grauman, K.,Cesa-Bianchi, N., and Garnett, R. (eds.), Advancesin Neural Information Processing Systems 31, pp.8410–8419. Curran Associates, Inc., 2018. URL http://papers.nips.cc/paper/8060-scaling-provable-adversarial-defenses.pdf.

Xiao, K. Y., Tjeng, V., Shafiullah, N. M. M., and Madry,A. Training for faster adversarial robustness verificationvia inducing reLU stability. In International Conferenceon Learning Representations, 2019. URL https://openreview.net/forum?id=BJfIVjAcKm.

Wasserstein Adversarial Examples via Projected Sinkhorn Iterations

A. Projected Sinkhorn derivationA.1. Proof of Lemma 3

Lemma 3. The dual of the entropy-regularized Wassersteinprojection problem in Equation (6) is

maximizeα,β∈Rn,ψ∈R+

g(α, β, ψ) (15)

where

g(α, β, ψ) =− 1

2λ‖β‖22 − ψε+ αTx+ βTw

−∑ij

exp(αi) exp(−ψCij − 1) exp(βj)(16)

Proof. For convenience, we multiply the objective by λ andsolve this problem instead:

minimizez∈Rn

+,Π∈Rn×n+

λ

2‖w − z‖22 +

∑ij

Πij log(Πij)

subject to Π1 = x

ΠT 1 = z

〈Π, C〉 ≤ ε.

(17)

Introducing dual variables (α, β, ψ) where ψ ≥ 0, the La-grangian is

L(z,Π, α, β, ψ)

=λ

2‖w − z‖22 +

∑ij

Πij log(Πij) + ψ(〈Π, C〉 − ε)

+ αT (x−Π1) + βT (z −ΠT 1).

(18)

The KKT optimality conditions are now

∂L

∂Πij= ψCij + (1 + log(Πij))− αi − βj = 0

∂L

∂zj= λ(zj − wj) + βj = 0

(19)

so at optimality, we must have

Πij = exp(αi) exp(−ψCij − 1) exp(βj)

z = −βλ

+ w(20)

Plugging in the optimality conditions, we get

L(z∗,Π∗, α, β, ψ)

=− 1

2λ‖β‖22 − ψε+ αTx+ βTw

−∑ij

exp(αi) exp(−ψCij − 1) exp(βj)

=g(α, β, ψ)

(21)

so the dual problem is to maximize g over α, β, ψ ≥ 0.

A.2. Proof of Lemma 4

Lemma 4. Suppose α∗, β∗, ψ∗ maximize the dual problemg in Equation (16). Then,

z∗i = wi − βi/λΠ∗ij = exp(α∗i ) exp(−ψ∗Cij − 1) exp(β∗j )

(22)

are the corresponding solutions that minimize the primalproblem in Equation (6).

Proof. These equations follow directly from the KKT opti-mality conditions from Equation (20).

A.3. Algorithm derivation and interpretation

Derivation To derive the algorithm, note that since this isa strictly convex problem to get the α and β iterates we solvefor setting the gradient to 0. The derivative with respect toα is

∂g

∂αi= xi − exp(αi)

∑j

exp(−ψCij − 1) exp(βj) (23)

and so setting this to 0 and solving for αi gives the α iterate.The derivative with respect to β is

∂g

∂βj= − 1

λβ+w− exp(βj)

∑i

exp(αi) exp(−ψCij−1)

(24)and setting this to 0 and solving for βj gives the β iterate(this step can be done using a symbolic solver, we usedMathematica). Lastly, the ψ updates are straightforwardscalar calculations of the derivative and second derivative.

Interpretation Recall from the transformation of dual toprimal variables from Lemma 4. To see how the ProjectedSinkhorn iteration is a (modified) matrix scaling algorithm,we can interpret these quantities before optimality as primaliterates. Namely, at each iteration t, let

z(t)i = wi − β(t)

i /λ

Π(t)ij = exp(α(t)) exp(−ψ(t)Cij − 1) exp(β(t))

(25)

Then, since the α and β steps are equivalent to setting Equa-tions (23) and (24) to 0, we know that after an update forα(t), we have that

xi =∑j

Π(t)ij (26)

so the α step rescales the transport matrix to sum to x.Similarly, after an update for β(t), we have that

z(t)i =

∑i

Π(t)ij (27)

Wasserstein Adversarial Examples via Projected Sinkhorn Iterations

which is a rescaling of the transport matrix to sum to theprojected value. Lastly, the numerator of the ψ(t) step canbe rewritten as

ψ(t+1) = ψ(t) + t · 〈Π(t), C〉 − ε

〈Π(t), C · C〉(28)

as a simple adjustment based on whether the current trans-port plan Π(t) is above or below the maximum thresholdε.

B. Experimental setupB.1. MNIST

Adaptive ε During adversarial training for MNIST, weadopt an adaptive ε scheme to avoid selecting a specific ε.Specifically, to find an adversarial example, we first let ε =0.1 on the first iteration of projected gradient descent, andincrease it by a factor of 1.4 every 5 iterations. We terminatethe projected gradient descent algorithm when either anadversarial example is found, or when 50 iterations havepassed, allowing ε to take on values in the range [0.1, 2.1]

Optimizer hyperparameters To update the modelweights during adversarial training, we use the SGD op-timizer with 0.9 momentum and 0.0005 weight decay, andbatch sizes of 128. We begin with a learning rate of 0.1,reduce it to 0.01 after 10 epochs.

B.2. CIFAR10

Adaptive ε We also use an adaptive ε scheme for adver-sarial training in CIFAR10. Specifically, we let ε = 0.01 onthe first iteration of projected gradient descent, and increaseit by a factor of 1.5 every 5 iterations. Similar to MNIST,we terminate the projected gradient descent algorithm wheneither an adversarial example is found, or 50 iterations havepassed, allowing ε to take on values in the range [0.01, 0.38].

Optimizer hyperparameters Similar to MNIST, to up-date the model weights, we use the SGD optimizer with 0.9momentum and 0.0005 weight decay, and batch sizes of 128.The learning rate is also the same as in MNIST, starting at0.1, and reducing to 0.01 after 10 epochs.

B.3. Motivation for adaptive ε

A commonly asked question of models trained to be robustagainst adversarial examples is “what if the adversary has aperturbation budget of ε+ δ instead of ε?” This is referringto a “robustness cliff,” where a model trained against anε strong adversary has a sharp drop in robustness whenattacked by an adversary with a slightly larger budget. Toaddress this, we advocate for the slightly modified versionof typical adversarial training used in this work: rather than

picking a fixed ε and running projected gradient descent,we instead allow for an adversarial to have a range of ε ∈[εmin, εmax]. To do this, we begin with ε = εmin, and thengradually increase it by a multiplicative factor γ until eitheran adversarial example is found or until εmax is reached.While similar ideas have been used before for evaluatingmodel robustness, we specifically advocate for using thisschema during adversarial training. This has the advantageof extending robustness of the classifier beyond a single εthreshold, allowing a model to achieve a potentially higherrobustness threshold while not being significantly harmedby “impossible” adversarial examples.

C. Auxiliary experimentsIn this section, we explore the space of possible parametersthat we treated as fixed in the main paper. While this is notan exhaustive search, we hope to provide some intuition asto why we chose the parameters we did.

C.1. Effect of λ and C

We first study the effect of λ and the cost matrix C. First,note that λ could be any positive value. Furthermore, notethat to construct C we used the 2-norm which reflects the1-Wasserstein metric, but in theory we could use any p-Wasserstein metric, where the the cost of moving from pixel(i, j) to (k, l) is

(|i− j|2 + |k − l|2

)p/2. Figure 8 shows

the effects of λ and p on both the adversarial example andthe radius at which it was found for varying values of λ =[1, 10, 100, 500, 1000] and p = [1, 2, 3, 4, 5].

We find that it is important to ensure that λ is large enough,otherwise the projection of the image is excessively blurred.In addition to qualitative changes, smaller λ seems to makeit harder to find Wasserstein adversarial examples, makingthe ε radius go up as λ gets smaller. In fact, for λ = (1, 10)and almost all of λ = 100, the blurring is so severe that noadversarial example can be found.

In contrast, we find that increasing p for the Wassersteindistance used in the cost matrixC seems to make the imagesmore “blocky”. Specifically, as p gets higher tested, morepixels seem to be moved in larger amounts. This seems tocounteract the blurring observed for low λ to some degree.Naturally, the ε radius also grows since the overall cost ofthe transport plan has gone up.

C.2. Size of local transport plan

In this section we explore the effects of different sized trans-port plans. In the main paper, we used a 5×5 local transportplan, but this could easily be something else, e.g. 3× 3 or7× 7. We can see a comparison on the robustness of a stan-dard and the `∞ robust model against these different sized

Wasserstein Adversarial Examples via Projected Sinkhorn Iterations

0.0 0.5 1.0 1.5 2.0ε radius for Wasserstein ball

0.0

0.2

0.4

0.6

0.8

1.0

Adve

rsar

ial a

ccur

acy

Standard 3x3Standard 5x5Standard 7x7Robust 3x3Robust 5x5Robust 7x7

Figure 7. Adversarial accuracy of a standard model and a modeltrained to be provably robust against `∞ attacks for different sizesof transport plans. In most cases the size of the transport plandoesn’t seem to matter, except for the 3 × 3 local transport plan.In this case, the adversary isn’t quite able to reach 0% accuracy forthe standard model, reaching 2.8% for for ε = 1.83. The adversaryis also unable to attack the robust MNIST model, bottoming out at41% adversarial accuracy at ε = 1.83.

transport plans in Figure 7, using λ = 1000. We observethat while 3 × 3 transport plans have difficulty attackingthe robust MNIST model, all other plan sizes seem to havesimilar performance.

D. Provable defenseIn this section we show how a Sinkhorn-like algorithm canbe derived for provable defenses, and that the resulting al-gorithm is actually just a simplified version of the ProjectedSinkhorn iteration, which we call the Conjugate Sinkhorniteration (since it solves the conjugate problem).

D.1. Conjugate Sinkhorn iteration

By subtracting the same entropy term to the conjugate ob-jective from Equation (14), we can get a problem similar tothat of projecting onto the Wasserstein ball.

minimizez∈Rn

+,Π∈Rn×n+

− λzT y +∑ij

Πij log(Πij)

subject to Π1 = x

ΠT 1 = z

〈Π, C〉 ≤ ε.

(29)

where again we’ve multiplied the objective by λ for con-venience. Following the same framework as before, weintroduce dual variables (α, β, ψ) where ψ ≥ 0, to con-

struct the Lagrangian as

L(z,Π, α, β, ψ)

=− λzT y +∑ij

Πij log(Πij) + ψ(〈Π, C〉 − ε)

+ αT (x−Π1) + βT (z −ΠT 1).

(30)

Note that since all the terms with Πij are the same, thecorresponding KKT optimality condition for Πij also re-mains the same. The only part that changes is the optimalitycondition for z, which becomes

β = λy (31)

Plugging the optimality conditions into the Lagrangian, weget the following dual problem:

L(z∗,Π∗, α, β, ψ)

=− ψε+ αTx

−∑ij

exp(αi) exp(−ψCij − 1) exp(βj)

=g(α,ψ)

(32)

Finally, if we minimize this with respect to α and ψ weget exactly the same update steps as the Projected Sinkhorniteration. Consequently, the Conjugate Sinkhorn iteration isidentical to the Projected Sinkhorn iteration except that wereplace the β step with the fixed value β = λy.

Wasserstein Adversarial Examples via Projected Sinkhorn Iterations

ε=N/A

p=1

λ=1

ε=N/A

λ=10

ε=1.83

λ=100

ε=0.71

λ=500

ε=0.58

λ=1000

ε=N/A

p=2

ε=N/A ε=N/A ε=1.04 ε=0.71

ε=N/A

p=3

ε=N/A ε=N/A ε=1.14 ε=0.78

ε=N/A

p=4

ε=N/A ε=N/A ε=1.25 ε=0.53

ε=N/A

p=5

ε=N/A ε=N/A ε=1.25 ε=0.71

Figure 8. A plot of the adversarial examples generated with different p-Wasserstein metrics used for the cost matrix C and differentregularization parameters λ. Note that when regularization is low, the image becomes blurred, and it is harder to find adversarial examples.In contrast, changing p does not seem to make any significant changes.