Dimensionality Reduction as a Defense against Evasion Attacks on MachineLearning Classifiers

Arjun Nitin BhagojiPrinceton University

Daniel CullinaPrinceton University

Prateek MittalPrinceton University

AbstractWe propose the use of dimensionality reduction as adefense against evasion attacks on ML classifiers. Wepresent and investigate a strategy for incorporating di-mensionality reduction via Principal Component Anal-ysis to enhance the resilience of machine learning, tar-geting both the classification and the training phase. Weempirically evaluate and demonstrate the feasibility ofdimensionality reduction of data as a defense mecha-nism against evasion attacks using multiple real-worlddatasets. Our key findings are that the defenses are (i) ef-fective against strategic evasion attacks in the literature,increasing the resources required by an adversary for asuccessful attack by a factor of about two, (ii) applica-ble across a range of ML classifiers, including SupportVector Machines and Deep Neural Networks, and (iii)generalizable to multiple application domains, includingimage classification, malware classification, and humanactivity classification.

1 Introduction

We are living in an era of ubiquitous machine learning(ML) and artificial intelligence. Machine learning is be-ing used in a number of essential applications such asimage recognition [1], natural language processing [2],spam detection [3], autonomous vehicles [4, 5] and evenmalware detection [6, 7]. In addition, recent advances indeep learning [8, 9] have demonstrated classification ac-curacy that is close to the accuracy obtained by humans,enabling the widespread deployment of ML systems.Given the ubiquity of ML applications, it is increasinglybeing deployed in adversarial scenarios, where an at-tacker stands to gain from the failure of a ML systemto classify inputs correctly. The question then arises: areML systems secure in adversarial settings?

Adversarial Machine Learning: Starting in the early2000s, there has been a considerable body of work [10–

12] exposing the vulnerability of machine learning al-gorithms to strategic adversaries. For example, poison-ing attacks [13] systematically introduce adversarial dataduring the training phase with the aim of causing themisclassification of data during the test phase. On theother hand, evasion attacks [14–16] aim to fool existingML classifiers trained on benign data by adding strategicperturbations to test inputs.

Evasion attacks: In this paper we focus on evasionattacks in which the adversary aims to perturb test in-puts to ML classifiers in order to cause misclassifica-tion. Evasion attacks have been proposed for a varietyof machine learning classifiers such as Support VectorMachines [14, 17], tree-based classifiers [17, 18] such asrandom forests and boosted trees and more recently forneural networks [15,16,19–22]. The vulnerability of ap-plications that use machine learning, such as face detec-tion [23], voice command recognition [24] and PDF mal-ware detection [25] has also been demonstrated, high-lighting the need for defenses. Surprisingly, it has alsobeen shown that the evasion properties of adversariallymodified data (for a particular classifier) persist acrossdifferent ML classifiers [16], which allows an adver-sary with limited knowledge of the ML system to at-tack it. Thus, it is crucial to consider the possibilityof adversarial data and evasion attacks while using MLsystems in adversarial contexts. However, very few de-fenses [18, 26] exist against these attacks, and the appli-cability of each is limited to only certain known attacksand specific types of ML classifiers (see Section 7 fora detailed description of and comparison with previouswork).

1.1 Contributions

Through extensive evaluations, we find that our defensemechanism demonstrably reduces the success of evasionattacks. To the best of our knowledge, it is the only de-fense against evasion attacks with the following prop-

1

arX

iv:1

704.

0265

4v1

[cs

.CR

] 9

Apr

201

7

erties: (1) applicability across multiple ML classifiers(such as SVMs, DNNs), (2) applicability across multi-ple application domains (image and activity classifica-tion), and (3) mitigation of multiple attack types, includ-ing strategic ones. Further, the tunability of our defenseallows a system designer to pick appropriate operatingpoints on the utility-security tradeoff curve depending onthe application.

1.1.1 Defense

In this paper, we propose the use of dimensionality re-duction of data as a defense against evasion attacks onML systems. Dimensionality reduction techniques suchas Principal Component Analysis aim to project high-dimensional data to a lower-dimensional space whilemeeting specific conditions [27, 28].We present and in-vestigate a strategy for incorporating dimensionality re-duction to enhance the resilience of machine learning,targeting both the classification and the training phase.We consider an approach in which dimensionality re-duction is applied to the training data to enhance the re-silience of the trained classifier, as well as to the test datafor resilient classification.

1.1.2 Empirical Evaluation

We empirically demonstrate the feasibility and effective-ness of our defenses using:

• multiple ML classifiers, such as Support Vector Ma-chines (SVMs) and Deep Neural Networks (DNNs);

• several distinct types of evasion attacks, such as anattack on Linear SVMs from Moosavi-Dezfooli et.al. [29], and on deep neural networks from Goodfel-low et. al. [19], as well as strategic attacks targetingour defense ;

• a variety of real-world datasets/applications: theMNIST image dataset [30] and the UCI Human Ac-tivity Recognition (HAR) dataset [31].

Our key findings are that even in the face of apowerful adversary with almost complete knowledgeof the ML system: i) our defense leads to significantincreases of up to 5× in the degree of modificationrequired for a successful attack and equivalently, re-duces adversarial success rates by around 2 − 50×at fixed degrees of modification, ii) the defense canbe used for different ML classifiers with minimalmodification of the original classifiers, while stillbeing effective at combating adversarial examples,and iii) there is a modest variation of about 1-4% inthe classification success on benign samples in mostcases. We also provide an analysis of the utility-security

tradeoff curve as well as the computational overheadsincurred by our defense. Our results may be repro-duced using the open source code available at https://github.com/inspire-group/ml_defense.

However, our defense does not completely solve theproblem of evasion attacks, since it reduces adversarialsuccess rates at fixed budgets, but does not make themnegligible in all cases. In Section 4, we discuss the rangeof budgets available to the adversary in different applica-tion scenarios and make explicit the scenarios in whichour defense is effective. We hope that our work inspiresfurther research in combating the problem of evasion at-tacks and securing machine learning based systems.

The rest of the paper is organized as follows: first, inSection 2 we present the necessary background on adver-sarial machine learning. Then, in Section 3 we describeour defense. We then set up and present our empiricalevaluation in Sections 4 and 5 respectively. We discussour results in Section 6. Finally, we provide a detailedoverview of the related work in Section 7 and concludein Section 8.

2 Adversarial machine learning

In this section, we present the required background onadversarial machine learning, focusing on both (a) MLclassifiers such as SVMs and DNNs as well as (b) eva-sion attacks that induce misclassification by perturbingthe test input.

Motivation and Running Examples: We use imagedata from the MNIST dataset [30] (see Section 4 for de-tails) for our running examples. Figure 1(a) depicts ex-ample test images from the MNIST dataset that are cor-rectly classified by a SVM classifier, while Figure 1(b)depicts adversarially crafted test images (perturbed im-ages using the evasion attack of Papernot et. al. [17]),which are misclassified by a SVM classifier.

2.1 Classification using machine learning

In this paper, we focus on supervised machine learning,where a classifier is trained on data with pre-existing la-bels. A trained, supervised machine learning classifieris a function that takes as input a data point x ∈ Rd (or{0,1}d in the case of binary data vectors), and producesa scalar output y ∈ C, where C is the set of all possibleclassification categories. For example, in the case of theMNIST dataset, x would be a 28×28 pixel grayscale im-age of a handwritten digit and C would be the finite set{0,1,2,3,4,5,6,7,8,9}.

2

(a) Typical test images from the MNIST dataset. Correctlyclassifed as 0, 4, 5 and 6 respectively.

(b) Corresponding adversarial images obtained using theevasion attack on Linear SVMs [29]. Now, misclassifiedas 9, 9, 3, 2 and 0. respectively.

Figure 1: Comparison of benign and adversarial imagestaken from the MNIST dataset.

2.2 Attacks against machine learning sys-tems

In this subsection we first discuss the adversarial modelunder consideration, after which we describe generalevasion attacks and finally, evasion attacks on specificML classifiers.

Notation: We denote the complete training data asStrain, the complete test data as Stest, the machine learningclassifier in use as f and the specific parameters chosenfor the ML system as θθθ . The original dimension of thedata is denoted as d. We then represent the adversary’sattack algorithm as A(xin|K), where xin denotes the in-put the adversary starts with and K denotes adversarialknowledge, which may be any subset of {Strain, f ,θθθ}.The adversarial sample generated by A is denoted by x.

2.2.1 Adversarial goals and capabilities

In this paper, we focus on attacks where the adversary’sobjective is either to modify a benign, existing input sothat it is misclassified as any other class or to cause it tobe classified in a targeted class different from the originalclass. Note that these aims are equivalent in the case ofbinary classifiers.

Our baseline assumptions are that the adversary hasthe following capabilities.

• The adversary has complete knowledge of the train-ing set the original classifier has been trained on,i.e. she knows the type of feature vectors which theclassifier takes as input.

• The adversary knows the classifier structure, hyper-parameters, and training procedure.

• Adversarial examples are created offline by the ad-versary, and submitted to the ML classifier duringthe test phase.

In summary, x = A(xin|Strain, f ,θθθ ,Kadd), where Kadddenotes any additional knowledge about the system theadversary may have.

Our assumptions on the adversary’s capabilities areconservative since from a security perspective, beingable to defend against attacks where the adversary has al-most complete knowledge of the system ensures that thedefense does not rely on security through obscurity andis robust even in the face of powerful attacks. Further,even adversaries with limited access (such as black-boxaccess) and knowledge of the ML system can infer theclassifier well enough to mount evasion attacks, whichmay be almost as successful as those mounted by an ad-versary with full access [17, 32–34], justifying our as-sumptions.

2.2.2 Evasion attacks

In normal operation, when there is no adversary, uponsome input xi ∈ S, f outputs y, where S is the set of in-puts. The fraction of the training set for which the outputclass matches the true class y 1 is α , i.e.

α(S) =#{(x,y) ∈ S : f (x) = y}

#S, (1)

where # gives the cardinality of a set.The adversary’s goal is to design an algorithm A act-

ing on x ∈ S which generates adversarial examples, i.e.A(x) = x. Let

Sadv = {(A(x),y) : (x,y) ∈ S},

the set of adversarially modified examples. These modi-fications should

• increase the misclassification percentage as com-pared to the normal operation of the classifier, i.e.α(Sadv)< α(S),

• not be detectable as anomalous by humans in thecase of human-interpretable data such as imagesand text, and by rule-based detection systems inthe case of data like malware samples, network andsystem logs etc. For example, in the case of mal-ware, the adversary is constrained by the fact thather modifications must ensure the final samples arestill malicious.

We next discuss adversarial perturbations and in thecase of image data, their perceptibility to humans.

1Assuming there are ground truth labels to compare against for allelements in S

3

2.2.3 Adversarial perturbations

Modeling human perception of image perturbations is adifficult problem. As a proxy for human perceptibility,we will measure the degree of modification as ‖A(x)−x‖for some norm ‖ · ‖. In particular, we will considerperturbations constrained in terms of the `2 norm, i.e.‖x−x‖2 ≤ ε , where ε dictates the strength of the pertur-bation. A detailed description of the relation between thevarious norms used to constrain adversarial perturbationsand their perception is given in [35].

Now, we define the minimal perturbations required fordifferent adversarial goals to be achieved. In order tocause misclassification in a specific class z, the minimumperturbation that has to be added to an input (x,y), wherez 6= y, is

∆(x,z) = infx{‖x−x‖ : f (x) = z}.

This is the minimum distortion required to cause x tobe classified as z. The mimimum distortion required tocause x to be misclassified in any class is

∆(x) = minz∈C\{y}

∆(x,z).

For image data, the relationship between these quan-tities and the minimum detectable distortion determinesthe robustness of the classifier f to adversarial perturba-tions. In Figure 1, the images correspond to a perturba-tion value that causes the Linear SVM to classify almostall inputs wrong but the perturbation is scarcely visible tothe human eye. This indicates that the standard form ofLinear SVMs is not robust to adversarial perturbations.Further discussion of the metrics used to constrain theadversary is given in Section 4.4.

2.3 Evasion attacks on specific classifiersWe now describe existing attacks from the literatureon specific ML classifiers and show some adversarialexamples from the MNIST dataset for each. A summaryof the various attacks is given in Table 1.

2.3.1 Optimal attacks on Linear SVMs

In the multi-class classification setting for Linear SVMs,a classifier gi is trained for each class i ∈C, where

gi : x 7→ wTi x+bi. (2)

x is then assigned to the class f (x) = argmaxi∈C gi(x).Given that the true class is t ∈ C, the objective of anuntargeted attack is to find the closest point x such thatf (x) 6= t.

From [29], we know that for an optimal untargeted at-tack on affine multi-class classifiers, i.e. if we only carethat f (x) 6= t with the smallest possible ‖x−x‖2, the op-timal choice of x is xk, where

k = argminj

|gt(x)−g j(x)|‖wt −w j‖

, (3)

and thus

x(ε) = x+ εwt −wk

‖wt −wk‖, (4)

where ε controls the magnitude of the perturbation. Theminimum ε required to cause a misclassification is ε∗ =|gt (x)−gk(x)|‖wt−wk‖

. It can easily be verified that f (x(ε∗)) = k.Note that this attack is constrained in terms of the `2norm, since ‖ε wt−wk

‖wt−wk‖‖= ε .

2.3.2 Gradient based attacks on neural networks

The Fast Gradient Sign (FGS) attack is an efficient at-tack against neural networks introduced in [19]. In thiscase, the adversarial examples are generated by addingadversarial noise proportional to the sign of the gradi-ent of the loss function (sign(∇J f (x,y,θ))), where J f (·)denotes the loss function and θ denotes the hyperparam-eters used for training. The gradient can be efficientlycalculated using backpropagation. Concretely,

x = x+η sign(∇J f (x,y,θ)), (5)

where η is a parameter that can be varied by the ad-versary to control the effectiveness of the adversar-ial examples. As η increases, the adversarial suc-cess rate typically increases as well. However, largerperturbations may make identification difficult for hu-mans as well in the case of images (see Appendixfor images modified with a range of η). The FSGattack is constrained in terms of the `∞ norm, since‖η sign(∇J f (x,y,θ))‖∞ = maxi |η sign(∇J f (x,y,θ))i|=η , which controls the strength of the perturbation.

The FGS attack and the attack on Linear SVMs areconstrained according to different norms. To facilitatea comparison of the robustness of various classifiers aswell as the effectiveness of our defense across them, wepropose a modification of the FGS attack which is con-strained in terms of the `2 norm. Denoting this as the FastGradient (FG) attack, we define the adversarial examplesto be equal to

x = x+η∇J f (x,y,θ)‖∇J f (x,y,θ)‖

. (6)

For the FG attack, η is a constraint on the `2 norm of theperturbation.

4

Table 1: Summary of attacks on Linear SVMs and neural networks.

Attack Classifier Constraint Intuition

Optimal attack on Linear SVMs [29] Linear SVMs `2 Move towards classifier boundary

Fast Gradient Neural Networks `2First order approximation to direction of

smallest perturbation

Fast Gradient Sign [19] Neural Networks `∞

Constant scaling for each pixelmodels perception better

2.4 Dimensionality reduction for machinelearning

While dealing with high-dimensional data (in the sensethat each sample has a large number of features), it is of-ten difficult to make sense of which features are impor-tant. Application constraints may also make it imprac-tical to carry out learning tasks on the data in the orig-inal, high-dimensional space. Dimensionality reductionis then a useful preprocessing step for high-dimensionaldata. It can also help alleviate the problems associatedwith the ‘curse of dimensionality’ [28]. In such cases,the data is first projected onto a lower-dimensional spacebefore providing it as input to the ML system. Commondimensionality reduction algorithms are PCA [27], ran-dom projections [36] and Kernel PCA [37]. In this paper,we make novel use of dimensionality reduction methodssuch as PCA to not only aid in interpretability and in-crease efficiency, but also to increase the robustness ofML systems to adversarial examples.

3 Dimensionality reduction based defense

In the previous section, we have seen that a number ofML classifiers are vulnerable to a variety of different eva-sion attacks. These vulnerabilities make it easy for ad-versaries to craft inputs to meet their desired objective,which could range from fooling a self-driving car [4, 5]and making it crash to escaping fraud and malware de-tection [6,7]. There is clearly a need for a defense mech-anism for ML systems which is effective against a varietyof attacks, since a priori, the owner of the system has noknowledge of the range of attacks possible against thesystem. Further, finding a defense that works across arange of classifiers can direct us to a better understandingof why ML systems are vulnerable in the first place. Inthis section, we present a defense that is agnostic at leastto common attacks from the literature, and remains ef-fective even in the presence of a stronger adversary withknowledge of the defense. Further, the defense makesmultiple types of ML classfiers operating in different ap-plication scenarios more robust as shown in Section 5.The defense is based on dimensionality reduction, which

was introduced briefly in Section 2.Now, we present our threat model and design goals,

followed by our dimensionality reduction based defensemechanism. In what follows, we refer to the originalML classification system with the addition of the defensemechanism as the ‘robust classification pipeline’.

3.1 Threat modelGiven the adversarial capabilities defined in Section 2.2,we consider two categories of evasion attacks in ourthreat model when evaluating the effectiveness of the de-fense mechanism:

1. Vanilla attacks: This category considers a rangeof existing attacks from the literature without anymodification. Since these attacks were designed formachine learning systems without taking into ac-count any defenses, the adversary is unaware ofthe defense mechanism. We note that the design ofdefense mechanisms that can successfully mitigateeven vanilla/known attacks is a significant challengefacing the security community.

2. Strategic attacks: In the second case, the adver-sary is aware of the defense mechanism. This im-plies that the adversary will take the effects of thedefense into account when crafting adversarial ex-amples, and possibly even switch to a different at-tack strategy optimized for the particular defense inuse.

In order to evaluate our defenses thoroughly, we investi-gate new attack strategies of the second type, optimizedfor our defense. We show that our defenses are effectiveeven in this case, albeit to a lesser extent.

3.2 Design goalsThe primary goal of any supervised machine learningclassifier is to achieve the best possible accuracy on thetest set. Further, it is desirable that the machine learningclassifier is as efficient as possible. In this light, we haveto ensure that any defense mechanism does not have an

5

overly large impact on either the efficiency or accuracy ofthe overall classification process. Thus, the design goalsfor a defense mechanism are:

1. High classification accuracy: Adding the defensemechanism to the overall classification pipelineshould maintain high classification accuracy on thebenign test set. In fact, an ideal defense mechanismmay even increase the classification accuracy on thetest set by reducing overfitting, increasing the ca-pacity of the classifer etc.

2. High efficiency: Both time and space complexityplay a role in determining the efficiency of an al-gorithm. In the worst case, the defense mecha-nism should not more add more than a polynomialoverhead to the running time and required space forthe original classifier. In an ideal case, the defensemechanism would make the overall pipeline moreefficient in both time and space.

3. Security: We define the vulnerability of a machinelearning classifier as the fraction of adversariallymodified input data that is misclassified (subject tothe constraint that the original input data was cor-rectly classified2). In order for a defense to be ef-fective, we need this misclassification fraction tobe lower than for the original classifier. Thus, fora particular attack, we can say that the pipeline ismore secure if the fraction of adversarial examplesthat is misclassifed is less than the original classi-fier, for a comparable attack parameter.

4. Tunability: Depending on the application, perfor-mance (i.e. classification accuracy), efficiency orsecurity may be the primary concern for the user ofthe machine learning system. The defense mecha-nism should give the user the option of navigatingdifferent points in the multi-dimensional tradeoffspace of performance, utility and security by modi-fying its parameters.

In Section 5 we quantify how well our defense satisfiesthe design goals laid out above. Now, we provide anoverview of the robust classification pipeline and how ithelps defend against existing attacks.

3.3 Overview of defenseIn our defense, we leverage dimensionality reductiontechniques to project high dimensional data onto a lowerdimensional space and to make the classifier more re-silient by modifying the training phase. In the first stepof our defense, the ML system user chooses a reduced

2See the Appendix for discussion on similar security definitionswithout this constraint

dimension k < d. Then, the training data Xtrain is trans-formed to a lower dimensional space using a dimension-ality reduction algorithm DR which takes k and Xtrain asinput. A new classifier fk is then trained on the reduceddimension training set. At test time, all inputs are trans-formed to a lower dimensional space using the same al-gorithm DR and the reduced dimension inputs xk are pro-vided as input directly to fk. We now describe a concreteinstantiation of the defense using PCA.

3.4 Defense using PCA

3.4.1 PCA in brief

PCA [27] is a linear transformation of the data that iden-tifies so-called ‘principal axes’ in the space of the data,which are the directions in which the data has maximumvariance, and projects the original data along these axes.The dimensionality of the data is reduced by choosingto project it along k principal axes. The choice of k de-pends on what percentage of the original variance is tobe retained. Intuitively, PCA identifies the directions inwhich the ‘signal’, or useful information in the data ispresent, and discards the rest as noise.

Concretely, let the data samples be xi ∈ Rd for i ∈{1, . . . ,n}. Let X′ be a d× n matrix such that each col-umn of X′ corresponds to a sample, and let 1 ∈Rn be thevector of all ones. Then, 1

n X′1 is the mean of the sam-ples and X =X′(I− 1

n 11T ) is the matrix of centered sam-ples. The principal components of X are the normalizedeigenvectors of its sample covariance matrix C = XT X.More precisely, because C is positive semidefinite, thereis a decomposition C = UΛUT where U is orthonormal,Λ = diag(λ1, . . . ,λn), and λ1 ≥ . . .≥ λn ≥ 0. In particu-lar, U is the d×d matrix whose columns are unit eigen-vectors of C, i.e. U = [u1, . . . ,ud ], where ui is the ith

normalized eigenvector of C. The eigenvalue λi is thevariance of X along the ith principal component.

Each column of UT X is a data sample represented inthe principal component basis. Let Xk be the projec-tion of the sample data in the k-dimensional subspacespanned by the k largest principal components. ThusXk = UIkUT X = UkUT

k X, where Ik is a diagonal matrixwith ones in the first k positions and zeros in the last d−kand Uk = [u1, . . . ,uk]. The amount of variance retainedis ∑

ki=1 λi, which is the sum of the k largest eigenvalues.

3.4.2 Implementing the defense

In our defense using PCA, the principal componentsare computed using an appropriately mean-centred andscaled matrix of training data. If there are n trainingsamples, then the matrix used to compute the principalcomponents is Xtrain, which is a d× n matrix. All sub-

6

sequent data samples are projected onto these principalcomponents that are initially computed.

Algorithm 1 DRtrain

Input: f 0k ,θθθ k, k, Xtrain, Train

Output: fk1: Use PCA to find the matrix Uk of the top k principal

components used to reduce the dimension of Xtrain

2: Compute the reduced dimension training setUT

k Xtrain

3: Train the reduced dimension classifier f ′k =Train( f 0

k ,θθθ k,UTk Xtrain)

4: Let fk = x 7→ f ′k(Ukx)

The resilient lower dimensional classifier fk is gener-ated using Algorithm 1. It takes a classifier training al-gorithm Train and adapts it to produce a variant that usesdimensionality reduction. The inputs to Algorithm 1 are:

• f 0k : This is the initial, untrained classifier of the de-

sired class initialized with appropriate parameters.

• θθθ k: These are the parameters used in the training offk which are identical to θθθ except for any dimensionspecific parameters which may change due to thereduced dimension training.

• k: This is the reduced dimension to be used. k maybe changed to adjust the utility-security trade-off.

• Train: This is the algorithm used to train classifiersof the desired class. An example of a commonlyused training algorithm is Stochastic Gradient De-scent [38] .

There is a second way to implement dimensionalityreduction through principal component analysis.

Algorithm 2 Alternative DRtrain

Input: f 0k ,θθθ k, k, Xtrain, Train

Output: fk1: Use PCA to find the matrix Uk of the top k principal

components used to reduce the dimension of Xtrain

2: Compute the projected training set UkUTk Xtrain

3: Let fk = Train( f 0k ,θθθ k,UkUT

k Xtrain)

For some choices of Train, Algorithms 1 and 2 areequivalent, i.e. they will output identical classifiers giventhe same inputs. One important example of this is LinearSVMs trained using `2 regularization. In this case, Al-gorithms 1 and 2 will clearly select linear functions withthe same behavior on each of the training points. Thisextends by linearity to the subspace of Rn containing allof the projected data points.

Among the linear functions that achieve this behavior,regularization will force Algorithm 2 to select the func-tion with the smallest weight vector in the `2 norm. Thiswill be the linear function that is invariant on all vectorsorthogonal to the subspace containing the projected data.In practice, we prefer Algorithm 1 because it operateson lower dimensional representations of the vectors andthus is more computationally efficient.

3.5 Intuition behind the defense

Now, we will give some intuition about why dimension-ality reduction should improve resilience for support vec-tor machines. We discuss the two-class case for simplic-ity, but the ideas generalize to the multiple class case.

The core of a linear classifier is a function g(x) =wT x+ b. Both x and w can be expressed in the princi-pal component basis as UT x and UT w. We expect manyof the largest coefficients of UT w to correspond to thesmallest eigenvalues of of C = UΛUT .

The reason for this is very simple: in order for differ-ent principal components to achieve the same level of in-fluence on the classifier output, the coefficient of (UT w)imust be proportional to 1/

√λi. To take advantage of

the information in a component with small variation, theclassifier must use a large coefficient. Of course, theprincipal components are not equally useful for classifi-cation. However, among the components that are useful,we expect a general trend of decreasing coefficients of(UT w)i as

√λi increases.

The top-most plot in Figure 2 validates this prediction.The first principal component3 is by far the most use-ful source of classification information. Consequently itdoes not fit the overall trend and actually has the largestcoefficient. However, among the other components thetrend holds.

Furthermore, we expect higher variance componentsto contain better information than lower variance compo-nents. By better information, we mean components thatgeneralize better from the training set to the test set, orequivalently components that mostly correspond to un-derlying features of the classes rather than spurious cor-relations driven by the specific training examples.

Since the optimal attack perturbation for a linear clas-sifier is a multiple of w, the principal components withlarge coefficients are the ones that the attacker takes ad-vantage of. The projection defense denies this opportu-nity to the attacker. By removing all variation from thelower components, it causes the classifier to assign noweight to them. This significantly changes the resultingw that the classifier learns. The classifier loses accessto some information, but accessing that information re-

3on the top right in each plot

7

Figure 2: The magnitudes of the coefficients of the weightvector w in the principal component basis. On the hori-zontal axis, we have

√λi. On the vertical axis, |(UT w)i|.

The top-most plot is from the classifier trained on the origi-nal MNIST data, with no projection (i.e. k = 728). The nextfour plots are the classifiers trained on the projected data withk ∈ {300,100,50,40}.

quired large weight coefficients, so the attacker is hurtfar more.

3.6 Complexity Analysis of DefensesThe defense using PCA adds a one-time O(d2n + d3)overhead for finding the principal components, with thefirst term arising from the covariance matrix computationand the second term from the eigenvector decomposition.There is also a one-time overhead for training a new clas-sifier on the reduced dimension data. The time needed totrain the new classifier will be less than that needed forthe original classifier since the dimensionality of the in-put data has reduced. Each subsequent input will incur aO(dk) overhead due to the matrix multiplication neededto project it onto the principal components.

4 Experimental setup

In this section we provide brief descriptions and imple-mentation details of the datasets, machine learning anddimensionality reduction algorithms and metrics usedin our experiments. We also discuss the reasons forthe choice of the adversarial budgets for various attacksand corroborate them with visual data in the case of theMNIST dataset.

4.1 DatasetsIn our evaluation, we use two datasets. The first is theMNIST image dataset and the second is the UCI HumanActivity Recognition dataset. We now describe each ofthese in detail.

MNIST dataset: This is a dataset of images of hand-written digits [30]. There are 60,000 training examplesand 10,000 test examples. Each image belongs to a sin-gle class from 0 to 9. The images have a dimension of28×28 pixels (total of 784) and are grayscale. The dig-its are size-normalized and centred. This dataset is usedcommonly as a ‘sanity-check’ or first-level benchmarkfor state-of-the-art classifiers. We use this dataset sinceit has been extensively studied from the attack perspec-tive by previous work. It is also easy to visualize theeffects of our defenses on this dataset.

UCI Human Activity Recognition (HAR) usingSmartphones dataset: This is a dataset of measure-ments obtained from a smartphone’s accelerometer andgyroscope [31] while the participants holding it per-formed one of six activities. Of the 30 participants, 21were chosen to provide the training data, and the remain-ing 9 the test data. There are 7352 training samples and2947 test samples. Each sample has 561 features, whichare various signals obtained from the accelerometer andgyroscope. The six classes of activities are Walking,

8

Walking Upstairs, Walking Downstairs, Sitting, Stand-ing and Laying. We used this dataset to demonstrate thatour defenses work across a variety of datasets and appli-cations.

4.2 Machine learning algorithmsWe have evaluated our defenses across a number of ma-chine learning algorithms including linear Support Vec-tor Machines (SVMs) and a variety of neural networkswith different configurations. All experiments were runon a desktop running Ubuntu 14.04, with an 4-coreIntel R© CoreTM i7-6700K CPU running at 4.00GHz, 24GB of RAM and a NVIDIA R© GeForce R© GTX 960 GPU.

Linear SVMs: Ease of training and the interpretabil-ity of separating hyperplane weights has led to the useof Linear SVMs in a wide range of applications [37, 39].We use the LinearSVC implementation from the Pythonpackage scikit-learn [40] for our experiments, which usesthe ‘one-versus-rest’ method for multi-class classifica-tion by default.

In our experiments, we obtained a classification accu-racy of 91.5% for the MNIST dataset and 96.7% for theHAR dataset.

Neural networks: Neural networks can be configuredby changing the number of layers, the activation func-tions of the neurons, the number of neurons in each layeretc. We have performed our experiments on a standardneural network used in previous work, for the purposeof comparison. The network we use is a standard onefrom [16] which we refer to as FC100-100-10. This neu-ral network has an input layer, followed by two hiddenlayers, each containing 100 neurons, and an output soft-max layer containing 10 neurons.FC100-100-10 is trained with a learning rate of 0.01

and momentum of 0.9 for 500 epochs. The size of eachminibatch is 500. On the MNIST test data, we get a clas-sification accuracy of 97.71% for FC100-100-10.

We use Theano [41], a Python library optimized formathematical operations with multi-dimensional arraysand Lasagne [42], a deep learning library that usesTheano, for our experiments on neural networks.

4.3 Dimensionality reduction techniquesPrincipal Component Analysis: We use the PCA mod-ule from scikit-learn [40]. Depending on the application,either the number of components to be projected onto,or the percentage of variance to be retained can be spec-ified. After performing PCA on the vectorized MNISTtraining data to retain 99% of the variance, the reduceddimension is 331, which is the first reduced dimensionwe use in our experiments on PCA based defenses. Afterperforming PCA on the HAR training data to retain 99%

of the variance, the reduced dimension is 155, which isthe first reduced dimension we use in our experiments onPCA based defenses.

4.4 Metrics

4.4.1 Adversarial success

There is a subtlety in the definition of adversarial successin the case of attacks which just aim to cause a misclas-sification, as opposed to a targeted misclassification. Inthis case, there are 3 possible notions of adversarial suc-cess that can be reported of which we explain one here.The other, similar notions are explained in the Appendix.

For each benign input x, we check two conditions: ifafter perturbation, f (x) = f (x) and if initially, f (x) = y,where y is the true label. Thus, the adversary’s attemptis successful if the original classification was correct butthe new classification on the adversarial sample is incor-rect. In a sense, this count represents the number of sam-ples that are truly adversarial, since it is only the adver-sarial perturbation that is causing misclassification, andnot an inherent difficulty for the classifier in classifyingthis sample. While reporting adversarial success rates,we divide this count by the total number of benign sam-ples correctly classified after they pass through the entirerobust classification pipeline.

Throughout Section 5, as we evaluate the performanceof our defense, we will emphasize the budget that theadversary needs to achieve a specified misclassificationrate over the success rate of an attack at a particular bud-get. There are a couple of reasons for this. First, thisallows us to somewhat sidestep the issue of precisely de-termining the amount of perturbation that is detectable bya human. Additionally, as will be seen in Section 5, theperturbation levels required to modify classifier outputsare often fairly concentrated across examples from thesame class. That is, adversarial success rate rises rapidlyat some perturbation budget. This means that successrate is somewhat sensitive to the choice of budget, butmedian required budget is relatively robust.

4.4.2 Perturbation magnitudes

When interpreting the concept of a “perturbation de-tectable by a human”, it is important to distinguish be-tween two variations. The first is a perturbation that willcause a human to misclassify the example. This providesa rough limit on the performance of any classifier: for anadversarial modification that produces an input that be-longs to a different class in some “ground truth” sense(the ground truth being a human in this case), that in-put will not count as a success for the adversary since itsground truth label has changed.

9

While this is a useful limit to keep in mind, it is notwhat is usually meant by “detectable”. The other levelof perturbation is that which a human will recognize asbeing unlikely to occur naturally. In other words, themodified sample looks like it has been tampered with,even though the original class may be more or less rec-ognizable. We illustrate this in Figure 3 using imagesfrom the MNIST dataset modified with a variety of at-tacks. These images validate the perturbation limits tillwhich our experiments are performed, as we demonstratethe effectiveness of our defense against attacks which arewell in the detectable range for humans.

When the ML system owner chooses to either operatein the reduced dimension space, the existence of adver-sarial perturbations may be checked (for image data) byprojecting the input to the ML system back to the orig-inal full dimensional space. A key point, highlighted inFigure 3, is that the adversarial perturbations can be de-tected much more easily when the defense is employed.The perturbations in the image projected back to the orig-inal pixel space from the reduced dimension subspaceare clearly visible, which indicates that while the `2 dis-tances of the perturbations are equal, they are furtherapart perceptually. Thus, evaluating the defense at per-turbation levels where the attack begins to be detected inthe full dimensional case leads to conservative estimatesof its effectiveness.

5 Experimental results

In this section we present an overview of our empiricalresults. The main questions we seek to answer with ourevaluations are:

i) Is the defense effective against strategic attacks?

ii) Is the defense able to defeat vanilla attacks?

iii) Does the defense work for different classifiers?

iv) Does the defense generalize across differentdatasets?

Our evaluation results confirm the effectiveness of ourdefense in a variety of scenarios, each of which has adifferent combination of datasets, machine learning algo-rithms, attacks and dimensionality reduction algorithms.For each set of evaluations, we vary a particular step ofthe classification pipeline and fix the others.

Baseline configuration: We start by considering aclassification pipeline with the MNIST dataset as inputdata, a Linear SVM as our classification algorithm andPCA as the dimensionality reduction algorithm used inour defense. Since we consider the Linear SVM asour classifier, we evaluate its susceptibility to adversar-ial samples generated using the Attack on Linear SVMs

(a) Benign and adversarial images of digit ‘9’ (againsta Linear SVM with no defense): The first image on theleft is the original while the others have been modifiedwith the attack on Linear SVMs with (from left to right),ε = 0.5,1.0,1.5 and 2.0. The perturbation begins to be visi-ble at ε = 1.5 and is very obvious in the image with ε = 2.0.The attack was carried out on a classifier f without any di-mensionality reduction.

(b) Adversarial images of digit ‘7’ (against a neural net-work with no defense): The images have been modifiedwith the Fast Gradient attack on neural networks with (fromleft to right), η ≈ 0.5,1.0,1.5,2.0 and 2.5. Again, the per-turbation begins to be visible at η = 1.5 and is very obviousin the images with η > 2.0. The attack was carried out on aclassifier f without any dimensionality reduction.

(c) Adversarial images of digit ‘7’ (against a neural net-work with a PCA based defense with k = 70): The im-ages have been modified with the Fast Gradient attack ona neural network taking in reduced dimension inputs with(from left to right), η ≈ 0.5,1.0,1.5,2.0 and 2.5. The re-duced dimension vectors were projected back to the imagespace for visualization. In this case, the perturbation beginsto be visible at η = 0.5 and is very obvious in the imageswith η > 1.5. This indicates that our defense also makesadversarial perturbations more easily detectable.

Figure 3: Adversarial images generated to evade LinearSVMs and neural networks.

10

described in Section 2.2. We evaluate our defenses onadversarial samples created starting from the test set foreach dataset. Unless otherwise noted, all security andutility results are for the complete test set. To empiri-cally demonstrate that our defense is resilient not only inthis baseline case, but also various configurations of it,we systematically investigate its effect as each compo-nent of the pipeline, as well as the attacks, are changed.

0

20

40

60

80

100

0 0.5 1 1.5 2 2.5

Adv

ersa

rial

succ

ess

(%)

ε

Vanilla attack on SVM

784200

907050

Figure 4: Effectiveness of the defense for the MNIST datasetagainst vanilla attacks on Linear SVMs. The adversarialexample success on the MNIST dataset is plotted versus theperturbation magnitude ε = ‖x− x‖. The attack is performedagainst the original classifier and the effect of the defense isplotted for each reduced dimension k.

5.1 Effect of defense on Support VectorMachines

In the baseline case, we begin by answering question ii),i.e. ‘Are the defenses able to reduce the effectiveness ofvanilla attacks?’ and i), i.e. ‘Are the defenses able toreduce the effectiveness of strategic attacks?’ for LinearSVMs.

5.1.1 Defense against vanilla attacks

Figure 4 shows the variation in adversarial success forthe defense against vanilla attacks on SVMs. The de-fense significantly reduces adversarial success rates. Forexample, at ε = 1.0, the defense using PCA with a re-duced dimension of k = 50 reduces adversarial successfrom 99.97% to 1.85%. This is a 98.12% or around54× decrease in the adversarial success rate. At ε = 0.5,where the adversarial success rate is 92.77% , the defensewith k = 50 brings the adversarial success rate down tojust 0.9%, which is a 103× (two orders of magnitude)decrease. Training with reduced dimension data leads to

0

20

40

60

80

100

0 0.5 1 1.5 2 2.5

Adv

ersa

rial

succ

ess

(%)

ε

Optimal attack on SVM

784200

9070503010

Figure 5: Effectiveness of the defense for the MNIST datasetagainst optimal attacks on Linear SVMs. The adversarialexample success on the MNIST dataset is plotted versus theperturbation magnitude ε = ‖x− x‖. The attack is performedagainst each reduced dimension classifier and the effect of thedefense is plotted.

0

5

10

15

20

25

0 0.2 0.4 0.6 0.8 1 1.2 1.4

Err

oron

beni

gnda

ta

ε needed for 50 % adv. success

SVM accuracy/robustness tradeoff

784 200 90 70 5030

10

Figure 6: Tradeoff between SVM classification performanceon benign test data, and adversarial performance. Adver-sarial robustness is the value of ‖x− x‖ that allows the adver-sary to achieve a 50% misclassification rate.

11

more robust Linear SVMs, and this can be seen in theadded effectiveness of the defense.

Again, we also notice that as we decrease the reduceddimension k used in the projection step of the defense,adversarial success decreases. At k = 331, the adver-sarial success rate is 48.75% at ε = 1.0, which drops to5.53% when k = 100. At k = 30, at the same ε , the ad-versarial success rate drops to just 2.63% with a modestdecrease to 2.52% with k = 10.

In the case of vanilla attacks, the defense also acts likea noise removal process, removing adversarial perturba-tions and leaving behind the clean input data. This ac-counts for the added robustness we see for vanilla attacksas compared to strategic attacks.

5.1.2 Effect of defenses on optimal attacks

Figure 5 shows the variation in adversarial success forthe defense against the optimal, strategic attack on Lin-ear SVMs. This plot corresponds to the case where theadversary is aware of the dimensionality reduction de-fense and inputs a sample to the pipeline which is de-signed to optimally evade the reduced dimension clas-sifier. At a perturbation magnitude of 0.5, where theclassifier with no defenses has a misclassification rate of99.04%, the reduced dimension classifier with k = 70 hasa misclassification rate of just 19.75%, which representsa 80.25% or 5.01× decrease in the adversarial successrates. However, since 0.5 is a small adversarial budget,the perturbation will simply not be visible even when thereduced dimension image is projected back to the pixelspace. At an adversarial budget of 1.3, where the pertur-bations begin to be clearly visible (see Section 4.4), themisclassification rate for the classifier with no defenses is100%, while it is about 77.11% for the classifier with areduced dimension of 70, which is almost a 23% dropin adversarial success rates. Recall that the perturba-tions required to evade the lower dimensional classifierare more clearly visible to the human eye, making thesefigures conservative.

We can also study the effect of our defense on the ad-versarial budget required to achieve a certain adversar-ial success rate. A budget of 0.3 is required to achievea 86.6% misclassification without the defense, while therequired budget for a classifier with a defense with k = 70is 1.6, which is a 5.33× increase. The correspondingnumbers to achieve a 90% misclassification rate are 0.4without the defense and 1.7 with, which represents a 4×increase. Thus, our defense clearly reduces the effective-ness of an attack carried out by a very powerful adversarywith full knowledge of the defense and the classifier aswell as the ability to carry out optimal attacks.

5.1.3 Utility-security tradeoff for defense

Figure 6 shows the tradeoff between performance underordinary and adversarial conditions. The optimal numberof dimensions for this dataset is clearly between 50 and30, where the kink in the tradeoff occurs. There is verylittle benefit in classification performance by using moredimensions, and essentially no benefit in robustness byusing fewer. At k = 50, we see a drop in classificationsuccess on the test set from 91.5% without any defenses,to 90.29% with the defense. Thus, there is a modest util-ity loss of about 1.2% at this value of k, as compared toa security gain of 5.9×, since the perturbation needed tocause 50 % of the test set to be misclassified increasesfrom 0.16 to 0.95.

With these results, we can conclude that our defenseis effective at least in the baseline case, for both vanillaand optimal attacks against Linear SVMs. Now, we in-vestigate the performance of our defenses on neural net-works, to substantiate our claim of the applicability ofour defenses across machine learning classifiers.

5.2 Effect of defense on neural networksWe now modify the baseline configuration by changingthe classifier used to FC100-100-10. We evaluate our de-fenses on both gradient-ased attacks for FC100-100-10:the Fast Gradient (FG) and Fast Gradient Sign (FGS) at-tacks. We continue to use the MNIST dataset and PCAfor dimensionality reduction. With these experiments,we answer question iii), i.e. ‘Do the defenses work fordifferent classifiers?’ and further strengthen our claimthat our defense is effective against strategic attacks.

5.2.1 Defense for strategic Fast Gradient attack

Figure 7 shows the variation in adversarial success forthe defense as η = ‖x− x‖2, the parameter governingthe strength of the strategic FG attack, increases. Thedefense also reduces adversarial success rates for this at-tack. At η = 1.44, the defense using PCA with a reduceddimension of k = 30 reduces adversarial success from91.95% to 43.41%. This is a 48.54% or around 2.1× de-crease in the adversarial success rate. Again, at η = 2.05,while the adversarial success rate is 98.02% without anydefense, the defense with k = 30 brings the adversar-ial success rate down to 82.25%, which is a 15.77%decrease. Thus, even for neural networks, the defensecauses significant reductions in adversarial success.

Again, we can study the effect of our defense on theadversarial budget required to achieve a certain misclas-sification percentage. A budget of 0.82 is required toachieve 60% misclassification without our defense, whilethe required budget for a classifier with a defense withk = 30 is 1.43, which is a roughly 1.6× increase. The

12

0

20

40

60

80

100

0 0.5 1 1.5 2 2.5

Adv

ersa

rial

succ

ess

(%)

η

Strategic FG Attack on NN

784200

9070503010

Figure 7: Effectiveness of the defense for the MNIST datasetagainst strategic FG attacks on FC100-100-10. The adver-sarial example success on the MNIST dataset is plotted versusthe perturbation magnitude η = ‖x− x‖. The attack is per-formed against each reduced dimension classifier and the effectof the defense is plotted.

0

20

40

60

80

100

0 0.1 0.2 0.3 0.4 0.5

Adv

ersa

rial

succ

ess

(%)

η

Strategic FGS Attack on NN

784200

9070503010

Figure 8: Effectiveness of the defense for the MNIST datasetagainst strategic FGS attacks on FC100-100-10. The adver-sarial example success on the MNIST dataset is plotted versusthe perturbation magnitude η . The attack is performed againsteach subspace classifier (obtained from Algorithm 2) and theeffect of the defense is plotted.

corresponding numbers to achieve a 90% misclassifica-tion rate are 1.43 without the defense and 2.45 with,which represents a roughly 2× increase.

5.2.2 Defense for strategic Fast Gradient Sign at-tack

The FGS attack is constrained in terms of the `∞ norm,so all features with non-zero gradient are perturbed byeither η or −η . The MNIST dataset has pixel valuesnormalized to lie in [0,1]. Thus, we restrict η to be lessthan 0.5, which is a conservative upper bound on the ad-versary’s capability, since the perturbations begin to beclearly visible at η = 0.25, which is also the value usedin previous work [19].

At η = 1.0, the adversarial success rate falls from84.21% to 69.20% for the defense with k = 50 which is a15% reduction. Further, the perturbation needed to cause90% misclassification is about 0.12 without the defensebut increases to 0.15 for the defense with k = 50, whichis a 25% increase.

With these results for neural networks, we concludethat our defenses are effective against a variety of differ-ent attacks, in each of which the nature of the adversarialperturbation is very different. Also, these results demon-strate that our defenses can be used in conjunction withdifferent types of classifiers, providing a general methodfor defending against adversarial inputs.

5.3 Applicability for different datasets

Next, we modify the baseline configuration by changingthe datasets used. We show results with Linear SVMsas the classifier and PCA as the dimensionality reductionalgorithm. We present results for the Human ActivityRecognition dataset.

5.3.1 Defense for the HAR dataset

In Figure 9, the reduction in adversarial success due tothe defense is shown. At ε = 1.0, the adversarial successrate drops from 99.56% without the defense to 91.75%with k = 70 and to 76.21% with k = 30. In order toachieve a misclassification rate of 90%, the amount ofperturbation needed is 0.65 without the defense, whichincreases to 0.876 with k = 70 and to 1.26 with k = 30.Thus, the adversarial budget increases 2× to achieve thesame adversarial success rate. The impact on utility ismodest, with a drop of 2.3% for k = 70 and 5.4% fork = 30, which are small in comparison to the gain in se-curity achieved.

13

MNIST data HAR data

FC100-100-10 Linear SVM Linear SVM

k (MNIST) k (HAR)No D.R. 97.47 91.52 No D.R. 96.67

784 97.32 91.54 561 96.57331 97.35 91.37 200 96.61200 97.04 91.28 100 92.43100 97.36 90.89 90 94.6090 97.14 90.58 80 94.5480 97.25 90.64 70 94.3770 97.52 90.76 60 93.7260 97.38 90.47 50 92.4750 97.26 90.18 40 92.0640 96.71 89.03 30 91.1130 96.56 88.37 20 88.6320 96.67 86.69 10 86.6710 93.22 77.79

Table 2: Utility values for the dimensionality reduction defense. For the MNIST and HAR datasets, the classification accuracyon the benign test set is provided for various values of reduced dimension k used for the PCA based defense, as well as the accuracywithout the defense.

0

20

40

60

80

100

0 0.5 1 1.5 2 2.5

Adv

ersa

rial

succ

ess

(%)

ε

Optimal attack on Linear SVM for HAR data

561200

9070503010

Figure 9: Adversarial example success on the HAR datasetversus perturbation magnitude ε for the Linear SVM attack(against original classifier). Plotted for each reduced dimen-sion k used in the defense.

5.4 Effect of defense on utilityTable 2 presents the effect of our defense on the classi-fication accuracy of benign data. The key takeaways arethat the decrease in accuracy for both neural networksand Linear SVMs for reduced dimensions down to k= 50is at most 4%. Further, we notice that dimensionality re-duction using PCA can actually increase classificationaccuracy as when k = 70, the accuracy on the MNISTdataset increases from 97.47% to 97.52%. More aggres-sive dimensionality reduction however, can lead to steepdrops in classification accuracy, which is to be expectedsince much of the information used for classification islost.

These results highlight the broad applicability of ourdefense across application domains. It is clear that theeffectiveness of our defenses is not an artifact of the par-ticular structure of data from the MNIST dataset, and thatthe intuition for their effect holds across different kindsof data.

6 Discussion and limitations

6.1 Meeting the design goalsIn Section 3.2, we laid out desirable goals that any de-fense should possess. First, the defense should main-tain high classification accuracy. From Table 2, it isclear that for both datasets and classifiers, there is a rangeof reduced dimensions which have a minimal impact onclassification accuracy, and in certain cases may even im-

14

0

20

40

60

80

100

0 1 2 3 4 5

Adv

ersa

rial

succ

ess

(%)

η

FG attack on CNN

None200

9070503010

Figure 10: Effectiveness of the defense for the MNISTdataset against strategic FG attacks on Papernot-CNN .The adversarial example success on the MNIST dataset is plot-ted versus the perturbation magnitude η = ‖x−x‖. The attackis performed against each subspace classifier (obtained fromAlgorithm 2) and the effect of the defense is plotted.

prove it. Secondly, the version of the defense based onPCA adds an overhead which is polynomial in the num-ber of samples n and the number of dimensions d duringboth test and . The time and space required to train thereduced dimension classifier is at most that for the classi-fier in the original high-dimensional space, thus our de-fense maintains high efficiency for the training and testphases. The added security conferred by our defense hasbeen illustrated for a variety of settings in the previoussection. From figure 6, it is clear that changing dimen-sions allow the ML system owner to navigate differentpoints in the utility-security space. Thus, our defensecreates a tunable system.

In summary, it is clear that our defense achieves all ofthe design goals that may be desired of a defense againstevasion attacks on ML systems. However, our defenseis not effective in all plausible scenarios when a systemmay be under attack, which leads to the limitations wediscuss below.

6.2 Limitations

Even though our defense reduces adversarial successrates in a number of cases, there are two main areaswhere it falls short of being a comprehensive defensemechanism against evasion attacks:

1. Insufficient on its own: While our defense causessignificant reductions in adversarial success rates ina variety of settings, there are cases where the ad-versarial success rate is still too high. In such cases,it is likely our defense would have to be combined

with other defenses such as adversarial training [19]and ensemble methods [43] for detection in order tocreate a ML system secure against evasion attacks.Our defense has the advantage that it can be usedin conjunction with a variety of ML classifiers andit will not interfere with the operation of other de-fense mechanisms. Further, as demonstrated in Sec-tion 4, our defense leads to the creation of adversar-ial perturbations that have greater visual percepti-bility. This may aid defenses which aim to detectadversarial perturbations.

2. Lack of universality: In certain settings, our de-fense has limited applicability. For example, in Fig-ure 10, we see that the PCA based defense offerslittle to no security improvements for the Papernot-CNN (See Section 9.3 for details). It is likely thatthis effect stems from the fact that CNNs already in-corporate domain-specific knowledge in their con-volutional layers, and an additional layer of pre-processing using PCA does not confer any addi-tional robustness. Further, PCA may reduce theamount of local information that the convolutionallayers in the CNN are able to use for the purposesof classification.

A key step in addressing these limitations of our de-fense is to use other dimensionality reduction techniqueswhich could reduce adversarial success to negligible lev-els and work better when combined with classifiers suchas CNNs. In future work we plan to explore techniquesfor reducing dimension such as autoencoders, kernelPCA and various compression schemes to better under-stand the relationship between dimensionality reductionand the robustness of classifiers.

7 Related work

7.1 Adversarial machine learningThe lack of robustness of machine learning algorithms toadversarially modified data was first pointed out in [44,45] in the context of spam classifiers. A clear taxon-omy for possible attacks and adversarial knowledge waslaid out in [10–12], with the main adversarial objectivesbeing integrity, availability and privacy violations. Theobjective most relevant to the current work is the in-tegrity violation, where the adversary attempts to causethe misclassification of data for her own benefit. In-tegrity violations can be broadly divided into two cate-gories, poisoning attacks and evasion attacks, and thispaper concentrates on combating the latter. In poison-ing attacks [13, 46–48], the adversary modifies the train-ing data before or during the training phase, in order toachieve objectives such as evasion at test time.

15

7.2 Evasion attacksOn the other hand, in evasion attacks, the adversary’sobjective is to construct samples that are misclassifiedby the ML system, which she does using varying de-grees of knowledge about the system and only at testtime. These attacks have been proposed for a varietyof machine learning classifiers such as Support VectorMachines [14, 17], tree-based classifiers [17, 18] such asrandom forests and boosted trees and more recently forneural networks [15,16,19–22]. The vulnerability of ap-plications that use machine learning, such as face detec-tion [23], voice command recognition [24] and PDF mal-ware detection [25] has also been demonstrated, high-lighting the need for defenses.

7.3 Analyses of evasion attacksThere have been several theoretical attempts to under-stand the reason for vulnerabilities in machine learningsystems. Nelson et al. [33] obtain bounds on the numberof classification queries to a convex-inducing classifierin order to find adversarial samples. Fawzi et al. [35, 49]study the relation between random noise and adversarialperturbations in order to understand why classifiers arerobust to random noise but not to adversarial perturba-tions.

Tanay et al. [50] provide a geometric perspective onadversarial examples, in terms of the distance betweenthe data submanifold and the classifier boundary, for lin-ear classifiers.

7.4 Comparison with previous defensesPrevious work on defenses against adversarial exampleshas largely focused on specific classifier families or ap-plication domains. Further, the existing defenses pro-vide improved security only against existing attacks inthe literature, and it is unclear if the defense mecha-nisms will be effective against adversaries with knowl-edge of their existence, i.e. strategic attacks exploitingweaknesses in the defenses. As a case in point, Paper-not et al. [51] demonstrated a defense using distillationof neural networks against the Jacobian-based saliencymap attack [15]. However, Carlini et al. [21] showed thata modified attack negated the effects of distillation andmade the neural network vulnerable again. Now, we givean overview of the existing defenses in the literature.

7.4.1 Classifier-specific

Russu et al. [26] propose defenses for SVMs byadding various kinds of regularization. Kantchelian etal. [18] propose defenses against optimal attacks de-signed specifically for tree-based classifiers. Existing

defenses for neural networks [52–56] make a variety ofstructural modifications to improve resilience to adver-sarial examples. These defenses do not readily general-ize across classifiers and may still be vulnerable to adver-sarial examples in spite of the modifications, as shown byGu and Rigazio [52].

7.4.2 Application-specific

Hendrycks and Gimpel [57] study transforming imagesfrom the RGB space to YUV space to enable better de-tection by humans and decrease misclassification rates.They also use whitening to make adversarial perturba-tions in RGB images more visible to the human eye. Theeffect of JPG compression on adversarial images has alsobeen studied [58]. Their conclusion was that it has asmall beneficial effect when the perturbations are small.These approaches are restricted to combating evasion at-tacks on image data, and by their nature, do not general-ize across applications.

7.4.3 General defenses

An ensemble of classifiers was used by Smutz andStavrou [43] to detect evasion attacks, by checking fordisagreement between various classifiers. However, anensemble of classifiers may still be vulnerable to adver-sarial examples since they generalize across classfiers.Further, Goodfellow et. al. [19] show that ensemblemethods have limited effectiveness for evasion attacksagainst neural networks. Goodfellow et. al. [19] re-trainon adversarial samples to improve the resilience of neuralnetworks. They find that this method reduces adversar-ial success somewhat, but still leads to high confidencepredictions on the adversarial samples. In our experi-ments, we find that re-training on adversarial samples hasan extremely limited effect on increasing the robustnessof linear SVMs, thus this defense may not be applicableacross classifiers. In [59], random feature nullification isused to reduce adversarial success rates for evasion at-tacks on neural networks. The applicability of this ideaacross classifiers is not studied. [60] use adversarial fea-ture selection to increase the robustness of SVMs. Theyfind and retain features that decrease adversarial successrates. This defense may be generalized across other clas-sifiers and is an interesting direction for future work.

8 Conclusion

In this paper, we considered the novel use of dimension-ality reduction as a defense mechanism against evasionattacks on ML classifiers. Our defenses rely on the in-sight that (a) that dimensionality reduction of data can

16

serve as noise reduction process, helping reduce the mag-nitude of adversarial perturbations, and (b) training clas-sifiers on reduced dimension data leads to enhanced re-silience of ML classifiers.

Using empirical evaluation on multiple real-worlddatasets, we demonstrated a 2x reduction in adversarialsuccess rates across a range of attack strategies (includ-ing strategic ones ), ML classifiers, and applications. Ourdefenses have a modest impact on the utility of the classi-fiers (1-2% reduction), and are computationally efficient.

Our work thus provides an attractive foundation forcountering the threat of evasion attacks.

Acknowledgements

We would like to thank Chawin Sitawarin for help withexperiments and useful discussions. Arjun Nitin Bhagojiis supported by NSF and DARPA. Daniel Cullina is sup-ported by DARPA.

References

[1] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learn-ing,” Nature, vol. 521, no. 7553, pp. 436–444,2015.

[2] R. Collobert, J. Weston, L. Bottou, M. Karlen,K. Kavukcuoglu, and P. Kuksa, “Natural languageprocessing (almost) from scratch,” Journal of Ma-chine Learning Research, vol. 12, no. Aug, pp.2493–2537, 2011.

[3] G. V. Cormack, “Email spam filtering: A system-atic review,” Foundations and Trends in Informa-tion Retrieval, vol. 1, no. 4, pp. 335–455, 2007.

[4] D. CiresAn, U. Meier, J. Masci, and J. Schmidhu-ber, “Multi-column deep neural network for trafficsign classification,” Neural Networks, vol. 32, pp.333–338, 2012.

[5] NVIDIA, “Self driving vehicles development plat-form.”

[6] N. Srndic and P. Laskov, “Hidost: a staticmachine-learning-based detector of maliciousfiles,” EURASIP Journal on Information Security,vol. 2016, no. 1, p. 22, 2016.

[7] G. E. Dahl, J. W. Stokes, L. Deng, and D. Yu,“Large-scale malware classification using randomprojections and neural networks,” in 2013 IEEE In-ternational Conference on Acoustics, Speech andSignal Processing. IEEE, 2013, pp. 3422–3426.

[8] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Im-agenet classification with deep convolutional neuralnetworks,” in Advances in neural information pro-cessing systems, 2012, pp. 1097–1105.

[9] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf,“Deepface: Closing the gap to human-level perfor-mance in face verification,” in Proceedings of theIEEE Conference on Computer Vision and PatternRecognition, 2014, pp. 1701–1708.

[10] L. Huang, A. D. Joseph, B. Nelson, B. I. Rubin-stein, and J. Tygar, “Adversarial machine learning,”in Proceedings of the 4th ACM workshop on Secu-rity and Artificial Intelligence. ACM, 2011, pp.43–58.

[11] M. Barreno, B. Nelson, R. Sears, A. D. Joseph, andJ. D. Tygar, “Can machine learning be secure?” inProceedings of the 2006 ACM Symposium on In-formation, computer and communications security.ACM, 2006, pp. 16–25.

[12] P. Laskov and M. Kloft, “A framework for quan-titative security analysis of machine learning,” inProceedings of the 2nd ACM workshop on Securityand artificial intelligence. ACM, 2009, pp. 1–4.

[13] B. Biggio, B. Nelson, and P. Laskov, “Poisoning at-tacks against support vector machines,” in Proceed-ings of the 29th International Conference on Ma-chine Learning (ICML-12), 2012, pp. 1807–1814.

[14] B. Biggio, I. Corona, D. Maiorca, B. Nelson,N. Srndic, P. Laskov, G. Giacinto, and F. Roli,“Evasion attacks against machine learning at testtime,” in Joint European Conference on MachineLearning and Knowledge Discovery in Databases.Springer, 2013, pp. 387–402.

[15] N. Papernot, P. McDaniel, S. Jha, M. Fredrikson,Z. B. Celik, and A. Swami, “The limitations of deeplearning in adversarial settings,” in 2016 IEEE Eu-ropean Symposium on Security and Privacy (Eu-roS&P). IEEE, 2016, pp. 372–387.

[16] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna,D. Erhan, I. Goodfellow, and R. Fergus, “Intrigu-ing properties of neural networks,” in InternationalConference on Learning Representations, 2014.

[17] N. Papernot, P. McDaniel, and I. Goodfellow,“Transferability in machine learning: from phe-nomena to black-box attacks using adversarial sam-ples,” arXiv preprint arXiv:1605.07277, 2016.

17

[18] A. Kantchelian, J. Tygar, and A. D. Joseph, “Eva-sion and hardening of tree ensemble classifiers,” inProceedings of the 33rd International Conferenceon Machine Learning (ICML-16), 2016.

[19] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Ex-plaining and harnessing adversarial examples,” inInternational Conference on Learning Representa-tions, 2015.

[20] A. Kurakin, I. Goodfellow, and S. Bengio, “Ad-versarial examples in the physical world,” arXivpreprint arXiv:1607.02533, 2016.

[21] N. Carlini and D. Wagner, “Towards evaluating therobustness of neural networks,” in IEEE Sympo-sium on Security and Privacy, 2017.

[22] A. Nguyen, J. Yosinski, and J. Clune, “Deep neu-ral networks are easily fooled: High confidencepredictions for unrecognizable images,” in 2015IEEE Conference on Computer Vision and PatternRecognition (CVPR). IEEE, 2015, pp. 427–436.

[23] M. McCoyd and D. Wagner, “Spoofing 2d face de-tection: Machines see people who aren’t there,”arXiv preprint arXiv:1608.02128, 2016.

[24] N. Carlini, P. Mishra, T. Vaidya, Y. Zhang,M. Sherr, C. Shields, D. Wagner, and W. Zhou,“Hidden voice commands,” in 25th USENIX Secu-rity Symposium (USENIX Security 16), Austin, TX,2016.

[25] W. Xu, Y. Qi, and D. Evans, “Automatically evad-ing classifiers,” in Proceedings of the 2016 Networkand Distributed Systems Symposium, 2016.

[26] P. Russu, A. Demontis, B. Biggio, G. Fumera,and F. Roli, “Secure kernel machines againstevasion attacks,” in Proceedings of the 2016ACM Workshop on Artificial Intelligence andSecurity, ser. AISec ’16. New York, NY, USA:ACM, 2016, pp. 59–69. [Online]. Available:http://doi.acm.org/10.1145/2996758.2996771

[27] J. Shlens, “A tutorial on principal component anal-ysis,” arXiv preprint arXiv:1404.1100, 2014.

[28] L. Van Der Maaten, E. Postma, and J. Van denHerik, “Dimensionality reduction: a comparativereview,” J Mach Learn Res, vol. 10, pp. 66–71,2009.

[29] S.-M. Moosavi-Dezfooli, A. Fawzi, andP. Frossard, “Deepfool: a simple and accuratemethod to fool deep neural networks,” arXivpreprint arXiv:1511.04599, 2015.

[30] Y. LeCun and C. Cortes, “The mnist database ofhandwritten digits,” 1998.

[31] D. Anguita, A. Ghio, L. Oneto, X. Parra, and J. L.Reyes-Ortiz, “A public domain dataset for humanactivity recognition using smartphones.” in ESANN,2013.

[32] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha,Z. Berkay Celik, and A. Swami, “Practical black-box attacks against deep learning systems usingadversarial examples,” in Proceedings of the 2017ACM Asia Conference on Computer and Commu-nications Security.

[33] B. Nelson, B. I. Rubinstein, L. Huang, A. D.Joseph, S.-h. Lau, S. J. Lee, S. Rao, A. Tran,and J. D. Tygar, “Near-optimal evasion of convex-inducing classifiers.” in AISTATS, 2010, pp. 549–556.

[34] B. Biggio, I. Corona, B. Nelson, B. I. Rubinstein,D. Maiorca, G. Fumera, G. Giacinto, and F. Roli,“Security evaluation of support vector machines inadversarial environments,” in Support Vector Ma-chines Applications. Springer, 2014, pp. 105–153.

[35] A. Fawzi, O. Fawzi, and P. Frossard, “Analysis ofclassifiers’ robustness to adversarial perturbations,”arXiv preprint arXiv:1502.02590, 2015.

[36] E. Bingham and H. Mannila, “Random projectionin dimensionality reduction: Applications to imageand text data,” in Proceedings of the Seventh ACMSIGKDD International Conference on KnowledgeDiscovery and Data Mining.

[37] B. Scholkopf and A. J. Smola, Learning with Ker-nels: Support Vector Machines, Regularization,Optimization, and Beyond. Cambridge, MA, USA:MIT Press, 2001.

[38] I. Goodfellow, Y. Bengio, and A. Courville, Deeplearning. MIT Press, 2016.

[39] D. Arp, M. Spreitzenbarth, M. Hubner, H. Gascon,and K. Rieck, “Drebin: Effective and explainabledetection of android malware in your pocket.” inNDSS, 2014.

[40] F. Pedregosa, G. Varoquaux, A. Gramfort,V. Michel, B. Thirion, O. Grisel, M. Blondel,P. Prettenhofer, R. Weiss, V. Dubourg, J. Van-derplas, A. Passos, D. Cournapeau, M. Brucher,M. Perrot, and E. Duchesnay, “Scikit-learn: Ma-chine learning in Python,” Journal of MachineLearning Research, vol. 12, pp. 2825–2830, 2011.

18

[41] Theano Development Team, “Theano: A Pythonframework for fast computation of mathematicalexpressions,” arXiv e-print arXiv:1605.02688.

[42] S. Dieleman and J. S. et.al., “Lasagne: Firstrelease.” Aug. 2015. [Online]. Available: http://dx.doi.org/10.5281/zenodo.27878

[43] C. Smutz and A. Stavrou, “When a tree falls: Us-ing diversity in ensemble classifiers to identify eva-sion in malware detectors,” in 23nd Annual Net-work and Distributed System Security Symposium,NDSS 2016.

[44] D. Lowd and C. Meek, “Adversarial learning,” inProceedings of the eleventh ACM SIGKDD interna-tional conference on Knowledge discovery in datamining. ACM, 2005, pp. 641–647.

[45] N. Dalvi, P. Domingos, S. Sanghai, D. Verma et al.,“Adversarial classification,” in Proceedings of thetenth ACM SIGKDD international conference onKnowledge discovery and data mining. ACM,2004, pp. 99–108.

[46] B. I. Rubinstein, B. Nelson, L. Huang, A. D.Joseph, S.-h. Lau, S. Rao, N. Taft, and J. Tygar,“Antidote: understanding and defending againstpoisoning of anomaly detectors,” in Proceedingsof the 9th ACM SIGCOMM conference on Internetmeasurement conference. ACM, 2009, pp. 1–14.

[47] ——, “Stealthy poisoning attacks on pca-basedanomaly detectors,” ACM SIGMETRICS Perfor-mance Evaluation Review, vol. 37, no. 2, pp. 73–74, 2009.

[48] M. Kloft and P. Laskov, “Online anomaly detectionunder adversarial impact.” in AISTATS, 2010, pp.405–412.

[49] A. Fawzi, S.-M. Moosavi-Dezfooli, andP. Frossard, “Robustness of classifiers: fromadversarial to random noise,” arXiv preprintarXiv:1608.08967, 2016.

[50] T. Tanay and L. Griffin, “A boundary tiltingpersepective on the phenomenon of adversarial ex-amples,” arXiv preprint arXiv:1608.07690, 2016.

[51] N. Papernot, P. D. McDaniel, X. Wu, S. Jha, andA. Swami, “Distillation as a defense to adversar-ial perturbations against deep neural networks,”in IEEE Symposium on Security and Privacy, SP2016, 2016, pp. 582–597.

[52] S. Gu and L. Rigazio, “Towards deep neural net-work architectures robust to adversarial examples,”arXiv preprint arXiv:1412.5068, 2014.

[53] U. Shaham, Y. Yamada, and S. Negahban, “Under-standing adversarial training: Increasing local sta-bility of neural nets through robust optimization,”arXiv preprint arXiv:1511.05432, 2015.

[54] Q. Zhao and L. D. Griffin, “Suppressing the un-usual: towards robust cnns using symmetric activa-tion functions,” arXiv preprint arXiv:1603.05145,2016.

[55] Y. Luo, X. Boix, G. Roig, T. Poggio, and Q. Zhao,“Foveation-based mechanisms alleviate adversar-ial examples,” arXiv preprint arXiv:1511.06292,2015.

[56] R. Huang, B. Xu, D. Schuurmans, andC. Szepesvari, “Learning with a strong adver-sary,” CoRR, abs/1511.03034, 2015.

[57] D. Hendrycks and K. Gimpel, “Visible progress onadversarial images and a new saliency map,” arXivpreprint arXiv:1608.00530, 2016.

[58] G. K. Dziugaite, Z. Ghahramani, and D. M. Roy,“A study of the effect of jpg compression on adver-sarial images,” arXiv preprint arXiv:1608.00853,2016.

[59] Q. Wang, W. Guo, K. Zhang, X. Xing, C. L. Giles,and X. Liu, “Random feature nullification for ad-versary resistant deep architecture,” arXiv preprintarXiv:1610.01239, 2016.

[60] F. Zhang, P. P. Chan, B. Biggio, D. S. Yeung, andF. Roli, “Adversarial feature selection against eva-sion attacks,” IEEE Transactions on Cybernetics,vol. 46, no. 3, pp. 766–777, 2016.

9 Appendix

9.1 Measuring adversarial successRecall that we used a particular notion of adversarial suc-cess in Section 4. There are two other related notionswhich may be used:

• For each x, we check if yadv(= f (xadv)) = f (x) ornot. This counts the total number of adversarialsamples for which the perturbation causes a changefrom the class assigned by the classifier for theclean sample x. It may be the case that neitherthe clean nor the adversarial sample are assignedthe true class y, since the classifier does not have100% accuracy on the test set (possibly on the train-ing set as well). However, it may also be the casethat adding the perturbation causes the classifier tocorrectly classify a previously erroneous input, i.e.

19

we may have the case that f (x) 6= y, but yadv = y.This is an unlikely but possible occurrence.

• For each x, we check if yadv = y or not. Thiscounts the total number of adversarial samples forwhich the class after perturbation is not equal to thetrue class. However, this number will also includethose adversarial samples for which the clean start-ing samples were already wrongly classified. In thiscase, regardless of the efficacy of the attack or de-fense, the percentage of incorrectly classified adver-sarial samples cannot be less than the baseline clas-sifier inaccuracy on the benign test samples, whichrepresents a lower limit on the effectiveness of anyattack.

It is unclear which of these 3 counts have been reportedas adversarial success in previous work. We computed all3 counts in our experiments and found they were similar.

9.2 Intuition for use of test data for defenseevaluation

Data used: Evasion attacks typically involve the mod-ification of existing samples so that they are classifiedincorrectly. An attack can be deemed to be effective ifit leads to high misclassification rates on adversarial ex-amples crafted from the training set. The classifier mayhave poor accuracy on the test set due to various reasons,and isolating the adversarial modification as the reasonfor misclassification may be problematic. Further, sinceclassifiers are typically trained till they have very highaccuracy on the training set, and their decision bound-aries reflect the distribution of the training data, an attackinvolving minimal modification of the training data is asuccessful one.

We evaluate the defense on adversarially modifiedsamples from the test set. The main reason for this isthat overfitting on the training set could be a possiblereason for the effectiveness of the defense on it, so anaccurate evaluation of the defense should involve adver-sarial samples crafted from the test set. So, a defensemechanism makes a classifier more secure if the classifi-cation accuracy of the pipeline on adversarially modifiedsamples from the test set is higher than that of the origi-nal classifier. The higher the accuracy, the more accuratethe defense is.

9.3 CNNsWe also run our experiments on a Convolutional NeuralNetwork [38] whose architecture we obtain from Paper-not et al. [51]. This CNNs architecture is as follows: ithas 2 convolutional layers of 32 filters each, followed bya max pooling layer, then another 2 convolutional layers

of 64 filters each, followed by a max pooling layer. Fi-nally, we have two fully connected layers with 200 neu-rons each, followed by a softmax output with 10 neurons(for the 10 classes in MNIST). All neurons in the hid-den layers are ReLUs. We call this network Papernot-CNN. It is trained with a learning rate of 0.1 (adjusted to0.01 for the last 10 epochs) and momentum of 0.9 for 50epochs. The batchsize is 500 samples for MNIST and onthe MNIST test data we get a classification accuracy of98.91% with the Papernot-CNN network.

20