Model Inversion Attacks that Exploit Confidence Informationand Basic Countermeasures

Matt FredriksonCarnegie Mellon University

Somesh JhaUniversity of Wisconsin–Madison

Thomas RistenpartCornell Tech

ABSTRACTMachine-learning (ML) algorithms are increasingly utilizedin privacy-sensitive applications such as predicting lifestylechoices, making medical diagnoses, and facial recognition. Ina model inversion attack, recently introduced in a case studyof linear classifiers in personalized medicine by Fredriksonet al. [13], adversarial access to an ML model is abusedto learn sensitive genomic information about individuals.Whether model inversion attacks apply to settings outsidetheirs, however, is unknown.

We develop a new class of model inversion attack thatexploits confidence values revealed along with predictions.Our new attacks are applicable in a variety of settings, andwe explore two in depth: decision trees for lifestyle surveysas used on machine-learning-as-a-service systems and neuralnetworks for facial recognition. In both cases confidence val-ues are revealed to those with the ability to make predictionqueries to models. We experimentally show attacks that areable to estimate whether a respondent in a lifestyle surveyadmitted to cheating on their significant other and, in theother context, show how to recover recognizable images ofpeople’s faces given only their name and access to the MLmodel. We also initiate experimental exploration of naturalcountermeasures, investigating a privacy-aware decision treetraining algorithm that is a simple variant of CART learn-ing, as well as revealing only rounded confidence values. Thelesson that emerges is that one can avoid these kinds of MIattacks with negligible degradation to utility.

1. INTRODUCTIONComputing systems increasingly incorporate machine learn-

ing (ML) algorithms in order to provide predictions of lifestylechoices [6], medical diagnoses [20], facial recognition [1],and more. The need for easy “push-button” ML has evenprompted a number of companies to build ML-as-a-servicecloud systems, wherein customers can upload data sets, trainclassifiers or regression models, and then obtain access toperform prediction queries using the trained model — all

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from Permissions@acm.org.CCS’15, October 12–16, 2015, Denver, Colorado, USA.Copyright is held by the owner/author(s). Publication rights licensed to ACM.ACM 978-1-4503-3832-5/15/10 ...$15.00.DOI: http://dx.doi.org/10.1145/2810103.2813677 .

over easy-to-use public HTTP interfaces. The features usedby these models, and queried via APIs to make predictions,often represent sensitive information. In facial recognition,the features are the individual pixels of a picture of a per-son’s face. In lifestyle surveys, features may contain sensitiveinformation, such as the sexual habits of respondents.

In the context of these services, a clear threat is thatproviders might be poor stewards of sensitive data, allow-ing training data or query logs to fall prey to insider at-tacks or exposure via system compromises. A number ofworks have focused on attacks that result from access to(even anonymized) data [18,29,32,38]. A perhaps more sub-tle concern is that the ability to make prediction queriesmight enable adversarial clients to back out sensitive data.Recent work by Fredrikson et al. [13] in the context of ge-nomic privacy shows a model inversion attack that is ableto use black-box access to prediction models in order to es-timate aspects of someone’s genotype. Their attack worksfor any setting in which the sensitive feature being inferredis drawn from a small set. They only evaluated it in a singlesetting, and it remains unclear if inversion attacks pose abroader risk.

In this paper we investigate commercial ML-as-a-serviceAPIs. We start by showing that the Fredrikson et al. at-tack, even when it is computationally tractable to mount, isnot particularly effective in our new settings. We thereforeintroduce new attacks that infer sensitive features used asinputs to decision tree models, as well as attacks that re-cover images from API access to facial recognition services.The key enabling insight across both situations is that wecan build attack algorithms that exploit confidence valuesexposed by the APIs. One example from our facial recogni-tion attacks is depicted in Figure 1: an attacker can producea recognizable image of a person, given only API access to afacial recognition system and the name of the person whoseface is recognized by it.

ML APIs and model inversion. We provide an overviewof contemporary ML services in Section 2, but for the pur-poses of discussion here we roughly classify client-side accessas being either black-box or white-box. In a black-box setting,an adversarial client can make prediction queries against amodel, but not actually download the model description.In a white-box setting, clients are allowed to download adescription of the model. The new generation of ML-as-a-service systems—including general-purpose ones such asBigML [4] and Microsoft Azure Learning [31]—allow dataowners to specify whether APIs should allow white-box orblack-box access to their models.

Figure 1: An image recovered using a new model in-version attack (left) and a training set image of thevictim (right). The attacker is given only the per-son’s name and access to a facial recognition systemthat returns a class confidence score.

Consider a model defining a function f that takes input afeature vector x1, . . . ,xd for some feature dimension d andoutputs a prediction y = f(x1, . . . ,xd). In the model in-version attack of Fredrikson et al. [13], an adversarial clientuses black-box access to f to infer a sensitive feature, sayx1, given some knowledge about the other features and thedependent value y, error statistics regarding the model, andmarginal priors for individual variables. Their algorithm isa maximum a posteriori (MAP) estimator that picks thevalue for x1 which maximizes the probability of having ob-served the known values (under some seemingly reasonableindependence assumptions). To do so, however, requirescomputing f(x1, . . . ,xd) for every possible value of x1 (andany other unknown features). This limits its applicabilityto settings where x1 takes on only a limited set of possiblevalues.

Our first contribution is evaluating their MAP estima-tor in a new context. We perform a case study showingthat it provides only limited effectiveness in estimating sen-sitive features (marital infidelity and pornographic viewinghabits) in decision-tree models currently hosted on BigML’smodel gallery [4]. In particular the false positive rate is toohigh: our experiments show that the Fredrikson et al. algo-rithm would incorrectly conclude, for example, that a per-son (known to be in the training set) watched pornographicvideos in the past year almost 60% of the time. This mightsuggest that inversion is not a significant risk, but in fact weshow new attacks that can significantly improve inversionefficacy.

White-box decision tree attacks. Investigating the ac-tual data available via the BigML service APIs, one sees thatmodel descriptions include more information than leveragedin the black-box attack. In particular, they provide thecount of instances from the training set that match eachpath in the decision tree. Dividing by the total number ofinstances gives a confidence in the classification. While apriori this additional information may seem innocuous, weshow that it can in fact be exploited.

We give a new MAP estimator that uses the confidenceinformation in the white-box setting to infer sensitive in-formation with no false positives when tested against twodifferent BigML decision tree models. This high precisionholds for target subjects who are known to be in the trainingdata, while the estimator’s precision is significantly worsefor those not in the training data set. This demonstratesthat publishing these models poses a privacy risk for thosecontributing to the training data.

Our new estimator, as well as the Fredrikson et al. one,query or run predictions a number of times that is linearin the number of possible values of the target sensitive fea-ture(s). Thus they do not extend to settings where featureshave exponentially large domains, or when we want to inverta large number of features from small domains.

Extracting faces from neural networks. An exampleof a tricky setting with large-dimension, large-domain datais facial recognition: features are vectors of floating-pointpixel data. In theory, a solution to this large-domain in-version problem might enable, for example, an attacker touse a facial recognition API to recover an image of a persongiven just their name (the class label). Of course this wouldseem impossible in the black-box setting if the API returnsanswers to queries that are just a class label. Inspecting fa-cial recognition APIs, it turns out that it is common to givefloating-point confidence measures along with the class label(person’s name). This enables us to craft attacks that castthe inversion task as an optimization problem: find the inputthat maximizes the returned confidence, subject to the clas-sification also matching the target. We give an algorithm forsolving this problem that uses gradient descent along withmodifications specific to this domain. It is efficient, despitethe exponentially large search space: reconstruction com-pletes in as few as 1.4 seconds in many cases, and in 10–20minutes for more complex models in the white-box setting.

We apply this attack to a number of typical neural network-style facial recognition algorithms, including a softmax clas-sifier, a multilayer perceptron, and a stacked denoising auto-encoder. As can be seen in Figure 1, the recovered imageis not perfect. To quantify efficacy, we perform experimentsusing Amazon’s Mechanical Turk to see if humans can usethe recovered image to correctly pick the target person out ofa line up. Skilled humans (defined in Section 5) can correctlydo so for the softmax classifier with close to 95% accuracy(average performance across all workers is above 80%). Theresults are worse for the other two algorithms, but still beatrandom guessing by a large amount. We also investigate re-lated attacks in the facial recognition setting, such as usingmodel inversion to help identify a person given a blurred-outpicture of their face.

Countermeasures. We provide a preliminary explorationof countermeasures. We show empirically that simple mech-anisms including taking sensitive features into account whileusing training decision trees and rounding reported confi-dence values can drastically reduce the effectiveness of ourattacks. We have not yet evaluated whether MI attacksmight be adapted to these countermeasures, and this sug-gests the need for future research on MI-resistant ML.

Summary. We explore privacy issues in ML APIs, showingthat confidence information can be exploited by adversar-ial clients in order to mount model inversion attacks. Weprovide new model inversion algorithms that can be usedto infer sensitive features from decision trees hosted on MLservices, or to extract images of training subjects from facialrecognition models. We evaluate these attacks on real data,and show that models trained over datasets involving surveyrespondents pose significant risks to feature confidentiality,and that recognizable images of people’s faces can be ex-tracted from facial recognition models. We evaluate prelim-inary countermeasures that mitigate the attacks we develop,and might help prevent future attacks.

2. BACKGROUNDOur focus is on systems that incorporate machine learning

models and the potential confidentiality threats that arisewhen adversaries can obtain access to these models.

ML basics. For our purposes, an ML model is simply adeterministic function f : Rd 7→ Y from d features to a setof responses Y . When Y is a finite set, such as the namesof people in the case of facial recognition, we refer to f as aclassifier and call the elements in Y the classes. If insteadY = R, then f is a regression model or simply a regression.

Many classifiers, and particularly the facial recognitionones we target in Section 5, first compute one or more re-gressions on the feature vector for each class, with the cor-responding outputs representing estimates of the likelihoodthat the feature vector should be associated with a class.These outputs are often called confidences, and the classi-fication is obtained by choosing the class label for the re-gression with highest confidence. More formally, we definef in these cases as the composition of two functions. Thefirst is a function f̃ : Rd 7→ [0, 1]m where m is a parame-ter specifying the number of confidences. For our purposesm = |Y | − 1, i.e., one less than the number of class labels.The second function is a selection function t : [0, 1]m → Ythat, for example when m = 1, might output one label ifthe input is above 0.5 and another label otherwise. Whenm > 1, t may output the label whose associated confidenceis greatest. Ultimately f(x) = t(f̃(x)). It is common amongAPIs for such models that classification queries return bothf(x) as well as f̃(x). This provides feedback on the model’sconfidence in its classification. Unless otherwise noted wewill assume both are returned from a query to f , with theimplied abuse of notation.

One generates a model f via some (randomized) trainingalgorithm train. It takes as input labeled training data db, asequence of (d+1)-dimensional vectors (x, y) ∈ Rd×Y wherex = x1, . . . ,xd is the set of features and y is the label. Werefer to such a pair as an instance, and typically we assumethat instances are drawn independently from some prior dis-tribution that is joint over the features and responses. Theoutput of train is the model1 f and some auxiliary informa-tion aux. Examples of auxiliary information might includeerror statistics and/or marginal priors for the training data.

ML APIs. Systems that incorporate models f will do sovia well-defined application-programming interfaces (APIs).The recent trend towards ML-as-a-service systems exem-plifies this model, where users upload their training datadb and subsequently use such an API to query a modeltrained by the service. The API is typically provided overHTTP(S). There are currently a number of such services,including Microsoft Machine Learning [31], Google’s Predic-tion API [16], BigML [4], Wise.io [40], that focus on analyt-ics. Others [21,26,36] focus on ML for facial detection andrecognition.

Some of these services have marketplaces within whichusers can make models or data sets available to other users.A model can be white-box, meaning anyone can downloada description of f suitable to run it locally, or black-box,meaning one cannot download the model but can only make

1We abuse notation and write f to denote both the functionand an efficient representation of it.

prediction queries against it. Most services charge by thenumber of prediction queries made [4,16,31,36].

Threat model. We focus on settings in which an adver-sarial client seeks to abuse access to an ML model API.The adversary is assumed to have whatever information theAPI exposes. In a white-box setting this means access todownload a model f . In a black-box setting, the attackerhas only the ability to make prediction queries on featurevectors of the adversary’s choosing. These queries can beadaptive, however, meaning queried features can be a func-tion of previously retrieved predictions. In both settings, theadversary obtains the auxiliary information aux output bytraining, which reflects the fact that in ML APIs this datais currently most often made public.2

We will focus on contexts in which the adversary doesnot have access to the training data db, nor the ability tosample from the joint prior. While in some cases trainingdata is public, our focus will be on considering settings withconfidentiality risks, where db may be medical data, lifestylesurveys, or images for facial recognition. Thus, the adver-sary’s only insight on this data is indirect, through the MLAPI.

We do not consider malicious service providers, and wedo not consider adversarial clients that can compromise theservice, e.g., bypassing authentication somehow or otherwiseexploiting a bug in the server’s software. Such threats areimportant for the security of ML services, but already knownas important issues. On the other hand, we believe that thekinds of attacks we will show against ML APIs are moresubtle and unrecognized as threats.

3. THE FREDRIKSON ET AL. ATTACKWe start by recalling the generic model inversion attack

for target features with small domains from Fredrikson etal. [13].

Fredrikson et al. considered a linear regression model fthat predicted a real-valued suggested initial dose of thedrug Warfarin using a feature vector consisting of patient de-mographic information, medical history, and genetic mark-ers.3 The sensitive attribute was considered to be the ge-netic marker, which we assume for simplicity to be the firstfeature x1. They explored model inversion where an at-tacker, given white-box access to f and auxiliary informa-

tion side(x, y)def= (x2, . . . ,xt, y) for a patient instance (x, y),

attempts to infer the patient’s genetic marker x1.Figure 2 gives their inversion algorithm. Here we assume

aux gives the empirically computed standard deviation σfor a Gaussian error model err and marginal priors p =(p1, . . . ,pt). The marginal prior pi is computed by firstpartitioning the real line into disjoint buckets (ranges of val-ues), and then letting pi(v) for each bucket v be the thenumber of times xi falls in v over all x in db divided by thenumber of training vectors |db|.

In words, the algorithm simply completes the target fea-ture vector with each of the possible values for x1, and thencomputes a weighted probability estimate that this is thecorrect value. The Gaussian error model will penalize val-

2For example, BigML.io includes this data on web pagesdescribing black-box models in their model gallery [4].3The inputs were processed in a standard way to make nom-inal valued data real-valued. See [13] for details.

adversary Af (err,pi,x2, . . . ,xt, y):1: for each possible value v of x1 do2: x′ = (v,x2, . . . ,xt)3: rv ← err(y, f(x′)) ·

∏i pi(xi)

4: Return arg maxv rv

Figure 2: Generic inversion attack for nominal tar-get features.

ues of x1 that force the prediction to be far from the givenlabel y.

As argued in their work, this algorithm produces the least-biased maximum a posteriori (MAP) estimate for x1 giventhe available information. Thus, it minimizes the adver-sary’s misprediction rate. They only analyzed its efficacy inthe case of Warfarin dosing, showing that the MI algorithmabove achieves accuracy in predicting a genetic marker closeto that of a linear model trained directly from the originaldata set.

It is easy to see that their algorithm is, in fact, black-box, and in fact agnostic to the way f works. That meansit is potentially applicable in other settings, where f is nota linear regression model but some other algorithm. Anobvious extension handles a larger set of unknown features,simply by changing the main loop to iterate over all possiblecombinations of values for the unknown features. Of coursethis only makes sense for combinations of nominal-valuedfeatures, and not truly real-valued features.

However, the algorithm proposed by Fredrikson et al. hasvarious limitations. Most obviously, it cannot be used whenthe unknown features cover an intractably large set. Oneexample of such a setting is facial recognition, where thefeature space is huge: in the facial recognition examples con-sidered later d ≈ 10, 304 with each feature a real number in[0, 1]. Even if one only wanted to infer a portion of the fea-tures this is computationally infeasible. A more subtle issue,explored in the next section, is that even when efficient itmay not be effective.

It turns out that we can give attacks that overcome bothissues above by giving inversion attacks that utilize the con-fidence information revealed by ML APIs. To do so will re-quire new techniques, which we explore using as case stud-ies decision trees (the next section) and facial recognition(Section 5).

4. MAP INVERTERS FOR TREESWe turn now to inversion attacks on decision trees. This

type of model is used on a wide range of data, and is oftenfavored because tree-structured rules are easy for users tounderstand. There are two types of decision trees commonin the literature: classification (where the class variable isdiscrete), and regression (where the class variable is contin-uous). In the following, we focus on classification trees.

Decision tree background. A decision tree model re-cursively partitions the feature space into disjoint regionsR1, . . . , Rm. Predictions are made for an instance (x, y) byfinding the region containing x, and returning the most likelyvalue for y observed in the training data within that region.

x1

0 x2

1 0

1 0

1 0

φ1(x) = x1 w1 = 0φ2(x) = (1− x1)(x2) w2 = 1φ3(x) = (1− x1)(1− x2) w3 = 0

Figure 3: Decision tree for the formula y = ¬x1 ∧ x2.

We characterize trees mathematically as follows:

f(x) =

m∑i=1

wiφi(x), where φi(x) ∈ {0, 1}

where each basis function φi is an indicator for Ri, and wicorresponds to the most common response observed in thetraining set within Ri. A simple example is shown in Fig-ure 3. Notice that there is exactly one basis function foreach path through the tree, and that a given x will “acti-vate” only one φi (and thus traverse only one path throughthe tree) because the basis functions partition the featurespace.

Decision trees can be extended to return confidence mea-sures for classifications by changing the form the wi coeffi-cients take. This is usually accomplished by setting wi toa vector corresponding to the distribution of class labels ina given region, as observed in the training set. Looking atthe example in Figure 3, we would set w1 = [89, 11] if therewere 89 training examples with x1 = 1 and x2 = 0, and 11with x1 = 1,x2 = 1. The classification and correspondingconfidences are given by:

f(x) = arg maxj(∑m

i=1 wi[j]φi(x))

, and

f̃(x) =

[wi∗ [1]∑i w1[i]

, . . . ,wi∗ [|Y |]∑i wm[i]

]where i∗ in the second formula takes the value in {1, . . . ,m}such that φi∗(x) = 1.

Decision tree APIs. Several web APIs expose trainingand querying routines for decision trees to end-users, in-cluding Microsoft Machine Learning [31], Wise.io [40], andBigML [4]. Users can upload their datasets to these services,train a decision tree to predict selected variables, and thenmake the resulting tree available for others to use for free orfor charge per-query. We use as running example the APIexposed by BigML, as it currently has the largest market-place of trained models and focuses on decision trees. Theresults carry over to services with a similar API as well.

BigML allows users to publish trees in either black-box orwhite-box mode. In black-box mode, other users are onlyallowed to query the decision tree for predictions using aREST API. In white-box mode, users are allowed to see theinternal structure of the tree, as well as download a JSONrepresentation of it for local use. The amount of availableinformation about the training set therefore varies betweenthe two modes. In both settings, the adversary has access tomarginal priors for each feature of the training set (see Sec-tion 3 for a description of this), in addition to a confusionmatrix C for which Ci,j gives the number of training in-stances with y = i for which the model predicted label j. Inthe white-box setting, the attacker also has access to a countni of the number of training set instances that match path

φi in the tree. This allows one to compute the confidence ofa particular prediction.

The inversion problem. Fix a tree f(x) =∑mi=1 wiφi(x)

and let (x, y) be a target instance that will either be fromthe training data db or not (we will be clear in the followingabout which setting we are in). We assume for simplicitythat there is one sensitive feature, the first, making the tar-get feature set in this case T = {1}; extending our attacksto more than one feature is straightforward. The side infor-mation side`(x, y) = (x`, . . . ,xd, y) for some to-be-specified` ≥ 2. We let K = {`, . . . , d} be the set of known featureindices, and we abuse notation slightly to let xK representthe d− 1 dimensional vector equal to x`, . . . ,xd.

Black-box MI. For black-box MI we turn to the genericalgorithm from Section 3 and adapt it to this setting. Themain difference from the prior work’s setting is that the de-cision tree models we consider produce discrete outputs, andthe error model information is different, being a confusionmatrix as opposed to the standard deviation of a Gaussian.For our purposes here, then, we use the confusion matrix Cand define err(y, y′) ∝ Pr [ f(x) = y′ | y is the true label ].In Section 4.1 we evaluate the algorithm of Figure 2, withthis error model, showing that it has unsatisfying perfor-mance in terms of a prohibitively high false positive rate.

White-box MI. Recall from before that in the white-boxsetting, we not only assume that the attacker knows each φi,but also the number of training samples ni that correspondto φi. From this, he also knows N =

∑mi=1 ni, the total

number of samples in the training set.The known values xK induce a set of paths S = {si}1≤i≤m:

S = {(φi, ni) | ∃x′ ∈ Rd . x′K = xK ∧ φi(x′)}. Each pathcorresponds to a basis function φi and a sample count ni.We let pi denote ni/N , and note that each pi gives us someinformation about the joint distribution on features used tobuild the training set. In the following, we write si as short-hand for the event that a row drawn from the joint priortraverses the path si, i.e., Pr [ si ] corresponds to the proba-bility of drawing a row from the joint prior that traverses si.Observe that pi is an empirical estimate for this quantity,derived from the draws used to produce the training set.

Recall that the basis functions partition the feature space,so we know that x traverses exactly one of the paths in S.Below we denote a specific value for the first feature by vand a specific value for the other d − 1 features as vK . Wewill abuse notation slightly and write φi(v) as shorthandfor the indicator function I(∃x′ ∈ Rd . x′1 = v ∧ φi(x′)).The following estimator characterizes the probability thatx1 = v given that x traverses one of the paths s1, . . . , smand xK = vK :

Pr [ x1 = v | (s1 ∨ · · · ∨ sm) ∧ xK = vK ]

∝m∑i=1

piφi(v) · Pr [ xK = vK ] · Pr [ x1 = v ]∑mj=1 pjφj(v)

∝ 1∑mj=1 pjφj(v)

∑1≤i≤m

piφi(v) · Pr [ x1 = v ] (1)

We refer to this as the white-box with counts (WBWC) esti-mator. The adversary then outputs a value for v that max-imizes (1) as a guess for x1. Like the Fredrikson et al. esti-mator, it returns the MAP prediction given the additionalcount information.

In the preceding analysis, we assumed that the attackerknew all of x except x1. The WBWC estimator can beextended to the general case where the attacker does notknow x2, . . . ,xl (so xK = {l+1, . . . , d−1}) by summing (1)over the unknown variables.

4.1 ExperimentsWe applied the black-box and white-box WBWC attacks

to decision trees trained on two datasets: FiveThirtyEight’s“How Americans Like Their Steak” survey [17], and a subsetof the General Social Survey (GSS) focusing on responses re-lated to marital happiness [33]. Each dataset contains rowsthat correspond to individuals, with attributes correspond-ing to survey responses. These datasets contain at leastone sensitive feature, and have been used to derive modelsthat are available on BigML’s marketplace. Additionally,the source data is public, which makes them appropriatesurrogates for our study—without source data, we cannotevaluate the effectiveness of our attacks.

FiveThirtyEight survey. In May 2014, Walt Hickey wrotean article for FiveThirtyEight’s DataLab section that at-tempted a statistical analysis of the connection between peo-ples’ steak preparation preferences and their propensity forrisk-taking behaviors [17]. To support the analysis, FiveThir-tyEight commissioned a survey of 553 individuals from Sur-veyMonkey, which collected responses to questions such as:“Do you ever smoke cigarettes?”, “Have you ever cheatedon your significant other?”, and of course, “How do you likeyour steak prepared?”. Demographic characteristics such asage, gender, household income, education, and census regionwere also collected. We discarded rows that did not containresponses for the infidelity question or the steak preparationquestion, resulting in a total of 332 rows for the inversionexperiments. We used model inversion on the decision treelearned from this dataset to infer whether each participantresponded “Yes” to the question about infidelity.

GSS marital happiness survey. The General Social Sur-vey (GSS) collects detailed information on the demograph-ics, interests, and attitudes of United States residents [37].We use a subset of the GSS data [33] created by JosephPrice for the purposes of studying various societal effects ofpornography. This subset corresponds to 51,020 individualsand 11 variables, including basic demographic informationand responses to questions such as, “How happy are you inyour marriage?” and “Have you watched X-rated movies inthe last year?” We discarded rows that did not contain re-sponses to either of these questions, resulting in 16,127 totalrows for the inversion experiments. We use model inversionto infer each participant’s response to the latter question.

Summary of results. For both datasets, we were able toidentify positive instances of the sensitive variable (i.e., a“Yes” answer to “Have you ever cheated on your significantother?” or “Have you watched X-rated movies in the lastyear?”) with high precision. Key findings are:

• Given white-box access to the BigML trees published byothers for these datasets, we are able to predict positiveinstances with perfect precision, i.e., no false positives.

• Individuals whose responses are used in the training dataare at significantly higher risk than individuals not in-cluded in the training data. The results are stark: onthe FiveThirtyEight survey, white-box inversion yields

on average 593× improved precision and 371× improvedrecall. Similar results hold for the GSS survey.

• White-box access to decision trees enhances the adver-sary’s advantage. On the BigML tree trained using GSSdata, the white-box adversary has a significant boost inprecision over the black-box adversary (158% greater)for a modest decrease in recall (32% less). In otherwords, the white-box adversary is able to identify slightlyfewer “Yes” responses than the black-box adversary, butwith much better (i.e., perfect) precision.

Our results indicate that releasing a decision tree trainedover either of these datasets substantially increases the riskof confidential data leakage for the individuals who providedresponses. Details are given below.

Methodology. We ran the white-box and black-box inver-sion attacks for each row of the datasets described above. Toevaluate the risk posed by existing, publicly-available mod-els, we used trees obtained from the BigML service [4]. Be-cause the sensitive feature in each dataset is binary-valued,we use precision and recall to measure the effectiveness ofthe attack. In our setting, precision measures the fraction ofpredicted “Yes” responses that are correct, and recall mea-sures the fraction of “Yes” responses in the dataset thatare predicted by the adversary. We also measure the ac-curacy which is defined as the fraction of correct responses.The trees published on BigML used the entire dataset fortraining. To evaluate the effect of training set inclusion,we trained trees locally by constructing 100 trees using de-fault parameters on randomly-sampled stratified trainingsets comprised of 50% of the available data. We downloadedthe models and ran the experiments locally to avoid placinga burden on BigML’s servers. We used a machine with 8Xeon cores running at 2.5 Ghz, with 16G of memory.

To establish a baseline for comparison, we use three pre-diction strategies corresponding to the capabilities of a ran-dom adversary, a baseline adversary, and an ideal adversary.

• The random adversary has access to no information asidefrom the domain of the sensitive attribute. This corre-sponds to an attacker who cannot access a model, andknows nothing about the prior. For both of our datasets,the sensitive feature is binary, so the adversary’s beststrategy is to flip a fair coin.

• The baseline adversary has access only to the marginalprobability distribution for the sensitive attribute, andnot the tree or any other information about the trainingset. The baseline adversary’s best strategy is to alwaysguess the most likely value according to the marginaldistribution (i.e., its mode, “No” on our data).

• The ideal adversary has access to a decision tree trainedfrom the original dataset to predict the sensitive at-tribute, and given the known features for an individual,uses the tree to make a prediction.

This strategy for the ideal adversary is the appropriate onein our setting as it inherits the limitations of the modelclass: because the attack inverts a decision tree to makepredictions, we do not expect those predictions to be moreaccurate than ones made by a decision tree trained to pre-dict the sensitive feature in the forward direction, from the

FiveThirtyEight GSSalgorithm acc. prec. rec. acc. prec. rec.whitebox 86.4 100.0 21.1 80.3 100.0 0.7blackbox 85.8 85.7 21.1 80.0 38.8 1.0random 50.0 50.0 50.0 50.0 50.0 50.0baseline 82.9 0.0 0.0 82.0 0.0 0.0ideal 99.8 100.0 98.6 80.3 61.5 2.3

Figure 4: MI results for for BigML models. Allnumbers shown are percentages.

same data. This kind of same-model-class comparison wasalso used in [13].

Performance. The amount of time needed to run boththe black-box and white-box attacks is negligible, with thewhite-box attack being the most expensive. This attack tookabout 6 milliseconds on average for both datasets, with themost expensive component being path enumeration througha fairly large (about 200 paths) tree. The number of callsneeded to run the black-box attack is small: 4 for the FiveThir-tyEight dataset (there are 2 unknown binary features), and2 for the GSS dataset. We conclude that for datasets similarto these, having few unknown features from small domains,the black-box attack is feasible to execute remotely.

Discussion. Figure 4 shows the full results for our ex-periments. The largest difference between black-box andwhite-box accuracy is precision: the white-box attack gaveno false positives, whereas black-box yielded about 15% inFiveThirtyEight and about 60% on GSS. This shows thatthe instance counts on tree paths gives useful informationabout the joint prior, allowing the attacker to better iden-tify negative instances of the sensitive feature. Measuredby both accuracy and precision, both attacks significantlyoutperform the random-guessing strategy, by at least 30%.However, neither attack achieves higher recall. This is dueto the skewed prior distribution on the sensitive attribute(about 80% “No” in both cases), which leads the attack tofavor answers which do not contribute to higher recall. Theupshot of this condition is much higher precision, which sup-ports greater confidence in positive predictions.

Figure 5a shows the advantage of the MI algorithms overthe baseline strategy, with advantage defined as the increasein accuracy, precision, or recall. Both algorithms comparefavorably in terms of precision on FiveThirtyEight (at least80% improvement) and recall (at least 20% improvement),but are only slightly better in terms of accuracy (3-5% im-provement). However, on GSS accuracy is slightly lower,whereas precision and recall are greater. Figure 5b showsthe results as a percentage of the ideal strategy’s perfor-mance. Reaching 100% of the ideal strategy’s performancemeans that the attack performs as well as one could rea-sonably expect it to. The whitebox attack achieves this forprecision, and comes close (within 15%) for accuracy.

Figure 5c compares the performance of the attack on in-stances from the training set versus instances outside it.Both attacks dramatically increase the attacker’s chancesof predicting the sensitive attribute, by as much as 70% pre-cision and 20% recall. This demonstrates that inclusion inthe training set poses a significant risk to the confidentialityof these individuals’ responses.

acc. prec. rec. acc. prec. rec.0

20

40

60

80

100

FiveThirtyEight GSS

Ad

v.

over

base

lin

e

Black-box White-box Ideal

(a) Results as advantage over baseline.

acc. prec. rec. acc. prec. rec.0

20

40

60

80

100

FiveThirtyEight GSS

%of

Ideal

Baseline Black-box White-box

(b) Results as a percentage of ideal.

acc. prec. rec. acc. prec. rec.0

20

40

60

80

100

GSSFiveThirtyEight

%of

Ideal

BB Test BB Train WB Test WB Train

(c) Training vs. test attack performance.

Figure 5: BigML model inversion comparison to the baseline and ideal prediction strategies. Although thewhite-box attack achieved greater than 100% improvement over the ideal strategy on the FiveThirtyEightmodel, all percentages shown here are cropped at 100% to make the other results legible.

5. FACIAL RECOGNITION MODELSFacial recognition models are functions that label an im-

age containing a face with an identifier corresponding to theindividual depicted in the image. These models are used in-creasingly often for a wide range of tasks such as authenti-cation [8], subject identification in security and law enforce-ment settings [35], augmented reality [9], and photo organi-zation (e.g., Facebook’s “DeepFace” [1]). A growing numberof web APIs support facial recognition, such as those offeredby Lambda Labs [26], Kairos [21], and SkyBiometry [36]. Anumber of local libraries also expose APIs for facial recog-nition, such as OpenCV [5] and OpenBR [24]. Common toall these APIs is the ability to train a model using a set ofimages labeled with the names of individuals that appear inthem, and the ability to perform classification given somepreviously trained model. Notice that classification may beperformed by a larger set of individuals, including ones whodo not have access to the labeled training set; use of the APIsin this way has been suggested, for example, in augmentedreality applications [9].

In this section we discuss two MI attacks on the modelsused by these APIs to violate the privacy of subjects in thetraining set. Both attacks assume that the adversary hasaccess only to a trained model, but not to any of the originaltraining data.

1) In the first attack we assume an adversary who knowsa label produced by the model, i.e. a person’s name orunique identifier, and wishes to produce an image of theperson associated with that label (i.e., the victim). Theadversary uses MI to “reconstruct” an image of the victim’sface from the label. This attack violates the privacy of anindividual who is willing to provide images of themselves astraining data, as the adversary can potentially reconstructimages of every individual in the training set. The adversary“wins” an instance of this attack if, when shown a set of faceimages including the victim, he can identify the victim. Insubsequent text, we refer to this as the reconstruction attack.

2) In the second attack we assume an adversary who hasan image containing a blurred-out face, and wishes to learnthe identity of the corresponding individual. The adversaryuses the blurred image as side information in a series ofMI attacks, the output of which is a deblurred image of thesubject. Assuming the original image was blurred to protectanonymity, this attack violates the privacy of the person inthe image. Note that the image has been blurred to anextent that the model no longer classifies it correctly. Theadversary can only hope to succeed if the individual was in

the training set for the model. Here the adversary wins if heidentifies the victim from a set of face images taken from thetraining set, or if the subject of the blurred image was notin the training set, and the adversary determines that theimage produced by the attack does not correspond to anyof the faces. We refer to this as the deblurring attack. Thisattack builds directly on the reconstruction attack. Dueto space constraints we defer discussing it in detail, withexperimental results, to the companion technical report [14].

5.1 BackgroundThere are many proposed algorithms and models for per-

forming facial recognition. Traditional approaches often re-lied on complex, hand-coded feature extraction, alignment,and adjustment algorithms, but more recently a number ofpromising systems have achieved better performance usingneural networks [1,19,27]. These models are quickly becom-ing the standard by which facial recognition systems areevaluated, so we consider three types of neural network mod-els: softmax regression, a multilayer perceptron network,and a stacked denoising autoencoder network. These mod-els vary in complexity, with softmax regression being thesimplest and the stacked denoising autoencoder network be-ing the most complex.

Softmax regression. This classifier is a generalization oflogistic regression that allows the class variable to take morethan two values—in our case, there are 40 individuals in thedataset, so the classifier needs to distinguish between 40 la-bels. Softmax regression is often used as the final layer indeep neural network architectures, so on its own this classi-fier can be seen as a neural network with no hidden layers.

Multilayer perceptron network. We use a multilayerperceptron network with one hidden layer of 3000 sigmoidneurons (or units) and a softmax output layer. This clas-sifier can be understood as performing softmax regressionafter first applying a non-linear transformation to the fea-ture vector. The point of this transformation, which corre-sponds to the hidden layer, is to map the feature vector intoa lower-dimensional space in which the classes are separableby the softmax output layer.

Stacked denoising autoencoder network. This classi-fier is an example of a deep architecture, and consists of twohidden layers and one softmax output layer. The two hid-den layers, which have 1000 and 300 sigmoid units, are eachinstances of a denoising autoencoder. An autoencoder is aneural network that maps its feature vector to a latent rep-

Algorithm 1 Inversion attack for facial recognition models.

1: function MI-Face(label , α, β, γ, λ)

2: c(x)def= 1− f̃label(x) + AuxTerm(x)

3: x0 ← 04: for i← 1 . . . α do5: xi ← Process(xi−1 − λ · ∇c(xi−1))6: if c(xi) ≥ max(c(xi−1), . . . , c(xi−β)) then7: break8: if c(xi) ≤ γ then9: break

10: return [arg minxi(c(xi)),minxi(c(xi))]

resentation (typically in a smaller space), and then maps itback (i.e., reconstructs) into the original feature space. Au-toencoders are trained to minimize reconstruction error, sovectors in the latent space can be thought of as compressedencodings of the feature space that should decode well for in-stances in the training data. The hope with this architectureis that these encodings capture the main factors of variationin feature space (much like Principal Component Analysis),leading to greater separability for the softmax layer.

Model ErrorSoftmax 7.5%MLP 4.2%DAE 3.3%

Figure 6: Modelaccuracy.

Dataset. We trained each typeof model over the AT&T Lab-oratories Cambridge database offaces [2]. This set contains tenblack-and-white images of 40 indi-viduals in various lighting condi-tions, facial expressions, and de-tails (e.g., glasses/no glasses), fora total of 400 images. We dividedthe images of each person into a training set (7 images) anda validation set (3 images), and trained each model usingpylearn2’s stochastic gradient descent algorithm [15] untilthe model’s performance on the training set failed to im-prove after 100 iterations. The error rate for each model isgiven in Figure 6.

Basic MI attack. We now turn to inversion attacks againstthe models described above. The features that we will at-tempt to invert in this case are the full vector of pixel intensi-ties that comprise an image, and each intensity correspondsto a floating-point value in the range [0, 1]. In all of the at-tacks we consider, we do not assume that the attacker knowsexact values for any of the pixels in the vector he is trying toinfer. These factors combine to make this type of inversionsubstantially different from the previous cases we consider,so these attacks require new techniques.

Assuming feature vectors with n components and m faceclasses, we model each facial recognition classifier as a func-tion, f̃ : [0, 1]n 7→ [0, 1]m. Recall that the output of themodel is a vector of probability values, with the ith compo-nent corresponding to the probability that the feature vectorbelongs to the ith class. We write f̃i(x) as shorthand for theith component of the output.

We use gradient descent (GD) to minimize a cost func-

tion involving f̃ to perform model inversion in this setting.Gradient descent finds the local minima of a differentiablefunction by iteratively transforming a candidate solution to-wards the negative of the gradient at the candidate solution.Our algorithm is given by the function MI-Face in Algo-rithm 1. The algorithm first defines a cost function c in

Algorithm 2 Processing function for stacked DAE.

function Process-DAE(x)encoder .Decode(x)x← NLMeansDenoise(x)x← Sharpen(x)return encoder .Encode(vecx)

Figure 7: Reconstruction without using Process-DAE (Algorithm 2) (left), with it (center), and thetraining set image (right).

terms of the facial recognition model f̃ and a case-specificfunction AuxTerm, which uses any available auxiliary in-formation to inform the cost function. We will describe aninstantiation of AuxTerm when we discuss facial deblur-ring. MI-Face then applies gradient descent for up to αiterations, using gradient steps of size λ. After each step ofgradient descent, the resulting feature vector is given to apost-processing function Process, which can perform vari-ous image manipulations such as denoising and sharpening,as necessary for a given attack. If the cost of the candidatefails to improve in β iterations, or if the cost is at least asgreat as γ, then descent terminates and the best candidateis returned.

MI-Face needs to compute the gradient of the cost func-tion c, which in turn requires computing the gradient of thefacial recognition model f̃ . This means that f̃ must be dif-ferentiable for the attack to succeed. ∇f̃ can be computedmanually or automatically using symbolic techniques. Ourexperiments use symbolic facilities to implement the latterapproach.

5.2 Reconstruction AttackThe first specific attack that we consider, which we will re-

fer to as Face-Rec, supposes that the adversary knows one ofthe labels output by the model and wishes to reconstruct animage depicting a recognizable face for the individual corre-sponding to the label. This attack is a fairly straightforwardinstantiation of MI-Face (Algorithm 1). The attacker hasno auxiliary information aside from the target label, so we

define AuxTerm(x)def= 0 for all x. Our experiments set the

parameters for MI-Face to: α = 5000, β = 100, γ = 0.99,and λ = 0.1; we arrived at these values based on our ex-perience running the algorithm on test data, and out ofconsideration for the resources needed to run a full set ofexperiments with these parameters.

In all cases except for the stacked DAE network, we setProcess to be the identity function. For stacked DAE net-work, we use the function Process-DAE in Algorithm 2.Since the adversary can inspect each of the model’s layers,he can isolate the two autoencoder layers. We configure theattack to generate candidate solutions in the latent space of

the first autoencoder layer, and at each iteration, Process-DAE decodes the candidate into the original pixel space,applies a denoising filter followed by a sharpening filter, andre-encodes the resulting pixels into the latent space. Wefind that this processing removes a substantial amount ofnoise from the final reconstruction, while enhancing recog-nizable features and bringing relative pixel intensities closerto the images from the training set. An example is shownin Figure 7. The image on the left was reconstructed usingthe label of the individual from the right image without theuse of Process-DAE, while the center picture was recon-structed using this processing step.

Experiments. To evaluate the effectiveness of the attack,we ran it on each of the 40 labels in the AT&T Face Database,and asked Mechanical Turk workers to match the recon-structed image to one of five face images from the AT&Tset, or to respond that the displayed image does not corre-spond to one of the five images. Each batch of experimentswas run three times, with the same test images shown toworkers in each run. This allowed us to study the consis-tency of responses across workers. The images shown toTurk workers were taken from the validation set, and thuswere not part of the training data.

In 80% of the experiments, one of the five images con-tained the individual corresponding to the label used in theattack. As a control, 10% of the instances used a plain im-age from the data set rather than one produced by MI-Face.This allowed us to gauge the baseline ability of the workersat matching faces from the training set. In all cases, the im-ages not corresponding to the attack label were selected atrandom from the training set. Workers were paid $0.08 foreach task that they completed, and given a $0.05 bonus ifthey answered the question correctly. We found that work-ers were usually able to provide a response in less than 40seconds. They were allowed to complete at most three tasksfor a given experiment. As a safeguard against random orcareless responses, we only hired workers who have com-pleted at least 1,000 jobs on Mechanical Turk and achievedat least a 95% approval rating.

5.2.1 Performancealgorithm time (s) epochsSoftmax 1.4 5.6MLP 1298.7 3096.3DAE 692.5 4728.5

Figure 8: Attack runtime.

We ran the attack foreach model on an 8-core Xeon machine with16G memory. The re-sults are shown in Fig-ure 8. Reconstructingfaces out of the softmax model is very efficient, taking only1.4 seconds on average and requiring 5.6 epochs (i.e., itera-tions) of gradient descent. MLP takes substantially longer,requiring about 21 minutes to complete and on the orderof 3000 epochs of gradient descent. DAE requires less time(about 11 minutes) but a greater number of epochs. This isdue to the fact that the search takes place in the latent fea-ture space of the first autoencoder layer. Because this hasfewer units than the visible layer of our MLP architecture,each epoch takes less time to complete.

5.2.2 Accuracy resultsThe main accuracy results are shown in Figure 9. In

this figure, overall refers to all correct responses, i.e., theworker selected the image corresponding to the individualtargeted in the attack when present, and otherwise selected

“Not Present”. Identified refers to instances where the tar-geted individual was displayed among the test images, andthe worker identified the correct image. Excluded refers toinstances where the targeted individual was not displayed,and the worker correctly responded “Not Present”.

Figure 9a gives results averaged over all responses, whereas9b only counts an instance as correct when a majority (atleast two out of three) users responded correctly. In bothcases, Softmax produced the best reconstructions, yielding75% overall accuracy and up to an 87% identification rate.This is not surprising when one examines the reconstructionproduced by the three algorithms, as shown in Figure 10.The reconstruction given by Softmax generally has sharp,recognizable features, whereas MLP produces a faint out-line of these features, and DAE produces an image that isconsiderably more blurry. In all cases, the attack signifi-cantly outperforms, by at least a factor of two, randomlyguessing from the six choices as this would give accuracyjust under 20%.

5.3 Black-Box AttacksIt may be possible in some cases to use numeric approxi-

mations for the gradient function in place of the explicit gra-dient computation used above. This would allow a black-boxadversary for both types of attack on facial recognition mod-els. We implemented this approach using scipy’s numericgradient approximation, and found that it worked well forSoftmax models—the reconstructed images look identical tothose produced using explicit gradients. Predictably, how-ever, performance suffers. While Softmax only takes about aminute to complete on average, MLP and DAE models takesignificantly longer. Each numeric gradient approximationrequires on the order of 2d black-box calls to the cost func-tion, each of which takes approximately 70 milliseconds tocomplete. At this rate, a single MLP or DAE experimentwould take 50–80 days to complete. Finding ways to opti-mize the attack using approximate gradients is interestingfuture work.

6. COUNTERMEASURESIn this section, we explore some potential avenues for

developing effective countermeasures against these attacks.Ideally one would develop full-fledged countermeasures forwhich it can be argued that no future MI attack will suc-ceed. Here we do a more limited preliminary investigation,showing discussing simple heuristics that prevent our cur-rent attacks and which might guide future countermeasuredesign.

Decision Trees. Just as the ordering of features in a de-cision tree is crucial for accuracy, it may also be relevant tothe tree’s susceptibility to inversion attacks. In particular,the level at which the sensitive feature occurs may affect theaccuracy of the attack. To test this hypothesis, we imple-mented a variant of CART learning that takes a parameter` which specifies the priority at which the sensitive feature isconsidered: the feature is only considered for splitting after`−1 other features have already been selected, and removedfrom consideration afterwards. We then sampled 90% of theFiveThirtyEight dataset 100 times, and used this algorithmto train a decision tree on each sample for every value of `.We evaluated the classification accuracy of the tree alongsidewhite-box inversion performance.

Softmax MLP DAE

overall identified excluded

20

40

60

80

100

%correct

(a) Average over all responses.

overall identified excluded

20

40

60

80

100

(b) Correct by majority vote.

overall identified excluded

20

40

60

80

100

(c) Accuracy with skilled workers.

Figure 9: Reconstruction attack results from Mechanical Turk surveys. “Skilled workers” are those whocompleted at least five MTurk tasks, achieving at least 75% accuracy.

Target Softmax MLP DAE

Figure 10: Reconstruction of the individual on theleft by Softmax, MLP, and DAE.

1 2 3 4 5 6 7 8 9 10 11 120.8

0.82

0.84

0.86

0.88

0.9

`

MIaccura

cy

1 2 3 4 5 6 7 8 9 10 11 120.3

0.32

0.34

0.36

0.38

0.4

Class.accura

cy

MI accuracy Class. accuracy

Figure 11: White-box MI vs. classification accuracyon decision trees trained on FiveThirtyEight datawith the sensitive feature at each priority level `.For this data, the optimal placement of the sensi-tive feature is at the first level, achieving the bestclassification accuracy while admitting MI accuracyonly 1% greater than baseline.

The results are displayed in Figure 11 (we observed similartrends for black-box performance). The effectiveness of theattack in this case is clearly affected by the depth at whichthe sensitive feature appears in the tree. When the featureappears near the top or bottom of the tree, the attack failswith greater probability than otherwise. Furthermore, al-though prioritizing placement of the sensitive feature at aparticular level does impact model accuracy, there is an op-timal placement in this case: when the feature is placed atthe top of the tree, classification accuracy is maximized whileinversion accuracy is only 1% greater than baseline guess-ing. This suggests that it may be possible to design moresophisticated training algorithms that incorporate model in-version metrics into the splitting criteria in order to achieveresistance to attacks without unduly sacrificing accuracy.

no rounding r = 0.001 r = 0.005 r = 0.01 r = 0.05

Figure 12: Black-box face reconstruction attackwith rounding level r. The attack fails to produce anon-empy image at r = 0.1, thus showing that round-ing yields a simple-but-effective countermeasure.

To understand why attack performance is not monotonein `, we counted the number of times each tree used thesensitive feature as a split. This measure increases until itreaches its maximum at ` = 8, and steadily decreases until` = 12. The difference in split frequency between ` = 8 and` = 12 is approximately 6×. This is most likely because oncemost of the features have been used, the training algorithmdeems further splitting unnecessary, thus omitting the sen-sitive feature from many subtrees. The inversion algorithmis unable to do better than baseline guessing for individu-als matching paths through these subtrees, thus making theattack less effective.

Facial Recognition. Our attacks on facial recognitionmodels are all based on gradient descent. One possible de-fense is to degrade the quality or precision of the gradientinformation retreivable from the model. There is no obviousway to achieve this in the white-box setting while preserv-ing model utility, but in the black-box setting this might beachieved by reducing the precision at which confidence scoresare reported. We tested this approach by rounding the scoreproduced by the softmax model, and running the black-boxreconstruction attack. The results are presented in Figure 12for rounding levels r = {0.001, 0.005, 0.01, 0.05}; the attackfailed to produce an image for r = 0.1. “No rounding” corre-sponds to using raw 64-bit floating-point scores to computenumeric gradients. Notice that even at r = 0.05, the attackfails to produce a recognizable image. This suggests thatblack-box facial recognition models can produce confidencescores that are useful for many purposes while remainingresistant to reconstruction attacks.

7. RELATED WORKMachine-learning techniques are used in a variety of ap-

plications, such as intrusion detection, spam filtering, andvirus detection. The basic idea behind these systems is totrain a classifier that recognizes “normal behavior”, so thatmalicious behaviors as can be labeled as abnormal. An ac-tive adversary can try to subvert these systems in a varietyof ways, such as crafting malicious inputs that the classifiermistakenly labels as normal. This is called an evasion ormimicry attack. An adversary can also try to degrade theperformance of such systems by devising inputs that cre-ate a large number of false alarms, thus overwhelming thesystem administrator. This is a variant of classic denial-of-service attacks. Barreno et al. [3] consider such attacks onmachine learning algorithms. Lowd and Meek [30] extendthese attacks to scenarios where the attacker does not havea complete description of the classifier. In contrast to thisline of work, we target the privacy implications of exposingmachine learning model functionality. However, a connec-tion between these two lines of research will be interestingto explore in the future, as the mechanisms used for ourattacks might prove useful in their settings.

Several authors have explored linear reconstruction at-tacks [10,12,23], which work as follows: given some releasedinformation y and assuming a hidden vector s of sensi-tive features (e.g., Boolean indicators for disease status of agroup of patients), the attacker constructs a system of linearinequalities (a matrix A and a vector z) such that As ≈ z,and attempts to solve for s using techniques that minimizesome norm, such as LP decoding. Kasiviswanathan, Rudel-son, and Smith [22] extended these attacks to releases thatare non-linear, such as M -estimators. We also explore at-tacks on non-linear models, and we further investigate theseattacks in realistic settings. It will be interesting future workto investigate whether strategies such as LP decoding workin settings similar to those considered in this paper.

Many disclosure attacks that have been explored in the lit-erature. The classic attack by Sweeney [38] showed that itwas possible to correlate publicly-available anonymzied hos-pital visit data with voter registration data, leading to re-identification of some individuals. Narayananan [32] demon-strated that an adversary with some prior knowledge canidentify a subscriber’s record in the anonymized Netflix prizedataset. Wang et al. [39], Sankararaman et al. [34], andHomer et al. [18] consider disclosure attacks on datasets gen-erated from Genome-Wide Association Studies (GWAS) andother partial genetic information for an individual. Cormodeshowed that if an adversary is allowed to make queries thatrelate sensitive attributes to quasi-identifiers, then it is pos-sible to build a differentially-private Naive Bayes classifierthat accurately predicts a sensitive attribute [7]. Loukideset al. [29] show that one can infer sensitive patient infor-mation from de-identified clinical data. The main differencebetween this line of work and that presented in this pa-per is the type of information used by the adversary to infersensitive information. Whereas previous work relied primar-ily on de-identified datasets, our attacks operate entirely ontrained machine learning models and the metadata that iscommonly released with them.

Komarova et al. [25] studied the problem of partial disclo-sure, where an adversary is given fixed statistical estimatesfrom combined public and private sources, and attempts toinfer the sensitive feature of an individual referenced in those

sources. The key difference between theirs and our settingis that we assume the adversary is given a statistical estima-tor as a function, and can thus use it directly to make andevaluate predictions about individuals. It is worth studyingwhether the statistics used in their study can be used as ad-ditional side information to boost the performance of modelinversion.

Li et al. [28] explore a privacy framework called member-ship privacy. This framework consists of two notions: pos-itive membership privacy (PMP) and negative membershipprivacy (NMP). The framework gives rise to a number of se-curity notions, including variants of differential privacy [11].These security notions were not designed to resist MI at-tacks, and whether they might help guide countermeasuredesign is an open question. (See also the discussion of dif-ferential privacy in [13].) We also note that MI may serve asa primitive useful to adversaries seeking to violate these pri-vacy notions (e.g., a high MI accuracy might reveal whethera person was in the training set).

8. CONCLUSIONWe demonstrated how the confidence information returned

by many machine learning ML classifiers enables new modelinversion attacks that could lead to unexpected privacy is-sues. By evaluating our model inversion algorithms overdecision trees published on a ML-as-a-service marketplace,we showed that they can be used to infer sensitive responsesgiven by survey respondents with no false positives. Using alarge-scale study on Mechanical Turk, we showed they theycan also be used to extract images from facial recognitionmodels that a large majority of skilled humans are able toconsistently re-identify.

We explored some simple approaches that can be usedto build effective countermeasures to our attacks, initiatingan experimental evaluation of defensive strategies. Althoughthese approaches do not constitute full-fledged private learn-ing algorithms, they illustrate trends that can be used toguide future work towards more complete algorithms. Ourfuture efforts will follow this path, as we continue to worktowards new systems that are able to benefit from advancesin machine learning without introducing vulnerabilities thatlead to model inversion attacks.

9. REFERENCES[1] DeepFace: Closing the Gap to Human-Level

Performance in Face Verification. In Conference onComputer Vision and Pattern Recognition (CVPR).

[2] AT&T Laboratories Cambridge. The ORL database offaces. http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html.

[3] M. Barreno, B. Nelson, R. Sears, A. D. Joseph, andJ. D. Tygar. Can machine learning be secure? InProceedings of the 2006 ACM Symposium onInformation, computer and communications security,pages 16–25. ACM, 2006.

[4] BigML. https://www.bigml.com/.

[5] G. Bradski. The OpenCV library. Dr. Dobb’s Journalof Software Tools, Jan. 2000.

[6] C.-L. Chi, W. Nick Street, J. G. Robinson, and M. A.Crawford. Individualized patient-centered lifestylerecommendations: An expert system forcommunicating patient specific cardiovascular risk

information and prioritizing lifestyle options. J. ofBiomedical Informatics, 45(6):1164–1174, Dec. 2012.

[7] G. Cormode. Personal privacy vs population privacy:learning to attack anonymization. In KDD, 2011.

[8] M. Dabbah, W. Woo, and S. Dlay. Secureauthentication for face recognition. In IEEESymposium on Computational Intelligence in Imageand Signal Processing, pages 121–126, April 2007.

[9] C. Dillow. Augmented identity app helps you identifystrangers on the street. Popular Science, Feb. 23 2010.

[10] I. Dinur and K. Nissim. Revealing information whilepreserving privacy. In PODS, 2003.

[11] C. Dwork. Differential privacy. In ICALP. Springer,2006.

[12] C. Dwork, F. McSherry, and K. Talwar. The price ofprivacy and the limits of lp decoding. In STOC, 2007.

[13] M. Fredrikson, E. Lantz, S. Jha, S. Lin, D. Page, andT. Ristenpart. Privacy in pharmacogenetics: Anend-to-end case study of personalized warfarin dosing.In USENIX Security Symposium, pages 17–32, 2014.

[14] Fredrikson, M. and Jha, S. and Ristenpart, T. Modelinversion attacks and basic countermeasures( Technical Report). Technical report, 2015.

[15] I. J. Goodfellow, D. Warde-Farley, P. Lamblin,V. Dumoulin, M. Mirza, R. Pascanu, J. Bergstra,F. Bastien, and Y. Bengio. Pylearn2: a machinelearning research library. arXiv preprintarXiv:1308.4214, 2013.

[16] Google. Prediction API.https://cloud.google.com/prediction/.

[17] W. Hickey. FiveThirtyEight.com DataLab: Howamericans like their steak.http://fivethirtyeight.com/datalab/how-

americans-like-their-steak/, May 2014.

[18] N. Homer, S. Szelinger, M. Redman, D. Duggan,W. Tembe, J. Muehling, J. V. Pearson, D. A. Stephan,S. F. Nelson, and D. W. Craig. Resolving individualscontributing trace amounts of dna to highly complexmixtures using high-density snp genotypingmicroarrays. PLOS Genetics, 2008.

[19] G. Huang, H. Lee, and E. Learned-Miller. Learninghierarchical representations for face verification withconvolutional deep belief networks. In ComputerVision and Pattern Recognition (CVPR), June 2012.

[20] International Warfarin Pharmacogenetic Consortium.Estimation of the warfarin dose with clinical andpharmacogenetic data. New England Journal ofMedicine, 360(8):753–764, 2009.

[21] Kairos AR, Inc. Facial recognition API.https://developer.kairos.com/docs.

[22] S. P. Kasiviswanathan, M. Rudelson, and A. Smith.The power of linear reconstruction attacks. In SODA,2013.

[23] S. P. Kasiviswanathan, M. Rudelson, A. Smith, andJ. Ullman. The price of privately releasing contingencytables and the spectra of random matrices withcorrelated rows. In STOC, 2010.

[24] J. Klontz, B. Klare, S. Klum, A. Jain, and M. Burge.Open source biometric recognition. In IEEEInternational Conference on Biometrics: Theory,Applications and Systems, pages 1–8, Sept 2013.

[25] T. Komarova, D. Nekipelov, and E. Yakovlev.Estimation of Treatment Effects from Combined Data:Identification versus Data Security. NBER volumeEconomics of Digitization: An Agenda, To appear.

[26] Lambda Labs. Facial recognition API.https://lambdal.com/face-recognition-api.

[27] H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng.Convolutional deep belief networks for scalableunsupervised learning of hierarchical representations.In Proceedings of the 26th Annual InternationalConference on Machine Learning, ICML ’09, pages609–616, New York, NY, USA, 2009. ACM.

[28] N. Li, W. Qardaji, D. Su, Y. Wu, and W. Yang.Membership privacy: A unifying framework forprivacy definitions. In Proceedings of ACM CCS, 2013.

[29] G. Loukides, J. C. Denny, and B. Malin. Thedisclosure of diagnosis codes can breach researchparticipants’ privacy. Journal of the American MedicalInformatics Association, 17(3):322–327, 2010.

[30] D. Lowd and C. Meek. Adversarial learning. InProceedings of the eleventh ACM SIGKDDinternational conference on Knowledge discovery indata mining, pages 641–647. ACM, 2005.

[31] Microsoft. Microsoft Azure Machine Learning.

[32] A. Narayanan and V. Shmatikov. Robustde-anonymization of large sparse datasets. In IEEESymposium on Security and Privacy, pages 111–125,2008.

[33] J. Prince. Social science research on pornography.http://byuresearch.org/ssrp/downloads/

GSShappiness.pdf.

[34] S. Sankararaman, G. Obozinski, M. I. Jordan, andE. Halperin. Genomic privacy and limits of individualdetection in a pool. Nature Genetics, 41(9):965–967,2009.

[35] C. Savage. Facial scanning is making gains insurveillance. The New York Times, Aug. 21 2013.

[36] SkyBiometry. Facial recognition API. https://www.skybiometry.com/Documentation#faces/recognize.

[37] T. W. Smith, P. Marsden, M. Hout, and J. Kim.General social surveys, 1972-2012. National OpinionResearch Center [producer]; The Roper Center forPublic Opinion Research, University of Connecticut[distributor], 2103.

[38] L. Sweeney. Simple demographics often identify peopleuniquely. 2000.

[39] R. Wang, Y. F. Li, X. Wang, H. Tang, and X. Zhou.Learning your identity and disease from researchpapers: information leaks in genome wide associationstudies. In CCS, 2009.

[40] Wise.io. http://www.wise.io/.