White-box vs Black-box: Bayes Optimal Strategies for Membership Inference

Alexandre Sablayrolles 1 2 Matthijs Douze 2 Yann Ollivier 2 Cordelia Schmid 1 Herve Jegou 2

AbstractMembership inference determines, given a sam-ple and trained parameters of a machine learn-ing model, whether the sample was part of thetraining set. In this paper, we derive the optimalstrategy for membership inference with a few as-sumptions on the distribution of the parameters.We show that optimal attacks only depend on theloss function, and thus black-box attacks are asgood as white-box attacks. As the optimal strat-egy is not tractable, we provide approximations ofit leading to several inference methods, and showthat existing membership inference methods arecoarser approximations of this optimal strategy.Our membership attacks outperform the state ofthe art in various settings, ranging from a simplelogistic regression to more complex architecturesand datasets, such as ResNet-101 and Imagenet.

1. IntroductionAteniese et al. (2015) state that “it is unsafe to release

trained classifiers since valuable information about the

training set can be extracted from them”. The problemthat we address in this paper, i.e., to determine whether asample has been used to train a given model, is related tothe privacy implications of machine learning systems. Theywere first discussed in the context of support vector ma-chines (Rubinstein et al., 2009; Biggio et al., 2014). Theproblem of “unintended memorization” (Carlini et al., 2018)appears in most applications of machine learning, such asnatural language processing systems (Carlini et al., 2018)or image classification (Yeom et al., 2018).

More specifically, we consider the problem of membershipinference, i.e., we aim at determining if a specific image wasused to train a model, given only the (image, label) pair andthe model parameters. This question is important to protect

1University Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK2Facebook AI Research. Correspondence to: Alexandre Sablay-rolles <asablayrolles@fb.com>.

Proceedings of the 36 thInternational Conference on Machine

Learning, Long Beach, California, PMLR 97, 2019. Copyright2019 by the author(s).

both the privacy and intellectual property associated withdata. For neural networks, the privacy issue was recentlyconsidered by Yeom et al. (2018) for the MNIST and CI-FAR datasets. The authors evidence the close relationshipbetween overfitting and privacy of training images. This isreminiscent of prior membership inference attacks, whichemploy the output of the classifier associated with a particu-lar sample to determine whether it was used during trainingor not (Shokri et al., 2017).

At this stage, it is worth defining the different levels ofinformation to which the “attacker”, i.e., the membershipinference system, has access to. We assume that the attackerknows the data distribution and the specifications of themodel (training procedure, architecture of the network, etc),even though they are not necessarily required for all meth-ods. We refer to the white-box setting as the case where theattacker knows all the network parameters. On a side note,the setup commonly adopted in differential privacy (Dworket al., 2006) corresponds to the white-box setting, where theattacker additionally knows all the training samples exceptthe one to be tested.

The black-box setting is when these parameters are unknown.For classification models, the attacker has only access to theoutput for a given input, in one of the following forms:

(i) the classifier decision;(ii) the loss of the correct label;(iii) the full response for all classes.

Prior works on membership inference commonly assume(i) or (iii). Our paper focuses on the black-box case (ii),in which we know the loss incurred by the correct label.The state of the art in this setting are the shadow modelsproposed by Shokri et al. (2017).

In our work, we use a probabilistic framework to derivea formal analysis of the optimal attack. This frameworkencompasses both Bayesian learning, and noisy training,where the noise is injected (Welling & Teh, 2011) or comesfrom the stochasticity of SGD. Under mild assumptions onthe distribution of the parameters, we derive the optimalmembership inference strategy. This strategy only dependson the classifier through evaluation of the loss, thereby show-ing that black-box attacks will perform as well as white-boxattacks in this optimal asymptotic setting. This result mayexplain why, to the best of our knowledge, the literature

White-box vs Black-box: Bayes Optimal Strategies for Membership Inference

does not report white-box attacks outperforming the state-of-the-art black-box-(ii) attacks.

The aforementioned optimal strategy is not tractable,therefore we introduce approximations to derive anexplicit method for membership inference. As abyproduct of this derivation, we show that state-of-the-art approaches (Shokri et al., 2017; Yeom et al., 2018) arecoarser approximations of the optimal strategy. One ofthe approximation drastically simplifies the membershipinference procedure by simply relying on the loss and a cal-ibration term. We employ this strategy to the more complexcase of neural networks, and show that it outperforms allapproaches we are aware of.

In summary, our main contributions are as follows:

• We show, under a few assumptions on training, that theoptimal inference only depends on the loss function,and not on the parameters of the classifier. In otherterms, white-box attacks don’t provide any additionalinformation and result in the same optimal strategy.

• We employ different approximations to derive threeexplicit membership attack strategies. We show thatstate-of-the-art methods constitute other approxima-tions. Simple simulations show the superiority of ourapproach on a simple regression problem.

• We apply a simplified, tractable, strategy to infer themembership of images to the train set in the case ofthe public image classification benchmarks CIFAR andImagenet. It outperforms the state of the art for mem-bership inference, namely the shadow models.

The paper is organized as follows. Section 2 reviews relatedwork. Section 3 introduces our probabilistic formulationand derives our main theoretical result. This section alsodiscusses the connection between membership inferenceand differential privacy. Section 4 considers approximationsfor practical use-cases, which allow us to derive inferencestrategies in closed-form, some of which are connected withexisting methods from the literature. Section 5 summarizesthe practical algorithms derived from our analysis. Finally,Section 6 considers the more difficult case of membershipinference for real-life neural networks and datasets.

2. Related workLeakage of information from the training set. Ourwork is related to the topics of overfitting and memorizationcapabilities of classifiers. Determining what neural net-works actually memorize from their training set is not trivial.A few recent works (Zhang et al., 2017; Yeom et al., 2018)evaluate how a network can fit random labels. Zhang et al.(2017) replace true labels by random labels and show that

popular neural nets can perfectly fit them in simple cases,such as small datasets (CIFAR10) or Imagenet without dataaugmentation. Krueger et al. (2017) extend their analysisand argue in particular that the effective capacity of neuralnets depends on the dataset considered. In a privacy con-text, Yeom et al. (2018) exploit this memorizing propertyto watermark networks. As a side note, random labelingand data augmentation have been used for the purpose oftraining a network without any annotated data (Dosovitskiyet al., 2014; Bojanowski & Joulin, 2017).

In the context of differential privacy (Dwork et al., 2006),recent works (Wang et al., 2016; Bassily et al., 2016) sug-gest that guaranteeing privacy requires learning systems togeneralize well, i.e., to not overfit. Wang et al. (2015) showthat Bayesian posterior sampling offers differential privacyguarantees. Abadi et al. (2016) introduce noisy SGD tolearn deep models with differential privacy.

Membership Inference. A few recent works (Hayeset al., 2017; Shokri et al., 2017; Long et al., 2018) have ad-dressed membership inference. Yeom et al. (2018) proposea series of membership attacks and derive their performance.Long et al. (2018) observe that some training images aremore vulnerable than others and propose a strategy to iden-tify them. Hayes et al. (2017) analyze privacy issues arisingin generative models. Dwork et al. (2015) and Sankarara-man et al. (2009) provide optimal strategies for membershipinference in genomics data.

Shadow models. Shadow models were introduced byShokri et al. (2017) in the context of black-box attacks.In this setup, an attacker has black-box-(iii) access (full re-sponse for all classes) to a model trained on a private dataset,and to a public dataset that follows the same distribution asthe private dataset. The attacker wishes to perform member-ship inference using black-box outputs of the private model.For this, the attacker simulates models by training shadow

models on known splits from the public set. On this simu-lated models, the attacker can analyze the output patternscorresponding to samples from the training set and from aheld-out set. Shokri et al. (2017) propose to train an attackmodel that learns to predict, given an output pattern, whetherit corresponds to a training or held-out sample. If the attackmodel simply predicts “training” when the output activa-tions fire on the correct class, this strategy is equivalent toYeom et al. (2018)’s adversary. Salem et al. (2019) furthershow that shadow models work under weaker assumptionsthan those of Shokri et al. (2017).

3. Membership inference modelIn this section, we derive the Bayes optimal performancefor membership inference (Theorem 1). We then makethe connection with differential privacy and propose looserguarantees that prevent membership inference.

White-box vs Black-box: Bayes Optimal Strategies for Membership Inference

3.1. Posterior distribution of parameters

Let D be a data distribution, from which we sample n 2 Npoints z1, z2, . . . , zn ⇠

i.i.d.D. A machine learning algorithm

produces parameters ✓ that incur a low lossP

n

i=1 `(✓, zi).Typically in the case of classification, z = (x, y) where x isan input sample and y a class label, and the loss function `

is high on samples (x, y0) for which y0 6= y.

We assume that the machine learning algorithm has somerandomness, and we model it with a posterior distributionover parameters ✓|z1, . . . , zn. The randomness in ✓ eithercomes from the training procedure (e.g., Bayesian posteriorsampling), or arises naturally, as is the case with StochasticGradient methods.

In general, we assume that the posterior distribution follows:

P(✓ | z1, . . . , zn) / e� 1

T

Pni=1 `(✓,zi), (1)

where T is a temperature parameter, intuitively controllingthe stochasticity of ✓. T = 1 corresponds to the case of theBayesian posterior, T ! 0 the case of MAP (Maximum APosteriori) inference and a small T to the case of averagedSGD (Polyak & Juditsky, 1992). Note that we do not includea prior on ✓: we assume that the prior is either uniform on abounded ✓, or that it has been factored in the loss term `.

3.2. Membership inference

Given ✓ produced by such a machine learning algorithm,membership inference asks the following question: Whatinformation does ✓ contain about its training set z1, . . . , zn?

Formally, we assume that binary membership variablesm1,m2, . . . ,mn are drawn independently, with probability� = P(mi = 1). The samples for which mi = 0 are the testset, while the samples for which mi = 1 are the training set.Equation (1) becomes

P(✓ | z1, . . . , zn,m1, . . . ,mn) / e� 1

T

Pni=1 mi`(✓,zi),

(2)

Taking the case of z1 without loss of generality, membershipinference determines, given parameters ✓ and sample z1,whether m1 = 1 or m1 = 0.Definition 1 (Membership inference). Inferring the mem-

bership of sample z1 to the training set amounts to comput-

ing:

M(✓, z1) := P(m1 = 1 | ✓, z1). (3)

Notation. We denote by � the sigmoid function �(u) =(1 + e

�u)�1. We collect the knowledge about theother samples and their memberships into the set T ={z2, ..., zn,m2, ...,mn}.

3.3. Optimal membership inference

In Theorem 1, we derive the explicit formula for M(✓, z1).Theorem 1. Given a parameter ✓ and a sample z1, the

optimal membership inference is given by:

M(✓, z1) = ET

�

✓log

✓P(✓ | m1 = 1, z1, T )P(✓ | m1 = 0, z1, T )

◆+ t�

◆�,

(4)

with t� = log⇣

�

1��

⌘.

Proof. By the law of total expectation, we have:

M(✓, z1) = P(m1 = 1 | ✓, z1) (5)= ET [P (m1 = 1 | ✓, z1, T )] . (6)

Applying Bayes’ formula:

P (m1 = 1 | ✓, z1, T ) =P (✓ | m1 = 1, z1, T )P(m1 = 1)

P (✓ | z1, T ),

(7)

=↵

↵+ �= �

✓log

✓↵

�

◆◆(8)

where:

↵ := P (✓ | m1 = 1, z1, T )P(m1 = 1) (9)� := P (✓ | m1 = 0, z1, T )P(m1 = 0) (10)

Given that P(m1 = 1) = �,

log

✓↵

�

◆= log

✓P(✓ | m1 = 1, z1, T )

P(✓ | m1 = 0, z1, T )

◆+ log

✓�

1� �

◆,

(11)

which gives the expression for M(✓, z1).

Note that Theorem 1 only relies on the fact that ✓ given{z1, ..., zn,m1, ...,mn} is a random variable, but it doesnot make any assumption on the form of the distribution. Inparticular the loss ` does not appear in the expression.

Theorem 2 uses the assumption in Equation (2) to furtherexplicit M(✓, z1); we give its formal expression below,prove it, and analyze the expression qualitatively. Let usfirst define the posterior over the parameters given samplesz2, . . . , zn and memberships m2, . . . ,mn:

pT (✓) :=e� 1

T

Pni=2 mi`(✓,zi)

Rte� 1

T

Pni=2 mi`(t,zi)dt

. (12)

Theorem 2. Given a parameter ✓ and a sample z1, the

optimal membership inference is given by:

M(✓, z1) = ET [� (s(z1, ✓, pT ) + t�)] (13)

White-box vs Black-box: Bayes Optimal Strategies for Membership Inference

where we define the following score:

⌧p(z1) := �T log

✓Z

t

e� 1

T `(t,z1)p(t)dt

◆(14)

s(z1, ✓, p) :=1

T(⌧p(z1)� `(✓, z1)) . (15)

Proof. Singling out m1 in Equation (2) yields the followingexpressions for ↵ and �:

↵ = �e� 1

T `(✓,z1)e�1T

Pni=2 mi`(✓,zi)

Rte� 1

T `(t,z1)e�1T

Pni=2 mi`(t,zi)dt

(16)

= �e� 1

T `(✓,z1)pT (✓)Rte� 1

T `(t,z1)pT (t)dt, (17)

and

� = (1� �)e� 1

T

Pni=2 mi`(✓,zi)

Rte� 1

T

Pni=2 mi`(t,zi)dt

= (1� �)pT (✓).

(18)

Thus,

log

✓↵

�

◆= �`(✓, z1)

T� log

✓Z

t

e� 1

T `(t,z1))pT (t)dt

◆+ t�

= s(z1, ✓, pT ) + t�. (19)

Then, Equation (8) yields the expected result.

The first observation in Theorem 2 is that M(✓, z1) doesnot depend on the parameters ✓ beyond the evaluation ofthe loss `(✓, ·): this strategy does not require, for instance,internal parameters of the model that a white-box attackcould provide. This means that if we can compute ⌧p orapproximate it well enough, then the optimal membershipinference depends only on the loss. In other terms, asymptot-ically, the white-box setting does not provide any benefitcompared to black-box membership inference.

Let us analyze qualitatively the terms in the expression:Since T is a training set, pT corresponds to a posterior overthis training set, i.e., a typical distribution of trained param-eters. ⌧p(z1) is the softmin of loss terms `(·, z1) over thesetypical models, and corresponds therefore to the typical loss

of sample z1 under models that have not seen z1.

The quantity ⌧p(z1) can be seen as a threshold, to whichthe loss `(✓, z1) is compared. Around this threshold, when`(✓, z1) ⇡ ⌧p(z1), then s ⇡ 0: since �(t�) = �, the mem-bership posterior probability M(✓, z1) is equal to �, andthus we have no information on m1 beyond prior knowledge.As the loss `(·, z1) gets lower than this threshold, s becomespositive. Since � is non decreasing, when s(z1, ✓, pT ) > 0,M(✓, z1) > � and thus we gain non-trivial membershipinformation on z1.

Another consequence of Theorem 2 is that a higher temper-ature T decreases s, and thus decreases P(m1 = 1 | ✓, z1):it corresponds to the intuition that more randomness in ✓

protects the privacy of training data.

3.4. Differential privacy and guarantees

In this subsection we make the link with differential privacy.

Differential privacy (Dwork et al., 2006) is a framework thatallows to learn model parameters ✓ while maintaining theconfidentiality of data. It ensures that even if a maliciousattacker knows parameters ✓ and samples zi, i � 2, forwhich mi = 1, the privacy of z1 is not compromised.Definition 2 (✏-differential privacy). A machine learning

algorithm is ✏-differentially private if, for any choice of z1

and T ,

log

✓P(✓ | m1 = 1, z1, T )

P(✓ | m1 = 0, z1, T )

◆< ✏. (20)

Note that this definition is slightly different from the one ofDwork et al. (2006) in that we consider the removal of z1rather than its substitution with z

0. Additionally we considerprobability densities instead of probabilities of sets, withoutloss of generality.Property 1 (✏-differential privacy). If the training is ✏-

differentially private, then:

P(m1 = 1 | ✓, z1) �+✏

4. (21)

Proof. Combining Equation (20) and the fact that �(u) �(v) + max(u� v, 0)/4 (Appendix A.3), we have:

�

✓log

✓P(✓ | m1 = 1, z1, T )

P(✓ | m1 = 0, z1, T )

◆+ t�

◆ �(t�) +

✏

4

= �+✏

4. (22)

Combining this expression with Theorem 1 yields the result.

Note that this bound gives a tangible sense of ✏. In general,decreasing ✏ increases privacy, but there is no consensusover “good” values of ✏; this bound indicates for instancethat ✏ = 0.01 would be sufficient for membership privacy.

✏-differential privacy gives strong membership inferenceguarantees, at the expense of a constrained training proce-dure resulting generally in a loss of accuracy (Abadi et al.,2016). However, if we assume that the attacker knows thezi, i � 2 for which mi = 1, ✏-differential privacy is requiredto protect the privacy of z1. Depending on the informationwe have on zi, i � 2, there is a continuum between differen-tial privacy (all zi’s are known) and membership inference(only prior knowledge on zi). In the case of membershipinference, it suffices to have the following guarantee:

White-box vs Black-box: Bayes Optimal Strategies for Membership Inference

Definition 3 ((✏, �) membership privacy). The training is

(✏, �)-membership private for some ✏ > 0, � > 0 if with

probability 1� � over the choice of T :

Z

t

`(t, z1)pT (t)dt� `(✓, z1) ✏. (23)

Property 2. If the training is (✏, �)-membership private,

then:

P(m1 = 1 | ✓, z1) �+✏

4T+ �. (24)

Proof. Jensen’s inequality states that for any distribution p

and any function f :Z

t

f(t)p(t)dt log

✓Z

t

ef(t)

p(t)dt

◆, (25)

hence the score s from Equation (15) verifies:

s(z1, ✓, p) 1

T

✓Z

t

`(t, z1)p(t)dt� `(✓, z1)

◆. (26)

Thus, distinguishing the cases � and 1� � in the expectationin Equation (13),

P(m1 = 1 | ✓, z1) � + (1� �)⇣�+

✏

4T

⌘(27)

�+✏

4T+ �, (28)

which gives the desired bound.

Membership privacy provides a post-hoc guarantee on ✓, z1.Guarantees in the form of Equation (23) can be obtained byPAC (Probably Approximately Correct) bounds.

4. Approximations for membership inferenceEstimating the probability of Equation (15) mainly requiresto compute the term ⌧p. Since its expression is intractable,we use approximations to derive concrete membership at-tacks (MA). We now detail these approximations, referredto as MAST (MA Sample Threshold), MALT (MA LossThreshold) and MATT (MA Taylor Threshold).

4.1. MAST: Approximation of ⌧(z1)

We first make the mean-field assumption that pT (t) doesnot depend on T (we note it p), and define

⌧(z1) := log

✓Z

t

e� 1

T `(t,z1)p(t)dt

◆. (29)

The quantity ⌧(·) is a “calibrating” term that reflects thedifficulty of a sample. Intuitively, a low ⌧(z1) means thatthe sample z1 is easy to predict, and thus a low value of`(✓, z1) does not necessarily indicate that z1 belongs to thetrain set. Thus, Theorem 2 gives the optimal attack model:

sMAST(✓, z1) = �`(✓, z1) + ⌧(z1). (30)

4.2. MALT: Constant ⌧

If we further assume that ⌧(·) is constant, the optimal strat-egy reduces to predicting that z1 comes from the training setif its loss `(✓, z1) is lower than this threshold ⌧ , and fromthe test set otherwise:

sMALT(✓, z1) = �`(✓, z1) + ⌧. (31)

A similar strategy is proposed by Yeom et al. (2018) forGaussian models. Carlini et al. (2018) estimate a secrettoken in text datasets by comparing probabilities of the sen-tence “My SSN is X” with various values of X. Surprisingly,to the best of our knowledge, it has not been proposed inthe literature to estimate the threshold ⌧ on public data, andto apply it for membership inference. As we show in Sec-tion 6, this simple strategy yields better results than shadowmodels. Their attack models take as input the softmax acti-vation �✓(x) and the class label y, and predict whether thesample comes from the training or test set. For classificationmodels, `(✓, (x, y)) = � log(�✓(x)y). Hence the optimalattack performs:

sMALT(✓, (x, y)) = log(�✓(x)y) + ⌧. (32)

In Shokri et al. (2017), we argue that the attack modelessentially performs such an estimation, albeit in a non-explicit way. In particular, we believe that the gap betweenShokri et al. (2017)’s method and ours is due to instabilitiesin the estimation of ⌧ and the numerical computation of thelog, as the model is given only �✓(x). As a side note, theexpectation term in T = z2, . . . , zn,m2, . . . ,mn is verysimilar in spirit to the shadow models, and they can beviewed as a Monte-Carlo estimation of this quantity.An experiment with Gaussian data. We illustrate thedifference between a MALT (global ⌧ ) and MAST (per-sample ⌧(·)) on a simple toy example. Let’s assume weestimate the mean µ of Gaussian data with unit variance.

We sample n values z1, . . . , zn from D = N (µ, I). Theestimate of the mean is ✓ = 1

n0

Pn

i=1 mizi where n0 =

|{i |mi = 1}|. We have (see Appendix A.2 for derivations):

`(✓, zi) :=1

2kzi � ✓k2 (33)

⌧(zi) =n0

2(n0 + 1)kzi � µk2 (34)

⌧ =n0

2(n0 + 1)Ekz � µk2 =

n0

2(n0 + 1)d. (35)

The expression of ⌧(zi) shows that the “difficulty” of samplezi is its distance to µ, i.e., how untypical this sample is.

Figure 1 shows the results with a global ⌧ or a per-sample⌧ : the per-sample ⌧ better separates the two distributions,leading to an increased membership inference accuracy.

White-box vs Black-box: Bayes Optimal Strategies for Membership Inference

���� ���� ���� ��� � �� ��� ��� ���

�������

���� �����

��������������

��� ��� � �� ��

�������

���� �����

��������������

Figure 1. Comparison of MALT and MALT for membership inference on the mean estimator for Gaussian data (n = 100 samples in2000 dimensions). Distribution of scores s used to distinguish between samples seen or not at training. MALT: a single threshold is usedfor all the dataset; MAST: each sample gets assigned a different threshold. MAST better separates training and non-training samples.

MATT: Estimation with Taylor expansion We as-sume that the posterior induced by the loss L(✓) =P

n

i=1 mi`(✓, zi) is a Gaussian centered on ✓⇤, the mini-

mum of the loss, with covariance C. This corresponds tothe Laplace approximation of the posterior distribution. Theinverse covariance matrix C

�1 is asymptotically n timesthe Fisher matrix (van der Vaart, 1998), which itself is theHessian of the loss (Kullback, 1997):

P(✓ | z1, T ) =1p

det (2⇡H�1)e� 1

2 (✓�✓⇤)TH(✓�✓

⇤).

(36)

We denote by ✓⇤0 (resp. ✓⇤1) the mode of the Gaussian corre-

sponding to {z2, ..., zn} (resp. {z1, ..., zn}), and H0 (resp.H1) the corresponding Hessian matrix. We assume that His not impacted by removing z1 from the training set, andthus H := H0 ⇡ H1 (cf. appendix A.4 for a more precisejustification). The log-ratio is therefore

log

✓P(✓ | m1 = 1, z1, T )P(✓ | m1 = 0, z1, T )

◆(37)

= �12(✓ � ✓

⇤1)

TH(✓ � ✓

⇤1) +

12(✓ � ✓

⇤0)

TH(✓ � ✓

⇤0)

= (✓ � ✓⇤0)

TH(✓⇤1 � ✓

⇤0)�

12(✓⇤1 � ✓

⇤0)

TH(✓⇤1 � ✓

⇤0).

The difference ✓⇤1 � ✓

⇤0 can be estimated using a Taylor

expansion of the loss gradient around ✓⇤0 (see e.g. Koh &

Liang (2017)):

✓⇤1 � ✓

⇤0 ⇡ �H

�1r✓`(✓⇤0 , z1) (38)

Combining this with Equation (37) leads to

�(✓ � ✓⇤0)

Tr✓`(✓⇤0 , z1)�

12r✓`(✓

⇤0 , z1)

TH

�1r✓`(✓⇤0 , z1).

(39)

We study the asymptotic behavior of this expression whenn ! 1. On the left-hand side, the parameters ✓ and ✓

⇤0 are

estimates of the optimal ✓⇤, and under mild conditions, the

error of the estimated parameters is of order 1/pn. There-

fore the difference ✓ � ✓⇤0 is of order 1/

pn. On the right-

hand side, the matrix H is the summation of n sample-wiseHessian matrices. Therefore, asymptotically, the right-handside shrinks at a rate 1/n, which is negligible comparedto the other, which shrinks at 1/

pn. In addition to the

asymptotic reasoning, we verified this approximation exper-imentally. Thus, we approximate Equation (39) to give thefollowing score:

sMATT(✓, z1) = �(✓ � ✓⇤0)

Tr✓`(✓⇤0 , z1). (40)

Equation (40) has an intuitive interpretation: parameters ✓were trained using z1 if their difference with a set of parame-ters trained without z1 (i.e. ✓⇤0) is aligned with the directionof the update �r✓`(✓⇤0 , z1).

5. Membership inference algorithmsIn this section, we detail how the approximations ofs(✓, z1, p) are employed to perform membership inference.

We assume that a machine learning model has been trained,yielding parameters ✓. We assume also that similar modelscan be re-trained with different training sets. Given a samplez1, we want to decide whether z1 belongs to the training set.

5.1. The 0-1 baseline

We consider as a baseline the “0-1” heuristic, which predictsthat z1 comes from the training set if the class is predictedcorrectly, and from the test set if the class is predicted incor-rectly. We note ptrain (resp. ptest) the classification accuracyon the training (resp. held-out) set. The accuracy of theheuristic is (see Appendix A.1 for derivations):

pbayes = �ptrain + (1� �)(1� ptest). (41)

For example when � = 1/2, since ptrain � ptest this heuristic

White-box vs Black-box: Bayes Optimal Strategies for Membership Inference

is better than random guessing (accuracy 1/2) and the im-provement is proportional to the overfitting gap ptrain � ptest.

5.2. Making hard decisions from scores

Variants of our method provide different estimates ofs(✓, z1, p). Theorem 2 shows that this score has to be passedthrough a sigmoid function, but since it is an increasing func-tion, the threshold can be chosen directly on these scores.Estimation of this threshold has to be conducted on simu-lated sets, for which membership information is known. Weobserved that there is almost no difference between chosingthe threshold on the set to be tested and cross-validatingit. This is expected, as a one-dimensional parameter thethreshold is not prone to overfitting.

5.3. Membership algorithms

MALT: Threshold on the loss. Since ⌧ in Equation (31) isconstant, and using the invariance to increasing functions,we need only to use loss value for the sample, `(✓, z1).

MAST: Estimating ⌧(z1). To estimate ⌧(z1) in Equa-tion (29), we train several models with different subsamplesof the training set. This yields a set of per-sample losses forz1 that are averaged into an estimate of ⌧(z1).

MATT: the Taylor approximation. We run the trainingon a separate set to obtain ✓

⇤0 . Then we take a gradient step

over the loss to estimate the approximation in Equation (40).Note that this strategy is not compatible with neural net-works because the assumption that parameters lie arounda unique global minimum does not hold. In addition, pa-rameters from two different networks ✓ and ✓

⇤0 cannot be

compared directly as neural networks that express the samefunction can have very different parameters (e.g. becausechannels can be permuted arbitrarily).

6. ExperimentsIn this section we evaluate the membership inference meth-ods on machine-learning tasks of increasing complexity.

6.1. Evaluation

We evaluate three metrics: the accuracy of the attack, andthe mean average precision when detecting either from train(mAPtrain) or test (mAPtest) images. For the mean averageprecision, the scores need not to be thresholded, the metricis invariant to mapping by any increasing function.

6.2. Logistic regression

CIFAR-10 is a dataset of 32⇥ 32 pixel images grouped in10 classes. In this subsection, we consider two of the classes(truck and boat) and vary the number of training images

Table 1. Accuracy (top) and mAP (bottom) of membership infer-ence on the 2-class logistic regression with simple CNN features,for different types of attacks. Note that 0-1 corresponds to thebaseline (Yeom et al., 2018). We do not report Shokri et al. (2017)since Table 2 shows MALT performs better. Results are averagedover 100 different random seeds.

Model accuracy Attack accuracyn train validation 0� 1 MALT MATT

400 97.9 93.8 52.1 54.4 57.01000 97.3 94.5 51.4 52.6 54.52000 96.8 95.2 50.8 51.7 53.04000 97.7 95.6 51.0 51.4 52.16000 97.5 96.0 50.7 51.0 51.8

mAPtest mAPtrain

n MALT MATT MALT MATT

400 55.8 60.1 51.9 57.11000 53.2 56.6 50.5 54.82000 51.8 54.4 50.4 53.44000 51.9 53.7 50.1 52.66000 51.4 53.0 50.2 52.2

from n = 400 to 6, 000.

We train a logistic regression to separate the two classes.The logistic regression takes as input features extracted froma pretrained Resnet18 on CIFAR-100 (a disjoint dataset).The regularization parameter C of the logistic regression iscross-validated on held-out data.

We assume that � = 1/2 (n/2 training images and n/2 testimages). We also reserve n/2 images to estimate ✓

⇤0 for the

MATT method. In both experiments, we report the peakaccuracy obtained for the best threshold (cf. Section 5.2).

Table 1 shows the results of our experiments, in terms ofaccuracy and mean average precision. In accuracy, theTaylor expansion method MATT outperforms the MALTmethod, for any number of training instances n, which itselfobtains much better results than the naive 0-1 attack.

Interestingly, it shows a difference between MALT andMATT: both perform similarly in terms of mAPtest, butMATT slightly outperforms MALT in mAPtrain. The mainreason for this difference is that the MALT attack is asym-metric: it is relatively easy to predict that elements comefrom the test set, as they have a high loss, but elements witha low loss can come either from the train set or the test set.

White-box vs Black-box: Bayes Optimal Strategies for Membership Inference

Table 2. Accuracy of membership attacks on the CIFAR-10 classi-fication with a simple neural network. The numbers for the relatedworks are from the respective papers.

Method Accuracy

0-1 (Yeom et al., 2018) 69.4Shadow models (Shokri et al., 2017) 73.9MALT 77.1MAST 77.6

6.3. Small convolutional network

In this section we train a small convolutional network1 withthe same architecture as Shokri et al. (2017) on the CIFAR-10 dataset, using a training set of 15, 000 images. Our modelis trained for 50 epochs with a learning rate of 0.001. Weassume a balanced prior on membership (� = 1/2).

We run the MALT and MAST attacks on the classifiers.As stated before, the MATT attack cannot be carried outon convolutional networks. For MAST, the threshold isestimated from 30 shadow models: we train these 30 shadowmodels on 15, 000 images chosen among the train+held-outset (30, 000 images). Thus, for each image, we have onaverage 15 models trained on it and 15 models not trainedon it: we estimate the threshold for this image by taking thevalue ⌧(z) that separates the best the two distributions: thiscorresponds to a non-parametric estimation of ⌧(z).

Table 2 shows that our estimations outperform the relatedworks. Note that this setup gives a slight advantage toMAST as the threshold is estimated directly for each sampleunder investigation, whereas MALT first estimates a thresh-old, and then applies it to never-seen data. Yet, in contrastwith the experiment on Gaussian data, MAST performs onlyslightly better than MALT. Our interpretation for this is thatthe images in the training set have a high variability, so itis difficult to obtain a good estimate of ⌧(z1). Furthermore,our analysis of the estimated thresholds ⌧(z1) show thatthey are very concentrated around a central value ⌧ , so theirimpact when added to the scores is limited.

Therefore, in the following experiment we focus on theMALT attack.

6.4. Evaluation on Imagenet

We evaluate a real-world dataset and tackle classificationwith large neural networks on the Imagenet dataset (Denget al., 2009; Russakovsky et al., 2015), which contains 1.2million images partitioned into 1000 categories. We divideImagenet equally into two splits, use one for training andhold out the rest of the data.

1 2 convolutional and 2 linear layers with Tanh non-linearity.

Table 3. Imagenet classification with deep convolutional networks:Accuracy of membership inference attacks of the models.

Model Augmentation 0-1 MALT

Resnet101 None 76.3 90.4Flip, Crop ±5 69.5 77.4Flip, Crop 65.4 68.0

VGG16 None 77.4 90.8Flip, Crop ±5 71.3 79.5Flip, Crop 63.8 64.3

We experiment with the popular VGG-16 (Simonyan &Zisserman, 2014) and Resnet-101 (He et al., 2016) archi-tectures. The model is learned in 90 epochs, with an initiallearning rate of 0.01, divided by 10 every 30 epochs. Param-eter optimization is conducted with SGD with a momentumof 0.9, a weight decay of 10�4, and a batch size of 256.

We conduct the membership inference test by running the0-1 attack and MALT. An important factor for the success ofthe attacks is the amount of data augmentation. To assess theeffect of data augmentation, we train different networks withvarying data augmentation: None, Flip+Crop±5, Flip+Crop(by increasing intensity).

Table 3 shows that data augmentation reduces the gap be-tween the training and the held-out accuracy. This decreasesthe accuracy of the Bayes attack and the MALT attack. Aswe can see, without data augmentation, it is possible toguess with high accuracy if a given image was used to traina model (about 90% with our approach, against 77% forexisting approaches). Stronger data augmentation reducesthe accuracy of the attacks, that still remain above 64%.

7. ConclusionThis paper has addressed the problem of membership infer-ence by adopting a probabilistic point of view. This led usto derive the optimal inference strategy. This strategy, whilenot explicit and therefore not applicable in practice, doesnot depend on the parameters of the classifier if we have ac-cess to the loss. Therefore, a main conclusion of this paperis to show that, asymptotically, white-box inference doesnot provide more information than an optimized black-boxsetting.

We then proposed two approximations that lead to threeconcrete strategies. They outperform competitive strategiesfor a simple logistic problem, by a large margin for ourmost sophisticated approach (MATT). Our simplest strat-egy (MALT) is applied to the more complex problem ofmembership inference from a deep convolutional networkon Imagenet, and significantly outperforms the baseline.

White-box vs Black-box: Bayes Optimal Strategies for Membership Inference

ReferencesMartin Abadi, Andy Chu, Ian Goodfellow, Brendan McMa-

han, Ilya Mironov, Kunal Talwar, and Li Zhang. Deeplearning with differential privacy. In CCS, 2016.

Giuseppe Ateniese, Luigi V Mancini, Angelo Spognardi,Antonio Villani, Domenico Vitali, and Giovanni Felici.Hacking smart machines with smarter ones: How to ex-tract meaningful data from machine learning classifiers.IJSN, 2015.

Raef Bassily, Kobbi Nissim, Adam Smith, Thomas Steinke,Uri Stemmer, and Jonathan Ullman. Algorithmic stabilityfor adaptive data analysis. In STOC, 2016.

Battista Biggio, Igino Corona, Blaine Nelson, Benjamin I. P.Rubinstein, Davide Maiorca, Giorgio Fumera, GiorgioGiacinto, and Fabio Roli. Security Evaluation of Support

Vector Machines in Adversarial Environments. SpringerInternational Publishing, 2014.

Piotr Bojanowski and Armand Joulin. Unsupervised learn-ing by predicting noise. In ICML, 2017.

Nicholas Carlini, Chang Liu, Jernej Kos, Ulfar Erlingsson,and Dawn Song. The secret sharer: Measuring unintendedneural network memorization & extracting secrets. arXiv

preprint arXiv:1802.08232, 2018.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, andLi Fei-Fei. Imagenet: A large-scale hierarchical imagedatabase. In CVPR, 2009.

Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Ried-miller, and Thomas Brox. Discriminative unsupervisedfeature learning with convolutional neural networks. InNIPS, 2014.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and AdamSmith. Calibrating noise to sensitivity in private dataanalysis. In TCC, 2006.

Cynthia Dwork, Adam Smith, Thomas Steinke, JonathanUllman, and Salil Vadhan. Robust traceability from traceamounts. In Proceedings of the Symposium on the Foun-

dations of Computer Science, 2015.

Jamie Hayes, Luca Melis, George Danezis, and EmilianoDe Cristofaro. Logan: evaluating privacy leakage ofgenerative models using generative adversarial networks.arXiv preprint arXiv:1705.07663, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.Deep residual learning for image recognition. In CVPR,2016.

Pang Wei Koh and Percy Liang. Understanding black-boxpredictions via influence functions. In ICML, 2017.

David Krueger, Nicolas Ballas, Stanislaw Jastrzebski, De-vansh Arpit, Maxinder S. Kanwal, Tegan Maharaj, Em-manuel Bengio, Asja Fischer, Aaron Courville, SimonLacoste-Julien, and Yoshua Bengio. A closer look atmemorization in deep networks. In ICML, 2017.

S. Kullback. Information Theory And Statistics. DoverPublications, 1997.

Yunhui Long, Vincent Bindschaedler, Lei Wang, DiyueBu, Xiaofeng Wang, Haixu Tang, Carl A Gunter,and Kai Chen. Understanding membership inferenceson well-generalized learning models. arXiv preprint

arXiv:1802.04889, 2018.

B. T. Polyak and A. B. Juditsky. Acceleration of stochasticapproximation by averaging. SIAM J. Control Optim.,1992.

Benjamin IP Rubinstein, Peter L Bartlett, Ling Huang,and Nina Taft. Learning in a large function space:Privacy-preserving mechanisms for SVM learning.arXiv:0911.5708, 2009.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause,Sanjeev Satheesh, Sean Ma, Zhiheng Huang, AndrejKarpathy, Aditya Khosla, Michael Bernstein, Alexan-der C. Berg, and Li Fei-Fei. ImageNet Large Scale VisualRecognition Challenge. IJCV, 2015.

Ahmed Salem, Yang Zhang, Mathias Humbert, PascalBerrang, Mario Fritz, and Michael Backes. Ml-leaks:Model and data independent membership inference at-tacks and defenses on machine learning models. In NCSS,2019.

Sriram Sankararaman, Guillaume Obozinski, Michael I. Jor-dan, and Eran Halperin. Genomic privacy and limits ofindividual detection in a pool. Nature Genetics, 2009.

Reza Shokri, Marco Stronati, and Vitaly Shmatikov. Mem-bership inference attacks against machine learning mod-els. IEEE Symp. Security and Privacy, 2017.

Karen Simonyan and Andrew Zisserman. Very deep convo-lutional networks for large-scale image recognition. InICLR, 2014.

A. W. van der Vaart. Asymptotic statistics. Cambridge Seriesin Statistical and Probabilistic Mathematics. CambridgeUniversity Press, 1998.

Yu-Xiang Wang, Stephen Fienberg, and Alex Smola. Pri-vacy for free: Posterior sampling and stochastic gradientmonte carlo. In ICML, 2015.

Yu-Xiang Wang, Jing Lei, and Stephen E Fienberg. On-average KL-privacy and its equivalence to generalizationfor max-entropy mechanisms. In PSD. Springer, 2016.

White-box vs Black-box: Bayes Optimal Strategies for Membership Inference

Max Welling and Yee Whye Teh. Bayesian learning viastochastic gradient langevin dynamics. In ICML, 2011.

Samuel Yeom, Irene Giacomelli, Matt Fredrikson, andSomesh Jha. Privacy risk in machine learning: Analyzingthe connection to overfitting. In CSF, 2018.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, BenjaminRecht, and Oriol Vinyals. Understanding deep learningrequires rethinking generalization. In ICLR, 2017.