A Probabilistic Approach for Learning with LabelProportions Applied to the US Presidential Election

Tao Sun1, Dan Sheldon1,2, Brendan O’Connor11College of Information and Computer Sciences, University of Massachusetts Amherst

2Department of Computer Science, Mount Holyoke CollegeEmail: {taosun, sheldon, brenocon}@cs.umass.edu

Abstract—Ecological inference (EI) is a classical problem frompolitical science to model voting behavior of individuals given onlyaggregate election results. Flaxman et al. recently formulatedEI as machine learning problem using distribution regression,and applied it to analyze US presidential elections. However,distribution regression unnecessarily aggregates individual-levelcovariates available from census microdata, and ignores knownstructure of the aggregation mechanism. We instead formulatethe problem as learning with label proportions (LLP), and developa new, probabilistic, LLP method to solve it. Our model is thestraightforward one where individual votes are latent variables.We use cardinality potentials to efficiently perform exact inferenceover latent variables during learning, and introduce a novelmessage-passing algorithm to extend cardinality potentials tomultivariate probability models for use within multiclass LLPproblems. We show experimentally that LLP outperforms distri-bution regression for predicting individual-level attributes, andthat our method is as good as or better than existing state-of-the-art LLP methods.

I. INTRODUCTION

Ecological inference (EI) is the problem of making inferencesabout individuals from aggregate data [13]. EI originates in po-litical science, where its history is closely intertwined with thespecific application of inferring voting behavior of individualsor demographic groups from vote totals for different regions.EI maps onto a growing number of problem settings withinmachine learning—including distribution regression [7], [27],optimal transport [17], learning with label proportions [14],[18], [29], multiple-instance learning [4], [10], and collectivegraphical models [23], [24], [26]—where one wishes to performsupervised learning, but supervision is only available at anaggregate level, e.g., as summary statistics for “bags” ofinstances.

We consider the classical EI problem of analyzing votingbehavior, motivated in particular by US presidential elections.Although there has been vigorous historical debate aboutthe inherent limitations of EI [8], [12], [13], [22], work inmachine learning makes it clear that, if one is careful to stateassumptions and goals clearly, it is indeed possible to learnindividual-level models from aggregate data [2], [18], [27], atleast asymptotically. One must still be careful to map theseresults back to the application at hand; for example, a typicalresult may be that it is possible to consistently estimate theparameters of an individual-level model given enough group-level observations. This would not imply the ability to inferthe outcome or behavior of a single individual.

We will focus on the EI voting problem formulated in [6], [7],where a collection of individual-level demographic covariates isavailable for each geographical region in addition to the region-level voting totals. In the US, individual-level demographicdata for different regions is readily available from the CensusBureau [1]. In [6], [7], the EI problem is then formulatedas distribution regression, where a function is learned tomap directly from the distribution of covariates within eachregion to voting proportions. This is accomplished by creatingkernel mean embedding vectors for each region, and learninga standard regression model to map mean embedding vectorsto voting proprortions. Theoretical results about distributionregression support the ability of such an approach to correctlylearn a model to make region-level predictions for new regions,assuming they are distributed in the same way as the regionsused to train the regression model [27].

We argue that EI is more naturally modeled as a learningwith label proportions (LLP) problem. Distribution regressiontreats voting proportions as generic labels associated withthe covariate distributions for each region, which ignores agreat deal of information about the problem. First, we knowthe precise aggregation mechanism: there is one vote perindividual and these votes are added to get the totals. Second,in solving the problem, distribution regression unnecessarilyaggregates the information we do know about individuals (thecovariates) by constructing a mean embedding. In contrast, LLPacknowledges that the voting proportions come from countingthe number of times each label appears in a bag of instances.It is able to use individual covariates and reason relative to theactual aggregation mechanism.

We posit a simple and natural probabilistic latent variablemodel for EI that places it within an LLP framework. Inour model, each individual possesses an observed covariatevector xi and an unobserved label yi (the candidate they votedfor), and, within each region, the total number of votes foreach candidate is obtained by aggregating the yi values. Wethen learn a logistic regression model mapping directly fromcovariates to individual-level labels. Because the individuallabels are unobserved at training time, we use expectationmaximization (EM) for learning. The key computationalchallenge is therefore to perform inference over the yi valuesgiven the totals. We show that techniques for graphical modelswith counting potentials [9], [28] solve this problem exactlyand efficiently. Furthermore, we develop a novel message-

passing algorithm to extend counting potentials to multivariateprobability models, and thus multiclass classifiction. The resultis the the first direct maximum-likelihood approach to LLPbased on the “obvious” latent variable model.

To evaluate different EI methods, we design a realistictestbed for designing synthetic problems that mimic thevoting prediction problem. We use geographic boundaries andindividual-level covariates that match those used in analysisof the US presidential elections. We then design a variety ofsynthetic tasks where we withhold one covariate and treatthis as the variable to be predicted. At training time, onlyaggregated per-region counts are provided for the withheldvariable. Within this framework we control factors such as thenumber of individuals per region and the number of classes toobtain a variety of realistic EI tasks. For the task of learningmodels to make individual-level predictions, we show that LLPmethods significantly outperform distribution regression, andthat our fully probabilistic approach to LLP outperforms otherexisting state-of-the-art methods. We also assess the ability ofdifferent LLP methods as well as distribution regression topredict the voting behavior of different demographic groups inthe 2016 US Presidential Election by making predictions usingEI and then comparing the results with exit poll data. We findthat EI methods do better on qualitative tasks, such as orderingsubgroups by their level of support for Hillary Clinton, thanthey do in predicting precise voting percentages.

II. RELATED WORK

LLP has been applied to many practical applications suchas object recognition [14], ice-water classification [16], frauddetection [21], and embryo selection [11].

Early approaches to LLP do not atempt to infer individuallabels: the Mean Map approach of [19] directly estimatesthe sufficient statistics of each bag (i.e., region) by solving alinear system of equations. The sufficient statistics summarizethe information from each bag that is relevant for estimatingmodel parameters. The Inverse Calibration method of [21]treats the mean of each bag as a “super-instance” (similar tothe kernel mean embedding used in the distribution regressionapproach to EI [7]) and treats label proportions for each bag astarget variables within a variant of Support Vector Regression.In contrast, our work explicitly models individual labels andthe structural dependency between individual labels and theiraggregate class counts.

Several recent LLP approaches reason explicitly aboutindividual labels, but not in a fully probabilistic manner. [25]first clusters training data given label proportions, and classifiesnew instance using either the closest cluster label, or a newclassifier trained from cluster-predicted training data labels.Alter-∝SVM [29] poses a joint large-margin optimizationproblem over individual labels and model parameters, andsolves it using alternating optimization. One step in thealternating optimization imputes individual labels. The Alter-CNN [16] and Alternating Mean Map (AMM) [18] methodsalso alternate between inferring individual lables and updatingmodel parameters. However, all of these approaches infer

“hard” labels for each instance (either 0 or 1). Alter-CNN isformulated probabilistically, but uses “Hard”-EM for learning,which is a heuristic approximate version of EM. In contrast,our method is conventional maximum-likelihood estimation inthe straightforward probability model, and we conduct marginalinference instead of MAP inference over missing individuallabels.

Several other papers formulate probabilistic models for LLP,but, unlike our method, resort to some form of approximateinference, such as “hard”-EM [16] or MCMC [11], [14]. Theauthors of [11] also propose an EM approach with exactprobability calculations in the E step, but using brute-forcealgorithms that do not scale beyond very small problems; forlarger problems they instead resort to approximate inference.In contrast to all previous work, we apply exact and efficientinference algorithms using counting potentials [9], [28] todirectly maximize the likelihood in the natural probabilitymodel.

III. BACKGROUND AND PROBLEM STATEMENT

We now formally introduce the ecological inference problemand describe how it fits within an LLP context. Recall thatwe assume the availability of individual-level covariates andregion-level voting totals for each voting region. In this section,we restrict our attention to binary prediction problems, i.e.,we assume there are only two voting options (e.g., candidatesfrom the two major US political parties). We will generalizeto multiclass problems in Section IV-C.

Within a single voting region, let xi denote a vector ofdemographic features for the ith individual, and let yi ∈ {0, 1}denote that individual’s vote, e.g., yi = 1 if the individual votesfor the Purple party. Note that we never observe yi, and weobtain a sample of xi values from the Public Use MicrodataSample [1]. We also observe z, the total number of votes forthe Purple party, and n, the total number of voters in the region.

Assume there are B regions in total, and, following LLPterminology, refer to regions as “bags” of instances. Theunderlying data for bag b is {(xi, yi)}i∈Ib where Ib is theindex set for bag b. In the training data, instead of observingindividual yi values, we observe zb =

∑i∈Ib yi, the number

of positive instance in the bag. The overall training data is({xi}i∈I1 , z1

), . . . ,

({xi}i∈IB , zB

),

where each example consists of a bag of feature vectors andtotal number of positive votes for the bag. The goal is to learna model to do one of two tasks. The first task is individual-level prediction: predict yi given a new xi. The second taskis bag-level prediction: predict zb given a new bag {xi}i∈Ibwithout any labels.

A. Comparison Between LLP and Distribution Regression

The generative model for LLP is illustrated in Figure 1a.The figure shows a single bag, for which the feature vectors xiand vote total z are observed, but the individual labels yi areunobserved. The conditional distribution of z is specified bythe deterministic relationship z =

∑i yi. In a probabilistic LLP

x1 y1

x2 y2

xn yn

z...

(a) LLP

x1 y1

x2 y2

xn yn

z

!

...

(b) Distribution Regression

Fig. 1. LLP and distribution regression models for EI. (a) In LLP, there is alatent variable yi for each individual, and z =

∑i yi is the number of positive

instances. (b) In distribution regression, yi is ignored; µ is an aggregatedsummary of the xi’s, and a regression model is learned to map directly fromµ to z.

model, p(yi | xi) is Bernoulli, and the modeler may choose anyregression model for the success probability, such as logisticregression, as we do below, or a CNN [16].

For comparison, Figure 1b illustrates the mean embeddingapproach to distribution regression [7], [27]. Here, the yivariables are ignored, as is the known relationship betweenthe yi variables and z. Instead, the (empirical) distribution ofthe xi values in the bag is summarized by a mean embeddinginto a reproducing kernel Hilbert space. In practice, thismeans computing the average µ = 1n

∑ni=1 φ(xi) of expanded

features vectors φ(xi) for each individual. Then, a standardregression model is learned to predict z directy from µ.Distribution regression introduces a tradeoff between the abilityto preserve information about the individual-level covariates andcomplexity of the model. A feature expansion correspondingto a characteristic kernel preserves information about thedistribution of the covariates, but is necessarily infinite andmust be approximated; a very high-dimensional approximationwill likely lead to increased variance in the following regressionproblem. If a simple feature expansion, such as a linear one, isused, it is clear the approach discards significant informationabout the individual-level covariates by simply computing theirmean. LLP avoids this tradeoff by leveraging known structureabout the problem.

IV. OUR APPROACH

We now present our approach, which is based on thegenerative model in Figure 1a and the EM algorithm. Weadopt the logistic regression model p(yi = 1 | xi; θ) = σ(xTi θ)where σ(u) = 1/(1 + e−u). It is straightforward to considermore complex regression models for p(yi = 1 | xi; θ) withoutchanging the overall approach. This completes the specificationof the probability model

We now turn to learning. A standard approach is to find θto maximize the conditional likelihood:

p(y, z | x; θ) =B∏

b=1

(nb∏i=1

p(yi | xi; θ)

)p(zb | yb).

In this equation, let yb = {yi}nbi=1 denote the set of labels in

bag b, let xb = {xi}nbi=1 denote the set of feature vectors, and

let y, z, and x denote the concatenation of the yb, zb, and xbvariables from all bags. Also recall that p(zb | yb) = 1[zb =∑nb

i=1 yi]. The obvious challenge to this learning problem isthat the y variables are unobserved.

A. EM

EM is a classical approach to address the problem of missingvariables within maximum-likelihood estimation [3]. A detailedderivation of EM for our model is given in Appendix A. Inthe tth iteration, we will select θ to maximize the followingfunction, which is a constant plus a lower bound of the log-likelihood:

Qt(θ) = Ey|z,x[log p(y, z | x; θ)]

=∑b

Eyb|zb,xb

(log p(zb | yb) +

∑i

log p(yi | xi; θ)

)=∑b

∑i

Eyi|zb,xb log p(yi | xi; θ) + const (1)

The expectation is taken with respect to the distributionparameterized by θt. The term log p(zb | yb) is constant withrespect to θ and is ignored during the optimization. Specializingto our logistic regression model, the lower bound simplifies to:

Q(θ)=∑b

∑i

qi log σ(xTi θ) + (1−qi) log(1−σ(xTi θ)) (2)

where qi := p(yi = 1 | zb,xb; θt), and we have dropped anadditional constant term from Equation (1).

The M step, which requires maximizing Q(θ) given theqi values, is straightforward. Equation (2) is the same cross-entropy loss function that appears in standard logistic regression,but the with “soft” labels qi appearing in place of the standard0-1 labels. It can be optimized with standard solvers.

The E step, however, is challenging. It requires computingthe posterior distribution p(yi | xb, zb; θt) over a single yi valuegiven the observed data xb and zb and the current parametersθt. This corresponds exactly to inference in the graphical modelshown in Figure 1a. Note that all variables are coupled bythe hard constraint 1[zb =

∑nbi=1 yi], and that this is a factor

involving nb +1 variables, so it is not clear based on standardgraphical model principles that efficient inference is possible.

B. Efficient Exact Inference with Cardinality Potentials

Tarlow et al. [28] showed how to perform efficient marginalinference for a set of n binary random variables y1, . . . , yndescribed by a probability model of the form:

q(y1, . . . , yn) ∝∏i

ψi(yi)φ(∑

i

yi), (3)

where each variable has a unary potential ψi(yi), and thereis a single cardinality potential φ(

∑i yi), which couples all

of the variables but depends only on the number that take apositive value. Our model fits in this form. Consider the modelfor a single bag, and dispense with the bag index, so that thevariables are x = {xi}, y = {yi} and z. Our model for thebag has unary potentials ψi(yi) = p(yi | xi; θ) and counting

potential φ(∑

i yi)= 1[z =

∑yi]. The method of [28] can

compute the marginal probability q(yi) for all i in O(n log2 n)time. In our model q(yi) = p(yi | x, z; θ) is exactly what wewish to compute during the E step, so this yields an E stepthat runs efficiently, in O(n log2 n) time.

We now give details of the inference approach, but presentthem in the context of a novel generalization to the case wheny1, . . . , yn ∈ {0, 1}k are binary vectors of length k. Such ageneralization is necessary for the most direct extension of ourLLP approach to multiclass classification, in which p(yi | xi; θ)is a categorical distribution over three or more alternatives. InSection IV-C, we describe this approach in more detail as wellas a different and faster approach to multiclass LLP.

Henceforth, assume that y1, . . . , yn are binary vectors thatfollow a joint distribution in the same form as Equation (3). Topreview the meaning of the multivariate model, the binaryvector yi will be the “one-hot” encoding of the class forindividual i, the unary potential is ψi(yi) = p(yi | xi; θ),1and the counting potential φ

(∑i yi)= 1[

∑i yi = z] will

encode the constraint that the total number of instances in eachclass matches the observed total, where z is now a vector ofcounts for each class. The description of the multivariate modelis symbolically nearly identical to the scalar case.

The key observation of [28] is that it is possible to introduceauxiliary variables that are sums of hierarchically nested subsetsof the yi variables, and arrange the auxiliary variables in abinary tree with z =

∑i yi at the root. Then, inference is

performed by message-passing in this binary tree.

z1

y1 y2

z2

y3 y4

z3

φ

Fig. 2. Illustration auxiliary variables arranged in a binary tree factor graphfor inference with a cardinality potential. Each non-leaf node is a deterministicsum of its children. The root node z is equal to

∑i yi.

Figure 2 illustrates the construction as a factor graph foran example with n = 4. The nodes are arranged in a binarytree. Each internal node zp is connected to two children, whichwe will denote zl and zr (for left and right), by a factorwhich encodes that zp is deterministically equal to the sumof zl and zp, i.e., ψ(zp, zl, zr) = 1[zp = zl + zr]. The unaryfactors at each leaf are the original unary potentials ψi(yi).The auxiliary nodes and factors enforce that the root node zsatisfies z =

∑i yi. Then, the factor attached to the root node

1Note: this factor has 2k entries indexed by the binary values yi1, . . . , yik .In this particular model, the binary vector is a one-hot vector, soψi(yi1, . . . , yik) is nonzero if and only if there is a single nonzero yij .The inference technique also applies to arbitrary distributions over binaryvectors, for which potentials would not have this structure.

is the cardinality potential φ(z) = 1[z = zobs], where zobs isthe observed total.

This model is a tree-structured factor graph. Hence, exactinference can be performed by message passing using a leaf-to-root pass followed by a root-to-leaf pass. Although there areonly O(n) messages, the support of the variables grows withheight in the tree. A node zl at height i is a sum over 2i of theyi values, so it is a vector with entries in {0, 1, . . . , 2i}. We willwrite this as zl ∈ [m]k where m = 2i and [m] = {0, 1, . . . ,m}.Note that m is never more than n.

zp

zl zr

ψ

αp(zp)

αl(zl) αr(zr)

zp

zl zr

ψ

βp(zp)

βl(zl) αr(zr)

(a) (b)

Fig. 3. Illustration of messages from the factor ψ = 1[zp = zl + zr]. (a)The upward message αp(zp) is computed from αl(zl) and αr(zr); (b) Thedownward message βl(zl) is computed from βp(zp) and αr(zr), similarlyfor βr(zr). See text for details.

The message passing scheme is illustrated in Figure 3.For any internal node zu, let αu(zu) denote the incomingfactor-to-variable message from the factor immediately belowit. Similarly, let βu(zu) be the incoming factor-to-variablemessage from the factor immediately above it. Because eachinternal variable is connected to exactly two factors, thevariables will simply “forward” their incoming messages asoutgoing messages, and we do not need separate notation forvariable-to-factor messages.

The message operation for the upward pass is illustrated inFigure 3(a). The factor ψ connects children zl and zr to parentzp. We assume that zl, zr ∈ [m]k for some m, and thereforezp ∈ [2m]k. The upward message from ψ to zp is

αp(zp) =∑

zl∈[m]k

∑zr∈[m]k

αl(zl)αr(zr)1[zp = zl + zr]

=∑

zl∈[m]kαl(zl)αr(zp − zl). (4)

Upward message computation:

αp(zp) =∑

zl∈[m]kαl(zl)αr(zp − zl)

Downward message computation:

βl(zl) =∑

zp∈[2m]kβp(zp)αr(zp − zl)

βl(zr) =∑

zp∈[2m]kβp(zp)αl(zp − zr)

Fig. 4. Summary of message passing for cardinality potentials. Eachmessage operation is a convolution; the entire message can be computedin O(mk logm) time by the multidimensional FFT. The overall running timeto compute all messages is O(nk log2 n).

Similarly, the downward message to zl, illustrated in Fig-ure 3(b), has the form

βl(zl) =∑

zp∈[2m]k

∑zr∈[m]k

βp(zp)αr(zr)1[zp = zl + zr]

=∑

zp∈[2m]kβp(zp)αr(zp − zl). (5)

Note that the upward and downward message operationsin Equations (4) and (5) both have the form of a con-volution. Specifically, if we let ∗ denote the convolutionoperation, then αp = αl ∗ αr, and βl = βp ∗ α̂r, whereα̂r(zr1, . . . , zrk) = αr(m − zr1, . . . ,m − zrk) is the factorwith the ordering of entries in every dimension reversed. Whilea direct iterative convolution implementation can computeeach message in O(m2k) time, a more efficient convolutionusing multidimensional fast Fourier transform (FFT) takes onlyO(mk logm) time.

The maximum computation time for a single message isO(nk log n), for the messages to and from the root. It can beshown that the total amount of time to compute all messages foreach level of the tree is O(nk log n), so that the overall runningtime is O(nk log2 n). The upward and downward message-passing operations are summarized in Figure 4.

C. Multiclass Classification

We explore two different methods to extend our LLPapproach to multi-class classification: softmax or multinomialregression and one-vs-rest logistic regression. Consider thecase when there are C ≥ 3 classes. It is convenient to assumeyi is encoded using the “one-hot” encoding, i.e., as a binaryvector of length C with yic = 1 if and only if the label is c.For each bag, we now observe the vector z =

∑i yi; the entry

zc is the total number of instances of class c in the bag.a) Softmax regression: The obvious generalization of

our logistic regression model to multiclass classification ismultinomial or softmax regression. In this case, p(yi | xi; θ)is a categorical distribution with probability vector µi = E[yi]obtained through a regression model. The entry µic is theprobability that yi encodes class c, and is given by:

γic = exp(θTc xi), µic = γic/

(∑c′

γic′).

The parameters θ of the model now include a separate parametervector θc for each class c.

Our EM approach generalizes easily to this model. TheM step remains a standard softmax regression problem. TheE step requires computing the posterior probability vectorqi = E[yi | x, z; θ] for every instance in the bag. This isexactly the problem we solved in the previous section forcardinality potentials over binary vectors. Since each yi, µi,and qi vector sums to one, we may drop one entry prior toperforming inference, and complete the E step for a bag withn instances in O(nC−1 log2 n) time.

This approach is appealing because it is follows a widelyused multiclass classification model, which is the naturalgeneralization of logistic regression. However, a drawback

is that the running time of the E step grows exponentially withthe number of classes, which may be too slow in practice whenthe numbers of instances or classes grows large.

b) One-vs-Rest Classification: An obvious alternative tosoftmax regression is one-vs-rest logistic regression, in whicha separate logistic regression model is learned for each class cby training on the binary task of whether or not an instancebelongs to class c. At test time, the class that predicts thehighest probability is selected. This model has been observedto work well in many settings [20]. For our LLP model, ithas a significant running-time advantage: each classifier canbe trained in the binary prediction setting, so the E step willalways run in O(n log2 n) time, regardless of the number ofclasses.

V. EXPERIMENTS

A. Overview

Our approach is designed for the setting where the learner hasaccess to individual-level covariates, but the outcome variablehas been aggregated at the bag level. We apply our model tothe problem of inferring how demographic subgroups votedin an election—from voting results and Census demographicdata, we would like to, for example, infer what proportion ofa particular minority group voted for a particular candidate. Inthis setting,

• The outcome is aggregated at the bag-level: voting isanonymous, and proportions of how people voted areknown only at coarse aggregations by geographic region,from officially released precinct-level vote totals.

• Demographics are individual-level: the U.S. Census re-leases anonymized “microdata,” which consists of covari-ate vectors of demographic attributes of individuals. It isnot known how individuals voted, of course.

Flaxman et al. [6], [7] apply distribution regression for thisproblem, performing aggregation on microdata demographicsas a preprocessing step. In order to test our hypothesis thatindividual-level modeling can improve these inferences, weconduct a series of experiments:

• (§V-B): Synthetic experiments. We follow [30] and hidea known attribute from the individual-level data, and attraining time our model accesses it only as aggregatedproportions per region. We evaluate models in their abilityto accurately predict the hidden attribute for individuals.

• (§V-C): 2016 presidential elections. Here, we look at thesame task as in [6]: trying to infer the individual-levelconditional probability p(vote Dem | f) for any arbitraryfeature f(x) of an individual’s covariates x (e.g., “personis Hispanic/Latino and lives in Illinois”). We train themodel with official per-geography voting tallies for theoutcome, and a large set of Census demographic covariatesfor individuals, and perform custom aggregations of themodel’s inferences to analyze f(x) selections. Quantitativeperformance is evaluated by comparing to separate surveyevidence (exit polls) for the same election.

Individual-level census data (x) is obtained from AmericanCommunity Survey’s Public Use Microdata Sample (PUMS),which covers all of the United States (including D.C. andPuerto Rico).2 Its millions of records each represents a singleperson or demographically typical person, along with a surveyweight representing how many people are represented by thatrecord, based on the Census’ statistical inferences to correctfor oversampling, non-response, and to help preserve privacy.We use Flaxman et al.’s open-source preprocessor (used for[6])3 to select and process the Census data. It merges PUMSdata from 2012–2015, resulting in 9,222,638 records, with anaverage survey weight of 24.2.

PUMS is coded and partitioned geographically by severalhundred Public Use Microdata Areas (PUMAs), which arecontiguous geographic units with at least 100,000 peopleeach. Since PUMAs do not exactly coincide with counties forwhich official electoral statistics are reported, the processingscripts merge them with overlapping counties (taking connectedcomponents of the bipartite PUMA-county overlap graph),resulting in 979 geographical regions. On average each regioncontains 9,420 PUMS records, representing on average 228,342(stdev 357,204) individuals per region, when accounting forsurvey weights.

Each raw individual record x is comprised of 23 continuouscovariates such as income and age, and 97 categorical covariatessuch as race, education, and spoken language. [7] usedFastFood [15] to approximate a kernel map φ(x), then averagedφ(x) vectors to produce per-region mean embeddings fordistribution regression. Materials posted for later work [6]suggest that linear embeddings may perform similarly asnonlinear, random Fourier-based embeddings. To simplify ourcurrent investigation, we only consider linear embeddings inthis work.

We use the same preprocessing routine to further processthe covariates; it standardizes continuous covariates to z-scores,binarizes categorical covariates, and adds regions’ geographicalcentroids, resulting in 3,881 dimensions for x in the model.

For reference, descriptions of several covariates are shownin Table I, including ones inferred for the synthetic predictionexperiments as well as exit poll analysis.4 In some cases,the number of categories results from coarsening the originalCensus categories (e.g. SCHL has 25 categories in the originaldata, but is coarsened to 4 for prediction purposes), usingthe same preprocessing rules as in previous work. (The3,881 dimensions for modeling use the original non-coarsenedcovariates.)

Finally, for computational convenience, we perform twofinal reductions on the x data before model fitting. First, formost experiments we subsample a fixed number of individualsper region, sampling them with replacement according to thePUMS survey weights. Second, we use PCA to reduce the

2https://www.census.gov/programs-surveys/acs/data/pums.html3https://github.com/dougalsutherland/pummeler4Details of all variables are available at: https://www2.census.gov/

programs-surveys/acs/tech_docs/pums/data_dict/PUMSDataDict15.txt

TABLE ICOVARIATES

Num.Classes

Covariate Description

2 SEX Gender2 DIS With or without disability

3 WKL When last worked

4 SCHL Educational attainment (high school or less, somecollege/assoc degree, college graduate, postgraduatestudy)

5 RAC1P Race (White, Black, Asian, Hispanic/Latino, Other)

covariates to 50 dimensions, which preserves approximately80% of the variance in x.

B. Synthetic experiments

1) Partial synthetic data: We create partial synthetic datafollowing the same procedure as [30]: we hide a known discretevariable from x and try to predict it from the other variables. Attraining time, we supply it as supervision only as an aggregatedcount by region (zb =

∑i∈Ib yi). We evaluate models in their

ability to predict the hidden variable for individuals in held-outdata.

The training data is prepared by sub-sampling either 10or 100 individual records per region (as described in §V-A);we test both settings since prior literature has occasionallyexamined performance as a function of bag size. The test set isconstructed to include 10,000 records sampled from all regions(by survey weight), from records that that were never selectedfor the training set.

For certain hidden response variables, some covariates areduplicates or near-duplicates, which makes the predictionproblem too easy. We make the problem harder by removingattributes in x that have at least one value with Pearsoncorrelation higher than 0.7 with the response (hidden attribute).For example, if NATIVITY (native or foreign born) was theresponse variable, it has high absolute correlations to twodifferent values of CIT (citizenship status): binarized valuesCIT_4 (US citizen by naturalization) and CIT_1 (born in theUS) have Pearson correlations of 1 and 0.85, respectively.Furthermore, DECADE_nan (Decade of entry is N/A, meaningborn in the US), and WAOB_1 (world area of birth is US state)also have high absolute correlations (both 0.85). Thus all CIT,DECADE, and WAOB attributes are removed. Depending onwhich hidden attribute is used as response and how manycovariates are highly correlated, the number of covariates(out of 3,881) we removed ranges from 0 for HICOV (healthinsurance available) to 965 for WKL (when last worked).

2) Models: We test a series of logistic regression models,all of which can make predictions for individuals.

• individual: an oracle directly trained on the indi-vidual labels. This is expected to outperform all othermethods, which can only access aggregated counts.

• mean-emb: logistic regression trained with meanembedding vectors and label proportions for each

https://www.census.gov/programs-surveys/acs/data/pums.htmlhttps://github.com/dougalsutherland/pummelerhttps://www2.census.gov/programs-surveys/acs/tech_docs/pums/data_dict/PUMSDataDict15.txthttps://www2.census.gov/programs-surveys/acs/tech_docs/pums/data_dict/PUMSDataDict15.txt

100 200 300 400 500 600 700 800 900

n regions

0.86

0.88

0.90

0.92

0.94

0.96

0.98

accu

racy

10 instances / region

individualcardinalityAMMmean emb

50 150 250 350 450 550 650 750 850 950

n regions

0.86

0.88

0.90

0.92

0.94

0.96

0.98100 instances / region

2 classes (DIS): predictive accuracy vs. n regions

50 150 250 350 450 550 650 750 850 950

n regions

0.725

0.750

0.775

0.800

0.825

0.850

0.875

0.900

accu

racy

10 instances / region

individualAMMmean embcard exactcard ovr

50 150 250 350 450 550 650 750 850 950

n regions

0.725

0.750

0.775

0.800

0.825

0.850

0.875

0.900

100 instances / region3 classes (WKL): predictive accuracy vs. n regions

200 400 600 800n regions

0

1000

2000

3000

4000

5000

seco

nds

10 instances / bag

exactovrAMM

200 400 600 800n regions

0

2000

4000

6000

8000

10000

12000

14000

16000

100 instances / bag

3 classes (WKL): runtime vs. n regions

Fig. 5. Predictive accuracies of trained models for 2, 3, or 4 labels classification tasks. The hidden attributes we chosen are Disability (DIS), When lastworked (WKL), and Ancestry recode (ANC). We consider a small bag and a larger bag (10 or 100 instances per bag)) for each hidden attribute. Shaded errorbars are 95% confidence intervals computed from 10 repeated trials.

region. (Since the sampling already accounts for surveyweights, the mean embedding vector for one region isthe simple average µ̂b =

1n

∑i∈Ib xi.)

• AMM: logistic regression trained on bags of instancesand label proportions. For multiclass problems, we usea one-vs-rest (ovr) approach [18].

• cardinality: our method, trained on bags ofinstances and label proportions, for binary labels(§IV-A,IV-B).

• card-exact: our method for multiclass problems,with exact inference (§IV-C).

• card-ovr: our method for multiclass problems, withan alternative one-vs-rest formulation, using binarycardinality inference.

Following [18], we initialize the LLP methods (AMM, car-dinality, card-exact, card-ovr) from mean-emb’s learnedparameters.

3) Results: Figure 5 shows predictive accuracies of alltrained models on the test set, for several hidden attributes(with 2, 3, and 4 categories each), with the mean and standarddeviation of performance across 10 different trials. Each trialconsists of one sampled training set, shared between all models(all trials use the same test set). Results are broadly similar forother hidden attributes (omitted for space). The results show:

1) LLP outperforms mean embeddings: AMM and thecardinality models substantially improve on mean-emb,

presumably because they can exploit individual-levelcovariate information and the structure of the aggregationmechanism.

2) Our cardinality method is the most accurate LLP method:It outperforms AMM, the previous state-of-the-art forLLP with statistically significant differences for mostsample sizes of regions and individuals. The cardinalitymethod is a little slower than AMM, though the differenceis minimal in the larger bag setting. Asymptotically, therunning time for the “E”-steps of AMM and card-ovrare nearly the same: O(kn log n) and O(kn log2 n),respectively.

3) For multiclass, card-ovr is performs similarly ascard-exact, and is computationally advantageous:card-ovr takes O(kn log2 n) in runtime and only re-quires a 1D FFT, versus card-exact’s O(nk−1 log2 n)runtime and additional memory usage for a multidimen-sional FFT. The exact method also has some precisionissues (numerical underflow of downward messages) whenrunning message passing in larger binary trees. Futurework could pursue improvements to exact multiclasscardinality inference; in the meantime, we recommendone-vs-rest as an effective practical solution.

4) General observations: as expected, the oracle individualmodel outperforms all others. Also note that the larger per-bag samples (100) result in a harder problem than smaller

(10) per-bag samples, since the training-time aggregationhides more information when bags are larger.

C. 2016 US Presidential Election AnalysisFor real-world experiments, our goal is to infer an individual-

level conditional probability p(vote Dem | f) for any arbitraryfeature f(x) of an individual’s covariates x. We will compareour predictions for such subgroups to an alternative, well-established source of information, exit polls at state and nationallevels.

1) Experiment: This experiment requires exit polls forvalidation, and voting data for model training. Exit polls aresurveys conducted on a sample of people at polling stations,right after they finish voting; we use the widely reported 2016exit poll data collected by Edison Research.5

Voting data (z) is based on county-level official results,6

aggregated to the 979 regions. This results in a tuple of votetotals (vD, vR, voth) for each region: how many people votedfor Clinton (D), Trump (R), or another candidate. Since thePUMS data includes information on all people—includingnonvoters—we add in nonvoters to the third category, resultingin the following count vector for each region:

z = (vD, vR, S − vD − vR)

where S is the PUMS estimate of the number of persons in theregion (sum of survey weights). This is a somewhat challengingsetting for the model, requiring it to learn what type of peoplevote, as well as their voting preference.

We test the mean embedding model, AMM, and the one-vs-rest cardinality model, using all 3,881 covariates. Unlikethe previous section, we give the mean embedding modeladditional access to all instances in the data (mean embeddingsare constructed from a weighted average of all PUMS recordsper region), following previous work. By contrast, for the LLPmodels we sample 1000 individuals per region. PCA is againapplied for all models in the same manner as before, and theLLP models are again initialized with mean-emb’s learnedparameters.

For evaluation, we prepare a held-out dataset with a 1%subsample from all regions in the 28 exit poll states. Aftertraining each model, we make predictions on held-out recordsand aggregate them to state-level and nation-level statistics, sothey can be compared against exit polls. We specifically inferfraction of the two-party vote

p(vote D | vote D or R, f(x)) = nD,fnD,f + nR,f

where nD,f and nR,f are counts of the model’s (hard)predictions for individuals in the test set with property f(x): forexample, nD,f (and nR,f ) might be the number of Clinton (andTrump) voters among Hispanics in Illinois. These quantitiesare calculated from exit polls as well for comparison.

5This questionnaire was completed by 24,537 voters at 350 voting placesthroughout the US on Election Day, from 28 states intended to be representativeof the U.S. We use data scraped from the CNN website, available at: https://github.com/Prooffreader/election_2016_data.

6We use [6]’s version of the data, scraped from NBC.com the day after theelection: https://github.com/flaxter/us2016

2) Results: The scatter plots in Figure 6 show predictionsmade by different methods vs. the exit poll results. The columnscorrespond to methods, and the rows correspond to the featureused to define subgroups. Each data point represents thesubgroup for one feature value (e.g., males) in one state. Thereare up to 28 points per feature value; there may be fewer dueto missing statistics in some state-level exit polls. For example,only 1% of respondents in Iowa exit polls were Asian, and theIowa exit polls do not report the voting breakdown for Asians.

The scatter plots show that EI methods are indeed able tomake correct inferences, but also make certain mistakes. Formost methods and feature values (e.g., mean-emb, males),the state-level predictions are strongly linearly correlated withthe exit polls—that is, the method correctly orders the statesby how much males in the state supported Clinton. However,subgroups are often clustered at different locations away fromthe 1:1 line, indicating systematic error for that group—thisis especially prominent for SCHL, where all methods tend tooverestimate support for Clinton among college graduates, andunderestimate support among individuals with high school orless education or with some college. In other examples, suchas mean-emb for ETHNICITY=white, the overall positioningof the subgroup is correct and the state-level predictions arewell correlated with the exit polls, but the slope is wrong. Thissuggests that the model has not correctly learned the magnitudeof other features that vary by state. Overall, the LLP methodsappear qualitatively better (predictions more clustered aroundthe 1:1 line) for SEX and ETHNICITY, while there is no clear“winner” for SCHL.

It is also interesting to aggregate the state-level predictionsto national-level predictions. Table II shows national-levelpredictions as well exit polls for subgroups defined by gender(SEX), race (RAC1P), and educational attainment (SCHL).We see here that the models make mostly correct qualitativecomparisons, such as: Which subgroup has a higher-level ofsupport for Clinton? Does the majority of a subgroup supportClinton or Trump? However, the models make notable errorspredicting the majority among men and women. Moreover, themodels have a difficult time predicting the exact percentageseven when the qualitative comparisons are correct.

To quantify these issues further and to gain a bettercomparison between the methods, Table III evaluates themethods based on national-level predictions according to threedifferent metrics for each feature:

1) Binary prediction is the number of subgroups for whichthe method correctly predicts which candidate receivesthe majority (e.g., “a majority of males supported Trump”,“a majority of women supported Clinton”).

2) AUC measures the ability of the methods to ordersubgroups by their level of support for Clinton (e.g.,“females showed higher support for Clinton than males”).It is measured by ordering the groups by predictedsupport for Clinton, and then measuring the fractionof pairs of groups that are in the correct order relative tothe exit polls; this is related to the area under the ROCcurve [5].

https://github.com/Prooffreader/election_2016_datahttps://github.com/Prooffreader/election_2016_datahttps://github.com/flaxter/us2016

0.0 0.5 1.0exit

0.0

0.2

0.4

0.6

0.8

1.0

pred

mean emb

F M

0.0 0.5 1.0exit

0.0

0.2

0.4

0.6

0.8

1.0

pred

AMM

F M

0.0 0.5 1.0exit

0.0

0.2

0.4

0.6

0.8

1.0

pred

cardinality

F M

(a) SEX

0.0 0.5 1.0exit

0.0

0.2

0.4

0.6

0.8

1.0

pred

mean emb

Asian Black Latino OthersWhite

0.0 0.5 1.0exit

0.0

0.2

0.4

0.6

0.8

1.0

pred

AMM

Asian Black Latino OthersWhite

0.0 0.5 1.0exit

0.0

0.2

0.4

0.6

0.8

1.0

pred

cardinality

Asian Black Latino OthersWhite

(b) ETHNICITY

0.0 0.5 1.0exit

0.0

0.2

0.4

0.6

0.8

1.0

pred

mean emb

College graduate High school or less Postgraduate Some college

0.0 0.5 1.0exit

0.0

0.2

0.4

0.6

0.8

1.0

pred

AMM

College graduate High school or less Postgraduate Some college

0.0 0.5 1.0exit

0.0

0.2

0.4

0.6

0.8

1.0

pred

cardinality

College graduate High school or less Postgraduate Some college

(c) SCHL

Fig. 6. Model predictions versus exit polls, by demographic group and state.Each color (demographic category) has up to 28 points for its subpopulationper state (for subgroups large enough such that the exit poll results showvoting numbers).

3) Weighted RMSE measures the numerical accuracy ofthe predictions. It is the square root of the weightedmean-squared error between the predicted and exit pollpercentages, with weights given by the size of eachsubgroup.

The results show that the models are indeed generally good atthe comparison tasks, as shown by the binary prediction andAUC metrics. However, they have considerable error (RMSEmore than 5% in all cases) predicting percentages. There isno clear winner among the methods across all metrics. Themean embedding model has the lowest AUC for two out ofthree variables, and is tied on the third variable.

TABLE IINATIONAL-LEVEL VOTING PREDICTIONS FOR CLINTON PER DEMOGRAPHIC

GROUP

mean-emb AMM card exit

SEXM 0.57 0.51 0.41 0.44F 0.44 0.47 0.45 0.57

RAC1P

Latino/Hispanic 0.63 0.69 0.77 0.70White 0.42 0.38 0.31 0.39Black 0.74 0.93 0.99 0.92Asian 0.58 0.91 1.00 0.71Others 0.77 0.83 0.94 0.61

SCHL

High school or less 0.44 0.32 0.18 0.47Some college 0.34 0.34 0.26 0.46

College graduate 0.71 0.73 0.71 0.53Postgraduate 0.60 0.68 0.68 0.61

TABLE IIIDEMOGRAPHIC-LEVEL MODEL ACCURACY IN PREDICTING VOTING

PROPORTIONS, COMPARED TO EXIT POLLS.

Binaryprediction

AUC WeightedRMSE

SEXembed 0/2 0 0.13AMM 0/2 0 0.09card 1/2 1 0.09

RAC1P

embed 5/5 0.7 0.08AMM 5/5 0.9 0.06card 5/5 0.8 0.11

SCHLembed 4/4 0.83 0.12AMM 4/4 0.83 0.15card 4/4 0.83 0.20

VI. CONCLUSION

In this paper we formulated the ecological inference problem,motivated by analysis of US presidential elections, in theframework of learning with label proportions. Comparedwith previous approaches, this allows us to use more knownstructure of the problem, and preserve information in individual-level covariates available to us from Census microdata. Wecontributed a novel, fully probabilistic, LLP method thatoutperforms distribution regression and a state-of-the-art LLPmethod on a range of synthetic tasks. Our probabilistic approachis enabled by adapting message-passing inference algorithmsfor counting potentials to a natural latent-variable model forLLP. We also applied several methods to analyze the 2016 USpresidential election, and found that models frequently makecorrect comparisons among different choices or groups, butmay not predict percentages accurately. Here, no method was aclear winner, but state-level results suggest that LLP methodsare in closer agreement with exit polls.

A direction for further exploration is the potential of non-linear methods to improve performance. Previous work usednon-linear feature embeddings, and, to a lesser extent, non-linear kernels for classification [7]. In this work we havefocused on linear embeddings and linear classifiers. However,our method can support arbitrary non-linear regression models,e.g., neural networks, for the individual-level model to predictyi from xi. Exploration of such models is a worthwhile avenuefor future research.

ACKNOWLEDGMENTS

This material is based on work supported by the NationalScience Foundation under Grant No. 1522054. Thanks to SethFlaxman and Dougal Sutherland for feedback.

REFERENCES

[1] “Public Use Microdata Sample (PUMS), 2010-2014 ACS 5-year PUMSand 2015 ACS 1-year PUMS.” U.S. Census Bureau; American Com-munity Survey (ACS). Available from https://www.census.gov/programs-surveys/acs/data/pums.html.

[2] G. Bernstein and D. Sheldon, “Consistently estimating markov chainswith noisy aggregate data,” in Proceedings of the 19th InternationalConference on Artificial Intelligence and Statistics, 2016, pp. 1142–1150.

[3] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihoodfrom incomplete data via the EM algorithm,” Journal of the RoyalStatistical Society. Series B (Methodological), vol. 39, no. 1, pp. 1–38,1977.

[4] T. G. Dietterich, R. H. Lathrop, and T. Lozano-Pérez, “Solving themultiple instance problem with axis-parallel rectangles,” ArtificialIntelligence, vol. 89, no. 1, pp. 31 – 71, 1997.

[5] T. Fawcett, “An introduction to ROC analysis,” Pattern recognition letters,vol. 27, no. 8, pp. 861–874, 2006.

[6] S. Flaxman, D. Sutherland, Y.-X. Wang, and Y. W. Teh, “Understandingthe 2016 US presidential election using ecological inference anddistribution regression with census microdata,” stat.AP arXiv:1611.03787,2016.

[7] S. R. Flaxman, Y.-X. Wang, and A. J. Smola, “Who supported Obamain 2012?: Ecological inference through distribution regression,” inProceedings of the 21th ACM SIGKDD International Conference onKnowledge Discovery and Data Mining. ACM, 2015, pp. 289–298.

[8] D. A. Freedman, S. P. Klein, M. Ostland, and M. Roberts, “On “solutions”to the ecological inference problem,” Journal of the American StatisticalAssociation, vol. 93, no. 444, pp. 1518–22, 1998.

[9] R. Gupta, A. A. Diwan, and S. Sarawagi, “Efficient inference withcardinality-based clique potentials,” in Proceedings of the 24th interna-tional conference on Machine learning. ACM, 2007, pp. 329–336.

[10] H. Hajimirsadeghi and G. Mori, “Multi-instance classification by max-margin training of cardinality-based markov networks,” IEEE Transac-tions on Pattern Analysis and Machine Intelligence, 2016.

[11] J. Hernández-González, I. Inza, and J. A. Lozano, “Learning bayesiannetwork classifiers from label proportions,” Pattern Recognition, vol. 46,no. 12, pp. 3425–3440, 2013.

[12] G. King, “The future of ecological inference research: A reply to freedmanet al.” Journal of the American Statistical Association, vol. 94, pp. 352–355, March 1999.

[13] ——, A solution to the ecological inference problem: Reconstructingindividual behavior from aggregate data. Princeton University Press,2013.

[14] H. Kück and N. de Freitas, “Learning about individuals from groupstatistics,” in Proceedings of the Twenty-First Conference on Uncertaintyin Artificial Intelligence. AUAI Press, 2005, pp. 332–339.

[15] Q. Le, T. Sarlós, and A. Smola, “Fastfood-approximating kernelexpansions in loglinear time,” in Proceedings of the 30th InternationalConference on Machine Learning (ICML-13), 2013.

[16] F. Li and G. Taylor, “Alter-CNN: An approach to learning from labelproportions with application to ice-water classification,” in NIPS 2015Workshop on Learning and privacy with incomplete data and weaksupervision, 2015.

[17] B. Muzellec, R. Nock, G. Patrini, and F. Nielsen, “Tsallis regularizedoptimal transport and ecological inference.” in AAAI, 2017, pp. 2387–2393.

[18] G. Patrini, R. Nock, T. Caetano, and P. Rivera, “(Almost) no label nocry,” in Advances in Neural Information Processing Systems, 2014, pp.190–198.

[19] N. Quadrianto, A. J. Smola, T. S. Caetano, and Q. V. Le, “Estimatinglabels from label proportions,” Journal of Machine Learning Research,vol. 10, no. Oct, pp. 2349–2374, 2009.

[20] R. Rifkin and A. Klautau, “In defense of one-vs-all classification,” Journalof machine learning research, vol. 5, no. Jan, pp. 101–141, 2004.

[21] S. Rueping, “SVM classifier estimation from group probabilities,” inProceedings of the 27th International Conference on Machine Learning(ICML-10), 2010, pp. 911–918.

[22] A. A. Schuessler, “Ecological inference,” Proceedings of the NationalAcademy of Sciences, vol. 96, no. 19, pp. 10 578–10 581, 1999.

[23] D. Sheldon, T. Sun, A. Kumar, and T. G. Dietterich, “Approximateinference in collective graphical models,” in International Conferenceon Machine Learning (ICML), vol. 28, no. 3, 2013, pp. 1004–1012.

[24] D. R. Sheldon and T. G. Dietterich, “Collective graphical models,” inAdvances in Neural Information Processing Systems, 2011, pp. 1161–1169.

[25] M. Stolpe and K. Morik, “Learning from label proportions by optimizingcluster model selection,” Machine Learning and Knowledge Discoveryin Databases, pp. 349–364, 2011.

[26] T. Sun, D. Sheldon, and A. Kumar, “Message passing for collectivegraphical models,” in Proceedings of the 32nd International Conferenceon Machine Learning (ICML-15), 2015, pp. 853–861.

[27] Z. Szabó, B. K. Sriperumbudur, B. Póczos, and A. Gretton, “Learningtheory for distribution regression,” Journal of Machine Learning Research,vol. 17, no. 152, pp. 1–40, 2016.

[28] D. Tarlow, K. Swersky, R. S. Zemel, R. P. Adams, and B. J. Frey,“Fast exact inference for recursive cardinality models,” in Proceedings of

the Twenty-Eighth Conference Conference on Uncertainty in ArtificialIntelligence. AUAI Press, 2012.

[29] F. Yu, D. Liu, S. Kumar, J. Tony, and S.-F. Chang, “∝SVM forlearning with label proportions,” in Proceedings of The 30th InternationalConference on Machine Learning, 2013, pp. 504–512.

[30] F. X. Yu, K. Choromanski, S. Kumar, T. Jebara, and S.-F. Chang, “Onlearning from label proportions,” arXiv preprint arXiv:1402.5902, 2014.

APPENDIX

A. EM Derivation

The model is:

p(y, z | x; θ) =B∏

b=1

(p(zb | yb)

nb∏i=1

p(yi | xi; θ)

)(6)

where p(zb | yb) = 1[zb =∑nb

i=1 yi].We wish to maximize the marginal log-likelihood of the

observed data z conditioned on x, i.e., maximize

`(θ) = log p(z | x; θ) = log∏b

p(zb | xb; θ)

=∑b

log∑yb

p(zb,yb | xb; θ)

We may introduce a variational distribution µb(yb) and applyJensen’s inequality to obtain a lower bound Q(θ) for `(θ):

`(θ) =∑b

log∑yb

µb(yb)p(zb,yb | xb; θ)

µb(yb)

≥∑b

∑yb

µb(yb) logp(zb,yb | xb; θ)

µb(yb):= Q(θ)

It is well known that, for a particular value θt, thislower bound is maximized when µb(yb) = p(yb |zb,xb; θt). After making this substitution and dropping the term∑

b

∑yb−µb(yb) logµb(yb),which is constant with respect to

θ, we may rewrite Q(θ) as:

Q(θ) =∑b

Eyb|zb,xb log p(zb,yb | xb; θ)

=∑b

Eyb|zb,xb[log p(zb | yb) +

∑i

log p(yi | xi; θ)]

=∑b

∑i

Eyi|zb,xb log p(yi | xi; θ) + const

In these equations, the expectation is with respect to thedistribution p(yb | zb,xb; θt) parameterized by the fixed valueθt. In the last line, log p(zb | yb) is constant with respect toθ and is ignored. Specializing now to our logistic regressionmodel, we have:

Q(θ)=∑b

∑i

Eyi|zb,xb[yi log σ(x

Ti θ) + (1−yi) log

(1−σ(xTi θ)

)]=∑b

∑i

qi log σ(xTi θ) + (1− qi) log

(1− σ(xTi θ)

)where qi := p(yi = 1 | zb,xb; θt) is the posterior probability ofyi given the observed data, and Q(θ) remains a cross-entropyloss with the “soft” labels qi taking the place of yi.