GUPT: Privacy Preserving Data Analysis Made Easy

Prashanth MohanUC Berkeley

prmohan@cs.berkeley.edu

Abhradeep Thakurta∗Pennsylvania State University

azg161@cse.psu.edu

Elaine ShiUC Berkeley

elaines@cs.berkeley.eduDawn SongUC Berkeley

dawnsong@cs.berkeley.edu

David E. CullerUC Berkeley

culler@cs.berkeley.edu

ABSTRACTIt is often highly valuable for organizations to have theirdata analyzed by external agents. However, any programthat computes on potentially sensitive data risks leaking in-formation through its output. Differential privacy providesa theoretical framework for processing data while protectingthe privacy of individual records in a dataset. Unfortunately,it has seen limited adoption because of the loss in outputaccuracy, the difficulty in making programs differentiallyprivate, lack of mechanisms to describe the privacy bud-get in a programmer’s utilitarian terms, and the challengingrequirement that data owners and data analysts manuallydistribute the limited privacy budget between queries.

This paper presents the design and evaluation of a newsystem, GUPT, that overcomes these challenges. Unlikeexisting differentially private systems such as PINQ andAiravat, it guarantees differential privacy to programs notdeveloped with privacy in mind, makes no trust assump-tions about the analysis program, and is secure to all knownclasses of side-channel attacks.

GUPT uses a new model of data sensitivity that degradesprivacy of data over time. This enables efficient allocationof different levels of privacy for different user applicationswhile guaranteeing an overall constant level of privacy andmaximizing the utility of each application. GUPT also in-troduces techniques that improve the accuracy of outputwhile achieving the same level of privacy. These approachesenable GUPT to easily execute a wide variety of data anal-ysis programs while providing both utility and privacy.

Categories and Subject DescriptorsD.4.6 [Operating Systems]: Security and Protection; H.3.5[Information storage and retrieval]: Online InformationServices

KeywordsAlgorithms, Differential Privacy, Data Mining, Security

∗Part of this work was done while visiting UC Berkeley.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.SIGMOD ’12, May 20–24, 2012, Scottsdale, Arizona, USA.Copyright 2012 ACM 978-1-4503-1247-9/12/05 ...$10.00.

1. INTRODUCTIONOrganizations frequently allow third parties to perform

business analytics and provide services using aggregated data.For instance, social networks have enabled many applica-tions to be built on social data providing services such asgaming, car-pooling and online deals. Retailers often sharedata about their customers’ purchasing behavior with prod-uct merchants for more effective and targeted advertisingbut do not want to reveal individual customers. While shar-ing information can be highly valuable, companies providelimited access to this data because of the risk that an indi-vidual’s privacy would be violated. Laws such as the HealthInsurance Portability and Accountability Act (HIPAA) haveconsidered the dangers of privacy loss and stipulate thatpatient’s personally identifiable information should not beshared. Unfortunately, even in datasets with “anonymized”users, there have been cases where user privacy was breached.Examples include the deanonymization of AOL search logs [3],the identification of patients in a Massachusetts hospital bycombining the public voters list with the hospital’s anonymizeddischarge list [23] and the identification of the users in ananonymized Netflix prize data using an IMDB movie ratingdataset [18].

Often the value in sharing data can be obtained by al-lowing analysts to run aggregate queries spanning a largenumber of entities in the dataset while disallowing analystsfrom being able to record data that pertains to individual en-tities. For instance, a merchant performing market researchto identify their next product would want to analyze cus-tomer behavior in retailers’ databases. The retailers mightbe willing to monetize the dataset and share aggregate an-alytics with the merchant, but would be unwilling to allowthe merchant to extract information specific to individualcustomers in the database. Researchers have invented tech-niques that ranged from ad-hoc obfuscation of data entries(such as the removal of Personally Identifiable Information)to more sophisticated anonymization mechanisms satisfyingprivacy definitions like k-anonymity [25] and `-diversity [15].However, Ganta et al. [8] and Kifer [13] showed that practi-cal attacks can be mounted against all these techniques. Arecent definition called differential privacy [5] formalizes thenotion of privacy of an individual in a dataset. Unlike earliertechniques, differentially private mechanisms use statisticalbounds to limit the privacy loss of an individual incurredby running statistical queries on datasets. It is designed toperturb the result of a computation in a manner that has lit-tle effect on aggregates, yet obscures the data of individualconstituents.

349

While differential privacy has strong theoretical proper-ties, the shortcomings of existing differentially private dataanalysis systems have limited its adoption. For instance, ex-isting programs cannot be leveraged for private data anal-ysis without modification. The magnitude of the pertur-bation introduced in the final output is another cause ofconcern for data analysts. Differential privacy systems op-erate using an abstract notion of privacy, called the ‘privacybudget’. Intuitively a lower privacy budget implies betterprivacy. However, this unit of privacy does not easily trans-late into the utility of the program and is thus difficult fordata analysts who are not experts in privacy to interpret.Further, analysts would also be required to efficiently dis-tribute this limited privacy budget between multiple queriesoperating on a dataset. An inefficient distribution of theprivacy budget would result in inaccurate data analysis andreduce the number of queries that can be safely performedon the dataset.

We introduce GUPT1, a platform that allows organiza-tions to allow external aggregate analysis on their datasetswhile ensuring that data analysis is performed in a differ-entially private manner. It allows the execution of existingprograms with no modifications, eliminating the expensiveand demanding task of rewriting programs to be differen-tially private. GUPT enables data analysts to specify adesired output accuracy rather than work with an abstractprivacy budget. Finally, GUPT automatically parallelizesthe task across a cluster ensuring scalability for concurrentanalytics. We show through experiments on real datasetsthat GUPT overcomes many shortcomings of existing dif-ferential privacy systems without sacrificing accuracy.

1.1 ContributionsWe design and develop GUPT, a platform for privacy-

preserving data analytics. We introduce a new model fordata sensitivity which applies to a large class of datasetswhere the privacy requirement of data decreases over time.As we will explain in Section 3.3, using this model is ap-propriate and allows us to overcome significant challengesthat are fundamental to differential privacy. This approachenables us to analyze less sensitive data to get reasonableapproximations of privacy parameters that can be used fordata queries running on the newer data.

GUPT makes the following technical contributions thatmake differential privacy usable in practice:

1. Describing privacy budget in terms of accuracy:Data analysts are accustomed to the idea of workingwith inaccurate output (as is the case with data sam-pling in large datasets and many machine learning al-gorithms have probabilistic output). GUPT uses theaging model of data sensitivity, to allow analysts todescribe the abstract ‘privacy budget’ in terms of ex-pected accuracy of the final output.

2. Privacy budget distribution: GUPT automati-cally allocates a privacy budget to each query in or-der to match the data analysts’ accuracy requirements.Further, the analyst also does not have to distributethe privacy budget between the individual data oper-ations in the program.

3. Accuracy of output: GUPT extends a theoreticaldifferential privacy framework called “sample and ag-gregate” (described in Section 2.1) for practical ap-

1GUPT is a Sanskrit word meaning ‘Secret’.

plicability. This includes using a novel data resam-pling technique that reduces the error introduced bythe framework’s data partitioning scheme. Further,the aging model of data sensitivity allows GUPT toselect an optimal partition size that reduces the per-turbation added for differential privacy.

4. Prevent side channel attacks: GUPT defends againstside channel attacks such as the privacy budget at-tacks, state attacks and timing attacks described in [10].

2. BACKGROUNDDifferential privacy places privacy research on a firm the-

oretical foundation. It guarantees that the presence or ab-sence of a particular record in a dataset will not signifi-cantly change the output of any computation on a statisticaldataset. An adversary thus learns approximately the sameinformation about any individual record, irrespective of itspresence or absence in the original dataset.

Definition 1 (ε-differential privacy [5]). Arandomized algorithm A is ε-differentially private if for alldatasets T, T ′ ∈ Dn differing in at most one data record andfor any set of possible outputs O ⊆ Range(A), Pr[A(T ) ∈O] ≤ eε Pr[A(T ′) ∈ O] . Here D is the domain from whichthe data records are drawn.

The privacy parameter ε, also called the privacy budget [16],is fundamental to differential privacy. Intuitively, a lowervalue of ε implies stronger privacy guarantee and a highervalue implies a weaker privacy guarantee while possibly achiev-ing higher accuracy.

2.1 Sample and Aggregate

Algorithm 1 Sample and Aggregate Algorithm [24]

Input: Dataset T ∈ Rn, length of the dataset n, privacyparameters ε, output range (min,max).

1: Let ` = n0.4

2: Randomly partition T into ` disjoint blocks T1, · · · , T`.3: for i ∈ {1, · · · , `} do4: Oi ← Output of user application on dataset Ti.5: If Oi > max, then Oi ← max.6: If Oi < min, then Oi ← min.7: end for8: A← 1

`

∑`i=1Oi + Lap( |max−min |

`·ε )

GUPT leverages and extends the “sample and aggregate”framework [24, 19] (SAF) to design a practical and usablesystem which will guarantee differential privacy for arbitraryapplications. Given a statistical estimator P(T ) , where Tis the input dataset , SAF constructs a differentially privatestatistical estimator P(T ) using P as a black box. Moreover,

theoretical analysis guarantees that the output of P(T ) con-verges to that of P(T ) as the size of the dataset T increases.

As the name “sample and aggregate” suggests, the algo-rithm first partitions the dataset into smaller subsets; i.e.,` = n0.4 blocks (call them T1, · · · , T`) (see Figure 1). Theanalytics program P is applied on each of these datasets Tiand the outputs Oi are recorded. The Oi’s are now clampedto within an output range that is either provided by the an-alyst or inferred using a range estimator function. (Refer to

350

T

T1 T2 T3 Tl …

Average

DATASET

BLOCKS

f … f f f PROGRAM

+ Laplacian noise

Private output

f(T1) f(T2) f(T3) f(Tl)

Figure 1: An instantiation of the Sample and Ag-gregate Framework [24].

Section 4.1 for more details.) Finally, a differentially privateaverage of the Oi’s is calculated by adding Laplace noise(scaled according to the output range). This noisy final out-put is now differentially private. The complete algorithm isprovided in Algorithm 1. Note that the choice of number ofblocks ` = n0.4 is from [24], used here for completeness. Forimproved choices of `, see Section 4.3.

GUPT extends the conventional SAF described above inthe following ways: i) Resampling: GUPT introduces theuse of data resampling to improve the experimental accuracyof SAF, without degrading the privacy guarantee; ii) Opti-mal block allocation: GUPT further improves experimen-tal accuracy by finding the better block sizes (as comparedto the default choice of n0.6) using the aging of sensitivitymodel explained in Section 3.3.

2.2 Related WorkA number of advances in differential privacy have sought

to improve the accuracy of very specific types of data queriessuch as linear counting queries [14], graph queries [12] andhistogram analysis [11]. A recent system called PASTE [21]allows queries on time series data where the data is storedon distributed nodes and no trust is laid on the central ag-gregator. In contract to PASTE, GUPT trusts the aggre-gator with storing all of the data but provides a flexiblesystem that supports many different types of data analysisprograms.

While systems tailored for specific tasks could potentiallyachieve better output accuracy, GUPT trades this for thegenerality of the platform. We show through experimentalresults that GUPT achieves reasonable accuracy for prob-lems like clustering and regression, and can even performbetter than the existing customized systems.

Other differential privacy systems such as PINQ [16] andAiravat [22] have also attempted to operate on a wide va-riety of data queries. PINQ (Privacy INtegrated Queries)proposed programming constructs which enable applicationdevelopers to write differentially private programs using ba-sic functional building blocks of differential privacy (e.g.,exponential mechanism [17], noisy counts [5] etc.). It doesnot consider the application developer to be an adversary.It further requires the developers to rewrite the applicationto make use of the PINQ primitives. On the other hand,Airavat was the first system that attempted to run unmodi-fied programs in a differentially private manner. It howeverrequired the programs to be written for the Map-Reduce

programming paradigm [4]. Further, Airavat only considersthe map program to be an “untrusted” computation whilethe reduce program is “trusted” to be implemented in a dif-ferentially private manner. In comparison, GUPT allows forthe private analysis of a wider range of unmodified programs.GUPT also introduces techniques that allow data analyststo specify their privacy budget in units of output accuracy.Section 7.3 presents a detailed comparison of GUPT withPINQ, Airavat and the sample and aggregate framework.

Similar to iReduct [28], GUPT introduces techniques thatreduce the relative error (in contrast to absolute error). Bothsystems use a smaller privacy budget for programs that pro-duce larger outputs, as the relative error would be smallas compared programs that generate smaller values for thesame absolute error. While iReduct optimizes the distri-bution of privacy budget across multiple queries, GUPTmatches the relative error to the privacy budget of individualqueries.

3. PROBLEM SETUPThere are three logical parties:1. The analyst/programmer, who wishes to perform ag-

gregate data analytics over sensitive datasets. Ourgoal is to make GUPT easy to use for an average pro-grammer who is not a privacy expert.

2. The data owner, who owns one or more datasets, andwould like to allow analysts to perform data analyticsover the datasets without compromising the privacy ofusers in the dataset.

3. The service provider, who hosts the GUPT service.The separation between these parties is logical; in reality,

either the data owner or a third-party cloud service providercould host GUPT.

Trust assumptions: We assume that the data owner andthe service provider are trusted, and that the analyst is un-trusted. In particular, the programs supplied by the analystmay act maliciously and try to leak information. GUPTdefends against such attacks using the security mechanismsproposed in Section 6.

3.1 GUPT Overview

����������

� ������ ����

1. Data Set 2. Privacy ↵Budget (ε)

���������� ������������

���������� ������������

���������� ������������

����� ���� ����

������������ ���

������������������ ����

�

1.  Computation 2.  Accuracy 3.  Output Range

Differentially Private Answer

�

����� ��������� �������������

Data Analyst

Data Owner

Figure 2: Overview of GUPT’s Architecture

The building blocks of GUPT is shown in Figure 2:• The dataset manager is a database that registers in-

351

stances of the available datasets and maintains theavailable privacy budget.• The computation manager instantiates computations

and seamlessly pipes data from the dataset to the ap-propriate instances.• The isolated execution chambers isolate and prevent

any malicious behavior by the computation instances.These building blocks allow the principals of the system

to easily interact with many parameters being automaticallyoptimized or are left as optional arguments for experts.

Interface with the data owner: (a) A multi-dimensionaldataset (such as a database table) that for the purposeof our discussion, we assume is a collection of real valuedvectors , (b) a total privacy budget that can be allocatedfor computations on the dataset and (c) [Optional] inputattribute ranges, i.e. the lower and upper bound for eachdimension of the data. A detailed discussion on these boundsis presented in Section 4.1.

For privacy reasons, the input ranges provided should notcontain any sensitive information. For example, it is rea-sonable to expect that a majority of the household’s annualincome should fall within the range [0; 500,000] dollars, thusit is not considered sensitive. On the other hand if the rich-est household’s income is used as the upper bound privateinformation could be leaked. In this case, a public informa-tion source such as the national GDP could be used.

Interface with the analyst: (a) Data analytics pro-gram, (b) a reference to the data set in the datasetmanager, (c) either a privacy budget or the desired accu-racy for the final answer and finally (d) an output rangeor a helper function for estimating the output range.

A key requirement for the analytics program is that itshould be able to run on any subset of the original dataset.As GUPT executes the application in a black-box fashion,the program may also be provided as a binary executable.

Privacy budget distribution: In order to guarantee ε-differential privacy for a dataset T , the privacy budget shouldbe distributed among the various applications (call themA1, · · · ,Ak) that operate on T . A composition lemma [5]states that if A1, · · · ,Ak guarantee ε1, · · · , εk-differentialprivacy respectively, then T is ε-differential private, whereε =

∑ki=1 εi. Thus judicious allocation of the privacy budget

is extremely important. Unlike existing differential privacysolutions, GUPT relieves the analyst and the data ownerfrom that task of distributing this limited privacy budgetbetween multiple data analytics programs.

3.2 Privacy and Utility GuaranteesGUPT guarantees ε-differential privacy to the final out-

put. It provides similar utility guarantees as the originalsample and aggregate algorithm from [24]. This guaranteeapplies to the queries satisfying “approximate2 normality”condition defined by Smith [24], who also observed that awide class of queries satisfy this normality condition. Someof the examples being various maximum-likelihood estima-tors and estimators for regression problems. The formalutility guarantees are deferred to Appendix A.GUPT provides the same level of privacy for queries that

are not approximately normal. Reasonable utility could be

2By approximately normal statistic we refer to the genericasymptotically normal statistic in Definition 2 of [24].

expected even from queries that are not approximately nor-mal, even though no theoretical guarantees are provided.

3.3 Aging of SensitivityIn real life datasets, the potential privacy threat for each

record is different. A privacy mechanism that considers thiscan obtain good utility while satisfying the privacy con-straints.

We introduce a new model called aging of sensitivity ofdata where older data records are considered to have lowerprivacy requirements. GUPT uses this model to optimizesome parameters of the sample and aggregate frameworklike block size (Section 4.3) and privacy budget allocation(Section 5.1). Consider the following motivating example:

Example 1. Let T70 yrs and Tnow be two datasets contain-ing the ages of citizens in a particular region 70 years earlierand at present respectively. It is conceivable that the privacythreat to T70 yrs is much lower as many of the participatingpopulation may have deceased. Although T70 yrs might not beas useful as Tnow for learning specific trends about the cur-rent population, it can be used to learn some general conceptsabout Tnow. For example, a crude estimate of the maximumage present in Tnow can be obtained from T70 yrs.

GUPT estimates such general trends in data distributionand uses them to optimize the performance of the system.The optimization results in a significant reduction in error.More precisely, the aged data is used for the following: i)to estimate an optimal block size for use in the sample andaggregate framework, ii) to identify the minimum privacybudget needed to estimate the final result within a givenaccuracy bound, and iii) to appropriately distribute the pri-vacy budget ε across various tasks and queries.

For simplicity of exposition, the particular aging model inour analysis is that a constant fraction of the dataset hascompletely aged out, i.e. the privacy of the entries in thisconstant fraction is no more of a concern. In reality, if theaged data is still weakly privacy sensitive, then it is possibleto privately estimate these parameters by introducing anappropriate magnitude of noise into these calculations. Theweak privacy of aged data allows us to keep the noise lowenough such that the estimated parameters are still useful.Existing techniques [1, 26] have attempted to use progressiveaggregation of old data in order to reduce its sensitivity. Theuse of differentially private operations for aggregation canpotentially be exploited to generate our training datasets.The use of these complementary approaches offer excitingopportunities that have not been explored in this paper.

It is important to mention that GUPT does not requirethe aging model for default functionality. The default pa-rameter choices allow it work well in a generic setting. How-ever, our experiments show that the aging model providesan additional improvement in performance.

4. ACCURACY IMPROVEMENTSThe sample and aggregate framework (Algorithm 1) in-

troduces two sources of error:• Estimation error: This arises because the query is eval-

uated on a smaller data block, rather than the en-tire dataset. Typically, the larger the block size, thesmaller the estimation error.• Noise: Another source of error is due to the Laplace

noise introduced to guarantee differential privacy.

352

Intuitively, the larger the number of blocks, the lowerthe sensitivity of the aggregation function – since theaggregation function has sensitivity s

`, where s denotes

the output range of each block, and ` denotes the num-ber of blocks. As a result, given a fixed privacy param-eter ε, with a larger number of blocks, the magnitudeof the Laplace noise is lowered.

GUPT uses two strategies (resampling and selecting theoptimal block size) to reduce these types of errors. Beforedelving into details of the techniques, the following sectionexplains how the output range for a given analysis programis computed. This range is used to decide the amount ofnoise to be added to the final output.

4.1 Output Range EstimationThe sample and aggregate framework described in Algo-

rithm 1 does not describe a mechanism to obtain the rangewithin which the output can lie. This is needed to esti-mate the noise that should be added for differential privacy.GUPT implements this requirement by providing the fol-lowing mechanisms:

1. GUPT-tight: The analyst specifies a tight range for theoutput.

2. GUPT-loose: The analyst only provides a loose rangefor the output. In this case, the computation is run oneach data block and their outputs are recorded. A dif-ferentially private percentile estimation algorithm [24]is then applied on the set of outputs to privately com-pute the 25-th and the 75-th percentile values. Thesevalues are used as the range of the output and aresupplied to Algorithm 1.

3. GUPT-helper: The analyst could also provide a rangetranslation function. If either (a) input range is notpresent or (b) only very loose range for the input (e.g.,using the national GDP as an upper-bound on annualhousehold income) is available, then GUPT runs thesame differentially private percentile estimation algo-rithm on the inputs to privately compute the 25-th andthe 75-th percentile (a.k.a, lower and upper quartiles)of the inputs. This is used as a tight approximationof the input range. The analyst-supplied range trans-lation function is then invoked to convert the “tight”input range into an estimate of the output range.

Our experiments demonstrate that one can get good re-sults for a large class of problems using the noisy lower andupper quartiles as approximations of the output range. If theinput dataset is multi-dimensional, the range estimation al-gorithm is run independently for each dimension. Note thatthe choice of 25-th and 75-th percentile above is somewhatarbitrary. In fact, one can choose a larger inter-percentilerange (e.g., 10-th and 90-th percentile) if there are moredata samples. However, this does not affect the asymptoticbehavior of the algorithm.

4.2 ResamplingThe variance in the final output is due to two sources

of randomness: i) partitioning the dataset into blocks andii) the Laplace noise added. The following resampling tech-nique3 can be used to reduce the variance due to partitioningthe dataset into blocks. Instead of requiring each data entryto reside in exactly one block (as described in the originalsample and aggregate framework [24]), each data entry can

3A variant of this technique was suggested by Adam Smith.

now reside in multiple blocks. The resampling factor γ de-notes the number of blocks in which each data entry resides.

If the number of records in the dataset is n, block size is βand each data entry resides in γ blocks, it is easy to see thatthe number of blocks ` = γn/β. To incorporate resampling,we make the following modifications to Algorithm 1. Lines1 and 2 are modified as follows. Consider ` = γn/β bins ofsize β each. The ith entry from the dataset T is picked andrandomly placed into γ bins that are not full. This processis performed for all the entries in the dataset T . In Line 8,

the Laplace noise is changed to Lap(β|max−min |n·ε ). The rest

of Algorithm 1 is left intact.The main benefit of using resampling is that it reduces the

variance due to partitioning the dataset into blocks withoutincreasing the noise needed for the same level of privacy.

Claim 1. With the same privacy level ε, resampling withany γ ∈ Z+, does not increase the Laplace noise being added(for fixed block size β).

Proof. Since each record appears in γ blocks, a Laplacenoise of magnitude O( γs

ε`) = O( sβ

εn) should be added to pre-

serve ε-differential privacy. This means that once the blocksize is fixed, the noise is independent of the factor γ.

Intuitively, the benefit from resampling is that the vari-ance due to partitioning of the dataset into blocks is reducedwithout increasing the Laplace noise (added in Step 8 of Al-gorithm 1) needed for the same level of privacy with theinclusion of γ > 1. Consider the following example to get abetter understanding of the intuition.

Example 2. Let T be a dataset (with n records) of theages of a population and max be the maximum age in thedataset. The objective is to find the average age (Av) in

this dataset. Let Av be the average age of a dataset formedwith n0.6 uniformly random samples drawn from T with re-placement. The expectation of Av equals the true averageAv. However, the variance of Av will not be zero (unless all

the entries in T are same). Let O = 1ψ

∑ψi=1 Av(i), where

Av(i) is the i-th independent computation of Av mentionedabove and ψ is some constant. Notice that O has the sameexpected value as Av but the variance has reduced by a fac-tor of ψ. Hence, resampling reduces the variance in the finaloutput O without introducing bias.

The above example is a simplified version of the actualresampling process. In the actual resampling process eachdata block of size n0.6 is allowed to have only one copy ofeach data entry of T . However, even with an inaccuraterepresentation, the above example captures the essence ofthe underlying phenomenon.

In practice, the resampling factor γ is picked such that it isreasonably large, without increasing the computation over-head significantly. Notice that the increase in accuracy withthe increase of γ becomes insignificant beyond a threshold.

4.3 Selecting the Optimal Block SizeIn this section, we address the following question: Given

a fixed privacy budget, how do we pick the optimal block sizeto maximize the accuracy of the private output?

Observe that increasing the block size β increases the noisemagnitude, but reduces the estimation error. Therefore, thequestion boils down to: how do we select an optimal block

353

size that will allow us to balance the estimation error andthe noise? The following example elucidates why answeringthe above question is important.

Example 3. Consider the same age dataset T used inExample 2. If our goal is to find the average of the entriesin T while preserving privacy, then it can be observed that(ignoring resampling) the optimal size of each block is onewhich attains the optimal balance between the estimation er-ror and noise. If the block size was one, then the expectederror will be O(1/n), where n is the size of the dataset. How-ever, if we use the default block size ( i.e., n0.6), the expectederror will be O(1/n0.4) which is much higher.

As a result getting the optimal block size based on thespecific task helps to reduce the final error to a large extent.The optimal block size varies from problem to problem. Forexample, in k-means clustering or logistic regression the op-timal block size has to be much larger than one.

Let ` = nα be the optimal number of blocks, where α isa parameter to be ascertained. Hence, n1−α is the blocksize. (For the simplicity of exposition we do not considerresampling.) Let f : Rk×n → R be the query which isto be computed on the dataset T . Let the data blocks berepresented as T1, · · · , T`. Let s be the sensitivity of thequery, i.e., the absolute value of maximum change that anyf(Ti) can have if any one entry of T is changed. With theabove parameters in place, the ε-differentially private outputfrom the sample and aggregate framework is

f(T ) =1

nα

nα∑i=1

f(Ti) + Lap(s

εnα) (1)

Assume that the entries of the dataset T are drawn i.i.d,and that there exists a dataset T np (with entries drawn i.i.d.from the same distribution as T ) whose privacy we do notcare for under the aging of sensitivity model. Let nnp be thenumber of entries in T np. We will use the aged dataset T np

to estimate the optimal block size. Specifically, we partitionT np into blocks of size β = n1−α. The number of blocks`np in T np is therefore `np =

nnp

n1−α . Notice that `np is afunction of α, whose optimal value has to be found. Alsonote that the minimum value of α must satisfy the followinginequality: nnp ≥ n1−α.

One possible approach for achieving a good value of α isby minimizing the empirical error in the final output. The

empirical error in the output of f is defined as∣∣∣∣∣∣ 1

`np

`np∑i=1

f(Tinp)− f(Tnp)

∣∣∣∣∣∣︸ ︷︷ ︸A

+

√2s

εnα︸ ︷︷ ︸B

(2)

Here A characterizes the estimation error, and B is due tothe Laplace noise added. We can minimize Equation 2 w.r.t.α when α ∈ [1− lognnp/ logn, 1]. Conventional techniqueslike hill climbing can be used to obtain a local minima.

The α computed above is used to obtain the optimal num-ber of blocks. Since, the computation involves only the non-private database T np, there is no effect on overall privacy.

5. PRIVACY BUDGET MANAGEMENTIn differential privacy, the analyst is expected to specify

the privacy goals in terms of an abstract privacy budgetε. The analyst performs the data analysis task optimizing

it for accuracy goals and the availability of computationalresources. These metrics do not directly map onto the ab-stract privacy budget. It should be noted that even a privacyexpert might be unable to map the privacy budget into accu-racy goals for arbitrary problems. In this section we describemechanisms that GUPT use to convert the accuracy goalsinto a privacy budget and to efficiently distribute a givenprivacy budget across different analysis tasks.

5.1 Estimating Privacy Budget for AccuracyGoals

In this section we seek to answer the question: How canGUPT pick an appropriate ε, given a fixed accuracy goal?Specifically, we wish to minimize the ε parameter to maxi-mally preserve the privacy budget. It is often more intuitiveto specify an accuracy goal rather than a privacy parameterε, since accuracy relates to the problem at hand.

Similar to the previous section, we assume the existenceof an aged dataset T np (drawn from the same distributionas the original dataset T ) whose privacy is not a concern.

Consider an analyst who wishes to guarantee an accuracyρ with probability 1 − δ, i.e. the output should be withina factor ρ of the true value. We wish to estimate an ap-propriate ε from an aged data set T np of size nnp. Let βdenote the desired block size. To estimate ε, first the per-missible standard deviation in the output σ is calculatedfor a specified accuracy goal ρ and then the following opti-mization problem is solved. Solve for ε, under the followingconstraints: 1) the expression in Equation 3 equals σ2, 2)α = max{0, log(n/β)}.

1

nα

1

`np

`np∑i=1

f(Tnpi )−

1

`np

`np∑i=1

f(Tnpi )

2︸ ︷︷ ︸

C

+2s2

ε2n2α︸ ︷︷ ︸D

(3)

In Equation 3, C denotes the variance in the estimationerror and D denotes the variance in the output due to noise.

To calculate σ from the accuracy goal, we can rely onChebyshev’s inequality: Pr[|f(T ) − E(f(Ti))| > φσ] < 1

φ2 .

Furthermore, assuming that the query f is a approximatelynormal statistic, we have |E(f(Ti)) − Truth| = O (1/β).

Therefore: Pr[|f(T ) − Truth| > φσ + O (1/β)] < (1/φ2) Tomeet the output accuracy goal of ρ with probability 1 − δ,we set σ '

√δ|1 − ρ|f(T np). Here, we have assumed that

the true answer is f(T np) and 1/β � σ/√δ.

Since in the above calculations we assumed that the trueanswer is f(T np), an obvious question is “why not outputf(T np) as the answer?”. It can be shown that in a lot ofcases, the private output will be much better than f(T np).

If the assumption that 1/β � σ/√δ does not hold, then

the above technique for selecting privacy budget would pro-duce suboptimal results. This however does not compromisethe privacy properties that GUPT wants to maintain, as itexplicitly limit the total privacy budget allocated for queriesaccessing a particular dataset.

5.2 Automatic Privacy Budget DistributionDifferential privacy is an alien concept for most analysts.

Further, the proper distribution of the limited privacy bud-get across multiple computations require significant mathe-matical expertise. GUPT eliminates the need to manuallydistribute privacy budget between tasks. The following ex-ample will highlight the requirement of an efficient privacy

354

budget distribution rather than distributing equally amongvarious tasks.

Example 4. Consider the same age census dataset T fromExample 2. Suppose we want to find the average age andthe variance present in the dataset while preserving differ-ential privacy. Assume that the maximum possible humanage is max and the minimum age is zero. Assume that thenon-private variance is computed as 1

n

∑ni=1(T (i)−Avpriv)2,

where Avpriv is the private estimate of the average and n isthe size of T . If an entry of T is modified, the average Avchanges by at most max/n, however the variance can changeby at most max2/n.

Let ε1 and ε2 be the privacy level expected for average andvariance respectively, with the total privacy budget being ε =ε1 + ε2. Now, if it is assumed that ε1 = ε2, then the errorin the computation of variance will be in the order of maxmore than in the computation of average. Whereas if privacybudget were distributed as ε1 : ε2 = 1 : max, then the noisein both the average and variance will roughly be the same.

Given privacy budget of ε and we need to use it for com-puting various queries f1, · · · , fm privately. If the privateestimation of query fi requires εi privacy budget, then thetotal privacy budget spent will be

∑mi=1 εi (by composition

property of differential privacy [5]). The privacy budget is

distributed as follows. Let ζiεi

be the standard deviation of

the Laplace noise added by GUPT to ensure privacy level εi.Allocate the privacy budget by setting εi = ζi∑m

i=1 ζiε. The

rationale behind taking such an approach is that usually thevariance in the computation by GUPT is mostly due to thevariance in the Laplace noise added. Hence, distributing εacross various tasks using the technique discussed above en-sures that the variance due to Laplace noise in the privateoutput for each fi is the same.

6. SYSTEM SECURITYGUPT is designed as a hosted platform where the an-

alyst is not trusted. It is thus important to ensure thatthe untrusted computation should not be able to access thedatasets directly. Additionally, it is important to prevent thecomputation from exhausting resources or compromising theservice provider. To this end, the “computation manager”is split into a server component that interacts with the userand a client component that runs on each node in the clus-ter. The trusted client is responsible for instantiating thecomputation in an isolated execution environment. The iso-lated environment ensures that the computation can onlycommunicate with a trusted forwarding agent which sendsthe messages to the computation manager.

6.1 Access ControlGUPT uses a mandatory access control framework (MAC)

to ensure that (a) communication between different instancesof the computation is disallowed and (b) each instance of thecomputation can only store state (or modify data) within itsown scratch space. This is the only component of GUPTthat depends upon a platform dependent implementation.On Linux, the LSM framework [27] has enabled many MACframeworks such as SELinux and AppArmor to be built.GUPT defines a simple AppArmor policy for each instanceof the computation, setting its working directory to a tem-porary scratch space that is emptied upon program termi-nation. AppArmor does not yet allow fine grained control to

limit network activity to individual hosts and ports. Thusthe “computation manager” is split into a server and clientcomponent. The client component of the computation man-ager allows GUPT to disable all network activity for theuntrusted computation and restrict IPC to the client.

We determined an empirical estimate of the overhead in-troduced by the AppArmor sandbox by executing an imple-mentation of k-means clustering on GUPT 6, 000 times. Wefound that the sandboxed version of GUPT was only 0.012times slower than the non-sandboxed version (overhead of1.26%).

6.2 Protection against side-channel attacksHaeberlen et al. [10] identified three possible side-channel

attacks against differentially private systems. They are i)state attack, ii) privacy budget attack, and iii) timing attack.GUPT is not vulnerable to any of these attacks.

State attacks: If the adversarial program can modify someinternal state (e.g., change the value of a static variable)when encountered with a specific data record. An adver-sary can then look at the state to figure out whether therecord was present in the dataset. Both PINQ (in it’s cur-rent implementation) and Airavat are vulnerable to stateattacks. However, it is conceivable that operations can beisolated using .NET AppDomains in PINQ to isolate datacomputations. Since GUPT executes the complete analysisprogram (which may be adversarial) in isolated executionchambers and allows the analyst to access only the final dif-ferentially private output, state attacks are automaticallyprotected against.

Privacy budget attack: In this attack, on encountering aparticular record, the adversarial program issues additionalqueries that exhausts the remaining privacy budget. [10]noted that PINQ is vulnerable to this attack. GUPT pro-tects against privacy budget attacks by managing the pri-vacy budget itself, instead of letting the untrusted programperform the budget management.

Timing attacks: In a timing attack, the adversarial pro-gram could consume an unreasonably long amount of timeto execute (perhaps get into an infinite loop) when encoun-tered with a specific data record. GUPT protects againstthis attack by setting a predefined bound on the number ofcycles for which the data analyst program runs on each datablock. If the computation on a particular data block com-pletes before the predefined number of cycles, then GUPTwaits for the remaining cycles before producing an outputfrom that block. In case the computation exceeds the prede-fined number of cycles, the computation is killed and a con-stant value within the expected output range is produced asthe output of the program running on the data block underconsideration.

Note that with the scheme above, the runtime of GUPTis independent of the data. Hence, the number of executioncycles does not reveal any information about the dataset.The proof that the final output is still differentially privateunder this scheme follows directly from the privacy guar-antee of the sample and aggregate framework and the factthat a change in one data entry can affect only one datablock (ignoring resampling). Thus GUPT is not vulnerableto timing attacks. Both PINQ and Airavat do not protectagainst timing attacks [10].

355

7. EVALUATIONFor each data analysis program, the program binary and

interfaces with the GUPT “computation manager” shouldbe provided. For arbitrary binaries, a lean wrapper programcan be used for marshaling data to/from the format of thecomputation manager.

In this section, we show using results from running com-mon machine learning algorithms (such as k-means cluster-ing and logistic regression on a life sciences dataset) thatGUPT does not significantly affect the accuracy of dataanalysis. Further, we show that GUPT not only relieves theanalysts from the burden of distributing a privacy budgetbetween data transformation operations, it also manages toprovide superior output accuracy. Finally, we show throughbenchmarks the scalability of the GUPT architecture andthe benefits of using aged data to estimate optimal values ofprivacy budget and block sizes.

7.1 Case Study: Life Sciences datasetWe evaluate the efficacy of GUPT using the ds1.10 life

sciences dataset taken from http://komarix.org/ac/ds as amotivating example for data analysis. This dataset containsthe top 10 principal components of chemical/biological com-pounds with each of the 26, 733 rows representing differentcompounds. Additionally, the reactivity of the compound isavailable as an additional component. A k-means clusteringexperiment enables us to cluster compounds with similar fea-tures together and logistic regression builds a linear classifierfor the experiment (e.g., predicting carcinogens). It shouldbe noted that these experiments only provide estimates asthe final answer, e.g., the cluster centroids in the case ofk-means. We show in this section that the perturbation in-troduced by GUPT only affects the final result marginally.

7.1.1 Output Accuracy

2 4 6 8 10Privacy Budget (ε)

0.5

0.6

0.7

0.8

0.9

1.0

Acc

urac

y

GUPT-tight

Non private baseline

Figure 3: Effect of privacy budget on the accuracyof prediction using Logistic Regression on the lifesciences dataset

As mentioned in Section 4, any analysis performed usingGUPT has two sources of error – (a) an estimation error,introduced because each instance of the computation workson a smaller subset of the data and (b) Laplace noise thatis added in order to guarantee differential privacy. In thissection, we show the effect of these errors when runninglogistic regression and k-means on the life sciences dataset.

GUPT can be used to run existing programs with no mod-ifications, thus drastically reducing the overhead of writingprivacy preserving programs. Analysts using GUPT are freeto use their favorite software packages written in any lan-guage. To demonstrate this property, we evaluate black box

0.4 0.5 0.6 0.7 0.8 0.9 1.0 2.0 3.0 4.0

Privacy Budget (ε)

0

20

40

60

80

100

Nor

mal

ized

Intra

Clu

ster

Varia

nce

Baseline ICVGUPT-loose

GUPT-tight

Figure 4: Intra-cluster variance for k-means cluster-ing on the life sciences dataset

implementations of logistic regression and k-means cluster-ing on the life sciences dataset.

Logistic Regression: The logistic regression software pack-age from Microsoft Research (Orthant-Wise Limited-memoryQuasi-Newton Optimizer for L1-regularized Objectives ) wasused to classify the compounds in the dataset as carcinogensand non-carcinogens. Figure 3 shows the accuracy of GUPTfor different privacy budgets.

When the package was run on the dataset directly, a base-line accuracy of 94% was obtained. The same package whenrun using the GUPT framework classified carcinogens withan accuracy between 75 ∼ 80%. To understand the sourceof the error, when the non-private algorithm was executedon a data block of size n

n0.4 records, the accuracy reducedto 82%. It was thus determined that much of the errorstems from the loss of accuracy when the algorithm is runon smaller blocks of the entire dataset reduced. For datasetsof increasingly large size, this error is expected to diminish.

k-means Clustering: Figure 4 shows the cluster variancecomputed from a k-means implementation run on the lifesciences dataset. The x-axis is various choices of the pri-vacy budget ε, and the y-axis is the normalized Intra-ClusterVariance (ICV) defined as 1

n

∑Ki=1

∑~x∈Ci |~x−~ci|

22, where K

denotes the number of clusters, Ci denotes the set of pointswithin the ith cluster, and ~ci denotes the center of the ith

cluster. A standard k-means implementation from the scipypython package is used for the experiment.

The k-means implementation was run using GUPT withdifferent configurations for calculating the output range (Sec-tion 4.1). For GUPT-tight, a tight range for the output istaken to be the exact minimum and the maximum of eachattribute (for all 10 attributes). For GUPT-loose, a looseoutput range is fixed as [min ∗2,max ∗2], where min andmax are the actual minimum and maximum for that at-tribute. Figure 4 shows that with increasing privacy budgetε, the amount of Laplace noise added to guarantee differ-ential privacy decreases, thereby reducing the intra-clustervariance, i.e. making the answer more accurate. It can alsobe seen that when GUPT is provided with reasonably tightbounds on the output range (GUPT-tight), the output of thek-means experiment is very close to a non-private run of theexperiment even for small values of the privacy budget. Ifonly loose bounds are available (GUPT-loose), then a largerprivacy budget is required for the same output accuracy.

7.1.2 Budget Distribution between OperationsIn GUPT, the program is treated as a black box and

noise is only added to the output of the entire program.

356

20 80 200k-means iteration count

0

20

40

60

80

100

120

Nor

mal

ized

Intra

Clu

ster

Varia

nce

PINQ-tight ε=2PINQ-tight ε=4GUPT-tight ε=1GUPT-tight ε=2

Figure 5: Total perturbation introduced by GUPTdoes not change with number of operations in theutility function

Thus the number of operations performed in the programitself is irrelevant. A problem with writing specialized dif-ferentially private algorithms such as in the case of PINQ isthat given a privacy budget ε for the task, it is difficult todecide how much ε to spend on each query, since it is diffi-cult to determine the number of iterations needed ahead oftime. PINQ requires the analyst to pre-specify the numberof iterations in order to allocate the privacy budget betweeniterations. This is often hard to do, since many data analy-sis algorithms such as PageRank [20] and recursive relationqueries [2] require iterative computation until the algorithmreaches convergence. The performance of PINQ thus de-pends on the ability to accurately predict the number ofiterations. If the specified number of iterations is too small,then the algorithm may not converge. On the other hand,if the specified number of iterations is too large, then muchmore noise than is required will be added which will bothslow down the convergence of the algorithm as well as harmits accuracy. Figure 5 shows the effect of PINQ on accu-racy when performing k-means clustering on the dataset.In this example, the program output for the dataset con-verges within a small number of iterations, e.g., n = 20.Whereas if a larger number of iterations (e.g., n = 200) wasconservatively chosen, then PINQ’s performance degradessignificantly. On the other hand, GUPT produces the sameamount of perturbation irrespective of the number of iter-ations in k-means. Further, it should be noted that PINQwas subjected to a weaker privacy constraint (ε = 2 and 4)as compared to GUPT (ε = 1 and 2).

7.1.3 Scalability

20 80 100 200Iterations

0.0

0.5

1.0

1.5

2.0

2.5

Tim

e(s

econ

ds)

Non PrivateGUPT-helper

GUPT-loose

Figure 6: Change in computation time for increasednumber of iterations in k-means

Using a server with two Intel Xeon 5550 quad-core CPUsand the entire dataset loaded in memory, we compare the

0 20 40 60 80 100Portion of queries (%)

80

85

90

95

100

Res

ulta

ccur

acy

(%)

GUPT-helper constant ε=1GUPT-helper constant ε=0.3GUPT-helper variable εExpected Accuracy

Figure 7: CDF of query accuracy for privacy budgetallocation mechanisms

execution time of an unmodified (non-private) instance anda GUPT instance of the k-means experiment.

If tight output range (i.e. , GUPT-tight) is not available,typically, the output range estimation phase of the sampleand aggregate framework takes up most of the CPU cy-cles. When only loose range for the input is available (i.e. ,GUPT-helper), a differentially private percentile estimationis performed on all of the input data. This is a O(n lnn)operation, n being the number of data records in the origi-nal dataset. On the other hand, if even loose range for theoutput is available (i.e. , GUPT-loose), then the percentileestimation is performed only on the output of each of theblocks in sample and aggregate framework, which is typicallyaround n0.4. This results in significantly reduced run-timeoverhead. The overhead introduced by GUPT is irrespec-tive of the actual computation time itself. Thus as the com-putation time increases, the overhead introduced by GUPTdiminishes in comparison. Further, there is an additionalspeed up since each of the computation instances work on asmaller subset of the entire dataset. It should be noted thatthe reduction in computation time thus achieved could alsopotentially be achieved by the computational task runningwithout GUPT. Figure 6 shows that the overall completiontime of the private versions of the program increases slowlycompared to the non-private version as we increase the num-ber of iterations of k-means clustering.

7.2 Using Aged DataGUPT uses an aged dataset (that is no longer considered

privacy sensitive) drawn from a similar distribution as thereal dataset. Section 4.3 describes the use of aged data toestimate an optimal block size that reduces the error in-troduced by data sampling. Section 5.1 describes how dataanalysts who are not privacy experts can continue to only de-scribe their accuracy goals yet achieve differentially privateoutputs. Finally, Section 5.2 uses aged data to automati-cally distribute a privacy budget between different querieson the same data set. In this section, we show experimentalresults that support the claims made in Sections 4.3 and 5.1.

7.2.1 Privacy Budget EstimationTo illustrate the ease with which GUPT can be used by

data analysts, we evaluate the efficiency of GUPT by ex-ecuting queries that are not provided with a privacy bud-get. We use a census income dataset from the UCI machinelearning repository [7] which consists of 32561 entries. Theage data from this dataset is used to calculate the aver-age age. A reasonably loose range of [0, 150] was enforced

357

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Nor

mal

ized

priv

acy

budg

etlif

etim

e

GUPT-helper constant ε=1GUPT-helper variable εGUPT-helper constant ε=0.3

Figure 8: Increased lifetime of total privacy budgetusing privacy budget allocation mechanism

0 10 20 30 40 50 60 70Block size (β)

0.1

0.2

0.3

0.4

0.5

Nor

mal

ized

RM

SE

Median ε=2Median ε=6Mean ε=2Mean ε=6

Figure 9: Change in error for different block sizes

on the output whose true average age is 38.5816. Initially,the experiment was run with a constant privacy budgets ofε = 1 and ε = 0.3. GUPT allows the analyst to providelooser constraints such as “90% result accuracy for 90% ofthe results” and allocates only as much privacy budget as isrequired to meet these properties. In this experiment, the10% of the dataset was assumed to be completely privacy in-sensitive and was used to estimate ε given a pre-determinedblock size. Figure 7 shows the CDF of the output accuracyboth for constant privacy budget values as well as for the ac-curacy requirement. Interestingly, not only does the figureshow that the accuracy guarantees are met by GUPT, butalso it shows that if the analyst was to define the privacybudget manually (as in the case of ε = 1 or ε = 0.3), theneither too much or too little privacy budget is used. Theprivacy budget estimation technique thus has the additionaladvantage that the lifetime of the total privacy budget for adataset will be extended. Figure 8 shows that if we were torun the average age query with the above constraints overand over again, GUPT will be able to run 2.3 times morequeries than using a constant privacy budget of ε = 1.

7.2.2 Optimal Block Size EstimationSection 4.3 shows that the estimation error decreases with

an increase in data block size, whereas the noise decreaseswith an increased number of blocks. The optimal trade offpoint between the block size and number of data blockswould be different for different queries executed on the dataset.To illustrate the tradeoff, we show results from queries ex-ecuted on an internet advertisement dataset also from theUCI machine learning repository [7]. Figure 9 shows thenormalized root mean square error (from the true value) inestimating the mean and median aspect ratio of advertise-ments shown on Internet pages with privacy budgets ε of 2and 6. In the case of the “mean” query, since the averagingoperation is already performed by the sample and aggre-gate framework, smaller data blocks would reduce the noise

GUPT PINQ Airavat

Works with unmodified Yes No NoprogramsAllows expressive programs Yes Yes NoAutomated privacy budget Yes No NoallocationProtection against privacy Yes No Yesbudget attackProtection against state Yes No NoattackProtection against timing Yes No Noattack

Table 1: Comparison of GUPT, PINQ and Airavat

added to the output and thus provide more accurate results.As expected, we see that the ideal block size would be one.

For the “median” query, it is expected that increasing theblock size would generate more accurate inputs to the av-eraging function. Figure 9 shows that when the “median”query is executed with ε = 2, the error is minimal for ablock size of 10. With increasing block sizes, the noise addedto compensate for the reduction in number of blocks wouldhave a dominating effect. On the other hand, when execut-ing the same query with ε = 6, the error continues to dropfor increased block sizes, as the estimation error dominatesthe Laplace noise (owing to the increased privacy budget).It is thus clear that GUPT can significantly reduce the to-tal error by estimating the optimal block size for the sampleand aggregate framework.

7.3 Qualitative AnalysisIn this section, GUPT is contrasted with both PINQ and

Airavat on various fronts (see Table 1 for a summary). Wealso list the significant changes introduced by GUPT in or-der to mold the sample and aggregate framework (SAF) [24]into a practically useful one.

Unmodified programs: Because PINQ [16] is an API thatprovides a set of low-level data manipulation primitives, ap-plications will need to be re-written to perform all operationsusing these primitives. On the other hand, Airavat [22] im-plements the Map-Reduce programming paradigm [4] andrequires that the analyst splits the user’s data analysis pro-gram into an “untrusted” map program and a reduce aggre-gator that is “trusted” to be differentially private.

In contrast, GUPT treats the complete application pro-gram as a black box and as a result the entire applicationprogram is deemed untrusted.

Expressiveness of the program: PINQ provides a lim-ited set of primitives for data operations. However, if therequired primitives are not already available, then a privacyunaware analyst would be unable to ensure privacy for theoutput. Airavat also severely restricts the expressiveness ofthe programs that can run in it’s framework: a) the “un-trusted” map program is completely isolated for each dataelement and cannot save any global state and b) it restrictsthe number of key-value pairs generated from the mapper.Many machine learning algorithms (such as clustering andclassification) require global state and complex aggregationfunctions. This would be infeasible in Airavat without plac-ing much of the logic in the “trusted” reducer program.

GUPT places no restriction on the application program,and thus does not degrade the expressiveness of the program.

Privacy budget distribution: As was shown in Section

358

7.1.2, PINQ requires the analyst to allocate a privacy budgetfor each operation on the data. An inefficient distribution ofthe budget either add too much noise or use up too much ofthe budget. Like GUPT, Airavat spends a constant privacybudget on the entire program. However, neither Airavat norPINQ provides any support for distributing an aggregateprivacy budget across multiple data analysis programs.

Using the aging of sensitivity model, GUPT provides anmechanism for efficiently distributing an aggregate privacybudget across multiple data analysis programs.

Side Channel Attacks: Current implementation of PINQlays the onus of protecting against side channel attacks onthe program developer. As was noted in [10], although Aira-vat protects against privacy budget attacks, it remains vul-nerable to state attacks. GUPT defends against both stateattacks and privacy budget attacks (see Section 6.2).

Other differences: GUPT extends SAF to use a noveldata resampling mechanism to reduce the variance in theoutput induced via data sub-sampling. Using the aging ofsensitivity model, GUPT overcomes a fundamental issue indifferential privacy not considered previously (to the best ofour knowledge): for an arbitrary data analysis application,how do we describe an abstract privacy budget in terms ofutility? The model also allows us to further reduce the errorin SAF by estimating a reasonable block size.

8. DISCUSSIONApproximately normal computation: GUPT does notprovide any theoretical guarantees on the accuracy of theoutputs, if the applied computation is not approximatelynormal, or if the data entries are not independently andidentically distributed (i.i.d). However, GUPT still guaran-tees that the differential privacy of individual data recordsis always maintained. In practice, GUPT provides reason-ably accurate results for a lot of queries that do not satisfyapproximate normality.

For some real world datasets, such as sensing and stream-ing data that have temporal correlation and do not satisfythe i.i.d property, the current implementation of GUPT failsto provide reasonable output. In our future work we intendto design GUPT for catering to these kind of datasets.

Ordering of multiple outputs: When data analysis pro-gram yields multi-dimensional outputs, the program maynot be able to guarantee the same ordering of outputs fordifferent data blocks. In this case, we have to sort the out-puts according to some canonical form before applying theaveraging operation. For example, in the case of k-meansclustering, each block can yield different ordering of the kcluster centers. In our experiments, the cluster centers weresorted according to their first coordinate.

8.1 LimitationsForeknowledge of output dimension: GUPT assumesthat the output dimensions are known in advance. This mayhowever not always be true in practice. For example, Sup-port Vector Machines (SVM) output an indefinite numberof support vectors. Unless the dimension is fixed in advance,private information may leak through the dimensionality it-self. Another problem is that if the output dimensions fordifferent blocks are different, then performing an average ofthe outputs will not be meaningful. A partial solution is to

clamp or pad the output dimension to a predefined number.We note that this limitation is not unique to GUPT. Forexample, Airavat [22] also suffers from the same problem inthe sense that their mappers have to output a fixed numberof (key, value) pairs.

Inherits limitations of differential privacy: GUPT in-herits some of the issues that are common to many differ-ential privacy mechanisms. For high dimensional data out-puts, the privacy budget needs to be split across the multipleoutputs. The privacy budget also needs to be split acrossmultiple data mining tasks and users. Given a fixed totalprivacy budget, the more tasks and queries we divide thebudget over, the more noise there is – and at some point, themagnitude of noise may cause the data analysis to becomeunusable. In Section 5.2, we outlined a potential method formanaging privacy budgets. Alternative approaches such asauctioning of privacy budget [9] can also be used.

While differential privacy guarantees the privacy of indi-vidual records in the dataset, many real world applicationwill want to enforce higher level privacy concepts such asuser-level privacy [6]. In cases where multiple records maycontain information about the same user, user-level privacyneeds to be accounted for accordingly.

9. CONCLUSION AND FUTURE WORKGUPT makes privacy-preserving data analytics easy for

privacy non-experts. The analyst can upload arbitrary datamining programs and GUPT guarantees the privacy of theoutputs. We propose novel improvements to the sample andaggregate framework to enhance the usability and accuracyof the data analysis. Through a new model that reduces thesensitivity of the data over time, GUPT is able to representprivacy budgets as accuracy guarantees on the final output.Although this model is not strictly required for the defaultfunctioning of GUPT, it improves both usability and accu-racy. Through the efficient distribution of the limited pri-vacy budget between data queries, GUPT is able to ensurethat more queries that meet both their accuracy and pri-vacy goals can be executed on the dataset. Through exper-iments on real-world datasets, we demonstrate that GUPTcan achieve reasonable accuracy in private data analysis.

AcknowledgementsWe would like to thank Adam Smith, Daniel Kifer, FrankMcSherry, Ganesh Ananthanarayanan, David Zats, PiyushSrivastava and the anonymous reviewers of SIGMOD fortheir insightful comments that have helped improve thispaper. This project was funded by Nokia, Siemens, Intelthrough the ISTC for Secure Computing, NSF awards (CPS-0932209, CPS-0931843, BCS-0941553 and CCF-0747294) andthe Air Force Office of Scientific Research under MURI Grants(22178970-4170 and FA9550-08-1-0352).

10. REFERENCES[1] N. Anciaux, L. Bouganim, H. H. van, P. Pucheral, and

P. M. Apers. Data degradation: Making private dataless sensitive over time. In CIKM, 2008.

[2] F. Bancilhon and R. Ramakrishnan. An amateur’sintroduction to recursive query processing strategies.In SIGMOD, 1986.

359

[3] M. Barbaro and T. Zeller. A face is exposed for aolsearcher no. 4417749. The New York Times, Aug.2006.

[4] J. Dean and S. Ghemawat. Mapreduce: Simplifieddata processing on large clusters. Operating SystemsDesign and Implementation, October 2004.

[5] C. Dwork, F. McSherry, K. Nissim, and A. Smith.Calibrating noise to sensitivity in private dataanalysis. In TCC, 2006.

[6] C. Dwork, M. Naor, T. Pitassi, and G. N. Rothblum.Differential privacy under continual observation. InSTOC, 2010.

[7] A. Frank and A. Asuncion. UCI machine learningrepository, 2010.

[8] S. R. Ganta, S. P. Kasiviswanathan, and A. Smith.Composition attacks and auxiliary information in dataprivacy. In KDD, 2008.

[9] A. Ghosh and A. Roth. Selling privacy at auction.http://arxiv.org/abs/1011.1375.

[10] A. Haeberlen, B. C. Pierce, and A. Narayan.Differential privacy under fire. In USENIX Security,2011.

[11] M. Hay, V. Rastogi, G. Miklau, and D. Suciu.Boosting the accuracy of differentially privatehistograms through consistency. Proc. VLDB Endow.,3:1021–1032, September 2010.

[12] V. Karwa, S. Raskhodnikova, A. Smith, andG. Yaroslavtsev. Private analysis of graph structure.In VLDB, 2011.

[13] D. Kifer. Attacks on privacy and definetti’s theorem.In SIGMOD, 2009.

[14] C. Li, M. Hay, V. Rastogi, G. Miklau, andA. McGregor. Optimizing linear counting queriesunder differential privacy. In PODS, 2010.

[15] A. Machanavajjhala, J. Gehrke, D. Kifer, andM. Venkitasubramaniam. l-diversity: Privacy beyondk-anonymity. In ICDE, 2006.

[16] F. McSherry. Privacy integrated queries: an extensibleplatform for privacy-preserving data analysis. InSIGMOD, 2009.

[17] F. McSherry and K. Talwar. Mechanism design viadifferential privacy. In FOCS, 2007.

[18] A. Narayanan and V. Shmatikov. Robustde-anonymization of large sparse datasets. In IEEESymposium on Security and Privacy, 2008.

[19] K. Nissim, S. Raskhodnikova, and A. Smith. Smoothsensitivity and sampling in private data analysis. InSTOC, 2007.

[20] L. Page, S. Brin, R. Motwani, and T. Winograd. Thepagerank citation ranking: Bringing order to the web.Technical Report 1999-66, Stanford InfoLab, 1999.

[21] V. Rastogi and S. Nath. Differentially privateaggregation of distributed time-series withtransformation and encryption. In SIGMOD, 2010.

[22] I. Roy, S. T. V. Setty, A. Kilzer, V. Shmatikov, andE. Witchel. Airavat: security and privacy formapreduce. In NSDI, 2010.

[23] A. Serjantov and G. Danezis. Towards an informationtheoretic metric for anonymity. In PET, 2002.

[24] A. Smith. Privacy-preserving statistical estimationwith optimal convergence rates. In STOC, 2011.

[25] L. Sweeney. k-anonymity: A model for protectingprivacy. International Journal on Uncertainty,Fuzziness and Knowledge-based Systems, 2002.

[26] H. H. van, M. Fokkinga, and N. Anciaux. Aframework to balance privacy and data usability usingdata degradation. In CSE, 2009.

[27] C. Wright, C. Cowan, S. Smalley, J. Morris, andG. Kroah-Hartman. Linux security modules: Generalsecurity support for the linux kernel. In USENIXSecurity, 2002.

[28] X. Xiao, G. Bender, M. Hay, and J. Gehrke. ireduct:differential privacy with reduced relative errors. InSIGMOD, 2011.

APPENDIXA. THEORETICAL GUARANTEES

We combine the privacy guarantees of the different piecesused in the system to present a final privacy guarantee. Weprovide three different privacy guarantees based on how theoutput range is being estimated.

Theorem 1 (Privacy Guarantee for GUPT). LetT ∈ Rk×n be a dataset and f : Rk×n → Rp be the query.GUPT is ε-differentially private if the following holds:

1. GUPT uses GUPT-helper with loose range for theinput: Execute percentile estimation algorithm definedin [24] for each of the k input dimensions with privacyparameter ε/(2k) and then run the sample and aggre-gate framework (SAF) with privacy parameter ε/(2p)for each output dimension.

2. GUPT uses GUPT-tight: Run SAF with privacy pa-rameter ε/p for each output dimension.

3. GUPT uses GUPT-loose: For each output dimension,run the percentile estimation algorithm defined in [24]with privacy parameter ε/(2p) and then run SAF withprivacy parameter ε/(2p).

The proof of this theorem directly follows from the pri-vacy guarantees for each of the module of GUPT and thecomposition theorem of [5]. In terms of utility, we claim thefollowing about GUPT.

Theorem 2 (Utility Guarantee for GUPT). Letf : Rk×n → Rp be a generic asymptotically normal statistic(see Definition 2 in [24]). Let T be a dataset of size n drawn

i.i.d. from some underlying distribution F . Let f(T ) be thestatistic computed by GUPT.

1. If GUPT uses GUPT-tight, then f(T ) converges tof(T ) in distribution.

2. Let f be a statistic which differ by at most γ (un-der some distance metric d) on two datasets T and

T , where T is obtained by clamping the dataset T oneach dimension by the 75-th percentile and the 25-thpercentile for that dimension. If GUPT uses GUPT-

helper with loose input range, then we have

d(f(T ), f(T )) ≤ γ as n→∞.

3. If GUPT uses GUPT-loose, then f(T ) converges indistribution to f(T ) as n → ∞ as long as k, 1

εand

log(|max−min |) are bounded from above by sufficientlysmall polynomials in n, where |max−min | is the looseoutput range provided.

The proof follows using a similar analysis used in [24].

360