Speeding Up Secure Computations via Embedded Cachingkzhai.github.io/paper/2009_sdm.pdf · tify the...

Speeding Up Secure Computations via Embedded Caching

K. Zhai∗ W. K. Ng† A. R. Herianto‡ S. Han§

AbstractMost existing work on Privacy-Preserving Data Min-ing (PPDM) focus on enabling conventional datamining algorithms with the ability to run in a se-cure manner in a multi-party setting. Although var-ious algorithms in data mining have been enhancedto incorporate secure mechanisms for data privacypreservation, their computation performance is fartoo high to allow them to be practically useful. Thisis especially true for those algorithms that make useof common cryptosystems. In this paper, we addressthe efficiency issue of PPDM algorithms by propos-ing to cache result data that are used more than onceby secure computations. For this to be possible, wecarefully examine the micro steps of secure compu-tations to identify the repetitive or iterative portionsand reduce the overall computational cost by cachingintermediate results/data. We have applied this todecision tree induction, association rule mining andk-means clustering that make use of secure buildingblocks such as secure multi-party sum, secure ma-trix multiplication, and secure inverse of matrix sum.We show empirically that the computational costs ofsecure computations can be reduced without affect-ing the quality of the data mining result in general.Our experiments show that the caching technique isgeneralizable to common data mining algorithms andthe efficiency of PPDM algorithms can be greatly im-proved without compromising data privacy.

1 IntroductionPPDM enables conventional data mining algorithmsto be securely performed in a distributed, multi-partyenvironment with sensitive data. To date, a hostof data mining algorithms have been retrofitted toincorporate privacy preservation. Generally, thereare two approaches for PPDM algorithm: random-ization [2] and secure multi-party computations [9].In this paper, we focus on PPDM algorithms that

∗School of Computer Engineering, Nanyang TechnologicalUniversity, Singapore, [email protected]

†School of Computer Engineering, Nanyang TechnologicalUniversity, Singapore, [email protected]

‡School of Computer Engineering, Nanyang TechnologicalUniversity, Singapore, [email protected]

§School of Computer Engineering, Nanyang TechnologicalUniversity, Singapore, [email protected]

make use of secure computations as the outcomeof this category of algorithms does not compro-mise the correctness of the algorithm to a large ex-tent. Approaches based on secure multi-party com-putation are too costly for practical applications.In the Workshop on Practical Privacy-PreservingData Mining (P3DM’08) [10] held in conjunctionwith SIAM SDM’08, it has been acknowledged thatimproving the efficiency of PPDM is a challenge.Our focus in this paper is to improve the com-putational/communication efficiency of PPDM algo-rithms that make use of secure multi-party compu-tations.

We observe that in general, many data mining al-gorithms are iterative in nature and execute certaindata operations repetitively. When such data are dis-tributed among multiple parties and the data are sen-sitive, the parties invoke secure computations that in-volve encryptions/decryptions and network commu-nications. Hence, the cost of secure computations ishigher than the non-secure version, and it rendersPPDM algorithms less practical for general usage.

We illustrate with two examples. In the Privacy-Preserving k-means Clustering (PPKC) algorithmthat was proposed by Jagannathan and Wright [8],one needs to compute the distance between each datapoint and the cluster center in each iteration. Thisincurrs high computation/communication costs. Asthis distance is computed using the Secure ScalarProduct (SSP) protocol, encryption of the input datacan be performed once and reused many times subse-quently. To achieve this, we could cache the essentialintermediate results to increase the efficiency. Fromthe experiments performed in this paper, we haveindeed shown that the running time can be dramat-ically reduced. Moreover, the modified secure proto-col that incorporates caching is much more scalablein terms of efficiency.

Similar inefficiency is also observed in thePrivacy-Preserving Association Rule Mining(PPARM) algorithm by Vaidya and Clifton [13]. Ateach iteration of computing the frequency count ofevery itemset, the same data vectors that are dis-tributed among different parties are used. Each timesuch vectors need to be used by the parties, severalSSP operations are invoked. As pointed out by Sub-ramaniam et al. [12], high computational overheads

461 Copyright © by SIAM. Unauthorized reproduction of this article is prohibited.

have become a major performance bottleneck of SSPprotocols. These repeated computational overheadsreduce the efficiency of PPARM. We performeda careful micro-inspection of the algorithm andassociated protocols and found ways to embedcaching and avoid redundant computations; thus,improving the overall efficiency.

Based on the above observations, we want toimprove the efficiency of the PPDM algorithms bycutting down the computational and communicationoverheads incurred by secure computations. To ac-complish this objective, we identify a set of securecomputations and examine their micro steps to iden-tify the possibility of embedding caching. Some ofthese micro steps involve cryptographic operations ondata. In this paper, we perform a micro-inspectionof the Secure Sum, SSP and Secure Matrix Multi-plication (SMM) protocols. These secure computa-tions are used in high-level PPDM algorithms suchas association rule mining, decision tree, and k-meansclustering.

We believe our work contributes to an im-portant aspect of privacy-preserving data mining—computational/communication efficiency—that haslargely been ignored by the PPDM community. Thework is part of our larger efforts to develop a prac-tically feasible PPDM system that supports multi-ple distributed parties, much like a secure, privacy-preserving version of Weka that can be plugged intodatabase systems. Our work generalizes various com-putational properties used in secure computations,such as the downward closure property and homo-morphic encryption, so that computation results arecachable. We theoretically and experimentally showthat the caching approach is generalizable to otherdata mining algorithms.

This paper is organized as follows: The next sec-tion summarizes a number of existing PPDM algo-rithms that make use of secure computations. Sec-tion 3 presents details on several existing securecomputations that include observations of proper-ties of computations and whether these computa-tions support caching. We also explain how thecaching technique can be applied to algorithms elab-orated in Section 2. In Section 4, we analyze theefficiency improvements of caching and make com-parisons with conventional data mining algorithmswithout caching. We conclude the paper in the finalsection.

2 Related WorkIn this section, we review some secure building blocksand conventional data mining algorithms that arewidely used. To date, research in privacy-preservingdata mining has produced a host of secure compu-

tation protocols such as secure multi-party sum [3],secure comparison [15], secure scalar product [5] fordata mining algorithms such as decision tree [14],association rule mining [13], k-means clustering [8],and machine learning algorithms such as fisher dis-criminant analysis [6], and singular values decompo-sition [7]. In the following sections, we describe somespecific data mining algorithms. In the next section,we show how the caching idea can be applied to se-cure building blocks to increase their efficiency.

2.1 Secure Scalar Product The SSP protocolis a protocol to compute the scalar product of twovectors from two parties without any private datadisclosure. It is one of the most important securecomputations protocols that are frequently used inmany PPDM algorithms [7, 6, 8, 13]. One of themost well known SSP protocols is the one proposedby Goethals et al. [5] which makes use of a cryptosys-tem that supports homomorphic addition property:E(x+y) = E(x)E(y). This property gives rise to thefollowing:

E(x·y) = E

(n∑

i=1

(xiyi)

)=

n∏

i=1

E(xiyi) =n∏

i=1

E(xi)yi

The algorithm goes as follows: Party A holdingvector x uses the public key to encrypt all elementsand sends them to Party B. Party B computes∏n

i=1(E(xi)yi) and then sends them back to PartyA. Party A decrypts and obtains the scalar productvalue.

2.2 Secure Multi-party Sum Yao [15] first in-troduced the general concept of Secure Multi-partyComputation (SMC). The Secure Multi-party Sum(SMS) was demonstrated by Clifton et al. [3] to allowmultiple parties to compute the sum of their privateshares together without disclosing the information toany other party. The main idea of the algorithmis that the first party uses one random number tohide its private number, and then send the sum ofprivate number and random number to the secondparty. The second party adds its private number tothe sum received from the first party and then sendsthe new sum to third party. After one round, thefinal party sends the sum to the first party. The firstparty subtracts the random number and obtains thetotal sum.

2.3 Privacy-Preserving Cooperative LinearSystem of Equations The Privacy-Preserving Co-operative Linear System of Equations Protocol (PP-CLSE) was proposed by Du and Atallah [4] to findthe solution of linear systems of equations collabora-tively. This protocol is applicable to linear regression;


thus, further applicable to finding classification ruleswith disclosure of each party’s private data. Thedataset may be partitioned among parties vertically,horizontally, arbitrarily or even overlapping. Thisprotocol works securely and accurately regardless ofthe partition scheme adopted by the parties.

As an example, suppose Alice holds an n × nmatrix M1 and an n dimension vector b1. Bob holdsan n × n matrix M2 and an n dimension vector b2.The protocol is based on the solution to the linearsystem of equations P (M1+M2)QQ−1x = P (b1+b2),which is equivalent to the solution of the systemof equations (M1 + M2)x = b1 + b2. In order tosolve the system of equations, Bob generates tworandom matrices P and Q and compute the matrixM

′= P (M1 + M2)Q and b

′= P (b1 + b2) without

disclosing any private information. They both obtainthe solution x consequently.

2.4 Privacy-Preserving Association RuleMining Vaidya and Clifton [13] proposed an algo-rithm for PPARM on vertically partitioned data.Attribute values in this algorithm are binary where 0represents the absence and 1 represents the presenceof the attribute value. This algorithm is based onthe conventional Apriori algorithm for multi-partyprivacy-preserving Apriori. The cornerstone ofPPARM is to compute the frequency of itemsets. Asan itemset could be partially owned by two parties,a SSP operation is required.

2.5 Privacy-Preserving Decision Tree Induc-tion Vaidya and Clifton [14] proposed a Privacy-Preserving ID3 (PPID3) algorithm for constructing adecision tree classifier on vertically partitioned data.The core operation of ID3, as well as the PPID3 al-gorithm, is to compute the information gain for aspecific set of attributes in order to determine thebest splitting attribute. A SSP computation is alsorequired to evaluate the entropy on certain attributesto further build the decision tree.

2.6 Privacy-Preserving k-Means ClusteringJagannathan and Wright [8] proposed an algorithmfor distributed PPKC on arbitrarily partitioned datathat retrofits the conventional k-means algorithm fora two-party scenario. The basic idea is to constantlyupdate every cluster center until a stable state isachieved. The critical operation is to compute theEuclidean distance between a data point and thecluster center; this requires invocation of SSP asthe data point is partially owned by two parties.Furthermore, the termination state also needs a seriesof SSP computations to be performed.

2.7 Secure Matrix Multiplication & Inverseof Matrix Sum Computation Han and Ng [6]addressed the secure multiplication of two matrices.Party A and B each hold a private m×N matrix Aand N × n matrix B respectively. Using of SSP pro-tocol in Section 2.1, the matrix multiplication couldbe easily computed. We will discuss more of this inSection 3.6. This observation is further applied toderive a Secure Inverse of Matrix Sum Protocol us-ing the formula PP−1(A + B)−1 = (A + B)−1. Theoverall process involves two SMM operations.

3 Embedding Caching in Data MiningAlgorithms

In this section, we investigate how one can applycaching to improve the computational efficiency ofPPDM algorithms. We analyze the efficiency im-provement and memory required to cache data thatare reused repeatedly.

3.1 Secure Multi-party Sum If the samedataset that is held by multiple parties are repeatedlyused in a data mining algorithm, we can simply cachethe results that were computed before. Although theresult caching of the SMS protocol is rather straight-forward, it improves the computation when there arenew parties joining the system. The original set ofparties may be conceptually treated as a single partyand the SMS protocol can be applied in the usualmanner to all parties, using the cached results of thesingle party.

Analysis The intermediate result caching of theSMS protocol reduces the amount of network mes-sages and the computational overheads of each party.The memory used for caching is relatively small andthus negligible for each party involved. However, ifonly one or a few parties are in the system, it maynot be very secure to use this caching technique. Inthis case, we may have to perform the SMS protocolanew.

3.2 Privacy-Preserving Cooperative LinearSystem of Equations Protocol With reference tothe earlier description in Section 2.3, we want to com-pute the value of P (M1 + M2)Q, where M1 (respec-tively M2) is held privately by Party A (respectivelyB). Matrices P and Q are randomly generated matri-ces known to Party B only. By decomposing M1 andM2 into several sub-matrices by both parties, theyexchange information securely using the 1-out-of-NOblivious Transfer protocol [11]. The overall resultfollows.

As illustrated in Algorithm 1, the result cachingof this protocol can be applied on matrix sequences


Algorithm 1 Private Evaluation of M ′ =P (M1 + M2)Q.

Input: Alice has a matrix M1, and Bob has a matrixM2 and two random matrices P and Q.Output: M ′ = P (M1 + M2) Q

1: Alice and Bob agree on two large numbers p andm.

2: Alice generates m random matrices X1, . . . , Xm,such that M1 = X1 + · · ·+ Xm.

3: Bob generates m random matrices Y1, . . . , Ym,such that M2 = Y1 + · · ·+ Ym.

4: for each j = 1, . . . , m, Alice and Bob conduct thefollowing sub-steps: do

5: Alice sends the following sequence to Bob:(H1, . . . ,Hp) where for a secret 1 6 k 6p,Hk = Xj ; the rest of the sequence are ran-dom matrices. k is a secret random numberknown only by Alice, namely Bob does notknow the position of Xj in the whole sequence.Bob could cache the matrix sequence(H1, . . . ,Hp) for future convenience dur-ing this process.

6: Bob computes P (Hi + Yj)Q + Rj for eachi = 1, . . . , p, where Rj is a random matrix.

7: Using the 1-out-of-N Oblivious Transfer pro-tocol, Alice gets back the result of Gj =P (Hk + Yj)Q+Rj = P (Xj + Yj) Q+Rj . Al-ice could cache the matrix Gj for futureconvenience during this process.

8: end for9: Bob sends Σm

j=1Rj to Alice.10: Alice computes M ′ =

Σmj=1 (P (Xj + Yj) Q + Rj) − Σm

j=1Rj =P (M1 + M2)Q.

Hjs in Party B and Gjs in Party A. In the case whenthe matrix input from B is modified from M2 to M

′2

in future, B can choose any ` (` ∈ [1,m]) out ofm Yjs for suitable adjustment. The total number `and the index of matrices can be chosen and selectedarbitrarily, depending on the security and difficultylevel for a party to speculate the information. Inother words, the number of matrices that Party Bchoose to modify greatly affects the data privacyof its own data. The more matrices that Party Bmodifies, the lower is the possibility of Party A’sability to speculate the information. However, thisresult caching scheme is relatively secure for ` > 2 ingeneral.

For example, suppose B decides to modify thevalue of matrices Y1 & Y2 to Y

′1 & Y

′2 such that

the equation M′2 = Y

′1 + Y

′2 + Y3 + · · · + Ym is

satisfied. Then B will only re-compute the sequence

G′1 & G

′2 based on the sequence H1 & H2 cached.

After Party A receives the updated matrix sequencesG′1 & G

′2, it simply obtains the result back with a

secret random number k and performs the rest of thecomputations with sequence G3, G4, . . . , Gm cachedin the last iteration. Hence, before applying thiscaching approach, we would like to choose the partythat has a higher probability of changing the matrixinput, such as Party B above. In the case of twoparties changing their inputs together, A could alsomodify its matrix sequence Xi in same manner aswell.

Analysis The intermediate result caching on this al-gorithm greatly reduces the computational complex-ity in terms of the number of oblivious transfers aswell as matrix multiplications. By caching the previ-ous result, the total number of oblivious transfers isreduced from m to 2, following the above description.Also, the number of matrix multiplications carriedout is 2m instead of m2, a significant reduction intotal computation time. This also reduces the over-all network communication cost.

Memory usage for the caching scheme dependson the number of matrices and their dimensions. Ifwe assume m − ` out of m matrices are maintainedin subsequent iterations and every matrix dimensionis d × d, the minimum memory (in bytes) requiredfor Alice is (m − `) × d2 × b bytes, where b denotesthe number of bytes for storing a single element in amatrix. As Bob stores a series of matrix sequences,the overall memory required by him is relatively largecompared to Alice; it requires (m − `) × d2 × b × pbytes in total.

Data privacy is preserved during caching. Giventhat the original protocol is secure, Bob would not beable to derive any solvable linear system of equationsby changing its matrix to probe Alice’s private ma-trix. This is true for Alice as well. Both parties agreeon the number of matrices in the sequence that theyare going to modify. Theoretically speaking, morenumber of matrices modified leads to better security.In general, two matrices are more than sufficient toguarantee the privacy of both parties, as it can beproved that both parties are not able to establishany linear system of equations to derive the otherparty’s private data. If this algorithm with cachingis executed many times, two parties could come to aconsensus to modify different portions of their matrixsequence based on the cached results in different it-erations to avoid the derivation of any solvable linearsystem of systems by any party.

Depending on pre-protocol agreement, Party Amay cache

∑mj=1 Rj if Party B maintains the same

Σmj=1Rj but with modifications on corresponding

items in the random number series Rj in each it-


Table 1: Data partitions among 3 parties

TID A1 A2 B1 C1 Class

T1 1 0 1 0 1T2 1 1 0 1 1T3 0 1 0 1 1T4 1 0 1 1 0T5 1 1 1 1 0

eration. Even though the improvement is minor, itreduces network communication cost if the networkdata transfer overhead is relatively high.

3.3 Privacy-Preserving Association RuleMining As described in Section 2.4, multipleparties performing the Apriori algorithm iterativelygenerate frequent itemsets using PPARM protocol.We observe that intermediate result caching can beapplied throughout the entire algorithm.

Let us elaborate this process using the examplein Table 1. Suppose after the first round of compu-tations, attributes A1 and A2 are held by Party A,attribute B1 is held by Party B, and attribute C1 isheld by Party C. In the next iteration, A and B com-pute the frequency counts of itemsets {A1, B1} and{A2, B1} collaboratively. Within each iteration, oneparty caches the encrypted sequence from the otherparty. In this case, B sends the encrypted sequenceof B1 to A, and Party A caches the sequence and useit throughout the following computations.

Consider the case when the frequency counts ofall itemsets generated in the 2nd iteration are greaterthan the support threshold. The three parties willfurther proceed to compute the frequency count of{A1, B1, C1} and {A2, B1, C1} in the next iteration.Consider the computational process {A1, B1, C1}.Party C requires the encrypted vector {A2, B1},which was computed in the previous round. If PartyA or B properly caches the sequence, the overall num-ber of cryptography computations is greatly reduced.

Analysis The exact number of SSP operationstrimmed by caching is dependent on the number offrequent itemsets involving a particular input vectorgenerated by the Apriori algorithm. Let us assumein one iteration, the number of frequent itemsets in-volving an input data vector ~V is k. The length of~V is `. If proper caching is applied, the number ofcryptography computations is reduced from (k + 1)`to k + `.

To apply this caching scheme, additional mem-ory blocks for storing completely encrypted vectorsfrom previous iterations are required. For example,suppose at iteration i there are c candidate frequentitemsets satisfying the support threshold. Then at it-

eration i + 1, the overall caching memory required isc×` in the worst case, where ` denotes the size of eachvector. In practice, many vectors overlapped one an-other, it is a lot less than c× `. As can be observed,the memory required is sequentially reducing, as thenumber of frequent itemsets is decreasing. As a re-sult, the system should reserve the necessary amountof memory for caching according to the number ofsingle attributes that satisfy the support thresholdduring the first iteration, as the number of frequentitemsets is maximum at that time.

Security is not an issue in this algorithm as longas the number of tuples is larger than the dimensionof dataset (which is true in real applications) asboth parties are not able to build a linear systemto derive the private data of other parties. Thisproblem is associated with the original PPARMalgorithm without caching. If the original PPARMprotocol is proved to be secure with input datasets,applying the caching scheme is secure as we onlycache intermediate results that are proven to besecure.

3.4 Privacy-Preserving Decision Tree Induc-tion The part of PPID3 that requires SSP is thecomputation of the information gain (i.e., entropy)in order to determine the best splitting attribute.Intermediate result caching scheme can be appliedthroughout the entire algorithm in a similar man-ner as PPARM as the input data are all strictly bi-nary sequences. In this section, we introduce threecaching strategies that improves the efficiency of thealgorithm. Referring to the data in Table 1, supposethe class attribute is owned by all parties and thealgorithm proceeds to node (B1 = 0). Let n be thedomain size of the class attribute and k be the num-ber of values of the observed attributes (n and k areequal to 2 in this case). If we want to compute theinformation gain of attribute A1, we evaluate:

Entropy(DB1=0, A1 = 0)

= −PClass=0

|DA1=0| logPClass=0

|DA1=0| −PClass=1

|DA1=0| logPClass=1

|DA1=0|and

Entropy(DB1=0, A1 = 1)

=− PClass=0

|DA1=1| logPClass=0

|DA1=1| −PClass=1

|DA1=1| logPClass=1

|DA1=1|where PClass=0 (respectively PClass=1) refers to theprobability of Class = 0 (respectively Class =1). All the computations of the above entropycomputations are within the domain B1 = 0. Tocompute the probability of every possible values ofthe class attribute involves a SSP computation. Intotal, for every observed attribute, n × k = 4 SSP


operations involving the same input vector B1 arecarried out. By caching vector B1, we only need tosend this vector once.

Party A has another attribute A2 in addition toA1. A 2nd caching strategy is applicable when thesystem computes the information gain of attributeA2. As this computation is similar to the processdescribed above, it also requires the vector on at-tribute B1. So caching on vector B1 is still applicableto other attributes for further reducing the crypto-graphic computational overheads.

A 3rd caching strategy is based on the iterativecharacteristic of the ID3 algorithm. It is known thatthe SSP protocol executed in each iteration alwaysdepends on its result from the previous iterations(except the first iteration). Continuing from theexample above, assume that in the second iteration,we choose attribute A2 as the splitting attribute andnow we are at tree node (B1 = 0, A2 = 0). Theinformation gain at this point, as stated previously,is obtained by performing SSP between the B1 vectorowned by B and the A2 vector owned by A. Inthis process, if we assume Party B initiates theSSP protocol, Party A retrieves a vector denoted as~V = A2 ×B1 in encrypted form.

The next step of the protocol is to computethe information gain on attribute C1 of party C,which again requires a SSP computation involving allthree parties. Now, the encrypted vector ~V from theprevious iteration cached by Party A can be treatedas the input to Party C directly. Eventually, afterthe protocol is completed, the scalar product of allthree vectors A2 ·B1 · C1 can be computed easily.

Analysis The number of cryptographic computa-tions reduced via caching depends on characteristicsof the constructed decision tree. However, the im-provement is predictably significant as almost all re-peated cryptographic computations are eliminated.Similar to the caching scheme applied on PPARM al-gorithm, the security of the overall algorithm is guar-anteed with this caching scheme.

The memory requirement for caching schemeon PPID3 would be relatively large compared toPPARM, as splitting attribute for every ancestorshould be cached in order to implement the thirdcaching strategy that is to efficiently compute theinformation gain for child and sibling node. Supposed is the depth of the tree and n represents the numberof data. If we denote k as the approximate sizeof an encrypted number (in byte). In total, wewill need memory of d × n × k bytes to implementthe third caching strategy. In addition, for everyiteration to efficiently compute information gain onevery attribute, we also need n×k bytes memory forimplementing the first and second caching strategy.

However, this memory can be released when theparty has not had any attribute to be considered inthat iteration. Therefore, in the worst case scenariowe will require additional (d + 1) × n × k bytes toimplement all caching strategies.

Algorithm 2 Privacy Preserving k-means Cluster-ing Protocol.

Input: Database D consisting of n data objectsarbitrarily partitioned by Alice and Bob.Integer k denotes the number of cluster centers.Output: Assignment of the cluster centers to input

data objects.1: Randomly select k objects from D as initial

cluster centers µ′1 . . . µ′k.2: Randomly generate all cluster center shares for

Alice and Bob:Alice’s share =

(αA

1 . . . αAk

)Bob’s share =

(αB

1 . . . αBk

)3: repeat4:

(µA

1 . . . µAk

)=

(αA

1 . . . αAk

)(µB

1 . . . µBk

)=

(αB

1 . . . αBk

)5: for each di in D do6: Compute and assign di to the closest cluster.

Alice (respectively Bob) could cachethe vector sequence (µB

1 . . . µBk ) (respec-

tively (µA1 . . . µA

k )) for convenience dur-ing this process.

7: end for8: Alice and Bob securely recompute random

shares of the centers of the k clusters as(αA

1 . . . αAk

)and

(αB

1 . . . αBk

)respectively.

9: until(µA

1 + µB1 , · · · , µA

k + µBk

)is close enough to(

αA1 + αB

1 , · · · , αAk + αB

k

)

3.5 Privacy-Preserving k-means ClusteringAs described in Section 2.6, intermediate resultcaching can be applied. Algorithm 2 illustratesthe PPKC protocol incorporating intermediate re-sult caching. The core portion of the PPKC algo-rithm is to compute the distance between a datapoint and every cluster center. This process involvesSSP computations among different parties iteratively.Consider a data point di which is represented by `distinct attributes di = (xi,1, xi,2, . . . , xi,`). With-out loss of generality, assume xi,1, . . . , xi,s belong toParty A and the rest of the `−s attributes belong toParty B. The equation used to compute the distancedist (di, µj) between the data input di and j-th clus-


ter mean µj is as follows:

(dist (di, µj))2

=(xi,1 −

(µA

j,1 + µBj,1

))2+ · · · (xi,` −

(µA

j,` + µBj,`

))2

=∑̀m=1

x2i,m +

∑̀m=1

(µA

j,m

)2+

∑̀m=1

(µB

j,m

)2

− 2∑̀m=1

µAj,mxi,m − 2

l∑m=1

xi,mµBj,m

+ 2∑̀m=1

µAj,mµB

j,m

as µj,t = µAj,t +µB

j,t, where µAj,t (respectively µB

j,t) rep-resents the t-th item of A’s (respectively B’s) share ofthe j-th cluster center. The first three terms can becomputed individually by Party A and B but the lastthree terms need a SSP protocol between the two par-ties. Within the same iteration, µA

j,t and µBj,t are con-

stant until all data inputs have been successfully ex-amined and assigned. Therefore, they are amenableto caching. After they exchange the encrypted vec-tors of all cluster centers at the beginning of eachiteration, the distance between any input data ob-ject and every cluster center is easily obtained. Inother words, the cryptographic overheads of the lastthree items are reduced to two decryption operationsas the result of

∑`m=1 µA

j,mµBj,m can be reused within

the same iteration for the same j value.Caching efficiency involving cluster centers is fur-

ther seen in the termination condition of the algo-rithm. At the end of every iteration, cluster centersare updated accordingly. The algorithm terminateswhen there are no substantial improvement in reduc-ing the error in approximation, which is measured interms of the Euclidean distance between the originaland updated cluster centers as follows:(dist(µold

j , µnewj )

)2

=(dist(µA,old

j + µB,oldj , µA,new

j + µB,oldj )

)2

=∑̀m=1

(µA,new

j,m + µB,newj,m

)2

+∑̀m=1

(µA,old

j,m + µB,oldj,m

)2

− 2∑̀m=1

(µA,old

j,m + µA,oldj,m

)(µA,new

j,m + µA,newj,m

)

=∑̀m=1

(µA,new

j,m + µB,newj,m

)2

+∑̀m=1

(µA,old

j,m + µB,oldj,m

)2

− 2∑̀m=1

µA,oldj,m µA,new

j,m − 2∑̀m=1

µB,oldj,m µB,new

j,m

− 2∑̀m=1

µA,oldj,m µB,new

j,m − 2∑̀m=1

µB,oldj,m µA,new

j,m

and the notation old (respectively new) simplystands for the original (respectively updated) clustercenter. When result caching is applied, we observethat only the first item requires a SSP while all oth-ers items require a decryption instead. As a result,the total number of cryptographic computation is re-duced significantly.

Analysis As stated previously, caching is applicableto the clustering centers. Let us assume the inputdataset to the PPKC system contains n input tuplesdistributed over ` number of attributes. At everyiteration, before we assign each data input to thenearest cluster center, the distances between any twoof them are computed. Thus, there are 3 × n × kSSP operations in total. To test for convergence, thedistance (also referred as error rate) between the oldand new cluster centers are computed. This requires4 SSP computations. In total, there are 4 × k SSPoperations. Based on these observations, the totalnumber of encryptions (Senc) are:

Senc = t× (3× n× k + 4× k)× `

where k represents the total number of clusters and tis the number of iterations required before it reachesconvergency.

As elaborated above, both parties are required tocache the encrypted vectors of all other party’s shareof cluster centers. This means that both parties needto encrypt all their cluster centers of length equals tok × ` concurrently. Furthermore, after both partiesobtain the new cluster center, they need to computethe distance between the previous cluster center andthe new cluster center. In this computation, Bobneeds to cache the encrypted value of the partial newcluster center held by Alice and so does Alice. So, inthe first iteration, we need 2 × k × ` encryptions.However, this new cluster center will be used againfor the next iteration if the iteration is not the last tocompute the distance between all cluster centers anddata tuples. This is why for the following iterations,we need only 1 × k × ` encryptions; thus, yielding atotal of (t+1)×k×` encryptions. After intermediateresult caching scheme is applied, the number ofencryption computations is essentially reduced to

Senc = (t + 1)× k × `

. The number of decryptions is ignored as it remainsconstant regardless of whether result caching schemeis applied.

The extra memory required for caching is k × `vectors at both sites, as both parties use it holdthe share held by the other party for every clustercenter. If one encrypted number requires b bytes,both parties reserve k × ` × b bytes in advance to


perform caching. There is no data privacy violationwhen caching is used as the private data will alwaysbe kept by the corresponding party. The informationthey exchanged is simply the shares of every clustercenter, which is considered relatively less sensitivedata.

3.6 Secure Matrix Multiplication & Inverseof Matrix Sum Computation To show the gen-eralizable feature of the proposed technique, we in-clude the efficient secure matrix multiplication pro-tocol here as well [7]. And we also explore the secureinverse of matrix sum computation using embeddedcaching technique here.

Let us denote A1, A2, . . . , Am as m row vectorsof A and B1, B2, . . . , Bn as n column vectors of B.Recall the definition of matrix multiplication:

M = AB

=

A1 ·B1 A1 ·B2 · · · A1 ·Bn

A2 ·B1 A2 ·B2 · · · A2 ·Bn

......

. . ....

Am ·B1 Am ·B2 · · · Am ·Bn

Essentially, every element in M is a scalar product oftwo vectors Ai and Bj (i ∈ [1,m], j ∈ [1, n]), whichcan be computed using any SSP protocol introducedin Section 2.1. To compute every element of everyrow i (respectively column j), Party B (respectivelyA) caches the encrypted vector of Ai (respectivelyBj) sent by Party A (respectively B). The waywe compute the matrix M row-wise or column-wise depends on the matrix dimension. In order toincrease the efficiency of result caching, we alwayschoose the loop with less iteration as the outer loop.

Efficient secure matrix multiplication is used toobtain the inverse of matrix sum as discussed inAlgorithm3. In the secure inverse of matrix sum,Alice and Bob each hold private d× d matrix A andB respectively. They want to compute the inverse ofA + B.

In Steps 2 and 5 of this protocol, the multipli-cation between two matrices is performed. We alsonotice that they are over the same matrix P . Thus,intermediate result caching is applied to P . However,take note that in these two matrix multiplications, Pis involved in different ways. It is first passed column-wise and during the second multiplication, every rowof P is exchanged among parties. Clearly, in Step 2,if we let Party B initiate the SSP mentioned in Sec-tion 2.1, B will send the entire matrix P in encryptedform to A before executing this protocol.

Analysis For matrix multiplication, there are n2

encryptions and n decryptions in the computationprocess in total at the cryptographic level. This

Algorithm 3 Secure Inverse of Matrix Sum Proto-col.

Input: Private n × n matrix A and B holding byParty A and B respectively.Output: Private n × n matrix Ma and Mb holding

by Party A and B respectively where Ma + Mb =(A + B)−1.1: B randomly generates a non-singular n×n matrix

P .2: A and B jointly perform secure matrix multipli-

cation to compute AP , at end of which, A and Bwould hold value Sa and Sb correspondingly suchthat Sa + Sb = AP . Party A could cache thematrix P for future references during thisprocess.

3: B send Sb + BP to A.4: A computes Sa + Sb + BP which equals to (A +

B)P , and then compute inverse P−1(A + B)−1.5: After B and A jointly perform secure matrix

multiplication on P and P−1(A + B)−1, theywould hold value Mb and Ma where Ma + Mb =PP−1(A + B)−1 = (A + B)−1.

is more effective than the one without caching (n3

encryptions and n decryptions).The memory used for this caching scheme would

be relative small, simply the size of a n × n matrix.If we denote k as the memory (in byte) for holdingone encrypted number, the total memory required isn× n× k bytes.

Data privacy during matrix multiplication ispreserved provided the SSP is secure, as in theprocess, Party A only sees an encrypted n×n matrixfrom B if we assume Party B initiates the protocol.For the inverse of matrix sum computation, as itis built upon several secure building blocks, we caneasily prove that there is no data leakage and privacydisclose to other parties.

4 Experimental EvaluationsIn the following sections, we present experiment re-sults of incorporating caching. The system is imple-mented in Java using Netbeans 6.1 and the platformis a Windows XP system running on an Intel Pentium4 3.40 GHz processor with 1 GB of main memory.The sample dataset is the Nurse dataset [1] from theUCI depository. This dataset has 9 attributes and12,960 tuples. We perform some preprocessing onthe data, such as converting it to binary data. Thecommunication network is simulated by passing datain a self loop. We compared the total number of cryp-tographic operations required for PPDM algorithmswith and without incorporating caching.


0 20 40 60 80 1000

0.5

1

1.5

2

2.5

3

3.5x 10

7

Input Tuples (%)

No.

of

Cyp

togr

aphy

Ove

rhea

ds

Without CachingWith Caching

Figure 1: Cryptographic computations for differentnumber of tuples in association rule mining.

4.1 Privacy-Preserving Association RuleMining Figure 1 illustrates the effects of cachingefficiency versus the number of tuples. The systemfirst extracts some portions of the entire dataset.Then the algorithm is run several times with differ-ent number of input tuples. All of the tuples areselected randomly according to uniform distributionto avoid using clustered data for each experiment.As expected, when the number of data tuplesincreases, we have more itemsets that satisfy thesupport threshold. Thus, more SSP and crypto-graphic computations are involved. With caching,similar behavior appears, but at a much slower rate.Hence, the overall performance of the PPDM systemgreatly improved, especially when the dataset is verylarge, as intermediate result caching provides a moreeffective way to handle this situation.

Figure 2 elaborates the relationship betweenthe number of parties in the PPDM versus theamount of cryptographic computations. We splittedall attributes in the Nurse dataset randomly amongdifferent parties. The number of SSP operationsdoes not fully depend on the number of parties,but more so on the frequent itemsets generated atevery iteration. If the itemsets generated are fullyowned by the same party, we will not need anySSP computation. As a result, the total numberof cryptographic computations is affected by thepartition scheme that parties choose over the entiredataset. The behavior of the graph does not showmuch pattern. Instead, it fluctuates within a certainrange.

The SSP protocol (Section 2.1) that we usefor this experiment does not require any additionalencryptions when the number of parties increases. Asshown, for the two-party setting, the SSP protocolneeds n+1 encryptions where n is the vector length.In our case, vector length is the number of tuplesinvolved in the experiment. In the three-party

2 3 4 5 6 7 80

0.5

1

1.5

2

2.5x 10

7

No. of Party

No.

of

Cry

ptog

raph

y O

verh

eads


Figure 2: Cryptographic computation for differentnumbers of party in association rule mining.

scenario, the protocol needs only n + 3 encryptionsin total. Additional encryptions do not affect theoverall performance if n is substantially large.

Still, when caching is applied, we observe majorimprovement in the overall processing speed of thePPDM system. The total number of cryptographiccomputations after we apply caching is about 1

10 ofthe original protocol.

In Figure 3, we address the impact to cachingefficiency under different support thresholds. In thisexperiment, we choose different threshold percent-ages in the range between 1% and 25%. The supportthreshold is defined as the number of total tuples in-volved in the experiment times the percentage. Werandomly choose 6,473 tuples from the Nurse datasetas the input data. In general, there is a negative cor-relation between the support threshold value and to-tal number of cryptographic computations. A highersupport threshold essentially reduces the number offrequent itemsets. Hence, it reduces the number ofiterations and SSP computations required.

Meanwhile, as we can observe from the figure,the intermediate result caching scheme indeed greatlyimproves system efficiency. This is especially truewhen the support threshold is relatively low, whilea comparably larger amount of SSP computations isrequired.

4.2 Privacy-Preserving Decision Tree In Fig-ure 4, we illustrate the relationship between the over-all number of cryptographic computations and thenumber of input tuples. Like experiment on PPARM,the system feeds the program with only a portion ofthe total Nurse dataset each time; thus, simulatingdifferent number of input tuples to the system. Asobserved, an increase in the total number of input tu-ples eventually leads to an increase in the complexityof constructing a decision tree classifier. However, aswe can see from the diagram, caching shows better


0 5 10 15 20 250

2

4

6

8

10

12x 10

7

Threshold (%)

No.

of

Cry

ptog

raph

y O

verh

eads


Figure 3: Cryptographic computations under differ-ent support thresholds for association rule mining.

0 20 40 60 80 1000

0.5

1

1.5

2x 10

8

Input Tuples (%)

No.

of

Cry

ptog

raph

y O

verh

eads


Figure 4: Cryptographic computations for differentnumber of tuples in decision tree induction.

performance compared to the original protocol with-out caching irrespective of the number of tuples tothe system. Since intermediate result caching mainlyfocuses on the reduction of the number of crypto-graphic computations, the more SSP operations areinvolved, the better is the caching scheme perfor-mance. In fact, experimental results indicate thatcaching technique works a lot better when the inputdataset is large.

PPID3 algorithm is able to handle multi-partysituations as well. The total number of partiesinvolved in building a decision tree classifier is takeninto account in Figure 5. In this figure, even withdifferent number of parties, the outputs are quitestable with only minor fluctuations in both curvesrepresenting the cases with and without cachingrespectively. A comparison of the two results showsthat the overall system performance is improvedtremendously with intermediate result caching.

4.3 Privacy-Preserving k-Means Clusteringin Arbitrarily Partitioned Data Unlike PPARMor PPID3 where classification execute on binary data

2 3 4 5 6 7 80

2

4

6

8

10

12x 10

7

No. of Party

No.

of

Cry

ptog

raph

y O

verh

eads


Figure 5: Cryptographic computations for differentnumber of parties in decision tree induction.

inputs, PPKC performs on any real-valued data. Weuse the Nurse dataset as input. The total numberof cryptographic computations can be calculatedexplicitly. Hence, the efficiency of intermediate resultcaching is easy to observe.

Experimental results on the PPKC algorithmprovides supporting evidence on previous computa-tions. In Figure 6, we illustrate the impact of thenumber of cryptographic computations under differ-ent number of input tuples. Figure 7 shows the rela-tionship between the number of clusters and the totalnumber of cryptographic computation required. Thediagram in Figure 8 shows the comparison betweendifferent number of cryptographic computations ac-cording to the different number of iterations in bothintermediate result caching scheme applied and origi-nal algorithm. All three graphs indicate the expectedlinear outputs irrespective of the underlying input pa-rameter that fit the previous computation formulas.After result caching scheme is applied, the numberof decryptions is the same as the original algorithm.However, the total number of encryptions is reducedsignificantly. As a result, the larger the input datasetis, the more efficient and effective is the intermediateresult caching scheme. In practical applications, ifwe have an extremely large dataset, the caching tech-nique greatly improves the overall processing speedof the system.

4.4 Secure Matrix Multiplication We imple-mented the SMM protocol to illustrate caching ef-ficiency in C# language under Microsoft Visual Stu-dio 2005 environment. The experiments are tested onWindows XP operating system with Intel Pentium 43.40GHz and 3GB memory. The input matrices tothe system are of m×N and N ×n dimension corre-spondingly. The values are generated randomly anddistributed uniformly in a mathematical field F.

Figure 9 illustrates the effect of total processing


0 20 40 60 80 1000

2

4

6

8

10

12x 10

9

Input Tuples (%)

No.

of

Cry

ptog

raph

y O

verh

eads

WithoutCachingWithCaching

Figure 6: Cryptographic computations for differentnumber of tuples in k-means clustering.

0 5 10 15 200

1

2

3

4

5x 10

10

No. of Cluster

No.

of

Cry

ptog

raph

y O

verh

eads


Figure 7: Cryptographic computations for differentnumber of clusters for k-means clustering.

0 50 100 150 2000

2

4

6

8

10

12x 10

9

No. of Iteration

No.

of

Cry

ptog

raph

y O

verh

eads


Figure 8: Cryptographic computations under differ-ent number of iterations for k-means clustering.

time with respect to different dimensions of inputmatrices. With result caching, the overall systemperformance is improved greatly. This is especiallytrue when input matrices are large.

0 200 400 600 800 10000

2000

4000

6000

8000

10000

12000

Value of n (m=10, N=10)

Com

puta

tion

Tim

e (M

illis

econ

d)


Figure 9: Processing time for different input matrixdimension n in secure matrix multiplication.

0 200 400 600 800 10000

2000

4000

6000

8000

10000

Value of N (m=10, n=10)

Com

puta

tion

Tim

e (M

illis

econ

d)


Figure 10: Processing time for different input matrixdimension N in secure matrix multiplication.

5 ConclusionsSecure computation is the most critical part ofPPDM algorithms. Although it preserves the pri-vacy of each party’s data, it also incurs the majorityof computational cost of algorithm execution. In thispaper, we addressed the inefficiency issues in exist-ing PPDM algorithm such as PPARM, PPID3, andPPKC and proposed an approach to resolve ineffi-ciency using result caching.

We have implemented and experimentally eval-uated our approach in PPDM algorithms and com-pared its performance with PPDM algorithms thatdo not incorporate caching. All of our experimentalresults show very high degree of reduction in numberof cryptographic computations incurred.

References

[1] UCI machine learning repository: Nursery data sethttp://archive.ics.uci.edu/ml/datasets/nursery.

[2] R. Agrawal and R. Srikant. Privacy-preserving datamining. In Proceedings of the ACM internationalConference on Management of Data, pages 439–450,Dallas, Texas, United States, 2000.


[3] C. Clifton, M. Kantarcioglu, J. Vaidya, X. Lin, andM. Y. Zhu. Tools for privacy preserving distributeddata mining. In SIGKDD Explorations, 4(2):28–34,December 2002.

[4] W. Du and M. J. Atallah. Privacy-preservingcooperative scientific computations. In Proceedingsof the 14th IEEE workshop on Computer SecurityFoundations, page 273, Washington, DC, USA,2001. IEEE Computer Society.

[5] B. Goethals, S. Laur, H. Lipmaa, andT. Mielikainen. On private scalar productcomputation for privacy-preserving data mining. InProceedings of the 7th Annual International Con-ference in Information Security and Cryptology,pages 104–120, Seoul, Korea, December 2–3 2004.

[6] S. Han and W. K. Ng. Privacy-preserving linearfisher discriminant analysis. In Proceedings of the12th Pacific-Asia Conference on Knowledge Discov-ery and Data Mining, Osaka, Japan, May 2008.

[7] S. Han, W. K. Ng, and P. Yu. Privacy-preservingsingular value decomposition. In Proceedings ofthe 25th IEEE International Conference on DataEngineering, Shanghai, China, April 2009.

[8] G. Jagannathan and R. N. Wright. Privacy-preserving distributed k-means clustering over ar-bitrarily partitioned data. In Proceedings of the 8thACM International Conference on Knowledge Dis-covery in Data Mining, pages 593–599, Chicago, Illi-nois, USA, 2005.

[9] Y. Lindell and B. Pinkas. Privacy preserving datamining. In Advances in Cryptology, volume 1880of Lecture Notes in Computer Science, pages 36–53.Springer-Verlag, 2000.

[10] K. Liu and R. Wolff. International workshop onpractical privacy-preserving data mining held inconjunction with siam sdm’08, 2008.

[11] M. Naor and B. Pinkas. Oblivious transfer andpolynomial evaluation. In Proceedings of the AnnualACM Symposium on Theory of Computing, pages245–254, Atlanta, Georgia, United States, 1999.

[12] H. Subramaniam, R. Wright, and Z. Yang. Experi-mental analysis of a privacy-preserving scalar prod-uct protocol. In Proceedings of the Workshop onSecure Data Management (held in conjunction withVLDB’04), 2004.

[13] J. Vaidya and C. Clifton. Privacy preserving as-sociation rule mining in vertically partitioned data.In Proceedings of the 8th ACM International Con-ference on Knowledge Discovery and Data Mining,pages 639–644, Edmonton, Alberta, Canada, July23-26 2002.

[14] J. Vaidya and C. Clifton. Privacy-preserving deci-sion trees over vertically partitioned data. In Pro-ceedings of the 19th Annual IFIP WG 11.3 Work-ing Conference on Data and Applications Security,Storrs, Connecticut, 2005. Springer.

[15] A. C. Yao. How to generate and exchange secrets.In Proceedings of the Annual IEEE Symposium onFoundations of Computer Science, pages 162–167,1986.


Speeding Up Secure Computations via Embedded Cachingkzhai.github.io/paper/2009_sdm.pdf · tify the...

Documents

Transcript of Speeding Up Secure Computations via Embedded Cachingkzhai.github.io/paper/2009_sdm.pdf · tify the...