The Second AAAI Workshop on Privacy-Preserving Artificial Intelligence (PPAI-21)

Dopamine: Differentially Private Federated Learning on Medical Data

Mohammad Malekzadeh, Burak Hasircioglu, Nitish Mital, Kunal Katarya,Mehmet Emre Ozfatura, Deniz Gunduz*

Department of Electrical and Electronic Engineering, Imperial College London.{m.malekzadeh, b.hasircioglu18, n.mital, kunal.katarya15, m.ozfatura, d.gunduz}@imperial.ac.uk

Abstract

While rich medical datasets are hosted in hospitals distributedacross the world, concerns on patients’ privacy is a barrieragainst using such data to train deep neural networks (DNNs)for medical diagnostics. We propose Dopamine, a system totrain DNNs on distributed datasets, which employs federatedlearning (FL) with differentially-private stochastic gradientdescent (DPSGD), and, in combination with secure aggre-gation, can establish a better trade-off between differentialprivacy (DP) guarantee and DNN’s accuracy than other ap-proaches. Results on a diabetic retinopathy (DR) task showthat Dopamine provides a DP guarantee close to the cen-tralized training counterpart, while achieving a better clas-sification accuracy than FL with parallel DP where DPSGDis applied without coordination. Code is available at https://github.com/ipc-lab/private-ml-for-health.

1 IntroductionDeep neural networks facilitate disease recognition frommedical data, particularly for patients without immediate ac-cess to doctors. Medical images are processed with DNNsfor faster diagnosis of skin disease (Esteva et al. 2017),lung cancer (Dunnmon et al. 2019), or diabetic retinopa-thy (DR) (Gulshan et al. 2016). However, the memoriza-tion capacity of DNNs can be exploited by adversariesfor reconstruction of a patient’s data (Carlini et al. 2019),or the inference of a patient’s participation in the trainingdataset (Shokri et al. 2017; Dwork et al. 2017). Due to suchprivacy risks and legal restrictions, medical data can rarelybe found in one centralized dataset; thus, there has been asurge of interest in privacy and utility preserving training ondistributed medical datasets (Kaissis et al. 2020).

Federated learning (FL) (McMahan et al. 2017) trainsa DNN where hospitals collaborate with a central serverin training a global model on their local datasets. At eachround, the server sends the current model to each hospi-tal, then hospitals update the model on their private datasetsand send the model back to the server. The hospitals’ up-dates are susceptible to information leakage about the pa-tients’ data due to model over-fitting to training data (Car-lini et al. 2019). Differential privacy (DP) (Dwork and Roth2014) limits an adversary’s certainty in inferring a patient’s

*This work was funded by the European Research Council (ERC) through Start-ing Grant BEACON (no. 677854) and by the UK EPSRC (grant no. EP/T023600/1).

presence in the training dataset. Before optimizing the DNN,Gaussian random noise is added to the computed gradi-ents on the patients’ data to achieve differentially-privatestochastic gradient descent (DPSGD) (Abadi et al. 2016).

We propose Dopamine, a customization of DPSGD forFL, which, in combination with secure aggregation by ho-momorphic encryption, can establish a better privacy-utilitytrade-off than the existing approaches, as elaborated in Sec-tion 3. Experimental results on the DR dataset (Choi et al.2017), using SqueezeNet (Iandola et al. 2016) as a bench-mark DNN, show that Dopamine can achieve a DP boundclose to the centralized training counterpart, while achiev-ing better classification accuracy than FL with parallel DPwhere the hospitals apply typical DPSGD on their sideswithout any specific coordination. We provide theoreticalanalysis for the guaranteed privacy by Dopamine, and dis-cuss the differences between Dopamine and the seminalwork proposed by (Truex et al. 2019): our solution allowsto properly keep track of the privacy loss at each round aswell as taking advantage of the momentum (Qian 1999) inFL-based DPSGD, which improves the DNN’s accuracy.

The main contribution of this paper is the design andimplementation of FL on medical images while satisfy-ing record-level DP. While previous works on medicaldatasets (as discussed in Appendix, Section B) either do notguarantee a formal notion of privacy, e.g. (Li et al. 2020),or apply weaker notion of DP, e.g. parameter-level DP (Liet al. 2019), to the best of our knowledge, Dopamine is thefirst system that implements FL-based DPSGD that guaran-tees record-level DP for a dataset of medical images. Finally,we publish a simulation environment to facilitate further re-search on privacy-preserving training on distributed medicalimage datasets.

2 Dopamine’s Methodology

Problem Formulation1 Let each hospital k ∈{1, . . . ,K} own a dataset, Dk, with an unknown numberof patients, where each patient i participates with a labeleddata (xi, yi). The global server owns a validation dataset

1As notation, we use lower-case italic, e.g. x, for variables; upper-case italic, e.g.X , for constants; bold font, e.g. X, for vectors, matrices, and tensors; blackboard font,e.g. X, for sets; and calligraphic font, e.g. X , for functions and algorithms.

arX

iv:2

101.

1169

3v2

[cs

.LG

] 2

9 Ja

n 20

21

DG, and we consider a DNN’s utility as its prediction ac-curacy on DG. The goal is to collaboratively train a DNNwhile satisfying record-level DP. We assume a threat modelwhere patients only trust their local hospital, and hospitalsare non-malicious and non-colluding. The global server ishonest but curious. Finally, hospitals and the server do nottrust any other third parties. We assume the patient’s privacy,defined by (ε, δ), is a bound on the record-level DP loss.The hospitals aim to ensure their patients a computationalDP against the server during training, and an information-theoretical DP against the server and any other third partiesafter training. It is called computational DP as a cryptosys-tem is only robust against computationally-bounded adver-saries. Dopamine assumes that adversaries are computation-ally bounded during training, which is a typical assumption,but after training they can be computationally unbounded2.

Dopamine’s TrainingThe training procedure is given in Algorithm 1, where weperform federated SGD among K hospitals. At each roundt, each hospital k samples a batch of samples from its lo-cal dataset, Dtk ⊂ Dk, where each local sample is chosenindependently and with probability q. Due to independentsampling, the batch size is not fixed, and is a binomial ran-dom variable with parameters q and |Dk|. Let |Dtk| and ηdenote the batch size and the learning rate, respectively. LetC denote the maximum value of the L2-norm of per-samplegradients. If a per-sample gradient has a norm greater thanthis, then its norm is clipped to C (Abadi et al. 2016).

As hospitals do not trust the server, a potential solutionis, for each hospital, to add a large amount of noise to themodel updates, wt

k, to keep them differentially private fromthe server. However, adding a large amount of noise has theundesirable effect of decreasing the accuracy of the trainedmodel. A better solution is to employ secure aggregation ofthe model updates, which prevents the server from discov-ering the hospital’s model updates. Since the model updatesare now hidden from the server, the hospitals can add lessamount of noise to keep their model updates differentiallyprivate from the server.

Lemma 1. In Algorithm 1, if each hospital k adds Gaussiannoise nk ∼ N

(µ = 0, σ2 = 2 ln (1.25/δ)C2

ε2|Dtk|2K

)to the aver-

age of (clipped) gradients, then wtG is (ε, δ)-DP against the

server, and(ε√K/K − 1, δ

)-DP against any hospital.

Proof. We compute the effect of the presence or the absenceof a single sample at hospital k on wt

G. Let wtk−w

t−1G be the

(noiseless) model update of hospital k at round t. Due to se-cure aggregation, the server and other hospitals receive (un-encrypted) wt

G−wt−1G = 1

K

∑Kk=1 w

tk−w

t−1G . Since wt−1

G

is already known to all, it is sufficient to only consider wtk.

We define w′tk as the local model if D′tk is used by hospital

k instead of Dtk, where Dtk and D′tk only differ in one sam-ple. Since L2-norm of each per-sample gradient is boundedby C and the hospital k averages all per-sample gradients of

2Background materials are provided in Appendix, Section B.

Algorithm 1 Dopamine’s Training

1: Input: K: number of hospitals, D: distributed dataset,w: model’s trainable parameters, L(·, ·): loss function,q: sampling probability, σ: noise scale,C: gradient normbound, η: learning rate, β: momentum, T : number ofrounds, (ε, δ): bounds on record-level DP loss.

2: Output: wG: optimized global model.3: w0

G = random initialization.4: ε = 05: for t : 1, . . . , T do6: for k : 1, . . . ,K do7: Dtk = Sampling(Dk) // by uniformly sampling

each item in Dk independently with probability q.8: for xi ∈ Dtk do9: gt(xi) = ∇wL(wt−1

G ,xi)

10: gt(xi) = gt(xi)/max(1, ||g

t(xi)||2C

)11: end for12: gtk = 1

|Dtk|(∑

xi∈Dtkgt(xi) +N (0, σ

2·C2·IK )

)13: gtk = gtk + βgt−1k //g0

k = 0

14: wtk = wt−1

G − ηgtk15: end for16: ε = CalculatePrivacyLoss(δ, q, σ, t) // by Mo-

ments Accountant (Abadi et al. 2016)17: if ε > ε then18: return wt−1

G19: end if20: wt

G = 1K

(SecureAggregation(

∑kw

tk))

21: Broadcast(wtG)

22: end for

|Dtk| samples, we have

maxDt

k,D′tk

∥∥∥wtk −w′

tk

∥∥∥2

= ηC/|Dtk|,

for all k ∈ [K]. As adding or removing one sample at hos-pital k only changes the gradients of that hospital, we have

maxDt

k,D′tk

∥∥∥wtG −w′

tG

∥∥∥2

=1

KmaxDt

k,D′tk

∥∥∥wtk −w′

tk

∥∥∥2

=ηC

|Dtk|K,

for all k ∈ [K], where w′tG is the global model when D′tk

is used. Thus, to guarantee a record-level (ε, δ)-DP, the vari-ance of effective noise added to each model update must be

σ2effective =

2 ln(1.25/δ)η2C2

ε2K2|Dtk|2.

If hospitals use noise nk in Lemma 1, then the effectivenoise added to the global model update, wt

G − wt−1G , is

ηK

∑Kk=1 nk, with a variance ησ2

K . When the expression forσ2 in Lemma 1 is put in place, the variance of the ef-fective noise is 2 ln(1.25/δ)η2C2

ε2|Dtk|2K2 = σ2

effective, which proves(ε, δ)-DP against the server. From a hospital’s perspective,since it knows its share in the effective noise, the variance ofthe noise after canceling its share is 2 ln(1.25/δ)η2C2(K−1)

ε2|Dtk|2K3 ,

which results in (ε√K/(K − 1), δ)-DP against hospi-

tals.

Single vs. Multiple Local UpdatesAn algorithm, similar to our Algorithm 1, is previously pro-posed by (Truex et al. 2019) (Section 5.2), where they al-low each party to carry out multiple local SGD steps beforesharing the updated model with the server. We argue thatthis approach has an important drawback: it may violate thecritical assumption in the moments accountant (MA) pro-cedure (Abadi et al. 2016) for tracking the privacy loss ateach round, and for the same reason, it does not allow us-ing momentum for SGD (line 13 in our Algorithm 1), whichhelps to improve the model’s accuracy. The MA is intro-duced by (Abadi et al. 2016) for keeping track of a boundon the moments of the privacy loss random variable, inthe sense of DP, that depends on the random noise addedto the algorithm’s output. In the proof provided by (Abadiet al. 2016), the MA is updated by sequentially applyingDP mechanism M. Particularly, (Abadi et al. 2016) mod-els the system by letting the mechanism at round t, Mt,to have access to the output of all the previous mecha-nismsMt−1,Mt−2, ...,M1, as the auxiliary input. Hence,to properly calculate the privacy loss at round t, one needsto make sure that the amount of privacy loss in the previ-ous t− 1 rounds are properly bounded by adding the properamount of noise.

In Algorithm 1, we allow each hospital to add a Gaus-sian noise of variance σ2/K, at each round, and then sharethe updated model to securely perform FedSGD (McMahanet al. 2017). Thus, before running any other computation onthe DNN, we add the required amount of effective noisethat is needed by MA to calculate the privacy loss ε (seeLemma 1). Moreover, the effect of each record on the localmomentum is the same as its effect on the aggregated model,thus, due to the post-processing property, using momentumsdoes not incur a privacy cost (see Appendix C). However,(Truex et al. 2019) allow the local model to be updated formultiple (e.g. 100) local iterations while at each iteration theDNN model that is used as the input is the output of the pre-vious local iterations, and not the aggregated model. Thus,the sequential property of the proof in MA may be violated.Moreover, when running multiple local iterations, it is notclear how one can ensure a proper sampling similar to whatis needed in the central DPSGD.

Fixing these issues, our algorithm allows to take advan-tage of both privacy loss tracking with the MA and usingSGD with momentums. We emphasize that although our al-gorithm needs more communications due to sharing modelupdates after every iteration, this cost can be tolerated con-sidering the importance of the other two aspects in medicaldata processing: model accuracy and privacy guarantee.

3 EvaluationDatasetDiabetic retinopathy is an eye condition that can cause vi-sion loss in diabetic patients. It is diagnosed by exam-ining scans of the blood vessels of a patient’s retina -thereby making this an image classification task. A DRdataset (Choi et al. 2017), available at https://www.kaggle.com/c/aptos2019-blindness-detection/data, exists to solve

this task. The problem is to classify the images of patients’retina into five categories: No DR, Mild DR, Moderate DR,Severe DR, and Proliferative DR. The dataset consists of2931 variable-sized images for training, and 731 imgaes fortesting. To deal with the variation in image dimensions, allimages were resized into the same dimensions of 224×224.

DNN ArchitectureWe use SqueezeNet (Iandola et al. 2016): a convolutionalneural network that has 50x fewer parameters than the fa-mous AlexNet, but is shown to achieve the same level ofaccuracy on the ImageNet dataset. The size of the model is avery important factor for both training and inference, whichis one of the main motivations for using SqueezeNet. More-over, SqueezeNet can achieve a classification accuracy ofabout 80% on the DR dataset while the current best accu-racy reported on the Kaggle’s competition3 on this dataset isabout 83% using a much larger DNN, EfficientNet-B3, thathas 18x more parameters than SqueezeNet4.

BaselinesThere are several approaches for training a DNN that pro-

vide us different trade-offs between utility, privacy, and com-plexity. We compare Dopamine against the following well-known approaches:

Non-Private Centralized Training (C). Every hospitalshares their data with the server, such that a centralizeddataset D is available for training. In this method the prob-lem is reduced to architecture search and optimization pro-cedure for training a DNN on D. This method provides thebest utility, a moderate complexity, but no privacy. It resultsin a single point of failure in the case of data breach. More-over, in situations where patients are from different coun-tries or private organizations, is almost impossible reachingan agreement on a single trusted curator.

Centralized Training with DP ( CDP). When a trustedcurator is available, one can trade some utility in themethod C to guarantee central DP for the patients byusing a differentially private training mechanism such asDPSGD (Abadi et al. 2016). The complexity of CDP issimilar to C, except for the fact that existing DP mechanismsmake the training of DNNs slower and less accurate, as theyrequire performing additional tasks, such as per-sample gra-dient clipping and noise addition.

Non-Private FL ( F). Considering that a trusted curator isnot realistic in many scenarios due to legal, security, andbusiness constraints, hospitals can collaborate via FL, in-stead of directly sharing their patients’ sensitive data. In F,every hospital has its own dataset that may differ in sizeand quality. We assume the server runs FedAvg (McMahanet al. 2017). While F introduces some serious communi-cation complexities, it removes the need for trusted curatorand enables an important layer of privacy protection. Basi-cally, after sharing the data in the methods C and CDP,

3https://www.kaggle.com/c/aptos2019-blindness-detection/notebooks4The Pytorch implementation of is available at https://pytorch.org/docs/stable/

modules/torchvision/models/squeezenet.html

hospitals have no more control on the type and amount ofinformation that can be extracted from their patients’ data.However, in F, hospitals do not share the original data, butsome aggregated statistics or the results of a computationover the dataset. Even more, hospitals can decide to stop re-leasing information at any time, thus having more controlover the type and amount of information sharing. Althoughit seems that the utility of F might be lower than that of C,there are studies (Bonawitz et al. 2019) showing that witha more sophisticated algorithms, one can achieve the sameutility as C.

FL with Parallel DP (FPDP). Similarly to CDP, one canapply a DP mechanism to the local training procedure at thehospital side. Thus, every hospital locally provides a cen-tral DP guarantee. Notice that this is different form a localDP guarantee (Dwork and Roth 2014), as each hospital isa trusted curator. In FPDP, a much better privacy can beprovided than the previous methods. However, as the size ofeach local dataset Di is much smaller than the size of thewhole dataset D, more noise should be added to achieve thesame level of privacy, which results in a utility loss.

Results

Dopamine proposes FL with a Central DP guarantee thatis similar to F, except the hospitals add noise to their gradi-ents before running a secure aggregation procedure to aggre-gate the models at the server. Dopamine is similar to FPDPin the sense that the hospitals add noise to their gradients,but thanks to secure aggregation, the variance of the noisehospitals add is less than FPDP by an order of K.

Figure 1 compares the classification accuracy (top plot)and DP bound (bottom plot) of all the introduced meth-ods for SqueezeNet trained on the DR dataset. We showthe mean and standard deviation of these 5 runs for eachmethod. To be fair, for each method we tuned the hyper-parameters to achieve the best accuracy of that method. ForFL scenarios, we have divided the dataset equally among 10hospitals (K = 10) in an independent and identically dis-tributed (iid) manner, thus each hospital owns 293 images.For F and FPDP we perform FedAvg (McMahan et al.2017), where each hospital performs 5 local epochs in be-tween each global epoch, and at each round half of the hos-pitals are randomly chosen to participate. The details of thehyper-parameters used are provided at https://github.com/ipc-lab/private-ml-for-health/tree/main/private training/

We see that Dopamine’s bound is very close to that ofthe centralized training counterpart CDP, and even betterthan CDP at early epochs. Dopamine achieves a better clas-sification accuracy than FPDP and more importantly muchbetter record-level privacy guarantee. In FPDP, each hos-pital needs to add noise with a variance K times larger toget the same privacy level of CDP. In our experiments, wefound that this amount of noise makes learning impossible.Finally, we see that F, compared to C, has a competitiveperformance, but there still is a considerable accuracy gapbetween F and Dopamine due to privacy protection pro-vided. Such a gap can be mitigated if we could get access

Figure 1: Comparison of (top) classification accuracy (big-ger better) vs. (bottom) DP bound (smaller better) on DRdataset. Dopamine achieves a DP bound very close to thecentralized training counterpart, while it also achieves a bet-ter classification accuracy than FPDP. For all DP methods,δ = 10−4. For Dopamine, each epoch is equal to one round(see Figure 2 in Appendix for a more details)

to a larger dataset and proportionally reduce the amount ofnoise without weakening the privacy protection.

4 Conclusion and Future WorkWe proposed Dopamine, a system for collaboratively train-ing DNNs on private medical images distributed across sev-eral hospitals. Experiments on a benchmark dataset of DRimages show that Dopamine can outperform existing base-lines by providing a better utility-privacy trade-off.

The important future directions are: (i) we aim to combinethe proposed secure aggregation functionality (discussed inAppendix D) to the Dopamine’s training pipeline. (ii) Cur-rent privacy-analysis only allows us to have one local it-eration. We observed that allowing local hospitals to per-form more than one iteration can improve the accuracy byabout 3%. We will carry out the privacy analysis for thiscase in the future. (iii) Existing DP libraries do not allowusing arbitrary DNN architectures. They are either not com-patible with more recent architectures, like EfficientNet, orthey need too much resources. Making DP training faster isstill an open area of research and engineering. (iv) Finally,we aim to evaluate Dopamine on other medical datasets, andinvestigate novel methods for accuracy improvement whileretaining the privacy protection provided.

ReferencesAbadi, M.; Chu, A.; Goodfellow, I.; McMahan, H. B.;Mironov, I.; Talwar, K.; and Zhang, L. 2016. Deep learningwith differential privacy. In Proceedings of the 2016 ACMSIGSAC Conference on Computer and Communications Se-curity, 308–318.

Bittau, A.; Erlingsson, U.; Maniatis, P.; Mironov, I.; Raghu-nathan, A.; Lie, D.; Rudominer, M.; Kode, U.; Tinnes, J.;and Seefeld, B. 2017. Prochlo: Strong privacy for analyt-ics in the crowd. In Proceedings of the 26th Symposium onOperating Systems Principles, 441–459. ACM.

Bonawitz, K.; Eichner, H.; Grieskamp, W.; Huba, D.; Inger-man, A.; Ivanov, V.; Kiddon, C.; Konecny, J.; Mazzocchi,S.; McMahan, H. B.; Van Overveldt, T.; Petrou, D.; Ram-age, D.; and Roselander, J. 2019. Towards federated learningat scale: System design. In Proceedings of the 2nd SysMLConference, Palo Alto, CA, USA.

Carlini, N.; Liu, C.; Erlingsson, U.; Kos, J.; and Song, D.2019. The secret sharer: Evaluating and testing unintendedmemorization in neural networks. In 28th USENIX SecuritySymposium (USENIX Security 19), 267–284.

Choi, J. Y.; Yoo, T. K.; Seo, J. G.; Kwak, J.; Um, T. T.;and Rim, T. H. 2017. Multi-categorical deep learning neuralnetwork to classify retinal images: A pilot study employingsmall database. PLOS ONE 12: 1–16.

Choudhury, O.; Gkoulalas-Divanis, A.; Salonidis, T.; Sylla,I.; Park, Y.; Hsu, G.; and Das, A. 2020. Anonymizing Datafor Privacy-Preserving Federated Learning. arXiv preprintarXiv:2002.09096 .

Cormode, G.; Jha, S.; Kulkarni, T.; Li, N.; Srivastava, D.;and Wang, T. 2018. Privacy at scale: Local differential pri-vacy in practice. In Proceedings of the 2018 InternationalConference on Management of Data, 1655–1658.

Corrigan-Gibbs, H.; and Boneh, D. 2017. Prio: Private, ro-bust, and scalable computation of aggregate statistics. In14th USENIX Symposium on Networked Systems Design andImplementation (NSDI’17), 259–282.

Ding, B.; Kulkarni, J.; and Yekhanin, S. 2017. Collectingtelemetry data privately. In Advances in Neural InformationProcessing Systems, 3571–3580.

Doroz, Y.; Cetin, G. S.; and Sunar, B. 2016. On-the-fly Ho-momorphic Batching/Unbatching. In Financial Cryptogra-phy Workshops.

Dunnmon, J. A.; Yi, D.; Langlotz, C. P.; Re, C.; Rubin,D. L.; and Lungren, M. P. 2019. Assessment of convolu-tional neural networks for automated classification of chestradiographs. Radiology 290(2): 537–544.

Dwork, C.; McSherry, F.; Nissim, K.; and Smith, A. 2006.Calibrating noise to sensitivity in private data analysis. InTheory of cryptography conference, 265–284. Springer.

Dwork, C.; and Roth, A. 2014. The algorithmic foundationsof differential privacy. Foundations and Trends in Theoreti-cal Computer Science 9(3-4): 211–407.

Dwork, C.; Smith, A.; Steinke, T.; and Ullman, J. 2017. Ex-posed! a survey of attacks on private data .

Erlingsson, U.; Feldman, V.; Mironov, I.; Raghunathan,A.; Talwar, K.; and Thakurta, A. 2019. Amplifica-tion by shuffling: From local to central differential pri-vacy via anonymity. In Proceedings of the Thirtieth An-nual ACM-SIAM Symposium on Discrete Algorithms, 2468–2479. SIAM.

Erlingsson, U.; Pihur, V.; and Korolova, A. 2014. Rap-por: Randomized aggregatable privacy-preserving ordinalresponse. In Proceedings of the 2014 ACM SIGSAC con-ference on computer and communications security, 1054–1067.

Esteva, A.; Kuprel, B.; Novoa, R. A.; Ko, J.; Swetter, S. M.;Blau, H. M.; and Thrun, S. 2017. Dermatologist-level clas-sification of skin cancer with deep neural networks. nature542(7639): 115–118.

Fan, J.; and Vercauteren, F. 2012. Somewhat Practical FullyHomomorphic Encryption. Cryptology ePrint Archive, Re-port 2012/144.

Gao, D.; Ju, C.; Wei, X.; Liu, Y.; Chen, T.; and Yang, Q.2019. HHHFL: Hierarchical Heterogeneous Horizontal Fed-erated Learning for Electroencephalography. arXiv preprintarXiv:1909.05784 .

Gentry, C. 2009. A Fully Homomorphic Encryption Scheme.Ph.D. thesis, Stanford, CA, USA.

Gulshan, V.; Peng, L.; Coram, M.; Stumpe, M. C.; Wu, D.;Narayanaswamy, A.; Venugopalan, S.; Widner, K.; Madams,T.; Cuadros, J.; Kim, R.; Raman, R.; Nelson, P. C.; Mega,J. L.; and Webster, D. R. 2016. Development and Valida-tion of a Deep Learning Algorithm for Detection of DiabeticRetinopathy in Retinal Fundus Photographs. JAMA 316(22):2402–2410.

Iandola, F. N.; Han, S.; Moskewicz, M. W.; Ashraf, K.;Dally, W. J.; and Keutzer, K. 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and¡ 0.5 MBmodel size. arXiv preprint arXiv:1602.07360 .

Kaissis, G. A.; Makowski, M. R.; Ruckert, D.; and Braren,R. F. 2020. Secure, privacy-preserving and federated ma-chine learning in medical imaging. Nature Machine Intelli-gence 1–7.

Kifer, D.; Messing, S.; Roth, A.; Thakurta, A.; and Zhang,D. 2020. Guidelines for Implementing and Auditing Differ-entially Private Systems. arXiv preprint arXiv:2002.04049.

Li, W.; Milletarı, F.; Xu, D.; Rieke, N.; Hancox, J.; Zhu, W.;Baust, M.; Cheng, Y.; Ourselin, S.; Cardoso, M. J.; et al.2019. Privacy-preserving federated brain tumour segmen-tation. In International Workshop on Machine Learning inMedical Imaging, 133–141. Springer.

Li, X.; Gu, Y.; Dvornek, N.; Staib, L.; Ventola, P.; and Dun-can, J. S. 2020. Multi-site fmri analysis using privacy-preserving federated learning and domain adaptation: Abideresults. arXiv preprint arXiv:2001.05647 .

Lyubashevsky, V.; Peikert, C.; and Regev, O. 2013. On IdealLattices and Learning with Errors over Rings. J. ACM 60(6).ISSN 0004-5411.

McMahan, B.; Moore, E.; Ramage, D.; Hampson, S.; andy Arcas, B. A. 2017. Communication-efficient learning ofdeep networks from decentralized data. In Artificial Intelli-gence and Statistics, 1273–1282. PMLR.

Pfohl, S. R.; Dai, A. M.; and Heller, K. 2019. Federatedand Differentially Private Learning for Electronic HealthRecords. arXiv preprint arXiv:1911.05861 .

Qian, N. 1999. On the momentum term in gradient descentlearning algorithms. Neural networks 12(1): 145–151.

Regev, O. 2009. On Lattices, Learning with Errors, RandomLinear Codes, and Cryptography. J. ACM 56(6). ISSN 0004-5411.

SEAL. 2020. Microsoft SEAL (release 3.5). https://github.com/Microsoft/SEAL. Microsoft Research, Redmond, WA.

Sheller, M. J.; Reina, G. A.; Edwards, B.; Martin, J.; andBakas, S. 2018. Multi-institutional deep learning model-ing without sharing patient data: A feasibility study on braintumor segmentation. In International MICCAI BrainlesionWorkshop, 92–104. Springer.

Shokri, R.; Stronati, M.; Song, C.; and Shmatikov, V. 2017.Membership inference attacks against machine learningmodels. In 2017 IEEE Symposium on Security and Privacy(SP), 3–18. IEEE.

Titus, A. J.; Kishore, S.; Stavish, T.; Rogers, S. M.; and Ni,K. 2018. PySEAL: A Python wrapper implementation of theSEAL homomorphic encryption library.

Truex, S.; Baracaldo, N.; Anwar, A.; Steinke, T.; Ludwig,H.; Zhang, R.; and Zhou, Y. 2019. A Hybrid Approach toPrivacy-Preserving Federated Learning. In Proceedings ofthe 12th ACM Workshop on Artificial Intelligence and Secu-rity, AISec’19, 1–11. New York, NY, USA: Association forComputing Machinery. ISBN 9781450368339.

Wang, T.; Blocki, J.; Li, N.; and Jha, S. 2017. Locally differ-entially private protocols for frequency estimation. In 26thUSENIX Security Symposium (USENIX Security 17), 729–745.

A Related Work(Sheller et al. 2018) apply FL, without any privacy guar-antee, for the segmentation of brain tumor images using adeep convolutional neural network, known as U-Net. Theycompare FL with a baseline collaborative learning scenariowhere hospitals iteratively train a shared model in turn, andshow that FL outperforms this baseline.

(Pfohl, Dai, and Heller 2019) employ FL with FedAvgand DPSGD algorithms to train logistic regression and feed-forward networks with one hidden layer for in-hospital mor-tality and prolonged length of stay prediction using the pa-tients electronic health records. Comparing FL with a cen-tralized approach, when DP is used in both cases, their ex-perimental results show that, while centralized training can

achieve a good DP bound with ε ≈ 1, FL with DP performspoorly in terms of both accuracy and ε.

Considering the differences in type of the device, numberof sensors, and the placement of the sensors that is used ineach hospital for collecting electroencephalography (EEG)data, (Gao et al. 2019) propose an application of FL for train-ing a classifier over heterogeneous data; however, they donot consider any formal privacy guarantee. (Li et al. 2020)use FL among hospitals for training an fMRI image classifierand add some noise to the trained parameters before sharingthem with the server which does not follow the requirementof a valid DP mechanism, thus it does not provide a formalprivacy guarantee.

(Choudhury et al. 2020) offer a k-anonymity based pri-vacy guarantee to generate a syntactic tabular dataset, in-stead of the original dataset, and use it for participating inFL. (Li et al. 2019) propose an FL system for brain tu-mour segmentation in a method where each hospital ownsa dataset of MRI scans. However, their implementation doesnot satisfy record-level DP, but parameter-level DP, whichfor neural networks is not a meaningful privacy protection.

B BackgroundDifferential Privacy (DP)When we want to publish the results of a computationF(D),over a private dataset D, DP helps us to bound the ability ofan adversary in distinguishing whether any specific sampledata was included in the dataset D or not. DP has a worst-case threat model, assuming that all-but-one can collude andadversaries can have access to any source of side-channelinformation.

Definition 1 (Differential Privacy). Given ε, δ ≥ 0, amechanism (i.e. algorithm) M satisfies (ε, δ)−differentialprivacy if for all pairs of neighboring datasets D and D′ dif-fering in only one sample, and for all Y ⊂ Range(M), wehave

Pr(M(D) ∈ Y

)≤ eε Pr

(M(D′) ∈ Y

)+ δ, (1)

where ε is the parameter that specifies the privacy loss (i.e.eε), δ is the probability of failure in assuring the upper-bound on the privacy loss, and the probability distributionis over the internal randomness of the mechanismM, hold-ing the dataset fixed (Dwork et al. 2006; Dwork and Roth2014). When δ = 0, it is called pure DP, which is strongerbut less flexible in terms of the mechanism design.

For approximating a deterministic function F , such as themean or sum, a typical example of M in Equation (1) isa zero-mean Laplacian or Gaussian noise addition mecha-nism (Dwork and Roth 2014), where the variance is chosenaccording to the sensitivity of F that is the maximum of theabsolute distance |F(D)−F(D′)| among all pairs of neigh-boring datasets D and D′ (Dwork et al. 2006). In general, tobe able to calculate the sensitivity of F , we need to know,or to bound, the Range(F); otherwise we cannot design areliable DP mechanism.

Definition 2 (Gaussian Mechanism). Gaussian mecha-nism adds a Gaussian noise to a function F(x) so that the

response to be released is (ε, δ)-differentially private. It isdefined as

M(x) = F(x) +N(µ = 0, σ2 =

2 ln (1.25/δ)(∆2F)2

ε2

),

where ∆2F is the L2 sensitivity of F(x) to the neighboringdatasets (Dwork and Roth 2014).

DP mechanisms are in two types: central DP and localDP. In central DP, we consider a model where there is atrusted server, thus the patients original data is shared withthat server. The trusted server then runM on the collecteddata before sharing the result with other parties. In local DP,every patient randomly transforms data at the patient’s sidewithout needing to trust the server or other parties (Erlings-son, Pihur, and Korolova 2014; Wang et al. 2017; Erlings-son et al. 2019). Thus, in terms of privacy guarantee, lo-cal DP ensures that the probability of discovering the truevalue of the user’s shared data is limited to a mathemati-cally defined upper-bound. However, central DP gives suchan upper-bound for the probability of discovering whether aspecific user has shared their data or not.

DP mechanisms are mostly used for computing aggre-gated statistics, such as the sum, mode, mean, most frequent,or histogram of the dataset (Bittau et al. 2017; Ding, Kulka-rni, and Yekhanin 2017; Cormode et al. 2018). DP mecha-nisms can also be used for training machine learning modelson sensitive datasets while providing DP guarantees for thepeople that are included in the training dataset (Abadi et al.2016; Bonawitz et al. 2019; Kifer et al. 2020).

Federated Learning (FL)Many parameterized machine learning problems can bemodeled as a stochastic optimization problem

minw∈Rd

:= ED∼DL(w,D), (2)

where w denotes the model parameters, D is a randomdataset sampled from the true, but unknown, data distribu-tion D, and L is a task-specific empirical loss function.

The core idea of the FL is to solve the stochastic opti-mization problem in (2) in a distributed manner such thatthe participating users do not share their local dataset witheach other but the model parameters to seek a consensus.Accordingly, the objective of K users that collaborate tosolve Equation (2) is reformulated as the sum of K lossfunctions defined over different sample datasets, where eachcorresponds to a different user:

minwL(w) =

1

K

K∑k=1

EDk∼DkL(w,Dk), (3)

whereDk denotes the portion of the dataset allocated to userk. SGD is a common approach to solve machine learningproblems that are represented in the form of Equation (2).FL considers the problem in Equation (3), and aims to solveit using SGD in a decentralized manner.

Homomorphic Encryption (HE)

Fully Homomorphic Encryption (FHE) allows computationof arbitrary functions F on encrypted data, thus enablinga plethora of applications like private computation offload-ing. Gentry (Gentry 2009) was the first to show that FHEis possible with a method that includes the following steps:(i) construct a somewhat homomorphic encryption (SHE)scheme that can evaluate functions of low degree, (ii) sim-plify the decryption circuit, and (iii) evaluate the decryptioncircuit on the encrypted ciphertexts homomorphically to ob-tain new ciphertexts with a reduced inherent noise. The thirdstep is called bootstrapping, and this allows computations offunctions of arbitrary degree. Here, the degree of a functionis the number of operations that must be performed in se-quence to compute the function. For example, to computex2, one multiplication is necessary, therefore its degree is 1,while to compute x3, 2 multiplications (x∗x and x∗x2) arenecessary, and therefore its degree is 2.

The security of homomorphic encryption schemes isbased on the learning with errors problem (Regev 2009),or its ring variant called ring learning with errors (Lyuba-shevsky, Peikert, and Regev 2013), the hardness of whichcan be shown to be equivalent to that of classical hard prob-lems on ideal lattices, which are the basis for a lot of post-quantum cryptography. All FHE schemes add a small noisecomponent during encryption of a message with a publickey, which makes the decryption very hard unless one hasaccess to a secret key. The decrypted message obtained con-sists of the message corrupted by a small additive noise,which can be removed if the noise is “small enough”. Eachcomputation on ciphertexts increases the noise, which ulti-mately grows large enough to fail the decryption. The boot-strapping approach is used to lower the noise in the cipher-text to a fixed level. Addition of ciphertexts increases thenoise level very slowly, while multiplication of ciphertextsincreases the noise very fast. When FHE is only used for ag-gregation, the bootstrapping step, which is a computation-ally expensive procedure, is not usually necessary since thenoise never grows to a large enough size.

While machine learning models compute on floating pointparameters, encryption schemes work only on fixed pointparameters, or equivalently, integer parameters, which canbe obtained by appropriately scaling and rounding fixedpoint values. Given an integer u to encode an integer baseb, a base-b expansion of u is computed, and represented as apolynomial with the same coefficients. The expansion usesa ‘balanced’ representation of integers modulo b as the coef-ficients, that is, when b is odd, the coefficients are chosen tolie in the range

[−(b−1)

2 , (b−1)2

], and when b is even, the co-

efficients are chosen to lie in the range[−b2 ,

(b−1)2

], except

if b = 2, when the coefficients are chosen to lie in the set{0, 1}. For example, if b = 2, the integer 26 = 24+23+21 isencoded as the polynomial x4+x3+x. If b = 3, 26 = 33−30

is encoded as the polynomial x3 − 1. Decoding is done bycomputing the polynomial representation at x = b. To mapan integer to a polynomial ring R = Z[x]/f(x), where f(x)is a monic polynomial of degree d, the polynomial represen-

tation of the integer is viewed as being a plaintext polyno-mial in R. The base b is called the plaintext modulus. Themost popular choice of f(x) is xd + 1, with d = 2n calledthe polynomial modulus. Let q > 1 be a positive integer,then we denote the set of integers [− q2 ,

q2 ] by Zq , and the

set of polynomials in R with coefficients in Zq by Rq . Theinteger q is called the coefficient modulus, and is usually setmuch larger than the plaintext modulus.

For multi-dimensional vectors, the single-instruction-multiple-data technique, also known as batching (Doroz,Cetin, and Sunar 2016), encodes a given array of multipleintegers into a single plaintext polynomial in Rq . A singlecomputation instruction on such a plaintext polynomial isequivalent to simultaneously executing that instruction onall the integers in that array, thus speeding up the computa-tion by many orders of magnitude. The plaintext modulus isassumed to be larger than any integer in the given array. Thelength of the array is assumed to be equal to the degree of thepolynomial modulus d, and in case that its size is less thand, then it is padded with zeros to make its length equal to d.Lagrange interpolation is performed over the array of inte-gers to obtain a polynomial of degree d − 1, thus obtainingthe plaintext polynomial in Rq encoding d integers.

C On the Effect of MomentumsConsider the momentum term (Algorithm 1, line 13) at

round t. For user k, we have

gtk = gtk + βgt−1k =

t−1∑i=0

βigt−ik , (4)

where gtk’s are the noisy gradients (Algorithm 1, line 12).After every round, gtk’s are securely averaged at the

server, such that the averaged gradient at round t becomes

gt =1

K

K∑k=1

gtk =

K∑k=1

t−1∑i=0

βigt−ik =1

K

t−1∑i=0

βiK∑k=1

gt−ik .

(5)Since the term

∑Kk=1 g

t−ik is computed securely, and the

noise of each gt−ik adds up, as it is shown in Lemma 1,∀i ∈ {0, 1, . . . , t − 1}, gt−i is (ε, δ)-DP against the serverand (ε

√K/(K − 1), δ)-DP against other hospitals, where

ε ≤ ε is the DP-bound at round i.Thanks to the post-processing property (Dwork and Roth

2014) of DP, use of the same noisy gradients repetitivelydoes not incur any additional privacy costs. Thus, use of mo-mentum in our setting does not incur additional privacy costeither.

D Secure AggregationSecure aggregation helps in providing a computational DPguarantee, even during the training procedure, so that thehospitals do not need to trust the server. We use theBrakerski-Fan-Vercauteren (BFV) scheme (Fan and Ver-cauteren 2012) for homomorphic encryption. For securityparameter λ, random element s ∈ Rq , ∆ = b qb c, and a dis-crete probability distribution χ = χ(λ), the BFV scheme isimplemented as follows:

• SecretKeyGen (1λ): sample s← χ, and output sk = s.

• PublicKeyGen (sk): sample a← Rq , e← χ, and output

pk =(

[−(as + e)]q ,a). (6)

• Encrypt(pk,m): To encrypt a message m ∈ Rb, letp0 = pk[0] and p1 = pk[1], and sample u, e1, e2 ← χ,and output

ct =([

p0u + e1 + ∆.m]q,[p1u + e2

]q

). (7)

• Decrypt(sk, ct): Set s = sk, c0 = ct[0], and c1 = ct[1],and compute [[b.[c0 + c1s]q

q

]]b. (8)

We use the open source PySEAL library (Titus et al.2018), which provides a python wrapper API for the orig-inal SEAL library published by Microsoft (SEAL). We sim-ulate the FL environment over the message passing inter-face (MPI), with the rank 0 process modeling the server, andthe rank 1, . . . ,K processes model the hospitals that partic-ipate in the training. The plaintext modulus b is chosen tobe 40961, while the polynomial modulus d is chosen to bex4096 + 1. Therefore, each ciphertext can pack d = 4096integers with magnitude less than 40961 for SIMD opera-tions, thus speeding up the communication and computationon encrypted data by 4096 times. The methodology is as fol-lows:

1. Key generation and distribution: A public key and secretkey are generated by the rank 1 process (hospital 1). Thepublic key is broadcasted to all the involved parties, whilethe secret key is shared with the hospitals but not theserver.

2. The server initializes the model to be trained, and sendsthe model to the hospitals in the first round.

3. Each hospital performs an iteration on its local modelbased on Algorithm 1, at the end of which each hospitalhas the updated model.

4. Each active hospital:

• flattens the tensor of model parameters into a 1D arrayw,

• converts the floating point values in the array to 3 digitintegers by multiplying each value by 103, and thenrounding it to the nearest integer,

• partitions the array w into chunks of d = 4096 param-eters, denoted by w1, . . . ,wl, where l = d length(w)

4096 e,• encrypts the chunks w1, . . . ,wl with the public key,

into ciphertexts ct1, . . . , ctl, respectively, using thebatching technique, and sends the list of ciphertexts tothe server.

5. The server adds the received encrypted updates and sendsthe encrypted aggregated model to the hospitals for thenext round.

Figure 2: The top plot in Figure 1, but starting from epoch 1.

6. The hospitals decrypt the received aggregate array w,scale and convert each integer value to a floating pointvalue of correct precision, divide each value by the num-ber of hospitals that were active in the previous round,and finally reshape w back to the structure of the originalmodel.

7. Go to step 3.

At the end of the above procedure, the hospitals send the fi-nal unencrypted updated models to the server, which aggre-gates them and updates the global model for the final time toobtain the final trained model. Notice that we assume hos-pitals are non-malicious, otherwise one needs to use othertechniques, e.g. (Corrigan-Gibbs and Boneh 2017). How-ever, methods such as (Corrigan-Gibbs and Boneh 2017) usesecure multi-party computation, and not homomorphic en-cryption. The method used in (Corrigan-Gibbs and Boneh2017) requires multiple non-colluding aggregators (at leastone honest aggregator). That is different from our setting,where we consider only one global aggregator. Another al-ternative is for the hospitals to send secret shares to the otherhospitals acting as aggregators. However, that would requirethe hospitals to be able to communicate with each other,which is not the best solution in dynamic settings.