Communication Efficiency in Federated Learning:Achievements and Challenges

Osama Shahid∗, Seyedamin Pouriyeh∗, Reza M. Parizi†, Quan Z. Sheng‡, Gautam Srivastava§, Liang Zhao∗∗ Department of Information Technology, Kennesaw State University, Marietta, GA, USA

oshahid@students.kennesaw.edu, {spouriye, lzhao10}@kennesaw.edu† Department of Software Engineering and Game Development, Kennesaw State University, Marietta, GA, USA

rparizi1@kennesaw.edu‡Department of Computing, Macquarie University, Sydney, Australia

michael.sheng@mq.edu.au§Department of Math and Computer Science, Brandon University, Canada

§Research Centre for Interneural Computing, China Medical University, Taichung Taiwansrivastavag@brandonu.ca

Abstract—Federated Learning (FL) is known to perform Ma-chine Learning tasks in a distributed manner. Over the years,this has become an emerging technology especially with variousdata protection and privacy policies being imposed FL allowsperforming machine learning tasks whilst adhering to thesechallenges. As with the emerging of any new technology, thereare going to be challenges and benefits. A challenge that exists inFL is the communication costs, as FL takes place in a distributedenvironment where devices connected over the network have toconstantly share their updates this can create a communicationbottleneck. In this paper, we present a survey of the researchthat is performed to overcome the communication constraints inan FL setting.

Index Terms—Security, Privacy, Federated learning, Commu-nication, Machine Learning.

I. INTRODUCTION

A Machine Learning (ML) model for a specific task iscreated utilizing the unprecedented amount of data that isgenerated from devices. This data can be used to achieveprocess optimization [1], gain insight discovery [2] and aid indecision making [3]. Some examples of devices that generateddata can be smartphones and wearable devices, smart homes,etc. Traditionally, to implement ML predictive models the datawould need to be transferred to a central server where it couldbe stored and used for training and testing ML models thatare designed for specific tasks [4]. However, the imposition ofprivacy and data-sharing policies like Global Data ProtectionRegulations (GDPR) [5] the traditional centralized method oftransferring data to the server could present more challenges.In addition to this, other computational challenges are presentwith this traditional approach [4]. An alternative solution forcreating reliable ML models for tasks needs to be considered.

Federated Learning (FL) is an innovative way to implementML algorithms and models over decentralized data. Firstintroduced by the research team at Google [6]. FL attempts toanswer the question [7] can we train machine learning models

Corresponding author: S. Pouriyeh (email: spouriye@kennesaw.edu).

without needing to transfer user data onto a central server?Since its’ introduction, there has been a growing interest inthe research community to explore the opportunities and ca-pabilities of FL. The technology enables a more collaborativeapproach of ML whilst also preserving user privacy by havingthe data decentralized over the device itself rather than haveit over a central server [8].

This method of collaborative learning can be explainedby using data generated from hospitals as an example. Eachhospital generates data from its’ smart devices and equipment,using this data could be useful to create an ML model for aspecific task.

However, data from a single hospital may not generateenough data [9]. Limitation of data could limit the overallknowledge and performance of the ML model. To create amore robust model that would be able to obtain a higherprediction accuracy access to a larger data set would be better.Other hospitals are also generating their data, however, due todata-sharing policies, it would not be possible to have accessto that data. This is where FL and its collaborative nature canthrive where each hospital can train its’ own independent MLmodel that gets uploaded to a central server where all modelsaverage out as a global ML model for the specific task.

This is just one example of the capability of FL is beingfurther researched by a range of industries to maximize thisdecentralized approach. Industries and applications such astransportation [10], autonomous vehicles [11], and a rangeof other Internet of Things (IoT) applications [12], [13].Its implementation can be seen on the mobile applicationGboard by Google for predictive texting; the FederatedAver-aging algorithm to train the ML model over the on-devicedecentralized data available on mobile devices can improvepredictive texting results whilst also reducing the privacy andsecurity risks of the sensitive user data compared to a centralserver [14]–[17].

In addition to preserving data privacy and decentralizingthe data, theoretically, the computational power is also split

arX

iv:2

107.

1099

6v1

[cs

.LG

] 2

3 Ju

l 202

1

Fig. 1. How FL can be used to improve predictive healthcare ML usingsensitive data across multiple hospitals [18].

amongst parties that are within the federated network usingthe decentralized algorithms. Rather than having to rely onthe centralized server for training the ML process, the trainingprocess can take place on end devices directly.

FL and more of its capabilities are still being discovered,though, as with the introduction of any new technology themore benefits that are reaped of its capabilities the morechallenges are exposed. Concerning FL, there are still someprivacy, security, communication, and algorithmic challengesthat still exist [19].

In this survey paper, we solely discuss the challengesthat are present in Communication concerning FL [20].The challenges that arise in an FL setting can come frommultiple sources creating a bottleneck. Factors such as end-user network connections that operate at substantially lowerrates when compared to network connections that are availableat a data center. This form of unreliability could be potentiallyexpensive [20]. Therefore there has been some research donetowards making the Communication factor of an FL environ-ment more efficient. Quantization and Sparsification are twomodel compression techniques that can be integrated withthe FL averaging algorithms to make communication moreefficient, along with other techniques that are further discussedin this paper. There has been some research towards this butvery limited surveys are found on the topic, therefore, thissurvey paper aims to present itself by providing a summary ofrecent research done on the subject so further researchers canbenefit from it. Here: The remainder of the paper is structuredas follows. Section II provides an introduction to FL systemand how it functions. Section III introduces and answers theResearch Questions (RSQs). Sections IV and V review thepapers where we share our thoughts and navigate towardssharing the future expectations towards this topic of reasearch.

II. PROBLEM STATEMENT

A. Background

Federated learning was first introduced by the research teamat Google [6], the team at Google had the motive to createmachine learning models that would be applicable for thewealth of data that is available on mobile devices. Federated

Learning was introduced so users could retain hold on to theirdata and retain privacy. In FL the ML models can be directlytrained on the device itself. As the data can be sensitive andalso large in quantity in many cases the approach of bringingthe model to the data could suit better. This decentralizedapproach integrated with collaborative learning was given theterm Federated Learning.

In this subsection, a general overview and the main com-ponents/entities that are available in an FL environment areintroduced. Additionally, we will discuss the FL processesfrom the communication angel.

B. The components of FL systems

FL system is a learning process where users/participants cancollaboratively train the ML model [21]. As described by [22]there are two main entities or components that are present inan FL environment (i) The data owners, or participants and(ii) The central server. The data owners produce the data ontheir end devices (e.g. mobile phones), this data generated bythe owners is kept private to them and does not leave thedevice therefore each participant or device has their privatedataset. The central server is where the global model stays. Itis where the original model is stored and shared across with allthe participants that are connected across the FL environment.This model is then a local model for each device i.e. beingtrained on that independent device’s dataset. Once all thedevice training is complete and model parameters are obtainedeach device uploads its local model back to the central server.Here at the central is also where each updated local modelis uploaded by the participating device and the overall modelaggregation takes place by applying aggregation algorithmstakes place to generate a new global model.

1) The processes of an FL system: FL allows a promisingapproach towards collaborative machine learning whilst pre-serving privacy [23]. The process of an FL has communicationbetween the two main entities. The two entities, the partici-pating devices and the central server communicate first whendevices download a global model from the central server [24].Each device trains the model based on their private datasetand improves the global model, the computation takes placedirectly on end devices. Each device then uploads its retrainedversion of the global model to the central server where anaggregation algorithm merges the model parameters from eachparticipating device into one new generated global model [23].

Fig. 2. A general overview of how FL works [8].

2) The different types of FL systems: The base concept ofFL is to be able to utilize data that is shared across multipledevices and domains. There are a few ways this data can bedistributed across devices, ways such as partitioned by exam-ples or partitioned by features [20]. The categorization of thisdata can be a prerequisite step for building an FL environment[25]. Some characteristics of data distribution are factors suchas heterogeneous data and the clients’ participation. In thissubsection, a brief introduction is done about the few FLsettings that can be applied based on the distribution of dataand other characteristics.

There are a few types of FL systems, based on the way thedata is distributed across the environment [26], they can becategorized as:

• Horizontal Federated Learning: This type of FL systemis when the data from various devices have a similarset of features in terms of the domain but with differentinstances. This is the original sense of FL learning wheredata from each domain is homogeneous, see in figure 3,and contributing together to train the global ML modeltogether [25]. This can be explained using the originalexample that is presented by Google [6] wherein theglobal model is an aggregate of locally trained multipleparticipating devices [27].

Fig. 3. Horizontal FL [27].

• Vertical Federated Learning: The data that is distributedacross in a Vertical FL setting is data that is commonbetween unrelated domains. This could perhaps be calleda feature-based learning scheme as the datasets involvedin the training process perhaps share the same sampleID space but may differ in feature space. An examplecould be where a bank and an e-commerce businessin the same city have some form of the mutual userbase, shown in figure 4. The bank has user-sensitive datasuch as credit card rating, or revenue. Whereas, the e-commerce business has a purchase and browsing history.Here, two different domains can use their data to maybecreate a prediction model based on the user and productinformation [21].

• Federated Transfer Learning: This type of system isdifferent from the aforementioned systems where neitherthe samples nor the features have many similarity [28].An example could be where two data sources such as abank in the United States and an e-commerce business in

Fig. 4. Vertical FL [27].

Canada are restricted by geography but still have a smallintersection with each other being different institutionssimilar to a vertical FL. However, this is just how the datais partitioned by the ML model is similar to the traditionalML method of transfer learning where the ML model isa pre-trained model on a similar dataset is used. Thismethod can provide better results in some cases comparedto a newly built ML model [25], this is further shown infigure 5.

Fig. 5. Federated Transfer Learning [25].

C. Publication Analysis

Over the recent years, Federated Learning has becomea driving research topic for many researchers. Especiallythose who are interested in decentralized Machine Learningtechniques and adhering to the privacy policies that have beenimposed. The catalog of research can often times be presentedin the form of survey papers. In this subsection we review andintroduce the multiple survey papers that have been conductedrecently and made available for this domain. Additionally, wehighlight our contribution and how our work is different fromothers.

Authors in [8] provide a comprehensive and detailed surveyon FL. This research done by authors is quite thorough withintroducing FL and the different types of FL systems that exist.The authors provide a study with a focus on the software thatallows FL to happen, the hardware that platforms, the protocol,and the possible applications for the emerging technology.

Similarly the authors in [29] introduce the implicationsand the practically of FL in healthcare providing general

solutions. The authors discuss and introduce some open ques-tions towards data quality, model precision, integrating expertknowledge, etc. A survey that presents some challenges thatcan be present in the integration of FL is presented in [22].The authors discuss on Mobile Edge Computing (MEC) andthe implementation of FL and how it could act as a dependabletechnology for the collaborative machine learning approachover edge networks.

The authors in [30] conduct a survey on federated learn-ing and the potential threats that might be available to thisapproach. As with any new discovery of research and ad-vancements in technology there are always vulnerabilities andpotential security threats that can provide a lack of robustness.The survey paper [30] introduces the risk of privacy andhow that could potentially be exploited in a FL setting.Furthering, the paper the authors present a brief review on howPoisoning Attacks can interfere and manipulate the outcomeof a FL training model, attacks such as data poisoning andmodel poisoning. Similarly how inference attacks which canalso lead towards privacy leakage. Furthering this topic ofresearch, another survey conducted by [25] expands moreon the threats towards FL as a survey by creating ResearchQuestions with regards to security and privacy and answeringthem by providing a comprehensive list of threats and attacks,the authors [25] also surface some unique security threatsexist in the FL environment and what defensive techniquescan be implemented towards such vulnerabilities. Defensetechniques such as proactive defenses that is a way to guess thethreats and the potential risks associated with it. The authorspresent solutions like Sniper, Knowledge distillation, anomalydetection, and moving target defense.

There are more surveys that are conducted on the topic [31][32] and even though they introduce the topic of communica-tion and the challenges that may be present. There has not beena dedicated research or survey paper that is written towardsthe challenges that are present in communication and ways tomake it efficient. This paper aims to bridge that gap by solelyfocusing on communication.

A list of recent surveys over the past few years are men-tioned in table I. There is a common theme with most surveypapers with survey papers with introducing the technology,presenting the applications and addressing the security andprivacy benefits and concerns. In most papers the topic ofcommunication is introduced too, though, it is only a briefsurface level part of the paper. This survey paper aims to bethe bridge in that gap and present a survey paper that focusessolely on the communication component for FL.

III. RESEARCH QUESTIONS & COMMUNICATIONEFFICIENT METHODS

Communication between the participating devices and thecentral server is an essential step for an FL environment.The model is communicated over rounds of communicationthat download, upload and train the ML model. However, asmentioned, this comes with its’ challenges. To better gainmore insight into the challenges that are present in an FL

environment with regards to communication we pose twoResearch Questions (RSQs). The first RSQ1 aims to providea brief analysis of What are some of the challenges that arepresented in FL with regards to communication? The secondRSQ2 provides an analysis in-depth about various methodsand approaches that can be implemented to answer How canwe make communication more efficient in an FL environment?

A. RSQ1 - What are some of the challenges that are presentedin FL with regards to communication?

As aforementioned in Section II about the generalworking of the FL environment the central server sharesthe global model across the network to all the devicesthat are participating. The total number of devices that areparticipating in this FL environment can sometimes be inmillions and the bandwidth over which the devices areconnected to the environment could be relatively slow orunstable [33]. In an FL training environment, there can bemany rounds of communication that exist between the centralserver and all the participating devices. Over a singularcommunication round the global model is shared across allthe devices in the FL environment and each participatingdevice downloads the global model to train it on their localdataset. A version of that is uploaded back to the environmentto the central server. Therefore there is a constant downlinkand uplink during the communication rounds [34]. Thoughdue to limited bandwidth, energy and power on the deviceend these rounds of communication can be slow [31]. Evenwith these challenges, the overall communication cost ofsharing model updates is relatively lower than sharing copiousamounts of data from the devices to a central server; it isstill important to preserve the communication bandwidthfurther to make it more efficient [25]. In this subsection, weintroduce the overheads [35] or challenges that could createa communication bottleneck in an FL environment.

• Number of Participating Devices: Though having a highnumber of participating devices in an FL environment hasits’ advantage wherein the ML model could be trainedon more data and a possible increase in performance andaccuracy. However, the large number of devices that areparticipating in multiple FL training rounds at the sametime could create a communication bottleneck too. Insome cases, a high number of clients could lead to anincrease in the overall computational cost too [25], [36].

• Network Bandwidth: Though in contrast to the traditionalML approach, the FL approach does reduce the costsubstantially, however, the communication bandwidth stillneeds to be preserved [25]. The participating devicesmay not always have the bandwidth needed and could beparticipating under unreliable network conditions. Factorssuch as having a difference between upload speed anddownload speed that could result in delays such modeluploads by participants [22] to the central server whichcould lead to potential bottleneck leading to disruptingthe FL environment.

TABLE IPUBLICATION ANALYSIS OF SURVEY PAPERS OVER THE PAST COUPLE OF YEARS.

Reference Year Objective Security Privacy Communication Challenges Future Direction

[32] 2019 A comprehensive review of the FL systems, touching and introducing range of FL components. 3 3 3 3 3[21] Providing a survey on existing works on FL discussing frameworks, concepts and applications. 3 3 7 3 3[30]

2020

Introduction of concept of FL, covering threat model attacks predominantly. 3 3 7 3 3[8] A FL survey focusing on hardware, software and technologies and real-life applications 3 3 3 3 3[22] Introduction of FL presenting some existing challenges and their solutions 3 3 3 3 3[31] A detailed survey introducing FL and the challenges. 3 3 3 3 3[29] 2021 A survey of FL in healthcare, covering common topics of introduction of technology, challenges, etc. 3 3 3 3 3[25] A comprehensive survey posing Research Questions with regards to FL and privacy and security 3 3 3 3 3

• Limited Edge Node Computation: The computation isnow dependant on edge devices rather than powerfulGPUs and CPUs. The edge devices could have limitationstowards computation, power resources, storage and asaforementioned limited link bandwidth [37]. The authorsin [37] give a comparison in training time between acentral server and an edge device. They elaborate thatan Image Classification model with over 60 millionparameters can be trained in just a few minutes over aGPU reaching speeds of 56Gbps. However, even witha powerful smartphone connected over 5G it could takemuch longer reaching an average speed of 50Mbps.

• Statistical Heterogeneity: Another possible source for acommunication bottleneck or where communication costscan rise could be statistical heterogeneity where the datais non independent and identically (non-i.i.d.) distributed[38]. In a FL environment, the data is only locallypresent on each participating device. It is gathered andcollected by the participants on their independent devicesbased on their usage pattern and local environment. Anindividual participant’s dataset in an FL environmentcould not be representative of the population distributionof the other participants and their datasets [6]. Thesize of data gathered and distributed amongst devicescan typically vary heavily [39]. Therefore, this typeof fluctuation in the size of the dataset could affectcommunication by causing a delay in model updatesand other attributes. A device with a larger dataset couldtake longer to update, whereas a device with a smallerdataset could be done with updates. However, the globalmodel might not be aggregated until all individual clientmodels are trained and uploaded causing a bottleneck.

The aforementioned constraints listed are just some of thediscovered challenges that could be possible sources towardscreating a communication bottleneck.

B. RSQ2 - How can we make communication more efficientin an FL environment?

To train the global ML model with decentralized data, theglobal model needs to be downloaded on the participating de-vices on that federated network. This allows the data generatedby those remote devices to remain preserved on the devicewhilst subsequently also making it possible to improve the MLmodel. The steps that make this possible can be summarizedinto three communication steps (1) the global model needs

to be shared across devices within the federated network,these can sometimes be millions of IoT devices of mobilesphone (2) The model is then downloaded by the devicesand trained locally on-device on the private dataset that isgenerated by those devices (3) the ML model is uploaded backto the central server where it is pooled with numerous othermodels that have been uploaded to aggregate them all togetherand find a federated average to generate a new and updatedglobal model. Considering the steps of communication thatpart-take to obtain an improvement in the global ML modelit is important to seek out the most communication-efficientmethods that could make the transfer of data from (1) to (2)and onto (3).

In this subsection, we present the recent findings and effortsthat are made by the research community to improve thecommunication of the ML model over the federated network.Findings that are tackling constraints where devices can dropout due to poor or limited network bandwidth. In addition tothis, other constraints too where data is sampled differentlyand how these could affect communication. We list someresearch towards discovering different methods that couldhelp with making communication more efficient. Methodssuch as Local Updating, Client Selection, Reduction in ModelUpdates, Decentralized Training & Peer-to-Peer Learning, andvarious Compression Schemes.

Fig. 6. [39] illustrate a complete round of distributed Stochastic GradientDescent (SGD) model. In (a) clients on the federated network synchronizewith the server and download the global ML model. In (b) the global modelis trained by each client on their local data, adjusting the weighted averageof the ML model as per. In (c) the new average achieved by each client isupdated onto the server where each update is used to obtain a new weightedaverage for the global model.

1) Local Updating: Distributed ML that exists in datacenters have become popular with integrating mini-batch op-

timization methods, however, there are still some constraintsand limitation on flexibility. This form of batch optimizationhas constraints over a more federated setting that includesboth communication and computation challenges [31], [40].Though the overall objective of local updating is mostlyfocusing on the aforementioned point (2) i.e. to fit andtrain the ML model locally on-device using the data thatis generated by those devices, however, having to computelocally over a singular communication round and then applyingthose updates to the central server is not that efficient dueunexpected dropouts of devices or synchronization latencydue to poor network connectivity [41]. Authors in [6] alsohighlight that it is very important to have a form of flexibilitywhen considering optimization method. Flexibility towardsclient participation and local updates. There are a few waystowards achieving this flexibility and in turn improving theoverall communication efficiency. Techniques such as primal-dual methods can offer a form of flexibility where local devicescan use local parameters to train the global model locally tofind arbitrary approximations based [42]. This leveraging andbreaking down the overall objective of the global model canallow for problems or training to be solved in parallel overeach communicated round [40]. In this subsection, we reviewsome of the methods that have come surface with regards tolocal updating techniques that can make communication moreefficient.

In an FL setting the data can be distributed unevenlyacross devices and may not always be identical. This can beconsidered that the distributed data across the FL environmentis in a non-independent and identically distributed manner i.e.non-iid. Having the datasets in a non-id state could challengethe testing accuracy of an FL model [41]. The testing accuracyof a locally trained ML model is important as it will contributeto the global model. Therefore having an accuracy of a localmodel that performs poorly could also deteriorate the overallperformance of the global model.

In [43] the authors introduce their way of combining FLwith Hierarchical Clustering (HC) techniques to reduce theoverall communication rounds within the FL protocol. Theclustering step introduced can cluster and separate the clientsby the similarity of the local updates to the global model.These similarities are the weights that are achieved by locallyupdating the local model on each device. Upon testing andcomparing their integrated FL + HC technique the authorsconcluded the communication rounds are reduced as per acomparative on the Manhattan distance metric [43], [44].

Due to the large number of connected devices that areattempting to communicate their locally updated model pa-rameters with the central server, (2), this could cause acommunication bottleneck due to bandwidth in some instances[45]. The authors in [45] introduce a new way to counterthis problem, their FedPAQ framework i.e. Federated PeriodicAveraging and Quantization framework tackles this problemwith its’ features. Its feature of Periodic Averaging feature, inparticular, helps reduce the number of communication rounds.In contrast to other training methods where each participating

device sends their ML models to synchronize through theparameter servers over each iteration resulting in increasingthe number of communication rounds between the devicesand central sever. However, this method of Periodic Averagingmethod can allow a solution using the Stochastic GradientDescent (SGD). The parameters and local updates of eachdevice can be synchronized with the central server wherea periodic average of models takes place. This is done byadjusting the parameter that corresponds to the number ofiterations that occur locally on the device itself. Other featuressuch as Partial Node Participation and Quantized Message-Passing of the FedPAQ also reduce communication overhead.The Quantized Message-Passing is further discussed in theCompression Schemes subsection.

The aforementioned FedAvg algorithm is used for its sim-plicity and in turn, is about to reduce communication costs inan FL environment. The algorithm tackles the challenges pre-sented over communication by performing numerous updateson available devices before communicating with the centralserver. However, in some cases such as the authors in [46] statethat over heterogeneous data the FedAVG algorithm couldintroduce ’drift’ towards the updates of each client or devicethat is participating. This ’client drift’ if will result in slow andunstable convergence and could persist if all clients participatein training rounds with full batch gradients. The authors runtheir analysis and determine that the FedAvg algorithm fullbatch gradients and matching lower bounds over no clientsampling to conclude that it is slower than the StochasticGradient Descent (SGD) over some parameters [46]. To overthis challenge the authors created their framework calledStochastic Controlled Averaging (SCAFFOLD) algorithm thatcan fix the drift [46]. For their analysis, the SCAFFOLDalgorithm performs well and provides reliable convergencerates as SGD even for non-iid data. It also takes the advantageof the similarity that exists within the clients and reducecommunication overload [47], [48]

An approach introduced by authors in the [49] calledFedDANE tackles some of the practical constraints that arepresent with FL. The approach is a culmination of methodsintroduced in [47], [48]. FedDANE collects gradient updatesfrom subset of devices during each of the communicationrounds. FedDANE works well with low client participationsettings too.

Local updating is an essential part of Federated Learning.Each device needs to compute local updates so a better overallglobal model can be generated and all participating devices canbenefit from it. Making the process of local updating moreefficient could indeed make communication more efficient.

2) Client Selection: Client Selection is an approach thatcan be implemented to make communication more efficient byreducing the costs by restricting the number of participatingdevices so only a fraction of the parameters is updated over thecommunication round [29]. In this subsection, we introducesome of the research that is done with regards to this and howbeneficial it could be towards easing communication overload.

An approach introduced by authors [50] tackles the limita-

tions that are present in communication over an FL setting.The authors [50] create a multi-criteria client selection modelfor IoT devices over an FL setting. Their FedMCCS approachconsiders all the specifications and network conditions of theIoT device for client selection. Things such as CPU, Memory,Energy and Time are all considered for the client resources todetermine whether the client would be able to participate inthe FL task. The FedMCCS approach considers more than oneclient and over each round, the number of clients participatingis increased. From their analysis FedMCCS in comparison toother approaches can outperform by reducing the total numberof communication rounds to achieve reasonable accuracy.

Another factor that is a vital property of FL accordingto authors in [51] is the varying significance of learningrounds. This comes to the surface when the authors realizethat the learning rounds are temporally interdependent but alsohave varying significance towards achieving the final learningoutcome. This conclusion comes from running numerousdata-driven experiments, the authors create an algorithm thatutilizes the wireless channel information but can achieve long-term performance guarantee; the algorithm results in providinga desired client selection pattern that is adapted to networkenvironments.

Authors in [52] introduced a framework that specificallytackles the challenges when it comes to client selection.The authors called their framework FedCS. Their goal isthat when it comes to Client Selection in a standard FLsetting it can sometimes be a random selection of clients (ordevices), however, with their approach of Client Selection theybreak into two steps. First, Resource Request, where clientinformation such as state of the wireless channel and computa-tional capacities, etc are requested and shared with the centralserver. Second, to estimate the time required Distribution andScheduled Update and Upload steps are taken. With theirframework, the overall approach of the Client Selection is toallow the server to aggregate as many clients within a certaintime frame. Knowing these factors such as data, energy, andcomputational resources i.e. used by the devices could bettermeet the requirements of a training task and possibly affectenergy consumption and bandwidth cost.

To better assess whether a client can sufficiently participateor not a resource management algorithm introduced by authorsin [53] adopts a deep-Q learning algorithm that could allow theservers to learn and make optimal decisions without having toknow prior network knowledge. Their Mobile Crowd MachineLearning (MCML) algorithm addresses the constraints thatare present in mobile devices reducing the energy consumedalong with making the training and communication time moreefficient too.

Authors run their analysis providing a method that practicesbiased client selection strategies [54]. As per their analysis, theselection of bias can affect the convergence speed, the bias oftheir client selection works towards clients that have a higherlocal loss and are achieving faster error convergence. With thisknowledge, the authors a framework that is a communicationand computation efficient framework. Calling it Power-Of-

Choice. Their framework has the agility to trade-off betweenthe bias and the convergence speed. The results the authorsachieve are computed relatively faster and provide higheraccuracy of results than the baseline random selection methods[54].

Uplink and downlink communication with participatingclients in a federated network is necessary. Having to com-municate with clients that have dependable network bandwidthand energy sources could aid towards achieve a well-trainedglobal model more efficiently. Client selection techniquesimplemented can aid in reducing the cost of overall commu-nication that is needed to achieve a dependable global model.

3) Reducing Model Updates: Once the global model isdownloaded by connected devices in the FL environment eachdevice starts to train the devices locally. As each device com-putes and updates the model these updates are communicatedback to the central server [6]. The number of communicationrounds between the devices and central server can be costlyand perhaps having fewer but more efficient model updatescould be a solution. In this section, we introduce sometechniques that discuss a possible reduction in communicationfor these model updates and potentially reduce the cost.

Authors in [55] introduce an efficient way of trainingmodels, their proposed approach adapts to the concept of driftstrains models equally well through different phases of modeltraining. Their approach leads to reducing the communicationsubstantially without depreciating the predictive performanceof the model.

In another study [56], authors introduce a Partitioned Varia-tional Inference (PVI) for probabilistic models that work wellover federated data. They train a Bayesian Neural Network(BNN) over an FL environment that is allowed for bothsynchronous or asynchronous model updates across manymachines, their proposed approach along with the integrationof other methods could allow a more communication-efficienttraining of BNN on non-iid federated data.

In contrast to most of the current FL methods that includeiterative optimization techniques over numerous communica-tion rounds [6], [57], authors in [58] introduce a one-shotcommunication round approach where only a single round ofcommunication is done between the central server and thenumber of connected devices [58]. The authors do suggest asopposed to computing increments, each device trains a localmodel to completion and then applies ensemble methods to ef-fectively capture information regarding device-specific models.Applying ensemble learning techniques could be better suitedfor global modelling than averaging techniques [58].

In an FL setting the rate at which model convergence occurscan sometimes take a large number of communication roundscreating a delay towards model training whilst simultaneouslyincreasing network resources [59]. An intelligent algorithmcalled FOLB is introduced by authors in [59]. The algorithmperforms smart sampling of participating devices over eachround to optimize the expected convergence speed. The algo-rithm can estimate the participating device’s capabilities, thisis done by adapting to devices aggregations.

If the mere frequency in which the model updates are sharedis reduced that means the overall communication between thedevices and the central server is reduced too. Having an effec-tive way of determining how these updates are computed andreducing the overall communication rounds that are needed.Fewer rounds could result in more efficiency.

4) Decentralized Training & Peer-to-Peer Learning: FLin a way is like practicing ML in a decentralized manner.However, FL does allow a more peer-to-peer learning approachwherein each node that is trained can benefit from the othernode that is trained on the FL network too. Even in decentral-ized training similar challenges of communication exist and totackle it different methods like compression [60] can be used.In this subsection how decentralized or peer-to-peer learningis utilized or integrated into an FL environment.

In an FL environment, as aforementioned, there is a centralserver that has the original model. The central server is whereall the devices that are connected to the network update theirmodel too. Essentially, all devices that are participating in theFL environment are connected to the central server. As statedby [31] the star network is the predominant communicationtopology in an FL setting shown in Figure 7. However, there issome research on whether a different communication topologywhere the participating devices only communicate with oneanother. A Peer-to-Peer learning experience or decentralizedtraining network, and whether this would be a more efficientway to communicate in an FL environment. Authors in [31]state that in traditional data center environments decentralizedtraining can appear to be faster than centralized trainingespecially when constraints of low bandwidth and high latencyare faced when operating on networks. Though this is not tosay that decentralized training does not have its’ constraintsauthors in [61] state that nodes computation time on nodes canslow down the convergence of a decentralized algorithm, inaddition to this, the sometimes, large communication overheadcould inalso further mitigate this. To overcome these, theauthors in [61] proposed QuanTimed-DSGD algorithm, adecentralized and gradient-based optimization, that imposesiteration deadlines for nodes and nodes exchange their quan-tized version of the models.

In the FL environment, peer-to-peer learning could bewhere each device only communicates with its neighboursand updates. The participating devices or clients updates theirmodel on their dataset, and aggregates it along with themodel updates from their neighbours [62]. Furthering that,authors in [63] build a framework for an FL environmenttowards a generic social network scenario. Their Online Push-Sum (OPS) method handles the complex topology whilstsimultaneously having optimal convergence rates.

Similarly, [64] propose a distributed learning algorithmwhere there is no central server but instead, the participatingdevices practice peer-to-peer learning algorithm to iterate andaggregate model information with their neighbour to collabora-tively estimate the global model parameters. The authors basean assumption suggesting that FL setting wherein a centralserver exists and communicates the global model can incur

large communication costs. In their approach, the devices arealready distributed over the network where communicationoccurs only with their one-hop neighbours.

The authors in [65] that further provides another avenuefor peer-to-peer learning and not depending on the centralserver as a single trusting authority. The authors [65] createtheir framework BrainTorrent that does not rely on a centralserver for training, their proposed peer-to-peer framework canis designed to motivate a collaborative environment. Accordingto their analysis, the absence of a central server makes theenvironment resistant to failure but also precludes the needfor a governing central body that every participant trusts.

Theoretically, the application of decentralized training in anFL setting could reduce the communication cost when com-pared with a central server [31] though there more researchdone on this and there are further avenues it can take too.

Fig. 7. An overview of star-network topology. [31]

5) Compression Schemes: In addition to the local updatingmethods, some ML compression techniques may also beimplemented towards reducing the total number of rounds be-tween the centralized server and the devices that are connectedon the federated network. Compression schemes like quanti-zation and sparsification that are sometimes integrated withthe aggregation algorithms can sometimes provide reasonabletraining accuracy results whilst at the same time also reducingthe overall communication cost. The authors in [20] list theobjectives set for compression schemes and federated learningas such:

a. Reducing the size of the object / ML model from theclients to the server i.e. used to update the overall globalmodel

b. Reducing the size of the global model that shared withthe clients on the network, the model on which the clientsstart local training using the available data

c. Any changes that are made to the overall training algo-rithm that makes training the global training model morecomputationally efficient

TABLE IIRESEARCH DONE TOWARDS REDUCING THE OVERALL COMMUNICATION COSTS AND OVERHEADS IN A FL.

Reference Section Model and Technology Remarks[43]

Local UpdatingHierarchical Clustering Technique A FL+HC technique that separates client clusters similarity of local updates

[45] FedPAQ Using periodic averaging to aggregate and achieve global model updates[46] SCAFFOLD Algorithm An algorithm that provides better convergence rates over non-iid data[39]

CompressionSchemes -Sparsification

STC method Providing compression for both upstream and downstream communications[66] FetchSGD Compresses the gradient based off of client’s local data[67] General Gradient Sparsification

(GSS)The batch normalization layer with local gradients achieved from GSS could mitigate the impact ofdelayed gradients and not increase the communication overhead

[68] CPFed A sparsified masking model that can provide compression and differential privacy[69] Sparse Binary Compression (SBS) Introducing temporal sparsity where gradients are not communicated after every local iteration[45]

CompressionSchemes -Quantization

FedPAQ Using quantization techniques based upon model accuracy[70] Lossy FL algorithm (LFL) Quantizing models before broadcasting[71] Hyper-Sphere Quantization (HSQ)

frameworkAbility to reduce the cost of communication per iteration

[72] UVeQFed With the algorithm convergence of model minimizes the loss function[73] Heir-Local-QSGD A technique that naturally leverages client-edge-cloud network hierarchy and quantized models updates[65] Decentralized

Training orPeer-to-peer learning

BrainTorrent A peer-to-peer learning framework where models converge faster and reach good accuracy[61] QuanTimed-DSGD decentralized gradient-based optimization imposing iteration deadlines for devices[50]

Client Selection

FedMCCS A multi-criteria client selection that considers that considers IoT device specification and networkcondition

[51] Resource Allocation Model Optimizing learning performance on how clients are selected and how bandwidth is allocated[52] FedCS The framework allows the server to aggregate as many clients within a certain time-frame[54] Power-Of-Choice A communication and computation efficient client selection framework[55]

Reduced ModelUpdates

A decentralized deep learningmodel

Ability to handle different phases of the model training well

[56] a Partitioned Variational Inference(PVI)

A Bayesian Neural Network over FL that is synchronous and asynchronous for model updates acrossmachines

[58] One-shot federated learning A single round of communication done between central server and connected devices[59] FOLB Intelligent sampling of devices in each round of model training to optimize the convergence speed

From the list of objectives A could have the highest effecton the overall running time of around therefore in reducing thatwould directly result in reducing the overall communicationcost; the clients on a federated network generally have aslower upload bandwidth compared to a download. There-fore compressing the ML models and potentially reducingthe uplink/downlink exchanges could result in reducing thecommunication cost [62].

In this subsection, we review the research that is doneby the community towards compression schemes to makecommunication more efficient in a federated network. Webegin by sharing the research that is done towards compressionschemes, then we further divide the research concerningmethods of sparsification and quantization.

Sparsification: Sparsification techniques is a type of com-munication technique that can be implemented in an FL settingto compress the model when they are being communicatedacross the server, in this section, we review the research thathas been conducted towards making integrating sparsificationin an FL environment and making communication more effi-cient.

The authors in [39] present a sparse ternary compression(STC) method that not only adheres to the compression re-quirements of an FL setting and environment but also providescompression for both upstream and downstream communi-cations. They run an analysis over FL models over variousdatasets and architectures, in their analysis, they conclude thatsome factors of an FL environment are hugely dependant onthe convergence rate of the averaging algorithm. From theiranalysis, the authors deduce that factors where training is done

on non-iid small portions of data or when only a subset ofclients participate in communication rounds can reduce theconvergence rate. However, the proposed model of STC isa protocol that compresses communication via sparsification,ternanization, error accumulation and the optimal Golombencoding. The robust technique provided by authors [39]converges relatively faster when compared to other averagingalgorithms like FedAVG over both factors of non-iid data anda lesser number of iterations that are communicated. The pro-posed method is also highly effective when the communicationbandwidth is constrained.

Sparse client participation is another challenge that needsto be overcome in an FL environment, authors introduce aFetchSGD algorithm that can help towards achieving this [66].FetchSGD overall aids with communication constraints of anFL environment by compressing the gradient that is basedon the client’s local data. The data structure Count Sketch[74] is used to compress the gradient before it is uploadedto the central server. The Count Sketch is also used forerror accumulation. According to authors in [67], one keyproblem with regards to federated setting and communicationis the communication overhead that is involved in parameterssynchronization. The authors inform that this overhead wastesbandwidth and increases training time whilst impacting theoverall model accuracy. To tackle this, the authors propose aGeneral Gradient Sparsification (GGS) framework for adaptiveoptimizers. The framework consists of two important mecha-nisms that are batch normalization updates with local gradients(BN-LG), and gradient correction. The authors determine thatupdating the batch normalization layer with local gradients

could mitigate the impact of delayed gradients and not increasethe communication overhead. The authors run their analysisover several models such as AlexNet, DenseNet-121, CifarNet,etc, and achieve high accuracy results concluding gradientsparsification does have a significant impact in reducing thecommunication overhead. The authors in [68] introduce a com-pression scheme that is both a communication efficient anda deferentially private federated learning scheme. The authorscall it CPFed. The challenge the authors face when addressingboth issues together is that one that rises from data com-pression. Techniques used for compression could sometimeslead to an increased number of training iterations required forachieving some desired training loss due to the compressionerrors, however, differential privacy could deteriorate withregards to the training iterations. To overcome this paradigm,their proposed CPFed is based on a sparsified privacy maskingtechnique that adds random noise to model updates along withan unbiased random sparsifier before updating the model. Theauthors can achieve high communication efficiency throughtheir proposed model.

Authors in [69] propose a compression technique thatcould drastically reduce the communication cost for a dis-tributed training environment. The framework introduced bythe authors is a Sparse Binary Compression (SBC) technique,their method integrated techniques that are already presentin communication delay and gradient sparsification with anovel binarization method. According to their research, theauthors find the current gradient information for training neuralnetworks with SGD is redundant. Instead, the authors utilizecommunication delay methods introduced in [6] to introducetemporal sparsity where gradients are not communicated afterevery local iteration. From their findings, the authors concludethat in a distributed SGD setting both communication delayand gradient sparsity can both be treated as independenttypes of sparsity techniques. These methods provide highercompression gains, though, in their findings the authors alsofind a slight decrease in Accuracy.

Quantization: The communication between the devicesand the central server includes sharing model updates andparameters that have occurred on the device end. This updateof parameters can be strenuous when it comes to up-linking themodel. To aid with this, another compression method referredto as quantization can bring the model parameters to a rea-sonable size without compromising much on model accuracy[75]. In this subsection, we introduce different techniques andframeworks that have integrated quantization towards makingcommunication more efficient for an FL setting.

The authors in [45] utilize their framework FedPAQ toreduce the overall communication rounds and the bearing cost.A feature of their framework is to Quantize Message-Passing.Communication bandwidth will mostly be limited in an FLsetting, with limited uplink bandwidth on participating deviceend could increase the communication cost making it moreexpensive. Authors [45] employ quantization techniques [76]on the up-links; every local model is quantized before beinguploaded hence reduce the overall communication overhead.

Furthering and employing quantization techniques authorsin [70] use them for their FL analysis. In contrast to typicalFL models where the global model from a central server isdownloaded by all devices and then subsequently updated andso and so forth. The authors from their analysis discoveredthat applying quantization techniques to the global modelcan help towards making communication more efficient. TheLossy FL algorithm (LFL) created by the authors quantizes theglobal model before it broadcasts and shares it across with thedevices. The local updates that take place on the device areuploaded on the global central server are also quantized. TheFL environment is hugely dependant on bandwidth and theauthors [70] for their analysis study how well the quantizedcompressed global model performs to provide an estimate forthe new global model from the local updates of the devices.They compare their results with another analysis of a losslessstandard approach of FL to conclude that their LFL andthe technique of quantizing global model updates provide asignificant reduction in communication load.

Traditionally, compression algorithms are trained for asetting where there is a high-speed network, places suchas data centers [77] but in an FL setting these algorithmsmight not be directly very effective. To further tackle thecommunication bottleneck that is created due to the networkand the other aforementioned reasons another quantizationcompression technique is proposed by [71]. To achieve a trade-off between communication efficiency and accuracy authors in[71] propose a Hyper-Sphere Quantization (HSQ) framework.HSQ can reduce communication cost low per-iteration whichis ideal for an FL environment, the framework utilizes vectorquantization techniques that shows an effective approach to-wards achieving gradient compression and simultaneously notcomprising over on convergence accuracy.

In another study, the authors [72] suggest that the commu-nication channel and transfer of model parameters betweenthe users to the central server has a throughput that can betypically constrained. The authors whilst doing their researchencountered that alternative methods to aid with this couldprovide dominant distortion of results. This lead [72] tocreate a quantization method that is more efficient towardsfacilitating the model transfer in an FL setting. Utilizingquantization theory methods, the authors design quantizersthat are suitable for distributed deep network training. Under-standing the requirements that are needed for quantization FLsetting, the authors can propose an encoding-decoding strategy.Their proposed scheme shows potential for an FL setting asit performs relatively well compared to previously proposedmethods. Furthering their research towards quantization theo-ries the authors in their review approach it from identifyingunique characteristics regarding the conveyed trained modelsover rate-constrained channels [78]. The authors propose aUniversal Vector Quantization technique for FL; a quantizationtechnique that would be suitable for such settings calling itUVeQFed. For their research, the authors demonstrate thatcombining universal vector quantization methods can yield asystem where compression of trained models induces only

a minimum distortion. Analyzing the distortion further theauthors determine that the distortion reduces substantially asthe number of users grows.

Authors in [73] introduce a Hierarchical Quantized Feder-ated Learning technique that can leverage client-edge-cloudnetwork hierarchy and quantized model updates. Their Heir-Local-QSGD algorithm performs partial edge aggregation andquantization on the model updates that can result in improvingthe communication efficiency in an FL environment.

IV. DISCUSSION

This paper provides an introduction towards FL whilstretaining focus on the communication component of FL. Com-munication is quite an essential part of the Federated Learningenvironment. It is essentially how the global machine learningmodel is transmitted from the central server to all the par-ticipating devices. Similarly, the model is then trained on thelocal data that is available on the devices, and then uploadedit back to the central server. This constant communicationrequires a lot of download and upload using a reliable networkbandwidth. As it is not always the case, limited bandwidth orpoor client participation this can create a communication lagin completing the FL training. Techniques to make the roundsof communication more efficient are shared in this paper.For example a wide range of compression scheme methodsmentioned in this paper can help towards reducing the com-munication overheads and reducing the costs to some factor.Even though the methods presented in this paper do make thecommunication front more efficient they are all implementedusing a range of resources such as different computationalpower and datasets. There are dependable ways towards whichcommunication rounds can be addressed, however, due to therange of solutions presented it could be concluded that aculmination of techniques implemented together could resultin the most communication robust solution for a FL setting.

V. CONCLUSION & FUTURE EXPECTATIONS

This survey paper aimed to provide a bridge between thegap that was present on the topic of communication in FL. Thispaper presented the challenges and constraints that are presentconcerning this. Such as bandwidth and limited computation.We posed these challenges in the form of two RSQs thatboth introduced the problems and provided a list of solutionsthat have been applied towards making communication moreefficient. We believe that communication is an important factorin an FL setting. It is after all how all the training of MLmodels occur. Despite its’ importance there is still limitedresearch done in comparison to other aspects such as privacy,security and resources. For future expectations we hope tosee research on this topic to further grow towards making theefforts making them more efficient.

REFERENCES

[1] D. Weichert, P. Link, A. Stoll, S. Ruping, S. Ihlenfeldt, and S. Wrobel,“A review of machine learning for the optimization of productionprocesses,” The International Journal of Advanced Manufacturing Tech-nology, vol. 104, no. 5, pp. 1889–1902, 2019.

[2] M. Pazzani, “Comprehensible knowledge discovery: gaining insightfrom data,” in First Federal Data Mining Conference and Exposition.Citeseer, 1997, pp. 73–82.

[3] G. Meyer, G. Adomavicius, P. E. Johnson, M. Elidrisi, W. A. Rush,J. M. Sperl-Hillen, and P. J. O’Connor, “A machine learning approachto improving dynamic decision making,” Information Systems Research,vol. 25, no. 2, pp. 239–263, 2014.

[4] A. L’heureux, K. Grolinger, H. F. Elyamany, and M. A. Capretz,“Machine learning with big data: Challenges and approaches,” IeeeAccess, vol. 5, pp. 7776–7797, 2017.

[5] J. P. Albrecht, “How the gdpr will change the world,” Eur. Data Prot.L. Rev., vol. 2, p. 287, 2016.

[6] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas,“Communication-efficient learning of deep networks from decentralizeddata,” in Artificial Intelligence and Statistics. PMLR, 2017, pp. 1273–1282.

[7] S. Truex, N. Baracaldo, A. Anwar, T. Steinke, H. Ludwig, R. Zhang, andY. Zhou, “A hybrid approach to privacy-preserving federated learning,”in Proceedings of the 12th ACM Workshop on Artificial Intelligence andSecurity, 2019, pp. 1–11.

[8] M. Aledhari, R. Razzak, R. M. Parizi, and F. Saeed, “Federated learning:A survey on enabling technologies, protocols, and applications,” IEEEAccess, vol. 8, pp. 140 699–140 725, 2020.

[9] A. Yazdinejad, G. Srivastava, R. M. Parizi, A. Dehghantanha, K.-K. R. Choo, and M. Aledhari, “Decentralized authentication of dis-tributed patients in hospital networks using blockchain,” IEEE Journalof Biomedical and Health Informatics, vol. 24, no. 8, pp. 2146–2156,2020.

[10] Y. Liu, J. James, J. Kang, D. Niyato, and S. Zhang, “Privacy-preservingtraffic flow prediction: A federated learning approach,” IEEE Internet ofThings Journal, 2020.

[11] S. R. Pokhrel and J. Choi, “Federated learning with blockchain for au-tonomous vehicles: Analysis and design challenges,” IEEE Transactionson Communications, 2020.

[12] Y. Zhao, J. Zhao, L. Jiang, R. Tan, and D. Niyato, “Mobile edgecomputing, blockchain and reputation-based crowdsourcing iot federatedlearning: A secure, decentralized and privacy-preserving system,” arXivpreprint arXiv:1906.10893, 2019.

[13] V. Mothukuri, P. Khare, R. M. Parizi, S. Pouriyeh, A. Dehghantanha,and G. Srivastava, “Federated learning-based anomaly detection for iotsecurity attacks,” IEEE Internet of Things Journal, pp. 1–1, 2021.

[14] A. Hard, K. Rao, R. Mathews, S. Ramaswamy, F. Beaufays, S. Augen-stein, H. Eichner, C. Kiddon, and D. Ramage, “Federated learning formobile keyboard prediction,” arXiv preprint arXiv:1811.03604, 2018.

[15] M. Chen, R. Mathews, T. Ouyang, and F. Beaufays, “Federated learningof out-of-vocabulary words,” arXiv preprint arXiv:1903.10635, 2019.

[16] T. Yang, G. Andrew, H. Eichner, H. Sun, W. Li, N. Kong, D. Ram-age, and F. Beaufays, “Applied federated learning: Improving googlekeyboard query suggestions,” arXiv preprint arXiv:1812.02903, 2018.

[17] S. Ramaswamy, R. Mathews, K. Rao, and F. Beaufays, “Federatedlearning for emoji prediction in a mobile keyboard,” arXiv preprintarXiv:1906.04329, 2019.

[18] Tian Li, “Federated learning: Challenges, methods and futuredirections,” https://blog.ml.cmu.edu/2019/11/12/federated-learning-challenges-methods-and-future-directions/, 2019, Accessed November13, 2020.

[19] S. Niknam, H. S. Dhillon, and J. H. Reed, “Federated learning forwireless communications: Motivation, opportunities, and challenges,”IEEE Communications Magazine, vol. 58, no. 6, pp. 46–51, 2020.

[20] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N.Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings et al.,“Advances and open problems in federated learning,” arXiv preprintarXiv:1912.04977, 2019.

[21] Q. Yang, Y. Liu, T. Chen, and Y. Tong, “Federated machine learning:Concept and applications,” ACM Transactions on Intelligent Systems andTechnology (TIST), vol. 10, no. 2, pp. 1–19, 2019.

[22] W. Y. B. Lim, N. C. Luong, D. T. Hoang, Y. Jiao, Y.-C. Liang, Q. Yang,D. Niyato, and C. Miao, “Federated learning in mobile edge networks:A comprehensive survey,” IEEE Communications Surveys & Tutorials,vol. 22, no. 3, pp. 2031–2063, 2020.

[23] R. Xu, N. Baracaldo, Y. Zhou, A. Anwar, and H. Ludwig, “Hybridalpha:An efficient approach for privacy-preserving federated learning,” inProceedings of the 12th ACM Workshop on Artificial Intelligence andSecurity, 2019, pp. 13–23.

[24] J. Kang, Z. Xiong, D. Niyato, Y. Zou, Y. Zhang, and M. Guizani,“Reliable federated learning for mobile networks,” IEEE Wireless Com-munications, vol. 27, no. 2, pp. 72–80, 2020.

[25] V. Mothukuri, R. M. Parizi, S. Pouriyeh, Y. Huang, A. Dehghantanha,and G. Srivastava, “A survey on security and privacy of federatedlearning,” Future Generation Computer Systems, vol. 115, pp. 619–640,2021.

[26] H. Zhu, H. Zhang, and Y. Jin, “From federated learning to federatedneural architecture search: a survey,” Complex & Intelligent Systems,pp. 1–19, 2021.

[27] Z. Tian, R. Zhang, X. Hou, J. Liu, and K. Ren, “Federboost: Privatefederated learning for gbdt,” arXiv preprint arXiv:2011.02796, 2020.

[28] Y. Chen, X. Qin, J. Wang, C. Yu, and W. Gao, “Fedhealth: A federatedtransfer learning framework for wearable healthcare,” IEEE IntelligentSystems, vol. 35, no. 4, pp. 83–93, 2020.

[29] J. Xu, B. S. Glicksberg, C. Su, P. Walker, J. Bian, and F. Wang,“Federated learning for healthcare informatics,” Journal of HealthcareInformatics Research, vol. 5, no. 1, pp. 1–19, 2021.

[30] L. Lyu, H. Yu, and Q. Yang, “Threats to federated learning: A survey,”arXiv preprint arXiv:2003.02133, 2020.

[31] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated learning:Challenges, methods, and future directions,” IEEE Signal ProcessingMagazine, vol. 37, no. 3, pp. 50–60, 2020.

[32] Q. Li, Z. Wen, Z. Wu, S. Hu, N. Wang, Y. Li, X. Liu, and B. He, “Asurvey on federated learning systems: vision, hype and reality for dataprivacy and protection,” arXiv preprint arXiv:1907.09693, 2019.

[33] J. Konecny, H. B. McMahan, F. X. Yu, P. Richtarik, A. T. Suresh, andD. Bacon, “Federated learning: Strategies for improving communicationefficiency,” arXiv preprint arXiv:1610.05492, 2016.

[34] S. Zheng, C. Shen, and X. Chen, “Design and analysis of uplink anddownlink communications for federated learning,” IEEE Journal onSelected Areas in Communications, 2020.

[35] W. Luping, W. Wei, and L. Bo, “Cmfl: Mitigating communicationoverhead for federated learning,” in 2019 IEEE 39th InternationalConference on Distributed Computing Systems (ICDCS). IEEE, 2019,pp. 954–964.

[36] Y. Chen, X. Sun, and Y. Jin, “Communication-efficient federated deeplearning with layerwise asynchronous model update and temporallyweighted aggregation,” IEEE transactions on neural networks andlearning systems, vol. 31, no. 10, pp. 4229–4238, 2019.

[37] Y. Shi, K. Yang, T. Jiang, J. Zhang, and K. B. Letaief, “Communication-efficient edge ai: Algorithms and systems,” IEEE CommunicationsSurveys & Tutorials, vol. 22, no. 4, pp. 2167–2191, 2020.

[38] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith,“Federated optimization in heterogeneous networks,” arXiv preprintarXiv:1812.06127, 2018.

[39] F. Sattler, S. Wiedemann, K.-R. Muller, and W. Samek, “Robust andcommunication-efficient federated learning from non-iid data,” IEEEtransactions on neural networks and learning systems, 2019.

[40] S. Zhang, A. E. Choromanska, and Y. LeCun, “Deep learning with elasticaveraging sgd,” in Advances in neural information processing systems,2015, pp. 685–693.

[41] Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra, “Federatedlearning with non-iid data,” arXiv preprint arXiv:1806.00582, 2018.

[42] V. Smith, S. Forte, C. Ma, M. Takac, M. I. Jordan, and M. Jaggi,“Cocoa: A general framework for communication-efficient distributedoptimization,” The Journal of Machine Learning Research, vol. 18, no. 1,pp. 8590–8638, 2017.

[43] C. Briggs, Z. Fan, and P. Andras, “Federated learning with hierarchicalclustering of local updates to improve training on non-iid data,” arXivpreprint arXiv:2004.11791, 2020.

[44] A. Singh, A. Yadav, and A. Rana, “K-means with three different distancemetrics,” International Journal of Computer Applications, vol. 67,no. 10, 2013.

[45] A. Reisizadeh, A. Mokhtari, H. Hassani, A. Jadbabaie, and R. Pedarsani,“Fedpaq: A communication-efficient federated learning method withperiodic averaging and quantization,” in International Conference onArtificial Intelligence and Statistics. PMLR, 2020, pp. 2021–2031.

[46] S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T.Suresh, “Scaffold: Stochastic controlled averaging for federated learn-ing,” in International Conference on Machine Learning. PMLR, 2020,pp. 5132–5143.

[47] O. Shamir, N. Srebro, and T. Zhang, “Communication-efficient dis-tributed optimization using an approximate newton-type method,” in

International conference on machine learning. PMLR, 2014, pp. 1000–1008.

[48] S. J. Reddi, J. Konecny, P. Richtarik, B. Poczos, and A. Smola,“Aide: Fast and communication efficient distributed optimization,” arXivpreprint arXiv:1608.06879, 2016.

[49] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smithy,“Feddane: A federated newton-type method,” in 2019 53rd AsilomarConference on Signals, Systems, and Computers. IEEE, 2019, pp.1227–1231.

[50] S. AbdulRahman, H. Tout, A. Mourad, and C. Talhi, “Fedmccs: multi-criteria client selection model for optimal iot federated learning,” IEEEInternet of Things Journal, vol. 8, no. 6, pp. 4723–4735, 2020.

[51] J. Xu and H. Wang, “Client selection and bandwidth allocation inwireless federated learning networks: A long-term perspective,” IEEETransactions on Wireless Communications, 2020.

[52] T. Nishio and R. Yonetani, “Client selection for federated learningwith heterogeneous resources in mobile edge,” in ICC 2019-2019 IEEEInternational Conference on Communications (ICC). IEEE, 2019, pp.1–7.

[53] T. T. Anh, N. C. Luong, D. Niyato, D. I. Kim, and L.-C. Wang,“Efficient training management for mobile crowd-machine learning: Adeep reinforcement learning approach,” IEEE Wireless CommunicationsLetters, vol. 8, no. 5, pp. 1345–1348, 2019.

[54] Y. J. Cho, J. Wang, and G. Joshi, “Client selection in federated learning:Convergence analysis and power-of-choice selection strategies,” arXivpreprint arXiv:2010.01243, 2020.

[55] M. Kamp, L. Adilova, J. Sicking, F. Huger, P. Schlicht, T. Wirtz, andS. Wrobel, “Efficient decentralized deep learning by dynamic modelaveraging,” in Joint European Conference on Machine Learning andKnowledge Discovery in Databases. Springer, 2018, pp. 393–409.

[56] T. D. Bui, C. V. Nguyen, S. Swaroop, and R. E. Turner, “Partitionedvariational inference: A unified framework encompassing federated andcontinual learning,” arXiv preprint arXiv:1811.11206, 2018.

[57] V. Smith, C.-K. Chiang, M. Sanjabi, and A. Talwalkar, “Federated multi-task learning,” arXiv preprint arXiv:1705.10467, 2017.

[58] N. Guha, A. Talwalkar, and V. Smith, “One-shot federated learning,”arXiv preprint arXiv:1902.11175, 2019.

[59] H. T. Nguyen, V. Sehwag, S. Hosseinalipour, C. G. Brinton, M. Chiang,and H. V. Poor, “Fast-convergent federated learning,” IEEE Journal onSelected Areas in Communications, vol. 39, no. 1, pp. 201–218, 2020.

[60] H. Tang, S. Gan, C. Zhang, T. Zhang, and J. Liu, “Communication com-pression for decentralized training,” arXiv preprint arXiv:1803.06443,2018.

[61] A. Reisizadeh, H. Taheri, A. Mokhtari, H. Hassani, and R. Pedarsani,“Robust and communication-efficient collaborative learning,” arXivpreprint arXiv:1907.10595, 2019.

[62] J. Xu, B. S. Glicksberg, C. Su, P. Walker, J. Bian, and F. Wang,“Federated learning for healthcare informatics,” Journal of HealthcareInformatics Research, pp. 1–19, 2020.

[63] C. He, C. Tan, H. Tang, S. Qiu, and J. Liu, “Central server free fed-erated learning over single-sided trust social networks,” arXiv preprintarXiv:1910.04956, 2019.

[64] A. Lalitha, O. C. Kilinc, T. Javidi, and F. Koushanfar, “Peer-to-peerfederated learning on graphs,” arXiv preprint arXiv:1901.11173, 2019.

[65] A. G. Roy, S. Siddiqui, S. Polsterl, N. Navab, and C. Wachinger,“Braintorrent: A peer-to-peer environment for decentralized federatedlearning,” arXiv preprint arXiv:1905.06731, 2019.

[66] D. Rothchild, A. Panda, E. Ullah, N. Ivkin, I. Stoica, V. Braverman,J. Gonzalez, and R. Arora, “Fetchsgd: Communication-efficient feder-ated learning with sketching,” in International Conference on MachineLearning. PMLR, 2020, pp. 8253–8265.

[67] S. Li, Q. Qi, J. Wang, H. Sun, Y. Li, and F. R. Yu, “Ggs: General gradientsparsification for federated learning in edge computing,” in ICC 2020-2020 IEEE International Conference on Communications (ICC). IEEE,2020, pp. 1–7.

[68] R. Hu, Y. Gong, and Y. Guo, “Sparsified privacy-masking forcommunication-efficient and privacy-preserving federated learning,”arXiv preprint arXiv:2008.01558, 2020.

[69] F. Sattler, S. Wiedemann, K.-R. Muller, and W. Samek, “Sparse binarycompression: Towards distributed deep learning with minimal commu-nication,” in 2019 International Joint Conference on Neural Networks(IJCNN). IEEE, 2019, pp. 1–8.

[70] M. M. Amiri, D. Gunduz, S. R. Kulkarni, and H. V. Poor, “Feder-ated learning with quantized global model updates,” arXiv preprintarXiv:2006.10672, 2020.

[71] X. Dai, X. Yan, K. Zhou, H. Yang, K. K. Ng, J. Cheng, and Y. Fan,“Hyper-sphere quantization: Communication-efficient sgd for federatedlearning,” arXiv preprint arXiv:1911.04655, 2019.

[72] N. Shlezinger, M. Chen, Y. C. Eldar, H. V. Poor, and S. Cui, “Federatedlearning with quantization constraints,” in ICASSP 2020-2020 IEEEInternational Conference on Acoustics, Speech and Signal Processing(ICASSP). IEEE, 2020, pp. 8851–8855.

[73] L. Liu, J. Zhang, S. Song, and K. B. Letaief, “Hierarchical quantizedfederated learning: Convergence analysis and system design,” arXivpreprint arXiv:2103.14272, 2021.

[74] R. Spring, A. Kyrillidis, V. Mohan, and A. Shrivastava, “Com-pressing gradient optimizers via count-sketches,” arXiv preprintarXiv:1902.00179, 2019.

[75] J. Nicholls, “Quantization in deep learning,” https://medium.com/@joel 34050/quantization-in-deep-learning-478417eab72b/, 2018, Ac-cessed April 28th, 2021.

[76] P. Jiang and G. Agrawal, “A linear speedup analysis of distributed deeplearning with sparse and quantized communication,” Advances in NeuralInformation Processing Systems, vol. 31, pp. 2525–2536, 2018.

[77] A. Koloskova, T. Lin, S. U. Stich, and M. Jaggi, “Decentralized deeplearning with arbitrary communication compression,” arXiv preprintarXiv:1907.09356, 2019.

[78] N. Shlezinger, M. Chen, Y. C. Eldar, H. V. Poor, and S. Cui, “Uveqfed:Universal vector quantization for federated learning,” IEEE Transactionson Signal Processing, 2020.