1

Blockchain-based Federated Learning forFailure Detection in Industrial IoT

Weishan Zhang∗, Qinghua Lu†, Qiuyu Yu∗, Zhaotong Li∗,Yue Liu∗, Sin Kit Lo†, Shiping Chen†, Xiwei Xu†, Liming Zhu†

∗College of Computer Science and Technology, China University of Petroleum (East China), China†Data61, CSIRO, Australia

Abstract—Federated learning is an emerging privacy-preserving machine learning paradigm which has attracted greatinterests from the community of Industrial Internet of Things(IIoT). Blockchain has been recently leveraged in IIoT federatedlearning to provide data integrity and incentives to attractsufficient client data and computation resources for training.However, there is a lack of systematic architecture design forblockchain-based federated learning systems to support method-ical development and tackle the challenge of data heterogeneityin failure detection of IIoT. Also, the current solutions do notconsider the incentive mechanism design and scalability issueof blockchain. Therefore, in this paper, we present a platformarchitecture of blockchain-based federated learning systems forfailure detection in IIoT. To address the data heterogeneity issuein IIoT failure detection, we propose a novel centroid distanceweighted federated averaging (CDW FedAvg) algorithm takinginto account the distance between positive class and negative classof each client dataset. To enable verifiable integrity of client datain a scalable way, each client server periodically creates a Merkletree in which each leaf node represents a client data record,and stores the tree root on a blockchain. An on-chain incentivemechanism is designed based on the size of client data usedin local model training to accurately and timely calculate eachclient’s contribution. A prototype of the proposed architecture isimplemented with our industry partner, and evaluated in termsof feasibility, accuracy and performance. The results show thatthe approach ensures data integrity and has satisfactory detectionaccuracy and performance.

Index Terms—Blockchain; Federated Learning; MachineLearning; Edge Computing; IoT; AI; Failure.

I. INTRODUCTION

INTERNET of Things (IoT) has been widely employed inindustry, particularly in manufacturing, to enable smart in-

dustrial IoT (IIoT) applications. There will be 75.44 billion IoTdevices worldwide by 2025, which are expected to generate79.4 Zettabytes of data [1]. How to manage and make useof these IoT data to derive intelligence in efficient, secureand economic ways becomes an essential research question.Failure detection is one of the most essential problems inIIoT [2], [3].

Federated learning is an emerging machine learningparadigm [4], [5] in a way that preserves privacy and reducesbias in model training. In each round of federated learning,multiple clients (e.g. organizations, data centers, IoT or mobiledevices) are selected to train models locally to produce a

Qinghua Lu is the corresponding author. Email: qinghua.lu@data61.csiro.au

global model while raw data are stored in clients and notexchanged or transferred.

Blockchain is an innovative distributed ledger technologyfor building new forms of decentralized software architecture,which enables agreements on transactional data sharing acrossa large network of untrusted participants, without relying on acentral trusted authority [6]. The data structure of blockchainis a list of identifiable blocks, and all the blocks are linkedto the previous block and thus form a chain. The blocks arecontainers for storing transactions, which are identifiable pack-ages carrying the changing states of data. A smart contract is auser-defined program, deployed and executed on a blockchainnetwork [7]. After deployment, a smart contract can expresstriggers, conditions to enable complex business logic.

Blockchain has been recently leveraged in IIoT federatedlearning to provide data integrity and incentives to attractsufficient client data and computation resources for training[8]–[10]. However, there is a lack of systematic and holisticarchitecture design for blockchain-based federated learningsystems to support methodical development and tackle thechallenge of data heterogeneity in failure detection of IIoT.An important aspect of such gap is demonstrated in currentsolutions, which do not consider the incentive mechanismdesign and scalability issue of blockchain.

Therefore, in this paper, we present a platform architectureof blockchain-based federated learning systems in IIoT. Ineach round of training, a number of clients are selectedby the central server to train models locally and uploadmodel updates to the central server instead of raw data.The central server aggregates the model updates to producea comprehensive and unbiased global model. To produce aunbiased global model, the model updates are aggregatedusing a novel averaging algorithm called centroid distanceweighted federated averaging (CDW FedAvg), which takesinto account the distance between positive class and negativeclass of each client dataset in weight calculation. Client dataare hashed and stored periodically on blockchain to ensure dataintegrity while addressing the scalability issue of blockchain.Incentives are calculated based on clients’ data contributionsand rewarded to motivate the clients via a smart contract.A proof-of-concept prototype of the proposed architectureis implemented with our industry partner and evaluated interms of feasibility, accuracy, and performance. The evaluationresults show that the proposed architecture is feasible and hassatisfactory detection accuracy and performance.

arX

iv:2

009.

0264

3v1

[cs

.DC

] 6

Sep

202

0

2

The contributions of the paper are as follows.

• A platform architecture that provides a systematic view ofsystem interactions and serves as a guidance for the de-sign of the blockchain-based federated learning systemsin IIoT failure detection. The architecture design involvesthe following architectural design decisions: placement ofmodel training, storage of monitored client data, incentivemechanisms for clients, and deployment of blockchain.

• A federated averaging algorithm called centroid distanceweighted federated averaging (CDW FedAvg) taking intoaccount the distance between positive class and negativeclass of each client dataset to reduce the impact of thedata heterogeneity issue in IIoT failure detection.

• A blockchain anchoring protocol which builds a customMerkle tree in which each leaf node represents a recordof data collected by a client’ edge device, and places theroot of the custom Merkle tree at a pre-configured intervalon blockchain to ensure verifiable integrity of client datain a scalable way.

• An incentive mechanism that rewards each client withtokens based on the size of the data contributed to modeltraining via the designed incentive smart contract tomotivate clients to contribute their data and computationresources to model training.

• Feasibility, predication accuracy and performance eval-uation using a real industry use case which predictsfailures for water-cooled magnetic levitation chillers inair-conditioners.

The remainder of this paper is organized as follow. SectionII summarises the related works. Section III presents theproposed solutions. Section IV discusses the implementationand evaluation. Section V concludes the paper and outlinesthe future work.

II. RELATED WORK

IoT has been integrated into industrial systems to allowa higher degree of automation and improve the economicbenefits including productivity and efficiency. In industrial IoT,traditional centralized cloud-based computing approach maybe inefficient, since tasks are pushed to cloud from numerousIoT devices to the cloud which can cause data leakage andnetwork overload [11]. To address the issue of data privacy andlimited bandwidth in cloud computing, edge computing hasbeen widely applied in IIoT systems to shift part of computingtasks from cloud servers to edge nodes, which are closer towhere data are generated.

The concept of federated learning is first proposed byMcMahan et al. [12], which enables edge computing fromthe machine learning perspective. In a border definition offederated learning, multiple clients are coordinated to trainmodel locally and aggregate model updates to achieve thelearning objective, while raw data are stored in clients andnot exchanged or transferred [5], [13]. Compared with moststudies of federated learning with central server, a few researchworks [8], [14] build a federated learning framework withoutcentral server, such as applying p2p structure.

Federated learning is classified into two types: cross-devicefederated learning and cross-silo federated learning. The for-mer is introduced on mobile and edge device applications.For example, Google has applied federated learning setting tothe prediction of search suggestions, next words and emojis,and the learning of out-of-vocabulary words [15]–[17]. Thelatter represents another type of federated learning setting inwhich clients are different organizations or geo-distributeddatacenters [13]. For example, Sheller et al. [18] adopt cross-silo federated learning to build a segmentation model on thebrain tumor data from multiple institutions.

Blockchain was first introduced as the technology support-ing Bitcoin [19], and has been generalized to an append-only data store which is distributed across computationalnodes and structured as a linked list of blocks containing aset of transactions. Currently, many studies have integratedblockchain into their federated learning approaches. Sincefederated learning relies on a single server, which is vulnerableto malfunction, some studies are focused on eliminating single-point failure in federated learning using blockchain [8], [14].Zhao et al. [14] design a system based on federated learningand blockchain to let clients empower full nodes and competeto serve as a central server by turns for aggregation. Kimet al. [8] proposed a blockchained-federated learning archi-tecture in which participant clients are able to store localmodel updates on blocks and all clients as miners can accessand aggregate the updates through smart contracts, withouta single central server. Some works leverage blockchain infederated learning for auditability and data privacy [10], [20].Lu et al. [20] incorporate federated learning and blockchaininto a data sharing scheme to preserve privacy. Majeed etal. [10] records local model parameters in each iterationinto blocks, and introduces “the global model state trie” tosecurely store global model. Incentive mechanism is anotherimportant research direction, which is imported to fedeartedlearning to encourage participants and standardize the behaviorof participants [9], [21]. Weng et al. [9] employs incentivemechanism and blockchain transactions to protect the privacyof local gradients, and enable auditability of the training pro-cess. Kang et al. [21] propose an incentive mechanism basedon blockchain combing the reputation which is introducedto measure the trustworthiness and reliability of clients, tostimulate honest clients with high-quality data to participate infederated learning. Moreover, there are some recently proposedplatforms which increase more novel participant roles, such asbuyers who can pay for the clients in order to complete theirtraining tasks [22]. Bao et al. [22] provides a platform forbuyers to buy federated learning models and designs DDCBFto reduce the time cost of buyers querying and verifyingmodels on a blockchain.

However, the above studies do not present the architecturedesign details of blockchain-based federated learning systemswith consideration of the data heterogeneity issue in IIoTfailure detection. Additionally, how to ensure the integrityof client data is not discussed. In this paper, we propose anarchitecture of federated learning for IIoT failure detectionusing blockchain in a way that motivates clients to providesufficient data and computing resources for model training

3

while reducing the impact of data heterogeneity and preservingclient data privacy and integrity.

III. ARCHITECTURE

We adopt the standard requirements elicitation methodology[23] for our industry partner’s specific requirements and de-rived a list of general application-agnostic functional and non-functional requirements for blockchain-based federated learn-ing systems for IIoT failure detection. The gathered functionalrequirements include: 1) Training local models at clients; 2)Formulating a global model based on the local model updateswithout exchanging the local data; 3) Making detection usinga trained model; 4) Resolving disputes between platformowner and clients; 5) Rewarding clients for their contributionsto the model training process. The identified non-functionalrequirements include: 1) Preserving edge data privacy; 2)Maintaining satisfactory detection accuracy; 3) Ensuring edgedata integrity; 4) Keeping availability of blockchain infrastruc-ture.

In this section, we present a platform architecture forblockchain-based federated learning systems to meet the abovegeneral application-independent requirements for federatedlearning systems. Section III-A provides an overview of thearchitecture and discusses how the components and theirinteractions. Section III-B discusses the main architecturaldecisions. Section III-C proposes a novel federated averagingalgorithm called centroid distance weighted federated aver-aging (CWD FedAvg). Section III-D presents a protocol foranchoring the client data to the blockchain to ensure dataintegrity. Section III-E designs an incentive mechanism toencourage the clients to contribute to model training.

A. Overall Architecture

Fig. 1 illustrates the overall platform architecture, whichconsists of two types of participating organizations: centralorganization and client organization. The central organizationis the federated learning platform owner (e.g. manufacturercompany) which maintains all industrial services based ondetection results analyzed by the platform, while the clientorganizations are the organizations that contribute local dataand computation resources for local model training.

Each client organization maintains sensors, a client device,a client server, and a blockchain full node, while the centralorganization hosts a central server and a blockchain full node.Each client device (e.g., Raspberry Pi) gathers the environmentdata detected by sensors through client data collection andclean the data (e.g., noise reduction) via client data pre-processing. All the collected client environment data are storedin database of client data hosted by the client server. If a clientorganization is selected by the central server for a certain roundof model training, the client server downloads the currentversion of the global model from the central server to furthertrain the model and compute model updates using the localdata via model training. The model updates are sent to theaggregator on the central server at a pre-configured round.Aggregator combines the model updates received from theclients and produces a global model, which is then sent back to

client servers for Detector. The detection results are encryptedand sent to the central organization, where the operator candecrypt the received detection results via data decryption andmake the final decision.

Each client server periodically creates a Merkle tree inwhich each leaf node represents a data record collected bysensors. The information of a client in each selection round(including the Merkle tree root, the status of training, andthe size of client data for model training) are all stored inthe pre-deployed smart contracts on the blockchain throughclient data anchoring. If a dispute (e.g. about failure cause)occurs, the operators of both the central organization and theclient organization can use dispute resolution to verify theintegrity of client historical data in a certain anchoring period,by comparing the data with the respective Merkle tree rootstored on-chain. To reward the client organizations for theircontribution to model training, tokens are distributed to themaccording to the contributed size of client data used for modeltraining.

B. Architectural Design Decisions

1) Local Model Training: To preserve the privacy of clientdata (e.g. usage frequency and time), which is often sensitiveto each client organization’s business, the architecture adoptsfederated learning to train the models locally on client serversvia model training while computing training results on thecentral server to formulate a global model for detection viaaggregator.

2) Anchoring Monitored Data to Blockchain: To resolvedisputes between the central organization and client organi-zations (e.g. about failure causes), the architecture leveragesthe blockchain to enable verifiable integrity of client data viaclient data anchoring on the client server. Since blockchainhas limited storage capabilities, the client server periodicallycreates a Merkle tree in which each leaf node represents arecord of data collected by sensors, and stores its root in thepre-deployed root registry smart contract (SC) through clientdata anchoring.

3) Smart Contract based Incentive Mechanism: The archi-tecture provides incentives to encourage client organizationsfor model training. Tokens are rewarded to client organizationsbased on the size of data taken in model training. The rewardedtokens are recorded in incentive registry SC.

4) Blockchain Deployment: In a blockchain network, a fullnode maintains a complete list of every single transaction thathad occurred on the blockchain. In the proposed architecture,each participating organization, including central organizationand client organization, hosts one blockchain full node. Hence,each organization has a full replica of the data stored on theblockchain which can be used for auditing, and ensure theavailability of the whole system.

C. Centroid Distance Weighted Federated Averaging

The federated averaging (FedAvg) algorithm proposed byGoogle [4] is not suitable for industrial IoT systems since theheterogeneity between the different client local datasets existsheavily in industrial IoT systems (i.e., the distribution and size

4

Client organization 1

Client serverClient device

Dispute resolution

Client data collection

Model training

Client data anchoring

Fullnode

Blockchain

Central organization

Fullnode

Operator

Data decryption

Database of detection results

Operator

Aggregator

Detector

Model updates

Updated model

Encrypted result

Database of client data

Clientorganization 2

Client organization 3

Clientorganization N

...

Predictionresult

Root

registry SC

Incentive

registry SC

Central server

Data/model contribution data, Merkle tree root

Sensor1.1 Sensor1.2

Client data pre-processing Current model

Fig. 1: Architecture of blockchain-based federated learning systems for failure detection in IIoT.

Algorithm 1 CWD FedAvg Algorithm/*Central server*/Initialize w0

for each round t = 1, 2, ... doSelect K clientsfor each client k ∈ K clients do

wkt ← UpdateClient( )

dkt ← the distance between two classes in training datasetend forf(dkt )←

1dkt

wt ←∑K

k=1 nk ∗ f(dkt ) ∗ wt−1/∑K

k=1 nk ∗ f(dkt )end for/*Client update*/UpdateClient( ){Initialize local minibatch size B, local epochs E, learning rate ηfor each epoch i ∈ E do

randomly sample Si based on size Bwi ← wi−1 - η5g(wi−1 ; Si )

end forreturn wi

}

of the dataset on each client may be different). Take the air-conditioner failure detection as an example, the deployed air-conditioners might have different working environments (e.g.,weather) and different users (e.g., age).

To reduce the impact of data heterogeneity and improve themodel performance, we propose a centroid distance weightedfederated averaging (CDW FedAvg) algorithm taking intoaccount the distance between positive class and negative classof each client dataset. The weighted average process arecalculated as:

wt =

∑Kk=1 s

kt ∗ f(dkt ) ∗ wt∑K

k=1 skt ∗ f(dkt ))

, h(dkt ) = 1/dkt (1)

where skt represents the size of k th client training dataset

contributed in t th round. dkt denotes the distance betweenthe centroid of positive class and the centroid of negativeclass from the k th client dataset in t th round. f(dkt ) is thefunction to process dkt , outputting the reciprocal of dkt . Here,we sacrifice the model accuracy for client datasets which havegreater distance between the centroid of positive class andcentroid of negative class since most of the client datasets inthe real-world have smaller distance. The detailed process isillustrated in Algorithm 1.

In Algorithm 1, for each round, the central server selectsa subset of clients to participate in the round based on somepredefined criteria (e.g., idle and charging) and sets the localminibatch size B, local epochs E and learning rate η onthe selected clients. Then, the selected clients execute localStochastic Gradient Descent (SGD) on B batches in E epochs,calculate the distance dkt and return the update wt and dkt tothe central server. Finally, on the central server, the updatesare weighted averaged according to the size of local trainingdataset and the reciprocal of dkt .

D. Anchoring Protocol

An anchoring protocol is designed to build a Merkle treebased on the monitored data collected by a client and anchorthe Merkel tree root to the blockchain for client data integrityand auditing (i.e., dispute resolution between the central orga-nization and the client organization). The incentive mechanismis also supported by the anchoring protocol to reward the clientorganizations according to the anchored contribution recordedon the incentive registry SC.

Algorithm 2 describes how the anchoring protocol works.The anchoring protocol is scheduled according to the agree-ment between the central organization and client organizations(e.g. every hour). For each anchoring period pi, the protocol

5

Algorithm 2 Anchoring Protocol1: for each period pi = 1, 2, ... do2: for cj ∈ C do3: for dcpn do4: dcpn ← String( dcpn)5: dcpn ← Hash( dcpn)6: pNodeList[0..N ]← dcpn7: end for8: repeat9: for (k = 0 ; k < length(pNodeList) ; k+2) do

10: l← pNodeList[k]11: if k = length(pNodeList) then12: r ← l13: else14: r ← pNodeList[k + 1]15: end if16: r ← SHA256(l + r)17: temp[j]← r18: (each j ∈ [0,dk/2e])19: end for20: pNodeList[0..j]← temp[0..j]21: until length(pNodeList) = 122: root← pNodeList[0]23: time← current time24: UpdRootSC(time, root)25: end for26: end for27: for each training round do28: model training29: UpdStaSC(rNo, dataSize)30: CalIncentiveSC( )31: end for

goes through each client organization cj , and queries the clientdata dcpn collected within the anchoring period. The protocolconverts each data record dcpn to a string format. All the datacollected within the anchoring period are hashed and used toconstruct a Merkle tree where each leaf node represents thehash value of a data record.

To construct the Merkle tree structure, the protocol firstcreates a pNodeList to store the parent node values and atemp to put the results of the combined hash values of thetwo child nodes. All the leaf node values are initially placedin the pNodeList. To compute the parent node of each pair oftwo leaf nodes which are adjacent to each other, the hashvalues of them are assigned to l which denotes left childnode, and r which denotes right child node respectively, andcombined to produce the hash value for the parent node, whichis then added to temp. After going through all the elementsin pNodeList, the temp is converted to pNodeList. The treeconstruction process is executed recursively until the lengthof the pNodeList becomes one. The root is exactly the onlyvalue in the pNodeList.

UpdRootSC() is used to anchor the Merkle tree root andtimestamp of the current period to root registry SC. Once theclient servers finish their work and sent the model updates tothe central server, dataSize (the size of data for training) ofeach participant client server and the serial number of traininground rNo is anchored in the blockchain by UpdStaSC().The corresponding tokens are then calculated and rewardedto client organizations through CalIncentiveSC(). Both Up-dStaSC() and CalIncentiveSC() are encoded in Incentive reg-istry SC.

E. Incentive Mechanism

Algorithm 3 describes how the incentive mechanism works.The incentive mechanism aims to reward client servers withtokens, according to their contribution (the size of data appliedfor model training). For every training round, the centralserver chooses a fixed number of client servers, and storestheir blockchain addresses in Clist. Besides, the central servershould also define a struct training to enroll finished trainingtasks of client servers, including the status of training workfinished and the size of data for training dataSize. The chosenclient servers train models locally, and upload correspondinginformation into training, which is then converted to contri.Finally, the incentive tokens are distributed to each address ofthe client server that Clist records, in accordance with theircontri respectively. A smaller value is computed by datasizedividing by a constant C, to apply for token distribution. Theultimate goal of this incentive mechanism is to encourage moreclient servers for participation, to eliminate the model bias.

Algorithm 3 Incentive Calculation and Distribution1: /*Central server executes*/2: Clist ← selected clients address3: /*Client server executes*/4: struct {finished, dataSize} training5: for each training round do6: if address ∈ Clist then7: model traning8: UpdStaSC(rNo, dataSize)9: end if

10: CalIncentiveSC( )11: end for12: /*Update Status of Training work */13: UpdStaSC(rNo, dataSize){14: training.finished← true15: training.dataSize← dataSize16: contri[addr][rNo]← work17: }18: /*Calculation Incentive*/19: CalIncentiveSC( ){20: if address ∈ Clist then21: if contri[addr][rNo].finished == true then22: token[addr] += contri[addr][rNo].dataSize / C23: end if24: end if25: }

IV. IMPLEMENTATION AND EVALUATION

In this section, we implement a proof-of-concept prototypeusing a real-world use case, and evaluate the prototype in termsof feasibility accuracy, and performance.

A. Use Case

Our industry partner is one of the largest air-conditionermanufacturers in the world, and a large portion of theirclients are hotels which are distributed across the globe. Thefailures of the hotel air-conditioning systems directly influencecustomer satisfaction. In the collaborated project, we aimto predict failures for air-conditioning systems. In the usecase, we claim that the manufacturer owns the servers/devicesplaced in the hotels. The ownership of hotel servers/deviceshere means only the manufacturer has the write-access author-ity to the firmware/software on the devices/servers.

6

In particular, the project seeks to place most of the machinelearning tasks within the hotels to preserve the data privacy ofhotels. For example, room occupancy percentage is sensitiveto a hotel and can be calculated based on the air-conditioningsystem data. Also, the project aims to encourage hotels tocontribute their air-conditioner data and train failure detectionmodels locally. Tokens are rewarded to hotels based on theircontribution which can later be used to buy new products orservices.

The water-cooled magnetic levitation chiller is the keycomponent in a hotel air-conditioning systems. We developeda proof-of-concept prototype for the failure detection systemof water-cooled magnetic levitation chillers using the proposedarchitecture.

B. Implementation

Fig. 2 shows the design of two on-chain smart contractsfor client data anchoring and model training incentives: rootregistry SC stores Merkle tree roots and corresponding times-tamps, while incentive registry SC maintains incentive infor-mation including training round number, the status of trainingtask, size of client data applied for training work, and rewardedtokens based on data size.

We adopt a federated learning framework named Leaf [24]in the implementation. We use Ethereum [25] as the under-lying blockchain network, in which the consensus algorithmis Proof-of-Work (PoW), and the difficulty is configured to0x4000. The smart contracts are written in Solidity andcompiled with Solidity compiler version 0.4.20. The clientdata anchoring component is developed in Java 1.8. We selectMySQL 5.7.25 as the supported database to store the off-chaindata.

We deployed the implemented prototype on 5 AlibabaCloud1 virtual machines (2vCPUs, 8G RAM) as client/centralservers (1 cloud server used as the central server and 4 cloudservers used as client servers) and 4 Raspberry Pi 3 boardsconfigured to Raspbian 2018-11-3 as 4 client devices. Eachclient server is allocated to one hotel where installed onelarge air conditioner (as client device). The data collected bythe sensors in each air conditioner includes 70 parameters.After reducing the noisy data and removing the redundantparameters (from two compressors), 18 parameters are kept formachine learning as client data, which are listed as follows.

• Evaporator inlet water temperature• Evaporator outlet water temperature• Condenser inlet water temperature• Condenser outlet water temperature• Evaporator cooling capacity• Compressor inlet air temperature• Compressor outlet air temperature• Evaporator inlet air pressure• Condenser outlet air pressure• Exhaust air overheat temperature• Main circuit’s coolant level• Opening size of the main coolant pipe valve• Compressor’s load

1https://www.aliyun.com/

Root Registry SC

timestamp: uintroot: stringroots: mapping(msgSender=>mapping(timestamp=>root))

UpdRoot(ID: address, timestamp: uint, root: string)getRoot(ID: address, timestamp: uint): (root: string)

Incentive Registry SC

rNo: uinttoken: uintstruct {finished: bool, dataSize: uint} workdataContribution: mapping(msgSender => mapping(rNo => work))balances: mapping(msgSender => token)

setselectedClient(rNo: uint, ID: address)UpdSta(rNo: uint, datasize: uint)CalIncentive(ID: address)getselectedClient(rNo: uint): (ID:address)getSta(rNo: uint, ID: address): (Contribution: work)getIncentive(ID: address): (token: uint)

Fig. 2: On-chain data structure for anchoring

• Compressor’s current• Compressor’s rotational speed• Compressor’s voltage• Compressor’s power• Compressor’s inverter’s temperatureFor the experiments, each data record consists of 18 features

as mentioned above and 1 label which indicates whether anyfault occurred. A client server has one classifier, which is afour-layer neural network. The first layer consists of 18 unitscorresponding to the 18 features. Then there are two hiddenlayers which both have 150 units. The output layer is a Softmaxfunction to classify normal data and abnormal data. Please notethat the hidden layers apply ReLu function as the activationfunction. The learning rate is set to 0.005.

C. Feasibility

To evaluate the feasibility of the proposed approach, we ex-amined the implemented proof-of-concept of proposed archi-tecture against the functional and non-functional requirementsidentified in Section III.

For the functional requirements, 1) the platform prototypeis able to train local models on the client server hosted byeach hotel and aggregate the local model updates into a globalmodel for failure detection. 2) The operators of the hotels andthe air-conditioner platform are able to verify the integrity ofclient data for a certain anchoring period by comparing withthe Merkle tree root stored on blockchain. 3) In addition, thehotels can be allocated with tokens as the reward for theircontribution via Incentive Registry SC.

Regarding the non-functional requirements, 1) the air-conditioner data privacy is preserved by the federated learningsetting, in which the central server only needs air-conditionermodel updates rather than all air-conditioner usage informa-tion. 2) In particular, the applied federated learning is moreadvantageous than centralized model training on the bandwidth

7

train local model on client server

send model updatesto central server

aggregate updates on central server

updated models back to client server

query client data from client database

predict failuresusing updated model

send results to central server

store results in central database

create Merkle tree

store Merkle tree root on chain

store contribution on chain

query client data from client database

Fig. 3: Machine learning process and blockchain anchoringprocess.

cost, if the size of training data is large. 3) The detectionaccuracy is maintained by a shared model, which is formulatedby aggregating updates from models trained on each hotel(client server). 4) Air-conditioner data integrity is achievedvia the anchoring protocol. When data auditing is needed, thedata stored locally in each hotel can be again constructed tothe Merkle tree structure and verified by comparing it withthe Merkle tree root stored on the blockchain. Each hotel andthe air-conditioner manufacturer maintains a full node to keepthe blockchain infrastructure available, in the case that somenodes maybe out-of-service.

D. Quantitative Analysis - Accuracy and Performance

In the quantitative analysis, each client server stores about7,000 training records and 2,300 testing records. The trainingrecords are distributed on client servers, while a global modelis learnt on the central server by aggregating local modelupdates. We first tested accuracy of the global model of failuredetection on each individual client server with their own test-ing records. Moreover, we compared the accuracy of the globalmodel trained using federated learning with the respectivemodel trained using the traditional centralized learning andlocal learning. We tested two classification models includingLogistic Regression (LR) and Neural Networks (NN).

15

17

19

21

23

25

27

29

31

33

35

0 10 20 30 40 50 60 70 80 90 100

Exec

utio

n ti

me

(s)

Number of test

Writing Merkle tree root Writing contribution data

Fig. 4: Blockchain execution time.

Test 1: Measuring blockchain anchoring timeThe executing time of anchoring protocol includes querying

data from the database, creating a Merkle tree, and writingthe Merkle tree root and data/model contribution data onthe blockchain. Fig. 4 only illustrates the execution time forwriting Merkle tree root and contribution data to blockchain,since it costs most of the execution time for the anchoringprotocol. We measured the transaction inclusion time whichis for the transaction to be included in a block. Writing theMerkle tree root takes around 18s while the average time forwriting contribution data is around 21s. The writing latencyon the blockchain is much longer than in the conventionaldatabase as it takes time to propagate transactions/blocks andachieve consensus. Since blockchain is used for auditing andincentives provision, the writing latency of blockchain is not aconcern in this work. We have also examined the Merkle treecreation time, which takes about 250ms. As the Merkle treecreation time is much less than transaction inclusion time, itis not included in Fig. 4.

Test 2: Measuring model detection accuracyBefore we measure the model prediction, we first examine

the distribution of each client dataset to check their distributiondifference. Fig. 5 shows the distribution of positive class andnegative class in each client dataset. We observed that thereis a distance from positive class to negative class in client #3,while other clients show a different pattern.

We measure the accuracy of Logistic Regression (LR)and Neural Networks (NN) using proposed centroid dis-tance weighted (CDW) federated learning method usingCDW FedAvg, standard federated learning using FedAvg,centralized learning and local learning respectively for 100training rounds. For each round of federated learning, themodel is trained locally on each client through 40 epochs andthe model updates are sent to central server to produce a globalmodel. Each client uses the global model to make failureprediction. For each round of centralized learning, a centralserver trains the model through 40 epochs and each clientuses this model to predict failures. Compared with federatedlearning model which is trained on local training dataset, thecentralized training model is trained on an aggregated trainingdataset which consist of local datasets from all clients. Wetested the centralized model and federated learning modelusing the same test dataset. For each round of local learning,

8

Dataset #1 Dataset #2

Dataset #3 Dataset #4

Fig. 5: Train data distribution after dimension reduction.

each client trains a local model using local datasets through40 epochs and uses the local model to make prediction.

The measurement results are shown in Fig. 6 and Fig. 7.The x-axis represents the number of training round, andthe y-axis means the percentage of correct detection (bothtrue positives and true negatives). In Fig. 6 and Fig. 7,each sub-graph includes 4 lines, which show the accuracyof model using different learning methods and tested on thecorresponding client datasets. For example, in Fig. 6 Dataset#1, CDW Federated LR represents the accuracy of the LRmodel, which was trained using the proposed centroid distanceweighted federated learning method using CDW FedAvg andtested on client #1 dataset. Overall, Fig. 6 and Fig. 7 show thatthe proposed CDW federated learning and standard federatedlearning can achieve satisfactory accuracy as local learningand centralized learning in most of the times. The accuracy ofthe models tested on clients differs from each other becausethe data distribution of the respective clients are different.Specifically, for both LR and NN, the accuracy of both theproposed CDW federated learning and standard federatedlearning tested on client 3 is highest, while the accuracy offederated learning performed on client 4 is lowest. Further,we observe that NN performs better than LR in all types oflearning. In addition, the proposed CDW federated learningperforms better than the standard federated learning for LR,while the two federated learning algorithms perform similarfor NN.

In addition to accuracy, we also measured three othermetrics in the experiments, including precision, recall, and F1-score. Precision means the proportion of truly classified pos-itive instances to the total classified positive instances, whilerecall means the ratio of truly classified positive instances tothe total number of practical positive instances. F1-Score is theharmonic mean of precision and recall. The formal definitionsof Precision, Recall and F1-Score are shown in Formula (2)-(4) respectively. We measured these three metrics of LR andNN using both federated learning and centralized learning.

Data Size 103 106 109

Centralized LR 103 106 109

Fed LR(1) 28800 28800 28800Fed LR(2) 57600 57600 57600Fed LR(3) 86400 86400 86400Fed LR(4) 115200 115200 115200

TABLE I: Communication overhead.

The results of are illustrated in Fig. 8 and Fig. 9. The x-axis includes four metrics, while the y-axis is the the valueof three measured metrics. The results show that the proposedCDW federated learning method and the standard federatedlearning method performed best on the dataset of client #3,which is similar to the results of accuracy tests. Compared withthe standard federated learning method, the proposed CDWfederated learning method shows an improvement in Fig. 8,while the improvement is slight in Fig. 9.

Accuracy =TP + TN

TP + TN + FP + FN(2)

Precision =TP

TP + FP(3)

Recall =TP

TP + FN(4)

F1-Score =2 ∗ Precision ∗RecallPrecision+Recall

(5)

TP, TN, FP, FN are the abbreviations of True Positive, TrueNegative, False Positive, False Negative respectively.

Test 3: Measuring the communication overheadWe measured the communication overhead of LR. The

results are shown in Table I, where Fed LR(N) means Nclients are chosen randomly to train the model in each round.In our implementation, the size of the LR model is 144bytes and the number of training round is set to 100. Forinstance, in Fed LR(3), the total communication overhead is144*3*100*2=86400 bytes, where emph*2 means the modelsare gathered from every chosen client and then the globalmodel is sent back to the clients after aggregating the clientmodel updates. The table shows the overhead of centralizedlearning is much more than federated learning, because allthe data need to be sent to the central server from the clientservers in centralized learning, while only the model updatesare sent to the central server in federated learning. Therefore,the communication overhead of the federated learning isirrelevant to the size of the data. When the size of dataset isextremely large, the communication overhead can be reducedsignificantly.

V. CONCLUSION

This paper presents a platform architecture of blockchain-based federated learning for IIoT failure detection. In the de-sign, a central server coordinates client servers to train a sharedglobal model for detection, with raw data stored locally. Thearchitecture adopts blockchain to enable verifiable integrity ofclient data and incentives for client contribution. To reduce

9

Dataset #1 Dataset #2

Dataset #3 Dataset #4

Fig. 6: The accuracy of LR model.

the impact of data heterogeneity, a novel centroid distanceweighted federated averaging (CDW FedAvg) algorithm isproposed based on the distance between positive class andnegative class of each client dataset.

A proof-of-concept prototype is implemented using the pro-posed architecture using a real-world use case. We evaluatedthe approach in terms of feasibility, detection accuracy, andperformance. The evaluation results show that the architectureachieved all the proposed objectives.

Although most of the federated learning modules are placedon client servers in this paper, we plan to adapt the design bymoving all the modules from client servers to client deviceswith more powerful computing and storage capabilities in ourfuture work. Also, we plan to explore how to increase thetrustworthiness of client devices for the aggregation processin further study.

VI. ACKNOWLEDGEMENT

This research is supported by the National Key R&D Pro-gram (2018YFE0116700), the Shandong Provincial NaturalScience Foundation (ZR2019MF049, Parallel Data DrivenFault Prediction under Online-Offline Combined Cloud Com-puting Environment), the supporting project from China

Petroleum Group (2018D-5010-16) for Big Data IndustryDevelopment Pilot Demonstration Project from Ministry ofIndustry and Information Technology, the National Major Sci-ence and Technology Project (2017ZX05013-002), the ChinaPetroleum Group Science and Technology Research InstituteCo., Ltd. Innovation Project (Grant No. 2017ycq02) and theFundamental Research Funds for the Central Universities(2015020031).

REFERENCES

[1] IDC, “The growth in connected iot devices is expected to generate79.4zb of data in 2025, according to a new idc forecast,” https://www.idc.com/getdoc.jsp?containerId=prUS45213219, 2019, accessed 2-April-2020.

[2] M. Banerjee, C. Borges, K. R. Choo, J. Lee, and C. Nicopoulos,“A hardware-assisted heartbeat mechanism for fault identification inlarge-scale iot systems,” IEEE Transactions on Dependable and SecureComputing, pp. 1–1, 2020.

[3] S. Seshadri, D. Rodriguez, M. Subedi, K. R. Choo, S. Ahmed, Q. Chen,and J. Lee, “Iotcop: A blockchain-based monitoring framework for de-tection and isolation of malicious devices in internet-of-things systems,”IEEE Internet of Things Journal, 2020, in press.

[4] J. Konecny, H. B. McMahan, F. X. Yu, P. Richtarik, A. T.Suresh, and D. Bacon, “Federated learning: Strategies for improvingcommunication efficiency,” CoRR, vol. abs/1610.05492, 2016. [Online].Available: http://arxiv.org/abs/1610.05492

10

Dataset #1 Dataset #2

Dataset #3 Dataset #4

Fig. 7: The accuracy of NN model.

0.6

0.7

0.8

0.9

1

accuracy precision recall F1-Score

dataset #1

0.6

0.7

0.8

0.9

1

accuracy precision recall F1-Score

dataset #2

0.6

0.7

0.8

0.9

1

accuracy precision recall F1-Score

dataset #3

0.6

0.7

0.8

0.9

1

accuracy precision recall F1-Score

dataset #4

Local_LR Centralized_LR Federated_LR CDW_Federated_LR

Fig. 8: Precision, recall, and F1-score of LR model.

[5] S. K. Lo, Q. Lu, C. Wang, H. Paik, and L. Zhu, “A systematic literaturereview on federated machine learning: From a software engineeringperspective,” ArXiv, vol. abs/2007.11354, 2020.

[6] F. Tschorsch and B. Scheuermann, “Bitcoin and beyond: A technicalsurvey on decentralized digital currencies,” IEEE Communications Sur-veys & Tutorials, vol. 18, no. 3, p. 464, 2016.

11

0.6

0.7

0.8

0.9

1

accuracy precision recall F1-Score

dataset #1

0.6

0.7

0.8

0.9

1

accuracy precision recall F1-Score

dataset #2

0.6

0.7

0.8

0.9

1

accuracy precision recall F1-Score

dataset #3

0.6

0.7

0.8

0.9

1

accuracy precision recall F1-Score

dataset #4

Local_NN Centralized_NN Federated_NN CDW_Federated_NN

Fig. 9: Precision, recall, and F1-score of NN model.

[7] S. Omohundro, “Cryptocurrencies, smart contracts, and artificialintelligence,” AI Matters, vol. 1, no. 2, pp. 19–21, Dec. 2014. [Online].Available: http://doi.acm.org/10.1145/2685328.2685334

[8] H. Kim, J. Park, M. Bennis, and S. Kim, “Blockchained on-devicefederated learning,” IEEE Communications Letters, pp. 1–1, 2019.

[9] J. Weng, J. Weng, J. Zhang, M. Li, Y. Zhang, and W. Luo, “Deepchain:Auditable and privacy-preserving deep learning with blockchain-basedincentive,” IEEE Transactions on Dependable and Secure Computing,pp. 1–1, 2019.

[10] U. Majeed and C. S. Hong, “Flchain: Federated learning via mec-enabledblockchain network,” in 2019 20th Asia-Pacific Network Operations andManagement Symposium (APNOMS), Sep. 2019, pp. 1–4.

[11] W. Shi and S. Dustdar, “The promise of edge computing,” Computer,vol. 49, no. 5, pp. 78–81, 2016.

[12] H. B. McMahan, E. Moore, D. Ramage, and B. A. y Arcas,“Federated learning of deep networks using model averaging,” CoRR,vol. abs/1602.05629, 2016. [Online]. Available: http://arxiv.org/abs/1602.05629

[13] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N.Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings et al.,“Advances and open problems in federated learning,” arXiv preprintarXiv:1912.04977, 2019.

[14] Y. Zhao, J. Zhao, L. Jiang, R. Tan, and D. Niyato, “Mobile edgecomputing, blockchain and reputation-based crowdsourcing iot federatedlearning: A secure, decentralized and privacy-preserving system,” 2019.

[15] T. Yang, G. Andrew, H. Eichner, H. Sun, W. Li, N. Kong,D. Ramage, and F. Beaufays, “Applied federated learning: Improvinggoogle keyboard query suggestions,” CoRR, vol. abs/1812.02903, 2018.[Online]. Available: http://arxiv.org/abs/1812.02903

[16] M. Chen, R. Mathews, T. Ouyang, and F. Beaufays, “Federated learningof out-of-vocabulary words,” 2019.

[17] S. Ramaswamy, R. Mathews, K. Rao, and F. Beaufays, “Federatedlearning for emoji prediction in a mobile keyboard,” 2019.

[18] M. J. Sheller, G. A. Reina, B. Edwards, J. Martin, and S. Bakas, “Multi-institutional deep learning modeling without sharing patient data: Afeasibility study on brain tumor segmentation,” in Brainlesion: Glioma,Multiple Sclerosis, Stroke and Traumatic Brain Injuries, A. Crimi,S. Bakas, H. Kuijf, F. Keyvan, M. Reyes, and T. van Walsum, Eds.Cham: Springer International Publishing, 2019, pp. 92–104.

[19] S. Nakamoto, “Bitcoin: A peer-to-peer electronic cash system,” https://bitcoinsv.io/bitcoin, 2008, accessed 15-February-2020.

[20] Y. Lu, X. Huang, Y. Dai, S. Maharjan, and Y. Zhang, “Blockchain andfederated learning for privacy-preserved data sharing in industrial iot,”IEEE Transactions on Industrial Informatics, pp. 1–1, 2019.

[21] J. Kang, Z. Xiong, D. Niyato, S. Xie, and J. Zhang, “Incentive mech-anism for reliable federated learning: A joint optimization approachto combining reputation and contract theory,” IEEE Internet of ThingsJournal, vol. 6, no. 6, pp. 10 700–10 714, 2019.

[22] X. Bao, C. Su, Y. Xiong, W. Huang, and Y. Hu, “Flchain: A blockchainfor auditable federated learning with trust and incentive,” in 2019 5thInternational Conference on Big Data Computing and Communications(BIGCOM), Aug 2019, pp. 151–159.

[23] G. Kotonya and I. Sommerville, Requirements Engineering: Processesand Techniques, 1st ed. Wiley Publishing, 1998.

[24] S. Caldas, S. M. K. Duddu, P. Wu, T. Li, J. Konen, H. B. McMahan,V. Smith, and A. Talwalkar, “Leaf: A benchmark for federated settings,”2018.

[25] V. Buterin, “Ethereum white paper: a next generation smart contract &decentralized application platform,” 2013, accessed: 30 Jan 2019. [On-line]. Available: https://github.com/ethereum/wiki/wiki/White-Paper/.