Empirical Vulnerability Analysis of Automated Smart ContractsSecurity Testing on Blockchains

Reza M. PariziDepartment of Soware Engineering and Game

Development, Kennesaw State UniversityGA, USA

rparizi1@kennesaw.edu

Ali DehghantanhaSchool of Computer Science, University of Guelph

Ontario, Canadaa.dehghan@uoguelph.ca

Kim-Kwang Raymond ChooDepartment of Information Systems and Cyber Security,

University of Texas at San AntonioTexas, USA

raymond.choo@fulbrightmail.org

Amritraj SinghDepartment of Soware Engineering and Game

Development, Kennesaw State UniversityGA, USA

amritra@students.kennesaw.edu

ABSTRACTe emerging blockchain technology supports decentralized com-puting paradigm shi and is a rapidly approaching phenomenon.While blockchain is thought primarily as the basis of Bitcoin, itsapplication has grown far beyond cryptocurrencies due to the in-troduction of smart contracts. Smart contracts are self-enforcingpieces of soware, which reside and run over a hosting blockchain.Using blockchain-based smart contracts for secure and transparentmanagement to govern interactions (authentication, connection,and transaction) in Internet-enabled environments, mostly IoT, is aniche area of research and practice. However, writing trustworthyand safe smart contracts can be tremendously challenging becauseof the complicated semantics of underlying domain-specic lan-guages and its testability. ere have been high-prole incidentsthat indicate blockchain smart contracts could contain variouscode-security vulnerabilities, instigating nancial harms. Whenit involves security of smart contracts, developers embracing theability to write the contracts should be capable of testing their code,for diagnosing security vulnerabilities, before deploying them tothe immutable environments on blockchains. However, there areonly a handful of security testing tools for smart contracts. isimplies that the existing research on automatic smart contractssecurity testing is not adequate and remains in a very stage of in-fancy. With a specic goal to more readily realize the application ofblockchain smart contracts in security and privacy, we should rstunderstand their vulnerabilities before widespread implementation.Accordingly, the goal of this paper is to carry out a far-reachingexperimental assessment of current static smart contracts securitytesting tools, for the most widely used blockchain, the Ethereumand its domain-specic programming language, Solidity, to providethe rst body of knowledge for creating more secure blockchain-based soware.

Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor prot or commercial advantage and that copies bear this notice and the full citationon the rst page. Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).CASCON’18, Canada© 2018 Copyright held by the owner/author(s). xxxx. . .$15.00DOI: xxxx

CCS CONCEPTS•Security and privacy→ Soware security engineering;

KEYWORDSBlockchain, Smart Contracts, Ethereum, Solidity, EVM, SecurityTesting Tools, Experiments, Internet of ings, IoT Security, Vul-nerability Detection.ACM Reference format:Reza M. Parizi, Ali Dehghantanha, Kim-Kwang Raymond Choo, and Am-ritraj Singh. 2018. Empirical Vulnerability Analysis of Automated SmartContracts Security Testing on Blockchains. In Proceedings of 28th AnnualInternational Conference on Computer Science and Soware Engineering,Canada, October 2018 (CASCON’18), 11 pages.DOI: xxxx

1 INTRODUCTIONSmart Contracts (SC) have grown in popularity in the recent pastyears and are believed to be the next generation of automation ininter-party agreements in the blockchain-based systems. Smartcontracts are self-directing agreements implemented through apiece of soware whose autonomous execution applies the termsof the selement and measurements. e main idea behind SC isto urge obviate traditional trusted third parties (authority, entityor organization) to be replaced by pieces of code running on adecentralized and immutable system. is new paradigm of SCapplications opens the door to many opportunities. One of thepromising areas is for the IoT security and forensics which hasdrawn a lot of aention from both academia and the enterpriseworld [14, 17, 20, 24, 39, 43]. In fact, the implementation and useof blockchain has far surpassed its original intended purpose asthe backbone to the world’s rst decentralized cryptocurrency.e value of a trustless, decentralized ledger that carries historicimmutability has been recognized by other industries looking toapply the core concepts to existing business processes. ese uniqueproperties of the blockchain make its application an aractive ideafor many areas of business, including IoT. e major issue in IoTsecurity is to control and know ‘who’ will be connecting to thenetwork across a large number of things (e.g. sensors and devices)without breaching data privacy [12]. Decentralized smart contracts

arX

iv:1

809.

0270

2v1

[cs

.CR

] 7

Sep

201

8

CASCON’18, October 2018, Canada Reza M. Parizi et al.

on blockchains seem to be a vital remedy [44], [11], [2], especiallyin dealing with security aws in widely distributed IoT nodes [5].e blockchain technology [42] is the essential means for deliveringthis trust model envisioned by smart contracts, and appears to holdgreat promise for future IoT advancement.

Although blockchain technology is perceived as secure by de-sign, its embedded applications (i.e. SC) in dynamic environments(such as IoT) may introduce vulnerabilities in real life situations[30]. Aer all, these smart contract applications controlling nodesand transactions are snippets of code wrien by fallible humandevelopers. Besides, because of their special nature, mistakes orbugs can have a signicant nancial impact, hence security is ofutmost importance.

To date, smart contracts have been aected by unfortunate inci-dents and aacks (e.g., reentrancy problem in ’splitDAO’ functioncaused $40 million loss in June 2016 [3], and $32 million was stolenby aackers due a bug within the code in November 2017 [13]).ese high-prole incidents demonstrate developers (even thosewith experiences) may put behind security bugs in smart contractsthat could create serious vulnerabilities for aackers’ misuse [31].is would be further extended in IoT environments because oftheir sheer size and velocity [40]. us, writing trustworthy andsafe smart contracts can be extremely dicult due to the compli-cated semantics of underlying domain-specic languages and itstestability. One eective way to mitigate this issue is, every devel-oper embracing the ability to write their smart contracts shouldbe capable of testing their own code, against vulnerabilities, viaautomated static code analysis tools [10], [35], and before deployingthem to the immutable environments on blockchains. Being awareof the security pitfalls involved with writing smart contracts as apreventive strategy to avoid known mistakes and vulnerabilitiesis a smart choice, since the cost of failure on a blockchain thathandles an IoT network can be very high. Currently, the state ofempirical knowledge within the area of smart contracts security isin embryonic form. us, the aim of this paper is to take this initia-tive by providing a rst-time empirical evaluation of static smartcontracts security testing tools, for Ethereum as the most widelyused blockchain in the community and its popular domain-specicprogramming language, Solidity. As the soware and security en-gineering research community has long relied on Free and OpenSource Soware (FOSS) tools for security testing, our main em-phasis would be on open source tools as well. With this empiricalanalysis, the objective of the research is to assess the four FOSS tools(namely Oyente [25], Mythril [29], Securify [22], and SmartCheck[38]) based on their vulnerability detection eectiveness and theaccuracy. To accomplish the stated objective, we ponder to examinethe accompanying research questions:

• RQ1 - How eective are the automated smart contractsecurity testing tools, in terms of vulnerability detectionability? And what is the most eective tool?

• RQ2 - What are the accuracy scores obtained by the auto-mated testing security tools in detecting true vulnerabili-ties?

To manage RQ1 and RQ2, we carried out large-scale experimentsamong the aforementioned tools. Based on the presented researchquestions, the subsequent hypotheses were detailed:

• Null hypothesis (H0Ef f ) = ere is no signicant dier-ence in vulnerability detection eectiveness between thefour smart contract security testing tools. It can be ex-pressed as: H0Ef f : µOyente = µMythr il = µSecur if y =µSmartCheck . Where, µt is the mean detection eective-ness of testing tool t measured on all the participatingsmart contracts (as shown in Eq. (1)).

• Alternative hypothesis (H1Ef f ) = ere is signicant dif-ference in vulnerability detection eectiveness betweenthe four automated tools. It can be expressed as: Givenz = {Oyente, Mythril, Securify, SmartCheck}, x and yas two single-valued variables belonging to z, therefore,H1Ef f : ∃x |µx > µyϵz−x .

• Null hypothesis (H0Acc ) = ere is no signicant dier-ence in accuracy scores between the four automated tools.It can be expressed as: H0Acc : θOyente = θMythr il =θSecur if y = θSmartCheck . Where, θt is the mean of theaccuracy of testing tool t measured on all the participatingsmart contracts (as shown in Eq. (2)).

• Alternative hypothesis (H1Acc ) = ere is signicant dier-ence in accuracy scores between the four automated tools.It can be expressed as: Given z = {Oyente, Mythril, Secu-rify, SmartCheck}, x and y as two single-valued variablesbelonging to z, therefore, H1Acc : ∃x |θx > θyϵz−x .

Within our exploration, there is only one independent variable,i.e. the security testing tool designed for smart contracts. For thismain factor there are four treatments (tools): Oyente, Mythril, Secu-rify, SmartCheck (See Section 2.1). e vulnerability detection ef-fectiveness and the accuracy are the dependent variables measuredin this work. We are collecting the following four building-blockmetrics to measure the dependent variables:

• True Positive - TPc: tool correctly identies a real vulnera-bility in contract c.

• False Negative - FNc: tool fails to identify a real vulnera-bility in contract c.

• True Negative - TNc: tool correctly ignores a false alarmin contract c.

• False Positive - FPc: tool fails to ignore a false alarm incontract c.

e vulnerability detection eectiveness of a tool is measured asfollowing (which is also referred to as Recall [37] or sensitivity):

E f fj = (n∑i=1

#TPi#TPi + #FNi

/n) ∗ 100 (1)

Where n is the number of smart contracts used, and j representsa given tool (i.e., j = 1, 2, 3, 4). e accuracy score of a tool ismeasured as following (which is essentially a Youden Index [41]):

Acc j = (E f fj + ((n∑i=1

#TNi#TNi + #FPi

/n) ∗ 100)) − 1 (2)

e rest of this paper is organized as follows. Section 2 providesan overview of the current tools and background in smart contracts.Section 3 gives and describes the experimental setup; Section 4 givesexperimental design and validity. Section 5 analyzes the obtained

Empirical Vulnerability Analysis of Automated Smart Contracts Security Testing on Blockchains CASCON’18, October 2018, Canada

Table 1: Summary of current tools and their characteristics

Tool Level ofrigor

Analysisbasis

InterfaceMeans

Description

Oyente Heuristic source code(.sol)

Commandline-basedtool

Oyente [25] is an automated security analysis tool for revealingsecurity vulnerabilities in smart contracts.

Mythril Analyticand Heuris-tic

Byte codeand sourcecode (.sol)

Commandline-basedtool

Mythril [29] is an automated security analysis tool forEthereum smart contracts. It uses Concolic analysis, taintanalysis and control ow checking to detect a variety of secu-rity vulnerabilities. e analysis is based on laser-Ethereum, asymbolic execution library for EVM bytecode.

Securify Formal Byte codeand sourcecode (.sol)

UI-basedtool

Securify [22] is an automated formal security analysis tool forEthereum smart contracts. It can detect various security issuessuch as input validation, reentrancy, and others.

SmartCheck Analyticand Heuris-tic

source code(.sol)

UI-basedtool

SmartCheck [38] is an automated static code analyzer devel-oped by SmartDec Security Team. It runs analysis in Soliditysource code and automatically checks smart contracts for se-curity vulnerabilities and bad practices.

data and discusses the results; Section 6 presents the related workin the literature; and nally, Section 7 reports the conclusions.

2 BACKGROUNDe concept of smart contract was initially perceived at the sourcecode level, ‘code is law’. e code-based testing not only in blockchainprogramming (including smart contracts development) but also inother computing paradigms is still favored and eective as it deter-mines a reasonable degree of reliance on the most comprehensiveartifact of development process, and before deploying it to the host-ing environment [32, 34]. In addition, due to the special nature ofsmart contracts on the blockchain, static security analysis prior todeployment seems to be the perfect t. Inspired by this, the paperprimarily focuses on the automated static smart contracts securitytools (see Section 2.1) in the Ethereum blockchain by focusing itsaention on Solidity (see Section 2.2).

2.1 Automated Smart Contract Security TestingTools

ere are a handful of tools for automated smart contract (wrien inSolidity) security vulnerability testing based on code-level analysis.We give a synopsis of the four most related FOSS tools that we usedin our experiments, namely Oyente [25], Mythril [29], Securify[22], and SmartCheck [38]. Table 1 shows the summary of theselected tools and their main properties. Level of rigor, ranging fromsyntactic, heuristic, analytic to fully formal, refers to underlyingsecurity testing technique of the given tool. e Analysis basiscolumn represents the type of artifact (source code, byte code, orbinary) that a tool uses as its input source.

2.2 Solidity, EVM, and Ethereum SmartContracts

Ethereum is a consensus-based framework that utilizes the blockchaintechnology to oer a globally open decentralized computing plat-form, referred to as Ethereum Virtual Machine (EVM). EVM pro-grams are implemented in bytecode that operates on a simple stackmachine supporting a handful of instructions. Contracts reside onthe blockchain in an Ethereum-specic binary format (EVM byte-code). Developers, however, do not usually write low-level EVMcodes for contracts. Instead, they typically use a high-level languagein an exceedingly JavaScript-like syntax called Solidity that com-piles into bytecode to be uploaded on the blockchain. Ethereum isgaining a substantial popularity in the blockchain community, sincethe major purpose of EVM programs is as smart contracts that man-age digital assets [8], and for creating decentralized applications(dApps).

Solidity [16] is a domain and platform-specic language that isdesigned for writing smart contracts on the Ethereum blockchain.It has been the most widely used open-source programming lan-guage in implementing public and private blockchains [36]. esyntax of Solidity is analogous to ECMAScript (JavaScript), wherethe resulting code associated with a smart contract is compiled tobytecode and then is run on the EVM to execute the functionalityof the contract.

Similar to dierent blockchain platforms (e.g., Bitcoin, Hyper-ledger Fabric), Ethereum provides a peer-to-peer (P2P) networkfor the users. e underlying Ethereum blockchain database issupported and updated by the participating nodes connected to thecurrent network. Each node on the network runs the EVM andexecutes identical set of directions. e Ethereum platform itself isfeatureless or value-agnostic. It is up to organizations and develop-ers to choose its utilization in view of their business objectives. Inany case, certain application composes assortments benet morethan others from Ethereum’s capabilities. In particular, Ethereum ised to applications that automate coordinate association between

CASCON’18, October 2018, Canada Reza M. Parizi et al.

peers or encourage facilitated bunch activity over a network. Forexample, applications for planning digital commercial centers, orthe computerization of complex nancial related contracts. Withregards to programming on Ethereum, there are some key focusesto note from the Ethereum Design Rationale (EDR) document [15].

In the context of Ethereum, a typical smart contract is recognizedby a unique contract address rendered aer a successful productionof a creation transaction. A blockchain state is hence a mappingfrom addresses to nodes’ accounts in the network, where each smartcontract account holds an associate quantity of virtual coins (Ether- the contract balance) by possessing its own non-public state andstorage.

Figure 1 shows an illustrative sample for a user-dened smartcontract, named Puzzle [25]. is contract manages a simplereward system in which whoever solves a problem (i.e. puzzle)will get rewards. To make this contract up and running, the EVMbytecode of the contract is rst sent to miners through a contract-creation transaction. Once the transaction is admied into a block(has to pass through mining process) in the hosting blockchain, aunique address for Puzzle is generated. en each miner instantiatesthe contract by executing its constructor (Line 8), and a local storageis assigned in the blockchain. Finally, the main code implementedin the anonymous function of Puzzle (Lines 15) is added to thecontract’s storage.

Figure 1: Smart contract example in Solidity

Once the contract is live on the blockchain, it can be used todeliver its service to users. Whenever a user submits his/her solu-tion, a contract-invoking transaction gets directed to the associatedaddress of the contract. is in turn triggers the execution of themain function dened at Line 15. In this case, all the sender’s in-formation, the Ether amount, and the input data of the invokingtransaction will be stored in the msg variable. e contract thenprocesses the receiving data and handles the reward according toits business logic wrien in the main function. When a correct

solution is received, the contract automatically redeems the rewardto the sender by executing the code at Line 24.

3 EXPERIMENTAL SETUPe general process of experiments is depicted in Figure 2. eoperation process starts with choosing a security testing tool in arandom manner, i.e. treatment, following by acquiring a chosenrandom smart contract from the pool of contracts.

Figure 2: e general experiment execution process

Tools without a Web-based UI were run from their source les onan online compiler, i.e. Remix1 as Solidity IDE, or solc command-lineSolidity compiler - whichever possible. We followed the instructionson tools’ respective GitHub pages to congure installation. Priorto the experiment, we compiled all contracts and made sure theywere all free from compilation errors.

Aer running the tools against each smart contract, we recordedthe security threats they raised, and assessed whether or not theyaccurately identied the known vulnerabilities, given the four-fundamental metrics of classication, TP, FN, TN, FP.

As with any experimental seing, we were required to collatea set of object programs (in our case ‘vulnerable smart contracts’)through observation of experiments execution process. One of thechallenges we faced was the diculty of nding representativecontracts to use in our study without geing involved in legal andprivacy issues. Historically, authors or developers have seen theavailability of deliberately vulnerable products (such as vulnerableWebsite for Web Apps security testing) as a practical solution to1hp://remix.ethereum.org/#optimize=false&version=soljson-v0.4.21+commit.dfe3193c.js.

Empirical Vulnerability Analysis of Automated Smart Contracts Security Testing on Blockchains CASCON’18, October 2018, Canada

pursue their studies without the fear of prosecution. In this spiritand aer an exhaustive search in the online resources, we foundtwo suitable publicly-available sources of intentionally vulnerablecontracts: (1) contracts developed by Trail of Bits2 , a dedicatedsecurity team on foundational tools and deep expertise in cryp-tography and Ethereum security; (2) contracts developed by theEthernaut3 wargame, an online smart contract hacking challengeby Zeppelin. Although one of the participating tool (Mythril) comeswith a collection of test contracts4, we decided not to use thosewith a specic end goal to make the evaluation fair-minded and togive a reasonable equivalent platform for analyzing all the tools.

Aerwards, we selected a suit of ten contracts wrien in Solidity,as presented in Table 2. ese contracts encompass a wide range ofreal-world vulnerabilities (including Integer Overow, Missing Con-structor, Reentrancy, Unchecked External Call, Unprotected Func-tion, Wrong Interface, Callstack Depth Aack, Assertion Failure,Timestamp Dependency, Parity Multisig Bug, Transaction-OrderingDependence (TOD, etc.) with many instances for each class of vul-nerability. In this way, the outcomes started from these contractsfor the proposed experiments will give premise to drawing moreconvincing bits of knowledge. We intentionally only chose opensource contracts to ensure that our experiments can be replicatedby public access to the source code.

Table 2: Smart contracts used in the experiment.

Smart contracts # function #LOC Source

Fallback 5 33 EthernautFallout 5 31 EthernautToken 3 20 EthernautKing 2 21 Ethernaut

Re-entrancy 4 25 EthernautRaceCondition 9 50 Trail of Bits

Rubixi 18 155 Trail of BitsGiBox 7 69 Trail of Bits

KingOfeEtherrone 7 170 Trail of BitsWalletLibrary 44 463 Trail of Bits

All experiments were conducted on a similar computer (to main-tain a strategic distance from ecological predisposition) utilizingsolc compiler (v0.4.21) on an Intel Core i7 at 2.9 GHz and 16 GbRAM, under Mac OS operating system. All applied statistical testswere run using IBM SPSS Statistics v.25 to decide the rejection oracceptance of formulated null hypotheses. To help selecting theproper statistical tests, we performed normal probability plots onthe experimental data using p-p and q-q plots and residual analysisto gain insights into the distribution of the data. In view of thedependent variables and the results of normality plots, we utilizedthe ANOVA test which is solid and reliable. In all hypothesis testing,a 5% signicance level was chosen, henceforth we acknowledge a5% likelihood of commiing a type-I-error that is rejecting a nullhypothesis when it is sure.

2hps://www.trailoits.com/3hps://ethernaut.zeppelin.solutions/4hps://github.com/ConsenSys/mythril/tree/master/solidity examples

4 EXPERIMENTAL DESIGN AND VALIDITYWe designed our experiments as a randomized block with no humansubjects included. Each smart contract was utilized specically onceto measure the impact of each treatment (tool). In other words, eachof four security testing tools were randomly applied on the sameten smart contracts. is design helped to divide the variabilityinto variability due to tools (i.e. treatments) and variability due tosmart contracts (i.e. blocks). us, the impact of the tools could beanalyzed without meddling from the impact of contracts coveringthe result of the investigations. e statistical analysis was based onthe formal model [33] underlying ANOVA for a randomized blockdesign of yi j = λ + ai + bj + εi j , where yi j is the eectiveness orthe accuracy score for contract i given treatment j, λ is the overallmean eectiveness or accuracy score (i.e. µt or θt ), ai is the impactof contract i (i=1,…,10), and bj is the impact of treatment j (j=1,…,4).

e threats to validity of this empirical research are as per thefollowing. Concerning external validity, we used a subset of well-known open-source smart contracts from the blockchain commu-nity in assessing and testing smart contracts. e chosen smartcontracts are generally suitable in terms of size and the number offunctionalities. Without a doubt, they are suciently enormous tobe reasonable and make the experiment practical. Moreover, theobject programs contain diverse types of common vulnerabilities.As for the security testing tools, the chosen ones are the most com-monly cited and used tools available in the literature at the time ofthis experiment. e threats to conclusion validity are concernedwith the statistical analysis underling the conclusions. We selectedour statistical tests based a thorough pre-analysis of the normalityassessment of experiment’s data (using p-p and q-q plots). As pre-scribed by plots, ANOVA and LSD (least signicant dierence) testswere chosen to investigate the null hypotheses (H0Ef f and H0Acc ),which are solid. All the assumptions required by ANOVA test werefurther met. erefore, the error rate would not be evident and thethreat to conclusion validity could be minor.

5 RESULTS AND DISCUSSIONe collected data are analyzed and discussed with respect to eachresearch question in this section. In addition to descriptive analysisof data, we further perform hypothesis testing using ANOVA andLSD statistical tests to report on the signicance of the results.

5.1 Analysis of Eectiveness (RQ1)e main visualization technique that we used to contrast the per-formance of tools was the Receiver Operating Characteristic (ROC5)analysis. e ROC curve shows how the recall vs. precision re-lationship changes, and we can learn a lot about a tool accuracyfrom this analysis. A ROC curve plots the true positive rate on they-axis while the false positive rate is shown on the x-axis. e truepositive rate (TPR) is the recall (sensitivity) and the false positiverate (FPR) is 1- specicity (i.e. F PcF Pc+T Nc the probability of a falsealarm by a tool on contract c).

Following OWASP benchmark project6, Figure 3 shows the in-terpretation guide for ROC plots. Each plot shows a point ploed

5hp://www.statisticshowto.com/c-statistic/6hps://www.owasp.org/index.php/Benchmark

CASCON’18, October 2018, Canada Reza M. Parizi et al.

on the chart which provides a visual indication of how well a tooldid in security vulnerability detection task.

Figure 3: Interpretation guide

As it can be seen from Figure 4, Oyente tool is not a good instru-ment since it is far from top le zone (ideal tool). is tool showstwo breaking points when it crossed the guessing line and scoredlower than random possibilities. is could be a strong disadvan-tage of this analyzer. Additionally, as we observed Oyente raisedseveral outdated vulnerabilities that deal with class-stack depthas well as various falsely transaction-ordering dependence (TOD)alarms.

Figure 5 shows the ROC plot for the Securify tool. Securifyshowed a steady performance throughout the trials. It had a largenumber of false positives, but is still catching more threats thatOyente was missing.

Figure 6 shows the ROC plot for the Mythril tool. By lookingat the graph, it is obvious that Mythril was able to detect morevulnerabilities compared to Oyente and Securify tools. It can beconcluded that Mythril has a reasonable sensitivity and can beconsidered a trustworthy tool.

As it can be seen from Figure 7, SmartCheck showed a highdegree of sensitivity compared to all other tested tools. Although,there was a steady ramp in the graph, this could be due to poordesign in some of the contracts’ logics.

Figure 8 contrast the percentages of eectiveness obtained fromthe testing tools over the ten smart contracts. e x-axis representsthe ten smart contracts used and the y-axis signies the eective-ness score for each single tool applied as result of the experiment.Neither tool catches everything, but it is clear that SmartCheckachieved much beer score compared to its peers. Overall, the E f fvalues of all smart contracts (10 out of 10) for SmartCheck tool havethe highest gures.

Figure 9 illustrates the total mean of the vulnerability detectioneectiveness each tool from all ten smart contracts. e y-axis

Figure 4: ROC plot for Oyente tool

Figure 5: ROC plot for Securify tool

shows the tools and the x-axis shows the percentage of the aver-age eectiveness. As it can be seen from the gure, SmartCheckaccounted for the highest eectiveness in detection ability whilethe rest of tools had much lower statistics in all.

e provided descriptive comparisons (Figure 8 and Figure 9)intuitively give some insights into the eectiveness of the auto-mated security testing tools in which not all data are identical(i.e. µOyente , µSecur if y , µMythr il , µSmartCheck ). WhileSmartCheck demonstrated beer results, this could make it con-ceivable to indicate that a distinction in vulnerability detectionability was noticed. Yet, it would not be adequate to reach con-clusions, including rejecting the null hypothesis (H0Ef f ), as it stillrequires conducting the statistical tests (given below) to decide thesignicant of the dierences.

Empirical Vulnerability Analysis of Automated Smart Contracts Security Testing on Blockchains CASCON’18, October 2018, Canada

Figure 6: ROC plot for Mythril tool

Figure 7: ROC plot for SmartCheck tool

Hypothesis testing (H0Ef f ): Table 3 shows the results of ANOVAtest on the eectiveness of the tools. Given Table 3 and based onthe decision rule, reject H0 if p-value ¡ α (whereas 0.0003 < 0.05).erefore, the null hypothesis (H0Ef f ) is rejected and we can ac-cept the alternative hypothesis, H1Ef f . In light of this analysis,the conclusion was drawn that there is a statistically signicantdistinction in the vulnerability capability of security testing toolsfor smart contracts.

Furthermore, a post-hoc statistical test by means of least signi-cant dierence (LSD) was performed to help ascertain where thedierences between tools lie and nd the most eective securitytesting tool. Because, the ANOVA tests alone was not able to indi-cate which specic testing tools were signicantly dierent fromeach other.

e results of this particular LSD test are shown in Table 4. Foreach pair of the tools the mean dierence between their E f f scores

Figure 8: Comparison of the vulnerability detectability oftools across all smart contracts

Figure 9: Comparison of the average eectiveness by secu-rity testing tools

Table 3: Results of ANOVA test on the eectiveness of thetools

Sum ofSquares

df MeanSquare

F Sig.

BetweenGroups

1491.176 3 497.059 8.214 0.0003

WithinGroups

1936.413 36 60.513

Total 3427.590 39

(Eq.1), the standard error of the dierence, the signicance level ofthe dierence, and a 95% condence interval are shown. Signicantoutcomes are set apart with a mark (*) by SPSS soware. In this case,SmartCheck tool demonstrated a signicant dierence comparedto all other tools (Sig. < 0.05).

CASCON’18, October 2018, Canada Reza M. Parizi et al.

Table 4: LSD multiple comparisons on the eectiveness of tools

(I) Factor (J) Factor Mean Dierence (I-J) Std. Error Sig.95% Condence IntervalLower Bound Upper Bound

OyenteSecurify -3.41111 3.66706 0.359 -10.8807 4.0584Mythril -6.88111 3.66706 0.07 -14.3507 0.5884SmartCheck -17.19111* 3.66706 0.002 -24.6607 -9.7216

SecurifyOyente 3.41111 3.66706 0.359 -4.0584 10.8807Mythril -3.47 3.66706 0.351 -10.9396 3.9996SmartCheck -13.78* 3.66706 0.001 -21.2496 -6.3104

MythrilOyente 6.88111 3.66706 0.07 -0.5884 14.3507Securify 3.47 3.66706 0.351 -3.9996 10.9396SmartCheck -10.31* 3.66706 0.008 -17.7796 -2.8404

SmartCheckOyente 17.19111* 3.66706 0.002 9.7216 24.6607Securify 13.78* 3.66706 0.001 6.3104 21.2496Mythril 10.31* 3.66706 0.008 2.8404 17.7796

* e mean dierence is signicant at the 0.05 level.

5.2 Analysis of Accuracy (RQ2)Only achieving the highest eectiveness rate at any precision wouldnot be a veritable pointer of best performance when it comes toreal-world applications and tools. In this manner, it would be asimperative to investigate the accuracy scores obtained by the toolsconcerning false alarms to arrive at a more solid assessment. Froma technical perspective, this two-facet evaluation would encouragefellow researchers and developers to pick the well-ed tool byobserving the trade-o between the eectiveness and the accu-racy. e experiment results related to this aspect of analysis arepresented in Figure 10 and Figure 11.

Figure 10: Comparison of the accuracy of tools across allsmart contracts

Figure 10 shows the percentages of accuracy scores obtained forthe tools over the ten smart contracts. e x-axis shows the tensmart contracts and the y-axis signies the percentage of accuracyfor each tool in the experiment. From the gure, Mythril andSmarkCheck tools almost outperformed all other peer tools in allten smart contracts with recording highest accuracy scores.

Figure 11: Comparison of the average accuracy obtained bysecurity testing tools

Table 5: Results of ANOVA test on the accuracy of the tools

Sum ofSquares

df MeanSquare

F Sig.

BetweenGroups

2348.290 3 782.763 14.909 0.0002

WithinGroups

1890.046 36 52.501

Total 4238.336 39

On the average basis and interestingly, Mythril tool showedthe highest accuracy score, though it had less eectiveness thanSmartCheck. Overall, it is safe to say that there were dierencesobserved in the accuracy of all the testing tools, but descriptiveresults could indicate that the Mythril and SmartCheck tools aremore accurate than Oyente and Securify with less false alarms.

Empirical Vulnerability Analysis of Automated Smart Contracts Security Testing on Blockchains CASCON’18, October 2018, Canada

Table 6: LSD multiple comparisons on the accuracy of the tools

(I) Factor (J) Factor Mean Dierence (I-J) Std. Error Sig.95% Condence IntervalLower Bound Upper Bound

1.002.00 -5.01500 3.24041 0.13 -11.5869 1.55693.00 -20.12500* 3.24041 0 -26.6969 -13.55314.00 -12.90200* 3.24041 0 -19.4739 -6.3301

2.001.00 5.01500 3.24041 0.13 -1.5569 11.58693.00 -15.11000* 3.24041 0 -21.6819 -8.53814.00 -7.88700* 3.24041 0.02 -14.4589 -1.3151

3.001.00 20.12500* 3.24041 0 13.5531 26.69692.00 15.11000* 3.24041 0 8.5381 21.68194.00 7.22300* 3.24041 0.032 0.6511 13.7949

4.001.00 12.90200* 3.24041 0 6.3301 19.47392.00 7.88700* 3.24041 0.02 1.3151 14.45893.00 -7.22300* 3.24041 0.032 -13.7949 -0.6511

* e mean dierence is signicant at the 0.05 level.

e ANOVA and LSD statistical tests performed below show thesignicant of the dierences.

Hypothesis testing (H0Acc ): To factually exhibit the accuracy’sdierences between the tools under evaluation, the ANOVA wasonce again carried out to check the respected hypothesis (H0Acc ).Likewise with the past ANOVA test on the eectiveness, the con-dence level of signicance for the hypothesis testing was set toα = 0.05 as well. Table 5 and Table 6 show the results of ANOVAstatistical test and LSD respectively.

As it can be seen from Table 5, p-value (0.0002) is less thanα = (0.05), thus the results suggest rejecting the null hypothesisH0Acc in favor of the alternativeH1Acc at the 0.05 signicance level.Following the recommendation, the choice was made to dismissthe null hypothesis and as needs be, acknowledge the alternativehypothesis.

As we were keen to nd out where the contrasts between thetesting tools lie, a LSD test was additionally performed to indicatethe dierences in a pair-wise manner. For each couple of the toolsthe mean dierence between their accuracy scores (Eq. (2)), thestandard error of the dierence, the signicance level of the dier-ence, and a 95% condence level are shown. Notably, Mythril andSmartCheck tools demonstrated signicant dierences in all pairswith the remaining tools (Sig. < 0.05).

Finally, it can be concluded that the results of these experimentsseemingly set up a trade-o between the vulnerability capabilitydetection and the accuracy of smart contract-specic security test-ing tools in the Ethereum blockchain, likewise with classic trade-obetween eectiveness and eciency reported in the literature inall realms of testing.

6 RELATEDWORKe nature of work given in this paper is generally related to secu-rity testing and vulnerability assessment of smart contracts on the

blockchain. is section presents the related work (including ap-proaches and tools) and secondary studies (including experimentsand surveys) in this area.

Most recently, Parizi et al. [31], conducted an empirical analysisof smart contract programming languages based on usability andsecurity from new developers point of view. ey selected threeprogramming languages for their experiment, i.e. Solidity, Pact7(a high-level language for the Kadena8 platform) and Liquidity9(a high-level language for the Tezos10 platform). e results oftheir experiment indicated that new contract developers foundSolidity to be the most ecient language with the highest usabilityscore and the shortest average implementation times. But, it wasfound that 73.33% of implemented Solidity contracts had securityvulnerabilities, while no known security vulnerabilities were foundin contracts implemented with Pact and Liquidity. In conclusion, thestudy concluded that although Solidity is the most usable languageto a new developer, it is also the most vulnerable to malicious aacksas new developers tend to leave behind security vulnerabilitieswhich can leave the contracts insecure.

Destefanis et al. [13] advocated the need for a discipline ofBlockchain Soware Engineering (BOSE), addressing the securityissues posed by smart contract programming and other applicationsrunning on the Ethereum blockchain. e authors presented a casestudy of the Parity11 wallet’s smart contract library, where poorprogramming practices led to a situation where an anonymous userwas able to freeze about 500K Ether (150M USD) in November 2017.e analysis of the case led to the authors concluding that vulnera-bility of the library was mainly due to a negligent programmingactivity rather than a problem in the Solidity language.

Atzei et al. [3] provided a systematic exposition of the securityvulnerabilities of Ethereum and Solidity. e authors presented a

7hp://kadena.io/docs/Kadena-PactWhitepaper.pdf8hp://kadena.io/9hps://github.com/OCamlPro/liquidity/blob/master/docs/liquidity.md10hps://tezos.com/11hps://www.parity.io/

CASCON’18, October 2018, Canada Reza M. Parizi et al.

taxonomy of causes of vulnerabilities, classifying them on threelevels namely Solidity (Call to the unknown, Gasless send, Excep-tion disorders, Type casts, Reentrancy, keeping secrets), EVM (Im-mutable bugs, Ether lost in transfer, Stack size limit) and Blockchain(Unpredictable state, Generating randomness, Time constraints).Additionally, the authors accompanied their taxonomy with actualaacks which exploit detected vulnerabilities except Type casts,Ether lost in transfer and generating randomness vulnerabilities.To conclude their survey, the authors recommended the process offormal verication of smart contracts to ensure that the intendedbehavior and the actual behavior of the smart contracts are ex-actly the same. Moreover, the authors suggested the use of Turing-incomplete, human-readable languages for formal verication asthe choice of Turing complete languages limits formal verication.

While the above-mentioned works primarily focused on com-paring and highlighting the possible security vulnerabilities withEthereum and Solidity, the works mentioned below focus on min-imizing and mitigating security vulnerabilities with various ap-proaches for verication of smart contracts [26] on the Ethereumblockchain.

Abdellatif and Brousmich [1] proposed a semantics-based formalmethod approach for vercation of smart contracts and blockchainproperties. e authors modeled a smart contract’s behavior and in-teractions with its hosting environment by applying this approachthrough an illustrative example. ey simulated these behaviors inthe BIP (Behavior Interaction Priorities) framework [6], equippedwith a series of runtime verication and simulation engines [7]. Inthis study, results were analyzed using a SMC (Statistical ModelChecking) [23] tool, which allowed the authors to reveal scenar-ios where the smart contract behaviors could be compromised bymalicious users.

Bhargavan et al. [8] designed a framework to examine and verifythe runtime safety and the functional correctness of the Ethereumcontracts by translation to F*, a functional programming languageaimed at program verication. e authors proposed two prototypetools which translate the contracts to F* programs for developingmore secure end-contracts. In a similar work, Grishchenko et al.[18] presented the rst complete small-step semantics of EVMbytecode, which they formalized in the F* proof assistant based ona combination of hyper- and safety properties, obtaining executablecode that they successfully validated against the ocial Ethereumtest suite.

Mavridou and Laszka [27] introduced FSolidM12 , a frameworkrooted in rigorous semantics for designing smart contracts as FiniteState Machines (FSM). e authors presented a tool for creating FSMon a highly usable GUI and for automatically generating Ethereumsmart contracts. e authors [28] provided a demonstration of theFSolidM tool in a later work.

Kalra et al. [21] proposed a framework ’ZEUS’ to verify thecorrectness and validate the fairness of smart contracts. ZEUSleverages both abstract interpretation and symbolic model check-ing, along with the power of constrained horn clauses to quicklyverify contracts for safety. e authors built a prototype of ZEUSfor Ethereum and Fabric13 blockchain platforms and evaluated it

12hps://github.com/anmavrid/smart-contracts13hps://www.hyperledger.org/projects/fabric

with smart contracts. e evaluation found 94.6% of contracts (con-taining cryptocurrency worth more than USD 0.5 billion) to besecurely vulnerable.

Breindenbach et al. [9] proposed the Hydra Framework, which isa principled approach for modeling and administering bug bountiesthat incentivize bug disclosure. e framework transforms pro-grams via N-of-N-version programming (NNVP), a variant of clas-sical N-version programming [4] that runs multiple independentprogram instances. e Hydra Framework was applied to samplesmart contracts and it was concluded it could greatly amplify thepower of bounties to incentivize bug disclosure by economicallyrational adversaries.

Lastly, Idelberger et al. [19] investigated the possibilities of us-ing logic-based smart contracts on blockchain systems to boosttheir general safety and security. e authors demonstrated thata logical-based approach could benecially complement its pro-cedural counterpart with respect to the negotiation, formation,storage/notarizing, enforcement, monitoring and activities relatedto dispute resolution. It was proven that the logic and proceduralapproaches are not incompatible, contrarily, they have the poten-tial to advantageously complement each other for more qualitycontracts.

In summary, it can be said that sound empirical studies in theeld of smart contract security is currently lagging as comparedto primary studies. To the extent of our knowledge, the workpresented in this paper was the rst of its kind that statistically ana-lyzes and evaluates security analysis tools for smart contracts basedon the eectiveness in vulnerability detection and the accuracy.

7 CONCLUSIONSIn this paper, we presented a comprehensive empirical evaluationof open source automatic security analysis tools for the securityvulnerability detection of Ethereum smart contracts wrien in So-lidity. We tested those tools on ten real-world smart contracts fromboth vulnerability eectiveness and accuracy of true detection view-points. e results of our experiments showed that SmartChecktool is statistically more eective than the other automated securitytesting tools at 95% signicance level (p < 0.05). Concerning theaccuracy, Mythril was found to be signicantly (p < 0.05) accuratewith issuing the lowest number of false alarms among peer tools.ese results could imply that SmartCheck could currently be themost eective static security testing tool for Solidity smart con-tracts on the Ethereum blockchain but perhaps less accurate thanMythril.

As a general conclusion, our work indicates that research onthe empirical knowledge evaluation of security testing for smartcontracts is scarce in the literature, especially in relation to IoT, asperhaps this domain is still in a state of infancy. Hence, our workcontributed towards lling this gap by providing: (1) a ne-grainedmethodology to conduct such empirical study for future use byfellow researchers, (2) comparable experimental results on the state-of-the-art smart contracts security testing tools, and (3) statisticaltests and constructive insights into the challenges associated withtesting smart contracts. We hope our work motivates researchersand developers to come up with more new and innovative ideas,frameworks, and tools that would result in writing safer, secure

Empirical Vulnerability Analysis of Automated Smart Contracts Security Testing on Blockchains CASCON’18, October 2018, Canada

and vulnerability-immune smart contracts in the future. In ourview, IoT combined with smart contracts on blockchains are helpfulin building more trustworthy and secure networks, and it wouldappear to hold great promise for future IoT security development.

REFERENCES[1] Tesnim Abdellatif and Kei-Léo Brousmiche. 2018. Formal verication of smart

contracts based on users and blockchain behaviors models. In New Technologies,Mobility and Security (NTMS), 2018 9th IFIP International Conference on. IEEE,1–5.

[2] Michael P Andersen, John Kolb, Kaifei Chen, Gabriel Fierro, David E. Culler,and Raluca Ada Popa. 2017. WAVE: A Decentralized Authorization System for IoTvia Blockchain Smart Contracts. Technical Report UCB/EECS-2017-234. EECSDepartment, University of California, Berkeley. hp://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-234.html

[3] Nicola Atzei, Massimo Bartolei, and Tiziana Cimoli. 2017. A survey of aackson ethereum smart contracts (sok). In Principles of Security and Trust. Springer,164–186.

[4] Algirdas Avizienis. 1995. e methodology of n-version programming. Sowarefault tolerance 3 (1995), 23–46.

[5] Amin Azmoodeh, Ali Dehghantanha, and Kim-Kwang Raymond Choo. 2018.Robust Malware Detection for Internet Of (Baleeld) ings Devices UsingDeep Eigenspace Learning. IEEE Transactions on Sustainable Computing (2018).

[6] Ananda Basu, Marius Bozga, and Joseph Sifakis. 2006. Modeling heterogeneousreal-time components in BIP. In Soware Engineering and Formal Methods, 2006.SEFM 2006. Fourth IEEE International Conference on. IEEE, 3–12.

[7] Saddek Bensalem, Marius Bozga, anh-Hung Nguyen, and Joseph Sifakis. 2009.D-nder: A tool for compositional deadlock detection and verication. In Inter-national Conference on Computer Aided Verication. Springer, 614–619.

[8] Karthikeyan Bhargavan, Antoine Delignat-Lavaud, Cédric Fournet, Anitha Gol-lamudi, Georges Gonthier, Nadim Kobeissi, Natalia Kulatova, Aseem Rastogi,omas Sibut-Pinote, Nikhil Swamy, et al. 2016. Formal verication of smartcontracts: Short paper. In Proceedings of the 2016 ACMWorkshop on ProgrammingLanguages and Analysis for Security. ACM, 91–96.

[9] Lorenz Breindenbach, Phil Daian, Florian Tramèr, and Ari Juels. 2018. En-ter the Hydra: Towards Principled Bug Bounties and Exploit-Resistant SmartContracts. In 27th {U SEN IX } Security Symposium ({U SEN IX } Security 18).{U SEN IX } Association, Baltimore, MD. hps://www.usenix.org/conference/usenixsecurity18/presentation/breindenbach

[10] Brian Chess and Gary McGraw. 2004. Static analysis for security. IEEE Security& Privacy 2, 6 (2004), 76–79.

[11] Konstantinos Christidis and Michael Devetsikiotis. 2016. Blockchains and smartcontracts for the internet of things. IEEE Access 4 (2016), 2292–2303.

[12] M. Conti, A. Dehghantanha, K. Franke, and S. Watson. 2018. Internet of ingssecurity and forensics: Challenges and opportunities. Future Generation ComputerSystems 78 (2018), 544–546.

[13] G. Destefanis, M. Marchesi, M. Ortu, R. Tonelli, A. Bracciali, and R. Hierons. 2018.Smart contracts vulnerabilities: a call for blockchain soware engineering?,.In 2018 International Workshop on Blockchain Oriented Soware Engineering(IWBOSE). 19–25.

[14] Gregory Epiphaniou, Petros Karadimas, Dhouha Kbaier Ben Ismail, HaiderAl-Khateeb, Ali Dehghantanha, and Kim-Kwang Raymond Choo. 2017. Non-reciprocity compensation combined with turbo codes for secret key generationin vehicular ad hoc social IoT networks. IEEE Internet of ings Journal (2017).

[15] Ethereum. [n. d.]. Design Rationale. hps://github.com/ethereum/wiki/wiki/Design-Rationale

[16] Ethereum. [n. d.]. Solidity. hp://solidity.readthedocs.io/en/v0.4.24/[17] Chong-zhi Gao, Qiong Cheng, Pei He, Willy Susilo, and Jin Li. 2018.

Privacy-preserving Naive Bayes classiers secure against the substitution-then-comparison aack. Information Sciences 444 (2018), 72–88.

[18] Ilya Grishchenko, Maeo Maei, and Clara Schneidewind. 2018. A SemanticFramework for the Security Analysis of Ethereum smart contracts. In Interna-tional Conference on Principles of Security and Trust. Springer, 243–269.

[19] Florian Idelberger, Guido Governatori, Régis Riveret, and Giovanni Sartor. 2016.Evaluation of logic-based smart contracts for blockchain systems. In InternationalSymposium on Rules and Rule Markup Languages for the Semantic Web. Springer,167–183.

[20] Rutvij H Jhaveri, Narendra M Patel, Yubin Zhong, and Arun Kumar Sangaiah.2018. Sensitivity Analysis of an Aack-Paern Discovery Based Trusted RoutingScheme for Mobile Ad-Hoc Networks in Industrial IoT. IEEE Access 6 (2018),20085–20103.

[21] Sukrit Kalra, Seep Goel, Mohan Dhawan, and Subodh Sharma. 2018. Zeus:Analyzing safety of smart contracts. NDSS.

[22] Soware Reliability Lab. [n. d.]. Securify. hps://securify.ch/[23] Axel Legay, Benoı̂t Delahaye, and Saddek Bensalem. 2010. Statistical model check-

ing: An overview. In International conference on runtime verication. Springer,

122–135.[24] Jin Li, Lichao Sun, Qiben Yan, Zhiqiang Li, Witawas Srisa-an, and Heng Ye.

2018. Signicant Permission Identication for Machine Learning Based AndroidMalware Detection. IEEE Transactions on Industrial Informatics (2018).

[25] Loi Luu, Duc-Hiep Chu, Hrishi Olickel, Prateek Saxena, and Aquinas Hobor.2016. Making smart contracts smarter. In Proceedings of the 2016 ACM SIGSACConference on Computer and Communications Security. ACM, 254–269.

[26] Daniele Magazzeni, Peter McBurney, and William Nash. 2017. Validation andverication of smart contracts: A research agenda. Computer 50, 9 (2017), 50–57.

[27] Anastasia Mavridou and Aron Laszka. 2017. Designing secure Ethereum smartcontracts: A nite state machine based approach. arXiv preprint arXiv:1711.09327(2017).

[28] Anastasia Mavridou and Aron Laszka. 2018. Tool demonstration: FSolidMfor designing secure Ethereum smart contracts. In International Conference onPrinciples of Security and Trust. Springer, 270–277.

[29] B. Mueller. [n. d.]. Mythril. hps://github.com/ConsenSys/mythril.[30] H.H. Pajouh, R. Javidan, R. Khayami, A. Dehghantanha, and K.K.R. Choo. 2016.

A two-layer dimension reduction and two-tier classication model for anomaly-based intrusion detection in IoT backbone networks. IEEE Transactions onEmerging Topics in Computing (2016).

[31] Reza M. Parizi, Amritraj, and Ali Dehghantanha. 2018. Smart Contract Pro-gramming Languages on Blockchains: An Empirical Evaluation of Usability andSecurity. In Blockchain – ICBC 2018, Shiping Chen, Harry Wang, and Liang-JieZhang (Eds.). Springer International Publishing, Cham, 75–91.

[32] Reza M. Parizi, AA Abdul Ghani, Rusli Abdullah, and Rodziah Atan. 2009. Onthe applicability of random testing for aspect-oriented programs. InternationalJournal of Soware Engineering and Its Application 3, 3 (2009), 1–19.

[33] Reza M. Parizi, Abdul Azim Abdul Ghani, Rusli Abdullah, and Rodziah Atan.2011. Empirical evaluation of the fault detection eectiveness and test eorteciency of the automated AOP testing approaches. Information and SowareTechnology 53 (2011), 1062–1083.

[34] Reza M. Parizi, Abdul Azim Abdul Ghani, and Sai Peck Lee. 2015. Automatedtest generation technique for aspectual features in AspectJ. Information andSoware Technology 57 (2015), 463–493.

[35] Reza M. Parizi, Kai Qian, Hossain Shahriar, Fan Wu, and Lixin Tao. 2018. Bench-mark Requirements for Assessing Soware Security Vulnerability Testing Tools.In 2018 IEEE 42nd Annual Computer Soware and Applications Conference (COMP-SAC). IEEE, 825–826.

[36] Ethereum Project. [n. d.]. Blockchain App Platform. hps://www.ethereum.org/[37] Gerard Salton and Christopher Buckley. 1988. Term-weighting approaches in

automatic text retrieval. Information processing & management 24, 5 (1988),513–523.

[38] SmartDec. [n. d.]. SmartCheck. hps://tool.smartdec.net[39] Chen Wang, Jian Shen, Qi Liu, Yongjun Ren, and Tong Li. 2018. A Novel Security

Scheme Based on Instant Encrypted Transmission for Internet of ings. Securityand Communication Networks 2018 (2018).

[40] Steve Watson and Ali Dehghantanha. 2016. Digital forensics: the missing pieceof the internet of things promise. Computer Fraud & Security 2016, 6 (2016), 5–8.

[41] William J Youden. 1950. Index for rating diagnostic tests. Cancer 3, 1 (1950),32–35.

[42] Yong Yuan and Fei-Yue Wang. 2016. Blockchain: the state of the art and futuretrends. Acta Automatica Sinica 42, 4 (2016), 481–494.

[43] X. Zhang, Y.A. Tan, C. Liang, Y. Li, and J. Li. 2018. A Covert Channel Over VoLTEvia Adjusting Silence Periods. IEEE Access 6, 1 (2018), 9292–9302.

[44] Yuanyu Zhang, Shoji Kasahara, Yulong Shen, Xiaohong Jiang, and JianxiongWan. 2018. Smart Contract-Based Access Control for the Internet of ings.arXiv preprint arXiv:1802.04410 (2018).

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-234.htmlhttp://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-234.htmlhttps://www.usenix.org/conference/usenixsecurity18/presentation/breindenbachhttps://www.usenix.org/conference/usenixsecurity18/presentation/breindenbachhttps://github.com/ethereum/wiki/wiki/Design-Rationalehttps://github.com/ethereum/wiki/wiki/Design-Rationalehttp://solidity.readthedocs.io/en/v0.4.24/https://securify.ch/https://github.com/ConsenSys/mythril.https://www.ethereum.org/https://tool.smartdec.net

Abstract1 Introduction2 Background2.1 Automated Smart Contract Security Testing Tools2.2 Solidity, EVM, and Ethereum Smart Contracts

3 Experimental Setup4 experimental Design and Validity5 Results and Discussion5.1 Analysis of Effectiveness (RQ1)5.2 Analysis of Accuracy (RQ2)

6 Related Work7 ConclusionsReferences