Post on 26-Apr-2022
Learning to Fuzz from Symbolic Executionwith Application to Smart Contracts
MislavBalunovic
NodarAmbroladze
PetarTsankov
MartinVechev
JingxuanHe
Random Fuzzing vs. Symbolic Execution
Random Fuzzing Symbolic Execution
Speed
Inputs
Coverage
Fast
Ineffective Effective
Slow
Low Low
1
2
Initial
. . .
. . .. . .
. . .
. . . . . .
, deposit(), P ether
, setOwner( )
, withdraw(P)
steals P ether from
Smart Contract Testing: Challenge
3
Smart Contract Testing: Challenge
Wanted: Transaction sequences that thoroughly explore the state space
4
Random Fuzzing vs. Symbolic Execution
Imitation Learning based Fuzzer
ILF (this work)
Fast
Effective
High
Random Fuzzing Symbolic Execution
Speed
Inputs
Coverage
Fast
Ineffective Effective
Slow
Low Low
5
- Analyzing Ethereum’s Contract Topology. Kiffer et al.. IMC ’18
~120K contracts ~16K clusters
Imitation Learning
6
Demonstration
Symbolic execution
Human expert
Demonstration
Robot
Fuzzer
Learning to Fuzz from Symbolic Execution
Training Transaction sequences
Symbolic execution expert
Smart contracts
≈ 15K contracts
Fuzzing
New contract Fuzzing policy(neural networks)
7
Coverage
VulnerabilityReport
Learning to Fuzz from Symbolic Execution
Training Transaction sequencesSmart contracts
Symbolic execution expert
Fuzzing
New contract
Coverage
VulnerabilityReportFuzzing policy
(neural networks)8
Smart Contract Fuzzing Policy
TestedContract
Transaction
Feedback
! = ($ &̅ , ()*+),, -./0*!)
= + + +$ &̅ ()*+), -./0*!
Example: a Uniformly Random Policy
2*3$/,.(4):
2*3$/,.(536*-!0,)($)):
72*3$/,. 0,9: $ is payableC(0) = 1 otherwise:
2*3$/,.(5JKLJM5):
may modify blockchain state
9
FuzzingPolicy
Neural Network Fuzzing Policy
GRUfuzzat step 3
FCNfunc
GRUint + FCNint
FCNsender
FCNamount
GRUfuzzat step 3 − 1
GRUfuzzat step 3 + 1
ℎQRS
ℎQ
10
Featureof $QRS ℎQ
hiddenstate
hidden state
$Q
&̅Q
()*+),Q
-./0*!Q
Features of 4
Neural Network Fuzzing Policy – Fuzzing State
GRUfuzzat step 3 − 1
GRUfuzzat step 3
Feature of $QRS Current hidden state
11
e.g., Coverage, opcodes, function name. (can be
dynamic)Last
hiddenstate
[3.5, 0.3, 4.0, …]
[1, 6.2, 5, …] [1.2, 8.7, 2.5, …]
Neural Network Fuzzing Policy
GRUfuzzat step 3
FCNfunc
GRUint + FCNint
FCNsender
FCNamount
GRUfuzzat step 3 − 1
GRUfuzzat step 3 + 1
ℎQRS
ℎQ
12
Featureof $QRS ℎQ
hiddenstate
hidden state
$Q
&̅Q
()*+),Q
-./0*!Q
Features of 4
Neural Network Fuzzing Policy – Function
FCNfunc+
Softmax
13
Current hidden state [1.2, 8.7, 2.5, …]
[[1, 6.2, 5, …],[4, 3.7, 6, …],…[2, 9.2, 7, …]]
Feature of 4 SetOwner
Deposit
Withdraw
Withdraw
Neural Network Fuzzing Policy
GRUfuzzat step 3
FCNfunc
GRUint + FCNint
FCNsender
FCNamount
GRUfuzzat step 3 − 1
GRUfuzzat step 3 + 1
ℎQRS
ℎQ
14
Featureof $QRS ℎQ
hiddenstate
hidden state
$Q
&̅Q
()*+),Q
-./0*!Q
Features of 4
One-hot
Neural Network Fuzzing Policy – Arguments
GRUintat step 0
GRUintat step 1
15
. . .
FCNint FCNint
[0, 0, 1, 0, …]
Current hidden state[1.2, 8.7, 2.5, …]
1
0x100x800
0x200. . .
11
0x100x800
0x200. . .
0x200
Distribution over 50 seed integer values from expert
Neural Network Fuzzing Policy
GRUfuzzat step 3
FCNfunc
GRUint + FCNint
FCNsender
FCNamount
GRUfuzzat step 3 − 1
GRUfuzzat step 3 + 1
ℎQRS
ℎQ
16
Featureof $QRS ℎQ
hiddenstate
hidden state
$Q
&̅Q
()*+),Q
-./0*!Q
Features of 4
Learning to Fuzz from Symbolic Execution
Training Transaction sequencesSmart contracts
Symbolic execution expert
Fuzzing
New contract
Coverage
VulnerabilityReportFuzzing policy
(neural networks)17
Symbolic Execution Expert
. . .ExecuteExecute
TQUQV TS TW
18
Symbolic: VerXS&P 2020
!S
!W
RevisitRevisit Revisit
Learning to Fuzz from Symbolic Execution
Training Transaction sequencesSmart contracts
Symbolic execution expert
Fuzzing
New contract
Coverage
VulnerabilityReportFuzzing policy
(neural networks)19
Training Neural Network Fuzzing Policy
20
$Q
&̅Q
()*+),Q
-./0*!Q
!Q by expert
Cross-Entropy loss
Back Prop.
NN Policyat step 3 − 1
InferenceNN Policyat step 3
Features
!Q by expert
HiddenState
Learning to Fuzz from Symbolic Execution
Training Transaction sequencesSmart contracts
Symbolic execution expert
Fuzzing
New contract
Coverage
VulnerabilityReportFuzzing policy
(neural networks)21
ILF System: Coverage & Vulnerability Detection
• Instruction coverage.• Basic block coverage.
22
• Locking: The contract cannot send out but can receive ether.• Leaking: An attacker can steal ether from the contract.• Suicidal: An attacker can deconstruct the contract.• Block Dependency: Ether transfer depends on block state variables.• Unhandled Exception: Root call does not catch exceptions from child calls.• Controlled Delegatecall: Transaction parameters explicitly flow into
arguments of a delegatecall instruction.
Evaluation
23
• 18,496 Contracts (5,013 Large & 13,483 Small)• 5-fold Cross Validation
• Echidna• UNIF• EXPERT • MAIAN
• ContractFuzzer
• Vulnerability Detection• Coverage & Speed• Fuzzing Components • Case Study
Coverage: ILF vs. Fuzzers
0%
20%
40%
60%
80%
100%
0 200 400 600 800 1000Number of Transactions
Instr. Coverage
ILF UNIF Echidna
24
0%
20%
40%
60%
80%
100%
0 200 400 600 800 1000Number of Transactions
Instr. Coverage
ILF UNIF Echidna
Large contractsSmall contracts
Coverage: ILF vs. Symbolic Expert
25
50%
60%
70%
80%
90%
100%
Small Contracts Large Contracts
Instr. Coverage
EXPERT
ILF (#tx same as EXPERT)
ILF (2k txs)
Small: 30 txs, 547sLarge: 49 txs, 2,580s
Small: 13sLarge: 17s148 txs/s
Vulnerability Detection
0%
20%
40%
60%
80%
100%
Leaking Suicidal Locking
% of True Vulnerabilities
ILF UNIF MAIAN
0%
20%
40%
60%
80%
100%
BlockDependency
UnhandledException
ControlledDelegatecall
% of True Vulnerabilities
ILF UNIF ContractFuzzer
26
∪ ∪
13 FPs
6 FPs
ILF: 0 FPs
Importance of Policy Components
27
50%
60%
70%
80%
90%
100%
Coverage Leaking
ILF
ILF-func
ILF-args
ILF-sender
ILF-amount
Mostimportant
All components are necessary
Summary
28
?Q & A