Automatic Generation of Inputs of Death and High-Coverage Tests
description
Transcript of Automatic Generation of Inputs of Death and High-Coverage Tests
![Page 1: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/1.jpg)
Automatic Generation of Inputs of Deathand High-Coverage Tests
Presented by Yoni Leibowitz
EXE & KLEE
![Page 2: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/2.jpg)
Automatically Generating Inputs of Death
David DillVijay GaneshCristian CadarDawson EnglerPeter Pawlowski
KLEEEXEUnassisted & Automatic Generation of High-Coverage Tests for Complex System ProgramsCristian CadarDaniel DunbarDawson Engler
![Page 3: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/3.jpg)
What if you could find all the bugs in
your code, automatically ?
![Page 4: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/4.jpg)
EXEEXecution generated Executions
The Idea
Code can automatically generate its own (potentially highly complex)
test cases
![Page 5: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/5.jpg)
EXEEXecution generated Executions
The Algorithm
Symbolic execution+
Constraint solving
![Page 6: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/6.jpg)
EXEEXecution generated Executions• As program runs
• Executes each feasible path, tracking all constraints
• A path terminates upon exit() crash
failed ‘assert’ error detection
• When a path terminates• Calls STP to solve the path’s constraints for concrete
values
![Page 7: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/7.jpg)
EXEEXecution generated Executions• Identifies all input values causing these errors
• Null or Out-of-bounds memory reference
• Overflow
• Division or modulo by 0
• Identifies all input values causing assert invalidation
![Page 8: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/8.jpg)
int main(void) {unsigned int i, t, a[4] = { 1, 3, 5,
2 };
if (i >= 4)exit(0);
char *p = (char *)a + i * 4;*p = *p − 1t = a[*p];t = t / a[i];if (t == 2)
assert(i == 1);else
assert(i == 3);return 0;
}
Example
![Page 9: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/9.jpg)
int main(void) {unsigned int i, t, a[4] = { 1, 3, 5,
2 };
if (i >= 4)exit(0);
char *p = (char *)a + i * 4;*p = *p − 1t = a[*p];t = t / a[i];if (t == 2)
assert(i == 1);else
assert(i == 3);return 0;
}
Marking Symbolic Data
make_symbolic(&i); Marks the 4 bytes
associated with 32-bit
variable ‘i’ as symbolic
![Page 10: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/10.jpg)
Compiling...example.c
EXE compiler
Inserts checks around every assignment, expression & branch, to determine if its operands are concrete or symbolic
example.outExecutable
unsigned int a[4] = {1,3,5,2} if (i >= 4)
1
int main(void) {unsigned int i, t, a[4] = { 1,
3, 5, 2 };
make_symbolic(&i);
if (i >= 4)exit(0);
char *p = (char *)a + i * 4;*p = *p − 1t = a[*p];t = t / a[i];if (t == 2)
assert(i == 1);
elseassert(i
== 3);return 0;
}
![Page 11: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/11.jpg)
Compiling...example.c
EXE compiler
Inserts checks around every assignment, expression & branch, to determine if its operands are concrete or symbolic
example.outExecutable
If any operand is symbolic, the operation is not performed, but is added as a constraint for the current path
1
int main(void) {unsigned int i, t, a[4] = { 1,
3, 5, 2 };
make_symbolic(&i);
if (i >= 4)exit(0);
char *p = (char *)a + i * 4;*p = *p − 1t = a[*p];t = t / a[i];if (t == 2)
assert(i == 1);
elseassert(i
== 3);return 0;
}
![Page 12: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/12.jpg)
Compiling...example.c
Inserts code to fork program execution when it reaches a symbolic branch point, so that it can explore each possibility
example.outExecutable
EXE compiler
if (i >= 4)(i ≥ 4) (i < 4)
2
int main(void) {unsigned int i, t, a[4] = { 1,
3, 5, 2 };
make_symbolic(&i);
if (i >= 4)exit(0);
char *p = (char *)a + i * 4;*p = *p − 1t = a[*p];t = t / a[i];if (t == 2)
assert(i == 1);
elseassert(i
== 3);return 0;
}
![Page 13: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/13.jpg)
Compiling...example.c
Inserts code to fork program execution when it reaches a symbolic branch point, so that it can explore each possibility
example.outExecutable
EXE compiler
For each branch constraint, queries STP for existence of at least one solution for the current path. If not – stops
executing path
2
int main(void) {unsigned int i, t, a[4] = { 1,
3, 5, 2 };
make_symbolic(&i);
if (i >= 4)exit(0);
char *p = (char *)a + i * 4;*p = *p − 1t = a[*p];t = t / a[i];if (t == 2)
assert(i == 1);
elseassert(i
== 3);return 0;
}
![Page 14: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/14.jpg)
Compiling...example.c
Inserts code for checking if a symbolic expression could have any possible value that could cause errors
example.outExecutable
EXE compiler
3
t = t / a[i]Division by
Zero ?
int main(void) {unsigned int i, t, a[4] = { 1,
3, 5, 2 };
make_symbolic(&i);
if (i >= 4)exit(0);
char *p = (char *)a + i * 4;*p = *p − 1t = a[*p];t = t / a[i];if (t == 2)
assert(i == 1);
elseassert(i
== 3);return 0;
}
![Page 15: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/15.jpg)
Compiling...example.c
Inserts code for checking if a symbolic expression could have any possible value that could cause errors
example.outExecutable
EXE compiler
3
If the check passes – the path has been verified as safe under all possible input values
int main(void) {unsigned int i, t, a[4] = { 1,
3, 5, 2 };
make_symbolic(&i);
if (i >= 4)exit(0);
char *p = (char *)a + i * 4;*p = *p − 1t = a[*p];t = t / a[i];if (t == 2)
assert(i == 1);
elseassert(i
== 3);return 0;
}
![Page 16: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/16.jpg)
int main(void) {unsigned int i, t, a[4] = { 1, 3, 5,
2 };
make_symbolic(&i);
if (i >= 4)exit(0);
char *p = (char *)a + i * 4;*p = *p − 1t = a[*p];t = t / a[i];if (t == 2)
assert(i == 1);else
assert(i == 3);return 0;
}
Running…
e.g. i = 8
EXE generates a test case
4 ≤ i
![Page 17: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/17.jpg)
i ≥ 4
8000test#1
add (i ≥ 4)add (i < 4)
i = *
![Page 18: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/18.jpg)
Running…e.g. i = 2
p → a[2] = 5
a[2] = 5 – 1 = 4
t = a[4]
EXE generates a test case
Out of bounds
0 ≤ i ≤ 4
int main(void) {unsigned int i, t, a[4] = { 1, 3, 5,
2 };
make_symbolic(&i);
if (i >= 4)exit(0);
char *p = (char *)a + i * 4;*p = *p − 1t = a[*p];t = t / a[i];if (t == 2)
assert(i == 1);else
assert(i == 3);return 0;
}
![Page 19: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/19.jpg)
in bounds (a[*p])
i ≥ 4
8000test#1
2000test#2
add (i ≥ 4)add (i < 4)
add (out of bounds(a[*p]))add (in bounds(a[*p]))
i = *
![Page 20: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/20.jpg)
Running…e.g. i = 0
p → a[0] = 1
a[0] = 1 – 1 = 0
t = a[0]
EXE generates a test case
Division by 0
0≤ i ≤ 4 , i ≠ 2
t = t / 0
int main(void) {unsigned int i, t, a[4] = { 1, 3, 5,
2 };
make_symbolic(&i);
if (i >= 4)exit(0);
char *p = (char *)a + i * 4;*p = *p − 1t = a[*p];t = t / a[i];if (t == 2)
assert(i == 1);else
assert(i == 3);return 0;
}
![Page 21: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/21.jpg)
in bounds (a[*p])
i ≥ 4
a[i] == 0
0000test#3
8000test#1
2000test#2
add (i ≥ 4)add (i < 4)
add (out of bounds(a[*p]))add (in bounds(a[*p]))
add (a[i] = 0)
“Division by Zero!”
“Memory overflow!”
i = *
add (a[i] ≠ 0)
![Page 22: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/22.jpg)
Running…i = 1
p → a[1]
a[1] = 2
t = a[2]
EXE determines neither ‘assert’
fails
0≤ i ≤ 4 , i ≠ 2 , i ≠ 0
t = 2
i = 3
p → a[3]
a[3] = 1
t = a[1]
t ≠ 2
2 valid test cases
int main(void) {unsigned int i, t, a[4] = { 1, 3, 5,
2 };
make_symbolic(&i);
if (i >= 4)exit(0);
char *p = (char *)a + i * 4;*p = *p − 1t = a[*p];t = t / a[i];if (t == 2)
assert(i == 1);else
assert(i == 3);return 0;
}
![Page 23: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/23.jpg)
in bounds (a[*p])
i ≥ 4
a[i] == 0
t == 2
i == 3 i == 1
3000test#4
0000test#3
8000test#1
2000test#2
add (i ≥ 4)add (i < 4)
add (out of bounds(a[*p]))add (in bounds(a[*p]))
add (a[i] ≠ 0) add (a[i] = 0)
valid
add (t ≠ 2) add (t = 2)
kill path
invalid
1000test#5
validkill path
invalid
“Division by Zero!”
“Memory overflow!”
i = *
![Page 24: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/24.jpg)
Output
test3.out
test3.forks
test3.err
# concrete byte values:0 # i[0], 0 # i[1], 0 # i[2], 0 # i[3]
ERROR: simple.c:16 Division/modulo by zero!
# take these choices to follow path0 # false branch (line 5)0 # false (implicit: pointer overflow check on line 9)1 # true (implicit: div−by−0 check on line 16)
i = 0
![Page 25: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/25.jpg)
EXEprocess
Optimizations
query string
ServercacheSTP
Solver
hash
1. Caching constraints to avoid calling STP• Goal – avoid calling STP when possible• Results of queries and constraint solutions are cached• Cache is managed by a server process• Naïve implementation – significant overhead
![Page 26: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/26.jpg)
EXEprocess
Optimizations1. Caching constraints to avoid calling STP
• Goal – avoid calling STP when possible• Results of queries and constraint solutions are cached• Cache is managed by a server process• Naïve implementation – significant overhead
query string
Servercache
hit
STPSolver
hash
![Page 27: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/27.jpg)
EXEprocess
Optimizations
query string
Servercache
miss
result
STPSolver
query
hash
1. Caching constraints to avoid calling STP• Goal – avoid calling STP when possible• Results of queries and constraint solutions are cached• Cache is managed by a server process• Naïve implementation – significant overhead
![Page 28: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/28.jpg)
Optimizations2. Constraint Independence
• Breaking constraints into multiple, independent, subsets
• Discard irrelevant constraints
• Small cost for computing independent subsets
• May yield additional cache hits
![Page 29: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/29.jpg)
Optimizations2. Constraint Independence
if (A[i] > A[i+1]) {…
}
if (B[j] + B[j-1] == B[j+1]) {…
} (A[i] > A[i+1]) && (B[j] + B[j-1] = B[j+1])
(A[i] ≤ A[i+1]) && (B[j] + B[j-1] = B[j+1])
(A[i] ≤ A[i+1]) && (B[j] + B[j-1] ≠ B[j+1])
(A[i] > A[i+1]) && (B[j] + B[j-1] ≠ B[j+1])
4 possible paths
2 consecutive independent
branches
![Page 30: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/30.jpg)
Optimizations2. Constraint Independence
1st “if”
A[i] ≤ A[i+1]
2nd “if”
A[i] > A[i+1]
2nd “if”
(A[i] ≤ A[i+1]) &&
(B[j] + B[j-1] ≠ B[j+1])
(A[i] ≤ A[i+1]) &&
(B[j] + B[j-1] = B[j+1])
(A[i] > A[i+1]) &&
(B[j] + B[j-1] ≠ B[j+1])
(A[i] > A[i+1])&&
(B[j] + B[j-1] = B[j+1])
no optimization
1 2
3 4 5 6
![Page 31: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/31.jpg)
Optimizations2. Constraint Independence
1st “if”
A[i] ≤ A[i+1]
2nd “if”
A[i] > A[i+1]
2nd “if”
(B[j] + B[j-1] ≠ B[j+1]) (B[j] + B[j-1] = B[j+1]) (B[j] + B[j-1] ≠ B[j+1]) (B[j] + B[j-1] = B[j+1])
with optimization
1 2
3 4
![Page 32: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/32.jpg)
Optimizations2. Constraint Independence
no optimization with optimization2(2n-1) queries to STP 2n queries to STP
‘n’ consecutive independent
branches
![Page 33: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/33.jpg)
Optimizations3. Search Heuristics – “Best First Search” & DFS
• By default, EXE uses DFS when forking for picking which branch to follow first
• Problem – Loops bounded by symbolic variables
• Solution• Each forked process calls search server, and blocks• Server picks process blocked on line of code which has
run the fewer number of times• Picked process and children are run with DFS
![Page 34: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/34.jpg)
Optimizations• Experimental Performance
• Used to find bugs in• 2 packet filters (FreeBSD & Linux)• DHCP server (udhcpd)• Perl compatible regular expressions library (pcre)• XML parser library (expat)
• Ran EXE without optimizations, with each optimization separately, and with all optimizations
![Page 35: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/35.jpg)
Optimizations• Experimental Performance
• Positive
• With both caching & independence – Faster by 7%-20%
• Cache hit rate jumps sharply with independence
• Cost of independence – near zero
• Best First Search gets (almost) full coverage more than
twice as fast than DFS
• Coverage with BFS compared to random testing: 92%
against 57%
![Page 36: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/36.jpg)
Optimizations• Experimental Performance
• Interesting
• Actual growth of number of paths is much smaller than
potentially exponential growth
• EXE is able to handle relatively complex code
• Negative
• Cache lookup has significant overhead, as conversion of
queries to string is dominant
• STP by far remains highest cost (as expected)
![Page 37: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/37.jpg)
Results• Found
• 2 buffer overflows in FreeBSD packet filter• 4 memory errors in Linux packet filter• 5 memory errors in DHCP server• Many out of bounds write in regular expressions
library
• All errors were confirmed after rerunning the concrete inputs on uninstrumented code
![Page 38: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/38.jpg)
Advantages• Automation – “competition” is manual and
random testing
• Coverage - can test any executable code path and (given enough time) exhaust them all
• Generation of actual attacks and exploits
• No false positives
![Page 39: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/39.jpg)
Limitations• Assumes deterministic code
• Cost – exponential ?• Forks on symbolic branches, most are concrete
(linear)• Loops – can get stuck…• “Happy to let run for weeks, as long as generating
interesting test cases”
• No support for floating point and double reference (STP)
• Source code is required, and needs adjustment
![Page 40: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/40.jpg)
Limitations• Optimizations – far from perfect
implementation
• Benchmarks – hand-picked, small-scaled
• Single threaded – each path is explored independently from others
• Code doesn’t interact with it’s surrounding environment
![Page 41: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/41.jpg)
2 years later…
![Page 42: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/42.jpg)
KLEE• Shares main idea with EXE, but completely
redesigned
• Deals with the external environment
• More optimizations, better implemented
• Targeted at checking system-intensive programs “out of the box”
• Thoroughly evaluated on real, more complicated, environment-intensive programs
![Page 43: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/43.jpg)
KLEE• A hybrid between an operating system for symbolic
processes and an interpreter
• Programs are compiled to virtual instruction sets in LLVM assembly language
• Each symbolic process (“state”) has a symbolic environment register file stack heapprogram counter path condition
• Symbolic environment of a state (unlike a normal process)• Refers to symbolic expressions and not concrete data values
![Page 44: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/44.jpg)
KLEE• Able to execute a large number of states
simultaneously
• At its core – an interpreter loop• Selects a state to run (search heuristics)
• Symbolically executes a single instruction in the context of the state
• Continues until no remaining states (or reaches user-defined timeout)
![Page 45: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/45.jpg)
Architectureint badAbs(int x) {
if (x < 0)
return -x;if (x
== 1234)
return -x;
return x;}
example2.c LLVMcompiler
LLVM bytecodeexample2.bc
KLEESTP
Solver
Test cases
x ≥ 0x ≠ 1234 x = 3
Symbolic environment
![Page 46: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/46.jpg)
Execution• Conditional Branches
• Queries STP to determine if the branch condition is true or false
• The state’s instruction pointer is altered suitably
• Both branches are possible?
• State is cloned, and each clone’s instruction pointer and path condition are updated appropriately
![Page 47: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/47.jpg)
Execution• Targeted Errors
• As in EXE
• Division by 0
• Overflow
• Out-of-bounds memory reference
![Page 48: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/48.jpg)
Modeling the Environment• Code reads/writes values from/to its environment
• Command line arguments• Environment variables• File data• Network packets
• Want to return all possible values for these reads
• How?• Redirecting calls that access the environment to
custom models
![Page 49: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/49.jpg)
Modeling the Environment• Example: Modeling the File System
• File system operations
• Performed on an actual concrete file on disk?• Invoke the corresponding system call in the OS
• Performed on a symbolic file?• Emulate the operation’s effect on a simple symbolic
file system (private for each state)
• Defined simple models for 40 system calls
![Page 50: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/50.jpg)
Modeling the Environment• Example: Modeling the File System
• Symbolic file system
• Crude
• Contains a single directory with N symbolic files
• User can specify N and size of files
• Coexists with real file system
• Applications can use files in both
![Page 51: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/51.jpg)
Optimizations1. Compact State Representation
• Number of concurrent states grows quickly (even >100,000)
• Implements copy-on-write at object level
• Dramatically reduces memory requirements per state
• Heap structure can be shared amongst multiple states
• Can be cloned in constant time (very frequent operation)
![Page 52: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/52.jpg)
Optimizations2. Simplifying queries
• Cost of constraint solving dominates everything else
• Make solving faster
• Reduce memory consumption
• Increase cache hit rate (to follow)
![Page 53: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/53.jpg)
Optimizations2. Simplifying queries
a. Expression Rewriting
• Simple arithmetic simplifications
• Strength reduction
• Linear simplification
x + 0 x
x * 2n x << n
2*x - x x
![Page 54: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/54.jpg)
Optimizations2. Simplifying queries
b. Constraint Set Simplification
• Constraints on same variables tend to become more specific
• Rewrites previous constraints when new, equality constraints, are added to the set
x < 10
![Page 55: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/55.jpg)
Optimizations
x < 10
x = 5
2. Simplifying queriesb. Constraint Set Simplification
• Constraints on same variables tend to become more specific
• Rewrites previous constraints when new, equality constraints, are added to the set
![Page 56: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/56.jpg)
Optimizations
x < 10
x = 5
2. Simplifying queriesb. Constraint Set Simplification
• Constraints on same variables tend to become more specific
• Rewrites previous constraints when new, equality constraints, are added to the set
true
![Page 57: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/57.jpg)
Optimizations
true
2. Simplifying queriesb. Constraint Set Simplification
• Constraints on same variables tend to become more specific
• Rewrites previous constraints when new, equality constraints, are added to the set
![Page 58: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/58.jpg)
Optimizations
true
2. Simplifying queriesb. Constraint Set Simplification
• Constraints on same variables tend to become more specific
• Rewrites previous constraints when new, equality constraints, are added to the set
![Page 59: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/59.jpg)
Optimizations2. Simplifying queries
c. Implied Value Concretization
• The value of a variable effectively becomes concrete
• Concrete value is written back to memoryx + 1 = 10 x = 9
![Page 60: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/60.jpg)
Optimizations2. Simplifying queries
d. Constraint Independence
• As in EXE
![Page 61: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/61.jpg)
Optimizations3. Counter-Example Cache
• More sophisticated than in EXE
• Allows efficient searching for cache entries for both subsets and supersets of a given set
Cache
{ i < 10, i = 10 } unsatisfiable
{ i < 10, j = 8 } ( i = 5, j = 8 )
(1)
(2)
![Page 62: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/62.jpg)
Optimizations3. Counter-Example Cache
• More sophisticated than in EXE
• Allows efficient searching for cache entries for both subsets and supersets of a given set
{ i < 10, i = 10, j = 12 } unsatisfiable
Superset of (1) Cache
{ i < 10, i = 10 } unsatisfiable
{ i < 10, j = 8 } ( i = 5, j = 8 )
(1)
(2)
![Page 63: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/63.jpg)
Optimizations3. Counter-Example Cache
• More sophisticated than in EXE
• Allows efficient searching for cache entries for both subsets and supersets of a given set
{ i < 10} ( i = 5, j = 8 )
Subset of (2) Cache
{ i < 10, i = 10 } unsatisfiable
{ i < 10, j = 8 } ( i = 5, j = 8 )
(1)
(2)
![Page 64: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/64.jpg)
Optimizations3. Counter-Example Cache
• More sophisticated than in EXE
• Allows efficient searching for cache entries for both subsets and supersets of a given set
( i = 5, j = 8 )
Superset of (2) Cache
{ i < 10, i = 10 } unsatisfiable
{ i < 10, j = 8 } ( i = 5, j = 8 )
(1)
(2)
{ i < 10, j = 8, i ≠ 3 }
![Page 65: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/65.jpg)
Optimizations4. Search Heuristics – State Scheduling
• The state to run at each instruction is selected by interleaving 2 strategies
• Each is used in a Round-Robin fashion
• Each state is run for a “time slice”
• Ensures a state which frequently executes expensive instructions will not dominate execution time
![Page 66: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/66.jpg)
Optimizations4. Search Heuristics – State Scheduling
a. Random Path Selection• Traverses tree of paths from root to leaves(internal nodes – forks, leaves – states)
• At branch points – randomly selects path to follow
• States in each subtree have equal probability of being selected
• Favors states higher in the tree – less constraints, freedom to reach uncovered code
• Avoids starvation (loop + symbolic condition = “forks bomb”)
![Page 67: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/67.jpg)
Optimizations4. Search Heuristics – State Scheduling
b. Coverage-Optimized Search
• Tries to select states more likely to cover new code
• Computes min. distance to uncovered instruction, call stack size & whether state recently covered new code
• Randomly selects a state according to these weights
![Page 68: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/68.jpg)
Optimizations• Experimental Performance
• Used to generate tests in• GNU COREUTILS Suite (89 programs)• BUSYBOX (72 programs)• Both have variety of functions, intensive interaction
with the environment• Heavily tested, mature code
• Used to find bugs in• Total of 450 applications
![Page 69: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/69.jpg)
Optimizations• Experimental Performance
• Query simplification + caching• Number of STP queries reduced to 5% (!) of original• Time spent solving queries to STP reduced from 92% of
overall time to 41% of overall time• Speedup
0
50
100
150
200
250
300
0 0.2 0.4 0.6 0.8 1
Base Irrelevant Constraint Elimination Caching Irrelevant Constraint Elimination + Caching
Tim
e (s
)
Executed instructions (normalized)
![Page 70: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/70.jpg)
Optimizations• Experimental Performance
• Paths ending with exit() are explored in average only a few times slower than random tests
(even faster in some programs)
• STP overhead – from 7 to 220 times slower than random tests
![Page 71: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/71.jpg)
Results• Methodology
• Fully automatic runs
• Max of 1 hour per utility, generate test cases
• 2 symbolic files, each holding 8 bytes of symbolic data
• 3 command line arguments up to 10 characters long
• Ran generated test cases on uninstrumented code
• Measured line coverage using ‘gcov’ tool
![Page 72: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/72.jpg)
Results – Line Coverage
0%
20%
40%
60%
80%
100%
1 12 23 34 45 56 67 78 89
GNU COREUTILSOverall: 84%, Average 91%, Median 95%
16 at 100%
Apps sorted by KLEE coverage
Cove
rage
(ELO
C %
)
![Page 73: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/73.jpg)
720%
20%
40%
60%
80%
100%
1 13 25 37 49 61
Results – Line Coverage 31 at 100%
Apps sorted by KLEE coverage
Cove
rage
(ELO
C %
)
BUSYBOXOverall: 91%, Average 94%, Median 98%
![Page 74: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/74.jpg)
Results – Line Coverage
9
-20%
0%
20%
40%
60%
80%
100% Avg/utilityKLEE 91%
Manual 68%
Apps sorted by KLEE coverage – Manual coverage
15 Years of manual testing beaten in
less than 89 hours
GNU COREUTILS
KLEE
cov
erag
e –
Man
ual c
over
age
![Page 75: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/75.jpg)
72
-20%
0%
20%
40%
60%
80%
100%
1 13 25 37 49 61
Results – Line Coverage KL
EE c
over
age
– M
anua
l cov
erag
e Avg/utilityKLEE 94%
Manual 44%
Apps sorted by KLEE coverage – Manual coverage
BUSYBOX
![Page 76: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/76.jpg)
Results – Line Coverage • High coverage with few test cases
• Average of 37 tests per tool in GNU COREUTILS
• “Out of the box” – utilities unaltered
• Entire tool suite (no focus on particular apps)
• However• Checks only low-level errors and violations• Developer tests also validate output to be as
expected
![Page 77: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/77.jpg)
Results – Bugs found• 10 memory error crashes in GNU COREUTILS
• More than found in previous 3 years combined• Generates actual command lines exposing crashes
![Page 78: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/78.jpg)
Results – Bugs found• 21 memory error crashes in BUSYBOX
• Generates actual command lines exposing crashes
![Page 79: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/79.jpg)
Checking Tool Equivalence
x = 21, y = 4
21 mod 4
y & -y = 4 ≠ 1
x & (y-1) = 1
unsigned int modOpt(unsigned int x, unsigned int y) {if ((y & −y) == y) // power of two?return x & (y−1);elsereturn x % y;}
unsigned int mod(unsigned int x, unsigned int y) {return x % y;}
int main() {unsigned int x,y;make symbolic(&x, sizeof(x));make symbolic(&y, sizeof(y));assert(mod(x,y) == modOpt(x,y));return 0;}
![Page 80: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/80.jpg)
Checking Tool Equivalence
Proved equivalenc
efor y ≠ 0
unsigned int modOpt(unsigned int x, unsigned int y) {if ((y & −y) == y) // power of two?return x & (y−1);elsereturn x % y;}
unsigned int mod(unsigned int x, unsigned int y) {return x % y;}
int main() {unsigned int x,y;make symbolic(&x, sizeof(x));make symbolic(&y, sizeof(y));assert(mod(x,y) == modOpt(x,y));return 0;}
![Page 81: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/81.jpg)
Checking Tool Equivalence• Able to test programs against one another
• If 2 functions compute different values along the path, and the assert fires – a test case demonstrating the difference is generated
• assert ( f(x) == g(x) )
• Useful for when• f is a simple implementations, g is an optimized version
• f is a patched version of g (doesn’t change functionality)
• f has an inverse – assert(uncompress(compress(x)) == x)
![Page 82: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/82.jpg)
Checking Tool Equivalence
unsigned int modOpt(unsigned int x, unsigned int y) {if ((y & −y) == y) // power of two?return x & (y−1);elsereturn x % y;}
unsigned int mod(unsigned int x, unsigned int y) {return x % y;}
int main() {unsigned int x,y;make symbolic(&x, sizeof(x));make symbolic(&y, sizeof(y));assert(mod(x,y) == modOpt(x,y));return 0;}
![Page 83: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/83.jpg)
Results – Tool Correctness• Checked 67 COREUTILS tools against (allegedly)
equivalent BUSYBOX implementations
Input BUSYBOX GNU COREUTILStee "" <t1.txt [infinite loop] [terminates]tee - [copies once to stdout] [copies twice]comm t1.txt t2.txt [doesn’t show diff] [shows diff]cksum / "4294967295 0 /" "/: Is a directory"split / "/: Is a directory"tr [duplicates input] "missing operand"[ 0 ‘‘<’’ 1 ] "binary op. expected"tail –2l [rejects] [accepts]unexpand –f [accepts] [rejects]split – [rejects] [accepts]t1.txt: a t2.txt: b (no newlines!)
corr
ectn
ess
erro
rsm
issi
ng
func
tion
alit
y
![Page 84: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/84.jpg)
Advantages• High coverage on a broad set of unaltered
complex programs
• Not limited to low-level programming errors – can find functional incorrectness
• Interaction with environment (somewhat limited)
• Better implemented optimizations
• Impressive results
![Page 85: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/85.jpg)
Limitations• No support for multi-threaded symbolic execution
• Requires compilation to LLVM intermediate representation – external libraries?
• Interprets programs instead of running them natively – slowdown
![Page 86: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/86.jpg)
Limitations• Requires user to specify properties of symbolic
environment
• First bug in path stops execution, following bugs in path are not explored
• STP – cost, no floating point support
![Page 87: Automatic Generation of Inputs of Death and High-Coverage Tests](https://reader031.fdocuments.in/reader031/viewer/2022013004/5681639c550346895dd49772/html5/thumbnails/87.jpg)
Questions ?