Cloud9: A Software Testing Service

CLOUD9: A SOFTWARE TESTING SERVICE

Liviu Ciortea, Cristian Zam!r,Stefan Bucur, Vitaly Chipounov, George Candea

SOFTWARE TESTING

• Software testing is laborious and expensive

• Bugs are still very common

• Human testing is prone to errors

• Current automatic test case generation is limited

2

GOALS

• Autonomy: Minimize intervention in test generation

• Usability: Minimize con!guration effort

• Performance: Maximize results relevance

3

OVERVIEW

• System Interface

• Parallel Symbolic Execution

• Cloud9 Design

• Preliminary Results

4

WHAT IS CLOUD9?

• Web service for automated testing

• Easy to use interface

• Relies on thorough testing technique

• Can operate autonomously

• Massive parallelization in the cloud

• Brings scalable performance5

SERVICE INTERFACE

7

OVERVIEW



• Cloud9 Design


9

SYMBOLIC EXECUTIONvoid read(int x) {

}

if (x < 0) {

} else {if (x < 5)bar(x);

else {...

}}

if (x > -3)foo(x);

else {...

} x = -2

Concrete value:

10


}

if (x < 0) {

} else {

}

if (x > -3)foo(x);

else {...

} x = 3

Concrete value:

if (x < 5)bar(x);

else {...

}

11


}

if (x < 0) {

} else {

}

if (x > -3)foo(x);

else {...

}

Symbolic execution tree:

if (x < 5)bar(x);

else {...

}

λ

λ<0 λ>=0

λ<0&& λ>-3

λ>=0&& λ<5

Symbolic value

Constraint

λ<0 && λ>-3 Constraintsolver λ=-2 (a solution)

12

PATH EXPLOSION

Large memory consumptionCPU Intensive

Large memory consumptionCPU Intensive

We massively parallelize in the cloud

CHALLENGES

• Tree structure is not known a priori

15

CHALLENGES


• Naive approach: pre-allocate workers equally on the tree

15

CHALLENGES


• Naive approach: pre-allocate workers equally on the tree

• Slightly better: examine the !rst h levels, then decide work allocation

h levels

15

CHALLENGES

• State transfer

• Avoid work and memory redundancy

• Coordination

16

OVERVIEW



• Cloud9 Design


17

TREE EXPLORATION

Explorationfrontier

Exploration strategy = which node to expand next?

w1

w2

18

LOAD BALANCING

w1 w2

LB

19

LOAD BALANCING

w1 w2

LB

19

negotiate

LOAD BALANCING

w1 w2

LB

19

STATE TRANSFER DECISIONS• State copying vs. state reconstruction

• Reconstruction optimizations

Copying(Network intensive)

Reconstruction(CPU intensive)

c1&&(c2||c3)&&...

w1 w2 c1&&(c2||c3)&&...

w1 w2

10010...

1

00

1

0

20

STRATEGY PORTFOLIO

• No “one size !ts all” exploration strategy

• Different workers with different strategies

• Invest in few workers, then select successful methods

21

OVERVIEW



• Cloud9 Design


22

CLOUD9 PROTOTYPE

• We built Cloud9 on KLEE

• State-of-the art sequential symbolic execution engine

• Tested real programs and found bugs

• Use Amazon EC2 as cloud computing platform

23

TESTING METHODOLOGY

• We compare with KLEE for testing Coreutils

• ls, cat, chmod, cp, mv, etc.

• Cloud9 and KLEE run for 1 hour

• 16 workers for Cloud9

24

CLOUD9 SPEEDUPFix code coverageand measure time

(Linear) 16 50

100

150

200

250

echojoinheadln cp ptx

du chgrp

cutdatecksum

ls pr commcsplit

fmtod factor

chmod

printf

basename

chown

expand

mv catnl dircolors

readlink

paste

chcon

rm mktemp

fold

Speedup(tKLEE/tCloud9)

Tool25

CODE COVERAGE

26

Fix resources (CPU time)and measure code coverage

0

20

40

60

80

100

ptxls chcon

cp headdate

joinpr fmt

du catfold

dircolors

csplit

echood chmod

expand

mv ln nl paste

cutcksum

chgrp

commrm chown

factor

basename

printf

mktemp

readlink

Cove

rage

(%)

Tool

Cloud9 (1h x 16w)KLEE (16h)

CONCLUSIONSAutonomy:

• Symbolic execution

Usability:

• Web service interface

• No local setup overhead

Performance: up to 250x speedup

• Parallel symbolic execution

• Dynamic load balancing

• Adaptive state transfer

• Strategy portfolio

✓

✓

✓

27

Cloud9: A Software Testing Service

Documents

Transcript of Cloud9: A Software Testing Service