Performance Observation Sameer Shende and Allen D. Malony {sameer,malony} @ cs.uoregon.edu.
AUTOMATED UNIT TEST GENERATION DURING SOFTWARE...
Transcript of AUTOMATED UNIT TEST GENERATION DURING SOFTWARE...
![Page 1: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/1.jpg)
AUTOMATED UNIT TEST GENERATION DURING
SOFTWARE DEVELOPMENT
José Miguel Rojas [email protected]
Joint work with Gordon Fraser and Andrea Arcuri
A Controlled Experiment and Think-aloud Observations
ISSTA 2015
![Page 2: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/2.jpg)
“Testing is a widespread validation approach in industry, but it is still largely ad hoc,
expensive, and unpredictably effective.”“Software Testing Research: Achievements, Challenges, Dreams,”
A. Bertolino. Future of Software Engineering. IEEE . 2007.
![Page 3: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/3.jpg)
“Testing is a widespread validation approach in industry, but it is still largely ad hoc,
expensive, and unpredictably effective.”
“Test case generation has a strong impact on the effectiveness and efficiency of testing.”
“…one of the most active research topics in software testing for several decades, resulting in many different approaches and tools.”
“Software Testing Research: Achievements, Challenges, Dreams,” A. Bertolino. Future of Software Engineering. IEEE . 2007.
”An orchestrated survey of methodologies for automated software test case generation,” S. Anand, E. K. Burke, T. Y. Chen, J. Clark, M.B. Cohen, W. Grieskamp, M.
Harman, M.J. Harrold, P. McMinn. J. Systems and Software. Elsevier. 2013.
![Page 4: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/4.jpg)
![Page 5: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/5.jpg)
BACK IN ISSTA 2013…
“Does automated white-box test generation really help software testers?,” G. Fraser, M. Staats, P. McMinn, A. Arcuri and F. Padberg
![Page 6: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/6.jpg)
BACK IN ISSTA 2013…
“Does automated white-box test generation really help software testers?,” G. Fraser, M. Staats, P. McMinn, A. Arcuri and F. Padberg
ARE UNIT TEST GENERATION TOOLS HELPFUL TO DEVELOPERS WHILE THEY ARE CODING?
![Page 7: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/7.jpg)
![Page 8: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/8.jpg)
CODE COVERAGE
![Page 9: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/9.jpg)
CODE COVERAGE
TIME SPENT ONTESTING
![Page 10: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/10.jpg)
CODE COVERAGE
TIME SPENT ONTESTING
IMPLEMENTATION QUALITY
![Page 11: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/11.jpg)
CONTROLLED EXPERIMENT
![Page 12: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/12.jpg)
CONTROLLED EXPERIMENT
Golden Implementation and Test Suite
![Page 13: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/13.jpg)
CONTROLLED EXPERIMENT
Class TemplateGolden Implementation and Test Suite
![Page 14: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/14.jpg)
CONTROLLED EXPERIMENT
Class Template Implementation and Test Suite
Golden Implementation and Test Suite
![Page 15: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/15.jpg)
CONTROLLED EXPERIMENT
Class Template Implementation and Test Suite
Golden Implementation and Test Suite
![Page 16: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/16.jpg)
CONTROLLED EXPERIMENT
Class Template Implementation and Test Suite
Manual
Golden Implementation and Test Suite
![Page 17: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/17.jpg)
CONTROLLED EXPERIMENT
1 hourClass Template Implementation and Test Suite
Manual
Golden Implementation and Test Suite
![Page 18: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/18.jpg)
CONTROLLED EXPERIMENT
1 hourClass Template Implementation and Test Suite
41
Manual
Golden Implementation and Test Suite
![Page 19: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/19.jpg)
CONTROLLED EXPERIMENT
1 hourClass Template Implementation and Test Suite
41
Manual
Golden Implementation and Test Suite
![Page 20: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/20.jpg)
CONTROLLED EXPERIMENT
1 hourClass Template Implementation and Test Suite
41 2
Manual
Golden Implementation and Test Suite
![Page 21: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/21.jpg)
CONTROLLED EXPERIMENT
1 hourClass Template Implementation and Test Suite
41 42
Manual
Golden Implementation and Test Suite
![Page 22: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/22.jpg)
DOES USING EVOSUITE DURING SOFTWARE DEVELOPMENT LEAD TO TEST SUITES WITH HIGHER CODE COVERAGE?
RQ 1
![Page 23: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/23.jpg)
Bran
ch C
over
age
0%
20%
40%
60%
80%
100%
FilterIterator FixedOrderComparator ListPopulation PredicatedMap
50%
26%
57%
39% 41%
83%
38%
63%
Assisted Manual
CODE COVERAGEparticipants’ test suites run on their own implementations
![Page 24: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/24.jpg)
CODE COVERAGETi
mes
Cov
erag
e w
as c
heck
ed
0
2
4
6
8
10
Category AxisFilterIterator FixedOrderComparator ListPopulation PredicatedMap
6.4
4
9.6
65.3
5.9
1.9
9
Assisted Manual
Times coverage was checked
![Page 25: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/25.jpg)
ListPopulation
Bran
ch C
over
age
(%)
0%
25%
50%
75%
100%
Time (min)0 10 20 30 40 50 60
ManualEvoSuiteAssisted
CODE COVERAGEparticipant’s test suites run on their own implementations
![Page 26: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/26.jpg)
ListPopulation
Bran
ch C
over
age
(%)
0%
25%
50%
75%
100%
Time (min)0 10 20 30 40 50 60
ManualEvoSuiteAssisted
CODE COVERAGEparticipant’s test suites run on their own implementations
![Page 27: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/27.jpg)
ListPopulation
Bran
ch C
over
age
(%)
0%
25%
50%
75%
100%
Time (min)0 10 20 30 40 50 60
ManualEvoSuiteAssisted
CODE COVERAGEparticipant’s test suites run on their own implementations
![Page 28: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/28.jpg)
ListPopulation
Bran
ch C
over
age
(%)
0%
25%
50%
75%
100%
Time (min)0 10 20 30 40 50 60
ManualEvoSuiteAssisted
CODE COVERAGEparticipant’s test suites run on their own implementations
![Page 29: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/29.jpg)
Bran
ch C
over
age
0%
20%
40%
60%
80%
100%
FilterIterator FixedOrderComparator ListPopulation PredicatedMap
50%
28%21%
30%
42%37%35%
41%
Assisted Manual
CODE COVERAGEparticipants’ test suites run on golden implementations
![Page 30: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/30.jpg)
FilterIterator
Bran
ch C
over
age
0%25%50%75%
100%
Time (min)0 10 20 30 40 50 60
Assisted Manual EvoSuite-generated
CODE COVERAGEparticipant’s test suites run on golden implementations, over time
ListPopulation
Bran
ch C
over
age
0%25%50%75%
100%
Time (min)0 10 20 30 40 50 60
FixedOrderComparator
Bran
ch C
over
age
0%25%50%75%
100%
Time (min)0 10 20 30 40 50 60
PredicatedMapBr
anch
Cov
erag
e
0%25%50%75%
100%
Time (min)0 10 20 30 40 50 60
![Page 31: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/31.jpg)
FilterIterator
Bran
ch C
over
age
0%25%50%75%
100%
Time (min)0 10 20 30 40 50 60
Assisted Manual EvoSuite-generated
CODE COVERAGEparticipant’s test suites run on golden implementations, over time
ListPopulation
Bran
ch C
over
age
0%25%50%75%
100%
Time (min)0 10 20 30 40 50 60
FixedOrderComparator
Bran
ch C
over
age
0%25%50%75%
100%
Time (min)0 10 20 30 40 50 60
PredicatedMapBr
anch
Cov
erag
e
0%25%50%75%
100%
Time (min)0 10 20 30 40 50 60
Coverage can be higher when using EvoSuite, depending on how the generated tests are used.
![Page 32: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/32.jpg)
DOES USING EVOSUITE DURING SOFTWARE DEVELOPMENT LEAD TO
DEVELOPERS SPENDING MORE OR LESS TIME ON TESTING?
RQ 2
![Page 33: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/33.jpg)
TESTING EFFORT
0
3
6
8
11
14
FilterIterator FixedOrderComparator ListPopulation PredicatedMap
5.6
11
12.913.7
4.4
8.27.27.8
Assisted Manual
Number of test runs
![Page 34: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/34.jpg)
TESTING EFFORT
0
5
10
16
21
26
FilterIterator FixedOrderComparator ListPopulation PredicatedMap
14.3
25
15.8
20
7.7
12.6
9.3
18.5
Assisted Manual
Minutes spent on testing
![Page 35: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/35.jpg)
TESTING EFFORT
0
5
10
16
21
26
FilterIterator FixedOrderComparator ListPopulation PredicatedMap
14.3
25
15.8
20
7.7
12.6
9.3
18.5
Assisted Manual
Minutes spent on testing
Using EvoSuite reduces the time spent on testing.
![Page 36: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/36.jpg)
DOES USING EVOSUITE DURING SOFTWARE DEVELOPMENT LEAD TO
SOFTWARE WITH FEWER BUGS?
RQ 3
![Page 37: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/37.jpg)
IMPLEMENTATION QUALITYGolden test suites run on participants’ implementations
Num
ber o
f Fail
ures
+Err
ors
0
3
6
10
13
16
FilterIterator FixedOrderComparator ListPopulation PredicatedMap
14.3
5.34.3
6.3
15.6
6.4
4.26.1
Assisted Manual
![Page 38: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/38.jpg)
IMPLEMENTATION QUALITYGolden test suites run on participants’ implementations
Num
ber o
f Fail
ures
+Err
ors
0
3
6
10
13
16
FilterIterator FixedOrderComparator ListPopulation PredicatedMap
14.3
5.34.3
6.3
15.6
6.4
4.26.1
Assisted Manual
Using EvoSuite during development did not lead to to better implementations.
![Page 39: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/39.jpg)
DOES SPENDING MORE TIME WITH EVOSUITE AND ITS TESTS LEAD TO
BETTER IMPLEMENTATIONS?
RQ 4
![Page 40: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/40.jpg)
PRODUCTIVITYTime spent with EvoSuite
Corr
elatio
n w
ith
num
ber o
f fail
ures
plus
err
ors
-0.50
-0.40
-0.30
-0.20
-0.10
0.00
0.10
0.20
0.30
0.40
FilterIterator FixedOrderComparator ListPopulation PredicatedMap
-0.490.02
-0.22-0.29
0.350.32
-0.03
0.35
Number of runsTime spent on tests
![Page 41: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/41.jpg)
PRODUCTIVITYTime spent with EvoSuite
Corr
elatio
n w
ith
num
ber o
f fail
ures
plus
err
ors
-0.50
-0.40
-0.30
-0.20
-0.10
0.00
0.10
0.20
0.30
0.40
FilterIterator FixedOrderComparator ListPopulation PredicatedMap
-0.490.02
-0.22-0.29
0.350.32
-0.03
0.35
Number of runsTime spent on tests
Implementation quality improves the more timedevelopers spend with EvoSuite-generated tests.
![Page 42: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/42.jpg)
Using automated unit test generation does impact developers’ productivity,
![Page 43: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/43.jpg)
Using automated unit test generation does impact developers’ productivity,
but…
![Page 44: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/44.jpg)
…how to make the most outof unit test generation tools?
Using automated unit test generation does impact developers’ productivity,
but…
![Page 45: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/45.jpg)
THINK ALOUD OBSERVATIONS
J. Hughes and S. Parkes, “Trends in the use of verbal protocol analysis in software engineering research,” Behaviour and Information Technology, vol. 22, no. 2, pp. 127–140, 2003.
K. A. Ericsson and H. A. Simon, Protocol Analysis: Verbal Reports as Data (revised edition). MIT Press, 1993.
![Page 46: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/46.jpg)
THINK ALOUD OBSERVATIONS
J. Hughes and S. Parkes, “Trends in the use of verbal protocol analysis in software engineering research,” Behaviour and Information Technology, vol. 22, no. 2, pp. 127–140, 2003.
K. A. Ericsson and H. A. Simon, Protocol Analysis: Verbal Reports as Data (revised edition). MIT Press, 1993.
Subject
![Page 47: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/47.jpg)
THINK ALOUD OBSERVATIONS
J. Hughes and S. Parkes, “Trends in the use of verbal protocol analysis in software engineering research,” Behaviour and Information Technology, vol. 22, no. 2, pp. 127–140, 2003.
K. A. Ericsson and H. A. Simon, Protocol Analysis: Verbal Reports as Data (revised edition). MIT Press, 1993.
Observer
Subject
![Page 48: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/48.jpg)
THINK ALOUD OBSERVATIONS
2 hoursClass Template Implementation and Test Suite
Golden Implementation and Test Suite
5 41
![Page 49: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/49.jpg)
THINK ALOUD OBSERVATIONS
2 hoursClass Template Implementation and Test Suite
Golden Implementation and Test Suite
5 41
![Page 50: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/50.jpg)
THINK ALOUD OBSERVATIONS
2 hoursClass Template Implementation and Test Suite
Golden Implementation and Test Suite
5 41
![Page 51: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/51.jpg)
THINK ALOUD OBSERVATIONS
2 hoursClass Template Implementation and Test Suite
Golden Implementation and Test Suite
5 41
![Page 52: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/52.jpg)
THINK ALOUD OBSERVATIONS
2 hoursClass Template Implementation and Test Suite
Golden Implementation and Test Suite
5 41
![Page 53: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/53.jpg)
THINK ALOUD OBSERVATIONS
2 hoursClass Template Implementation and Test Suite
Golden Implementation and Test Suite
5 41
![Page 54: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/54.jpg)
THINK ALOUD OBSERVATIONS
2 hoursClass Template Implementation and Test Suite
Golden Implementation and Test Suite
5 41
![Page 55: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/55.jpg)
THINK ALOUD OBSERVATIONS
2 hoursClass Template Implementation and Test Suite
Golden Implementation and Test Suite
5 41
![Page 56: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/56.jpg)
THINK ALOUD OBSERVATIONS
2 hoursClass Template Implementation and Test Suite
Golden Implementation and Test Suite
5 41
![Page 57: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/57.jpg)
THINK ALOUD OBSERVATIONS
2 hoursClass Template Implementation and Test Suite
Golden Implementation and Test Suite
5 41
![Page 58: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/58.jpg)
RESULTS
Images: http://jozef89.deviantart.com/
3 6 2 2 5
15 20 23 7 10
94% | 92% 97% | 72% 94% | 95% 100% | 94% 100% | 83%
FixedOrderComparator ListPopulationFilterIterator PredicatedMap PredicatedMap
![Page 59: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/59.jpg)
RESULTS
Images: http://jozef89.deviantart.com/
3 6 2 2 5
15 20 23 7 10
94% 97% 94% | 95% 100% 100%
FixedOrderComparator ListPopulationFilterIterator PredicatedMap PredicatedMap
![Page 60: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/60.jpg)
RESULTS
Images: http://jozef89.deviantart.com/
3 6 2 2 5
15 20 23 7 10
94% 97% | 72% 94% 100% 100%
FixedOrderComparator ListPopulationFilterIterator PredicatedMap PredicatedMap
![Page 61: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/61.jpg)
LESSONS LEARNED
![Page 62: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/62.jpg)
LESSONS LEARNED• There are different approaches to testing and test generation tools
should be adaptable to them
![Page 63: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/63.jpg)
LESSONS LEARNED• There are different approaches to testing and test generation tools
should be adaptable to them
• Developers’ behaviour is often not driven by code coverage
![Page 64: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/64.jpg)
LESSONS LEARNED• There are different approaches to testing and test generation tools
should be adaptable to them
• Developers’ behaviour is often not driven by code coverage
• Readability of generated unit tests is paramount
![Page 65: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/65.jpg)
LESSONS LEARNED• There are different approaches to testing and test generation tools
should be adaptable to them
• Developers’ behaviour is often not driven by code coverage
• Readability of generated unit tests is paramount
• Integration into development environments must be improved
![Page 66: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/66.jpg)
LESSONS LEARNED• There are different approaches to testing and test generation tools
should be adaptable to them
• Developers’ behaviour is often not driven by code coverage
• Readability of generated unit tests is paramount
• Integration into development environments must be improved
• Education/Best practices: Developers do not know how tobest use automated test generation tools!
![Page 67: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/67.jpg)
![Page 68: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/68.jpg)
![Page 69: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/69.jpg)
![Page 70: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/70.jpg)
![Page 71: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/71.jpg)
“Coverage is easy to assess because it isa number, while readability is a very non-
tangible property…
![Page 72: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/72.jpg)
“Coverage is easy to assess because it isa number, while readability is a very non-
tangible property…
… What is readable to me may not bereadable to you. It is readable to me justbecause I spent the last hour and a half
doing this.”
![Page 73: AUTOMATED UNIT TEST GENERATION DURING SOFTWARE …issta2015.cs.uoregon.edu/slides/rojas-issta15-evosuite.pdf · software testing for several decades, resulting in many different approaches](https://reader035.fdocuments.in/reader035/viewer/2022063022/5fea5848f34fe6600a462fd1/html5/thumbnails/73.jpg)
“Coverage is easy to assess because it isa number, while readability is a very non-
tangible property…
… What is readable to me may not bereadable to you. It is readable to me justbecause I spent the last hour and a half
doing this.”—Participant 5