Effective Testing and Debugging Methods and Its Supporting System with Program Deltas
description
Transcript of Effective Testing and Debugging Methods and Its Supporting System with Program Deltas
![Page 1: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/1.jpg)
Effective Testing and Debugging Methods
and Its Supporting System with Program Deltas
Makoto Matsushita, Masayoshi Teraguchi,
and Katsuro InoueOsaka University
![Page 2: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/2.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 2
Software Maintenance
Software maintenance procedures
– Code modification to apply changes of specs
– Regression testing and debugging if any bugs
Supporting tools for software maintenance• Version management system (VMS)• Regression testing tool
![Page 3: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/3.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 3
Debugging with Program Deltas
It is hard to debug features which are not changed during maintenance
Debugging with a code differences (deltas)• Using test results of each version
• The deltas may contain some bugs
![Page 4: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/4.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 4
Ness and Ngo: Regression Containment
• Using VMS and testing tool• Correct output (○) and wrong output (×)
doesn’t appear by turns• Only a single “wrong” case• Only removes codes; no debugging
![Page 5: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/5.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 5
Zeller: Delta Debugging
• Using test tool (no VMS is required)• Assumes “core” (△) output• Can’t consider development sequences• Only two “wrong” cases (△ and ×)• Only analysis; no debugging
![Page 6: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/6.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 6
Our Research Objectives
An improved method with program deltas– Consider more “wrong” cases– Support actual debugging activities
Debugging methods “DMET”, andits supporting environment “DSUS”
![Page 7: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/7.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 7
Today’s topic
• Our debugging method “DMET”• “DSUS” supporting environment• DMET evaluation with DSUS
prototype• Conclusion
![Page 8: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/8.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 8
An approach of DMET
DMET is composed of three phases:– Detection: a program delta (may contains
some bugs) to be detected– Indication: show a delta with latest code– Reflection: apply modification to old
versions
![Page 9: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/9.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 9
DMET: Detection Phase
• Using VMS and testing tools• Assumes wrong output ( - ) which is
different from the one of latest version• Detect the last correct output and the
oldest wrong output
![Page 10: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/10.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 10
DMET: Indication Phase
• Program deltas are arranged as “removed” and “added” program codes
• Detected deltas (maybe in older version) are mapped to the latest version
![Page 11: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/11.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 11
DMET: Reflection Phase
Modification to the latest version are also applied to older versions
– if application causes some errors, remove error version from test sequence
– new sequence are used for further testing
![Page 12: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/12.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 12
Supporting System DSUS
• A GUI-based SDE for debugging– Program editors– Compile, Execute a program
• Cooperates other tools– RCS for version management system– DejaGnu for testing tools
• Auto-execution support– check-in/check-out, regression testing– three DMET procedures
![Page 13: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/13.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 13
System Configuration
• GUI: user interaction
• DSUSmain: integration of other components
• RCS: wrapper for RCS commands
• DejaGnu: wrapper for DejaGnu commands
![Page 14: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/14.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 14
DSUS Screenshot
![Page 15: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/15.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 15
DSUS Evaluation
• Two experimentations– Experimentation A: correcting
programs which contain bugs in unchanged potion of program
– Experimentation B: debug programs corrected by experimentation A
• Preparation– Program specification and its program– Requirements for the new program– Test data
![Page 16: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/16.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 16
Experimentation Procedures
• Experimentation A– Apply new requirement to a (prepared)
program– Correct programs which has a bug in
unchanged portion of the program
• Experimentation B– Debug program collected in exp. A– Subjects are divided into two groups
• debugging without DMET• debugging with DMET
– Compare Elapsed time of debugging
![Page 17: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/17.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 17
Results of experiment A
• Three programs are collected• DMET detects bugs in two
programs; however, detected deltas of program Z doesn’t contain a bug
Program
# of vers
# of test
deltas detected?
X 28 25 1.3-1.4 yes
Y 91 10 1.81-1.88 yes
Z 72 70 1.1-1.46 no
![Page 18: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/18.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 18
Results of experiment B
Debugging time is reduced in X and Y, which bug is detected correctly.
X Y Z
w/o DMET
60.0 103.0 10.6
w/ DMET
39.8 27.2 39.4
0
20
40
60
80
100
120
X Y Zプログラム
平均所要時間(
分)
DMET利用なしDMET利用ありw/ DMETw/o DMET
Program
ela
pse
d t
ime (
min
)
![Page 19: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/19.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 19
Results of experiment B (cont.)
According to Welch’s test (5 percents differences), a significant difference of X+Y between both groups is detected
elapsed time
X+Y+Z X+Y
w/o DMET 60.0 103.0w/ DMET 39.8 27.2
DMET support effective debuggingif the detection is correct
![Page 20: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/20.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 20
Discussion
In program Z, elapsed time of w/ DMET takes longer time than w/o DMET
– A bug in program Z is the case that DMET doesn’t consider; it is caused since some modification should be needed
– DMET generates wrong deltas– Testees are confused with deltas
![Page 21: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/21.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 21
Discussion (cont.)
However, we may estimate that the deltas truly contains a bug since it’s doubtful that:
– the distance of deltas is longer, compared with total numbers of version
– the delta contains the “base version (1.1)”
program
# of vers
# of test
deltas detected?
Z 72 70 1.1-1.46 no
![Page 22: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/22.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 22
Conclusion
• DMET contributes to reduce debugging costs of the engineers, if the detected program delta contains bugs
• DMET doesn’t consider all cases, it may generates wrong delta– If wrong delta, debugging takes
longer time– However, it can be estimated
![Page 23: Effective Testing and Debugging Methods and Its Supporting System with Program Deltas](https://reader036.fdocuments.in/reader036/viewer/2022062409/56814e93550346895dbc3cbb/html5/thumbnails/23.jpg)
ISFST2000 at Kanazawa Nov/02/2000
Page 23
Further Research Topics
• Some estimation methods for checking DMET deltasIf the delta seems wrong, DSUS shows to the engineers that it maybe wrong.
• DMET extension for new programsCurrent DMET assumes that “base version” exists and it is correct. New program have no “base version”.