Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture...
Transcript of Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture...
![Page 1: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/1.jpg)
S f T i d A l iSoftware Testing and AnalysisProcess, Principles, and Techniques
JUNBEOM YOOJUNBEOM YOO
Dependable Software LaboratoryKONKUK University
http://dslab.konkuk.ac.kr
Ver. 1.7 (2014.06)
※ This lecture note is based on materials from Mauro Pezzè and Michal Young, 2007. ※ Anyone can use this material freely without any notification.
![Page 2: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/2.jpg)
IntroductionIntroduction
T• Text– Software Testing and Analysis : Process, Principles, and Techniques
• This book provides– a coherent view of the state of the art and practice
f f– technical and organizational approaches to push the state of practice toward the state of the art
• Part I Fundamentals of Test and Analysis• Part II Basic TechniquesPart II Basic Techniques• Part III Problems and Methods• Part IV Process
2
![Page 3: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/3.jpg)
Part I. Fundamentals of Test and Analysis
![Page 4: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/4.jpg)
Chapter 1Chapter 1. Software Test and Analysis in a Nutshell
![Page 5: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/5.jpg)
Learning ObjectivesLearning Objectives
Vi h “bi i '' f f li i h f f• View the “big picture'' of software quality in the context of a software development project and organization
• Introduce the range of software verification and validation activities
• Provide a rationale for selecting and combining them within a softwareProvide a rationale for selecting and combining them within a software development process
5
![Page 6: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/6.jpg)
Engineering ProcessesEngineering Processes
All i i h i i i• All engineering processes have two common activities– Construction activities– Checking activities
• In software engineering (purpose: construction of high quality software)– Construction activities– Verification activities
• We are focusing on software verification activities.
6
![Page 7: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/7.jpg)
Software Verification ActivitiesSoftware Verification Activities
S f ifi i i i i k i f• Software verification activities take various forms– for non-critical products for mass markets– for highly-customized products– for critical products
• Software verification is particularly difficult, becausep y ,– Many different quality requirements – Evolving structure– Inherent non-linearityInherent non linearity– Uneven distribution of faults
< An example of uneven distribution of software faults >< An example of uneven distribution of software faults >
If an elevator can safely carry a load of 1,000 kg, it can also safely carry any smaller load.
If a procedure can correctly sort a set of 256 elements, it may fail on a set of 255 or 53 elements, as well as on 257 or 1,023.
7
elements, as well as on 257 or 1,023.
![Page 8: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/8.jpg)
Variety of ApproachesVariety of Approaches
N il b ll f f ifi i• No silver bullet for software verification
• Software verification designers should– Choose and schedule a right blend of techniques
• to reach the required level of quality (concerned with product)to reach the required level of quality (concerned with product)
• within cost constraints (concerned with project)
– Design a specific solution of V&V activities which can suit toDesign a specific solution of V&V activities which can suit to • the problem• the requirements• the development environment• the development environment
8
![Page 9: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/9.jpg)
Basic QuestionsBasic Questions
T d di h k h bl f if i f• To start understanding how to attack the problem of verifying software
1. When do verification and validation start and end?
2 What techniques should be applied?2. What techniques should be applied?
3. How can we assess the readiness of a product?
4. How can we ensure the quality of successive releases?
5. How can the development process be improved?
9
![Page 10: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/10.jpg)
1. When Do Verification and Validation Start and End?
T• Test – A widely used V&V activity– Usually known as a last activity in software development process– But, not the test activity is “test execution”– Test execution is a small part of V&V process
• V&V start as soon as we decide to build a software product, or even before.
• V&V last far beyond the product delivery as long as the software is in use, to cope with evolution and adaptations to new conditions.use, to cope with evolution and adaptations to new conditions.
10
![Page 11: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/11.jpg)
Early Start: From Feasibility StudyEarly Start: From Feasibility Study
F ibili d f j k i• Feasibility study of a new project must take into account – Required qualities – Their impact on the overall cost
• Quality related activities include– Risk analysis – Measures needed to assess and control quality at each stage of development – Assessment of the impact of new features and new quality requirementsp q y q– Contribution of quality control activities to development cost and schedule
11
![Page 12: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/12.jpg)
Long Lasting: Beyond MaintenanceLong Lasting: Beyond Maintenance
M i i i i i l d• Maintenance activities include– Analysis of changes and extensions– Generation of new test suites for the added functionalities– Re-executions of tests to check for non regression of software functionalities
after changes and extensions– Fault tracking and analysis
12
![Page 13: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/13.jpg)
2 What Techniques Should Be Applied?2. What Techniques Should Be Applied?
N i l A&T h i ll• No single A&T technique can serve all purposes.
• The primary reasons for combining techniques are:p y g q– Effectiveness for different classes of faults
( analysis instead of testing for race conditions )– Applicability at different points in a project
( inspection for early requirements validation )– Differences in purpose
( statistical testing to measure reliability )d ff i d– Tradeoffs in cost and assurance
( expensive technique for key properties )
13
![Page 14: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/14.jpg)
nito
ring
Plan
ning
& m
oner
ific
atio
n of
spe
csVe
tion
of
test
sat
ion
Gen
era t
cuti
on a
nd s
w v
alid
ate
st c
ase
exec
14Proc
ess
impr
ovem
ent
![Page 15: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/15.jpg)
3. How Can We Assess the Readiness of a Product?
A&T i i i i li f l d i d l• A&T activities aim at revealing faults during development.– We cannot reveal or remove all faults.– A&T cannot last infinitely.
• We have to know whether products meet the quality requirements or not.– We must specify the required level of dependability. Measurement
– We can determine when that level has been attained. Assessment
15
![Page 16: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/16.jpg)
4. How Can We Ensure the Quality of Successive Releases?
A&T ti iti d t t t th fi t l• A&T activities does not stop at the first release.
• Software products operate for many years, and undergo many changes.d i h– To adapt to environment changes
– To serve new and changing user requirements
• Quality tasks after delivery include– Test and analysis of new and modified code
Re execution of system tests– Re-execution of system tests– Extensive record-keeping
16
![Page 17: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/17.jpg)
5. How Can the Development Process be Improved?
Th d f d i j f j• The same defects are encountered in project after project.• We can improve the quality through identifying and removing
weaknesses – in development process– in A&T process (quality process)
• 4 steps for process improvement1. Define the data to be collected and implementing procedures for collecting p g p g
them2. Analyze collected data to identify important fault classes3. Analyze selected fault classes to identify weaknesses in development and y y p
quality measures 4. Adjust the quality and development process
17
![Page 18: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/18.jpg)
SummarySummary
Th li h h diff l• The quality process has three different goals– Improving a software product– Assessing the quality of the software product– Improving the quality process
• We need to combine several A&T techniques through the software process.
• A&T depends on organization and application domain.A&T depends on organization and application domain.
18
![Page 19: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/19.jpg)
19
![Page 20: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/20.jpg)
Chapter 2Chapter 2. A Framework for Testing and Analysis
![Page 21: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/21.jpg)
Learning ObjectivesLearning Objectives
I d di i d d ff b d l i i i i• Introduce dimensions and tradeoff between test and analysis activities
• Distinguish validation from verification activitiesg
• Understand limitations and possibilities of test and analysis activities
21
![Page 22: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/22.jpg)
Verification and ValidationVerification and Validation
V lid i f• Validation: “Does the software system meets the user's real needs?”– Are we building the right software?
• Verification: “Does the software system meets the requirements specifications?”– Are we building the software right?
SWSpecs
ActualRequirements
Specs
System
Validation Verification
22
![Page 23: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/23.jpg)
V&V Depends on the SpecificationV&V Depends on the Specification
U ifi bl (b lid bl ) ifi i “If• Unverifiable (but validatable) specification: “If a user presses a request button at floor i, an available elevator must arrive at floor i soon.“
• Verifiable specification: “If a user presses a request button at floor i, an available elevator must arrive at floor i within 30 seconds“
1 2 3 4 5 6 7 8
23
![Page 24: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/24.jpg)
V Model of V&V ActivitiesV-Model of V&V Activities
24
![Page 25: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/25.jpg)
Undeciability of Correctness PropertiesUndeciability of Correctness Properties
C i d id bl• Correctness properties are not decidable.– Halting problem can be embedded in almost every property of interest.
Property
Pass/FailDecision Procedure
Program
Pass/Fail
25
![Page 26: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/26.jpg)
Verification Trade off DimensionsVerification Trade-off Dimensions
O i i i i• Optimistic inaccuracy– We may accept some programs
that do not possess the property.It t d t t ll i l ti
Perfect verification ofarbitrary properties bylogical proof or exhaustivetesting (Infinite effort)
M d l h ki
Theorem proving:Unbounded effort to
verify generalproperties.
– It may not detect all violations. – Example: Testing
• Pessimistic inaccuracyd
Model checking:Decidable but possiblyintractable checking of
simple temporalproperties.
Data flow– It is not guaranteed to accept a
program even if the program does possess the property being analyzed because of false alarms
Precise analysis ofsimple syntactic
Typical testingtechniques
analysis
analyzed, because of false alarms.– Example: Automated program analysis
• Simplified properties– It reduces the degree of freedom
yproperties.
– It reduces the degree of freedom by simplifying the property to check.
– Example: Model Checking
Optimisticinaccuracy
Pessimisticinaccuracy
Simplifiedproperties
26
![Page 27: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/27.jpg)
Terms related to Pessimistic and OptimisticTerms related to Pessimistic and Optimistic
Safe:A safe analysis has no optimistic inaccuracy; that is, it accepts only correct programscorrect programs.
Sound:A l i f i h f l i d if hAn analysis of a program P with respect to a formula F is sound, if the analysis returns True only when the program actually does satisfy the formula.
Complete:An analysis of a program P with respect to a formula F is complete, if y p g p pthe analysis always returns true when the program actually does satisfy the formula.
27
![Page 28: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/28.jpg)
SummarySummary
M t i t ti ti d id bl th i l t• Most interesting properties are undecidable, thus in general we cannot count on tools that work without human intevention.
A i liti i t l t t f• Assessing program qualities comprises two complementary sets of activities:
– Validation (Does the software do what it is supposed to do?)– Verification (Does the system behave as specificed?)– Verification (Does the system behave as specificed?)
• There is no single technique for all purposes.V&V designers need to select a suitable combination of techniques– V&V designers need to select a suitable combination of techniques.
28
![Page 29: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/29.jpg)
29
![Page 30: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/30.jpg)
Chapter 3Chapter 3. Basic Principles
![Page 31: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/31.jpg)
Learning ObjectivesLearning Objectives
U d d h b i i i l d l i A&T h i• Understand the basic principles undelying A&T techniques.
• Grasp the motivations and applicability of the main principles.p pp y p p
31
![Page 32: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/32.jpg)
Main A&T PrinciplesMain A&T Principles
P i i l f l i i• Principles for general engineering:– Partition: divide and conquer– Visibility: making information accessible– Feedback: tuning the development process
• Principles specific to software A&T:– Sensitivity: better to fail every time than sometimes– Redundancy: making intentions explicitRedundancy: making intentions explicit– Restriction: making the problem easier
32
![Page 33: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/33.jpg)
1 Sensitivity1. Sensitivity
“I i b f il i h i ”• “It is better to fail every time than sometimes.”• Consistency
• A test selection criterion works better if every selected test provides the same result.• i.e. if the program fails with one of the selected tests, then it fails with all ofi.e. if the program fails with one of the selected tests, then it fails with all of
them. (reliable criteria)
• Run time deadlock analysis works better if it is machine independent.Run time deadlock analysis works better if it is machine independent.• i.e. if the program deadlocks when analyzed on one machine, then it
deadlocks on every machine.
33
![Page 34: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/34.jpg)
2 Redundancy2. Redundancy
“M k i i li i ”• “Make intention explicit.”
• Redundant checks can increase the capabilities of catching specific faults p g pearly or more efficiently.
– Static type checking is redundant with respect to dynamic type checking, but it can reveal many type mismatches earlier and more efficiently.
– Validation of requirement specifications is redundant with respect to validation of the final software, but can reveal errors earlier and more efficiently.T ti d f f ti d d t b t ft d t th t– Testing and proof of properties are redundant, but are often used together to increase confidence.
34
![Page 35: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/35.jpg)
3 Restriction3. Restriction
“M k h bl i ”• “Make the problem easier.”
• Suitable restrictions can reduce hard (unsolvable) problems to simpler p p(solvable) problems.
• A weaker spec may be easier to check:A weaker spec may be easier to check: – It is impossible (in general) to show that pointers are used correctly, but the
simple Java requirement that pointers are initialized before use is simple to enforce.
• A stronger spec may be easier to check: – It is impossible (in general) to show that type errors do not occur at run-time– It is impossible (in general) to show that type errors do not occur at run-time
in a dynamically typed language, but statically typed languages impose stronger restrictions that are easily checkable.
35
![Page 36: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/36.jpg)
4 Partition4. Partition
“Di id d ”• “Divide and conquer.”
• Hard testing and verification problems can be handled by suitably g p y ypartitioning the input space.
– Both structural and functional test selection criteria identify suitable partitions of code or specifications.
– Verification techniques fold the input space according to specific characteristics, grouping homogeneous data together and determining partitions.
36
![Page 37: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/37.jpg)
5 Visibility5. Visibility
“M k i f i ibl ”• “Make information accessible.”
• The ability to measure progress or status against goalsy p g g g– X visibility = ability to judge how we are doing on X– schedule visibility = “Are we ahead or behind schedule?” – quality visibility = “Does quality meet our objectives?”q y y q y j
• Involves setting goals that can be assessed at each stage of development.
• The biggest challenge is early assessmente.g., assessing specifications and design with respect to product quality
37
![Page 38: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/38.jpg)
6 Feedback6. Feedback
“T h d l ”• “Tune the development process.”
• Learning from experience: g p– Each project provides information to improve the next.
• ExamplesExamples– Checklists are built on the basis of errors revealed in the past.– Error taxonomies can help in building better test selection criteria.
Design guidelines can avoid common pitfalls– Design guidelines can avoid common pitfalls.
38
![Page 39: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/39.jpg)
SummarySummary
Th di i li f A&T i h i d b 6 i i i l• The discipline of A&T is characterized by 6 main principles:– Sensitivity: better to fail every time than sometimes– Redundancy: making intentions explicit– Restriction: making the problem easier– Partition: divide and conquer– Visibility: making information accessible– Feedback: tuning the development process
• They can be used to understand advantages and limits of different ey ca be used to u de sta d ad a tages a d ts o d e e tapproaches and compare different techniques.
39
![Page 40: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/40.jpg)
40
![Page 41: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/41.jpg)
Chapter 4. pTest and Analysis Activities within a
Software ProcessSoftware Process
![Page 42: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/42.jpg)
Learning ObjectivesLearning Objectives
U d d h l f li i h d l• Understand the role of quality in the development process
• Build an overall picture of the quality processp q y p
• Identify the main characteristics of a quality process– Visibility– Visibility– Anticipation of activities– Feedback
42
![Page 43: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/43.jpg)
Software Quality and ProcessSoftware Quality and Process
Q li i b dd d f d l• Qualities cannot be added after development– Quality results from a set of inter-dependent activities.– Analysis and testing are crucial but far from sufficient.
• Testing is not a phase, but a lifestyle– Testing and analysis activities occur from early in requirements engineering
through delivery and subsequent evolution. – Quality depends on every part of the software process.
• An essential feature of software processes is that software test and panalysis is thoroughly integrated and not an afterthought
43
![Page 44: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/44.jpg)
Quality ProcessQuality Process
Q li• Quality process– A set of activities and responsibilities
• Focused on ensuring adequate dependability • Concerned with project schedule or with product usability
• Quality process provides a framework for – Selecting and arranging A&T activities – Considering interactions and trade-offs with other important goalsg p g
44
![Page 45: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/45.jpg)
An Example of Other Important GoalsAn Example of Other Important Goals
“Hi h d d bili ” “Ti k ”• “High dependability” vs. “Time to market”
• Mass market products: p– Better to achieve a reasonably high degree of dependability on a tight
schedule than to achieve ultra-high dependability on a much longer schedule
• Critical medical devices:– Better to achieve ultra-high dependability on a much longer schedule than a
reasonably high degree of dependability on a tight schedule
45
![Page 46: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/46.jpg)
Planning and MonitoringPlanning and Monitoring
Q li• Quality process – A&T planning– Balances several activities across the whole development process– Selects and arranges them to be as cost-effective as possible– Improves early visibility
• A&T planning is integral to the quality process.– Quality goals can be achieved only through careful planning.Quality goals can be achieved only through careful planning.
46
![Page 47: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/47.jpg)
Process VisibilityProcess Visibility
A i i ibl h h h i• A process is visible to the extent that one can answer the question:– How does our progress compare to our plan?– Example: Are we on schedule? How far ahead or behind?
• The quality process can achieve adequate visibility, if one can gain strong confidence in the quality of the software system, before it reaches final q y ytesting
– Quality activities are usually placed as early as possible• Design test cases at the earliest opportunityg pp y• Uses analysis techniques on software artifacts produced before actual
code – Motivates the use of “proxy” measures
• Example: the number of faults in design or code is not a true measure of reliability, but we may count faults discovered in design inspections as an early indicator of potential quality problems.
47
![Page 48: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/48.jpg)
A&T PlanA&T Plan
A h i d i i f h li h i l d• A comprehensive description of the quality process that includes:– objectives and scope of A&T activities– documents and other items that must be available – items to be tested– features to be tested and not to be tested– analysis and test activities – staff involved in A&T– constraints– pass and fail criteria– schedule– deliverables– hardware and software requirements– risks and contingencies
48
![Page 49: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/49.jpg)
Quality GoalsQuality Goals
G l b f h fi d i l d bl f bj i• Goal must be further refined into a clear and reasonable set of objectives.
• Product quality: goals of software quality engineeringq y g q y g g• Process quality: means to achieve the goals
• Product qualities• Product qualities– Internal qualities: invisible to clients
• maintainability, flexibility, reparability, changeabilityE t l liti di tl i ibl t li t– External qualities: directly visible to clients
• Usefulness:– usability, performance, security, portability, interoperability
D d bilit• Dependability:– correctness, reliability, safety, robustness
49
![Page 50: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/50.jpg)
Dependability PropertiesDependability Properties
C• Correctness– A program is correct if it is consistent with its specification.– Seldom practical for non-trivial systems
for
• Reliability– Likelihood of correct function for some ”unit” of behavior
for Normal
Operation
– Statistical approximation to correctness (100% reliable = correct)
• SafetySafety– Concerned with preventing certain undesirable behavior, called hazards
• Robustness
for Abnormal Operation• Robustness
– Providing acceptable (degraded) behavior under extreme conditions– Fail softly
&Situation
50
![Page 51: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/51.jpg)
An Example of Dependability PropertyAn Example of Dependability Property
C t R li bilit• Correctness, Reliability: – Let traffic pass according to
correct pattern and central schedulingscheduling
• Robustness, Safety: – Provide degraded function whenProvide degraded function when
it fails– Never signal conflicting greens
• Blinking red / blinking g ed / b gyellow is better than no lights.
• No lights is better than conflicting greens.
51
![Page 52: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/52.jpg)
Relationship among Dependability PropertiesRelationship among Dependability Properties
Robust but not Safe:Robust but not Safe:Catastrophic failures can occurReliable but not Correct:
Failures can occur rarely
Reliable Correct Safe Robust
Safe but not Correct: Annoying failures can occur
Correct but not Safe nor Robust: The specification is inadequate
52
y gThe specification is inadequate
![Page 53: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/53.jpg)
AnalysisAnalysis
D i l l i f d• Do not involve actual execution of program source code– Manual inspection– Automated static analysis
• Inspection technique– Can be applied to essentially any document – Takes a considerable amount of time – Re-inspecting a changed component can be expensive.p g g p p
• Automatic static analysis– Can be applied to some formal representations of requirements modelspp p q– Not to natural language documents– Substituting machine cycles for human effort makes them particularly cost-
effective.
53
![Page 54: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/54.jpg)
TestingTesting
E d l i d l b• Executed late in development, but• Start as early as possible
• Early test generation has several advantages:– Tests are generated independently from code, when the specifications are
fresh in the mind of analysts.y– Generation of test cases may highlight inconsistencies and incompleteness of
the corresponding specifications.
– Tests may be used as compendium of the specifications by the programmers.
54
![Page 55: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/55.jpg)
Improving the ProcessImproving the Process
L l i• Long lasting errors are common.• It is important to structure the process for
– Identifying the most critical persistent faults– Tracking them to frequent errors– Adjusting the development and quality processes to eliminate errors
• Feedback mechanisms are the main ingredient of the quality process for identifying and removing errors.
55
![Page 56: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/56.jpg)
Organizational FactorsOrganizational Factors
Diff f d l d li ?• Different teams for development and quality?– Separate development and quality teams is common in large organizations.
• Different roles for development and quality?– Test designer is a specific role in many organizations– Mobility of people and roles by rotating engineers over development and y p p y g g p
testing tasks among different projects is a possible option.
56
![Page 57: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/57.jpg)
An Example of Allocation of ResponsibilityAn Example of Allocation of Responsibility
All i k d ibili i i l j b• Allocating tasks and responsibilities is a complex job
• Unit testing g– to the development team (requires detailed knowledge of the code)– but the quality team may control the results (structural coverage)
• Integration, system and acceptance testingIntegration, system and acceptance testing – to the quality team– but the development team may produce scaffolding and oracles
• Inspection and walk-through• Inspection and walk-through – to mixed teams
• Regression testingt lit d i t t– to quality and maintenance teams
• Process improvement related activities – to external specialists interacting with all teams
57
![Page 58: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/58.jpg)
SummarySummary
A&T l ti ti th t t b ti bl l d d it d• A&Ts are complex activties that must be sutiably planned and monitored.
• A good quality process obeys some basic principles:i ibili– Visibility
– Early activities– Feedback
• Aims at– Reducing occurrences of faults
A i th d t d d bilit b f d li– Assessing the product dependability before delivery– Improving the process
Ch 4, slide 58
![Page 59: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/59.jpg)
59
![Page 60: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/60.jpg)
Part II. Basic Techniques
![Page 61: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/61.jpg)
Chapter 5Chapter 5. Finite Models
![Page 62: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/62.jpg)
Learning ObjectivesLearning Objectives
U d d l d i li i f fi i b i• Understand goals and implications of finite state abstraction
• Learn how to model program control flow with graphsp g g p
• Learn how to model the software system structure with call graphs
• Learn how to model finite state behavior with finite state machines
62
![Page 63: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/63.jpg)
ModelModel
A d l i• A model is– A representation that is simpler than the artifact it represents,– But preserves some important attributes of the actual artifact
• Our concern is with models of program execution.
63
![Page 64: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/64.jpg)
Directed GraphDirected Graph
Di d h• Directed graph:– N : set of nodes– E : set of edges (relation on the set of nodes)
aa
b a c
b c
N = { a, b, c }E = { (a, b), (a, c), (c, a) }
64
![Page 65: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/65.jpg)
Directed Graph with LabelsDirected Graph with Labels
W l b l d i h h d i i f h i i h• We can label nodes with the names or descriptions of the entities they represent.
– If nodes a and b represent program regions containing assignment t t t i ht d th t d d d ( b) tistatements, we might draw the two nodes and an edge (a, b) connecting
them in this way:
x = y + z;
a = f(x);
65
![Page 66: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/66.jpg)
Finite Abstractions of BehaviorFinite Abstractions of Behavior
T ff f b i• Two effects of abstraction
1. Coarsening of execution model
2. Introduction of nondeterminism
66
![Page 67: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/67.jpg)
Intraprocedural Control Flow GraphIntraprocedural Control Flow Graph
C ll d “C l Fl G h” “CGF”• Called “Control Flow Graph” or “CGF”– A directed graph (N, E)
• Nodes – Regions of source code (basic blocks)– Basic block = maximal program region with a single entry and single exit p g g g y g
point– Statements are often grouped in single regions to get a compact model.– Sometime single statements are broken into more than one node to model g
control flow within the statement.
• Directed edges g– Possibility that program execution proceeds from the end of one region
directly to the beginning of another
67
![Page 68: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/68.jpg)
An Example of CFGAn Example of CFG
public static String collapseNewlines(String argStr)public static String collapseNewlines(String argStr)
public static String collapseNewlines(String argStr){
char last = argStr.charAt(0);StringBuffer argBuf = new StringBuffer();
{char last = argStr.charAt(0);StringBuffer argBuf = new StringBuffer();
for (int cIdx = 0 ;
b2
for (int cIdx = 0 ; cIdx < argStr.length(); cIdx++){
char ch = argStr.charAt(cIdx); {char ch = argStr.charAt(cIdx);
cIdx < argStr.length();
TrueFalse
b4
b3
if (ch != '\n' || last != '\n'){
argBuf.append(ch);last = ch;
if (ch != '\n'
True
True
False
|| last != '\n') b5
last ch;}
}
return argBuf toString();
{argBuf.append(ch);last = ch;
}
}
False
b6
b7return argBuf.toString();}
}cIdx++)
return argBuf.toString();}
b7
b8
68
![Page 69: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/69.jpg)
The Use of CFGThe Use of CFG
CFG b d di l d fi h h i i f i• CFG may be used directly to define thoroughness criteria for testing.– Chapter 9. Test Case Selection and Adequacy– Chapter 12. Structural Testing
• Often, CFG is used to define another model which is used to define a thoroughness criterion
– Example: LCSAJ is derived from the CGF• Essential sub-paths of the CFG from one branch to another
69
![Page 70: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/70.jpg)
LCSAJ (Linear Code Sequence And Jump)LCSAJ (Linear Code Sequence And Jump)
From Sequence of Basic Blocks To
entry b1 b2 b3 jX
entry b1 b2 b3 b4 jT
entry b1 b2 b3 b4 b5 jE
entry b1 b2 b3 b4 b5 b6 b7 jL
jX b8 Return
jL b3 b4 jT
jL b3 b4 b5 jEjL b3 b4 b5 jE
jL b3 b4 b5 b6 b7 jL
70
![Page 71: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/71.jpg)
Call GraphsCall Graphs
“I d l C l Fl G h”• “Interprocedural Control Flow Graph”– A directed graph (N, E)
• Nodes– Represent procedures, methods, functions, etc.
• Edges– Represent ‘call’ relation
• Call graph presents many more design issues and trade-off than CFG.– Overestimation of call relation– Context sensitive/insensitiveContext sensitive/insensitive
71
![Page 72: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/72.jpg)
Overestimation in a Call GraphOverestimation in a Call Graph
Th i ll h i l d ll h h d i bi di h• The static call graph includes calls through dynamic bindings that never occur in execution.
public class C {public static C cFactory(String kind) {
if (kind == "C") return new C(); if (kind == "S") return new S(); return null;
}}void foo() {
System.out.println("You called the parent's method"); }public static void main(String args[]) {
(new A()).check(); }
A.check()}
}class S extends C {
void foo() {System.out.println("You called the child's method");
}}class A {
void check() { C myC = C.cFactory("S"); myC.foo();
}
C.foo() S.foo() CcFactory(string)
72
}}
never occur in execution
![Page 73: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/73.jpg)
Context Sensitive/Insensitive Call GraphsContext Sensitive/Insensitive Call Graphs
public class Context {public class Context {public static void main(String args[]) {
Context c = new Context(); c.foo(3); c bar(17); main mainc.bar(17);
}
void foo(int n) {int[] myArray = new int[ n ];
main main
int[] myArray = new int[ n ]; depends( myArray, 2) ;
}
void bar(int n) {
C.foo C.bar C.foo(3) C.bar(17)
void bar(int n) {int[] myArray = new int[ n ]; depends( myArray, 16) ;
} C.depends C.depends(int(3) a,2) C.depends (int(17) a,16)
void depends( int[] a, int n ) {a[n] = 42;
}}
< Context Insensitive > < Context Sensitive >
73
}
![Page 74: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/74.jpg)
Calling Paths in Context Sensitive Call GraphsCalling Paths in Context Sensitive Call Graphs
AA
B C 1 context AB
D
C
E
1 context A
2 contexts AB AC
F
E
G
2 contexts AB AC
4 contexts ABD ABE ACD ACE
H I
4 contexts ABD ABE ACD ACE
8 contexts …
J
8 contexts …
16 calling contexts … exponentially grow.
74
J g p y g
![Page 75: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/75.jpg)
Finite State MachinesFinite State Machines
CFG b d f• CFGs can be extracted from programs.• FSMs are constructed prior to source code, and serve as specifications.
– A directed graph (N, E)– CFG and FSM are duals.
• Nodes– A finite set of states
• Edges– A set of transitions among statesA set of transitions among states
LF CR EOF other char
e e / emit l / emit d / - w / append
w e / emit l / emit d / emit w / append
l / d / / d
75
l e / - d / - w / append
![Page 76: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/76.jpg)
Correctness Relations for FSM ModelsCorrectness Relations for FSM Models
76
![Page 77: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/77.jpg)
Abstract Function for Modeling FSMsAbstract Function for Modeling FSMs
Modeling with abstraction
77
![Page 78: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/78.jpg)
SummarySummary
M d l b h i l h h if h d ib i d• Models must be much simpler than the artifact they describe in order to be understandable and analyzable.
• Models must be sufficiently detailed to be useful.
• CFG are built from software programCFG are built from software program.
• FSM can be built before software program to documente behavior.
78
![Page 79: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/79.jpg)
79
![Page 80: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/80.jpg)
Ch t 6Chapter 6. Data Dependency and Data Flow Models
![Page 81: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/81.jpg)
Learning ObjectivesLearning Objectives
U d d b i f d fl d l d h l d (d f• Understand basics of data-flow models and the related concepts (def-use pairs, dominators…)
• Understand some analyses that can be performed with the data-flow model of a program
– Data flow analyses to build models– Analyses that use the data flow models
• Understand basic trade-offs in modeling data flowUnderstand basic trade offs in modeling data flow
81
![Page 82: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/82.jpg)
Why Data Flow Models Need?Why Data Flow Models Need?
M d l f Ch 5 h i d l fl l• Models from Chapter 5 emphasized control flow only.– Control flow graph, call graph, finite state machine
• We also need to reason about data dependence.– To reason about transmission of information through program variables– “Where does this value of x come from?”– “What would be affected by changing this? “– ...
• Many program analyses and test design techniques use data flow information and dependences
– Often in combination with control flow
82
![Page 83: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/83.jpg)
Definition Use PairsDefinition-Use Pairs
A d f (d ) i i i i h l i• A def-use (du) pair associates a point in a program where a value is produced with a point where it is used
• Definition: where a variable gets a value– Variable declaration– Variable initialization– Assignment– Values received by a parameter
• Use: extraction of a value from a variable– Expressions
Conditional statements– Conditional statements– Parameter passing– Returns
83
![Page 84: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/84.jpg)
Def Use PairsDef-Use Pairs
...if (...) {
x = ... ; if (...) {
...
D fi iti... }y = ... + x + ... ; x = ...
if (...) { Definition: x gets a value
… x ...
...
y = + x +
Use: the value of x is extractedDef-Use
pathy ... + x + ...
...
84
![Page 85: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/85.jpg)
Def Use PairsDef-Use Pairs
/** Euclid's algorithm */
bli i t d(i t i t ) {public int gcd(int x, int y) {int tmp; // A: def x, y, tmp while (y != 0) { // B: use y
tmp = x % y; // C: def tmp; use x, ytmp x % y; // C: def tmp; use x, yx = y; // D: def x; use yy = tmp; // E: def y; use tmp
}t // Freturn x; // F: use x
}
85
![Page 86: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/86.jpg)
Definition Clear & KillingDefinition-Clear & Killing
A d fi i i l h i h l h CFG f d fi i i• A definition-clear path is a path along the CFG from a definition to a use of the same variable without another definition of the variable between.
• If, instead, another definition is present on the path, then the latter definition kills the former
• A def-use pair is formed if and only if there is a definition-clear path between the definition and the use
86
![Page 87: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/87.jpg)
Definition Clear & KillingDefinition-Clear & Killing
x = ... // A: def xq = ... x = y; // B: kill x def x
... x = y; // B: kill x, def xz = ... y = f(x); // C: use x x = ...
Definition: x gets a valueA
... Definition: x gets a new value, oldPath A C is
x = y a new value, old value is killedB
Path A..C is not definition-clear
Use: the value of x is extracted
...
y = f(x)C
Path B..C is definition-clear
87
![Page 88: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/88.jpg)
(Direct) Data Dependence Graph(Direct) Data Dependence Graph
Di d d d h• Direct data dependence graph– A direct graph (N, E)
• Nodes: as in the control flow graph (CFG)• Edges: def-use (du) pairs, labelled with the variable name
/** Euclid's algorithm */
public int gcd(int x int y) {
xpublic int gcd(int x, int y) {
int tmp; // A: def x, y, tmp while (y != 0) { // B: use y
tmp = x % y; // C: def tmp; use x, yx = y; // D: def x; use yy = tmp; // E: def y; use tmpy p; y; p
}return x; // F: use x
}
88
![Page 89: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/89.jpg)
Control DependenceControl Dependence
D t d d• Data dependence– “Where did these values come from?”
Control dependence• Control dependence – “Which statement controls whether this statement executes?”– A directed graph
• Nodes: as in the CFG• Nodes: as in the CFG• Edges: unlabelled, from entry/branching points to controlled blocks
/** Euclid's algorithm *// Euclid s algorithm /
public int gcd(int x, int y) {int tmp; // A: def x, y, tmp while (y != 0) { // B: use y
tmp = x % y; // C: def tmp; use x, yx = y; // D: def x; use yy = tmp; // E: def y; use tmp
}return x; // F: use x
}
89
![Page 90: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/90.jpg)
DominatorDominator
P d i i d di d h b d k hi• Pre-dominators in a rooted, directed graph can be used to make this intuitive notion of “controlling decision” precise.
• Node M dominates node N, if every path from the root to N passes through M.
– A node will typically have many dominators, but except for the root, there is a unique immediate dominator of node N which is closest to N on any path from the root, and which is in turn dominated by all the other dominators of N. B h d ( t th t) h i i di t d i t th– Because each node (except the root) has a unique immediate dominator, the immediate dominator relation forms a tree.
Post dominators are calculated in the reverse of the control flow graph• Post-dominators are calculated in the reverse of the control flow graph, using a special “exit” node as the root.
90
![Page 91: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/91.jpg)
An Example of DominatorsAn Example of Dominators
A d i ll d• A pre-dominates all nodes.• G post-dominates all nodes.
A
• F and G post-dominate E.• G is the immediate post-
dominator of B
B
dominator of B.
• C does not post-dominate B.C E
• B is the immediate pre-dominator of G.
D F
• F does not pre-dominate G.
G
91
![Page 92: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/92.jpg)
More Precise Definition of Control DependenceMore Precise Definition of Control Dependence
W d i i i d fi i i f l• We can use post-dominators to give a more precise definition of control dependence
– Consider again a node N that is reached on some but not all execution paths.– There must be some node C with the following property:
• C has at least two successors in the control flow graph (i.e., it represents a control flow decision).
• C is not post-dominated by NC is not post dominated by N.• There is a successor of C in the control flow graph that is post-dominated by N.
– When these conditions are true, we say node N is control-dependent on node C.
• Intuitively, C is the last decision that controls whether N executes.
92
![Page 93: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/93.jpg)
An Example of Control DependenceAn Example of Control Dependence
AExecution of F is not inevitable at B
B
C E
not inevitable at B
Execution of F is C
D
E
F
inevitable at E
D F
GF i t l d d t B
GF is control-dependent on B,
the last point at which itsexecution was not inevitable
93
![Page 94: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/94.jpg)
Data Flow AnalysisData Flow Analysis
D ib h l i h d d fl i f i• Describes the algorithms used to compute data flow information. – Basic algorithms used widely in compilers, test and analysis tools, and other
software tools.
• Too difficult Skipped.
94
![Page 95: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/95.jpg)
SummarySummary
D fl d l d CFG• Data flow models detect patterns on CFGs.– Nodes initiating the pattern– Nodes terminating it– Nodes that may interrupt it
• Data dependence informationp– Pros:
• Can be implemented by efficient iterative algorithms• Widely applicable (not just for classic “data flow” properties)
– Limitations:• Unable to distinguish feasible from infeasible paths• Analyses spanning whole programs (e.g., alias analysis) must trade off precision
i t t ti l tagainst computational cost
95
![Page 96: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/96.jpg)
96
![Page 97: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/97.jpg)
Ch t 7Chapter 7. Symbolic Execution and Proof of Properties
![Page 98: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/98.jpg)
Learning ObjectivesLearning Objectives
U d d h l d i li i f b li ll i• Understand the goal and implication of symbolically executing programs
• Learn how to use assertions to summarize infinite executions• Learn how to reason about program correctness• Learn how to use symbolic execution to reason about program
propertiesproperties
• Understand limits and problems of symbolic execution
98
![Page 99: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/99.jpg)
Symbolic ExecutionSymbolic Execution
B ild di h h i• Builds predicates that characterize – Conditions for executing paths – Effects of the execution on program state
• Bridges program behavior to logic
• Finds important applications in – Program analysis
Test data generation– Test data generation– Formal verification (proofs) of program correctness
• Rigorous proofs of properties of critical subsystems– Example: safety kernel of a medical device
• Formal verification of critical properties particularly resistant to dynamic testing – Example: security properties
• Formal verification of algorithm descriptions and logical designs– less complex than implementationsp p
99
![Page 100: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/100.jpg)
Symbolic State and InterpretationSymbolic State and Interpretation
T i i i h b li l d i i h b i f• Tracing execution with symbolic values and expressions is the basis of symbolic execution.
– Values are expressions over symbols.– Executing statements computes new expressions with the symbols.
Execution with concrete values Execution with symbolic values
(before)low 12high 15
(before)low Lhigh H
mid -
mid = (high + low) / 2
gmid -
mid = (high + low) / 2
(after)low 12high 15
(after)Low Lhigh H
100
mid 13 mid (L+H) / 2
![Page 101: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/101.jpg)
Tracing Execution with Symbolic ExecutionsTracing Execution with Symbolic Executions
char *binarySearch( char *key char *dictKeys[ ] ∧∀k, 0 ≤ k < size : dictKeys[k] = key → L ≤ k ≤ Hchar *binarySearch( char *key, char *dictKeys[ ], char *dictValues[ ], int dictSize) {
int low = 0; int high = dictSize - 1;
Execution with symbolic values
y [ ] y∧ H ≥ M ≥ L
int high dictSize 1; int mid; int comparison;
while (high >= low) {
(before)low = 0
∧ high = (H-1)/2 -1∧ mid = (H 1)/2
supposed
while (high > low) {mid = (high + low) / 2; comparison = strcmp( dictKeys[mid], key );if (comparison < 0) {low = mid + 1;
∧ mid = (H-1)/2
while (high >= low) {
(after)low mid + 1;} else if ( comparison > 0 ) {high = mid - 1;
} else {return dictValues[mid];
(after)low = 0
∧ high = (H-1)/2 -1∧ mid = (H-1)/2∧ (H 1)/2 1 >= 0
when truereturn dictValues[mid];
}}return 0;
∧ (H-1)/2 - 1 >= 0... ∧ not((H-1)/2 - 1 >= 0)
when false
101
}
![Page 102: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/102.jpg)
Summary InformationSummary Information
S b li i f h b l l• Symbolic representation of paths may become extremely complex.
• We can simplify the representation by replacing a complex condition Pp y p y p g pwith a weaker condition W such that
P => W– W describes the path with less precisionW describes the path with less precision– W is a summary of P
102
![Page 103: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/103.jpg)
An Example of Summary InformationAn Example of Summary Information
If i b h f h bi h l i h• If we are reasoning about the correctness of the binary search algorithm, – In “ mid = (high+low)/2 “
Weaker condition:Complete condition:
low = L∧ high = H∧ mid = M∧ L < M < H
low = L∧ high = H∧ mid = M∧ M (L+H) / 2 ∧ L <= M <= H∧ M = (L+H) / 2
• The weaker condition contains less information, but still enough to reason about correctness.
103
![Page 104: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/104.jpg)
Weaker PreconditionWeaker Precondition
Th k di t L id H i h b d h t t b• The weaker predicate L <= mid <= H is chosen based on what must be true for the program to execute correctly.
– It cannot be derived automatically from source code.It depends on our understanding of the code and our rationale for believing– It depends on our understanding of the code and our rationale for believing it to be correct.
• A predicate stating what should be true at a given point can be p g g pexpressed in the form of an assertion
• Weakening the predicate has a cost for testingg p g– Satisfying the predicate is no longer sufficient to find data that forces
program execution along that path. • Test data satisfying a weaker predicate W is necessary to execute the
path but it may not be sufficientpath, but it may not be sufficient.• Showing that W cannot be satisfied shows path infeasibility.
104
![Page 105: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/105.jpg)
Loops and AssertionsLoops and Assertions
Th b f i h h h i h l i• The number of execution paths through a program with loops is potentially infinite.
• To reason about program behavior in a loop, we can place within the loop an invariant.
– Assertion that states a predicate that is expected to be true each time execution reaches that point
• Each time program execution reaches the invariant assertion, we can weaken the description of program state.
– If predicate P represents the program state and the assertion is W– We must first ascertain P => W – And then we can substitute W for P
105
![Page 106: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/106.jpg)
Precondition and PostconditionPrecondition and Postcondition
S d h• Supposed that– Every loop contains an assertion– There is an assertion at the beginning of the program
Th i fi l ti t th d– There is a final assertion at the end• Then
– Every possible execution path would be a sequence of segments from one assertion to the nextassertion to the next.
• Precondition: the assertion at the beginning of a segmentPrecondition: the assertion at the beginning of a segment• Postcondition: the assertion at the end of the segment
106
![Page 107: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/107.jpg)
Verification of Program CorrectnessVerification of Program Correctness
F h if if h• For each program segment, if we can verify that – Starting from the precondition,– Executing the program segment,– And postcondition holds at the end of the segment
• Then, we verify the correctness of an infinite number of program paths.
107
![Page 108: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/108.jpg)
An Example of Verification with AssertionsAn Example of Verification with Assertions
char *binarySearch( char *key char *dictKeys[ ]char *binarySearch( char *key, char *dictKeys[ ], char *dictValues[ ], int dictSize) {
int low = 0; int high = dictSize - 1;
Precondition: “should be sorted”∀i j 0 ≤ i < j < size : dictKeys[i] ≤ dictKeys[j]int high dictSize 1;
int mid; int comparison;
while (high >= low) {
∀i,j, 0 ≤ i < j < size : dictKeys[i] ≤ dictKeys[j]
Invariant: “should be in range”∀i, 0 ≤ i < size : dictKeys[i] = key → low ≤ i ≤ highwhile (high > low) {
mid = (high + low) / 2; comparison = strcmp( dictKeys[mid], key );if (comparison < 0) {low = mid + 1;
, y [ ] y g
low mid + 1;} else if ( comparison > 0 ) {high = mid - 1;
} else {return dictValues[mid];return dictValues[mid];
}}return 0;
108
}
![Page 109: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/109.jpg)
When Executing the LoopWhen Executing the Loop
low = L∧ high = H
∀i j 0 ≤ i < j < i di tK [i] ≤ di tK [j]
Initial values:
Instantiated invariant:
Precondition∀i,j, 0 ≤ i < j < size : dictKeys[i] ≤ dictKeys[j]
∀i, j, 0 ≤ i < j < size : dictKeys[i] ≤ dictKeys[j]∧∀k, 0 ≤ k < size : dictKeys[k] = key → L ≤ k ≤ H
Instantiated invariant:
After executing: mid = (high + low) / 2
Invariant∀i 0 ≤ i < size :low = L
∧ high = H ∧ mid = M∧∀i j 0 ≤ i j i di tK [i] ≤ di tK [j]
∀i, 0 ≤ i < size : dictKeys[i] = key → low ≤ i ≤ high
∧∀i, j, 0 ≤ i < j < size : dictKeys[i] ≤ dictKeys[j]∧∀k, 0 ≤ k < size : dictKeys[k] = key → L ≤ k ≤ H∧ H ≥ M ≥ L
109
![Page 110: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/110.jpg)
After executing the LoopAfter executing the Loop
After executing the loop :
low = M+1∧ high = H ∧ mid = M∧∀i, j, 0 ≤ i < j < size : dictKeys[i] ≤ dictKeys[j]∧∀k, 0 ≤ k < size : dictKeys[k] = key → L ≤ k ≤ H∧ H ≥ M ≥ L∧ dictkeys[M] < key
The new instance of the invariant:
∀i, j, 0 ≤ i < j < size : dictKeys[i] ≤ dictKeys[j]∧∀k, 0 ≤ k < size : dictKeys[k] = key → M+1 ≤ k <= H
If the invariant is satisfied, the loop is correct woth respect to the preconditions and the invariant.
110
p p p
![Page 111: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/111.jpg)
At the End of the LoopAt the End of the Loop
E h i i i i fi d b h di i i f lEven the invariant is satisfied, but the postcondition is false:
low = L∧ hi h H∧ high = H ∧∀i, j, 0 ≤ i < j < size : dictKeys[i] ≤ dictKeys[j]∧∀k, 0 ≤ k < size : dictKeys[k] = key → L ≤ k ≤ H∧ L > H
If the condition satisfies the post-condition, the program is correct with h d di i
∧ L > H
respect to the pre- and post-condition.
111
![Page 112: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/112.jpg)
Compositional ReasoningCompositional Reasoning
F ll h hi hi l f• Follow the hierarchical structure of a program– at a small scale (within a single procedure) – at larger scales (across multiple procedures)
• Hoare triple: [pre] block [post]
• If the program is in a state satisfying the precondition pre at entry to the block, then after execution of the block, it will be in a state satisfying the postcondition postpostcondition post
112
![Page 113: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/113.jpg)
Reasoning about Hoare Triples: InferenceReasoning about Hoare Triples: Inference
While loops:I : invariantC : loop conditionS : body of the loop
[ I ∧C ] S [ I ]premise
[ I ] while(C) { S } [I ∧ ¬C]conclusion
Inference rule says:if we can verify the premise (top)if we can verify the premise (top), then we can infer the conclusion (bottom)
113
![Page 114: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/114.jpg)
Other Inference RuleOther Inference Rule
if statement:
[P ∧ C] thenpart [Q] [P ∧ ¬C] elsepart [Q][P] if (C) {thenpart} else {elsepart} [Q][P] if (C) {thenpart} else {elsepart} [Q]
114
![Page 115: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/115.jpg)
Reasoning StyleReasoning Style
S i th ff t f bl k f d ( h l d ) b• Summarize the effect of a block of program code (a whole procedure) by a “contract == precondition + postcondition “
• Then use the contract wherever the procedure is called
• Summarizing binarySearch:
(∀i,j, 0≤i<j<size : keys[i]≤keys[ j]) <-- precondition
bi S h(k k l i )s = binarySearch(k, keys, vals, size)
(s=v and ∃i , 0≤i , size : keys[i]=k ∧ vals[i]=v) <-- postcondition( ∃i 0 i i k [i] k)∨ (s=v ∧ ¬∃i , 0≤i , size : keys[i]=k)
115
![Page 116: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/116.jpg)
Reasoning about Data Structures and ClassesReasoning about Data Structures and Classes
D t t t d l• Data structure module = Collection of procedures (methods) whose specifications are strongly interrelated
• Contracts: specified by relating procedures to an abstract model of their (encapsulated) inner state
• Example: – Dictionary can be abstracted as {<key, value>}– Implemented independently as a list tree hash table etc– Implemented independently as a list, tree, hash table, etc.
116
![Page 117: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/117.jpg)
Structural Invariant & Abstract FunctionStructural Invariant & Abstract Function
S l i i h l h i i h b• Structural invariants are the structural characteristics that must be maintained. (directly analogous to loop invariants)
– Example: Each method in a search tree class should maintain the ordering of k i th tkeys in the tree.
• Abstract function maps concrete objects to abstract model states.– Example: Dictionary
• [<k,v> ∈ Φ(dict) ]• o = dict.get(k)• [ o = v ]
117
![Page 118: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/118.jpg)
SummarySummary
S b li ti i b id f ti l i f• Symbolic execution is a bridge from an operational view of program execution to logical and mathematical statements.
• Basic symbolic execution technique is the execution using symbols.
• Symbolic execution for loops, procedure calls, and data structures: proceed hierarchically
compose facts about small parts into facts about larger parts– compose facts about small parts into facts about larger parts
• Fundamental technique forGenerating test data– Generating test data
– Verifying systems – Performing or checking program transformations
• Tools are essential to scale up.
118
![Page 119: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/119.jpg)
119
![Page 120: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/120.jpg)
Chapter 8Chapter 8. Finite State Verification
![Page 121: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/121.jpg)
Learning ObjectivesLearning Objectives
U d d h d i f fi i ifi i• Understand the purpose and appropriate uses of finite-state verification– Understand how FSV mitigates weaknesses of testing– Understand how testing complements FSV
• Understand modeling for FSV as a balance between cost and precision
• Distinguish explicit state enumeration from analysis of implicit models– Understand why implicit models are sometimes (but not always) more
effectiveeffective
121
![Page 122: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/122.jpg)
OverviewOverview
M i i f i d id bl• Most important properties of program execution are not decidable.• Finite state verification can automatically prove some significant
properties of a finite model of the infinite execution space.
• Need to balance trade-offs among – Generality of properties to be checkedGenerality of properties to be checked– Class of programs or models that can be checked– Computational effort in checking– Human effort in producing models and specifying propertiesHuman effort in producing models and specifying properties
122
![Page 123: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/123.jpg)
Resources and ResultsResources and Results
Properties to b dbe proved
complex and formal reasoningsymbolic execution
and formal reasoning
finite statefinite stateverification
Applies techniques from symbolicApplies techniques from symbolic execution and formal verification to models that abstract the potentially infinite state space of program behavior into finite representations
controland data flow
behavior into finite representations
Computational costhighlow
simple
and data flow models
123
highlow
![Page 124: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/124.jpg)
Cost of FSVCost of FSV
H ff d kill i d• Human effort and skill are required. – to prepare a finite state model – to prepare a suitable specification for automated analysis
• Iterative process of FSV1. Prepare a model and specify properties2. Attempt verification 3. Receive reports of impossible or unimportant faults4. Refine the specification or the model
124
![Page 125: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/125.jpg)
Finite State Verification FrameworkFinite State Verification Framework
125
![Page 126: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/126.jpg)
Applications for Finite State VerificationsApplications for Finite State Verifications
C ( l i h d d di ib d )• Concurrent (multi-threaded, distributed, ...) system– Difficult to test thoroughly (apparent non-determinism based on scheduler)– Sensitive to differences between development environment and field
i tenvironment– First and most well-developed application of FSV
• Data models– Difficult to identify “corner cases” and interactions among constraints, or to
thoroughly test them
• Security– Some threats depend on unusual (and untested) use
126
![Page 127: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/127.jpg)
An Example: Modeling Concurrent SystemAn Example: Modeling Concurrent System
D i i d fi i d l i h d• Deriving a good finite state model is hard.
• Example: FSM model of a program with multiple threads of controlp p g p– Simplifying assumptions
• We can determine in advance the number of threads.• We can obtain a finite state machine model of each thread.• We can identify the points at which processes can interact.
– State of the whole system model • Tuple of states of individual process models
i i– Transition • Transition of one or more of the individual processes, acting individually or in
concert
127
![Page 128: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/128.jpg)
State Space Explosion An ExampleState Space Explosion – An Example
O li h i• On-line purchasing system
• Specificationp– In-memory data structure initialized by reading configuration tables at system
start-up– Initialization of the data structure must appear atomic. – The system must be reinitialized on occasion.– The structure is kept in memory.
• Implementation (with bugs)
– No monitor (e.g. Java synchronized), because it’s too expensive.– But use double-checked locking idiom* for a fast system– But, use double checked locking idiom for a fast system
*Bad decision, broken idiom ... but extremely hard to find the bug through testing.
128
![Page 129: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/129.jpg)
On line Purchasing System ImplementationOn-line Purchasing System - Implementation
l T bl 1 {public void reinit() { needsInit = true; }
class Table1 {
private static Table1 ref = null; private boolean needsInit = true; private ElementClass [ ] theValues;
private synchronized void initialize() {. . . needsInit = false;
}private ElementClass [ ] theValues;private Table1() { }
public static Table1 getTable1() {if (ref == null)
}
public int lookup(int i) {if (needsInit) {
h i d(thi ) {if (ref == null){ synchedInitialize(); }
return ref;}
synchronized(this) {if (needsInit) {
this.initialize();}
}private static synchronized void synchedInitialize() {
if (ref == null) { ref = new Table1(); ref initialize();
}}return theValues[i].getX() + theValues[i].getY();
}ref.initialize();
} }
. . .}
129
![Page 130: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/130.jpg)
Analysis on On line Purchasing SystemAnalysis on On-line Purchasing System
S f d l f i di id l• Start from models of individual threads
– Systematically trace all the ibl i t l i f th d
(a)lookup()
needsInit==true
(x)reinit()
needsInit=truepossible interleaving of threads
– Like hand-executing all possible sequences of execution, but automated
(b)
obtain lock
(c)
(y)
E
automated
• Analysis begins by constructing an FSM model of each individual
(c)
needsInit==false(d)
modifyingneedsInit==false
needsInit==true
an FSM model of each individual thread.
(e)
needsInit=false
release lock
(f)reading
release lock
E
130
![Page 131: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/131.jpg)
Analysis (C ti d)Analysis (Continued)
J h di l• Java threading rules:– When one thread has obtained a monitor lock, the other thread cannot
obtain the same lock
• Locking – Prevents threads from concurrently calling initialize– Does not prevent possible race condition between threads executing the
lookup method
• Tracing possible executions by hand is completely impractical.• Use a finite state verification using the SPIN model checker
131
![Page 132: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/132.jpg)
Modeling the System in PROMELAModeling the System in PROMELA
...proctype Lookup(int id ) {
if :: (needsInit) -> atomic { ! locked -> locked = true; };
needsinit==true
{ ; };if :: (needsInit) ->
assert (! modifying); modifying = true; /* Initialization happens here */
acquire lock
/* Initialization happens here */modifying = false ; needsInit = false;
:: (! needsInit) ->
...
( )skip;
fi;locked = false ;
fi;fi;assert (! modifying);}
132
![Page 133: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/133.jpg)
Run SPIN and OutputRun SPIN and Output
S i• Spin– Depth-first search of possible executions of the model– Explores 51 states and 92 state transitions in 0.16 seconds
Finds a sequence of 17 transitions from the initial state of the model to a– Finds a sequence of 17 transitions from the initial state of the model to a state in which one of the assertions in the model evaluates to false
Depth=10 States=51 Transitions=92 Memory=2.302 pan: assertion violated !(modifying) (at depth 17)pan: wrote pan_in.trail(Spin Version 4.2.5 -- 2 April 2005)…0 16 real 0 00 user 0 03 sys0.16 real 0.00 user 0.03 sys
133
![Page 134: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/134.jpg)
Counterexample: Interpret the OutputCounterexample: Interpret the Outputproc 3 (lookup) proc 1 (reinit) proc 2 (lookup)
public init lookup(int i)if (needsInit) {
synchronized(this) {if (needsInit) {
(a)(b)(c)
if (needsInit) {this.initialize();
}}
}
(d)(e)
public void reinit(){ needsInit = true; }
(x)(y) { needsInit true; }(y)
public init lookup(int i)(a)
Read/writeRace condition
…return
theValues[i].getX()+ theValues[i].getY();
}
(f)
if (needsInit) {synchronized(this) {
if (needsInit) {this.initialize();
...
(b)(c)(d)
134
States (f) and (d)} ...
![Page 135: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/135.jpg)
The State Space Explosion ProblemThe State Space Explosion Problem
Di i hil h l ki f d dl k i h SPIN• Dining philosophers - looking for deadlock with SPIN
5 phils+forks 145 statespdeadlock found
10 phils+forks 18,313 stateserror trace too long to be usefulerror trace too long to be useful
15 phils+forks 148,897 stateserror trace too long to be useful
• Team Practice and Homework!!!– From 2015
135
![Page 136: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/136.jpg)
The Model Correspondence ProblemThe Model Correspondence Problem
V if i d b d l d• Verifying correspondence between model and program– Extract the model from the source code with verified procedures
• Blindly mirroring all details state space explosion • Omitting crucial detail “false alarm” reports
– Produce the source code automatically from the model• Most applicable within well-understood domains
– Conformance testing• Combination of FSV and testing is a good tradeoff
136
![Page 137: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/137.jpg)
Granularity of ModelingGranularity of Modeling
(a) (a) (w)(w)
(b)
t=i;
(x)
u=i;
i = i+1
(c)
t=t+1;
(y)
u=u+1;i = i+1
(d)
(c)
(d)
i=t;
(y)
(z)
i=u;
(z)(d)
E E
(d)
E
(z)(z)
E
137
![Page 138: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/138.jpg)
Analysis of Different ModelsAnalysis of Different Models
W fi d h l i h RacerP RacerQ
• We can find the race only with fine-grain models. t = i;
(a)
t = t+1;(b)
u = i;(w)
u = i;
u = u+1;(x)
(c)
i = u;(y)
i = t;(c)
(d) (z)
138
![Page 139: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/139.jpg)
Looking for Appropriate GranularityLooking for Appropriate Granularity
C il h d f i i• Compilers may rearrange the order of instruction.– A simple store of a value into a memory cell may be compiled into a store
into a local register, with the actual store to memory appearing later.T l d t t diff t l ti b d d f– Two loads or stores to different memory locations may be reordered for reasons of efficiency.
– Parallel computers may place values initially in the cache memory of a local processor and only later write into a memory areaprocessor, and only later write into a memory area.
• Even representing each memory access as an individual action is not always sufficient.
• Example: Double-check idiom only for lazy initialization– Spin assumes that memory accesses occur in the order given in the PROMELA
d d h i h d h Jprogram, and we code them in the same order as the Java program.– But, Java does not guarantee that they will be executed in that order.– And, SPIN would find a flaw.
139
![Page 140: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/140.jpg)
Intentional ModelsIntentional Models
E i ll h bl i li i i f f fi i• Enumerating all reachable states is a limiting factor of finite state verification.
• We can reduce the space by using intentional (symbolic) representations.– describe sets of reachable states without enumerating each one individually
• Example (set of Integers)– Enumeration {2, 4, 6, 8, 10, 12, 14, 16, 18}– Intentional representation: {x∈N | x mod 2 =0 and 0<x<20}Intentional representation: {x∈N | x mod 2 0 and 0<x<20} ← “characteristic function”
• Intentional models do not necessarily grow with the size of the set they• Intentional models do not necessarily grow with the size of the set they represent
140
![Page 141: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/141.jpg)
OBDD: A Useful Intentional ModelOBDD: A Useful Intentional Model
OBDD (O d d Bi D i i Di )• OBDD (Ordered Binary Decision Diagram)– A compact representation of Boolean functions
• Characteristic function for transition relations– Transitions = pairs of states– Function from pairs of states to Booleans is true, if there is a transition
between the pair.– Built iteratively by breadth-first expansion of the state space:
• Create a representation of the whole set of states reachable in k+1 steps from the t f t t h bl i k tset of states reachable in k steps
• OBDD stabilizes when all the transitions that can occur in the next step are already represented in the OBDD.
141
![Page 142: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/142.jpg)
From OBDD to Symbolic CheckingFrom OBDD to Symbolic Checking
I i l i i lf i h• Intentional representation itself is not enough.• We must have an algorithm for determining whether it satisfies the
property we are checking.
• Example: A set of communicating state machines using OBDD– To represent the transition relation of a set of communicating state machinesTo represent the transition relation of a set of communicating state machines– To model a class of temporal logic specification formulas
• Combine OBDD representations of model and specification to produce a• Combine OBDD representations of model and specification to produce a representation of just the set of transitions leading to a violation of the specification
– If the set is empty the property has been verifiedIf the set is empty, the property has been verified.
142
![Page 143: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/143.jpg)
Representing Transition Relations as Boolean Functions
b d• a b and cnot(a) or (b and c)
aF T
• BDD is a decision tree that has been transformed into an acyclic graph by merging nodes leading
bF Tg p y g g g
to identical sub-trees.F T
ccF T
F T
143
![Page 144: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/144.jpg)
Representing Transition Relations as Boolean Functions : Steps
A A i l b l h a ( 0 0) x0(A) (C)
A. Assign a label to each stateB. Encode transitions C. The transition tuples correspond
s0 (00)
b (x0=1)
a (x0=0) x00 1
x10 1
x10 1
( ) ( )
p pto paths leading to true, and all other paths lead to false.
s1 (01)
0 1 0 1
x2x2
0 1
b (x0=1)
0 1
x30 1
0 1
x30 1
s2 (10)
x30 1
0 0 0 00
x1 x2 x3 x4 x0
0 1
x40 1
0 1
x40 1
(B)0 1
0 0 0 0 0
0 0 0 1 1
sym from state to state
0 1 1 0 1
144
F Tsym from state to state
![Page 145: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/145.jpg)
Intentional vs Explicit RepresentationsIntentional vs. Explicit Representations
W• Worst case:– Given a large set S of states,– a representation capable of distinguishing each subset of S cannot be more
t th th t ti th t i l li t l t f thcompact on average than the representation that simply lists elements of the chosen subset.
I t ti l t ti k ll h th l it t t d• Intentional representations work well when they exploit structure and regularity of the state space.
145
![Page 146: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/146.jpg)
Model RefinementModel Refinement
C i f fi i d l• Construction of finite state models – Should balance precision and efficiency
• Often the first model is unsatisfactory – Report potential failures that are obviously impossible– Exhaust resources before producing any result
• Minor differences in the model can have large effects on tractability of the verification procedure.
• Finite state verification as iterative process is required.
146
![Page 147: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/147.jpg)
Iteration ProcessIteration Process
construct aninitial model
attempt verification
exhausts computational
spuriousltcomputational
resourcesresults
abstract the model further
make the modelmore precise
147
![Page 148: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/148.jpg)
Refinement 1: Adding Details to the ModelRefinement 1: Adding Details to the Model
M | P I i i l ( i ) d lM1 |= P Initial (coarse grain) model(The counter example that violates P is possible in M1, but does not correspond to an execution of the real program.)
M2 |= P Refined (more detailed) model(the counterexample above is not possible in M2 , but a newcounterexamples violates M2 , and does not correspond to anexecution of the real program too.)
....
Mk |= P Refined (final) model(the counter example that violates P in Mk corresponds to an(the counter example that violates P in Mk corresponds to anexecution in the real program.)
148
![Page 149: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/149.jpg)
Refinement 2: Add Premises to the PropertyRefinement 2: Add Premises to the Property
I i i l ( i ) d lInitial (coarse grain) model
M |= P|
Add a constraint C1 that eliminates the bogus behavior
M |= C1 P
M |= (C1 and C2) P....
Until the verification succeeds or produces a valid counter example
149
![Page 150: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/150.jpg)
Data Model Verification and Relational AlgebraData Model Verification and Relational Algebra
A h li i f FSV b id• Another application of FSV, besides concurrent systems
• Many information systems are characterized by y y y– Simple logic and algorithms– Complex data structures
• Key element of these systems is the data model (UML class and object diagrams + OCL assertions)= Sets of data and relations among them Sets of data and relations among them
• The challenge is to prove that I di id l t i t i t t– Individual constraints are consistent.
– They ensure the desired properties of the system as a whole.
150
![Page 151: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/151.jpg)
An Example: Simple Web SiteAn Example: Simple Web Site
• A set of pages divided among three kinds of pages• A set of pages divided among three kinds of pages– Unrestricted pages: freely accessible– Restricted pages: accessible only to registered users– Maintenance pages: inaccessible to both sets of usersp g
• A set of users: – administrator, registered, and unregistered
• A set of links relations among pages– Private links lead to restricted pages– Public links lead to unrestricted pagesPublic links lead to unrestricted pages– Maintenance links lead to maintenance pages
• A set of access rights relations between users and pages– Unregistered users can access only unrestricted pages– Registered users can access both restricted and unrestricted pages– Administrator can access all pages including maintenance pages
151
![Page 152: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/152.jpg)
Data Model of the Simple Web SiteData Model of the Simple Web Site
152
![Page 153: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/153.jpg)
Relational Algebra Specification (Alloy) for PageRelational Algebra Specification (Alloy) for Page
module WebSite
// P i l d th di j i t t f li k
signature: set Page
// Pages include three disjoint sets of links sig Page {disj linksPriv, linksPub, linksMain: set Page }
// Each type of link points to a particular class of page constraintsyp p p p gfact connPub {all p:Page, s: Site | p.linksPub in s.unres } fact connPriv {all p:Page, s: Site | p.linksPriv in s.res } fact connMain {all p:Page, s: Site | p.linksMain in s.main }
introduce relations
// Self loops are not allowed fact noSelfLoop {no p:Page| p in p.linksPriv+p.linksPub+p.linksMain }
153
![Page 154: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/154.jpg)
Relational Algebra Specification (Alloy) for UserRelational Algebra Specification (Alloy) for User
// Users are characterized by the set of pages that they can access sig User { pages: set Page } // Users are partitioned into three sets part sig Administrator Registered Unregistered extends User { }part sig Administrator, Registered, Unregistered extends User { }// Unregistered users can access only the home page, and unrestricted pagesfact accUnregistered {
all u: Unregistered, s: Site|u.pages = (s.home+s.unres) g , | p g ( )}// Registered users can access the home page,restricted and unrestricted pages fact accRegistered {
all u: Registered s: Site|u pages = (s home+s res+s unres)all u: Registered, s: Site|u.pages = (s.home+s.res+s.unres) }// Administrators can access all pages fact accAdministrator { {
all u: Administrator, s: Site|u.pages = (s.home+s.res+s.unres+s.main)
}
Constraints mapusers to pages
154
![Page 155: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/155.jpg)
Analysis of the Relational Algebra SpecificationsAnalysis of the Relational Algebra Specifications
O t i d ifi ti t ti fi bl b i l t ti• Overconstrained specifications are not satisfiable by any implementation. • Underconstrained specifications allow undesirable implementations.
• Specifications identify infinite sets of solutions. – Therefore, properties of a relational specification are undecidable.
• A (counter) example that invalidates a property can be found within a finite set of small models.
– Then, we can verify a specification over a finite set of solutions by limiting the cardinality of the setscardinality of the sets
155
![Page 156: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/156.jpg)
Checking a Finite Set of SolutionsChecking a Finite Set of Solutions
If l i f d• If an example is found,– There are no logical contradictions in the model.– The solution is not overconstrained.
• If no counterexample of a property is found,– No reasonably small solution (property violation) exists. – BUT, NOT that NO solution exists.– We depend on a “small scope hypothesis”: Most bugs that can cause failure
with large collections of objects can also cause failure with very small collections. (so it’s worth looking for bugs in small collections even if we can’t afford to look in big ones)
156
![Page 157: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/157.jpg)
Analysis of the Simple Web Site SpecificationAnalysis of the Simple Web Site Specification
Cardinality limit:Cardinality limit: Consider up to 5 objects of each type
run init for 5
// Can unregistered users visit all unrestricted pages? assert browsePub {assert browsePub {
all p: Page, s: Site | p in s.unres implies s.home in p.* linksPub}check browsePub for 3
*
Property to be checked Transitive closure(including home)
157
![Page 158: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/158.jpg)
Analysis ResultAnalysis Result
C lCounterexample:
• Unregistered User_2 cannot visit the User_2 Site_0
unrestricted page page_2.• The only path from the home page
to page_2 goes through the restricted page page 0Page 1
pages home unres
res restricted page page_0.• The property is violated because
unrestricted browsing paths can be interrupted by restricted pages or
Page_1
pages
P 0
linksPriv
li k P b
unres
res
interrupted by restricted pages or pages under maintenance.
Page_0linksPub
linksPub
Page_2
158
![Page 159: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/159.jpg)
Correcting the SpecificationCorrecting the Specification
W difi d h bl b li i i bli li k f• We can modified the problem by eliminating public links from maintenance or reserved pages:
fact descendant {fact descendant { all p:Pages, s:Site|p in s.main+s.res
implies no p. links.linkPub}}
• Analysis would find no counterexample of cardinality 3.y p y• We cannot conclude that no larger counter-example exists, but we may
be satisfied that there is no reason to expect this property to be violated only in larger models.only in larger models.
159
![Page 160: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/160.jpg)
SummarySummary
Fi i ifi i i l i• Finite state verification is complementary to testing.– Can find bugs that are extremely hard to test for
• Example: race conditions that happen very rarely, under conditions that h d t t lare hard to control
– But is limited in scope• Cannot be used to find all kinds of errors
• Checking models can be (and is) automated• But designing good models is challenging.g g g g g
• Requires careful consideration of abstraction, granularity, and the properties to be checked
• Often requires a cycle of model / check / refine until a useful result is q yobtained
160
![Page 161: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/161.jpg)
161
![Page 162: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/162.jpg)
Part III. Problems and Methods
![Page 163: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/163.jpg)
Chapter 9Chapter 9. Test Case Selection and Adequacy
![Page 164: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/164.jpg)
Learning ObjectivesLearning Objectives
U d d h f d fi i d i i d h i• Understand the purpose of defining test adequacy criteria and their limitations
• Understand basic terminology of test selection and adequacy• Know some sources of information commonly used to define adequacy
criteria• Understand how test selection and adequacy criteria are usedq y
164
![Page 165: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/165.jpg)
OverviewOverview
Wh ld lik k i• What we would like to know is– A real way of measuring effectiveness of testing– “If the system passes an adequate suite of test cases, then it must be correct.”
• But that’s impossible.– The adequacy of test suites is provably undecidable.
• Therefore we’ll have to settle on weaker proxies for adequacy• Therefore, we ll have to settle on weaker proxies for adequacy.
165
![Page 166: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/166.jpg)
Source of Test SpecificationSource of Test Specification
S f t t ifi tiTesting Other names
Source of test specification
Example
F ti l Bl k b t tiSoftware specification
Functional Testing
Black box testingSpecification-based testing
If specification requires robust recovery from power failure, test obligations should include simulated power failure.
Structural Source codeStructural Testing
White box testingSource code
Traverse each program loop one or more times
Models of system• Models used in specification or design
Model-based Testing
Models used in specification or design• Models derived from source code
Exercise all transitions in communication protocol model
Fault-based Testing
Hypothesized faults, common bugs
Check for buffer overflow handling (common vulnerability) by testing on very large inputs
166
![Page 167: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/167.jpg)
Terminologies in TestingTerminologies in Testing
Terms Descriptions
Test case a set of inputs, execution conditions, and a pass/fail criterionp , , p /
Test case specification (Test specification)
a requirement to be satisfied by one or more test cases
a partial test case specification requiring some property deemedTest obligation
a partial test case specification, requiring some property deemed important to thorough testing
Test suite a set of test cases
Test (Test execution)
the activity of executing test cases and evaluating their results
Adequacy criterion a predicate that is true (satisfied) or false of a program, test suite pair
167
![Page 168: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/168.jpg)
Adequacy CriteriaAdequacy Criteria
Ad i i S f bli i• Adequacy criterion = Set of test obligations
• A test suite satisfies an adequacy criterion, iffq y– All the tests succeed (pass), and– Every test obligation in the criterion is satisfied by at least one of the test
cases in the test suite.
– Example: • “The statement coverage adequacy criterion is satisfied by test suite S for g q y y
program P, if each executable statement in P is executed by at least one test case in S, and the outcome of each test execution was pass.”
168
![Page 169: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/169.jpg)
SatisfiabilitySatisfiability
S i i i f i i f i• Sometimes no test suite can satisfy a criterion for a given program.– Example:
• Defensive programming style includes “can’t happen” sanity checks.if ( 0) {– if (z < 0) {throw new LogicError (“z must be positive here!”)
}• For this program, no test suite can satisfy statement coverage.
• Two ways of coping with the unsatisfiability of adequacy criteria1. Exclude any unsatisfiable obligation from the criterion2. Measure the extent to which a test suite approaches an adequacy criterion
169
![Page 170: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/170.jpg)
Coping with the UnsatisfiabilityCoping with the Unsatisfiability
A h A• Approach A – Exclude any unsatisfiable obligation from the criterion– Example:
• Modify statement coverage to require execution only of statements which can be executed
– But, we can’t know for sure which are executable or not.
• Approach B – Measure the extent to which a test suite approaches an adequacy criterion
E l– Example• If a test suite satisfies 85 of 100 obligations, we have reached 85% coverage.
– Terms: A d it i i ti fi d t• An adequacy criterion is satisfied or not.
• A coverage measure is the fraction of satisfied obligations.
170
![Page 171: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/171.jpg)
CoverageCoverage
M i b f l i di f• Measuring coverage (% of satisfied test obligations) can be a useful indicator of – Progress toward a thorough test suite (thoroughness of test suite)
– Trouble spots requiring more attention in testing
• But, coverage is only a proxy for thoroughness or adequacy., g y p y g q y– It’s easy to improve coverage without improving a test suite (much easier
than designing good test cases)– The only measure that really matters is (cost-) effectiveness.y y
171
![Page 172: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/172.jpg)
Comparing CriteriaComparing Criteria
C di i i h f k d i i ?• Can we distinguish stronger from weaker adequacy criteria?
• Analytical approachy pp– Describe conditions under which one adequacy criterion is provably stronger
than another– Just a piece of the overall “effectiveness” question– Stronger = gives stronger guarantees
→ Subsumes relation
172
![Page 173: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/173.jpg)
Subsumes RelationSubsumes Relation
T d i i A b d i i B iff f• Test adequacy criterion A subsumes test adequacy criterion B iff, for every program P, every test suite satisfying A with respect to P also satisfies B with respect to P.
E E i i ll b h (b h ) b i i– E.g. Exercising all program branches (branch coverage) subsumes exercising all program statements.
• A common analytical comparison of closely related criteria– Useful for working from easier to harder levels of coverage, but not a direct
i di ti f litindication of quality
173
![Page 174: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/174.jpg)
174
![Page 175: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/175.jpg)
Use of Adequacy CriteriaUse of Adequacy Criteria
1 T l i h1. Test selection approaches (Selection)
– Guidance in devising a thorough test suite• E.g. A specification-based testing criterion may suggest test cases
i t ti bi ti f lcovering representative combinations of values.
2. Revealing missing tests (Measurement)
– Post hoc analysis: What might I have missed with this test suite?
• Often in combination– Design test suite from specifications, then use structural criterion (e.g.
coverage of all branches) to highlight missed logic
175
![Page 176: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/176.jpg)
SummarySummary
Ad i i id d fi i f “ h h ” i• Adequacy criteria provide a way to define a notion of “thoroughness” in a test suite.
– But, they don’t offer guarantees.– More like rules to highlight inadequacy
• Adequacy criteria are defined in terms of “covering” some information– Derived from many sources(specs, code, models, etc.)
• Adequacy criteria may be used for selection as well as measurement. q y y– But, an aid to thoughtful test design, not a substitute
176
![Page 177: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/177.jpg)
177
![Page 178: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/178.jpg)
Chapter 10Chapter 10. Functional Testing
![Page 179: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/179.jpg)
Learning ObjectivesLearning Objectives
U d d h i l f i ( d ) l i f• Understand the rationale for systematic (non-random) selection of test cases
• Understand why functional test selection is a primary, base-line technique
• Distinguish functional testing from other systematic testing techniques
179
![Page 180: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/180.jpg)
Functional TestingFunctional Testing
F i l i• Functional testing– Deriving test cases from program specifications – ‘Functional’ refers to the source of information used in test case design, not
t h t i t t dto what is tested.
• Also known as:– Specification-based testing (from specifications)– Black-box testing (no view of source code)
• Functional specification = description of intended program behavior– Formal or informal
180
![Page 181: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/181.jpg)
Systematic testing vs Random testingSystematic testing vs. Random testing
R d ( if ) i• Random (uniform) testing– Pick possible inputs uniformly– Avoids designer’s bias– But, treats all inputs as equally valuable
• Systematic (non-uniform) testingy ( ) g– Try to select inputs that are especially valuable– Usually by choosing representatives of classes that are apt to fail often or not
at all
• Functional testing is a systematic (partition-based) testing strategy.Functional testing is a systematic (partition based) testing strategy.
181
![Page 182: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/182.jpg)
Why Not Random Testing?Why Not Random Testing?
D if di ib i f f l• Due to non-uniform distribution of faults– Example:
• Java class “roots” applies quadratic equation
– Supposed an incomplete implementation logic: • Program does not properly handle the case in which b2 - 4ac =0 and a=0
– Failing values are sparse in the input space: needles in a very big haystack– Random sampling is unlikely to choose a=0 and b=0.
182
![Page 183: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/183.jpg)
Purpose of TestingPurpose of Testing
O l i fi d dl d h f h• Our goal is to find needles and remove them from hay. → Look systematically (non-uniformly) for needles !!!– We need to use everything we know about needles.
E A th h i th h ? D th ift t th b tt ?• E.g. Are they heavier than hay? Do they sift to the bottom?
• To estimate the proportion of needles to hay• To estimate the proportion of needles to hay → Sample randomly !!!– Reliability estimation requires unbiased samples for valid statistics. – But that’s not our goal– But that s not our goal.
183
![Page 184: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/184.jpg)
Systematic Partition TestingSystematic Partition Testing
Failure (valuable test case)Failures are sparse in But dense in some partsFailure (valuable test case)
No failurethe space of possible inputs.
But, dense in some parts of the space
esnp
ut v
alue
) po
ssib
le in
hays
tack
spac
e of
p(th
e
If we systematically test some cases from each part, we will include the Functional testing is one way of
drawing pink lines to isolate regions
The
184
p ,dense parts. drawing pink lines to isolate regions
with likely failures
![Page 185: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/185.jpg)
Principles of Systematic PartitioningPrinciples of Systematic Partitioning
E l i k l d h l h lik l• Exploit some knowledge to choose samples that are more likely to include “special” or “trouble-prone” regions of the input space
– Failures are sparse in the whole input space.– But, we may find regions in which they are dense.
• (Quasi-) Partition testing: separates the input space into classes whose union is the entire space
• Desirable case: Each fault leads to failures that are dense (easy to find) in es ab e case: ac au t eads to a u es t at a e de se (easy to d)some class of inputs
– Sampling each class in the quasi-partition selects at least one input that leads to a failure, revealing the fault.g
– Seldom guaranteed; We depend on experience-based heuristics.
185
![Page 186: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/186.jpg)
A Systematic Approach: Functional TestingA Systematic Approach: Functional Testing
F i l i h ifi i (f l i f l) i i• Functional testing uses the specification (formal or informal) to partition the input space.
– E.g. Specification of “roots” program suggests division between cases with d t l tzero, one, and two real roots.
• Test each category and boundaries between categories– No guarantees, but experience suggests failures often lie at the boundaries.
(as in the “roots” program)
• Functional Testing is a base-line technique for designing test cases.
186
![Page 187: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/187.jpg)
Functional TestingFunctional Testing
Th b li h i f d i i• The base-line technique for designing test cases– Timely
• Often useful in refining specifications and assessing testability before d i ittcode is written
– Effective• Find some classes of fault (e.g. missing logic) that can elude other
approachesapproaches– Widely applicable
• To any description of program behavior serving as specificationAt l l f l it f d l t t t ti• At any level of granularity from module to system testing
– Economical• Typically less expensive to design and execute than structural (code-
based) test casesbased) test cases
187
![Page 188: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/188.jpg)
Functional Test vs Structural TestFunctional Test vs. Structural Test
Diff t t ti t t i t ff ti f diff t l f• Different testing strategies are most effective for different classes of faults.
F ti l t ti i b t f i i l i f lt• Functional testing is best for missing logic faults.– A common problem: Some program logic was simply forgotten.– Structural (code-based) testing will never focus on code that isn’t there.
• Functional test applies at all granularity levels– Unit (from module interface spec)– Integration (from API or subsystem spec)– Integration (from API or subsystem spec)– System (from system requirements spec)– Regression (from system requirements + bug history)
• Structural test design applies to relatively small parts of a system– Unit and integration testing
188
![Page 189: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/189.jpg)
Main Steps of Functional Program TestingMain Steps of Functional Program Testing
Functional specificationsFunctional specifications
I d d tl T t bl F t
Identify independently testable featuresFinite State Machine,
Grammar,Algebraic Specification
Brute force testing
Independently Testable Feature
Derive a modelIdentify representative values
Algebraic Specification,Logic Specification,
CFG / DFG
Representative Values Model
Generate test case specifications Test selection criteria
Semantic Constraint,C bi ti l S l ti
Test Case Specification
Generate test cases
criteria
Manual Mapping
Combinational Selection,Exhaustive Enumeration,
Random Selection
Test Cases
Instantiate tests
Manual Mapping,Symbolic Execution,
A-posteriori Satisfaction
189
Scaffolding
![Page 190: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/190.jpg)
y ntly
e s
Iden
tify
Inde
pend
eTe
stab
leFe
atur
es
Generate Test-C
Specifica erate Test-Case
ifications
t-Case
cations Genera
Specifi
Gen
erat
e Te
st C
ases
Inst
antia
teTe
sts
190
![Page 191: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/191.jpg)
From Specifications to Test CasesFrom Specifications to Test Cases
1 Id if i d d l bl f1. Identify independently testable features– If the specification is large, break it into independently testable features.
2. Identify representative classes of values, or derive a model of behavior– Often simple input/output transformations don’t describe a system. – We use models in program specification, in program design, and in test
design too.
3. Generate test case specificationsp– Typically, combinations of input values or model behaviors
4. Generate test cases and instantiate tests4. Generate test cases and instantiate tests
191
![Page 192: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/192.jpg)
SummarySummary
F i l i i l bl d• Functional testing (generating test cases from specifications) is a valuable and flexible approach to software testing.
– Applicable from very early system specifications right through module ifi tispecifications
• Partition testing suggests dividing the input space into equivalent classes.– Systematic testing is intentionally non-uniform to address special cases, error
conditions and other small places.– Dividing a big haystack into small and hopefully uniform piles where the
needles might be concentratedneedles might be concentrated
192
![Page 193: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/193.jpg)
193
![Page 194: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/194.jpg)
Chapter 11Chapter 11. Combinatorial Testing
![Page 195: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/195.jpg)
Learning ObjectivesLearning Objectives
U d d h k id i bi i l h• Understand three key ideas in combinatorial approaches– Category-partition testing– Pairwise testing – Catalog-based testing
195
![Page 196: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/196.jpg)
OverviewOverview
C bi i l i id ifi di i ib h b i d• Combinatorial testing identifies distinct attributes that can be varied. – In data, environment or configuration– Example:
• Browser could be “IE” or “Firefox”• Operating system could be “Vista”, “XP” or “OSX”
C bi i l i i ll bi i b d• Combinatorial testing systematically generates combinations to be tested.– Example:
• IE on Vista, IE on XP, Firefox on Vista, Firefox on OSX, etc.
• Rationale: – Test cases should be varied and include possible “corner cases”.
196
![Page 197: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/197.jpg)
Key Ideas in Combinatorial ApproachesKey Ideas in Combinatorial Approaches
1 C i i i1. Category-partition testing– Separate (manual) identification of values that characterize the input space
from (automatic) generation of combinations for test cases
2. Pairwise testing – Systematically test interactions among attributes of the program input space
with a relatively small number of test cases
3. Catalog-based testing– Aggregate and synthesize the experience of test designers in a particular gg g y p g p
organization or application domain, to aid in identifying attribute values
197
![Page 198: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/198.jpg)
1 Category Partition Testing1. Category-Partition Testing
1 D h ifi i i i d d l bl f1. Decompose the specification into independently testable features– for each feature, identify parameters and environment elements– for each parameter and environment element, identify elementary
h t i ti ( t i )characteristics (→ categories)
2. Identify representative values– for each characteristic(category), identify classes of values
• normal values• boundary values
i l l• special values• error values
3 Generate test case specifications3. Generate test case specifications
198
![Page 199: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/199.jpg)
An Example: “Check Configuration”An Example: Check Configuration
I h W b i f f i D ll ‘ h ki• In the Web site of a computer manufacturer, i.e. Dell, ‘checking configuration’ checks the validity of a computer configuration.
– Two parameters:d l• Model
• Set of Components
199
![Page 200: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/200.jpg)
Informal Specification of ‘Model’Informal Specification of Model
Model: A model identifies a specific product and determines a set of constraints on available components. Models are characterized by logical slots for components which may or may not be implemented by physicalslots for components, which may or may not be implemented by physical slots on a bus. Slots may be required or optional. Required slots must be assigned with a suitable component to obtain a legal configuration, while optional slots may be left empty or filled depending on the customer’s p y p y p gneeds.
Example: The required “slots” of the Chipmunk C20 laptop computer include a screen, a processor, a hard disk, memory, and an operating system. (Of these, only the hard disk and memory are implemented using actual hardware slots on a bus.) The optional slots include external storage devices such as a CD/DVD writersuch as a CD/DVD writer.
200
![Page 201: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/201.jpg)
Informal Specification of ‘Set of Component’Informal Specification of Set of Component
Set of Components: A set of (slot, component) pairs, corresponds to the required and optional slots of the model. A component is a choice that can be varied within a model and which is not designed to be replaced by thebe varied within a model, and which is not designed to be replaced by the end user. Available components and a default for each slot is determined by the model. The special value empty is allowed (and may be the default selection) for optional slots. In addition to being compatible or incompatible ) p g p pwith a particular model and slot, individual components may be compatible or incompatible with each other.
Example: The default configuration of the Chipmunk C20 includes 20 gigabytes of hard disk; 30 and 40 gigabyte disks are also available. (Since the hard disk is a required slot, empty is not an allowed choice.) The default operating system is RodentOS 3 2 personal edition but RodentOS 3 2operating system is RodentOS 3.2, personal edition, but RodentOS 3.2 mobile server edition may also be selected. The mobile server edition requires at least 30 gigabytes of hard disk.
201
![Page 202: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/202.jpg)
Step 1: Identify Independently Testable Features and Parameter Characteristics
Ch i i• Choosing categories– No hard-and-fast rules for choosing categories!– Not a trivial task
• Categories reflect test designer's judgment. – Which classes of values may be treated differently by an implementation.
• Choosing categories well requires experience and knowledge of the C oos g catego es e equ es e pe e ce a d o edge o t eapplication domain and product architecture.
202
![Page 203: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/203.jpg)
Identify Independently Testable UnitsIdentify Independently Testable Units
M d l b
Model
Model number
Number of required slots for selected model (#SMRS)
Number of optional slots for selected model (#SMOS)p ( )
Correspondence of selection with model slots
N b f i d i h l i
Components
Number of required components with selection empty
Required component selection
Number of optional components with selection emptyNumber of optional components with selection empty
Optional component selection
Number of models in database (#DBM)Product Database
Number of models in database (#DBM)
Number of components in database (#DBC)
Parameters Categories
203
Parameters Categories
![Page 204: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/204.jpg)
Step 2: Identify Representative ValuesStep 2: Identify Representative Values
Id tif t ti l f l f h f th t i• Identify representative classes of values for each of the categories
• Representative values may be identified by applying B d l t ti– Boundary value testing
• Select extreme values within a class • Select values outside but as close as possible to the class• Select interior (non-extreme) values of the class
– Erroneous condition testing• Select values outside the normal domain of the program
204
![Page 205: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/205.jpg)
Representative Values: ModelRepresentative Values: Model
M d l b• Model numberMalformedNot in databaseValidValid
• Number of required slots for selected model (#SMRS)001Many
• Number of optional slots for selected model (#SMOS)011Many
205
![Page 206: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/206.jpg)
Representative Values: ComponentsRepresentative Values: Components
C d f l ti ith d l l t• Correspondence of selection with model slotsOmitted slotsExtra slotsMismatched slotsComplete correspondence
• Number of required components with non empty selection00< number required slots= number required slots
Req i ed component selection• Required component selectionSome defaultsAll valid 1 incompatible with slots
1 i ibl i h h l i 1 incompatible with another selection 1 incompatible with model 1 not in database
206
![Page 207: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/207.jpg)
Representative Values: ComponentsRepresentative Values: Components
N b f ti l t ith t l ti• Number of optional components with non empty selection0< #SMOS= #SMOS SMOS
• Optional component selectionSome defaultsAll valid 1 incompatible with slots 1 incompatible with another selection 1 incompatible with model 1 incompatible with model 1 not in database
207
![Page 208: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/208.jpg)
Representative Values: Product DatabaseRepresentative Values: Product Database
N b f d l i d t b (#DBM)• Number of models in database (#DBM)01ManyMany
• Number of components in database (#DBC)01Many
• Note 0 and 1 are unusual (special) values. – They might cause unanticipated behavior alone or in combination with
particular values of other parametersparticular values of other parameters.
208
![Page 209: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/209.jpg)
Step 3: Generate Test Case SpecificationsStep 3: Generate Test Case Specifications
A bi i f l f h d• A combination of values for each category corresponds to a test case specification.
– In the example, we have 314,928 test cases.– Most of which are impossible.– Example: zero slots and at least one incompatible slot
• Need to introduce constraints in order to– Rule out impossible combinations, and– Reduce the size of the test suite, if too large
– Example:• Error constraints• Property constraints• Single constraints
209
![Page 210: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/210.jpg)
Error ConstraintsError Constraints
[ ] i di l l h Model number• [error] indicates a value class that corresponds to an erroneous values.
N d b i d l
Model numberMalformed [error]Not in database [error]Valid
Correspondence of selection with model l– Need to be tried only once
• Error value class
Correspondence of selection with model slotsOmitted slots [error]Extra slots [error]Mismatched slots [error]Complete correspondence
– No need to test all possible combinations of errors, and one test is enough.
p p
Number of required comp. with non empty selection0 [error]< number of required slots [error]
Required comp. selection 1 not in database [error]
Number of models in database (#DBM)0 [error]
Number of components in database (#DBC)0 [error]
210
Error constraints reduce test suite from 314,928 to 2,711 test cases
![Page 211: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/211.jpg)
Property ConstraintsProperty Constraints
C i [ ] [if ] l i lid bi i f l• Constraint [property] [if-property] rule out invalid combinations of values.– [property] groups values of a single parameter to identify subsets of values
with common properties.[if t ] b d th h i f l f t th t b– [if-property] bounds the choices of values for a category that can be combined with a particular value selected for a different category.
211
![Page 212: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/212.jpg)
Property ConstraintsProperty Constraints
Number of required slots for selected model (#SMRS)Number of required slots for selected model (#SMRS)1 [property RSNE]Many [property RSNE] [property RSMANY]
Number of optional slots for selected model (#SMOS)1 [property OSNE]Many [property OSNE] [property OSMANY]
Number of required comp. with non empty selection0 [if RSNE] [error]
b i d l [if RSNE] [ ]< number required slots [if RSNE] [error]= number required slots [if RSMANY]
Number of optional comp with non empty selectionNumber of optional comp. with non empty selection< number required slots [if OSNE]= number required slots [if OSMANY]
from 2 711 to 908 test cases
212
from 2,711 to 908 test cases
![Page 213: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/213.jpg)
Single ConstraintsSingle Constraints
[ i l ] i di l l h d i h l• [single] indicates a value class that test designers choose to test only once to reduce the number of test cases.
• Example– Value some default for required component selection and optional
component selection may be tested only once despite not being an erroneous condition.
• Note – Single and error have the same effect but differ in rationale. – Keeping them distinct is important for documentation and regression testing.
213
![Page 214: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/214.jpg)
Single ConstraintsSingle Constraints
N b f i d l t f l t d d l (#SMRS)Number of required slots for selected model (#SMRS)0 [single]1 [property RSNE] [single]
Number of optional slots for selected model (#SMOS)0 [single]1 [single] [property OSNE]
Required component selectionSome default [single]
Optional component selectionOptional component selectionSome default [single]
Number of models in database (#DBM)1 [ i l ]1 [single]
Number of components in database (#DBC)1 [single] from 908 to 69 test cases
214
from 908 to 69 test cases
![Page 215: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/215.jpg)
Check Configuration Summary of CategoriesCheck Configuration – Summary of Categories
Parameter Model Parameter ComponentParameter Model• Model number
– Malformed [error]– Not in database [error]– Valid
f f
Parameter Component• Correspondence of selection with model slots
– Omitted slots [error]– Extra slots [error]– Mismatched slots [error]– Complete correspondence
• Number of required slots for selected model (#SMRS)– 0 [single]– 1 [property RSNE] [single] – Many [property RSNE] [property RSMANY]
• Number of optional slots for selected model (#SMOS)
p p• # of required components (selection empty)
– 0 [if RSNE] [error]– < number required slots [if RSNE] [error]– = number required slots [if RSMANY]
• Required component selection– 0 [single]– 1 [property OSNE] [single] – Many [property OSNE] [property OSMANY]
Environment Product data base
– Some defaults [single]– All valid– 1 incompatible with slots– 1 incompatible with another selection– 1 incompatible with model– 1 not in database [error]
• Number of models in database (#DBM)– 0 [error]– 1 [single]– Many
• Number of components in database (#DBC)
1 not in database [error]• # of optional components (selection empty)
– 0– < #SMOS [if OSNE]– = #SMOS [if OSMANY]
• Optional component selectionp ( )– 0 [error]– 1 [single]– Many
p p– Some defaults [single]– All valid– 1 incompatible with slots– 1 incompatible with another selection– 1 incompatible with model– 1 not in database [error]– 1 not in database [error]
215
![Page 216: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/216.jpg)
Category Partitioning Testing in SummaryCategory-Partitioning Testing, in Summary
C i i i i i h• Category partition testing gives us systematic approaches to – Identify characteristics and values (the creative step)– Generate combinations (the mechanical step)
• But, test suite size grows very rapidly with number of categories. • Pairwise (and n-way) combinatorial testing is a non-exhaustive approach.
– Combine values systematically but not exhaustively
216
![Page 217: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/217.jpg)
2 Pairwise Combination Testing2. Pairwise Combination Testing
C i i k ll h i i i i d h• Category partition works well when intuitive constraints reduce the number of combinations to a small amount of test cases.
– Without many constraints, the number of combinations may be blunmanageable.
• Pairwise combination– Instead of exhaustive combinations– Generate combinations that efficiently cover all pairs (triples,…) of classes– Rationale:
• Most failures are triggered by single values or combinations of a few values.• Covering pairs (triples,…) reduces the number of test cases, but reveals most faults.
217
![Page 218: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/218.jpg)
An Example: Display ControlAn Example: Display Control
N i d h l b f bi i 432 (3 4 3 4 3)• No constraints reduce the total number of combinations 432 (3x4x3x4x3) test cases, if we consider all combinations.
Display Mode Language Fonts Color Screen size
full-graphics English Minimal Monochrome Hand-held
text-only French Standard Color-map Laptop
limited-bandwidth Spanish Document-loaded 16-bit Full-size
Portuguese True-color
218
![Page 219: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/219.jpg)
Pairwise Combination: 17 Test CasesPairwise Combination: 17 Test CasesLanguage Color Display Mode Fonts Screen Size
English Monochrome Full-graphics Minimal Hand-held
English Color-map Text-only Standard Full-size
English 16-bit Limited-bandwidth - Full-size
E li h T l T t l D t l d d L tEnglish True-color Text-only Document-loaded Laptop
French Monochrome Limited-bandwidth Standard Laptop
French Color-map Full-graphics Document-loaded Full-size
French 16 bit Text only MinimalFrench 16-bit Text-only Minimal -
French True-color - - Hand-held
Spanish Monochrome - Document-loaded Full-size
Spanish Color-map Limited-bandwidth Minimal Hand-heldSpanish Color map Limited bandwidth Minimal Hand held
Spanish 16-bit Full-graphics Standard Laptop
Spanish True-color Text-only - Hand-held
Portuguese - - Monochrome Text-onlyPortuguese Monochrome Text only
Portuguese Color-map - Minimal Laptop
Portuguese 16-bit Limited-bandwidth Document-loaded Hand-held
Portuguese True-color Full-graphics Minimal Full-size
219
Portuguese True color Full graphics Minimal Full size
Portuguese True-color Limited-bandwidth Standard Hand-held
![Page 220: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/220.jpg)
Adding ConstraintsAdding Constraints
Si l i• Simple constraints– Example: “Color monochrome not compatible with screen laptop and full size”
can be handled by considering the case in separate tables.
Display Mode Language Fonts Color Screen size
full-graphics English Minimal Monochrome Hand-held
text-only French Standard Color-map
limited-bandwidth Spanish Document-loaded 16-bit
Portuguese True-color
Display Mode Language Fonts Color Screen size
full-graphics English Minimalg p g
text-only French Standard Color-map Laptop
limited-bandwidth Spanish Document-loaded 16-bit Full-size
Portuguese True-color
220
Portuguese True color
![Page 221: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/221.jpg)
Pairwise Combination Testing in SummaryPairwise Combination Testing, in Summary
C i i h i• Category-partition approach gives us– Separation between (manual) identification of parameter characteristics and
values, and (automatic) generation of test cases that combine themC t i t t d th b f bi ti– Constraints to reduce the number of combinations
• Pairwise (or n-way) testing gives us – Much smaller test suites, even without constraints– But, we can still use constraints.
• We still need help to make the manual step more systematic.
221
![Page 222: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/222.jpg)
3 Catalog based Testing3. Catalog-based Testing
D i i l l i h j d• Deriving value classes requires human judgment. • Therefore, gathering experience in a systematic collection can
– Speed up the test design process – Routinize many decisions, better focusing human effort– Accelerate training and reduce human error
• Catalogs capture the experience of test designers by listing important cases for each possible type of variable.
– Example: If the computation uses an integer variable, a catalog might indicate p p g , g gthe following relevant cases
• The element immediately preceding the lower bound • The lower bound of the interval• A non-boundary element within the interval• The upper bound of the interval• The element immediately following the upper boundy g pp
222
![Page 223: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/223.jpg)
Catalog based Testing ProcessCatalog-based Testing Process
1 Id if l i f h ifi i1. Identify elementary items of the specification– Pre-conditions– Post-conditions– Definitions– Variables– Operations
2. Derive a first set of test case specifications from pre-conditions, post-conditions and definitions
3. Complete the set of test case specifications using test catalogs
223
![Page 224: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/224.jpg)
An Example: ‘cgi decode’An Example: cgi_decode
A i f l ifi i f ‘ i d d ’• An informal specification of ‘cgi_decode’
Function cgi_decode translates a cgi-encoded string to a plain ASCII string, reversing the encoding applied by the common gateway interface (CGI) of most web servers.
CGI translates spaces to +, and translates most other non-alphanumeric characters to hexadecimal escape sequences.
cgi_decode maps + to spaces, %xy (where x and y are hexadecimal digits) to the corresponding ASCII character, and other alphanumeric characters to themselvesthemselves.
224
![Page 225: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/225.jpg)
‘cgi digicode’ Input/Outputcgi_digicode Input/Output
[INPUT] d d A t i f h t (th i t CGI )• [INPUT]: encoded A string of characters (the input CGI sequence) containing below and terminated by a null character
– alphanumeric charactersthe character +– the character +
– the substring “%xy” , where x and y are hexadecimal digits
• [OUTPUT]: decoded A string of characters (the plain ASCII characters• [OUTPUT]: decoded A string of characters (the plain ASCII characters corresponding to the input CGI sequence)
– alphanumeric characters copied into output (in corresponding positions)– blank for each ‘+’ character in the inputblank for each character in the input– single ASCII character with value xy for each substring “%xy”
• [OUTPUT]: return value cgi_decode returns[ ] g _– 0 for success – 1 if the input is malformed
225
![Page 226: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/226.jpg)
‘cgi digicode’ Definitionscgi_digicode Definitions
P di i C di i i h b b f h• Pre-conditions: Conditions on inputs that must be true before the execution
– Validated preconditions: checked by the system– Assumed preconditions: assumed by the system
• Post-conditions: Results of the execution
• Variables: Elements used for the computation
• Operations: Main operations on variables and inputs
• Definitions: Abbreviations
226
![Page 227: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/227.jpg)
Step 1: Identify Elementary Items of the SpecificationVAR 1 encoded: a string of ASCII charactersVAR 1 encoded: a string of ASCII charactersVAR 2 decoded: a string of ASCII charactersVAR 3 return value: a boolean
DEF 1 hexadecimal characters, in range ['0' .. '9', 'A' .. 'F', 'a' .. 'f'], g [ , , ]DEF 2 sequences %xy, where x and y are hexadecimal charactersDEF 3 CGI items as alphanumeric character, or '+', or CGI hexadecimal
OP 1 Scan the input string encoded
PRE 1 (Assumed) input string encoded null-terminated string of charsPRE 2 (Validated) input string encoded sequence of CGI items
POST 1 if encoded contains alphanumeric characters they are copied to the output stringPOST 1 if encoded contains alphanumeric characters, they are copied to the output stringPOST 2 if encoded contains characters +, they are replaced in the output string by ASCII SPACE
characters POST 3 if encoded contains CGI hexadecimals, they are replaced by the corresponding ASCII charactersPOST 4 if encoded is processed correctly, it returns 0POST 5 if encoded contains a wrong CGI hexadecimal (a substring xy, where either x or y are absent or
are not hexadecimal digits, cgi_decode returns 1 POST 6 if encoded contains any illegal character, it returns 1VAR 1 encoded: a string of ASCII characters
227
![Page 228: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/228.jpg)
Step 2: Derive an Initial Set of Test Case Specifications
V lid d di i• Validated preconditions:– Simple precondition (expression without operators)
• 2 classes of inputs: – inputs that satisfy the precondition– inputs that do not satisfy the precondition
– Compound precondition (with AND or OR):• apply modified condition/decision (MC/DC) criterion
• Assumed precondition:p– apply MC/DC only to “OR preconditions”
• Postconditions and Definitions:Postconditions and Definitions: – if given as conditional expressions, consider conditions as if they were
validated preconditions
228
![Page 229: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/229.jpg)
Test Cases from PRETest Cases from PRE
PRE 2 (V lid d) h i i d d i f CGI iPRE 2 (Validated): the input string encoded is a sequence of CGI items – TC-PRE2-1: encoded is a sequence of CGI items – TC-PRE2-2: encoded is not a sequence of CGI items
POST 1: if encoded contains alphanumeric characters, they are copied in the output string in the corresponding position
– TC-POST1-1: encoded contains alphanumeric characters – TC-POST1-2: encoded does not contain alphanumeric characters
POST 2: if encoded contains characters +, they are replaced in the output string by ASCII SPACE characters
– TC-POST2-1: encoded contains character +– TC-POST2-2: encoded does not contain character +
229
![Page 230: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/230.jpg)
Test Cases from POSTTest Cases from POST
POST 3 if d d i CGI h d i l h l d b hPOST 3: if encoded contains CGI hexadecimals, they are replaced by the corresponding ASCII characters
– TC-POST3-1 Encoded: contains CGI hexadecimals – TC-POST3-2 Encoded: does not contain a CGI hexadecimal
POST 4: if encoded is processed correctly, it returns 0
POST 5: if encoded contains a wrong CGI hexadecimal (a substring xy, where either x or y are absent or are not hexadecimal digits, cgi decode returns e t e o y a e abse t o a e ot e adec a d g ts, cg _decode etu s1
– TC-POST5-1 Encoded: contains erroneous CGI hexadecimals
POST 6 if encoded contains any illegal character, it returns 1– TC-POST6-1 Encoded: contains illegal characters
230
![Page 231: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/231.jpg)
Step 3: Complete the Test Case Specification using Catalog
S h l i ll• Scan the catalog sequentially – For each element of the catalog,
• Scan the specifications and apply the catalog entry
• Delete redundant test cases
• Catalogg– List of kinds of elements that can occur in a specification– Each catalog entry is associated with a list of generic test case specifications.
• Example: Catalog entry Boolean – Two test case specifications: true, false
Label in/out indicate if applicable only to input output both– Label in/out indicate if applicable only to input, output, both
231
![Page 232: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/232.jpg)
A Simple Test CatalogA Simple Test Catalog
• Boolean • Non Numeric Constant C• Boolean– True [in/out]– False [in/out]
• Enumeration
• Non-Numeric Constant C– C [in/out]– Any other constant compatible with C [in]– Some other compatible value [in]
• Sequence– Each enumerated value [in/out]– Some value outside the enumerated set
[in]
Range L U
• Sequence– Empty [in/out]– A single element [in/out]– More than one element [in/out]– Maximum length (if bounded) or very long
[in/out]• Range L ... U– L-1 [in]– L [in/out]– A value between L and U [in/out]– U [in/out]
[in/out]– Longer than maximum length (if bounded) [in]– Incorrectly terminated [in]
• Scan with action on elements P– P occurs at beginning of sequence [in]U [in/out]
– U+1 in]
• Numeric Constant C– C [in/out]
P occurs at beginning of sequence [in]– P occurs in interior of sequence [in]– P occurs at end of sequence [in]– PP occurs contiguously [in]– P does not occur in sequence [in]– pP where p is a proper prefix of P [in]
– C –1 [in]– C+1 [in]– Any other constant [in]
compatible with C
pP where p is a proper prefix of P [in]– Proper prefix p occurs at end of sequence [in]
232
![Page 233: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/233.jpg)
Catalog Entry: BooleanCatalog Entry: Boolean
B l• Boolean– True [in/out]– False [in/out]
• Application to return value generates 2 test cases already covered by TC-PRE2-1 and TC-PRE2-2.
233
![Page 234: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/234.jpg)
Catalog Entry: EnumerationCatalog Entry: Enumeration
E ti• Enumeration– Each enumerated value [in/out]– Some value outside the enumerated set [in]
• Applications to CGI item (DEF 3)– included in TC-POST1-1, TC-POST1-2, TC-POST2-1, TC-POST2-2, TC-POST3-1, TC-
POST3-2
• Applications to improper CGI hexadecimals– New test case specifications
TC POST5 2 d d t i t d ith “% ” h i h d i l di it• TC-POST5-2 encoded terminated with “%x”, where x is a hexadecimal digit• TC-POST5-3 encoded contains “%ky”, where k is not a hexadecimal digit and y is a
hexadecimal digit• TC-POST5-4 encoded contains “%xk”, where x is a hexadecimal digit and k is not
– Old test case specifications can be eliminated if they are less specific than the newly generated cases.
• TC-POST3-1 encoded contains CGI hexadecimals • TC-POST5-1 encoded contains erroneous CGI hexadecimals
234
![Page 235: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/235.jpg)
Catalog Entries: the OthersCatalog Entries: the Others
W l i hWe can apply in the same ways.
• rangeg• numeric constant• non-numeric constant• sequence• sequence• scan
235
![Page 236: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/236.jpg)
Summary of Generated Test CasesSummary of Generated Test Cases
TC-POST2-1: encoded contains + TC-DEF2-26: encoded contains %x`TC POST2 1: encoded contains + TC-POST2-2: encoded does not contain +TC-POST3-2: encoded does not contain a CGI-hexadecimal TC-POST5-2: encoded terminated with %xTC-VAR1-1: encoded is the empty sequenceTC-VAR1-2: encoded a sequence containing a single characterTC-VAR1-3: encoded is a very long sequence
TC DEF2 26: encoded contains %xTC-DEF2-27: encoded contains %xaTC-DEF2-28: encoded contains %xy (y in [b..e])TC-DEF2-29: encoded contains %xfTC-DEF2-30: encoded contains %xgTC-DEF2-31: encoded terminates with %TC-DEF2-32: encoded contains %xyzy g q
TC-DEF2-1: encoded contains %/yTC-DEF2-2: encoded contains %0yTC-DEF2-3: encoded contains '%xy' (x in [1..8]) TC-DEF2-4: encoded contains '%9y'TC-DEF2-5: encoded contains '%:y'TC-DEF2-6: encoded contains '%@y‘
d d i
yTC-DEF3-1: encoded contains /TC-DEF3-2: encoded contains 0TC-DEF3-3: encoded contains c in [1..8]TC-DEF3-4: encoded contains 9TC-DEF3-5: encoded contains :TC-DEF3-6: encoded contains @
d d iTC-DEF2-7: encoded contains '%Ay' TC-DEF2-8: encoded contains '%xy' (x in [B..E])TC-DEF2-9: encoded contains '%Fy'TC-DEF2-10: encoded contains '%Gy‘TC-DEF2-11: encoded contains %`y'TC-DEF2-12: encoded contains %ayTC DEF2 13: encoded contains %xy (x in [b e])
TC-DEF3-7: encoded contains ATC-DEF3-8: encoded contains c in[B..Y]TC-DEF3-9: encoded contains ZTC-DEF3-10: encoded contains [TC-DEF3-11: encoded contains`TC-DEF3-12: encoded contains aTC DEF3 13: encoded contains c in [b y]TC-DEF2-13: encoded contains %xy (x in [b..e])
TC-DEF2-14: encoded contains %fy'TC-DEF2-15: encoded contains %gyTC-DEF2-16: encoded contains %x/TC-DEF2-17: encoded contains %x0TC-DEF2-18: encoded contains %xy (y in [1..8])TC-DEF2-19: encoded contains %x9
TC-DEF3-13: encoded contains c in [b..y]TC-DEF3-14: encoded contains zTC-DEF3-15: encoded contains {TC-OP1-1: encoded starts with an alphanumeric characterTC-OP1-2: encoded starts with +TC-OP1-3: encoded starts with %xyTC-OP1-4: encoded terminates with an alphanumeric characterTC DEF2 19: encoded contains %x9
TC-DEF2-20: encoded contains %x:TC-DEF2-21: encoded contains %x@TC-DEF2-22: encoded contains %xATC-DEF2-23: encoded contains %xy(y in [B..E])TC-DEF2-24: encoded contains %xFTC-DEF2-25: encoded contains %xG
TC OP1 4: encoded terminates with an alphanumeric characterTC-OP1-5: encoded terminates with +TC-OP1-6: encoded terminated with %xyTC-OP1-7: encoded contains two consecutive alphanumeric charactersTC-OP1-8: encoded contains ++TC-OP1-9: encoded contains %xy%zwTC-OP1-10: encoded contains %x%yzy
236
![Page 237: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/237.jpg)
What Have We Got from Three Methods?What Have We Got from Three Methods?
F i i i• From category partition testing: – Division into a (manual) step of identifying categories and values, with
constraints, and an (automated) step of generating combinations
• From catalog-based testing: – Improving the manual step by recording and using standard patterns for
id if i i ifi lidentifying significant values
• From pairwise testing: – Systematic generation of smaller test suites
• Three ideas can be combined.
237
![Page 238: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/238.jpg)
SummarySummary
R i ifi i i ll b i i h f f l• Requirements specifications typically begin in the form of natural language statements.
– But, flexibility and expressiveness of natural language is an obstacle to t ti l iautomatic analysis.
• Combinatorial approaches to functional testing consist of – A manual step of structuring specifications into set of properties – An automatic(-able) step of producing combinations of choices
• Brute force synthesis of test cases is tedious and error prone.– Combinatorial approaches decompose brute force work into steps to attack
the problem incrementally by separating analysis and synthesis activities that can be quantified and monitored, and partially supported by tools.
238
![Page 239: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/239.jpg)
239
![Page 240: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/240.jpg)
Chapter 12Chapter 12. Structural Testing
![Page 241: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/241.jpg)
Learning ObjectivesLearning Objectives
U d d i l f l i• Understand rationale for structural testing – How structural testing complements functional testing
• Recognize and distinguish basic terms such as adequacy and coverage• Recognize and distinguish characteristics of common structural criteria• Understand practical uses and limitations of structural testingUnderstand practical uses and limitations of structural testing
241
![Page 242: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/242.jpg)
Structural TestingStructural Testing
J d i i h h b d h f h• Judging test suite thoroughness based on the structure of the programitself
– Also known as hi b i• White-box testing
• Glass-box testing • Code-based testing
Distinguish from functional (requirements based “black box”) testing– Distinguish from functional (requirements-based, black-box ) testing
St t l t ti i till t ti d t f ti lit i t it• Structural testing is still testing product functionality against its specification.
– Only the measure of thoroughness has changed.
242
![Page 243: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/243.jpg)
Rationale of Structural TestingRationale of Structural Testing
O f i h i “Wh i i i i i ?”• One way of answering the question “What is missing in our test suite?”– If a part of a program is not executed by any test case in the suite, faults in
that part cannot be exposed.B t h t’ th ‘ t’?– But what’s the ‘part’?
• Typically, a control flow element or combination• Statements (or CFG nodes), Branches (or CFG edges)• Fragments and combinations: Conditions paths• Fragments and combinations: Conditions, paths
• Structural testing complements functional testing.A h i h d diff l– Another way to recognize cases that are treated differently
• Recalling fundamental rationale– Prefer test cases that are treated differently over cases treated the same
243
![Page 244: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/244.jpg)
No GuaranteeNo Guarantee
E i ll l fl l d fi di ll f l• Executing all control flow elements does not guarantee finding all faults.– Execution of a faulty statement may not always result in a failure.
• The state may not be corrupted when the statement is executed with some data valuesvalues.
• Corrupt state may not propagate through execution to eventually lead to failure.
• What is the value of structural coverage?– Increases confidence in thoroughness of testing
244
![Page 245: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/245.jpg)
Structural Testing Complements Functional Testing
C l fl b d i i l d h b id ifi d• Control flow-based testing includes cases that may not be identified from specifications alone.
– Typical case: Implementation of a single item of the specification by multiple t f thparts of the program
– E.g. Hash table collision (invisible in interface specification)
i h i f l fl d i i ld f il i• Test suites that satisfy control flow adequacy criteria could fail in revealing faults that can be caught with functional criteria.
– Typical case: Missing path faults
245
![Page 246: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/246.jpg)
Structural Testing in PracticeStructural Testing, in Practice
C f i l i fi h l• Create functional test suite first, then measure structural coverage to identify and see what is missing.
• Interpret unexecuted elements– May be due to natural differences between specification and implementation– May reveal flaws of the software or its development processy p p
• Inadequacy of specifications that do not include cases present in the implementation
• Coding practice that radically diverges from the specificationI d t f ti l t t it• Inadequate functional test suites
• Attractive because structural testing is automatedC i i di– Coverage measurements are convenient progress indicators.
– Sometimes used as a criterion of completion of testing • Use with caution: does not ensure effective test suites
246
![Page 247: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/247.jpg)
An Example Program: ‘cgi decode’ and CFGAn Example Program: cgi_decode and CFG
1 #include “hex values h”1. #include hex_values.h
2. int cgi_decode(char* encoded, char* *decoded) {3. char *eptr = encoded;4. char *dptr = decoded;5. int ok = 0;
6 while (*eptr) {
{char *eptr = encoded;char *dptr = decoded;int ok = 0;
int cgi_decode(char *encoded, char *decoded)
A
6. while (*eptr) {7. char c;8. c = *eptr;
9. if (c == ‘+’) {10. *dptr = ‘ ‘;11. } else if (c = ‘%’) {
char c;c = *eptr;if (c == '+') {
while (*eptr) {TrueFalse
C
B
12. int digit_high = Hex_Values[*(++eptr)];13. int digit_low = Hex_Values[*(++eptr)];
14. if (digit_high == -1 || digit_low == -1) {15. ok = 1;16. } else {17 *dptr = 16 * digit high + digit low;
*dptr = ' ';}
True
int digit_high = Hex_Values[*(++eptr)];int digit low = Hex Values[*(++eptr)];
True
False
False
elseif (c == '%') {
else*dptr = *eptr;
D E
F G17. dptr = 16 digit_high + digit_low;18. }19. } else {20. *dptr = *eptr;21. }22. ++dptr;23. ++eptr;
int digit_low = Hex_Values[ (++eptr)];if (digit_high == -1 || digit_low == -1) {
ok = 1;}
True
else {*dptr = 16 * digit_high + digit_low;}
False
dptr = eptr;}
H I
p24. }
25. *dptr = ‘\0’;26. return ok;27. }
*dptr = '\0';return ok;}
++dptr;++eptr;}
LM
247
![Page 248: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/248.jpg)
Structural Testing TechniquesStructural Testing Techniques
1 S T i1. Statement Testing
2. Branch Testingg
3. Condition Testing– Basic– Basic– Compounded– MC/DC
4. Path Testing– Bounded interior– Loop boundary– LCSAJ– Cyclomatic
248
![Page 249: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/249.jpg)
1 Statement Testing1. Statement Testing
Ad i i• Adequacy criterion: – Each statement (or node in the CFG) must be executed at least once.
• Coverage:number of executed statements
number of statementsnumber of statements
• Rationale: f l i l b l d b i h f l– A fault in a statement can only be revealed by executing the faulty statement.
• Nodes in a CFG often represent basic blocks of multiple statements.– Some standards refer to ‘basic block coverage’ or ‘node coverage’.– Difference in granularity, but not in concept
249
![Page 250: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/250.jpg)
An Example: for Function “cgi decode”An Example: for Function cgi_decode
< Test cases >T0 = {“”, “test”, “test+case%1Dadequacy”}17/18 94% St t t17/18 = 94% Statement coverage
T1 = {“adequate+test%0Dexecution%7U”}{ q }18/18 = 100% Statement coverage
T2 = {“%3D”, “%A”, “a+b”, “test”}18/18 100% Statement coverage18/18 = 100% Statement coverage
T3 = {“ ”, “+%0D+%4J”}…
T4 = {“first+test%9Ktest%K9”}…
250
![Page 251: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/251.jpg)
Coverage is not a Matter of SizeCoverage is not a Matter of Size
C d d d h b f• Coverage does not depend on the number of test cases.– T0 , T1 : T1 >coverage T0 T1 <cardinality T0
– T1 , T2 : T2 =coverage T1 T2 >cardinality T1
• Minimizing test suite size is not the goal• Minimizing test suite size is not the goal.– Small test cases make failure diagnosis easier.– But, a failing test case in T2 gives more information for fault localization than
a failing test case in Ta failing test case in T1
251
![Page 252: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/252.jpg)
Complete Statement CoverageComplete Statement Coverage
C l int cgi decode(char *encoded char *decoded)• Complete statement coverage may not imply executing all branches in a program.
{char *eptr = encoded;char *dptr = decoded;int ok = 0;
while (*eptr) {
int cgi_decode(char encoded, char decoded)
A
B
• Example: – Suppose block F were missing
char c;c = *eptr;if (c == '+') {
*dptr = ' ';
while ( eptr) {
TrueFalse
TrueFalseelseif (c == '%') {
C
B
D E
– But, statement adequacy would not require false branch from D to L
dptr = ;}
int digit_high = Hex_Values[*(++eptr)];int digit_low = Hex_Values[*(++eptr)];if (digit_high == -1 || digit_low == -1) {
True
TrueFalse
False
elseif (c == % ) {
else {*dptr = *eptr;}
D E
F G
• T3 = {“ ”, “+%0D+%4J”}– 100% statement coverage
ok = 1;}
True
else {*dptr = 16 * digit_high + digit_low;}
False
H I
g– No false branch from D *dptr = '\0';
return ok;}
++dptr;++eptr;}
LM
252
![Page 253: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/253.jpg)
2 Branch Testing2. Branch Testing
Ad it i• Adequacy criterion: – Each branch (edge in the CFG) must be executed at least once.
Coverage:• Coverage:number of executed branches
number of branches
• Example:– T3 = {“”, “+%0D+%4J”}
100% St t C• 100% Stmt Cov.• 88% Branch Cov. (7/8 branches)
– T2 = {“%3D”, “%A”, “a+b”, “test”}• 100% Stmt Cov. • 100% Branch Cov. (8/8 branches)
253
![Page 254: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/254.jpg)
Statements vs BranchesStatements vs. Branches
T i ll d ll d b i i d• Traversing all edges causes all nodes to be visited.– Therefore, test suites that satisfy the branch adequacy also satisfy the
statement adequacy criterion for the same program.B h d b t t t d– Branch adequacy subsumes statement adequacy.
• The converse is not true (see T3)– A statement-adequate test suite may not be branch-adequate.
254
![Page 255: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/255.jpg)
All Branches CoverageAll Branches Coverage
“All b h ” ill i di i• “All branches coverage” can still miss conditions.
• Example: p– Supposed that we missed the negation operator of “digit_high == -1”
digit_high == 1 || digit_low == -1
• Branch adequacy criterion can be satisfied by varying only ‘digit_low’.The faulty sub expression might never determine the result– The faulty sub-expression might never determine the result.
– We might never really test the faulty condition, even though we tested both outcomes of the branch.
255
![Page 256: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/256.jpg)
3 Condition Testing3. Condition Testing
B h f lt i h t ti h b• Branch coverage exposes faults in how a computation has been decomposed into cases.
– Intuitively attractive: checking the programmer’s case analysisBut only roughly: grouping cases with the same outcome– But, only roughly: grouping cases with the same outcome
• Condition coverage considers case analysis in more detail.– Consider ‘individual conditions’ in a compound Boolean expressionConsider individual conditions in a compound Boolean expression
• E.g. both parts of ‘”igit_high == 1 || digit_low == -1”
• Adequacy criterion:Adequacy criterion: – Each basic condition must be executed at least once.
• Basic condition testing coverage:as c co d t o test g co e age:number of truth values taken by all basic conditions
2 * number of basic conditions
256
![Page 257: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/257.jpg)
Basic Conditions vs BranchesBasic Conditions vs. Branches
B i di i d i i b i fi d i h i f i• Basic condition adequacy criterion can be satisfied without satisfying branch coverage.
• T4 = {“first+test%9Ktest%K9”}– Satisfies basic condition adequacy– But, does not satisfy branch condition adequacyy q y
• Branch and basic condition are not comparable. – Neither implies the other– Neither implies the other.
257
![Page 258: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/258.jpg)
Covering Branches and ConditionsCovering Branches and Conditions
B h d diti d• Branch and condition adequacy: – Cover all conditions and all decisions
Compound condition adequacy:• Compound condition adequacy:– Cover all possible evaluations of compound conditions.– Cover all branches of a decision tree.
truefalse truefalse
258
![Page 259: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/259.jpg)
Compounded ConditionsCompounded Conditions
C d di i f h i l l i• Compound conditions often have exponential complexity.
• Example: (((a || b) && c) || d) && ep || ||
259
![Page 260: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/260.jpg)
Modified Condition/Decision (MC/DC)Modified Condition/Decision (MC/DC)
M ti ti• Motivation– Effectively test important combinations of conditions, without exponential
blowup in test suite size – “Important” combinations means:Important combinations means:
• Each basic condition shown to independently affect the outcome of each decision
• Requires– For each basic condition C, two test cases,– Values of all ‘evaluated’ conditions except C are the same.– Compound condition as a whole evaluates to ‘true’ for one and ‘false’ for the
thother.
260
![Page 261: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/261.jpg)
Complexity of MC/DCComplexity of MC/DC
MC/DC h li l i• MC/DC has a linear complexity.
• Example: (((a || b) && c) || d) && ep || ||
Test a b c d e outcomeCase(1) true -- true -- true true(2) false true true -- true true(3) true -- false true true true(6) true true false false(6) true -- true -- false false(11) true -- false false -- false(13) false false -- false -- false
• Underlined values independently affect the output of the decision.– Required by the RTCA/DO-178B standard
261
![Page 262: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/262.jpg)
Comments on MC/DCComments on MC/DC
MC/DC i• MC/DC is – Basic condition coverage (C)– Branch coverage (DC)– Plus one additional condition (M)
• Every condition must independently affect the decision’s output.
I i b d b d di i d b ll h i i• It is subsumed by compound conditions and subsumes all other criteria discussed so far.
– Stronger than statement and branch coverage
• A good balance of thoroughness and test size – Widely used
262
![Page 263: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/263.jpg)
4 Path Testing4. Path Testing
Th h h b h• There are many more paths than branches.– Decision and condition adequacy criteria consider individual decisions only.
• Path testing focuses combinations of decisions along paths.
• Adequacy criterion:Adequacy criterion: – Each path must be executed at least once.
• Coverage:• Coverage:number of executed paths
number of paths
263
![Page 264: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/264.jpg)
Path Coverage Criteria in PracticePath Coverage Criteria in Practice
Th b f h i i h l i b d d• The number of paths in a program with loops is unbounded. – Usually impossible to satisfy
• For a feasible criterion, – Should partition infinite set of paths into a finite number of classes
• Useful criteria can be obtained by limiting – Number of traversals of loops
Length of the paths to be traversed– Length of the paths to be traversed– Dependencies among selected paths
264
![Page 265: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/265.jpg)
Boundary Interior Path TestingBoundary Interior Path Testing
G h h h diff l i h b h h f ll h• Group together paths that differ only in the subpath they follow, when repeating the body of a loop
– Follow each path in the CFG up to the first repeated node– The set of paths from the root of the tree to each leaf is the required set of
subpaths for boundary interior coverage.
265Paths for boundary interior path testingPaths derived from the CFG
![Page 266: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/266.jpg)
Limitations of Boundary Interior AdequacyLimitations of Boundary Interior Adequacy
Th b f h ill i ll• The number of paths can still grow exponentially
• The subpaths through this control flow can include or if (a) { S1; }if (b) { S2; }if (c) { S3; }
p gexclude each of the statements Si, so that in total N branches result in 2N paths that must be traversed.
...if (x) { Sn; } • Choosing input data to force execution of one particular
path may be very difficult, or even impossible if the conditions are not independentconditions are not independent.
266
![Page 267: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/267.jpg)
Loop Boundary AdequacyLoop Boundary Adequacy
V i f h b d /i i i i• Variant of the boundary/interior criterion– Treats loop boundaries similarly, but is less stringent with respect to other
differences among paths.
• Criterion: – A test suite satisfies the loop boundary adequacy criterion iff, for every loop:
I l h l b d i i d i• In at least one test case, the loop body is iterated zero times.• In at least one test case, the loop body is iterated once.• In at least one test case, the loop body is iterated more than once.
• Corresponds to the cases that would be considered in a formal correctness proof for the loop
267
![Page 268: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/268.jpg)
LCSAJ AdequacyLCSAJ Adequacy
Li C d S A d J (LCSAJ)• Linear Code Sequence And Jumps (LCSAJ)– Sequential subpath in the CFG starting and ending in a branch
• TER1 = statement coverageTER b h• TER2 = branch coverage
• TERn+2 = coverage of n consecutive LCSAJs
– Essentially considering full path coverage of (short) sequences of decisions
• Data flow criteria considered in a later chapter provide a more i i l d f h i ti l b th i t tprincipled way of choosing some particular sub-paths as important
enough to cover in testing. – But, neither LCSAJ nor data flow criteria are much used in current practice.
268
![Page 269: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/269.jpg)
Cyclomatic AdequacyCyclomatic Adequacy
C l i b• Cyclomatic number– Number of independent paths in the CFG– A path is representable as a bit vector, where each component of the vector
t drepresents an edge.– “Dependence” is ordinary linear dependence between (bit) vectors
• If e = #edges, n = #nodes, c = #connected components of a graph, – e - n + c for an arbitrary graph– e - n + 2 for a CFG ← Cyclomatic complexityy p y
• Cyclomatic coverage counts the number of independent paths that have been exercised, relative to cyclomatic complexitybeen exercised, relative to cyclomatic complexity.
269
![Page 270: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/270.jpg)
Procedure Call TestingProcedure Call Testing
M f l fl i hi i di id l d• Measure coverage of control flow within individual procedures. – Not well suited to integration or system testing
• Choose a coverage granularity commensurate with the granularity of testing
– If unit testing has been effective, then faults that remain to be found in integration testing will be primarily interface faults, and testing effort should focus on interfaces between units rather than their internal details.
• Procedure entry and exit testing– Procedure may have multiple entry points (e.g., Fortran) and multiple exit
points.
• Call coverage– The same entry point may be called from many points.
270
![Page 271: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/271.jpg)
Comparing Structural Testing CriteriaComparing Structural Testing Criteria
271
Subsumption Relation among Structural Test Adequacy Criteria
![Page 272: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/272.jpg)
The Infeasibility ProblemThe Infeasibility Problem
S i i i b i fi bl• Sometimes criteria may not be satisfiable.– The criterion requires execution of
• Statements that cannot be executed as a result of – Defensive programming – Code reuse (reusing code that is more general than strictly required for the
application)
• Conditions that cannot be satisfied as a result of• Conditions that cannot be satisfied as a result of – Interdependent conditions
• Paths that cannot be executed as a result of – Interdependent decisionsInterdependent decisions
272
![Page 273: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/273.jpg)
Satisfying Structural CriteriaSatisfying Structural Criteria
L f ‘f il’ d i di i i i bili• Large amounts of ‘fossil’ code may indicate serious maintainability problems.
• But, some unreachable code is common even in well-designed and well-maintained systems.
• Solutions:1. Make allowances by setting a coverage goal less than 100%2. Require justification of elements left uncovered
• As RTCA-DO-178B and EUROCAE ED-12B for modified MC/DC
273
![Page 274: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/274.jpg)
SummarySummary
W d fi d b f d i i• We defined a number of adequacy criteria. – NOT test design techniques– Different criteria address different classes of errors.
• Full coverage is usually unattainable.– Attainability is an undecidable problem.y p
• Rather than requiring full adequacy the “degree of adequacy” of a testRather than requiring full adequacy, the degree of adequacy of a test suite is estimated by coverage measures.
274
![Page 275: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/275.jpg)
275
![Page 276: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/276.jpg)
Chapter 13Chapter 13. Data Flow Testing
![Page 277: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/277.jpg)
Learning ObjectivesLearning Objectives
U d d h d fl i i h b d i d d d• Understand why data flow criteria have been designed and used• Recognize and distinguish basic DF criteria
– All DU pairs, all DU paths, all definitions
• Understand how the infeasibility problem impacts data flow testing• Appreciate limits and potential practical uses of data flow testingAppreciate limits and potential practical uses of data flow testing
277
![Page 278: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/278.jpg)
MotivationMotivation
Middl d i l i• Middle ground in structural testing– Node and edge coverage don’t test interactions.– Path-based criteria require impractical number of test cases.
• Only a few paths uncover additional faults, anyway.
– Need to distinguish “important” paths
• Intuition: Statements interact through data flow.– Value computed in one statement, is used in another.– Bad value computation can be revealed only when it is used.p y
278
![Page 279: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/279.jpg)
Def Use PairsDef-Use Pairs
V l f 6 ld b• Value of x at 6 could be computed at 1 or at 4.x = ....
1
2 • Bad computation at 1 or 4 could be revealed only if they are used at 6.
if ....
4
2
3
• (1, 6) and (4, 6) are def-use (DU) pairs.
x = .... .... 43
5 pa s.– defs at 1, 4– use at 6
...
y = x +6
y = x + ...
279
![Page 280: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/280.jpg)
TerminologyTerminology
DU i• DU pair– A pair of definition and use for some variable, such that at least one DU path
exists from the definition to the use.“ ” i d fi iti f– “x = ...” is a definition of x
– “= ... x ...” is a use of x
• DU path – A definition-clear path on the CFG starting from a definition to a use of a
same variable– Definition clear: Value is not replaced on path.– Note: Loops could create infinite DU paths between a def and a use.
280
![Page 281: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/281.jpg)
Definition Clear PathDefinition-Clear Path
1 2 3 5 6 i d fi i i l h• 1,2,3,5,6 is a definition-clear path from 1 to 6.
– x is not re-assigned between 1 d 6
x = .... 1
and 6.
• 1,2,4,5,6 is not a definition-clear th f 1 t 6
if .... 2
path from 1 to 6.– the value of x is “killed”
(reassigned) at node 4.x = .... .... 43
5
• (1, 6) is a DU pair because 1,2,3,5,6 is a definition-clear path.
...
6
5
y = x + ... 6
281
![Page 282: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/282.jpg)
Adequacy CriteriaAdequacy Criteria
All DU i• All DU pairs – Each DU pair is exercised by at least one test case.
• All DU paths – Each simple (non looping) DU path is exercised by at least one test case.
• All definitions– For each definition, there is at least one test case which exercises a DU pair
containing it.g– Because, every computed value is used somewhere.
• Corresponding coverage fractions can be defined similarly.Corresponding coverage fractions can be defined similarly.
282
![Page 283: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/283.jpg)
Difficult CasesDifficult Cases
[i] [ j]• x[i] = ... ; ... ; y = x[ j]– DU pair (only) if i==j
• p = &x ; ... ; *p = 99 ; ... ; q = x– *p is an alias of x
• m.putFoo(...); ... ; y=n.getFoo(...); – Are m and n the same object?– Do m and n share a “foo” field?Do m and n share a foo field?
• Problem of aliases: Which references are (always or sometimes) the same?– Which references are (always or sometimes) the same?
283
![Page 284: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/284.jpg)
Data Flow Coverage with Complex StructuresData Flow Coverage with Complex Structures
A d i i i l f d fl l i• Arrays and pointers are critical for data flow analysis.– Under-estimation of aliases may fail to include some DU pairs.– Over-estimation may introduce unfeasible test obligations.
• For testing, it may be preferable to accept under-estimation of alias set rather than over-estimation or expensive analysis.
– Alias analysis may rely on external guidance or other global analysis to calculate good estimates.
– Undisciplined use of dynamic storage, pointer arithmetic, etc. may make the h l l i i f iblwhole analysis infeasible.
– But, in other applications (e.g., compilers), a conservative over-estimation of aliases is usually required.
284
![Page 285: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/285.jpg)
The Infeasibility Problem Why? – I don’t Know!The Infeasibility Problem
S ‘ d’ h h d
Why? I don t Know!
• Suppose ‘cond’ has not changed between 1 and 5.
– Or the conditions could be diff t b t th fi t i li th
if (cond)1
different, but the first implies the second.
Then (3 5) is not a (feasible) DU
x = .... .... 32
• Then (3,5) is not a (feasible) DU pair.
– But it is difficult or impossible to determine which pairs are
...4
5 determine which pairs are infeasible.
• Infeasible test obligations are a
if (cond)
6
5
7 • Infeasible test obligations are a problem.
– No test case can cover them.
y = x + ... .... 6 7
285
![Page 286: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/286.jpg)
Data Flow Coverage in PracticeData Flow Coverage in Practice
Th h i d f d fl l i k h i f ibili• The path-oriented nature of data flow analysis makes the infeasibility problem especially relevant.
– Combinations of elements matter.– Impossible to (infallibly) distinguish feasible from infeasible paths. – More paths = More work to check manually
• In practice, reasonable coverage is (often, not always) achievable.– Number of paths is exponential in worst case, but often linear.– All DU paths is more often impractical.
286
![Page 287: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/287.jpg)
SummarySummary
D fl i di i i h “i ” h I i• Data flow testing attempts to distinguish “important” paths: Interactions between statements.
– Intermediate between simple statement and branch coverage and more i th b d t t l t tiexpensive path-based structural testing
• Cover Def-Use (DU) pairs: From computation of value to its use– Intuition: Bad computed value is revealed only when it is used.– Levels: All DU pairs, all DU paths, all defs (some use)
• Limits: Aliases, infeasible paths– Worst case is bad (undecidable properties, exponential blowup of paths), so
pragmatic compromises are required.
287
![Page 288: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/288.jpg)
288
![Page 289: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/289.jpg)
Chapter 14Chapter 14. Model-Based Testing
![Page 290: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/290.jpg)
Learning ObjectivesLearning Objectives
U d d h l f d l i d i i• Understand the role of models in devising test cases– Principles underlying functional and structural test adequacy criteria, as well
as model-based testing
• Understand some examples of model-based testing techniques• Be able to understand, devise and refine other model-based testing
techniques
290
![Page 291: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/291.jpg)
y ntly
e s
Iden
tify
Inde
pend
eTe
stab
leFe
atur
es
Generate Test-C
Specifica erate Test-Case
ifications
t-Case
cations Genera
Specifi
Gen
erat
e Te
st C
ases
Inst
antia
teTe
sts
291
![Page 292: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/292.jpg)
OverviewOverview
M d l d i ifi i d i h• Models used in specification or design have structure.– Useful information for selecting representative classes of behavior – Behaviors that are treated differently with respect to the model should be
t i d b th h t t ittried by a thorough test suite.– In combinatorial testing, it is difficult to capture that structure clearly and
correctly in constraints.
• We can devise test cases to check actual behavior against behavior specified by the model.
“C ” i il l i b li d ifi i d– “Coverage” similar to structural testing, but applied to specification and design models
292
![Page 293: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/293.jpg)
Deriving Test Cases from Finite State MachinesDeriving Test Cases from Finite State Machines
Informal S ifi ti
FSM Test CasesSpecification
FSM Test Cases
293
![Page 294: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/294.jpg)
Informal Specification: Feature “Maintenance” of the Chipmunk Web Site
M i t Th M i t f ti d th hi t f it d i i tMaintenance: The Maintenance function records the history of items undergoing maintenance.
If the product is covered by warranty or maintenance contract, maintenance can be requested either by calling the maintenance toll free number, or through the web site, or by bringing the item to a d i t d i t t tidesignated maintenance station.If the maintenance is requested by phone or web site and the customer is a US or EU resident, the item is picked up at the customer site, otherwise, the customer shall ship the item with an express courier.If the maintenance contract number provided by the customer is not valid, the item follows the
d f i d bprocedure for items not covered by warranty.If the product is not covered by warranty or maintenance contract, maintenance can be requested only by bringing the item to a maintenance station. The maintenance station informs the customer of the estimated costs for repair. Maintenance starts only when the customer accepts the estimate. If the customer does not accept the estimate, the product is returned to the customer.Small problems can be repaired directly at the maintenance station. If the maintenance station cannot solve the problem, the product is sent to the maintenance regional headquarters (if in US or EU) or to the maintenance main headquarters (otherwise).If the maintenance regional headquarters cannot solve the problem, the product is sent to the maintenance main headquarters.Maintenance is suspended if some components are not available.Once repaired, the product is returned to the customer.p p
294
![Page 295: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/295.jpg)
Corresponding Finite State MachineCorresponding Finite State MachineNO
Maintenance
0
request at
maintenance station
(no warranty)
request by phone or web
[US or EU resident]
(contract number)t at
e st
atio
ns
cour
ier
umbe
r)
pick up return
1 2 Maintenance(no warranty)
estim
ate
cost
s
)
Wait forpick up
pick up
requ
est
mai
nten
ance
or b
y ex
pres
s(c
ontra
ct n
u
Wait for returning
reject esti
invalidcontractnumber
1 2 3
eRepair
(maintenance station)
Wait for acceptance
acceptestimate
timate
Repairedrepair completed
air
una
(USt (a)
4 5 6
Repair
successf
ul repairable to repair
US or EU resident)
sful r
epair
Wait for
lack co
mponent (
componentarrives (a)
7 8(regional
headquarters)
succ
essf
unable trepair
Wait for component lack component (b)
lack compon
componentarrives (b)
unable to repair(not US or EU resident)
295Repair(main
headquarters)
to
ponent (c)componentarrives (c)
9
![Page 296: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/296.jpg)
Test Cases Generated from the FSMTest Cases Generated from the FSM
FSM b d b h• FSM can be used both to1. Guide test selection (checking each state transition)2. Constructing an oracle that judge whether each observed behavior is correct
TC1 0 2 4 1 0
TC2 0 5 2 4 5 6 0TC2 0 5 2 4 5 6 0
TC3 0 3 5 9 6 0
TC4 0 3 5 7 5 8 7 8 9 6 0
• Questions:
TC4 0 3 5 7 5 8 7 8 9 6 0
– Is this a thorough test suite? – How can we judge? Coverage criteria require.
296
![Page 297: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/297.jpg)
Transition Coverage CriteriaTransition Coverage Criteria
S• State coverage– Every state in the model should be visited by at least one test case.
• Transition coverage– Every transition between states should be traversed by at least one test case. – Most commonly used criterion– A transition can be thought of as a (precondition, postcondition) pair
297
![Page 298: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/298.jpg)
Path Sensitive CriteriaPath Sensitive Criteria
B i i S f ll i hi• Basic assumption: States fully summarize history.– No distinction based on how we reached a state – But, this should be true of well-designed state machine models.
• If the assumption is violated, we may distinguish paths and devise criteria to cover them
– Single state path coverage: • Traverse each subpath that reaches each state at most once
– Single transition path coverage: • Traverse each subpath that reaches each transition at most once
– Boundary interior loop coverage: • Each distinct loop of the state machine must be exercised the minimum, an
intermediate and the maximum or a large number of timesintermediate, and the maximum or a large number of times• Of the path sensitive criteria, only boundary-interior is common.
298
![Page 299: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/299.jpg)
Deriving Test Cases from Decision StructuresDeriving Test Cases from Decision Structures
S ifi i d d i i bl d i i• Some specifications are structured as decision tables, decision trees, or flow charts.
• We can exercise these as if they were program source code.
Informal Specification
Decision Structures
Test Cases
299
![Page 300: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/300.jpg)
Informal Specification: Feature “Price” of the Chipmunk Web Site
Pricing: The pricing function determines the adjusted price of a configuration for aPricing: The pricing function determines the adjusted price of a configuration for a particular customer.
The scheduled price of a configuration is the sum of the scheduled price of the model and the scheduled price of each component in the configuration The adjustedmodel and the scheduled price of each component in the configuration. The adjusted price is either the scheduled price, if no discounts are applicable, or the scheduled price less any applicable discounts.There are three price schedules and three corresponding discount schedules, Business, Educational and IndividualEducational, and Individual.….
Educational prices: The adjusted price for a purchase charged to an educational i d di i h h d l d i f h d i l i h d laccount in good standing is the scheduled price from the educational price schedule.
No further discounts apply.…
Special-price non-discountable offers: Sometimes a complete configuration is offered at a special, non-discountable price. When a special, non-discountable price is available for a configuration, the adjusted price is the non-discountable price or the regular price after any applicable discounts, whichever is less.
300
![Page 301: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/301.jpg)
Corresponding Decision TableCorresponding Decision Table
Education IndividualEducation Individual
EduAc T T F F F F F F
BusAc - - F F F F F F
CP > CT1 - - F F T T - -CP CT1 F F T T
YP > YT1 - - - - - - - -
CP > CT2 - - - - F F T T
YP > YT2 - - - - - - - -
SP < Sc F T F T - - - -
SP < T1 - - - - F T - -
SP < T2 - - - - - - F T
Out Edu SP ND SP T1 SP T2 SP
…
ConstraintsConstraintsat-most-one (EduAc, BusAc) at-most-one (YP < YT1, YP > YT2) YP > YT2 → YP > YT1 at-most-one (CP < CT1, CP > CT2)CP > CT2 → CP > CT1 at-most-one (SP < T1, SP > T2SP > T2 → SP > T1
301
SP > T2 → SP > T1
![Page 302: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/302.jpg)
Test Cases Generated from the Decision TableTest Cases Generated from the Decision Table
B i di i• Basic condition coverage– A test case specification for each column in the table
• Compound condition adequacy criterion– A test case specification for each combination of truth values of basic
conditions
• Modified condition/decision adequacy criterion (MC/DC)– Each column in the table represents a test case specification.p p– We add columns that differ in one input row and in outcome, then merge
compatible columns.
302
![Page 303: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/303.jpg)
A Part of Test Cases by Applying MC/DCA Part of Test Cases by Applying MC/DC
C.1 C.1a C.1b C.10Generate C.1a and
C.1b by flipping one element of C 1EduAc T F T -
BusAc - - - T
CP > CT1 - - - F
element of C.1.
C.1b can be merged ith i ti
YP > YT1 - - - F
CP > CT2 - - - -
YP > YT2 - - - -
with an existing column (C.10) in the
specification. (?)
YP > YT2 - - - -
SP > Sc F F T T
SP > T1 - - - -
SP T2
Outcome of generated columns must differ from source columnSP > T2 - - - -
Out Edu * * SP
source column
303
![Page 304: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/304.jpg)
Deriving Test Cases from Control and Data Flow Graph
If h ifi i d l h b h d i i d i l l i• If the specification or model has both decisions and sequential logic, we can cover it like program source code.
• Flowgraph based testing
Informal Specification
Flowgraph Test Cases
304
![Page 305: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/305.jpg)
Informal Specification: Feature “Process Shipping Order” of the Chipmunk Web Site
Process shipping order: The Process shipping order function checks the validity of orders and prepares the receiptProcess shipping order: The Process shipping order function checks the validity of orders and prepares the receipt.
A valid order contains the following data:
cost of goods: If the cost of goods is less than the minimum processable order (MinOrder) then the order is invalid.g g p ( )shipping address: The address includes name, address, city, postal code, and country.preferred shipping method: If the address is domestic, the shipping method must be either land freight, expedited land freight, or overnight air; If the address is international, the shipping method must be either air freight, or expedited air freight.
type of customer which can be individual, business, educationalpreferred method of payment. Individual customers can use only credit cards, business and educational customers can choose between credit card and invoice
card information: if the method of payment is credit card, fields credit card number, name on card, expiration date, and billing address, if different than shipping address, must be provided. If credit card information is not valid the user can either provide new data or abort the order.
The outputs of Process shipping order areThe outputs of Process shipping order arevalidity: Validity is a boolean output which indicates whether the order can be processed.total charge: The total charge is the sum of the value of goods and the computed shipping costs (only if validity = true). payment status: if all data are processed correctly and the credit card information is valid or the payment is invoice, payment status is set to valid, the order is entered and a receipt is prepared; otherwise validity = false.p p p y
305
![Page 306: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/306.jpg)
Corresponding Control Flow GraphCorresponding Control Flow GraphProcess shipping order
preferred shipping method = land freight,OR expedited land freight OR overnight air
CostOfGoods < MinOrder
shipping address
no
domestic
preferred shipping method = airfreight OR expedited air freight
international
OR expedited land freight OR overnight airfreight OR expedited air freight
calculate domestic shipping chargecalculate international shipping charge
total charge = goods + shipping
yes
individual customer no
yes
bt i dit d d t b
method of payement
credit card
nono
obtain credit card data: number, nameon card, expiration date invoice
billing address = shipping address
no
yes
obtain billing address
valid credit cardinformation
no
yes
t t t lidno
306
payement status = validenter order
prepare receipt
invalid order
abort order?
yes
![Page 307: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/307.jpg)
Test Cases Generated from the CFGTest Cases Generated from the CFG
N d d i i• Node adequacy criteria
Case Too Small Ship Where Ship Method Cust Type Pay
Method Same Address CC valid
• Branch adequacy criteria
TC-1 No Int Air Bus CC No Yes
TC-2 No Dom Air Ind CC - No (abort)
• Branch adequacy criteria
Case Too Small Ship Where Ship Method Cust Type Pay
Method Same Address CC valid
TC 1 N I t Ai B CC N YTC-1 No Int Air Bus CC No Yes
TC-2 No Dom Land - - - -
TC-3 Yes - - - - - -
TC 4 No Dom AirTC-4 No Dom Air - - - -
TC-5 No Int Land - - - -
TC-6 No - - Edu Inv - -
TC-7 No - - - CC Yes -
307
TC 7 No CC Yes
TC-8 No - - - CC - No (abort)
TC-9 No - - - CC - No (no abort)
![Page 308: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/308.jpg)
Deriving Test Cases from GrammarsDeriving Test Cases from Grammars
G d• Grammars are good at:– Representing inputs of varying and unbounded size– With recursive structure and boundary conditions
• Examples: – Complex textual inputsp p– Trees (search trees, parse trees, ... )
• Example: XML and HTMl are trees in textual form
– Program structuresg• Which are also tree structures in textual format
Informal Specification
Grammar Test Cases
308
![Page 309: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/309.jpg)
Grammar Based TestingGrammar-Based Testing
T ‘ i ’ d f h• Test cases are ‘strings’ generated from the grammar
• Coverage criteria:g– Production coverage:
• Each production must be used to generate at least one (section of) test case.
– Boundary condition: • Annotate each recursive production with minimum and maximum number of
application, then generate:Mi i– Minimum
– Minimum + 1– Maximum - 1– MaximumMaximum
309
![Page 310: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/310.jpg)
Informal Specification: Feature “Check Configuration” of the Chipmunk Web Site
Check configuration: The Check configuration function checks the validity of a computer configurationCheck configuration: The Check-configuration function checks the validity of a computer configuration.
Model: A model identifies a specific product and determines a set of constraints on available components. Models are characterized by logical slots for components, which may or may not be implemented by physical slots on a bus. Slots may be required or optional. Required slots must be assigned with a suitable component to obtain a legal configuration while optional slots may be left empty or filledsuitable component to obtain a legal configuration, while optional slots may be left empty or filled depending on the customers' needs
Example: The required ``slots'' of the Chipmunk C20 laptop computer include a screen, a processor, a hard disk, memory, and an operating system. (Of these, only the hard disk and memory are implemented using actual hardware slots on a bus ) The optional slots include external storage devices such as ausing actual hardware slots on a bus.) The optional slots include external storage devices such as a CD/DVD writer.
Set of Components: A set of [slot,component] pairs, which must correspond to the required and optional slots associated with the model. A component is a choice that can be varied within a model, and which is not designed to be replaced by the end user Available components and a default for each slot isnot designed to be replaced by the end user. Available components and a default for each slot is determined by the model. The special value empty is allowed (and may be the default selection) for optional slots. In addition to being compatible or incompatible with a particular model and slot, individual components may be compatible or incompatible with each other.
Example: The default configuration of the Chipmunk C20 includes 20 gigabytes of hard disk; 30 and 40Example: The default configuration of the Chipmunk C20 includes 20 gigabytes of hard disk; 30 and 40 gigabyte disks are also available. (Since the hard disk is a required slot, empty is not an allowed choice.) The default operating system is RodentOS 3.2, personal edition, but RodentOS 3.2 mobile server edition may also be selected. The mobile server edition requires at least 30 gigabytes of hard disk.
310
※ It is not the example in the text.
![Page 311: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/311.jpg)
Corresponding Grammar with LimitsCorresponding Grammar with Limits
Model <Model>::= <modelNumber> <compSequence> <optCompSequence>
compSeq1 [0, 16] <compSequence> ::= <Component> <compSequence>
compSeq2 <compSequence> ::= empty
optCompSeq1 [0, 16] <optCompSequence> ::= <OptionalComponent> <optCompSequence>
optCompSeq2 <optCompSequence> ::= empty
Comp <Component> ::= <ComponentType> <ComponentValue>
OptComp <OptionalComponent> ::= <ComponentType>
modNum <modelNumber> ::= string
CompTyp <ComponentType> ::= string
CompVal <ComponentValue> ::= string
311
![Page 312: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/312.jpg)
Test Cases Generated from the GrammarTest Cases Generated from the Grammar
“M d000”• “Mod000”– Covers Model, compSeq1[0], compSeq2, optCompSeq1[0], optCompSeq2,
modNum
• “Mod000 (Comp000, Val000) (OptComp000)”– Covers Model, compSeq1[1], compSeq2, optCompSeq2[0], optCompSeq2,
C O C dN C T C V lComp, OptComp, modNum, CompTyp, CompVal
• Etc.
• Comments:– By first applying productions with nonterminals on the right side, we obtain
few, large test cases.g– By first applying productions with terminals on the right side, we obtain many,
small test cases.
312
![Page 313: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/313.jpg)
Grammar Testing vs Combinatorial TestingGrammar Testing vs. Combinatorial Testing
C bi i l ifi i b d i i d f “ l• Combinatorial specification-based testing is good for “mostly independent” parameters.
– We can incorporate a few constraints, but complex constraints are hard to t drepresent and use.
– We must often “factor and flatten.”– E.g. separate “set of slots” into characteristics “number of slots” and predicates
about what is in the slots (all together)about what is in the slots (all together)
• Grammar describes sequences and nested structure naturally.B l i diff b diffi l d ib d– But, some relations among different parts may be difficult to describe and exercise systematically, e.g. compatibility of components with slots.
313
![Page 314: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/314.jpg)
SummarySummary
M d l f l b i• Models are useful abstractions.– In specification and design, they help us think and communicate about
complex artifacts by emphasizing key features and suppressing details.M d l t t d h l f thi t ti– Models convey structure and help us focus on one thing at a time.
• We can use them in systematic testing.– If a model divides behavior into classes, we probably want to exercise each of
those classes.– Common model-based testing techniques are based on state machines,
d i i t t ddecision structures, and grammars.– But, we can apply the same approach to other models.
314
![Page 315: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/315.jpg)
315
![Page 316: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/316.jpg)
Chapter 15Chapter 15. Testing Object-Oriented Software
![Page 317: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/317.jpg)
317
![Page 318: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/318.jpg)
Chapter 16Chapter 16. Fault-Based Testing
![Page 319: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/319.jpg)
Learning ObjectivesLearning Objectives
U d d h b i id f f l b d i• Understand the basic ideas of fault-based testing– How knowledge of a fault model can be used to create useful tests and judge
the quality of test casesU d t d th ti l f f lt b d t ti ll h t di ti i h– Understand the rationale of fault-based testing well enough to distinguish between valid and invalid uses
d d l f f l b d• Understand mutation testing as one application of fault-based testing principles
319
![Page 320: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/320.jpg)
Estimating Test Suite QualityEstimating Test Suite Quality
S d h I h i h b• Supposed that I have a program with bugs.
• Add 100 new bugsg– Assume they are exactly like real bugs in every way– I make 100 copies of my program, each with one of my 100 new bugs.
• Run my test suite on the programs with seeded bugs– And the tests revealed 20 of the bugs.
h h i d f il– The other 80 program copies do not fail.
• What can I infer about my test suite’s quality?y y
320
![Page 321: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/321.jpg)
Basic AssumptionsBasic Assumptions
W j d ff i f i i fi di l f l• We want to judge effectiveness of a test suite in finding real faults, – by measuring how well it finds seeded fake faults.
• Valid to the extent that the seeded bugs are representative of real bugs– Not necessarily identical– But, the differences should not affect the selection
321
![Page 322: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/322.jpg)
Mutation TestingMutation Testing
A i f i h i• A mutant is a copy of a program with a mutation.
• A mutation is a syntactic change (a seeded bug).y g g– Example: change (i < 0) to (i <= 0)
• Run test suite on all the mutant programsRun test suite on all the mutant programs• A mutant is killed, if it fails on at least one test case. (The bug is found.)
• If many mutants are killed, infer that the test suite is also effective at finding real bugs.
322
![Page 323: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/323.jpg)
Assumptions on Mutation TestingAssumptions on Mutation Testing
C h h i• Competent programmer hypothesis– Programs are nearly correct.
• Real faults are small variations from the correct program.Th f t t bl d l f l b• Therefore, mutants are reasonable models of real buggy programs.
• Coupling effect hypothesis– Tests that find simple faults also find more complex faults.– Even if mutants are not perfect representatives of real faults, a test suite that
kills mutants is good at finding real faults too.
323
![Page 324: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/324.jpg)
Mutant OperatorsMutant Operators
S i h f l l ill l• Syntactic changes from legal program to illegal program– Specific to each programming language
• Examples: – crp: constant for constant replacementcrp: constant for constant replacement
• E.g. from (x < 5) to (x < 12)• Select constants found somewhere in program text
– ror: relational operator replacementp p• E.g. from (x <= 5) to (x < 5)
– vie: variable initialization elimination• E.g. change int x =5; to int x;
324
![Page 325: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/325.jpg)
Fault based Adequacy CriteriaFault-based Adequacy Criteria
M i l i i f h f ll i• Mutation analysis consists of the following steps:1. Select mutation operators2. Generate mutants3. Distinguish mutants
• Live mutants– Mutants not killed by a test suite
• Given a set of mutants SM and a test suite T, the fraction of G e a set o uta ts S a d a test su te , t e act o ononequivalence mutants killed by T measures the adequacy of T with respect to SM.
325
![Page 326: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/326.jpg)
Variations on Mutation AnalysisVariations on Mutation Analysis
P bl• Problem: – There are lots of mutants. – Running each test case to completion on every mutant is expensive.– Number of mutants grows with the square of program size.
• Solutions:– Weak mutation:
• Execute meta-mutant (with many seeded faults) together with original ( y ) g gprogram
– Statistical mutation• Just create a random sample of mutants
326
![Page 327: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/327.jpg)
SummarySummary
F l b d i i id l d i i d f i• Fault-based testing is a widely used in semiconductor manufacturing.– With good fault models of typical manufacturing faults, e.g. “stuck-at-one” for
a transistorB t f lt b d t ti f d i i h ll i ( i ft )– But, fault-based testing for design errors is more challenging (as in software).
Mutation testing is not widely used in industry• Mutation testing is not widely used in industry.– But, plays a role in software testing research, to compare effectiveness of
testing techniques
327
![Page 328: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/328.jpg)
328
![Page 329: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/329.jpg)
Chapter 17Chapter 17. Test Execution
![Page 330: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/330.jpg)
Learning ObjectivesLearning Objectives
A i h f i• Appreciate the purpose of test automation– Factoring repetitive and mechanical tasks from creative and human design
tasks in testing
• Recognize main kinds and components of test scaffolding
• Understand some key dimensions in test automation design– Design for testability: Controllability and observability– Degrees of generality in drivers and stubsg g y– Comparison-based oracles and self-checks
330
![Page 331: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/331.jpg)
Automating Test ExecutionAutomating Test Execution
D i i d i i i• Designing test cases and test suites is creative.– Demanding intellectual activity– Requiring human judgment
• Executing test cases should be automatic.– Design once, execute many times
• Test automation separates the creative human process from the p pmechanical process of test execution.
331
![Page 332: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/332.jpg)
From Test Case Specifications to Test CasesFrom Test Case Specifications to Test Cases
T d i f i ld ifi i h h d• Test design often yields test case specifications, rather than concrete data.– E.g. “a large positive number”, not 420,023– E.g. “a sorted sequence, length > 2”, not “Alpha, Beta, Chi, Omega”
• Other details for execution may be omitted.
• Test Generation creates concrete, executable test cases from test caseTest Generation creates concrete, executable test cases from test case specifications.
• A Tool chain for test case generation & execution• A Tool chain for test case generation & execution– A combinatorial test case generation to create test data
• Optional: Constraint-based data generator to “concretize” individual values, e.g., from “positive integer” to 42from positive integer to 42
– ‘DDSteps’ to convert from spreadsheet data to ‘JUnit’ test cases– ‘JUnit’ to execute concrete test cases
332
![Page 333: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/333.jpg)
ScaffoldingScaffolding
C d d d d l i i i• Code produced to support development activities– Not part of the “product” as seen by the end user– May be temporary (like scaffolding in construction of buildings)
• Scaffolding includes – Test harnesses– Drivers– Stubs
333
![Page 334: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/334.jpg)
ScaffoldingScaffolding
T d i• Test driver– A “main” program for running a test
• May be produced before a “real” main programP id t l th th “ l” i• Provide more control than the “real” main program
– To drive program under test through test cases
T b• Test stub– Substitute for called functions/methods/objects
• Test harness– Substitutes for other parts of the deployed environment– E.g. Software simulation of a hardware deviceg
334
![Page 335: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/335.jpg)
Controllability & ObservabilityControllability & Observability
E l W• Example: We want to automate tests, – But, interactive input provides limited control– Graphical output provides limited observability.
GUI input (MVC “Controller”)p
Program Functionality
Graphical output (MVC “View”)
335
Graphical output (MVC View )
![Page 336: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/336.jpg)
Controllability & ObservabilityControllability & Observability
S l i A d i f d id i f f l (API)• Solution: A design for automated test provides interfaces for control (API) and observation (wrapper on output)
GUI input (MVC “Controller”) Test driver
API
Program Functionality Log behavior
Capture wrapper
336
Graphical output (MVC “View”)
![Page 337: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/337.jpg)
Generic vs Specific ScaffoldingGeneric vs. Specific Scaffolding
H l h ld ff ldi b ?• How general should scaffolding be?– We could build a driver and stubs for each test case.– Or at least factor out some common code of the driver and test management
( JU it)(e.g. JUnit)– Or further factor out some common support code, to drive a large number of
test cases from data (as in DDSteps)Or f rther generate the data a tomaticall from a more abstract model (e g– Or further generate the data automatically from a more abstract model (e.g. network traffic model)
• It’s a question of costs and re-use, just as for other kinds of software.
337
![Page 338: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/338.jpg)
Test OraclesTest Oracles
N i 10 000 i ll if h l b• No use running 10,000 test cases automatically, if the results must be checked by hand.
• It’s a problem of ‘range of specific to general’, again– E.g. JUnit: Specific oracle (“assert”) coded by hand in each test case
• Typical approach– Comparison-based oracle with predicted output valuep p p– But, not the only approach
338
![Page 339: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/339.jpg)
Comparison based OracleComparison-based Oracle
Wi h i b d l d di d f h• With a comparison-based oracle, we need predicted output for each input.
– Oracle compares actual to predicted output, and reports failure if they differ.– Fine for a small number of hand-generated test cases– E.g. for hand-written JUnit test cases
339
![Page 340: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/340.jpg)
Self Checks as OraclesSelf-Checks as Oracles
A l l b i lf h k• An oracle can also be written as self-checks.– Often possible to judge correctness without predicting results
• Advantages and limits: Usable with large, automatically generated test suites, but often only a partial check
– E.g. structural invariants of data structures– Recognize many or most failures, but not all
340
![Page 341: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/341.jpg)
Capture and ReplayCapture and Replay
S i h i l i h i d b i• Sometimes there is no alternative to human input and observation.– Even if we separate testing program functionality from GUI, some testing of
the GUI is required.
• We can at least cut repetition of human testing.
• Capture a manually run test case, and replay it automatically– With a comparison-based test oracle: behavior be same as previouslyWith a comparison based test oracle: behavior be same as previously
accepted behavior– Reusable only until a program change invalidates it– Lifetime depends on abstraction level of input and output.Lifetime depends on abstraction level of input and output.
341
![Page 342: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/342.jpg)
SummarySummary
T i i i k f d i f• Test automation aims to separate creative task of test design from mechanical task of test execution.
– Enable generation and execution of large test suites– Re-execute test suites frequently (e.g. nightly or after each program change)
• Scaffolding: Code to support development and testing– Test drivers, stubs, harness, oracles– Ranging from individual, hand-written test case drivers to automatic
generation and testing of large test suites– Capture/replay where human interaction is required
342
![Page 343: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/343.jpg)
343
![Page 344: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/344.jpg)
Chapter 18Chapter 18. Inspection
![Page 345: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/345.jpg)
345
![Page 346: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/346.jpg)
Chapter 19Chapter 19. Program Analysis
![Page 347: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/347.jpg)
Learning ObjectivesLearning Objectives
U d d h d l i l i d• Understand how automated program analysis complements testing and manual inspection
– Most useful for properties that are difficult to test
• Understand fundamental approaches of a few representative techniques– Lockset analysisy– Pointer analysis– Symbolic testing– Dynamic model extractiony
• Recognize the same basic approaches and design trade-offs in other program analysis techniquesprogram analysis techniques
347
![Page 348: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/348.jpg)
OverviewOverview
A d l i h i l d i i• Automated program analysis techniques complement test and inspection in two ways:
– Can exhaustively check some important propertieshi h i l i i i l l ill i d• Which conventional testing is particularly ill-suited
– Can extract and summarize information for test and inspection design• Replacing or augmenting human efforts
• Automated analysisReplace human inspection for some class of faults– Replace human inspection for some class of faults
– Support inspection by • Automating extracting and summarizing information• Navigating through relevant information• Navigating through relevant information
348
![Page 349: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/349.jpg)
Static vs Dynamic AnalysisStatic vs. Dynamic Analysis
S i l i• Static analysis– Examine program source code
• Examine the complete execution spaceB t l d t f l l• But, may lead to false alarms
• Dynamic analysisE i ti t– Examine program execution traces
• No infeasible path problem• But, cannot examine the execution space exhaustively
• Example:– Concurrency faults
Memory faults– Memory faults
349
![Page 350: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/350.jpg)
Summarizing Execution PathsSummarizing Execution Paths
O i i fi d ll f l f i ki d• Our aim is to find all program faults of a certain kind.– We cannot simply prune exploration of certain program paths as in symbolic
testing.
• Instead, we must abstract enough to fold the state space down to a size that can be exhaustively explored.
– Example: analyses based on finite state machines (FSM)• data values by states• operations by state transitions
Th h t k i fl l i d fi it t t ifi ti– The approaches taken in flow analysis and finite state verification
350
![Page 351: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/351.jpg)
Memory AnalysisMemory Analysis
I d i ll• Instrument program to trace memory access dynamically– Record the state of each memory location– Detect accesses incompatible with the current state
• Attempts to access unallocated memory • Read from uninitialized memory locations
– Array bounds violations:Add l ti ith t t ll t d b f d ft h• Add memory locations with state unallocated before and after each array
• Attempts to access these locations should be detected immediately
– Example:• Purify• Purify• Garbage detector
351
![Page 352: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/352.jpg)
Pointer AnalysisPointer Analysis
P i i bl d b hi i h h• Pointer variables are represented by a machine with three states:– invalid value– possibly null value– definitely not null value
• Deallocation triggers transition from non-null to invalid.gg• Conditional branches may trigger transitions.
– E.g. testing a pointer for non-null triggers a transition from possibly null to definitely non-nully
• Potential misuse – Deallocation in possibly null state– Deallocation in possibly null state – Dereference in possibly null – Dereference in invalid states
352
![Page 353: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/353.jpg)
A C Program with “Buffer Overflow”A C Program with Buffer Overflow…
int main (int argc char *argv[]) {int main (int argc, char argv[]) { char sentinel_pre[] = "2B2B2B2B2B"; char subject[] = "AndPlus+%26%2B+%0D%"; char sentinel_post[] = "26262626"; char *outbuf = (char *) malloc(10); int ret rn code
Output parameter of fixed length can overrun theint return_code;
printf("First test, subject into outbuf\n");
return_code = cgi_decode(subject, outbuf);
length can overrun theoutput buffer.
printf("Original: %s\n", subject);printf("Decoded: %s\n", outbuf);printf("Return code: %d\n", return_code);
printf("Second test, argv[1] into outbuf\n"); printf("Argc is %d\n", argc); assert(argc == 2);
return code = cgi decode(argv[1] outbuf);return_code = cgi_decode(argv[1], outbuf);
printf("Original: %s\n", argv[1]); printf("Decoded: %s\n", outbuf); printf("Return code: %d\n", return_code);
353
}
![Page 354: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/354.jpg)
Dynamic Memory Analysis with PurifyDynamic Memory Analysis with Purify[I] Starting main[E] ABR: Array bounds read in printf {1 occurrence}
Reading 11 bytes from 0x00e74af8 (1 byte at 0x00e74b02 illegal)Address 0x00e74af8 is at the beginning of a 10 byte blockAddress 0x00e74af8 points to a malloc'd block in heap 0x00e70000Thread ID: 0xd64
...[E] ABR A b d d i i tf {1 }[E] ABR: Array bounds read in printf {1 occurrence}
Reading 11 bytes from 0x00e74af8 (1 byte at 0x00e74b02 illegal)Address 0x00e74af8 is at the beginning of a 10 byte blockAddress 0x00e74af8 points to a malloc'd block in heap 0x00e70000Thread ID: 0xd64
...[E] ABWL: Late detect array bounds write {1 occurrence}
Memory corruption detected, 14 bytes at 0x00e74b02Address 0x00e74b02 is 1 byte past the end of a 10 byte block at 0x00e74af8Address 0x00e74b02 points to a malloc'd block in heap 0x00e7000063 memory operations and 3 seconds since last-known good heap state63 memory operations and 3 seconds since last known good heap stateDetection location - error occurred before the following function call
printf [MSVCRT.dll]...
Allocation locationmalloc [MSVCRT.dll]
Identifies the problem
[ ]...[I] Summary of all memory leaks... {482 bytes, 5 blocks}...[I] Exiting with code 0 (0x00000000)
Process time: 50 milliseconds
354
[I] Program terminated ...
![Page 355: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/355.jpg)
Lockset AnalysisLockset Analysis
D h d l i h i• Data races are hard to reveal with testing.
• Static analysis:– Computationally expensive, and approximated
• Dynamic analysis:– Can amplify sensitivity of testing to detect potential data races
• Avoid pessimistic inaccuracy of finite state verification• Avoid pessimistic inaccuracy of finite state verification• Reduce optimistic inaccuracy of testing
355
![Page 356: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/356.jpg)
Dynamic Lockset AnalysisDynamic Lockset Analysis
L k di i li f l d• Lockset discipline: set of rules to prevent data races– Every variable shared between threads must be protected by a mutual
exclusion lock.– ….
• Dynamic lockset analysis detects violation of the locking discipline.– Identify set of mutual exclusion locks held by threads when accessing each
shared variable.– INIT: each shared variable is associated with all available locks– RUN: thread accesses a shared variable
• intersect current set of candidate locks with locks held by the thread
– END: set of locks after executing a test set of locks always held by threads accessing that variable= set of locks always held by threads accessing that variable
• empty set for v = no lock consistently protects v
356
![Page 357: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/357.jpg)
Simple Lockset Analysis: An ExampleSimple Lockset Analysis: An ExampleThread Program trace Locks held Lockset(x)
{} {lck1, lck2}
thread A lock(lck1)
{lck1}
INIT:all locks for x
lck1 held{lck1}
x=x+1
{lck1} Intersect with locks held
unlock(lck1}
{}
locks held
tread B lock{lck2}
{lck2}
x=x+1
lck2 held
x=x+1
{}
unlock(lck2}Empty intersection potential race
357
{}
![Page 358: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/358.jpg)
Lockset Analysis in PracticeLockset Analysis in Practice
Si l l ki di i li i l d b• Simple locking discipline violated by – Initialization of shared variables without holding a lock– Writing shared variables during initialization without locks– Allowing multiple readers in mutual exclusion with single writers
Delay analysistill after initialization
(second thread) Multiple writers
t i l tireport violations
Multiple readers single writerdo not report violations
358
do not report violations
![Page 359: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/359.jpg)
Extracting Behavior Model from ExecutionExtracting Behavior Model from Execution
B h i l i• Behavior analysis can– Gather information from executing several test cases– And synthesize a model that characterizes those execution,– To the extent that they are the representative of other executions as well.
• Using behavioral models for– Testing : validate tests thoroughness
– Program analysis : understand program behaviorg y p g
– Regression testing : compare versions or configurations
– Testing of component-based software : compare components in different contexts
– Debugging : Identify anomalous behaviors and understand causesgg g y
359
![Page 360: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/360.jpg)
SummarySummary
P l i l i d i i• Program analysis complements testing and inspection.– Addresses problems (e.g., race conditions, memory leaks) for which
conventional testing is ineffectiveC b t d t b l h ti i i d t ( th– Can be tuned to balance exhaustiveness, precision and cost (e.g., path-sensitive or insensitive)
– Can check for faults or produce information for other uses (debugging, documentation testing)documentation, testing)
• A few basic strategiesB ild b t t t ti f t t b it i l– Build an abstract representation of program states by monitoring real or simulated (abstract) execution
360
![Page 361: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/361.jpg)
361
![Page 362: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/362.jpg)
Part IV. Process
![Page 363: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/363.jpg)
Chapter 20Chapter 20. Planning and Monitoring the Process
![Page 364: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/364.jpg)
Chapter 21. pIntegration and Component-based
Software TestingSoftware Testing
![Page 365: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/365.jpg)
Chapter 22. pSystem, Acceptance, and Regression
TestingTesting
![Page 366: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/366.jpg)
Chapter 23Chapter 23. Automating Analysis and Test
![Page 367: Software Testing and Analysis - Konkukdslab.konkuk.ac.kr/Class/2015/15SV/Lecture Note/Software... · 2015-01-23 · – Software Testing and Analysis : Process, Principles, and Techniques](https://reader034.fdocuments.in/reader034/viewer/2022042712/5f8edea09334e2371a5546db/html5/thumbnails/367.jpg)
Chapter 24Chapter 24. Documenting Analysis and Test