Software Testing: Introduction
-
Upload
softwarecentral -
Category
Documents
-
view
333 -
download
0
Transcript of Software Testing: Introduction
Why? What? How?, When? This Course
Software Testing: Introduction
Mohammad Mousavi
Eindhoven University of Technology, The Netherlands
Software Testing, 2008
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
Outline
Why?
What?
How?, When?
This Course
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
Why?
“Bugs”
I Facts of life! (correct by construction: notalways possible / affordable)
I Serious consequences (Pentium bug, OVChipcard, etc.) Ariane 5 explosion report:
This loss of information was due to specification anddesign errors in the software of the inertial referencesystem. ... The internal SRI software exception wascaused during execution of a data conversion from 64-bitfloating point to 16-bit signed integer value. The floatingpoint number which was converted had a value greaterthan what could be represented by a 16-bit signedinteger.
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
Why?
“Bugs”
I 2002 Costs: 60 Billion USD (only USA).I
Coders introduce bugs at the rateof 4.2 defects per hour ofprogramming. If you crack thewhip and force people to movemore quickly, things get evenworse. [Watts Humphreys]
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
Why?
Quest for QualityI
Software quality will become thedominant success criterion in thesoftware industry.
[ACM Workshop on StrategicDirections in Software Quality]
I Testing:I a way to achieve better qualityI >50% of the development costs
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
Outline
Why?
What?
How?, When?
This Course
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
What?
Sorts of “Bug”
I Error: conceptual mistakeI commission: implement the wrong
specificationI omission: forget to implement a
specification(the more difficult one to find and resolve)
I Fault/Bug/Defect: implemented error
I Failure: error shown in the behavior (Alsoanomaly, incident: observable failure.)
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
What?
Sorts of “Bug”
I Error: conceptual mistakeI commission: implement the wrong
specificationI omission: forget to implement a
specification(the more difficult one to find and resolve)
I Fault/Bug/Defect: implemented error
I Failure: error shown in the behavior (Alsoanomaly, incident: observable failure.)
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
What?
Sorts of “Bug”
I Error: conceptual mistakeI commission: implement the wrong
specificationI omission: forget to implement a
specification(the more difficult one to find and resolve)
I Fault/Bug/Defect: implemented error
I Failure: error shown in the behavior (Alsoanomaly, incident: observable failure.)
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
How?
Spec: A program that inputs an integer, and outputs 2 ∗ i3.
int i;i << cin;i = 2 * i;i = exp(i,3);cout << i;
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
How?Spec: A program that inputs an integer, and outputs 2 ∗ i3.
1: int i;2: i << cin;3: i = 2 * i;4: i = exp(i,3);5: cout << i;
I Error: confusing the binding power of operators
I Bug: Statements 3 and 4 are in the wrong order
I Fault:Test-case: on input 1, the program must output 2.input 1 ... output 8!
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
How?Spec: A program that inputs an integer, and outputs 2 ∗ i3.
1: int i;2: i << cin;3: i = 2 * i;4: i = exp(i,3);5: cout << i;
I Error: confusing the binding power of operators
I Bug: Statements 3 and 4 are in the wrong order
I Fault:Test-case: on input 1, the program must output 2.input 1 ... output 8!
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
How?Spec: A program that inputs an integer, and outputs 2 ∗ i3.
1: int i;2: i << cin;3: i = 2 * i;4: i = exp(i,3);5: cout << i;
I Error: confusing the binding power of operators
I Bug: Statements 3 and 4 are in the wrong order
I Fault:Test-case: on input 1, the program must output 2.input 1 ... output 8!
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
What?
Testing
I Planned experiments to:
1. reveal bugs (turn bugs into failures, test to fail),
Testing can show the presence of bugs, but not theabsence. [Dijkstra]
2. gain confidence in software (test to pass)
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
What?
I Test case (the plan):input, (behavior) and output
I Testing: planning and executing test-cases (how?).
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
Outline
Why?
What?
How?, When?
This Course
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
How?
Testing
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
How?
I Testing: planning and executing test-cases.
1. designing test-cases (manual, automatic: models, formalspecs),
2. executing them (manual or automatic: scaffolding, fixture),3. distinguishing failures or correct executions (manual: experts,
automatic: oracles, models)4. giving feed back to debug / change specification
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
How?
Aspects of Testing
I Functional testing:assumption: software is a function from inputs to outputscovering aspects of specificationsuitable for black-box testing (but can be enhanced withinformation from the code)
+ program independent: tests can be planned early+ tests are re-usable- gaps: untested pieces of software- redundancies: the same statements may be tested several times
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
Functional Testing: Mortgage Example
Spec. Write a program that takesthree inputs: gender (boolean), age([18-55]), salary ([0-10000])and output the total mortgage for one person
Mortgage = salary * factor,where factor is given by the following table.
Category Male FemaleYoung (18-35 years) 75 (18-30 years) 70Middle (36-45 years) 55 (31-40 years) 50Old (46-55 years) 30 (41-50 years) 35
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
An Implementation
Mortgage (male:Boolean, age:Integer, salary:Integer): Integerif male then
return ((18 ≤ age < 35)?(75 ∗ salary) : (31 ≤ age <40)?(55 ∗ salary) : (30 ∗ salary))
else {female}return ((18 ≤ age < 30)?(75 ∗ salary) : (31 ≤ age <40)?(50 ∗ salary) : (35 ∗ salary))
end if
Is this implementation correct?
No way, 12 bugs!
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
An Implementation
Mortgage (male:Boolean, age:Integer, salary:Integer): Integerif male then
return ((18 ≤ age < 35)?(75 ∗ salary) : (31 ≤ age <40)?(55 ∗ salary) : (30 ∗ salary))
else {female}return ((18 ≤ age < 30)?(75 ∗ salary) : (31 ≤ age <40)?(50 ∗ salary) : (35 ∗ salary))
end if
Is this implementation correct? No way, 12 bugs!
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
Functional Testing
Mortgage (male:Boolean, age:Integer, salary:Integer): Integerif male then
return ((18 ≤ age < 35)?(75 ∗ salary) : (31 ≤ age <40)?(55 ∗ salary) : (30 ∗ salary))
else {female}return ((18 ≤ age < 30)?(75 ∗ salary) : (31 ≤ age <40)?(50 ∗ salary) : (35 ∗ salary))
end if
Possible test cases:inputs: representatives from each age range and for each genderand salary 1,output: factors as given by the table(similar to equivalence testing; wait till next sessions!)
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
How?
Aspects of Testing
I Structural testing:covering aspects of programexamples: code coverage, branch coverage
+ giving insight to the effectiveness of test- more complicated than functional testing- incapable of detecting errors of omission
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
Structural Testing
Spec.: input: an integer x [1..216]output: x incremented by two, if x is less than 50,x decremented by one, if x is greater than 50, and50, otherwise.
if x < 50 thenx = x + 1;
end ifif x > 50 then
x = x - 1;end ifreturn x
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
Structural Testingif x < 50 then
x = x + 1;end ifif x > 50 then
x = x - 1;end ifreturn x
Test-cases: sufficiently many random inputs until all statements areat least executed once, manually check the outputs with the spec.
Input Output Pass/Fail1540 1539 P2783 2782 P3222 3221 P30 31 F
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
Structural TestingFirst “Debugged” Version:
if x < 50 thenx = x + 2;
end ifif x > 50 then
x = x - 1;end ifreturn x
Input Output Pass/Fail1540 1539 P2783 2999 P3222 3221 P30 32 P
Have we tested enough?
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
Structural Testing
if x < 50 thenx = x + 2;
end ifif x > 50 then
x = x - 1;end ifreturn x
Input Output Pass/Fail49 50 F
Pesticide paradox: debugging old faults may produce new bugs (or“wake” old bugs up).
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
How?
Ideal Mix
I Functional testing for unit testing
I Structural testing for system-level testing
I Structural measures for the effectiveness of functionaltest-cases
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
When?
Waterfall Model
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
When?
Waterfall Model
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
When?
Dealing with Bugs
1-3 Putting errors in (producing bugs),
4-6 finding bugs:I testingI fault classificationI fault isolation
7 removing bugs
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
What Else?
Alternatives
I Static Analysis:test abstract properties without running the program,e.g., uninitialized/unused variables, empty/unspecified cases,coding standards, checking for design (anti)patterns.
+ automatic and scalable for generic and abstract properties- involves approximation (true negatives and false positives);
complicated (may involve theorem proving) for concrete andspecific properties (proving the abstraction function to be“correct”)
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
What Else?
Alternatives
I Model Checking:test the state-space for formally specified properties.
+ rigorous analysis, push-button technology;- not (yet) applicable to many industrial cases (state-space
explosion)
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
When
Alternatives
I Model Checking: test all the state-space for formally specifiedproperties.
+ rigorous analysis, push-button technology;- not (yet) applicable to many industrial cases (state-space
explosion)
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
Outline
Why?
What?
How?, When?
This Course
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
Goals
I Learning about practical testing techniques (functional as wellas structural). The ability to choose among them based onthe assumptions of the problem in hand.
I Theoretical basis of testing. The ability to solve theoreticalproblems using the mathematical theory.
I Brief familiarity with debugging and automated debugging.Being able to apply debugging techniques.
I Getting an impression about Software Testing in practice.
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
Outline of This Course
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
General Information
I Home Page:http://www.win.tue.nl/∼mousavi/testing.htm
I Text book: Software Testing: A Craftsman Approach, 2ndEd., CRC Press, 2002. (3rd Ed. due in Feb. 2008)
I Marks:I Final exam on 25-06-2008, 09:00-12:00 (register before
03-06-2008).I Re-exam: 08-08-2008, 14:00-17:00.
I Office hours: Wednesdays, 14:30-15:30 or with priorappointment.
Mousavi: Software Testing: Introduction
Why? What? How?, When? This Course
Acknowledgment
I have benefitted from the following resource in designing thiscourse:
I Judi Romijn’s material (slides, papers, handouts) for 2IW30
I Tim Willemse’s slides on LTS-based testing
I Other courses on Software Testing (named on the course webpage)
Mousavi: Software Testing: Introduction