Software Testing testing.f12.ppt CS 121 “Ordering Chaos” “Mike” Michael A. Erlinger.
-
Upload
octavia-benson -
Category
Documents
-
view
217 -
download
1
Transcript of Software Testing testing.f12.ppt CS 121 “Ordering Chaos” “Mike” Michael A. Erlinger.
Software Testing Software Testing
testing.f12.ppt
CS 121“Ordering Chaos”
“Mike” Michael A. Erlinger
– 2 – CS 121
AdministriviaAdministrivia
Prepare for Sycamore on the 30Prepare for Sycamore on the 30thth….….
Need schedule of meeting with graders for User TestNeed schedule of meeting with graders for User Test
– 3 – CS 121
Software Testing: Convention WisdomSoftware Testing: Convention Wisdom
95% of errors are in 5% of the code95% of errors are in 5% of the code
…or maybe 80% of the errors are in 20% of the code…
Testing: Can I find that 5% or 20% via a test suite
– 4 – CS 121
Software TestingSoftware Testing
GoalsGoals
Types of testsTypes of tests
Levels of testsLevels of tests
Test measuresTest measures
Test planTest plan
– 5 – CS 121
GoalsGoals
Verification: Have we built the software right?Verification: Have we built the software right?
Validation: Have we built the right software?Validation: Have we built the right software?
Bug-free, meets specs
Meets customers’ needs
Are Verification and Validation different or variation of the same idea?My argument: “meeting specs” should equal “meeting customer needs”... not generally true (my kitchen, satellite systems)
– 6 – CS 121
Software TestingSoftware Testing
GoalsGoals
Types of testsTypes of tests
Levels of testsLevels of tests
Test measuresTest measures
Test planTest plan
most of our discussion focuses on verification
(more specifically bug testing)
– 7 – CS 121
Types of testsTypes of tests
Black boxBlack box
White boxWhite box
Gray boxGray box
– 8 – CS 121
Black Box TestsBlack Box Tests
input output
interface
1. Does it perform the specified functions?2. Does it handle obvious errors in input?3. Ariane5 – lousy error handling4. Classic ints vs floats, yards vs meters Black box should catch these if there is
adequate “test coverage”
– 9 – CS 121
Example: Ordered List of intsExample: Ordered List of ints
class OrdInts
create
getFirst
getNext
insert
delete
L=create()L.insert(5)L.insert(-1)L.insert(-1)p=L.getFirst()print (p)L.delete(p)p=L.getFirst()print(p)p=L.getNext(p)print(p)p=L.getNext(p)
-1-15error
– 10 – CS 121
Black box testsBlack box tests
Advantage: black box testerAdvantage: black box tester≠developer≠developer is unbiased by is unbiased by implementation details. e.g., Use Case testing, just work implementation details. e.g., Use Case testing, just work through all the Use Cases, building tests for eachthrough all the Use Cases, building tests for each
Disadvantage: black box tester is uninformed about Disadvantage: black box tester is uninformed about implementation detailsimplementation details unnecessary tests – test same thing in different way insufficient tests – can miss the extremes, especially if actual use
follows a different pattern – generally impossible to check all inputs
– 11 – CS 121
Black Box TestsBlack Box Tests
Input Code
choose good distribution of input – hope good distribution of code tested
x
x
x x
– 12 – CS 121
Unnecessary TestsUnnecessary Tests
Input Code
large range of input may exercise a small part of codee.g., operator test of satellite control stations, run through each
input and output light/key option. Testing same functions, whereas no one had a test for my map function.
– 13 – CS 121
Insufficient TestsInsufficient Tests
Input Code
a small range of input may exercise a large range of codebut can you ‘know’ this without knowing the code? Did we miss the 20%
– 14 – CS 121
Sufficient TestsSufficient Tests
Input Code
complexcode
a small range of input may exercise a small but important/error-prone region of code
– 15 – CS 121
White Box TestsWhite Box Tests
Based on codeBased on code
test 1
test 2
– 16 – CS 121
Example: ordered list of intsExample: ordered list of ints
class ordInts {class ordInts {
public: …public: …
private:private:
int vals[1000];int vals[1000];
int maxElements=1000;int maxElements=1000;
……
}}
– 17 – CS 121
White Box TestExample: ordered list of intsWhite Box TestExample: ordered list of ints
bool testMax(){ L=create(); num=maxElements; for (int i=0; i<=num; i++)
print iL.insert(i)
print maxElements;}
– 18 – CS 121
White Box TestsWhite Box TestsAdvantage:Advantage:
design tests to achieve good code coverage and avoid duplication can stress complicated, error-prone code Can be driven from Design, e.g., Use Cases or other Models can stress boundary values (fault injection)
Disadvantage:Disadvantage: tester=developer may have bias (team alternatives to test developer) if code changes, tests may have to be redesigned (is this bad?)
– 19 – CS 121
Gray Box TestsGray Box Tests
Look at code to design testsLook at code to design tests
But test through interfaceBut test through interface
Best of both worlds - maybeBest of both worlds - maybe
– 20 – CS 121
Example: ordered list of intsExample: ordered list of ints
class OrdInts
create
getFirst
getNext
insert
delete
L=create()L.insert(1)L.insert(2)L.insert(3)
L.insert(1001)p=L.getFirst()print(p)p=L.getNext(p)print p
123
…
……
– 21 – CS 121
Types of TestsTypes of TestsBlack box: test based on interface, through interfaceBlack box: test based on interface, through interface
Gray box: test based on code, through interfaceGray box: test based on code, through interface
White box: test based on code, through codeWhite box: test based on code, through code
Testing strategy can/should include all approaches!
My experience: Black Box = non developer, outside testing “idiots” who follow
specfications/requirements. in your case who?White Box = developer, part of development process in your case who?Gray Box = hated, non developer within developer organization in your case who?
– 22 – CS 121
Levels of TestsLevels of Tests
UnitUnit
IntegrationIntegration
SystemSystem
System IntegrationSystem Integration
white
black
– 23 – CS 121
Testing measures (white box)Testing measures (white box)
Code coverage – individual modulesCode coverage – individual modules
Path coverage – sequence diagramsPath coverage – sequence diagrams
Code coverage based on complexity – test of the risks, tricky Code coverage based on complexity – test of the risks, tricky part of code (e.g., Unix part of code (e.g., Unix ““you are not expected to understand you are not expected to understand thisthis”” code) code)
– 24 – CS 121
Code CoverageCode Coverage
How much of code is How much of code is ““testedtested”” by tests? by tests? manually profiling tools
Design new tests to extend coverageDesign new tests to extend coverage
Is 100% good? Is it possible to achieve?Is 100% good? Is it possible to achieve?
– 25 – CS 121
Path CoveragePath Coverage
How many execution paths have been exercised by tests?How many execution paths have been exercised by tests?
100% path coverage is usually impossible100% path coverage is usually impossible
Aim to cover Aim to cover commoncommon paths and error prone ( paths and error prone (complexcomplex) paths) paths
Aim to break code with tests – good testers are not liked by Aim to break code with tests – good testers are not liked by developers....developers....
– 26 – CS 121
Convention WisdomConvention Wisdom
95% of errors are in 5% of the code95% of errors are in 5% of the code
…or maybe 80% of the errors are in 20% of the code…
– 27 – CS 121
Code Complexity MeasuresCode Complexity Measures
cyclomatic complexity measure (McCabe)cyclomatic complexity measure (McCabe) measures number of linearly independent paths through
program ignores asynchronous interaction, fallibility of services etc.
developerdeveloper’’s best guess – what problems are likely and which s best guess – what problems are likely and which will be hardest to diagnosewill be hardest to diagnose
started with Knuth and others who gathered stats on programs.started with Knuth and others who gathered stats on programs.
– 28 – CS 121
Test planTest plan
Collection of tests: unit, integration, systemCollection of tests: unit, integration, system
Rationale for test: why these tests?Rationale for test: why these tests?
Strategy for developing/performing testsStrategy for developing/performing tests e.g., test units as developed, test integration at each build, run the
same tests over each build, etc. Final test plan for Version 1 should do all the unit tests, integration
tests, etc.
– 29 – CS 121
Testing StrategyTesting Strategy
TDD – test driven developmentTDD – test driven development
RegressionRegression
Test harnessTest harness
Bug trackingBug tracking
User testsUser tests
Write test BEFORE your write the code!
– 30 – CS 121
TDD – test DrivenTDD – test Driven
Unit tests are written first by the SEs. Then as code is written it Unit tests are written first by the SEs. Then as code is written it passes incrementally larger portions of the test suites. The passes incrementally larger portions of the test suites. The test suites are continuously updated as new failure test suites are continuously updated as new failure conditions and corner cases are discovered, and integrated conditions and corner cases are discovered, and integrated with regression tests. Unit tests are maintained like with regression tests. Unit tests are maintained like software and integrated into the build process. The goal is software and integrated into the build process. The goal is to achieve continuous deployment with frequent updates.to achieve continuous deployment with frequent updates.
– 31 – CS 121
Testing StrategyTesting Strategy
TDD – test driven developmentTDD – test driven development
RegressionRegression
Test harnessTest harness
Bug trackingBug tracking
User testsUser tests
AutomateBuild up number of depth of testsRun tests before you check in codeMy mess up historically
– 32 – CS 121
Testing StrategyTesting Strategy
TDD – test driven developmentTDD – test driven development
RegressionRegression
Test harnessTest harness
Bug trackingBug tracking
User testsUser tests Useful for interactive graphics applicationsautomated test framework, built one for satellite console to run through all console interactions
– 33 – CS 121
Test HarnessTest Harness
everything else
input output
inject test inputquery state
Test harnesses allow for the automation of tests. They can call functions with supplied parameters and print out and compare the results to the desired value. The test harness is a hook to the developed code, which can be tested using an automation framework.
– 34 – CS 121
StrategyStrategy
TDD – test driven developmentTDD – test driven development
RegressionRegression
Test harnessTest harness
Bug trackingBug tracking
User testUser test
Design system for trackingScrupulously record problems,their context, the effect, ideas on the cause,attempts to fix, create new tests as bugs uncovered
– 35 – CS 121
StrategyStrategy
TDD – test driven developmentTDD – test driven development
RegressionRegression
Test harnessTest harness
User testUser testValidationNonfunctional requirementsNot best for discovering bugsNeed to know how they generated bugHistorically, I have seen these used to Validate the final deliverable….too late….
– 36 – CS 121
Test PlanTest Plan
Collection of tests: unit, integration, systemCollection of tests: unit, integration, system
Rationale for test: why these tests?Rationale for test: why these tests?
Strategy for developing/performing testsStrategy for developing/performing tests
Be thoughtful in designingBe diligent in executing
– 37 – CS 121
Python & PyGamePython & PyGame
What tools are provided to do tests?What tools are provided to do tests?
What kind of tests are the tools suited to?What kind of tests are the tools suited to?
– 38 – CS 121
The EndThe End