Silver Bullets and Software Testing CS3300 Fall 2015.
-
Upload
madison-wood -
Category
Documents
-
view
218 -
download
0
Transcript of Silver Bullets and Software Testing CS3300 Fall 2015.
![Page 1: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/1.jpg)
Silver Bullets and Software Testing
CS3300Fall 2015
![Page 2: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/2.jpg)
Fred Brooks
Author of Mythical Man Month -- Essays about development of OS360 (1975)
Overall Premise??
Essential ComplexityAccidental Complexity
![Page 3: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/3.jpg)
Thoughts from the Retrospective Panel (OOPSLA 2007)Fred Brooks -- biggest benefit was OO
David Parnas -- substitute tools for learning your trade, everybody wants an easy answer to hard problems. Coding genius is actually bad since no one else can understand or fix it.
Linda Northrop – the hardest thing about software is what to say, not how to say it. We focus on technologies instead of understanding the needs of users. Focus on essence, not accident.
Dave Thomas – Universities don’t teach key concepts, they teach specific languages
Ricardo Lopez – we are the silver bullets
Martin Fowler – Buy vs. Build, Growing great designers, rapid prototyping and incremental development, Agile – syntax over semantics
Werewolf – Every time someone makes a CPM/PERT chart or a Gantt chart, I get to eat a kitten
Audience – How do you measure productivity?
![Page 4: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/4.jpg)
Software Quality Assurance• Testing is not SQA.• SQA is a broad field that we will touch on later.
![Page 5: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/5.jpg)
Vocabulary Failure Fault/Defect/Bug Error Test Case Test Suite Driver Stub / Mock
![Page 6: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/6.jpg)
Testing Principles
• 1. Testing shows the presence of defects (cannot show absence of defects) (Dijkstra)
• 2. Exhaustive testing is impossible• a java routine to test ax2+bx+c exhaustively would take 10 billion
years to run 4 times.• 3. Confusing an absence of failures with product fitness is a fallacy• 4. Testing should begin as early as possible
![Page 7: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/7.jpg)
Testing Principles
• 5. Defects are not spread uniformly in modules (Defect Clustering – Pareto Principle)
• 6. Using the same set of test over and over will cease to find bugs (Pesticide Paradox)
• 7. Vary testing efforts based on circumstances
![Page 8: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/8.jpg)
Goals Uncover defects : Static and Dynamic Gain confidence in product and provide information about its quality Preventing defects : Uncover new defects after changes (regression
tests)
![Page 9: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/9.jpg)
Testing is in entire lifecycleRequirements : Reviews, WalkthroughsDesign : Analysis, Review, WalkthroughCode : Unit Tests, Integration and System testsQA : Formal TestsDeployment : Alpha/Beta Tests, Acceptance TestsMaintenance : Regression Tests
![Page 10: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/10.jpg)
How much testing?• Adequacy and Coverage criteria (measure)• Need to define a finite set of tests that is a subset of all possible
tests.• These tests need to be thorough, but how do we define
thorough?• So Adequacy and Coverage try to answer the question “Do I
have enough tests?”
• Adequacy: Our product should have all requirements tested.• Criteria Measure: Percent of requirements currently tested.
10
![Page 11: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/11.jpg)
White Box Testing• Dynamic• AKA Basis Path testing or Structural Testing• Faults can be discovered by exercising the different execution
paths of the code.• Many errors deal with incorrect branching conditions or
variable assignment errors• Usually used at the unit level (individual functions and classes).• Coverage criteria = Statement, Branch, Path
11
![Page 12: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/12.jpg)
A little more on CFG Basic Block Decision Point Junctions Compound Predicate Shapes don't matter, just graph Statement coverage = visit all nodes Branch coverage = visit all edges
![Page 13: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/13.jpg)
1 public int foo(int argc, string[] argv) {2 int x, y;3 if (argc < 4) {4 usage();5 System.Exit(-1);6 }7 x=Int32.TryParse(argv[1]) + 34;8 for (y = 0; y < 5; y++) {9 y += x + Int32.TryParse(argv[2]);10 }11 }
![Page 14: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/14.jpg)
Annotating CFG DataflowDefinition (d) : value of a variable is definedUse (u) : value of a variable is used
Computation (c) : used in right hand side of exprPredicate (p) : used in boolean expression
Definition-Use path (du path) : a path through the CFG from a d to a u.
Definition-Clear path : a path through the CFG from a d to a u that has no intervening d
![Page 15: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/15.jpg)
Program Slicing Focusing on statements that contribute to the values of certain
variables.int i;int sum = 0;int product = 1;for(i = 0; i < N; ++i) { sum = sum + i; product = product *i;}write(sum);write(product);
int i;int sum = 0; for(i = 0; i < N; ++i) { sum = sum + i; }write(sum);
![Page 16: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/16.jpg)
McCabe’s Measure
• AKA Cyclomatic Complexity• Based on idea that complexity is caused by complex control flow• Maximum complexity for a quality routine is 10• Three ways to compute:
• Number of Predicates + 1• Edges-Nodes+2 from CFG• Number of Regions in CFG
• Forms an upper bound on test cases required for branch coverage and lower bound on path coverage thus: branch coverage cases <= mccabe <= path coverage cases
![Page 17: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/17.jpg)
A problem for WhiteBox tests
17
public int divide(int num, int den) { return num / den;}
test case: divide(10, 2) expect 5
Statement, branch and all paths coverage with one test case.
![Page 18: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/18.jpg)
Black Box Testing• Dynamic• AKA Specification-Based Tests• We pretend not to know anything about how the code is
implemented
18
![Page 19: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/19.jpg)
Equivalence Partitions• Dynamic• Divide the inputs into partitions such that for every input value
in a partition, the system would treat it the same way.• Alternatively, if one test case in a partition succeeds, then we
assume all the cases would have succeeded. If one case fails, we assume all the cases in that partition would have failed.
19
![Page 20: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/20.jpg)
Equivalence Partitions Ranges = 1 valid, 2 invalid Set = 1 in set, 1 not in set
Example:Triangle analyzer: Returns Scalene, Isoceles, Equilateral or Not a
Triangle. valid lengths are 1 to 100.
TriType analyze(int a, int b, int c)
![Page 21: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/21.jpg)
Boundary Values• We don’t just want to pick values in a partition, we want to
focus on those right at the boundary between partitions. • We pick a good value on the boundary, and one off on the
invalid side.
21
![Page 22: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/22.jpg)
Decision Tables / Decision Condition Testing• Used when the inputs are not independent, but have some kind
of relationship• In our triangle example, there is a relationship between the
three inputs that governs the correct answer.• Testing Condition “A or B”
• TT and FF will provide both the true and false result• Is that adequate?
22
![Page 23: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/23.jpg)
Decision Table Testinga < b + c F T T T T T T T T T T
b < a + c - F T T T T T T T T T
c < a + b - - F T T T T T T T T
a= b - - - T T T T F F F F
a = c - - - T T F F T T F F
b = c - - - T F T F T F T F
NOT XX XX XX
SCALENE XX
ISOCELES XX XX XX
EQUILAT XX
IMPOSSIBLE XX XX XX
![Page 24: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/24.jpg)
Syntax Testing• For a particular grammar• Take each mandatory and optional path
• - name : string• name : string• - name : string = “Bob”
24
![Page 25: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/25.jpg)
Mutation Testing Common errors occur in coding like off-by-one Create Mutations of a program Run tests Mutations should fail Real program should succeed Add tests as necessary to kill all mutants Useful to develop complete test suite, or to validate existing tests. May have to prove equivalence for some mutants.
![Page 26: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/26.jpg)
Markov Chains, randomized testing Build Markov model of program Assign probabilities Generate and Run Tests
![Page 27: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/27.jpg)
Combinatorial Testing Pairwise Tests Cannot test all possible combinations Assumption that pairs of vars cause problems Test all pairs Use tool to generate Manual use Orthogonal Arrays
![Page 28: Silver Bullets and Software Testing CS3300 Fall 2015.](https://reader034.fdocuments.in/reader034/viewer/2022051316/56649e735503460f94b72ddc/html5/thumbnails/28.jpg)
Exploratory Testing• simultaneous learning, test design and test execution• style of software testing that emphasizes the personal freedom
and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project