rial Analysis Workshop

8/14/2019 rial Analysis Workshop

1/53

Copyright 2007 2008 Bj Rollison All rights reserved.

Bj RollisonTest Architect

Microsoft

http://www.TestingMentor.comhttp://blogs.msdn.com/imtesty


2/53


Define the combinatorial testing problemLearn coverage array basics

State advantages of combinatorial analysisIdentify common problemsOvercome common problems by learning to usethe right tool in the right context

Decompose complex feature setsDesign effective model filesLearn custom features of PICT

Discover how combinatorial analysis is a best

practice when used in the appropriate situation


3/53


Design an extremely small subset of tests fromall possibilitiesEvaluate important attributes and capabilitiesof the application under testIdentify issues and other potential risks

Exercise large portions of the code to reduce riskProvide accurate information to the decisionmakers for informed risk-assessment

Within a limited amount of timeProvides a high degree of confidence for theentire team


4/53


Parameters with multipleinterdependent variablesare likely to cause bugs!Testing problem...

24 parameters

2 5 variable statesper parameter > 500,000,000,000

combinations1 test/ms > 3300 years

So, how would you choose thetests?


5/53


Testing the combinations of variable stateswhen :

Parameters are directlyinterdependent

Parameters aresemi-coupled

Parameter input is unordered


6/53Copyright 2007 2008 Bj Rollison All rights reserved.

Functional testing technique in which the tester:systematically analyze parameterinteraction in a complex feature setin order to methodically select a finite subsetof tests from all possible combinations

Uses a mathematical approach based onhistorical failure indicators and empirical faultmodels

Helps reduce overall risk byHigh defect detection effectivenessHigh levels of structural coverage



Random evaluation approachesBest guess

Experience, customer data, intuition, luckRandom selectionRandomly selected from all combinations

Systematic evaluation approachesEach choice (EC)Base choice (BC)Orthogonal arrays (OA)Coverage arrays (CA)

Exhaustive (AC)



Finding OA is non-trivialUnrealistic expectationsequal number of variablesper parameter Redundant tuples in output

Test B I U S Col.5 Col.6 Col.7

1 unck unck unck unck 1 1 1

2 unck unck unck chk 2 2 2

3 unck chk chk unck 1 2 24 unck chk chk chk 2 1 1

5 chk unck chk unck 2 1 2

6 chk unck chk chk 1 2 1

7 chk chk unck unck 2 2 1

8 chk chk unck chk 1 1 2

L8 Orthogonal Array

Taguchi Array Selector http://www.freequality.org



Pairs of parameters(BI, BU, BS, IU, IS, US)Every variable combinationfor each pair of parametersMultiple pair combinationsper testMinimizes redundancy

Test 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Bold yes yes yes yes no no no yes no no yes yes no yes no noItalic no yes yes yes yes yes yes yes no no no no no no yes no

Underline no no yes yes no yes yes no yes yes yes yes no no no no

Strikethrough no no no yes no no yes yes no yes no yes yes yes yes no


10/53



Install PICT from www.pairwise.org/tools.aspCreate shortcut to the installation directory

Launch Notepad as editor, use file format asfollows

Open console windowCommand to run PICT tool is:

Use Excel to view output

Bold: Check, UncheckItalic: Check, UncheckStrikethrough: Check, UncheckUnderline: Check, Uncheck

[Pict_Path]:\PICT [path_to_file.txt] > [path_to_output.xls]



Objective:Create a model file of theNumbers format feature

Steps:Decompose the parametersIdentify interesting variable

statesCreate a PICT model fileGenerate and analyze output

Skills:Equivalence class partitioningof variable states

Basic PICT usage



Increased defect detection effectiveness withfewer number of testsIncreased code coverage with less number of testsProven results in complex and critical systems

Easy to incorporate into data-driven automationOutput almost instantaneousGreater creativity over the design of the testIncreased efficiency !



Pair-wise found 98 percent of all detectablefaults seeded in five applications

All faults in four applications

Evaluation of Combination Strategies for Test Case Selection (Grindal, et. al.)

Pairwise



In this study we found no significant differencein the FDE of n-way and random combinatorialtest suites. the result is not unexpected .Tests selectedrandomly fromall possiblecombinations# of tests equalto n-way # of tests

(Patrick J. Schroeder, Pankaj Bolaki, and Vijayram Gopu)



" a set of 29 pairwise tests gave 90% blockcoverage for the UNIX sort command. We alsocompared pairwise testing with random inputtesting and found that pairwise testing gavebetter coverage .

The AETG System: An Approach to Testing Based on Combinatorial Design (Cohen, et. al.)



Attrib.exe path + 6 optional argv.

Findstr.exe str + 19 optional argv. (421,200 EC tests)

Total # of blocks = 483 Default test suite Exhaustive coverage Pair-wise coverage

Number of test cases 9 972 13

Blocks covered 358 370 370

Code coverage 74% 77% 77%

Functions not covered 15 15 15

Total # of blocks = 1317 Hand-crafted tests N=2 coverage N=3 coverage N=4 coverage

Number of test cases 236 136 800 3533

Blocks covered 960 979 994 1006

Code coverage 73% 74% 75% 76%

Functions not covered 11 11 10 10



Regression test suite time reduced(104 man-days to 32 hours)Test design and implementation reduced(44 man-months to 4 man-months)Reduced OS configuration testing 50% with zerobug bounce (ZBB)Increase in automated test development adoptingdata-driven output of PICT

(@ 30 tests/week to 235 tests/week)3 show-stopper defects detected prior to releasesaved @ $375,000 (each QFE @ $125K)


19/53



The combinatorialfault model does notapply if

Parameters areindependent

Mathematical formulasVar Param1 + Var Param2 = Var ParamOut

Sequential operationsOrdered input



Objective:Decompose the dependentor semi-coupled

parameters and their variable statesSteps:

Decompose a featureCreate a PICT model fileGenerate and analyze output

This is important becauseParameter interactionincreases defect probabilityMisuse can provide falseconfidence



Objective:Identify dependentparameters

Steps:Decompose theparameters

Identify dependent or semi-coupledparameters

Skills:Feature decomposition



Misuse of technique (wrongfault model)

Unfamiliarity with thetechnique or feature

Right values not tested

Highly probable values gettoo little attention

Complex interactionsmissed



Interdependent parametersFonts

BoldItalicSemi-coupled parameters

Color (Black, white, red, green, blue, yellow)Size (1 1638, including half sizes)StrikethroughUnderline

Total number of tests = Cartesian product of variables = 1,257,600

tests (assuming all font size values are tested)


25/53


File format (for PICT)param : var 1, var 2, var 3

Use equivalence classes!

1 - 9 10 - 16 17 - 72 73 - 1638 1.5 1637.5Example:


26/53


Objective:Decompose the Replace feature of Wordpad for interdependent or semi-coupled parameters

Steps:Decompose a featureCreate a PICT model file

Generate outputSkills:

Feature decomposition

Variable equivalence class partitioningBasic PICT model design


27/53








28/53

Identify mutuallyexclusive variablecombinations

Analyze outputfor happy path

Analyze output

for failureindicators

Copyright 2008 Bj Rollison. All rights reserved.


29/53


Conditional orinvariantrelationships

Exclude mutuallyexclusive variable states;often increases total

number of tests

Dont change output!Not only is it foolish, butis completely negatesthe output of the tool!


30/53


Objective:Modify model file to handle conditional or invariantconstraints of variable state combinations

Steps:Decompose a featureCreate a PICT model file

Generate and analyze outputModify model file usingconditional or invariant constraint syntax

Skills:Verification of model outputUse simple script language to modify model for

conditional or invariant variable constraints


31/53





Highly probable valuesget too little attention



32/53


Weighting variables,Variable preference Give increased weighted

value to important

variables; MAY increaseprobability of occurrence


33/53


Objective:Weight variables toincrease probability

Steps:Decompose featureand create model file

Generate and analyzeoutputModify model withweights to important variable statesRegenerate output and compare

Skills:Certain variable states are more important that othersbut, ALL n-combinations should be tested first!


34/53


35/53


Objective:Seed variables toguarantee output

Steps:Decompose featureand create model file

Generate and analyzeoutputCreate seeded input file

Regenerate output and compareSkills:Certain variable states are more representative of realusage or customer requirements


36/53






Miss complex interactions


37/53


Sub-ModelsBundle specific parameters into groups

Each groups gets own combinatory ordersIncreases thoroughness of tested combinationsSyntax

{, ,, ... } @

Combinatory order cannot exceed number of its parameters


38/53


Randomize output toincrease breadth oftesting and minimize

probability of error due tomissed combinations

PICT/r: n switch


39/53


Majority (> 50%) of combinatorial defects aresimple pair interactions

Randomizing pair-wise combinations with PICT may improve t=2 detection effectiveness*

Subtle defects still being discovered at t=6

Effects are cumulativeNumber of tests increase

exponentially,but dont stop atpair-wise (t=2) 0%

20%40%60%80%

100%

t=2t=3 t=4 t=5 t=6


40/53

Increase order of n-wisecombinations for greater

depth of testing andminimize probability oferror due to complex

interactions

PICT/o: n switch 3-wise = 120 tests2-wise = 33 tests


41/53


Objective:Create a model file

Steps:

Decompose a featureCreate a PICT model fileGenerate and analyze

outputModify as necessaryRegenerate output and

compareSkills:Certain variable state combinations must berevalidateddepending on their criticality


42/53


Misuse of techniqueDomain knowledge beyond the user interface

Unfamiliarity with technique or toolTraining, in-depth technical knowledge,use the right tool for the right job!

Right values not testedIn-depth system knowledge, stochastic dataHighly probable values get too little attention

Weighted variables, include seeded inputsComplex interactions are missed

Random output, increase n-wise combinations


43/53


Setup/upgrade testing


44/53


Interoperability testing


45/53


Configuration testing


46/53


Game testing


47/53


Black-box API testing


48/53


Utility/command line testing


49/53


Compiler testing


50/53


Best Practice compared to other approaches for testing complex parameter interactionsProvides critical information earlier

High probability of early defect detectionHigh structural coverage; t=6 exhaustive

Increases depth and breadth of test coverageEasy to integrate into automation (data-driven)But,

Requires in-depth technical knowledgeMust verify output its only a tool!

Dont forget Beizers Pesticide Paradox!


51/53


http://[email protected]

http://hwtams.com


52/53


http://www.pairwise.orgPairwise Testing in the Real World: PracticalExtensions to Test-Case ScenariosJacek Czerwonka(http://msdn2.microsoft.com/en-us/testing/cc150619.aspx?topic=306121)

Comparing the Fault Detection Effectiveness of N-

way and Random Test SuitesPatrick J. Schroeder, Pankaj Bolaki, and Vijayram Gopuhttp://portal.acm.org/citation.cfm?doid=1159733.1159742

Software and Hardware Testing UsingCombinatorial Covering SuitesAlan Hartman, IBM Haifa Research Laboratoryhttp://www.haifa.il.ibm.com/projects/verification/mdt/papers/CombinatorialProblemsinSWTesting-finalDraft180703.pdf


53/53

rial Analysis Workshop

Documents

Transcript of rial Analysis Workshop