Adopting the pairwise test design technique to optimize printer driver test coverage v1.Adopting the...
-
Upload
hcl-technologies -
Category
Business
-
view
88 -
download
0
Transcript of Adopting the pairwise test design technique to optimize printer driver test coverage v1.Adopting the...
© 2015, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
Abstract
Abbreviations
Market Trends and Challenges
Solution
Case Study
Best Practices
Common IssuesCommon Issues
Conclusion
Reference
3
3
3
4
5
7
88
9
10
Table of Contents
© 2015, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
Testing the interactions between printer driver variables has always been a challenge. Considering that testing all the possible combinations of driver variable interactions is quite impractical, there is always a demand for optimizing the test coverage. Pairwise testing, which is a proven technique in the software testing industry, can be adopted to provide optimized test coverage for printer driver testing.
AA case study has been conducted to explore the effectiveness of using the pairwise test design technique against the legacy approach that used randomization, using operational profiles. The input data contained around 61 variables resulting in approximately 2800 pairwise test cases to completely test all variable interac-tions. The test cases generated by the pairwise approach and the tests that were generated using the legacy approach were executed on the product under test and it was observed that the pairwise test scenarios were able to detect 19% more of defects than the ones that were detected by the legacy test cases with an execu-tion effort savings of 70%.
For any application in the current world, due to the distinct varieties of user groups, there is a need for accom-modating more input variables in a software system. With this increase, the complexity multiplies in testing the interactions between all these variables. With only 10 variables each with 2 states per se, the total test sce-narios to test all the possible combinations will be 210 = 1024 test cases. If just one more variable is added to the software system with 2 states, the total test scenarios become 2048!
In a printer driver, there are more than 60 input variables leading to an unimaginable number of test combi-nations; a complete coverage of which, is nearly impossible to provide. Considering the competitive market
Sl.No
1
2
3
4
5
6
77
DRE
TTM
NASA
OS
PDL
PICT
OATSOATS
Defect Removal Efficiency
Time To Market
Operating System
Page Description Language
Orthogonal Array Test Strategy
The National Aeronautics and Space Administration, USA
Pairwise Independent Combinatorial Testing tool from Microsoft
Full FormAcronyms
Abstract
Abbreviations
Market Trends and Challenges
Adopting the Pairwise Test Design Technique to Optimize Printer Driver Test Coverage | 3
© 2015, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
environment and the need for a reduced time to market (TTM), the test teams are challenged to come up with various strategies to optimize testing without compromising on the quality of the deliverables. This may include reducing the overall number of test cases to accelerate product time to market
Sometimes, test teams choose option (2) which can efficiently unveil a more number of defects but not most of them. With this option, providing a quantitative number in terms of test coverage is a challenge as well. Most often, option (1) is chosen by many test teams which brings ineffective test coverage, leading to poor Defect Removal Efficiency (DRE). A poor DRE means more field defects and less satisfied customers.
By randomly picking a subset of tests from the overall count without adopting a strategy (pure ran-domization or guessing)
By randomly picking a subset of tests from the overall count using operational profiles (more focus on customer usage like paper sizes / paper types, highest customer base like Windows 8.1, Mac 10.9, etc.) or
By choosing a statistically-based test design technique.
The solution to the combinatorial testing problem stated in the previous page is to adopt the Pairwise Test Design Technique for testing the printer driver’s variable interactions.
PairwisePairwise testing is an economical alternative to testing all the possible combinations of a set of variables. It is an effective test case generation technique based on the historical data that most faults are caused by interac-tions of two actions. In pairwise testing, test cases are designed to execute all the possible discrete combina-tions of each pair of input parameters. Pairwise-generated test suites cover all the combinations of two; and are therefore much smaller than exhaustive ones, yet still very effective in finding defects. Pairwise testing is also referred to as all-pairs testing or 2-way testing.
Graph 1 shows the cumulative percentage of software defects detected by the n-tuple interactions where it is apparent that for medical device software and NASA applications, pairwise interactions are able to detect more than 90% of software failures. Higher-order interactions (3-way, 4-way, etc.) can still be adopted, if pre-ferred, but the benefit-cost ratio will be considerably less.
Graph 1. Error detection rates for interaction strengths 1 to 6
Cumulative percent of
software failures
Interactions
Mediacal Devices
Browser
Sensor
NASA
Network Security
Solution
Adopting the Pairwise Test Design Technique to Optimize Printer Driver Test Coverage | 4
© 2015, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
The process flow for the pairwise test generation goes as shown in Figure 1.
In terms of printer driver testing, the key suggestion is to combine the two forms of Combinatorial Testing – Configuration Testing (testing interactions of configuration related variables such as the OS, Applications, PDL, and Connectivity, etc.) and Input Parameter Testing (testing interactions of the printer driver features).
A case study was conducted where a launched printer driver version was chosen, and the effectiveness of the pairwise testing technique was studied against the legacy test approach that was adopted while launching the driver. The legacy approach used randomization using operational profiles. The detailed process flow of the case study is shown in Figure 2.
The input data contained around 61 variables with states ranging from 1 to more than 40 resulting in approxi-mately 2800 test cases to completely test all variable interactions. PICT (Pairwise Independent Combinatorial Testing tool) was used to generate the pairwise scenarios.
PICT is a free command line tool from Microsoft that takes a model file as input and generates a compact set of parameter value choices that represent the test cases to get comprehensive combinatorial coverage of the input parameters. PICT allows users to feed constraints while generating the pairwise scenarios. There were more than 75 constraints fed into PICT for this case study.
Figure 2. Detailed process flow of the case study
Figure 1. Process flow for the pairwisetest generation
Case Study
Adopting the Pairwise Test Design Technique to Optimize Printer Driver Test Coverage | 5
© 2015, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
Table 1 represents a sample of the input data used for generating pairwise scenarios.
Table 2 represents a sample of the pairwise data generated for the case study.
Variables
States
Letter (8.5 x 11”)
Legal (8.5 x 14”)
Statement (5.5 x 8.5”)
Executive (7.25 x 10.5”)
8.5” x 13”
Automatically Select
Printer Default Type
Plain
Hole Punched
Transparency
Automatically Select
White
Blue
Yellow
Green
1 Sided Print
2 Sided Print
2 Sided print, Flip
on Short Edge
Media Size Media Type Media Color 2-Sided Printing
Variables
Total # of test cases
Resource(s) Utilized
Duration (In Man Days)
Effort Savings Through Pairwise Strategy
Unique defectd identified (excl. invalid & duplicatedefects)
% of defects detected additionally by Pairwisescenories
~ 5000
6
396
59 70
70%
19%
~ 2800
3
114
Legacy Approach Media Type
Test case #
No 6 3/4 Envelop
B5 (176 x 250mm)
New
A4 (210 x 297 mm)
Letter (8.5 x 11”)
B5 (176 x 250 mm)
1
2
3
4
5
6
Letterhead
Plain
Custom Type 2
Custom Type 1
Custom Type 5
Hole Punched
Clear (Transparency)
Gray
Goldenrod
Custom Color 3
Pink
Custom Color 1
1 Sided Print
2 Sided Print
1 Sided Print
2 Sided Print Flip on Short Edge
2 Sided Print
2 Sided Print Flip on Short Edge
Media Size Media Type Media Color 2-Sided Printing
Adopting the Pairwise Test Design Technique to Optimize Printer Driver Test Coverage | 6
Table 1. Sample of input data
Table 2. Sample of pairwise data
Table 3. Comparison data between the Pairwise approach and the Legacy approach
© 2015, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
From Table 3, it is apparent that the pairwise approach found 19% more defects than the legacy approach with a savings in execution effort of 70%. It was also found that the pairwise scenarios were able to unveil all the defects that the legacy test approach discovered, leaving NO leakages.
Table 4 shows the key differences between the legacy approach and the pairwise approach.
Another approach in the industry that attempts to solve the problem defined in this paper is Orthogonal Array Test Strategy (OATS). Orthogonal arrays are two dimensional arrays of numbers which possess the interesting quality that by choosing any two columns in the array you receive an even distribution of all the pair-wise combinations of values in the array. OATS uses the concept of orthogonal arrays to design tests for software systems.
A sample orthogonal array [L9 (34)] has been shown in Table 5 for reference.
Best Practices
Legacy Approach
Randomized
Subjective
Coverage cannot be quantitatively measured as the
data is randomized
Cannot be automated easily
Optimized
Objective
Coverage can be quantitatively measured
Great candidate for automation!
Pairwise Approach
Adopting the Pairwise Test Design Technique to Optimize Printer Driver Test Coverage | 7
Table 4. Pairwise approach v/s Legacy approach
© 2015, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
Unfortunately, OATS becomes ineffective when the system under test includes constraints because there is no way constraints can be fed into orthogonal arrays while generating test scenarios; this causes OATS to generate tests with invalid pairs.
Another disadvantage with OATS is that the Orthogonal Arrays contain all pairs an equal number of times. That means, using orthogonal arrays will result in more number of tests compared to the pairwise strategy because they include pairs equally. The pairwise strategy covers all the pairs in a lesser number of tests than OATS.
The challenge with the pairwise test strategy is that the requirements (including the constraints) should be accurate and complete; right the first time. The requirements should NOT change once the pairwise scenarios are generated; the entire pairwise matrix will have to be re-generated if any one requirement changes post the pairwise test generation. This is because each row item in the pairwise matrix is unique, attempting to cover as many unique pairs as possible with the least number of tests.
AA dedicated effort is also required for generating pairwise scenarios for every product under test, even for similar products; even a slight change in the requirements between two products will call for separate pairwise matrices.
Another challenge concerns the selection of the right tool to be used. There are many free and paid tools available in the market and one needs to clearly define the tool evaluation criteria for picking up the right tool. The evaluation criteria should consider, but not be limited to:
Common Issues
Adopting the Pairwise Test Design Technique to Optimize Printer Driver Test Coverage | 8
Control Factors
Expt. No
1
2
3
4
5
6
77
8
9
1
1
1
2
2
2
33
3
3
1
2
3
1
2
3
11
2
3
1
2
3
2
3
1
33
1
2
1
2
3
3
1
2
22
3
1
A B C D
Table 5. Sample Orthogonal Array
© 2015, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
The budget allotted for the test activity, if a paid tool is to be used.
Tool support; most of the free tools will NOT have any tool support provided along with them.
The number of input variables and states.
The complexity of the constraints that come with the input data and the ability of the tool to accom-modate these constraints while generating pairwise scenarios.
Ease of use in terms of feeding the variables and constraints into the tool.
Support in importing the generated pairwise tests into a Test Management tool for test execution management.
The pairwise test design technique is very effective in providing optimized test coverage for testing the variable interactions of a printer driver. With just 2800 test scenarios derived using the pairwise test design technique as compared to approximately 5000 test cases using the legacy approach, the pairwise technique was able to detect 19% more of defects than the ones that were detected through the legacy approach with a test execution effort savings of 70%.
TheThe pairwise test design technique also enables the test teams to measure test coverage in a quantitative way as opposed to using other strategies like pure randomization, guessing and/or randomization using opera-tional profiles.
Adopting the pairwise test design technique for testing the variable interactions of the printer driver signifi-cantly reduces the execution effort, detects more defects with less number of tests, and provides us with a quantitative way of measuring test coverage. Thus, the pairwise technique proves itself to be the best combi-natorial test design technique with a reduced number of tests and an improved defect detection rate.
Conclusion
Adopting the Pairwise Test Design Technique to Optimize Printer Driver Test Coverage | 9
For more details contact: [email protected] us on twitter: http://twitter.com/hclers andOur blog http://www.hcltech.com/blogs/engineering-and-rd-servicesVisit our website: http://www.hcltech.com/engineering-rd-services
Hello, I’m from HCL’s Engineering and R&D Services. We enable technology led organizations to go to market with innovative products and solutions. We patner with our customers in building world class products and creating associated solution delivery ecosystems to help bring market leadership. We develop engineering products, solutions and platforms across Aerospace and Defense, Automotive, Consumer Electronics, Software, Online, Industrial Manufacturing, Medical Devices, Networking and Telecom, Office Automation, Semiconductor and Servers & Storage for our customers.
This whitepaper is published by HCL Engineering and R&D Services.
The views and opinions in this article are for informational purposes only and should not be considered as a substitute for professional business advice. The use herein of any trademarks is not an assertion of ownership of such trademarks by HCL nor intended to imply any association between HCL and lawful owners of such trademarks.
For more information about HCL Engineering and R&D Services,Please visit http://www.hcltech.com/engineering-rd-services
Copyright@ HCCopyright@ HCL TechnologiesAll rights reserved.
Swadish Aaryan HCL Engineering and R&D Services
Reference
Author Info
1) http://www.pairwise.org/
2) Pairwise Testing in the Real World:
Practical Extensions to Test-Case Scenarios. (http://msdn.microsoft.com/en-us/library/cc150619.aspx#Pairwise_topic1)
3) James Bach and Patrick J. Schroeder , Pairwise Testing: A Best Practice That Isn’t
4) Jacek Czerwonka, PICT 3.3 User’s Guide
5) Practical Combinatorial Testing. SP 800-142. (Report). Natl. Inst. of Standards and Technology. 2010.
Adopting the Pairwise Test Design Technique to Optimize Printer Driver Test Coverage | 10