Post on 13-Oct-2014
Minimization and Prioritization of Test Cases
Development may take 2 to 4 years
Regression Testing
Same may have to be maintained for 10 to 15 years
Software maintenance is expensive
Reason for software change
1. Errors during actual use
2. Additional functionality
3. Changes in external world
4. Restructuring work
5. Change in technologies
6. Obsolete capabilities to be deleted.
Regression Testing
When we modify software, we typically retest it.
This retesting is called regression testing.
Development testing
Regression testing is the process of retesting the modified parts of the software and ensuring that no new errors have been introduced into previously tested code.
Regression Testing
Purposes
Increase confidence in the correctness of the modified program.
Locate errors in the modified program.
Preserve the quality & reliability of software.
Ensure the software’s continued operation.
We may perform regression testing during latter stage of software development.
Regression Testing
Regression TestingDevelopment TestingS.No.
1 We create test suites and test plan
We test all software components
Budget give time for testing
Once on software product
Performed under the pressure of release date
2
3
4
5
We can make use of existing test suites and test plans
We retest modified components or affected by modifications
No time
Many times
Performed in crisis situations, under greater time constraint.
Regression Testing
Reducing the Number of Test Cases
Select all those test cases that traverse the modified portion of the program and the portion that is affected by the modification .
These test case minimization techniques attempts to find redundant test cases
MINIMIZATION OF TEST CASES
Reducing the Number of Test Cases
Prioritization of Test Cases
A test case with highest rank has the highest priority and second highest rank test case has second highest priority and as so on.
Reducing the Number of Test Cases
Prioritization of Test Cases
Test case prioritization
General test case prioritization
Version specific test case prioritization
Reducing the Number of Test Cases
Prioritization of Test Cases
In general test case prioritization, for a given program with its test suite, we prioritize the test cases that will be usefulover a succession of subsequent modified version of original program without any knowledge of modification(s).
In version specific test case prioritization, we prioritize the test cases, when the original program is changed to the modified program, with the knowledge of the changes that have been made in the original program.
Reducing the Number of Test Cases
Prioritization GuidelinesAny guidelines should address two fundamental issues like:
What functions of the software must be tested?What are the consequences if some functions are not tested?
All prioritization guidelines should be designed on the basis ofrisk analysis. All risky functions should be tested on higher priority.Most important is the “impact of failure” which may range from “no impact” to “loss of human life” and must be studied very carefully.
Reducing the Number of Test Cases
Priority Category SchemePriority code 1 : Essential test casePriority code 2 : Important test casePriority code 3 : Execute, if time permitsPriority code 4 : Not important test casePriority code 5 : Redundant test case
Reducing the Number of Test Cases
Priority Category Scheme
Priority code 1 : Important for the customerPriority code 2 : Required to increase customer satisfactionPriority code 3 : Help to increase market share of the product
Risk Analysis
What is risk? Probability of occurrence of a problem (i.e. an event)Impact of that problem
Risk analysis is a process of identifying the potential problems and then assigning a ‘probability of occurrence of problem’ value and ‘impact of that problem value for each identified problem.Both of these values are assigned on a scale of 1 (low) to 10 (high). A factor ‘risk exposure’ is calculated for every problem which is the product of ‘probability of occurrence of problem’ value and ‘impact of that problem’ value.
Risks may be ranked on the basis of its risk exposure.
Risk Analysis
4321
Risk Exposure
Impact of that Problem
Probability of occurrence of problem
Potential ProblemS.No.
Risk Analysis Table
Risk Analysis
842School not available in the data base10
212Issued login-id is not in specified format
9623Lists not in proper format8818Ambiguous documentations71892Database corrupted610101Unauthorised access5.422Printing mistake in registration card4933Wrong entry in scheme detail form31226Wrong entry in students detail form2632Issued password not available1
Risk Exposure
Impact of that Problem
Probability of occurrence of problem
Potential ProblemsS. No.
Risk Analysis
• The potential problems ranked by risk exposure are 6, 2, 5, 3, 7, 10, 1, 8, 4 and 9.
Risk Analysis
Risk matrix is used to capture identified problems, estimate their probability of occurrence with impact and rank the risks based on this information.We may use the risk matrix to assign thresholds that group the potential problems into priority categories.
Risk Matrix
Risk Analysis
Threshold by quadrant
Impact of the problem
Probability of occurrence of the problem
PC-3 PC-1
PC-4
PC-2
0
10
9
8
7
6
5
4
3
1 2 3 4 5 6 7 8 9 10
2
1
1 3
24
5
6
78
9
10
Risk Analysis
The priority category in defined as:
Priority category 1 (PC-1) = High probability value and high impact value
Priority category 2 (PC-2) = High probability value and low impact value
Priority category 3 (PC-3) = Low probability value and high impact value
Priority category 4 (PC-4) = Low probability value and low impact value
We may decide to give more importance to high impact value over the high probability value. Hence, PC-2 and PC-3 will swap, but PC-1 and PC-4 will remain the same.
Risk AnalysisAlternate Threshold by quadrant
Impact of the problem
Probability of occurrence of the problem
PC-2 PC-1
PC-4
PC-3
0
109
8
7
65
43
1 2 3 4 5 6 7 8 9 10
21
1 3
24
5
6
78
9
10
Risk AnalysisThreshold by diagonal quadrant
Impact of the problem
Probability of occurrence of the problem
0
10987
65
4
3
2
1
1 2 3 4 5 6 7 8 9 10
PC-4
PC-3
PC-2
PC-1
Risk AnalysisThreshold based on high “Impact of Problem” value
Impact of the problem
Probability of occurrence of the problem
10987
65
43
21
1 2 3 4 5 6 7 8 9 10
0
PC-1
PC-3 PC-2
PC-5 PC-4
Risk AnalysisThreshold based on high “probability of occurrence of problem”
Impact of the problem
Probability of occurrence of the problem
10987
65
43
2
1
1 2 3 4 5 6 7 8 9 10
0
PC-4 PC-2
PC-5 PC-3PC-1
Risk Analysis
After the risks are ranked, the high priority risks are identified. These risks are required to be managed first and then other priority risks and so on.
These risks should be discussed in a team and proper action should be recommended to manage these risks.
A risk matrix has become a powerful tool for designing prioritization schemes.
All that matter when using risk matrix is the relative order of the probability estimates (which risks are more likely to occur) on the scale of 1 to 10.
The impact of the problem may be critical, serious, moderate, minor or negligible.
Execution history is given below
Test Case Array
T1
T2
T3
T4
T5
T6
T7
T8
T9
T10
1, 2, 20, 30, 40, 50
1, 3, 4, 21, 31, 41, 51
5, 6, 7, 8, 22, 32, ,42, 52
6, 9, 10, 23, 24, 33, 43, 54
5, 9, 11, 12, 13, 14, 15, 20, 29, 37, 38, 39
15, 16, 17, 18, 19, 23, 24, 25, 34, 35, 36
26, 27, 28, 40, 41, 44, 45, 46
46, 47, 48, 49, 50, 53, 55
55, 56, 57, 58, 59
3, 4, 60
Regression Testing
Suppose 1,2,5,15,35,45 and 55 are modified.
Simplest is to execute all test cases that have any of the modified lines of code.
Selected test cases : T1, T2, T3, T5, T6, T7, T8 and T9
Total : 8
Regression Testing
No. of matchesModified linesTest casesT1
T2
T3
T4
T5
T6
T7
T8
T9
T10
1, 2
1
5
--
5, 15
15, 35
45
55
55
--
2
1
1
0
2
2
1
1
1
0
Regression Testing
Number of matches found (NFOUND) is stored in an array and elements are stored in descending order.
The test case that has the maximum number of matches is selected by making its candidate value = 1. This test case will be executed first & is shown on next slide.
Regression Testing
CandidateMatches found
No. of matches (nfound)
Test cases
T1
T5
T6
T2
T3
T7
T8
T9
2
2
2
1
1
1
1
1
1, 2
5, 15
15, 35
1
5
45
55
55
1
0
0
0
0
0
0
0
Regression Testing
Test cases in descending order
T1 is selected for execution and it covers line nos. 1 and 2LOCs that are still to be executed:
[ 1, 2, 5, 15, 35, 45, 55 ] – [ 1, 2 ]
= [5, 15, 35, 45, 55]
Again, we check for the number of modified lines of code covered by each test case and sort them in descending order & select the one with maximum number of matches.
Regression Testing
CandidateMatches found
No. of matches (nfound)
Test cases
T5
T6
T3
T7
T8
T9
2
2
1
1
1
1
5, 15
15, 35
5
45
55
55
1
0
0
0
0
0
Regression Testing
T5 is selected for execution and it covers line nos. 5 and 15.LOCs that are still to be executed:
[ 5, 15, 35, 45, 55 ] – [ 5, 15 ]
= [35, 45, 55]
Again, we check for the number of modified lines of code covered by each test case and sort them in descending order & select the test case with maximum number of matches found.
Regression Testing
CandidateMatches found
No. of matches (nfound)
Test cases
T6
T7
T8
T9
1
1
1
1
35
45
55
55
1
0
0
0
Regression Testing
T6 is selected and covers line no. 35.
LOCs still to be executed : [ 45, 55 ]
CandidateMatches found
No. of matches (nfound)
Test cases
T7
T8
T9
1
1
1
45
55
55
1
0
0
Regression Testing
T7 is selected for execution covering line no. 45.LOCs still to be executed : [ 55 ]
CandidateMatches found
No. of matches (nfound)
Test cases
T8
T9
1
1
55
55
1
0
T8 is selected for execution
Order : T1, T5, T6, T7, T8
Regression Testing
Out of 10 test cases, we need to run only 5 test cases for 100% code coverage of modified lines of code.
Second Case
Lines of code have been modified and also deleted.
Example has 20 lines of code. 5 Test Cases as given on next slide.
50% saving of test case
Regression Testing
Execution HistoryTest Cases
T1
T2
T3
T4
T5
1, 5, 7, 15, 20
2, 3, 4, 5, 8, 16, 20
6, 8, 9,10, 11, 12, 13, 14, 17, 18
1, 2, 5, 8, 17, 19
1, 2, 6, 8, 9, 13
Modified lines : 6, 13, 17, 20
Deleted lines : 4, 7, 15
Regression Testing
From the test history information, remove the deleted lines say : 4,7,15.
Test case array will be:
Execution HistoryTest Cases
T1
T2
T3
T4
T5
1, 5, 20
2, 3, 5, 8, 16, 20
6, 8, 9,10, 11, 12, 13, 14, 17, 18
1, 2, 5, 8, 17, 19
1, 2, 6, 8, 9, 13
Regression Testing
Now, we want to find redundant test case. This situation has become important due to deletion of few lines.,
If we see execution history carefully. We come to know that T1 & T5 are redundant test cases.
Regression Testing
Redundant Y/N
Found in test caseLine numbersTest cases
T1
T5
1
5
20
1
2
6
8
9
T4
T2
T2
T4
T2
T3
T3
T3
Y
Y
Y
Y
Y
Y
Y
Y
13 T3 Y
Regression Testing
Test case array is:
Execution HistoryTest Cases
T2
T3
T4
2, 3, 5, 8, 16, 20
6, 8, 9, 10, 11, 12, 13, 14, 17, 18
1, 2, 5, 8, 17, 19
The remaining test cases are = T2, T3, T4
Regression Testing
Number of matches for modified lines are
Modified lines are : 6, 13, 17, 20
No. of matches found (nfound)
Matches foundTest cases
T2
T3
T4
20
6, 13, 17
17
1
3
1
Regression Testing
Test cases are sorted on the basis of matches found (descending order)
CandidateLine numbersNo. of matches foundTest cases
T3
T2
3
1
6, 13, 17
20
1
0
T4 1 17 0
Regression Testing
T3 is selected for execution. Remaining modified line is : 20
T1 & T5 should be deleted from the test suite.
Saving : 60%
Hence T3 and T2 are sufficient to execute modified lines.
Regression Testing