CS4723 Software Validation and Quality Assurance Lecture 9 Bug Report Management.
CS4723 Software Validation and Quality Assurance Lecture 15 Advanced Topics Test Plans and...
-
Upload
sheena-carr -
Category
Documents
-
view
219 -
download
0
Transcript of CS4723 Software Validation and Quality Assurance Lecture 15 Advanced Topics Test Plans and...
Test Plan & Management
Something you are going to think about when you are a project manager instead of a tester / developer
About handle the testing progress of a whole software project
About quality assurance at highest level
Approach: risk based testing
Presentation Outline Introduction - The Challenge Why? Risk Analysis and Testing
Risk Based Test Approach What? Test Metrics
Risk Based Test Metrics
Risk Analysis in Risk Based Testing Risk Analysis in Practice
Plan and Manage - Metrics How? Quality and Progress Monitoring
Warning! Organization and Process...
The Challenges
Time Constraints Resource Constraints Quality Requirements Risk Factors:
New technology Lack of knowledge Lack of experience
Take Control!
Presentation Outline Introduction - The Challenge Risk Analysis and Testing What?
Risk Based Test Approach Test Metrics
Risk Based Test Metrics
Risk Analysis in Risk Based Testing Theory and Practice
Plan and Manage Quality and Progress Monitoring
Warning! Organization and Process...
Risk Analysis and Testing
Risk Identfication
Risk Strategy
Risk Assessment
Risk Mitigation
Risk Reporting
Risk Prediction
Testing, Inspection etc.
Test Plan
Metrics: Cost and Probability
Test Item Tree
Test Metrics
Risk Based Testing - Approach
Risk Identification / Test Items Identify the items to be tested
Functions, classes, features, transactions, … Risk Strategy / Test Plan:
Setup approaches to generating test cases Automatic / Manual Original / Regression Load / Security / Usability / ,,,
Risk Assessment / Metrics Assess risk of each test items
Risk Based Testing - Approach
Risk Mitigation / Testing Generate and execute the concrete test cases
Risk Reporting Measure how much have we mitigated the risk
Test coverage Found and fixed bugs
Risk Prediction Based on the risk reports, predict risks of test items for new
round of testing
Risk Based Testing - Metrics Identify Areas with “High Risk Exposure”
Probability and Cost All functions/modules should be tested to a
“minimum level” “Extra Testing” in areas with high risk exposure Establish Test Plan and Schedule
Monitor Quality Number of Faults per function and time
Monitor Progress Number of hours in test and fix -> ETC
The Formula
Re(f) - Risk Exposure of function f P(f) - Probability of a fault in function f C(f) - Cost related to a fault in function f
Risk Based Testing - Theory
P(f)*C(f)Re(f)
Simple Test Metrics
Test Planning Number of test cases per function Number of hours testing per function
Progress Tracking Number of tests planned, executed and
completed Number of faults per function Number of hours used for test and fix Estimated to Complete
Simple Test Metrics
Probability of faults - Indicators New functionality Size Complexity Quality of previous phases and documents
Cost of Faults
Presentation Outline Introduction - The Challenge Risk Analysis and Testing
Risk Based Test Approach Test Metrics
Risk Based Test Metrics
Risk Analysis in Risk Based Testing How? Theory and Practice
Plan and Manage Quality and Progress Monitoring
Warning! Organization and Process...
Risk Based Testing - Example
Other Probability Factors might include: Function Points, Frequency of Use etc.
Ranking the functions based on Risk ExposureThe Probability of a FaultThe Cost of a Fault
Example: 2
C(s)C(c)P(f)*Re(f)
Cost Probability Risk
Func. C( s ) C(c) Avr.New Func.
Design Quality Size
Com-plexity
Weighted Averag
Exp. func.
5 5 1 3 fInterest Calc. 3 3 3 2 3 3 3 37 111
Close Account 1 3 2 2 2 2 3 31 62
Cust. Profitab. 2 1 1,5 3 3 2 3 41 61,5
Consequence
Pro
bab
ilit
y
LowLow
High
High
1
2
34
Risk Based Testing - Reporting
510
440
10 1
TECHNICAL
INTERFACE
RISK
439
370
11 2 2
369
302
5
Low Medium High
BUSINESS RISK
Low Medium High
Low
M
ediu
m
Hig
h
Consequence
Pro
bab
ilit
y
Prior to test execution: identify critical transactions
1
Test Execution identifies
“bad” transactions
2
Extra Testing: - Additional testing by product specialist- Automated regression testing
3
Risk Based Testing - Practice
Presentation Outline Introduction - The Challenge Risk Analysis and Testing
Risk Based Test Approach Test Metrics
Risk Based Test Metrics
Risk Analysis in Risk Based Testing Theory and Practice
Plan and Manage How? Quality and Progress Monitoring
Warning! Organization and Process...
Two Strategies
On-line Risk Analysis Usually used for first time testing Cold start with little data Minimal testing and a small list of critical features
provide by customers Batch Risk Analysis
Assume a large amount of data at the beginning Different metrics
On-line Test Cases Completed
Date
Nu
mb
er o
f T
est
Cas
es
Planned
Executed
QAed
Planning and Progress Tracking
Progress Indicators - “To be vs. Actual”
“To be fixed” vs. “Actually fixed”
To be retested Act. retested
Rejected
To Be Restested, Actually Retested and Rejected
Nu
mb
er o
f F
ault
s“To be Retested” vs. “Actually Retested”
Actually fixed
To Be Fixed and Actually Fixed
Nu
mb
er o
f F
ault
s
To Be Fixed
Progress Indicators - Hours Used
BatchNumber of hours
for finding one fault and for fixing one
OnlineNumber of hours for finding one fault and for fixing one
Hours per Fault for Test and Fix
Date
Ho
urs
pe
r F
ault
Test
Fix
Test
Fix
Hours per Fault for Test and Fix
Ho
urs
pe
r F
ault
“Estimated to Complete” ETC for system test based on:
Number of hours testing per fault found Number of hours fixing per fault Number of
faults found per function
Number of fixes being rejected
Number of remaining tests (functions to be tested)
Calculated ETC and Actual Hours
Date
Ho
urs
ET
C
Estimated to Complete at Time t
Actual to Complete at Time t
Benefits of Risk Based Testing
Improved Quality? all critical functions tested
Reduced Time and Money in Testing effort not wasted on non critical or low risk
functions Improved customer confidence
due to customer involvement and good reporting and progress tracking
Presentation Outline Introduction - The Challenge Risk Analysis and Testing
Risk Based Test Approach Test Metrics
Risk Based Test Metrics
Risk Analysis in Risk Based Testing How? Theory and Practice
Plan and Manage Quality and Progress Monitoring
Warning! Organization and Process...
WARNING! Require a focused Organization…
Strong Test Management Testers to “drive” the Programmers Responsibility
Application based
groups
v222
Test log
Test procedure
TEST EXECUTION
Change
request
2
Test design
Test data
Function based
groups
DYNAMIC
TEST PLANNING
STATIC
TEST PLANNING
Test plan
Test item tree
Test item spec.
Test case spec.
Test
summary
report
LOG KEEPING
Event log
Error /
change log
Objectives,
policy and
strategy for the
test project
specific.
…and a well defined Process DocumentationPlanning and Execution ProceduresControl ProceduresPlanning and Progress Tracking
LD PD
Test Exec
PTDsRaised
Fix
CR
Fix Procedure
Problem Mngmnt.Procedure
Test Exec.Procedure
Case BuildProcedure
Case QualityStandards
Change Mngmnt.Procedure
Re-test
Good/Bad
TestCompleted
RegressionTest
Good
Bad
ProAte
QC / QABasic Test Data
Test Case
Risk Identification
Risk Assessment
Risk Mitigation
Risk Reporting
Risk Prediction
Test Process Work Flow