SENG 521 Software Reliability & Testing

SENG521 (Fall 2002) [email protected] 1

SENG 521SENG 521Software Reliability & Software Reliability & TestingTesting

Best Practice SRE

(Part 12c)Department of Electrical & Computer Engineering, University of Calgary

B.H. Far （ [email protected] ）http://www.enel.ucalgary.ca/~far/Lectures/SENG521/12c/


Incremental ImplementationIncremental Implementation Most projects

implement the SRE activities incrementally.

A typical implementation sequence:


Feasibility PhaseFeasibility Phase Activity 1:Activity 1: Define and classify failures

Define failure from customer’s perspective Group identified failures into a group of severity classes from

customer’s perspective Usually 3-4 classes are sufficient

Activity 2:Activity 2: Identify customer reliability needs What is the level of reliability that the customer needs? Who are the rival companies and what are rival products and what is

their reliability?

Activity 3:Activity 3: Determine operational profile Based on the tasks performed and the environmental factors


Feasibility Phase: BenefitsFeasibility Phase: Benefits Activity 1 and 2:Activity 1 and 2: Define and classify failures,

identify customer reliability needs Benefits:Benefits: Release software at a time that meets customer

reliability needs but is as early and inexpensive as possible

Activity 3:Activity 3: Determine operational profiles Benefits:Benefits: Speed up time to market by saving test time,

reduce test cost, have a quantitative measure for reliability


Requirements PhaseRequirements Phase Activity 4:Activity 4: Conduct trade-off studies

Reliability and functionality Reliability, cost and delivery date

Activity 5:Activity 5: Set reliability objectives based on Explicit requirement statements from a request for

proposal or standard document Customer satisfaction with a previous release or similar

product Capabilities of competition Trade-offs with performance, delivery date and cost Warranty, technology capabilities


Requirements Phase: Requirements Phase: BenefitsBenefits

Activity 4:Activity 4: Conduct trade-off studies Benefits: Benefits: Increase market share by providing a software

product that matches better to customer needs

Activity 5:Activity 5: Set reliability objectives Benefits:Benefits: Release software at a time that meets customer

reliability needs but is as early and inexpensive as possible


Design PhaseDesign Phase Activity 6:Activity 6: Allocate reliability among acquired software,

components, hardware and other systems Determine which systems and components are involved and how

they affect the overall system reliability

Activity 7:Activity 7: Engineer to meet reliability objectives Plan using fault tolerance, fault removal and fault avoidance

Activity 8:Activity 8: Focus resources based on operational profile Operational profile guides the designer to focus on features that are

supposed to be more critical


Design Phase : BenefitsDesign Phase : Benefits Activity 6:Activity 6: Allocate reliability among acquired software,

components, hardware and other systems Benefits:Benefits: Reduce development time and cost by striking better

balance among components Activity 7:Activity 7: Engineer to meet reliability objectives

Benefits:Benefits: Reduce development time and cost with better design Activity 8:Activity 8: Focus resources based on operational profile

Benefits:Benefits: Speed up time to market by guiding development priorities, reduce development cost


Implementation PhaseImplementation Phase Activity 8:Activity 8: Focus resources based on operational profiles

Develop more critical functions first in more detail

Activity 9:Activity 9: Manage fault introduction and propagation Practicing a development methodology; constructing modular

system; employing reuse; conducting inspection and review; controlling change

Activity 10:Activity 10: Measure reliability of acquired software, hardware and other systems

Certification test using reliability demonstration chart


Implementation Phase : BenefitsImplementation Phase : Benefits Activity 8:Activity 8: Focus resources based on operational profile

Benefits:Benefits: Speed up time to market by guiding development priorities, reduce development cost

Activity 9:Activity 9: Manage fault introduction and propagation Benefits: Benefits: Maximize cost-effectiveness of reliability improvement

Activity 10:Activity 10: Measure reliability of acquired software, hardware and other systems

Benefits:Benefits: Reduce risks to reliability, schedule, cost from unknown software and systems


System Test PhaseSystem Test Phase Activity 11:Activity 11: Determine operational profile used

for testing Decide upon critical operations Decide upon need of multiplicity of operational profile

Activity 12:Activity 12: Track testing progress and certify that reliability objectives are met Conduct feature test, regression test and performance and

load test Conduct reliability growth test


System Test Phase : BenefitsSystem Test Phase : Benefits Activity 11:Activity 11: Determine operational profile used

for testing Benefits:Benefits: Reduce the chance of critical operations going

unattended, speed up time to market by saving test time, reduce test cost

Activity 12:Activity 12: Conduct reliability growth testing, track testing progress Benefits:Benefits: Know exactly what reliability the customer

would experience at different points in time if the software is released at those points


Field Trial PhaseField Trial Phase Activity 13:Activity 13: Certify that reliability objectives and

release criteria are met Check accuracy of data collection Check whether test operational profile reflects field

operational profile Check customer’s definition of failure matches with what

was defined for testing the product


Field Trial Phase : BenefitsField Trial Phase : Benefits Activity 13:Activity 13: Certify that reliability

objectives are met Benefits:Benefits: Release software at a time that meets

customer reliability needs but is as early and inexpensive as possible; verify that the customer reliability needs are actually met


Post Delivery Phase /1Post Delivery Phase /1 Activity 14:Activity 14: Project post-release staff needs

Customer’s staff for system recovery; supplier’s staff to handle customer-reported failures and to remove faults

Activity 15:Activity 15: Monitor field reliability vs. objectives Collect post release failure data systematrically

Activity 16:Activity 16: Track customer satisfaction with reliability Survey product features with a sample customer set


Post Delivery Phase /2Post Delivery Phase /2 Activity 17:Activity 17: Time new feature introduction by

monitoring reliability New features bring new defects. Add new features

desired by the customers if they can be managed without sacrificing reliability of the whole system

Activity 18:Activity 18: Guide product and process improvement with reliability measures Root-cause analysis for the faults Why the fault was not detected earlier in the development

phase and what should be done to reduce the probability of introducing similar faults


Post Delivery Phase: Post Delivery Phase: BenefitsBenefits

Activity 14:Activity 14: Project post-release staff needs Benefits: Benefits: Reduce post-release costs with better planning

Activity 15-16:Activity 15-16: Monitor field reliability vs objectives, track customer satisfaction with reliability Benefits:Benefits: Maximize likelihood of pleasing customer with

reliability


Post Delivery Phase: Post Delivery Phase: BenefitsBenefits

Activity 17:Activity 17: Time new feature introduction by monitoring reliability Benefits:Benefits: Ensure that software continues to meet

customer reliability needs in the field

Activity 18:Activity 18: Guide product and process improvement with reliability measures Benefits: Benefits: Maximize cost-effectiveness of product and

process improvements selected

SENG 521 Software Reliability & Testing

Documents

Transcript of SENG 521 Software Reliability & Testing