DONE-01: Applying Quality Assurance and Testing to Application Development

Ademola Taiwo (AT)Empowerment Consultant

2 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Agenda

Introduction Quality Defined Why Quality Assurance Implementing QA Infrastructure Ensuring Quality Summary Questions

Cost of finding and fixing software errors

Co

st

3 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Achieving Quality

A Quality product meets or exceeds customer

expectations

Goal: Implement Quality

4 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

What is Quality?

Quality is– Based on customers’ perceptions of a

product’s design and how well the design matches the original specifications.

– The ability of a product and service to satisfy stated or implied needs.

– Achieved by conforming to established requirements within an organization.

http://www.asq.org/portal/page

Quality is All about the Customer

5 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

What is Quality Control

Control = An evaluation to indicate needed corrective responses; the act of guiding or the state of a process in which the variability is attributable to a constant system of chance causes.

Quality control = The operational techniques and activities used to fulfill requirements for quality.

Quality Control is NOT Quality Assurance

6 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

What is Quality Assurance?

Assurance = The act of giving confidence, the state of being certain, or the act of making certain.

Quality assurance = All the planned and systematic activities implemented within the quality system that can be demonstrated to provide confidence a product or service will fulfill requirements for quality

Quality Assurance is NOT Quality Control

7 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Cost of Poor quality

Costs associated with defects

found before the customer

receives the product or service

Internal Failure Cost

8 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Cost of Poor quality

Costs associated with defects

found after the customer receives

the product or service

External Failure Cost

9 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Cost of Poor quality

Costs incurred to determine the

degree of conformance to quality

requirements

Appraisal Cost

10 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Cost of Poor quality

Costs incurred to keep failure and

appraisal costs to a minimum

Prevention Cost

11 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Cost of Poor Quality

Quality Variables– DefectsDefects– Time To MarketTime To Market– FeaturesFeatures– Internal Cost / AffordabilityInternal Cost / Affordability

Quality Variables Drive Priorities

12 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Why is QA Necessary?

Cost of finding and fixing software errors

Co

st

Requirements Code Release TIME

“It takes 5 to 17hrs to fix bugs in released code, as compared to 1 to 1.5 hrs during code reviews and inspections”US National Institute for Standards and Technology, May 2002, Economic Impact of Inadequate Infrastructure of Software Testing.

The Boehm Curve

13 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Cost of Poor Quality

The National Institute of Standards and Technology study shows that Software defects cost the U.S. economy some $59.5 billion a year. (2002)

http://www.nist.gov/public_affairs/releases/n02-10.htm

To the US Economy

14 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Cost of Poor Quality

“On average developers make 100 – 150 errors every thousand lines. These statistics are different developer to developer and project to project and language to language.”

If 10% of the errors produced are serious in a small application of 20,000 lines, then we will have 200 serious errors!!

Watts. S. Humphrey. Multiyear Study by Software Engineering Institute (SEI) at Carnegie Mellon

University

Internal Cost to the Development Organization

15 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Cost of Poor Quality

Blue Cross Insurance, US – Sent $60 million in unwarranted and duplicate payments

Pepsi Promotion, Mexico – Instead of issuing 1 x $40,000 winning bottle, 800,000 were produced

Trader in London sold shares 100 times larger than he intended. Bank had to pay out millions to retrieve these shares. Validation requirements were not catered for

External Failure Cost (To You)

16 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Benefit of Good Quality

NIST's estimated that about $22 billion of the sunk costs could be recovered through better software testing

http://www.nist.gov/public_affairs/releases/n02-10.htm

To Everyone (the Economy)

17 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

What is Testing

Testing is the act of gathering metrics in pursuit of Quality. It generates supporting data for

assuring quality.

Testing is just another source of Quality metrics

Testing is NOT Quality Assurance

18 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Why Test?

“Software defects cost the U.S. economy almost $60 billion annually. 80% of the total development costs are consumed by developers identifying and correcting defects”

National Institute of Standards and Technology (NIST). www.nist.gov/public_affairs/releases/n02-10.htm

“Cancelled projects due to quality issues in the US, cost $55 billion dollars”

The Standish Group. CHAOS Chronicles v3.0 March 2003

A relatively inexpensive way to gather metrics

19 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

AgendaCost of finding and fixing software errors

Co

st

Introduction Quality Defined Why Quality Assurance Implementing QA Infrastructure Ensuring Quality Summary Questions

20 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Implementing QA Infrastructure

Verification Is it error free? Does it do what was specified?

Validation Is it valid? Is this what you really, really want?

Risk Mitigation Expect the Unexpected

http://www.its.bldrdoc.gov/fs-1037/dir-029/_4291.htm

NOT Test Implementation

21 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Benefits of Quality Assurance

Reduced Time to Market Lowered Risk Reduced Rework Increased Job Satisfaction Serving as a Differentiator Efficient Framework for Outsourcing“IT outsourcing will increase by 130 percent

in the next 5 years to $23 billion in 2008”Information Week 22nd October 2003

Better Customer Experience at Reduced Cost

22 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Quality is Everyone’s Job!

Quality

Performance

Interoperability

Integrity Efficiency

Maintainability

Reliability

Adaptability

Usability

Quality - or Lack of - is Ubiquitous

23 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Implementing Quality AssuranceEnsuring Quality

24 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Integrate QA into your Software Development Life Cycle (SDLC)

QA Process should begin with requirement gathering and continue through to the application release.

Quality improvements require discipline. People, processes and technology are only successful with proper follow through.

QA processes must be supported by technology to help troubleshoot and analyse problems.

Testing should be a function of lowering risk (risk-based testing).

QA begins at the Requirements Gathering Stage

25 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Measuring Quality

Understand what metrics you want to measure– Define and communicate

Clear defect severity rating Quality encompasses more than tests and defects! Think about:

– Customer satisfaction Features Platforms Defects Costs – internal and external Time to Market

Process Feature / Functionality Definition Spec reviews Code reviews Test scenario reviews Project plan reviews Measure and allow for improvements throughout the process

26 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Keys to Achieving High Quality

People Process Technology

Quality Product must Meet or Exceed Customer Expectation

27 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Key to Achieving High Quality

Development & QA Managers: Peers General Engineering Managers (GEM)

– Monitors Critical Quality Goals

– Cross Dept Communication

– Maintains Product Release Calendar

– Monitors & Improves Development Process

People: Quality / Development Organization

28 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Key to Achieving High Quality

Quality ownership: engineers and designers must think about how to test and validate– Design time requirements

Product feature design Test scenario design

Quality needs to be a ‘lifestyle’ not a checkbox item!– Design quality into the product

NOT

– Test to see if it is a quality product

Process: Guidelines

29 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

QA in Development Methodologies

QA Approach is determined by the Software Development Life Cycle (SDLC)

There are 3 main SDLC Models:

Requirements Specification

Functional Specification

Technical Specification

Program Specification

Code Acceptance Test

Business Objectives

1) Waterfall

Process: Guidelines

30 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Development Methodologies Cont….

Check

Check

Check

Check

2) V Model 3) RAD

Process: Guidelines

31 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Current View Point – Application TransformationApplying Quality Assurance and Testing to App. Dev.

32 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Key to Achieving High Quality

Testing– Functional Testing– Boundary testing– Negative testing– Platform Testing– Multi-user Testing– Stress / Performance Testing

Technology: Automated Data Gathering

33 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Test Plans

Define test scenarios in the design specs Define test cases in the test plans and automation Intent needs to be clear Review all test cases Include multiple test types Define measurable milestones

– Feature Freeze, Code Complete, Code Freeze Don’t forget performance testing

– 5X slower is a bug !

Technology: Automated Data Gathering

34 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Data Gathering

Three phase of testing– Phase 1 – Planning through Feature Freeze

– Phase 2 – Feature Freeze through Code Freeze

– Phase 3 – Code Freeze to customer delivery

Technology: Test, Test and Test again

35 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Summary

Quality is meeting customer expectation– Good quality is exceeding customer

expectation QA enhances IT effectiveness

– Increase Customer Satisfaction– involves People, Processes and

Technology Immediate ROI

– Lower cost / Increased revenue Found Money

– Start NOW

Quality is Everyone’s Job

36 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Questions?

37 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Thank you for your time!

38 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

You Can Begin NOW!

39 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

40 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

AgendaCost of finding and fixing software errors

Co

st

Introduction Quality Defined Why Quality Assurance Implementing QA Infrastructure Ensuring Quality Summary Questions

41 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Summary

Quality is meeting and exceeding customer expectations

QA involves People, Processes and Technology

Immediate ROI

Quality Assurance

42 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

The Quality Health Check Quality Health Check Focuses on Four Key

Area’s:– Software Testing– Software Configuration Management (SCM)– Incident Management – Process Management across QA

43 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Quality Matters

Bullets: Arial Bold, size 28, sentence case– Second level: Arial Bold, size 26

Third level: Arial Bold, size 24 Maximum 7 bullets per slide Maximum 7 words per bullet Type in charts and graphics

– Arial Bold, size 20 or above, initial caps Source code in examples

– ABSOLUTELY NO SMALLER THAN SIZE 20– See sample slides

Applying Quality Assurance and Testing to App. Dev.

44 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Proposed Break Down

Software Testing (IEEE 7925-2) Review and Highlight Improvements in Processes,

Procedures and Strategies: Incorporate Reviews – Walkthroughs, Informal

Reviews, Technical Reviews and Inspections Conduct a Risk Assessment's Look at Functional (Black Box), Structural (White

Box) and Non-Functional Testing in line with SDLC

Incorporate a Test Strategy Incorporate a Test Plan (IEEE 829-1998)

45 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Proposed Break Down

Software Configuration Management (SCM) Auditing Release Management Build Management Process Control Investigate the Workflow

46 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Proposed Break Down

Software Testing (IEEE 7925-2) Review and Highlight Improvements in Processes,

Procedures and Strategies: Incorporate Reviews – Walkthroughs, Informal

Reviews, Technical Reviews and Inspections Conduct a Risk Assessment's Look at Functional (Black Box), Structural (White

Box) and Non-Functional Testing in line with SDLC

Incorporate a Test Strategy Incorporate a Test Plan (IEEE 829-1998)

47 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Implementing Test Plans

Test at API Level as well as User Level Automate versus manual test Allow for multiple phases of development &

testing– Phase 1 – development of features, unit testing &

execution of automated regression tests.

– Phase 2 – regression testing of existing functionality, performance, platform, and stress testing

– Phase 3 – Final regression testing

48 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Regression Testing (example)

Criteria to Ship– Automation Passing at 95% or greater ( understand the

failures and evaluate the risk versus time to market)– Rerun of failed tests

Regression testing validates– No defects introduced– New features function

Weekend Automated Regression Tests– Execution on all platforms – 2 days to complete testing

Nightly Automated Regression Tests– Execution on all platforms– 1 night to complete testing

49 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Proposed Break Down

Software Configuration Management (SCM)– Auditing– Release Management– Build Management– Process Control– Investigate the Workflow

50 DONE-01 - Applying QA and Testing to App. Dev.© 2005 Progress Software Corporation

Measuring Quality

Understand what metrics you want to measure– Define and communicate

Clear defect severity rating Quality encompasses more than tests and defects! Think about:

– Customer satisfaction Features Platforms Defects Costs – internal and external Time to Market

Process Feature / Functionality Definition Spec reviews Code reviews Test scenario reviews Project plan reviews Measure and allow for improvements throughout the process