Quality Assurance and Testing - Group 3

8/6/2019 Quality Assurance and Testing - Group 3

1/38

Quality assurance andTesting

Group 39th May 2011

Andre Bruintjies

Desir Minnaar

Neil Jonkers

Adolf Lamprecht


2/38

Agenda

Quality and the History

Software Quality Models

Neils Topics

Adolfs Topics


3/38


4/38

What is Quality and Quality Assurance?

Quality

The totality offeatures and characteristics of a product orservice thatbear on its ability to satisfy stated orimplied needs

Computer Dictionary Online .(1995). Quality . Available: http://www.computer-dictionary-online.org/?q=quality [24 April 2011]

The standard of something as measured against other things of asimilar kind; the degree of excellence of something

Oxford dictionaries. (2011). Quality. Available: http://oxforddictionaries.com/view/entry/m_en_gb0678350#m_en_gb0678350 [24 April 2011]

QualityAssurance

Aplanned and systematic pattern of all actions necessary to provideadequate confidence that the product optimally fulfils customersexpectations, i.e. that it is problem-free and will able to perform the task itwas designed for.

Computer Dictionary Online .(1995). Quality Assurance . Available: http://www.computer-dictionary-online.org/?q=quality%20assurance .[24 April 2011]


5/38

The history of quality assurance

History (Evolution) of Quality Control .(2009).

Available: http://www.rajeshtimane.com/56/academics/history-of-quality-control.html . [24April 2011]

Inspection

Total Quality


6/38

Engineering Approaches evolved to IT.

First definition of software engineering (Bauer)

Set of principles seeking a solution to a common problem

Systematic planning

Hoares (1975) four characteristics of software engineering

Wassermann (1996) eight characteristics of software engineering

Yingxu Wang and Dilip Patel. (2000). Comparative Software Engineering. Available:http://www.google.co.za/url?sa=t&source=web&cd=3&ved=0CCgQFjAC&url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.90.9432%26rep%3Drep1%26type%3Dpdf&rct=j&q=Annals%20of%20Software%20Engineering%20Volume%2010&ei=_3bCTY3gOY3qOcO77J0I&usg=AFQjCNGBj0m2lrlRXWC8sAacoKxfDhKR5g&sig2=ZFf3XupDB8rEIqVpRZKbng&cad=rja[24 April 2011]


7/38

Why is quality assurance required in IT.

Conformance to specification

Meeting customers needs

Change (Industry)

Increased confidence (customers will be kept interested)


8/38

Software development life cycle

Planning Phase Analyze Phase Design Phase Test PhaseImplementation

Phase

Satzinger, et. al. 2004. Systems Analysis & Design in a changing world. Thomson, United states of America.


9/38

Desire Minnaar


10/38

What is Quality .?

IEEE [1990] defines quality as the degree to which a system,component or process meets specified requirements and customer(user) needs (expectations).

conformance to explicitly stated functional and performance

requirements, explicitly documented development standards, andimplicit characteristics that are expected of all professionallydeveloped software.

Petrasch [1999] defines it as the existence ofcharacteristics of aproduct which can be assigned to requirements.

[ISO, 1999] as the totality of characteristics of an entity that bear onits ability to satisfy stated and implied needs,

2 Important key points:

Conformance to specification

Meeting customer needs:

Nasib S. G. ,(2005), Factors affecting effective software quality management revisited., SIGSOFT Softw. Eng. Notes 30, 2 (March 2005), pp.1-4


11/38

McCalls Quality Model (1977)

By Jim McCall, also known asGeneral Electrics Model of 1977

Originates in the US military

Bridges the gap between usersand developers

3 major perspectives for definingand identifying quality of softwareproduct

Product Revision

Product Transition

Product operations

11 Quality Factors (To specify)

23 Quality Criteria (To build)

Metrics (To Control)

Milicic , D. 2005 , Software Quality Attributes and Trade-Offs, Chapter 1 - Software Quality Models and Philosophies, viewed 25 April 2100,http://www.bth.se/tek/besq.nsf/(WebFiles)/5A52350A52726F51C12570A8004CB613/$FILE/Software_quality_attributes.pdf

Rafa E. Al-Qutaish, PhD ,(2001), Quality Models in Software Engineering Literature : An Analytical and Comparative Study, Journal ofAmerican Science ,Vol 6 Iss. 3, pp. 166-175


12/38

McCalls Quality Model (1977)

Rafa E. Al-Qutaish, PhD ,(2001), Quality Models in Software Engineering Literature: An Analytical and Comparative Study, Journal ofAmerican Science , Vol 6Iss. 3, pp. 166-175


13/38

Boehms Quality Model (1978) By Boehm to automatically and

quantitatively evaluate the qualityof software

Similar to McCall Quality Model:hierarchical quality model structure

Consists of

3 High-level characteristicsAddress the main questions of thesoftware buyer

7 Intermediate-level

Represent the qualities expected

15 Lowest-level (primitive)

Primitive characteristics providefoundation for defining qualitiesmetrics

Milicic , D. 2005 , Software Quality Attributes and Trade-Offs, Chapter 1 - Software Quality Models and Philosophies,viewed 25 April 2100,http://www.bth.se/tek/besq.nsf/(WebFiles)/5A52350A52726F51C12570A8004CB613/$FILE/Software_quality_attributes.pdf

Rafa E. Al-Qutaish, PhD ,(2001), Quality Models in Software Engineering Literature: An Analytical and Comparative Study,Journal ofAmerican Science , Vol 6 Iss. 3, pp. 166-175


14/38

ISO 9126

Established 1991 by the International Organization forStandardization

Software Product Evaluation: Quality characteristics & Guidelines forTheir Use-Standard

Based on McCall and Boehm models

Currentversion has 1 International Standard, 3 Technical Reports

ISO IS 9126 -1:Quality Model [ISO,1991]

ISO TR 9126-2: External Metrics [ISO,2003]

ISO TR 9126-3: Internal Metrics [ISO,2003]

ISO TR 9126-4:Quality in Use Metrics [ISO,2004]

Quality Model contains 2 parts

1. Internal and External quality model :

6 characteristics & 27 sub-characteristics

2. Quality in use model 4 Quality in use characteristics

Rafa E. Al-Qutaish, PhD ,(2001), Quality Models in Software Engineering Literature: An Analytical and Comparative Study, Journal ofAmerican Science, Vol 6 Iss. 3, pp. 166-175


15/38

ISO 9126: Internal External Quality Model &Quality in Use Characteristics



16/38

ISO 9126: Quality in the Lifecycle [ISO, 2001]



17/38

Comparisons



18/38

ISO 9000 family of Standards Quality Management Process

Resource Management Process

Regulatory Research Process Market Research Process

Product Design Process

Purchasing Process

Production Process

Ser vice Provision Process

Product Protection Process Customer NeedsAssessment Process

Customer Communications Process

Internal Communications Process

Document Control Process

Record Keeping Process

Planning Process

Training Process

InternalAudit Process

Management Review Process

Monitoring and Measuring Process

Non conformance Management Proces

Continual Improvement Process

A series of international standards dealing withquality systems that can be used for externalquality assurances purposes

Established 1987

Refers to 3 standards

ISO 9000 : definitions & terminology

ISO 9001 : requirements for certification

ISO 9004 : guidelines for sustainedsuccess

ISO 9001 consists 3 main parts:

Framework, describes organization aspect of a

quality system for software production

Life cycle activities, defines required actions

Supporting activities, describes the minimalrequirements to support production, supply maintenance

of software

The requirements specified are aimed primarily

atachieving customer satisfaction

bypreventing non-conformityat all stages

from design through servicing

Bazzana, G.; Andersen, O.; Jokela, T. (1993), ISO 9126 and ISO 9000: friends or foes?, Software Engineering Standards Symposium, 1993. Proceedings., 1993 , 30 Aug-3 Sep 1993, pp 79 - 88



19/38

8 Steps to implement ISO 9001 requirements

Helio Yang, Y. (2001) "Software quality management and ISO 9000 implementation", Industrial Management & Data Systems, Vol. 101 Iss: 7, pp.329 - 338

1. Establish Commitment ( 4-12 weeks)

2. Define program structure ( 4 weeks)

3. Establish foundation elements (24 weeks)

4. Define development methodologies ( 32 weeks)

5. Define major supporting elements

6. Define remaining supporting elements

7. PreliminaryAudit

Audit (3 days)

Correction ( up to 12 weeks)

8. RegistrationAudit

Audit (3 days)

Correction ( up to 8 weeks)

International Organization for standardisation, , ISO 9001 - Quality management systems Requirements,viewed 2 May 2011,http://www.iso.org/iso/survey2009.pdf


20/38

ISO Challenges

The amount ofmoney, time and paperwork required for registration

ISO 9001 promotes specification, control, and procedures ratherthan understanding and improvement

The added cost to certify and then maintain certification may not bejustified if product end users do not require ISO 9000

The standard is seen as especially prone to failure when a companyis interested in certification before quality. The standard is seenas especially prone to failure when a company is interested in

certification before quality Another problem reported is the competition among the numerous

certifying bodies, leading to a softer approach to the defects noticedin the operation of the Quality System of a firm.

Barnes, F , 2000, Good Business Sense Is the Key to Confronting ISO 9000" , Review of Business, 22 March, p1-10, viewed : 03 May 2011,http://www.allbusiness.com/specialty-businesses/713376-1.htmlSeddon, J. ,(1997), "Ten arguments against ISO 9000", Managing Service Quality, Vol. 7 Iss: 4, pp.162 - 16


21/38

Other Software Quality Models & Frameworks


Rafa E. Al-Qutaish, PhD ,(2001), Quality Models in Software Engineering Literature: An Analytical and Comparative Study, Journal ofAmerican Science , Vol 6 Iss. 3, pp. 166-175


22/38

Adolf Lamprecht


23/38

Types of testingManual testing

According to Right Hand Technologies: Software testing isperformed to verify that the completed software packagefunctions according to the expectations defined by therequirements/ specifications. The overall objective to not to find everysoftware bug that exists, but to uncover situations that could

negatively impact the customer, usability and/or maintainability.

According to Myers "Software Testing is the process of executing aprogram or system with the intent of finding errors."

Automated testing

Definition (ApTest, the software testing specialists):

Testing using software tools to execute tests without manualintervention. Can be applied in GUI, performance,API, etc. testing.

The use of software to control the execution of tests, the comparisonof actual outcomes to predicted outcomes, the setting up of testpreconditions, and other test control and test reporting functions.

http://www.aptest.com/resources.html#app-func http:www.righthandtech.com/software-testing.php


24/38

When does manual testing make sense?

High per-test or maintenance costs are an indicator that a test shouldbe done manually.

The need for human judgment to assess the correctness of the resultor extensive, ongoing human intervention to keep the test running

Installation, setup, operations, and maintenance like loading ofCDs etc.

Configuration and compatibility.

Error handling and recovery

Localisation

Documentation and help

Usability

Black, R., (2002), Investing in Software Testing: Manual orAutomated? White Paper


25/38

Wild cards (auto & manual testing)

Functional testing can either be done manually or auto

Use cases (user scenarios)

User interface

Date and time handling

Higher per-test case costs and needs for human skills, judgment, andinteraction lean towards manual testing. If you need to repeat testsmany times or reduce the cycle time for test execution, lean towardsautomated testing



26/38

Automated testing

According to Fewster & Graham (2000) Automated Testing is not aseasy as buying a popular test execution tool, recording the manual testsand playing them back whenever you want to.

Fewster, M., (1999), Software Test Automation. Addison-Wesley, 1999


27/38

When does auto testing make sense?

According to Edith et.al (2010) they have between 50000 and 100000test cases at Ericsson per release with a time limit for testing.

Regression and confirmation testing

Monkey (or random). Load, volume, and capacity

Performance and reliability

Structural, especiallyAPI-based unit, component, integration

Edith, S., (2010), Towards fully automated test management for large complex systems, 2010 3rd International Conference on Software Testing,Verification and Validation, IEEE Computer societyBlack, R., (2002), Investing in Software Testing: Manual orAutomated? White Paper


28/38

Reasons why test automation fails

QA people are not programmers smart tests Not enough time and resources allocated for development of

tests inadequate planning

Tests are fragile

T

ests are too complicated and too interdependent Expensive and cost of maintenance too high

Expectations

Lack of usefulness

Learning curve too steep

Poor choices of which parts to automate

Lack of training

http://agilesoftwaretesting.com/?p=56 and http://productdevelop.blogspot.com/2010/07/some-reasons-why-test-automation.html


29/38

Types of auto testing tools

Source testing tools (24) Functional testing tools (32)

Performance testing tools (15)

Java testing tools (27)

Embedded testing tools (7)

Database testing tools (13)

Web testing tools (109)

Bug tracking tools (66)

Test management tools (9)

Communication testing tools (109)

Other (+- 25) thus in total > 300

http://www.aptest.com/resources.html#app-func


30/38

Choosing the right tool according to Fewster(1999)

The tool selection project

The tool selection team

Investigate which commercial tools are suitable for you- this order isv

ery important Start by evaluating your requirements

Identify your constraints

Build or buy?

Identify what is available on the market

Evaluate the shortlisted candidate tools

Make the decision



31/38

Major testing tools

Tool Vendor

Loadrunner Mercury / HP

Winnrunner 10 Mercury / HP

Quick Test Professional (QTP) Mercury / HPSilk Test / Silk Performer etc Borland formerly Segue (www.borland.com)

QARun 4.9 Compuware (http://www.compuware.com/qarun.htm)

QAWizard Pro 2011 Seapine Software (www.seapine.com/qawizard.html)

Visual Test Rational Software (developed by MS)


32/38

Lessons learned according to Dustin (1999)

The various tools used throughout the development lifecycle didnot easily integrate

Duplicate information was kept in multiple repositories

The automated testing tool drove the testing effort.

Everyone on the testing team was trying to automate scripts. Elaborate test scripts were developed, duplicating the

development effort.

Automated test script creation was cumbersome.

Training was too late in the process, so test engineers lackedtool knowledge.

Testers resisted the tool

There were expectations of early payback

Dustin, E., (1999), Software Testing & Quality Engineering, Lessons in Test automation, Sept/Oct 1999 (Vol. 1, issue 5).


33/38

Case studies by Dr Dieter Kreuer et.al.(1999)

Ericsson and the cellular network provider Mannesmann Mobilfunk

Both companies seek to improve testing efficiency by means of testautomation

Main focus was on load testing Test platform created by Ericssons and enhanced by Mannesmann

High reuse rate of test cases make auto testing even more attractive

Kreuer, D. (1999) Applying Test Automation to Type Acceptance Testing ofTelecom Networks: A Case Study with CustomerParticipation , 14th IEEE International Conference on Automated Software Engineering (ASE'99)pp. 216-224


34/38

Forecasted development of Mannesmann CSS TAcc effort

Kreuer, D. (1999) Applying Test Automation to Type Acceptance Testing ofTelecom Networks: A Case Study with CustomerParticipation , 14th IEEE International Conference on Automated Software Engineering (ASE'99)pp. 216-224.


35/38

Why is it still so difficult to get quality right inIT?

Testers are ill equipped

Ever more complex software solutions

Testers have to cover as many aspects as possible

Minimum criteria that testers use to judge the readiness of testedsoftware

Any code specific fix can

(a) fix only the problem that was reported

(b) fail to fix the problem

(c) fix the problem but break something that was previouslyworking,

(d) fail to fix the problem and break somethingelse

Whittaker, J.A., What Is Software Testing? And Why Is It So Hard? IEEE Software, January/February 2000, pp. 70-79


36/38

References

Barnes, F , 2000, Good Business Sense Is the Key to Confronting ISO 9000" , Review of Business, 22 March, p1-10, viewed : 03 May 2011, http://www.allbusiness.com/specialty-businesses/713376-1.html

Bazzana, G.; Andersen, O.; Jokela, T. (1993), ISO 9126 and ISO 9000: friends or foes?, Software EngineeringStandards Symposium, 1993. Proceedings., 1993 , 30 Aug-3 Sep 1993, pp 79 88


Clifford S, (2005), So many standards to follow, so little payoff,Inc. Magazine, 1 May,p.1 viewed 03 May 2011http://www.inc.com/magazine/20050501/management.html

Dustin, E., (1999), Software Testing & Quality Engineering, Lessons in Test automation, Sept/Oct 1999 (Vol. 1, issue5).

Edith, S., (2010), Towards fully automated test management for large complex systems, 2010 3rd InternationalConference on Software Testing, Verification and Validation, IEEE Computer society


Helio Yang, Y. (2001) "Software quality management and ISO 9000 implementation", Industrial Management & DataSystems, Vol. 101 Iss: 7, pp.329 - 338

http://agilesoftwaretesting.com/?p=56

http://www.aptest.com/resources.html#app-func

http://www.ece.cmu.edu/~koopman/des_s99/sw_testing/

http:www.righthandtech.com/software-testing.php


37/38

Referencehttp:www.stqemagazine.com

International Organization for standardisation, , ISO 9001 - Quality management systems Requirements,viewed 2 may 2011,http://www.iso.org/iso/survey2009.pdf

Milicic , D. 2005 , Software Quality Attributes and Trade-Offs, Chapter 1 - Software Quality Models andPhilosophies, viewed 25 April 2100,http://www.bth.se/tek/besq.nsf/(WebFiles)/5A52350A52726F51C12570A8004CB613/$FILE/Software_quality_attributes.pdf

Naik, S., Tripathy ,P.,(2008), Software Testing and Quality Assurance: Theory and Practice , John Wiley &Sons, INC., Hoboken, New Jersey

Nasib S. G. ,(2005), Factors affecting effective software quality management revisited., SIGSOFT Softw.Eng. Notes 30, 2 (March 2005), pp.1-4

Rafa E. Al-Qutaish, PhD ,(2001), Quality Models in Software Engineering Literature: An Analytical andComparative Study , Journal ofAmerican Science , Vol 6 Iss. 3, pp. 166-175

Seddon, J. ,(1997), "Ten arguments against ISO 9000", Managing Service Quality, Vol. 7 Iss: 4, pp.162 - 16

Kreuer, D. (1999) Applying Test Automation to Type Acceptance Testing ofTelecom Networks: A Case

Study with Customer Participation, 14th IEEE International Conference on Automated Software

Engineering (ASE'99)pp. 216-224.

Myers, G.J., The Art of Software Testing, Second Edition (2004), John Wiley & Sons, Inc., Hoboken,New Jersey

Whittaker, J.A., What Is Software Testing? And Why Is It So Hard? IEEE Software, January/February 2000,pp. 70-79


38/38

References

Barnes, F , 2000, Good Business Sense Is the Key to Confronting ISO 9000" , Review of Business, 22 March, p1-10, viewed : 03 May 2011, http://www.allbusiness.com/specialty-businesses/713376-1.html

Bazzana, G.; Andersen, O.; Jokela, T. (1993), ISO 9126 and ISO 9000: friends or foes?, Software EngineeringStandards Symposium, 1993. Proceedings., 1993 , 30 Aug-3 Sep 1993, pp 79 88

Clifford S, (2005), So many standards to follow, so little payoff,Inc. Magazine, 1 May,p.1 viewed 03 May 2011http://www.inc.com/magazine/20050501/management.html

Helio Yang, Y. (2001) "Software quality management and ISO 9000 implementation", Industrial Management & Data

Systems, Vol. 101 Iss: 7, pp.329 - 338

International Organization for standardisation, , ISO 9001 - Quality management systems Requirements, viewed 2may 2011,http://www.iso.org/iso/survey2009.pdf

Milicic , D. 2005 , Software Quality Attributes and Trade-Offs, Chapter 1 - Software Quality Models andPhilosophies, viewed 25 April 2100,http://www.bth.se/tek/besq.nsf/(WebFiles)/5A52350A52726F51C12570A8004CB613/$FILE/Software_quality_attributes.pdf

Naik, S., Tripathy ,P.,(2008), Software Testing and Quality Assurance: Theory and Practice , John Wiley & Sons,

INC., Hoboken, New Jersey

Nasib S. G. ,(2005), Factors affecting effective software quality management revisited., SIGSOFT Softw. Eng.Notes 30, 2 (March 2005), pp.1-4

Rafa E. Al-Qutaish, PhD ,(2001), Quality Models in Software Engineering Literature: An Analytical and ComparativeStudy, Journal ofAmerican Science , Vol 6 Iss. 3, pp. 166-175

Seddon, J. ,(1997), "Ten arguments against ISO 9000", Managing Service Quality, Vol. 7 Iss: 4, pp.162 - 16

Quality Assurance and Testing - Group 3

Documents

Transcript of Quality Assurance and Testing - Group 3