Quality Assessment I

Peter Dologdolog [at] cs [dot] aau [dot] dk2.2.05Intelligent Web and Information SystemsOctober 28, 2010

This Lecture

Performance and Reliability TestingData Flow Testing (White Box)Terminology in the quality assessment

2Peter Dolog, Web Engineering 2010, Quality Assesment I

Performance and Reliability Testing

Functional Steps

4Peter Dolog, Web Engineering 2010, Quality Assesment I

© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.

Thoughput with technologies

5Peter Dolog, Web Engineering 2010, Quality Assesment I

© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.

Typical tasks

Representation of workload model definitionTraffic generationData Collection and analysis

6Peter Dolog, Web Engineering 2010, Quality Assesment I

Representation of Workload Model

File lists with access frequences and analytical distributiongood enough for static resources, less flexible

File tracesDepends on representativeness of log

Finate State AutomataFlexible and yet suitable for complex patternsNeeds to be defined by hands which can be error prone and labourous

7Peter Dolog, Web Engineering 2010, Quality Assesment I

Traffic Generation

Emulates user interactionNeeds to generate and take into account cookies,

session management, and security (e.g. HTTPS)Needs to take into account network delaysGiven workflow model is taken as an input for traffic

generation

8Peter Dolog, Web Engineering 2010, Quality Assesment I

Data Collection and Analysis

You have to choose a metric and granularity samples

Mean, minimum and maximum may not be sufficient for modelling performance of advanced applications

Advanced statistics is needed thenSubject of white box or black box strategies:

resource vs. system based metrics

9Peter Dolog, Web Engineering 2010, Quality Assesment I

Black Box Testing

The whole web system as seen from outsideServed pages, hits, bytes per second ->

administrator to check the architectureResponse time: elapsed time between user click

until arrival of last object from the request -> interesting for users (not so much throughput)

Response time higher than 30seconds is unacceptable for users according to IBM

Less then 20 seconds is considered adequate

10Peter Dolog, Web Engineering 2010, Quality Assesment I

Testing Approach

Define several workload models as patterns for youVerify whether the performance stays stableIf not, analyse what could have caused problemsWeb usage logs could give you a hint of a

bottelnecksWeb application needs to operate with safe limits

11Peter Dolog, Web Engineering 2010, Quality Assesment I

Response time as a function of traffic volume

12Peter Dolog, Web Engineering 2010, Quality Assesment I

© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.

White Box testing

Evaluates internal web applications components for different client request rates (usually maximum)

You need to run it after black box in case of performane problems

Reveals what causes the problemMetrics look at resources such as CPU utilization,

network, central memory, file and socket descriptors, process table entries, …

Metrics look also at the functions such as function call frequency, percentage of time spent by program in each main function …

13Peter Dolog, Web Engineering 2010, Quality Assesment I

Performance improvement

See last lecture

14Peter Dolog, Web Engineering 2010, Quality Assesment I

Example workload models (400 concurrent users)

15Peter Dolog, Web Engineering 2010, Quality Assesment I

© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.

System Design

HTML, PhP (4), MySQL (4), Apache (2.0)Linux, 2.4GHz, 1GB main memory, 80GB disks. Fast

ethernet

16Peter Dolog, Web Engineering 2010, Quality Assesment I

Testing Loop

17Peter Dolog, Web Engineering 2010, Quality Assesment I

© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.

Results

18Peter Dolog, Web Engineering 2010, Quality Assesment I

© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.

Additional response time testing and stressing

19Peter Dolog, Web Engineering 2010, Quality Assesment I

© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.

White box testing (2)

20Peter Dolog, Web Engineering 2010, Quality Assesment I

© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.

White Box Testing

21Peter Dolog, Web Engineering 2010, Quality Assesment I

© Mauro Andreolini, Michele Colajanni, and Riccardo Lancellotti: Web System Reliability and Performance. In Web Engineering, 2006, Springer, pages: 181-218.

Data Flow White Box Testing

Control Flow Graph

ORD = (V, E)V … Set of nodes representing objectsE … VxV relationships betwen objects

Objects: client pages, server pages, componentsObjects have attributes: variables or document elementsObjects have operations: functions in scripting or prog.

Language

Graph can be defined recursively

Relationships: inheritance, aggregation, association, request, response, navigation, and redirect

23Peter Dolog, Web Engineering 2010, Quality Assesment I

Example: bidding web application

24Peter Dolog, Web Engineering 2010, Quality Assesment I

© Liu, C., Kung, D. C., Hsia, P., and Hsu, C. 2000. Object-Based Data Flow Testing of Web Applications. In Proceedings of the the First Asia-Pacific Conference on Quality Software (Apaqs'00) (October 30 - 31, 2000). APAQS. IEEE Computer Society, Washington, DC, 7.

Principles

25Peter Dolog, Web Engineering 2010, Quality Assesment I

© Liu, C., Kung, D. C., Hsia, P., and Hsu, C. 2000. Object-Based Data Flow Testing of Web Applications. In Proceedings of the the First Asia-Pacific Conference on Quality Software (Apaqs'00) (October 30 - 31, 2000). APAQS. IEEE Computer Society, Washington, DC, 7.

Composite CFG

26Peter Dolog, Web Engineering 2010, Quality Assesment I

© Liu, C., Kung, D. C., Hsia, P., and Hsu, C. 2000. Object-Based Data Flow Testing of Web Applications. In Proceedings of the the First Asia-Pacific Conference on Quality Software (Apaqs'00) (October 30 - 31, 2000). APAQS. IEEE Computer Society, Washington, DC, 7.

27Peter Dolog, Web Engineering 2010, Quality Assesment I

© Liu, C., Kung, D. C., Hsia, P., and Hsu, C. 2000. Object-Based Data Flow Testing of Web Applications. In Proceedings of the the First Asia-Pacific Conference on Quality Software (Apaqs'00) (October 30 - 31, 2000). APAQS. IEEE Computer Society, Washington, DC, 7.

OCFG for the example

28Peter Dolog, Web Engineering 2010, Quality Assesment I

© Liu, C., Kung, D. C., Hsia, P., and Hsu, C. 2000. Object-Based Data Flow Testing of Web Applications. In Proceedings of the the First Asia-Pacific Conference on Quality Software (Apaqs'00) (October 30 - 31, 2000). APAQS. IEEE Computer Society, Washington, DC, 7.

5-level Data flow testing

Function level (CFG)Function cluster level (ICFG)Object level (OCFG + criterion)Object cluster level (ICFG cross object, CCFG for

http)Application level (CCFG)

29Peter Dolog, Web Engineering 2010, Quality Assesment I

Def use chains from graphs

30Peter Dolog, Web Engineering 2010, Quality Assesment I

© Liu, C., Kung, D. C., Hsia, P., and Hsu, C. 2000. Object-Based Data Flow Testing of Web Applications. In Proceedings of the the First Asia-Pacific Conference on Quality Software (Apaqs'00) (October 30 - 31, 2000). APAQS. IEEE Computer Society, Washington, DC, 7.

Summary

Can uncover error at the lovest levelsProvides a systemHowever, does not scale for complex applicationsYou can use it in your project for the most important

functions

31Peter Dolog, Web Engineering 2010, Quality Assesment I

Terminology

Quality

According to ISO 8402-86 Quality is the totality of features and characteristics of software product that relate to its ability to satisfy stated or implied needs.

33Peter Dolog, Web Engineering 2010, Quality Assesment I

Quality Attributes (ISO/IEC 9126-1)

Functionality – correct and adequate functionsReliability – capability of ensuring given level of

performanceUsability – related to ease of use by end usersEfficiency – related to the ratio of the performance

to the number of used resourcesMaintanability – ease of modification even after

deploymentPortability – being able to run on different platforms

34Peter Dolog, Web Engineering 2010, Quality Assesment I

Quality Model

Quality model is a selection of criteria to assess properties related to quality characteristics

Quality assessment => measuring the properties related to the above according to specified quality metrics for those

35Peter Dolog, Web Engineering 2010, Quality Assesment I

Quality Characteristics (ISO/IEC 9126)

36Peter Dolog, Web Engineering 2010, Quality Assesment I

© Springer, Web Applications Engineering 2009

Quality Perspectives

37Peter Dolog, Web Engineering 2010, Quality Assesment I

© Springer, Web Applications Engineering 2009

Quality in Use

Effectiveness – capability of reaching the users goals with accuratness and completeness

Productivity – capability of spending appropriate amount of resources in relation to effectiveness to be achieved

Safety – capability of achieving acceptable level of risks for the users, the usage environment, and the users’ activities

Satisfaction – a capability to satisfy the user in a given context of use

38Peter Dolog, Web Engineering 2010, Quality Assesment I

Evaluation of Quality

Internal attributes – e. g. static measures on the code

External attributes – e. g. dynamic measures of behavior of the code

Quality in use attributes – e. g. reduction of time to produce an invoice

39Peter Dolog, Web Engineering 2010, Quality Assesment I

Web Application Functionality

ContentNavigationCorrectness and Adequateness

=> Functional Testing

40Peter Dolog, Web Engineering 2010, Quality Assesment I

Performance

Throughput:number of requests per unit of time

Response time:the time needed to serve a request

Evaluation in average and peak conditions

41Peter Dolog, Web Engineering 2010, Quality Assesment I

Usability

Easy-to-use navigation mechanismsQuickly reaching relevant content and operationNot getting lostAccessibility

42Peter Dolog, Web Engineering 2010, Quality Assesment I

Testing Web Applications

White Box TestingBlack Box TestingGray Box TestingUnit TestingIntegration TestingSystem Testing

43Peter Dolog, Web Engineering 2010, Quality Assesment I

Types of Web Applications Testing

Functional testing: Ghost pages, Unreachable pagesModel-based testing: using design models to find

out whether quality criteria are satifiedData-flow testing: to reveal invalid use of data

values due to coding errors based on control flow

44Peter Dolog, Web Engineering 2010, Quality Assesment I

Structural Model supporting Testing

45Peter Dolog, Web Engineering 2010, Quality Assesment I

© Springer, Web Applications Engineering 2009

Data-Flow testing

Path coverage Branch coverageNode coverage

46Peter Dolog, Web Engineering 2010, Quality Assesment I

Control Flow Model

47Peter Dolog, Web Engineering 2010, Quality Assesment I

© Springer, Web Applications Engineering 2009