Politecnico di Milano Web quality Luciano Baresi Politecnico di Milano – Dipartimento di...

PolitecnicoPolitecnicodi Milanodi Milano

Web qualityWeb quality

Luciano BaresiLuciano BaresiPolitecnico di Milano – Dipartimento di Elettronica e InformazionePolitecnico di Milano – Dipartimento di Elettronica e Informazione

Piazza L. da Vinci, 32 – 20133, Milano (Italy)Piazza L. da Vinci, 32 – 20133, Milano (Italy)[email protected]@elet.polimi.it

Web Quality - Buenos Aires (Argentina) July 21-25 2

Who am I?Who am I?

Research positionsAssociate professor at Politecnico di Milano (Italy)Visiting positions at

University of Oregon (USA)University of Paderborn (Germany)

Professional ActivitiesCo-chair of GT-VMT 01, ICECCS 02, UMICS 03PC member of several conferences (ICWE 04)

Co-founder and senior partner of INQuaS

www.elet.polimi.it/www.elet.polimi.it/~baresi~baresi


Politecnico di MilanoPolitecnico di Milano

The oldest technical university in ItalyOne of the oldest in Europe

5 schools1000 faculties25,000 students


QualityQuality


A definitionA definition

A methodological approach to the analysis and management of all company processesThis activity aims at

Reducing wastes Improving products and servicesImproving processes

We all know what quality is


Different perspectivesDifferent perspectives

Quality is quality but different roles have different views. For example

StakeholdersThey “see” the business

UsersThey want to “use” the system

DevelopersThey must develop and maintain the system


Process quality vs. product quality Process quality vs. product quality

Process qualityWe aim at introducing special-purpose characteristics to control

CostsTime-to-deliveryQualityCompetitiveness

Product qualityWe aim at products with particular quality indicators no matter of the process behind themThey must be able to match explicit and implicit requests from customers


Quality & development processQuality & development process

RUP


Quality dimensionsQuality dimensions

UsabilityDependability/ReliabilityCorrectnessRobustnessScalabilityPerformabilitySecurity

internal vs. external properties


Web applicationsWeb applications

(at least their architectures)(at least their architectures)


Web applications vs. Web sitesWeb applications vs. Web sites


Why Web applications?Why Web applications?

The Web is heterogeneous by definitionToo many technologies (Applet, Servlet, JSP, ASP, XML, …)Much more importance to presentation and communicationDistribution…

Internet

ClientFirewall

Internal network

internal users

SMTP serverWeb serverDB server

N-tier architecture


HeterogeneityHeterogeneity

ComponentsHTMLScripting languagesDatabasesMultimedia contents

ExpertiseSystem designersDB administratorsDesignersProgrammersTesters

Development and maintenance are complex (more complex?)


Client/ServerClient/Server

Entities that are logically distinct, but linked by a network, that work together to obtain a given task

Requests

Services


Logical componentsLogical components

PresentationUser interactions

Data managementAccess, persistency, queries

Application logicBusiness oriented computations

In some cases some of these elements can be absentEmbedded systems have no presentation layers

PresentationPresentation Applicationlogic

Applicationlogic

Datamanagement

Datamanagement


Client and server responsibilitiesClient and server responsibilities

PresentationPresentation

LogicLogic


LogicLogic

Serv

er

Clie

nt

Netw

ork


LogicLogic

PresentationPresentation PresentationPresentation

LogicLogic

LogicLogic

Data management

Data management


LogicLogic

Datamanagement

Datamanagement

A B C D E

Data management

Data management

Data management

Data management

Data management

Data management

Data management

Data management


Client and server responsibilitiesClient and server responsibilities

Distributed presentationThe server owns all the knowledgeThe client only interacts with the server

Example: HTML form (no control on the quality of data)Remote presentation

Presentation is fully in charge of the clientDistributed logic

The logic is partly on the server and partly on the clientRemote data access

Presentation and logic are on the client that interacts with a server to access data (SQL interface)

Distributed databaseThe functionality to manage data are partially on the client and partially on the server (e.g., Distributed Relational Database Architecture di IBM)


Fat client vs. Fat serverFat client vs. Fat server

Clients are fat if the implement part of the application logic (C, D, E)Why fat clients

The system can handle user inputs more quicklyThe system interacts with the user with a finer granularityDifferent clients can implement interfaces that are specific to different usersThe system is more scalable


Fat client vs. Fat serverFat client vs. Fat server

Why not fat clientsInteractions with the server can become too frequent during complex computations

Servers could better control data access if they control the computation

We loose data encapsulationThe client must know in detail how data are organized on the server

System management and maintenance become more complex

Clients and server must be both updated It would be easier if we could update only the server


Three-tier architecturesThree-tier architectures

In the ideal model the mid component contains all the application logic

In many real cases the logic is also spread on client and server

This way we can distinguish between presentation and application logic

Different distributed data sourcesDifferent client types


Applicationlogic

Datamanagement

Datamanagement

ServerServerClientClient

Intermediatecomponent



Another exampleAnother example

Client

Shared Application Services

DataSources

Order Entry

Customer Service

Scheduling

Credit Check

Customer Information

Distribution Service

Inventory Manager

Credit Service

CustomersDistribution Inventory


From two-tier to n-tierFrom two-tier to n-tier


Applicationlogic

Datamanagement

Datamanagement

ServerServerClientClient Intermediate

component


ServerServerClientClient

2-tier

n-tier


Intermediatecomponent Intermediate

component



Computational modelsComputational models

Computational modelComputational model

Host/terminalmodel

Host/terminalmodel

Models based onDistributed computation

Models based onDistributed computation

Client/serverClient/server

Models basedon events

Models basedon events

Models based onfile transfer

Models based onfile transfer Peer-to-PeerPeer-to-Peer

Object-OrientedObject-

Oriented


What can we do?What can we do?


Basically, we canBasically, we can

MeasureWe “understand” the quality of our applications by measuring itSeveral different measures

AnalyzeWe “understand” the quality by studying the artifacts we haveInteresting, but complex

TestWe can “understand” the quality by trying with some special executions

No magic solutions !!!


Design for testabilityDesign for testability


TestabilityTestability

proc foo ( ) x: integer; y: char; begin xxlskd ;

xxl;

Definition ofanalysis models

Direct control over source code(usually not feasible)

? P

Interestingproperties

Algorithm control ofproperty P’

? P´

Modelproperty

Implication


Important factorsImportant factors

Systems can be tested more easily if they:Work decently (operability)Can be controlled (controllability)Can be observed (observability)Do not add useless complexity (simplicity)Are documented consistently and completely (understandability)Are well-known (suitability)Are stable (stability)


Five dimensionsFive dimensions

RequirementsWe must consider the way we represent them

DesignWe should anticipate as many constraints as we can

ImplementationTest oracles should be “added” during the implementation

Test We should identify test cases as soon as we can

DocumentationBetter testability if we have good documentation


Some very preliminary commentsSome very preliminary comments

Design projects as simple as possibleNo added (useless) complexity

Use/add contracts (assertions) to our componentsMaximize the visibility of all productsConsider comments and documentationWrite “standard” codePrivilege formal (semi-formal) notations instead of informal models

Is this true with the Web?


nowebnoweb


Test and analysisTest and analysis


Why test and analysisWhy test and analysis

Software is never correctNo matter of the domainNo matter of the techniques we use

Any software must be verified/validatedTest and analysis are

Important to control and assess the quality of products

But impact on the processUsually expensiveDifficult, but interesting

Good compromises


IEEE terminologyIEEE terminology

ErrorWhat causes the problem (deviation) between the product and the ideal programErrors and faults are not consistentFor example, typos, cut&paste, wrong requirements

FaultProgram elements that do not correspond to the expectations Faults do not respect locality and are not consistent with failuresFor example, the program has a multiply operator instead of a sum operator

FailureBehavior not consistent with system specificationFor example, 4 + 3 = 12


PropertiesProperties

Process oriented (internal) propertiesReusabilityMaintainabilityModularity

External properties that can be verifiedInteroperabilityTimeliness

External properties that cannot be verifiedUser-friendlinessUsability

Dependability propertiesCorrectnessRobustnessSafetyReliability


DependabilityDependability

Robust

Safe

Robust but not safe: we can have catastrophic errors

Correct, but not safe or robust: the specification is not enough

Reliable

Reliable, but not correct: We can seldom have failures

Correct

Safe, but not correct: We can have “light” failures


Validation and verificationValidation and verification

Requirements

Formal description

system

validation verificationInclude usability test, user feedback

Includes test, inspection, static analysisbuilding the right systembuilding the system right


Validation vs. VerificationValidation vs. Verification

If we say that the page must display quickly, we cannot verify the property, but we can validate itIf we say that the page must display in 30 seconds, we can verify it.


The real problemThe real problem

Correctness properties are undecidable

Decisionprocedure

property

program

yes/no


What do they offer?What do they offer?

We must settle for some kind of inaccuracyto be able to deal with the problem

Perfect verification

Optimistic approximation(testing)

Pessimistic approximation(analysis, proof)

Simplified properties

Sample theinput space


Impact of the software typeImpact of the software type

The software type and its characteristics impact test and analysis activities in different ways

Different emphasis on the same propertyTimeliness“Correctness”

Different propertiesUsabilityUser-friendliness

New techniquesPresentationNavigation


PrinciplesPrinciples

The basic principles areSensitivity: it is better to fail every time than sometimeRedundancy: we should make our intentions explicitPartitioning: divide et imperaRestriction: we should try to reduce the scopeFeedback: we should use the experience to improve the process


SensitivitySensitivity

Better to fail every time than sometimesConsistency helps:

A test selection criterion is better if any selected test gives the same results, that is, if the program fails with a given test, it should fail with all tests selected with that criterionFor example, deadlock analysis at run-time is better if it does not depend on the machine, that is, if the program fails on a machine within a given execution, then it should fail on all machines within that execution


RedundancyRedundancy

Make decisions explicitRedundant control can increase the capability of capturing errors in advance or more efficiently

Static type check is redundant with respect to testing, but it can solve many problems in advanceThe validation of requirements is redundant with respect to the validation of the final product, but it can solve several problems in advance and more efficientlyTest and model checking are redundant, but they are used together in some cases to increase the confidence on the right behavior of the product


PartitionPartition

Divide et imperaDifficult problems can be treated by partitioning the input space

Criteria for both functional and structural test selection identify meaningful partitions of the input spaceVerification techniques partition the input space by grouping data that are homogeneous with respect to the properties that we want to prove


RestrictionRestriction

Simplify the problemClever restrictions can make problems that are difficult (and undecidable) simple and tractable

In some cases a weaker property can be easier to verify

For example, we cannot demonstrate that pointers are used in the right way, but if we use Java we can impose it easily

In other cases, an heavier property can be easier to verify

For example, in general we cannot demonstrate that we do not have type errors with languages that have a dynamic type system, but we can demonstrate it if the language is statically typed.


FeedbackFeedback

Fine tune the development processLearn from the experience:

Checklists are built on errors discovered in the pastThe way errors are classified can help define meaningful criteria to select test casesThe mechanisms to revise the process are based on the fact that we must improve the process to improve the product


Test and analysis in a development processTest and analysis in a development process

Activities related to quality control and types of development processesDegrees of freedom and compromises

How to balance budget, risks, and qualityError analysis and feedbackImpact of the development process on test activitiesResponsibilities of a test group


Course outlineCourse outline

UsabilityAccessibility

Functional correctnessTesting and analysis techniques

Robustness and scalabilityPerformanceSecurityTest processToolsConclusions


UsabilityUsability


UsabilityUsability

It is the measure of how a software system satisfies the needs of its users

Ease of useEfficacy and efficiencyEase of storing produced artifactsLow number of errors and ease to recoverSatisfaction while using the product

Jakob Nielsen


Some design principlesSome design principles

Design of pages based on the deviceCorrect visualization with different browsersCorrect visualization no matter of the screen characteristicsLight-to-load pages

Ease to access supplied contentsStructure of contents to facilitate their readingCoherence in the page styleVisibility of links and their meaningConsistency of the presentation style with respect to users

Ease of navigation


Page design and devicePage design and device


Correct visualizations and browsersCorrect visualizations and browsers

The semantics of tags can change in different browsersThe design can

Use the minimum common set, trying not to use the last version of the languageUse the latest version

To stimulate users to updateMany users may not to be able to access some contents

We should test our applications with all well-known browsersIf we knew the users, we could better understand what we need

Specific classes of usersSkilled users

We can start developing the application with old versions of the languages and

Add the extensions laterAll main contents must be published using such a format that can be accessed by all users


Screen-independent designScreen-independent design

800x600 is the standard resolutionMany displays are already set to this resolutionIf we used higher resolutions, the visualization would use horizontal sliding bars

Vertical bars are acceptable, but we should avoid horizontal ones

We could use relative dimensions (percentages)The visualization adapts to the current resolution and the current screen sizeBut we must be sure that all visualizations are meaningful


Design light pagesDesign light pages

Some studies have revealed that0.1 second is the limit to give the user the perception that the system is reacting

We do not need any message, but we need to start displaying the result

1 second is the limit within which any human thought is not interrupted

The user perceives the delay, but it is still acceptable

10 seconds is the limit to keep the user attention on the dialogue

If delays are higher, the user changes his focusIf we deliver pages in more than 10 seconds, this means that we loose the client


Possible problems and solutionsPossible problems and solutions

The speed can be influenced by the number and size of images

Light and few imagesMultimedia effects used only if they are really helpful

They improve the way information is perceivedNo pages should be loaded in more than 20 secondsWe should check page loading in the different situations

Maybe even with different cache settingsFirst screen at a glance

We should reduce the time needed to load the first pageThis page (the first part) should provide as much information as possible (more text than images)


Neat and clear informationNeat and clear information

We should use some 50% less text than “standard” newspapers

When we read on a screen, we are 25% slowerUsers do not love to scroll windows

Contents should be well-structured: titles, paragraphs, and itemized lists are useful to locate contents

Rule of the pyramid (taken from journalism)Each page should start with a short conclusion (summary) and then present all details


Hypertextual structureHypertextual structure

We should use hypertext to split long text in pagesThe hypertext should not be used to break linear informationEach information peace should focus on a specific and well-defined argument

We should be able to identify the entities that are more relevant and then identify their subcomponents

We should then identify the relationships that exist between entities and between subcomponents within the same entity


Homogeneous visualization styleHomogeneous visualization style

We should choose coherence and uniformityWe should plan the themes, information structures, and navigation paths that contribute to the uniformity of the siteWe should define a visual schema that should be adopted in all pages

Same fonts, colors, rendering of contents to communicate the adopted schema and facilitate the comprehension to users

We should avoid to change the styleLinear transitions and uniformity give the user the idea of being in the same site (known borders)Better ease of use and learning since users can find well-known models


An example (I)An example (I)


An example (II)An example (II)


Linear transitionsLinear transitions

Same backgroundSame fontsSame grid to partition the pageSame positioning of the different page elementsSame use of empty spaces

Example: national gallery of Washington (www.nga.gov)

http://www.nga.gov/


Facilitate access to contentsFacilitate access to contents

Ease the interactionIdentification of links and their meaning

Link typesStructural links: They define the structure of the information space and allow the user to move within itAssociation links: Textual links that let the user deeper the knowledge of the text used as anchorLists of alternative pointers: They can help users find what they need

Starting rhetoricThe user must be able to identify the added value that he will find in the target page

Ending rhetoricThe target page must clarify the ending context and give value to the source page


Descrivere i link testualiDescrivere i link testuali

We should not use long anchorsThe must only be pointersToo many words do not allow us to identify the meaning of the link (max 4 words)We should use simple words that clearly identify the target

Examples– To know about my job click here– Here is my job

We can add text to supply further informationIt can be used to differentiate similar links


Suitable presentation stylesSuitable presentation styles

The visual style must give the right impression that we want to communicateWe should choose the right styleFor example

http://www.nasa.govhttp://kids.msfc.nasa.gov


Ease of navigationEase of navigation

Where am I? Where was I? Where can I go?We should always define the page titleBreadcrumbs link identify the path in the site followed to reach a given pageA link to the home page (or to the starting pages of the different sections) from each page help the user to understand where he is We should be careful with the links defined so far


Site organizationSite organization

Each application must have a home page It provides a summary and links to some important pages

A Web site is the set of pages that are linked to the home page through links

The should supply an homogeneous set of information

HP


Linear structureLinear structure

It guides the user along a pathUseful when the presentation implies that the user follow a predefined path

examples: lessons, book chapters, collections, etc.

User navigate forward and backward by following a predefined pathEach page has a link to the initial page

HP

Argument Argument Argument…

SubArgument1 SubArgument2


Network structureNetwork structure

We need a navigation bar to guide the user

Typical in many simple applications (personal site)Links to and from any page in the application

Home Page

Link1Link2Link3....

LinkN

HP


Hierarchical structure Hierarchical structure

Areas identify and organize contentsIn each page we can have links to the home page and to the other pages of the area

Home Page

Sub areapage

Contentspage

…Area page Area page Area page

Sub areapage

Contentspage

Contentspage

Contentspage

Contentspage

Contentspage

Contentspage


Usability testUsability test


A simple exampleA simple example

Information density is inverse proportional to the search time and the capability of finding what we were searching forA useful page should not contain an excessive number of non homogenous dataWe have many pages, even professional ones, that are not useful according to this criterion



Test with sample users to study their interaction with the system

The human participation is basilarTest of HTML code to verify they are compliant with the W3C guidelines

It can be easily made in an automatic wayThey are complementary techniques



The tests with sample users apply techniques to collect qualitative and quantitative data while they interact with the product to stress particular featuresWe have two main approaches with sample users

The use of formally defined tests to validate or invalidate particular hypotheses (not so used)The use of iterative tests to gradually expose usability problems

The goals areIdentification and correction of usability problems before releasing the applicationFacilitate the creation of products that

Are easy to learn and useSatisfy user needsSupply useful functionality to the set of target users

Create a repository of tests to be used for future releases


Non technical goalsNon technical goals

Minimize the costs to support product maintenanceIncrease the number of sold productsAcquire a competitive advantage with respect to competitors

Usability is a key feature for many productsMinimize risks before the release


Types of usability testsTypes of usability tests

Analysis of user

requirementsRequirementsspecification

Preliminary design

Detailed design

Implementation

Explorative tests

Evaluation tests

Validation tests

Comparison tests


Explorative tests (I)Explorative tests (I)

WhenAt the beginning of the development processWhen we know the user profile and how the system will be used, but we are working on the functional specificationsBefore working on the detailed project

GoalsUnderstand the validity of the preliminary design Verify the conceptual idea that the user has of the product (high abstraction level)Verify the hypothesis on the user


Explorative tests (II)Explorative tests (II)

MethodologyWe use product prototypes to make them be evaluated by representative usersAt the beginning, we can also use static interfaces, maybe even on whiteboards (story boards)We must highly interact with participants to

Verify the efficacy of the concepts on which the preliminary project is basedHelp fill the gap as to not-yet-implemented functionality


Evaluation tests (I)Evaluation tests (I)

It is the most widely used usability testWhen

At the beginning or during the development cycleAfter the definition of the preliminary project

GoalsWe want to extend the results with explorative tests to evaluate the usability of low level operationsWhile explorative tests work on the skeleton of my product, these tests consider all characteristicsWe do not want to evaluate how intuitive the product is, but how a user can complete realistic tasks and maybe identify possible problems


Evaluation tests (II)Evaluation tests (II)

Methodology:The user always works on tasks, instead of surfing around and making commentsWe not want to understand mental processes, but we consider the actual behaviorWe collect quantitative measures


Validation tests (I)Validation tests (I)

Called also verification test When

Late in the development cycleIt is used to certify the usability of the productBefore releasing the product

GoalsWe want to compare the product against predefined standards or standard used by competitorsWe want our product to comply with standards before releasing it

If it is not compliant with want to understand whyThese standards are defined at the beginning of the development cycle along with the usability goals


Validation tests (II)Validation tests (II)

Usability goalsThey are defined by thinking of

Usability tests on previous versionsMarket analysisInterviews with users

They are characterized by means ofCriteria to measure performance (speed, user accuracy while working on a task)Criteria for user preferences


Validation test (III)Validation test (III)

MethodologySimilar to evaluation testBefore testing, we need to identify the reference standardsWe must define also the tolerance with want to use to accept results (e.g., % of failures)We propose specific tasks to participantsThey do not interact with the test monitorWe collect quantitative data


Comparison tests (I)Comparison tests (I)

WhenIt is not associated with any specific step in the development processIn the first phases it can be used to compare different alternatives through exploratory testsIt can be used to access the efficacy of a single componentIt can be used to compare the product with what developed by competitors

GoalsIt can be associated with any of the other testing methodsIt can be used to understand pros and cons of different projectsIt can be used to understand the effectiveness of different designs


Comparison tests (II)Comparison tests (II)

MethodologyWe propose the vis-a-vis comparison between two or more alternativesFor each alternative, we collect information and observations on performance and preferencesThe format we use depends on what the want to getIn many cases we discover that some alternatives can be the winning ones The best results come when we compare projects that are radically different (instead of just similar)


The phases of usability testThe phases of usability test

Definition of test planSelection and recruiting of participantsPreparation of test materialExecuting the testDebriefingConclusions and recommendations


Definition of test planDefinition of test plan

Objectives: We want to describe the reasons why we want to test the application

For example: We have user feedback on some particular problems with the application

Test objectives: We must describe the test questions in a clear and neat way

For example: is the on-line help easier through hot-keys or the mouse?Is the on-line help enough and self-contained?Is the navigation good enough to allow the user to always know where he is?Is the product usable?

This is wrong because it is too vague


Test plan: how to design testsTest plan: how to design tests

We must define the test procedure by identifying the steps that testers should follow and the “tools” they should useWe must decide the distribution of participants with respect to

Experience, background, age, sex, ..Execution order of tasks

They need a reference guide for all participants (test monitor and external observers)Different test monitors can manage the same test in equivalent ways


Test plan: final reportTest plan: final report

It must identify all elements that are significant to evaluate the application and that must be collectedFor example

For each task of group of themAssigned time framePercentage of participants that have completed their task successfullyPercentage of participants that have done some mistakesPercentage of participants that have not been able to complete their task


Selection and recruiting of Selection and recruiting of participantsparticipants

Identification of classes of possible usersFor example: professors, expert students, novice students

Definition of the optimum sample (10-12 participants)Some participants from each class according to the frequencies in the real worldFor example

3 professors, 1 professor of computer science5 students that are used to navigate in Internet2 novice students

Definition of the minimal sample (4-5 participants)Identify the minimum number for each class

1 professor, better if not expert2 expert students1 novice student


Preparation of test materialPreparation of test material

Screening questionnaire to select participantsQuestionnaire to study their backgroundTools to collect data

What dataPerformance: measures on the behavior of the application (e.g., number of errors, mean time to answer)Preferences: opinions on the product (e.g., preferences between two possible versions)

How to classify themDeclarations to comply with privacy rulesPre-testPost-test


Test executionTest execution

IntroductionPresentation of the system and testFilling of a questionnaire to know the participants

Identification of goalsLet the participants know what we want with the test

We want to evaluate the system not the participantWe should be careful with cameras and microphones

ExecutionEach participant must have a list of tasks that must be completed in a given time frameThe test monitor oversee the process, but does not influence it


DebriefingDebriefing

We interview participants on what they did during the testIt is useful to understand the reasons of discovered problems and how to solve themWe should not judge participants and try to score themWe must try to convince participants to tell their comments


Conclusions and recommendationsConclusions and recommendations

We collect and summarize data (mean values and accuracy)We analyze data to:

Identify those tests that did not match quality criteriaIdentify the origin/nature of the problemsDiscover new problemsClassify priorities: Criticality = Severity + Probability

Severity (to be defined with the development team):

– Unusable– Hard– Mild– Not important


Analysis of resultsAnalysis of results

After executing tests, the test monitor analysis results based on questioners and discussions with participantsHe produces a report to identify problems and limits of the system (as to usability)He can define new requirementsWho is responsible for the system can decide that the system must be reworkedThe test monitor decides if and what tests must be re-executed


AccessibilityAccessibility


W3C WAIW3C WAI

The WAI – Web Accessibility Initiative defines 14 steps that must be followed to make a Web application accessibleEach step is associated with a priority level based on the impact it has on the accessibilityAccording to the priority, we have 3 certification levels (A, AA, AAA)

A : This means that the application complies with all MUST stepsAA : This means that the application complies with all MUST and SHOULD stepsAAA : This means that the application complies with all MUST, SHOULD, LIKE TO steps


PrioritiesPriorities

[Priority 1] A Web content developer must satisfy this checkpoint. Otherwise, one or more groups will find it impossible to access information in the document. Satisfying this checkpoint is a basic requirement for some groups to be able to use Web documents.

[Priority 2] A Web content developer should satisfy this checkpoint. Otherwise, one or more groups will find it difficult to access information in the document. Satisfying this checkpoint will remove significant barriers to accessing Web documents.

[Priority 3] A Web content developer may address this checkpoint. Otherwise, one or more groups will find it somewhat difficult to access information in the document. Satisfying this checkpoint will improve access to Web documents.


ExampleExample

In General (Priority 1) 1.1 Provide a text equivalent for every non-text element (e.g., via "alt",

"longdesc", or in element content). This includes: images, graphical representations of text (including symbols), image map regions, animations (e.g., animated GIFs), applets and programmatic objects, ascii art, frames, scripts, images used as list bullets, spacers, graphical buttons, sounds (played with or without user interaction), stand-alone audio files, audio tracks of video, and video.

2.1 Ensure that all information conveyed with color is also available without color, for example from context or markup.

4.1 Clearly identify changes in the natural language of a document's text and any text equivalents (e.g., captions).

6.1 Organize documents so they may be read without style sheets. For example, when an HTML document is rendered without associated style sheets, it must still be possible to read the document.

6.2 Ensure that equivalents for dynamic content are updated when the dynamic content changes.

7.1 Until user agents allow users to control flickering, avoid causing the screen to flicker.

14.1 Use the clearest and simplest language appropriate for a site's content. And if you use images and image maps (Priority 1)

1.2 Provide redundant text links for each active region of a server-side image map.

9.1 Provide client-side image maps instead of server-side image maps except where the regions cannot be defined with an available geometric shape.

http://www.w3.org/TR/WCAG10/wai-pageauth.html#tech-text-equivalent

http://www.w3.org/TR/WCAG10/wai-pageauth.html#tech-color-convey

http://www.w3.org/TR/WCAG10/wai-pageauth.html#tech-identify-changes

http://www.w3.org/TR/WCAG10/wai-pageauth.html#tech-order-style-sheets

http://www.w3.org/TR/WCAG10/wai-pageauth.html#tech-dynamic-source

http://www.w3.org/TR/WCAG10/wai-pageauth.html#tech-avoid-flicker

http://www.w3.org/TR/WCAG10/wai-pageauth.html#tech-simple-and-straightforward

http://www.w3.org/TR/WCAG10/wai-pageauth.html#tech-redundant-server-links

http://www.w3.org/TR/WCAG10/wai-pageauth.html#tech-client-side-maps


Functional correctnessFunctional correctness

Traditional testingTraditional testing


Granularity levelsGranularity levels

Acceptance testing: the software behavior is compared with end user requirementsSystem testing: the software behavior is compared with the requirements specificationsIntegration testing: checking the behavior of module cooperation.Unit testing: checking the behavior of single modulesRegression testing: to check the behavior of new releases


The test case generation problemThe test case generation problem

How to generate test data Partition testing: divide program in (quasi-) equivalence classesrandomfunctional (black box)

based on specificationsstructural (white box)

based on codefault based

based on classes of faults


White vs black boxWhite vs black box

Black boxit depends on the specification notationit scales up(different techniques at different granularity levels)it cannot reveal code bases testing (same specification implemented with different modules)

White boxit is based on control or data flow coverageit does not scale up (mostly applicable at unit and integration testing level)it cannot reveal missing path errors(part of the specification that is not implemented)


Specification-based TestingSpecification-based Testing

From formal specificationscan be automatedEXAMPLES: Test case generation from

Algebraic specificationsFinite state automata (UML class diagrams)Grammars

From semi-formal specificationspartitions can be easily identifiedcan be partially automated


Test-case Generation from Informal Test-case Generation from Informal Specifications (Natural Language)Specifications (Natural Language)

cannot be automatedsome structure (e.g., organization standards) can helpguidelines to increase confidence level and reduce discretionality: at least on test case for each:

subsets of “valid” homogeneous data“non valid” (combination of) databoundary dataspecific data (treated independently, error prone,...)


Fault-based TestingFault-based Testing

Identify a set of program locations (related to specific faults)generate alternate programs by seeding faults in the original program in the identified locationsgenerate test cases to estimate adequacy in detecting real faults from adequacy in detecting seeded faults


Partition TestingPartition Testing

Basic idea: Divide program input space into (quasi-) equivalence classes

Underlying idea of specification-based, structural, and fault-based testing


The Category-Partition Method The Category-Partition Method

STEP 1: Analyze the specification: Identify individual functional units that can be tested separately. For each unit identify:

parameters and characteristicsenvironment and characteristics

classify units into categoriesSTEP 2: Partition the categories into choicesSTEP 3: Determine constraints among the choicesSTEP 4: Write tests and documentation


The Category-Partition Method: The Category-Partition Method: an examplean example

......... *Command:

findSyntax:

find <pattern> <file>Function:

The find command is used to locate one or more instances of a given pattern in a file. All lines in the file that contain the pattern are written to standard output. A line containing the pattern is written only once, regardless of the number of times the pattern occurs in it.The pattern is any sequence of characters whose length does not exceed the maximum length of a line in the file. To include a blank in the pattern, the entire pattern must be enclosed in quotes (“). To include a quotation mark in the pattern, two quotes in a row (““) must be used.

...........

* From Ostrand, Balcer, The Category-Partition Method for Specifying and Generating Functional Tests


Step A - analyze the specification: Step A - analyze the specification: identify categoriesidentify categories

find is an individual function that can be tested separatelyparameters: pattern, filecharacteristics (pattern)

explicit (immediately derivable from specs):pattern lengthpattern enclosed in quotespattern contains blankspattern contains enclosed quotes

implicit (“hidden” in specs):quoted patterns with/without blanksseveral successive quotes included in the pattern........


Step B - partition categoriesStep B - partition categories

Parameters:Pattern size:

emptysingle charactermany characterslonger than any line in the

fileQuoting:

pattern is quotedpattern is not quotedpattern is improperly

quotedEmbedded blanks:

none oneseveral

Parameters (cont.....)Embedded quotes: none one severalFile name:

....Environment:

Number of occurrences of pattern in a file:

noneoneseveral

Pattern occurrences on target line:

....


Step C: Determine ConstraintsStep C: Determine Constraints

Parameters:Pattern size:

empty [property Empty]single character [property NonEmpty]many characters [property NonEmpty]longer than any line in the file [single]

Quoting:pattern is quoted [property Quoted]pattern is not quoted [if NonEmpty]pattern is improperly quoted [error]

........

Environment:Number of occurrence of pattern in a file:

none [if NonEmpty] [single]one [if NonEmpty] [property Match]

.....


Some Considerations onSome Considerations onthe Category Partition Methodthe Category Partition Method

a practical implementation of general principles:partition testingboundary testingerroneous conditions

other approaches with similar goals, but different procedures:

condition tablescause effect graphsequivalence partitioning


Structural Coverage TestingStructural Coverage Testing

(In)adequacy criteria If significant parts of program structure are not tested, testing is surely inadequate

Control flow coverage criteriaStatement (node, basic block) coverageBranch (edge) coverageCondition coveragePath coverageData flow (syntactic dependency) coverage

Attempted compromise between the impossible and the inadequate


Statement CoverageStatement Coverage

One test datum (N=1, A[0]=-7, X=9) is enough to guarantee statement coverage of function selectFaults in handling positive values of A[i] would not be revealed

int select(int A[], int N, int X) {

int i=0;while (i<N and A[i] <X) {

if (A[i]<0) A[i] = - A[i];

i++;}return(1);

} i++

i<N and A[i] <X

A[i]<0

A[i] = - A[i];

return(1)

TrueFalse

TrueFalse

i=0


Branch CoverageBranch Coverage

i=0

i<N and A[i] <X

A[i]<0

A[i] = - A[i];return(1)

TrueFalse

TrueFalse

We must add a test datum (N=1, A[0]=7, X=9) to cover branch False of the if statement. Faults in handling positive values of A[i] would be revealed. Faults in exiting the loop with condition A[i] <X would not be revealed



if (A[i]<0) A[i] = - A[i];

i++;}return(1);

} i++


Condition CoverageCondition Coverage

i=0

i<N and A[i] <X

A[i]<0

A[i] = - A[i];return(1)

TrueFalse

TrueFalse

Both conditions (i<N), (A[i]<X) must be false and true for different tests. In this case, we must add tests that cause the while loop to exit for a value greater than X. Faults that arise after several iterations of the loop would not be revealed.



if (A[i]<0) A[i] = - A[i];

i++;}return(1);

} i++


Path CoveragePath Coverage

i=0

i<N and A[i] <X

A[i]<0

return(1)

TrueFalse

TrueFalse

The loop must be iterated given number of times.PROBLEM: uncontrolled growth of test sets. We need to select a significant subset of test cases.



if (A[i]<0) A[i] = - A[i];

i++;}return(1);

} i++;

A[i] = - A[i];


Data Flow CoverageData Flow Coverage

DEF={i}

USE={i,N,A,X}

USE={A,i}

USE={i}DEF={i}

TrueFalse

True

False

Exercise Def-Use paths: selects paths based on effects on the variables, rather than number of iteration of loops



if (A[i]<0) A[i] = - A[i];

i++;}return(1);

} USE={A,i}DEF{A}

DEF={A,N,X}


The Infeasibility ProblemThe Infeasibility Problem

Syntactically indicated behaviors (paths, data flows, etc.) are often impossible

Infeasible control flow, data flow, and data statesAdequacy criteria are typically impossible to satisfy Unsatisfactory approaches:

Manual justification for omitting each impossible test case (esp. for more demanding criteria)Adequacy “scores” based on coverage

example: 95% statement coverage, 80% def-use coverage


Regression TestingRegression Testing

Testing a new version (release): how can we minimize effort using results of testing of previous versions?On a previous release:

save scaffoldings (drivers,stubs,oracles)record test cases (<inputs,outputs>)

On the new release:keep track of changesevaluate impact of changes


Create ScaffoldingCreate Scaffolding

initialization of non-local variablesinitialization of parameters

activation of the unit

DRIVER

“templates” of modules used by the unit (functions called

by the unit)“templates” of any other entity used by the unit

STUB

PROGRAM UNIT

ORACLE

check the correspondence between

the produced and the

expected result


Problems and TradeoffsProblems and Tradeoffs

effort in developing drivers/stubs

effort in test execution and regression testing

poorly designed drivers/stubs

well designed drivers/stubs

high effort in test execution and regression testing

high effort in development

low effort in test execution and regression testing

low effort in development


What is an oracle?What is an oracle?

1324535968....

An “Inspector” of executions:do test executions produce

acceptable results?

An oracle can be:human being

machinea former version of the same program

another program.....


What is a good oracle?What is a good oracle?

Testing large, complex applications may require millions of test runs

ewfcdgvr09-4tfewv95gff9btbfd766dv0

erfjmvesf78-6rsdvv45fvf6bfdfb666gf7

gygirfnm55-8nyfng43nhynhum001bn4

hnhnghn88-7hgmh11vfnnhnhm888hh6

ewfcdgvr09-4tfewv95gff9btbfd766dv0

erfjmvesf78-6rsdvv45fvf6bfdfb666gf7

gygirfnm55-8nyfng43nhynhum001bn4

hnhnghn88-7hgmh11vfnnhnhm888hh6

................

................

...........

The size of the outputs to be

inspected exceed the capabilities of

human eyes

human eyes are slow and unreliable examiners even of small number of outputs

AUTOMATED ORACLES ARE ESSENTIAL!AUTOMATED ORACLES ARE ESSENTIAL!


How can we build acceptable How can we build acceptable oracles? oracles?

There is NO universal recipe

application domains development environments development phases

• GUI• protocols• .....

• no specifications• informal specifications• formal specifications• .......• system testing

• regression testing• ........

Different solutions for different


Oracles from DesignOracles from Design

Example: UML design notationsMessage sequence charts

A UML message sequence chart indicates a test case and expected outcome, which can be interpreted by a driver and oracle Typical of “scenario-based” oracles

scenarios combine test case with special oracleStateChart (finite state acceptor)

A UML finite state machine describes all permissible behaviors of a module

oracle can be used with large numbers of automatically generated test cases


Oracles from Code DocumentationOracles from Code Documentation

Parnas’ tabular annotations precisely describe the functional behavior of the unit. The table can be evaluated with respect to the produced outputs to check for their correctness .

* from: Parnas, Madey, Iglewski, Precise Documentation of Well-Structured Programs, IEEE-TSE Vol.. 20 N. 12 Dec 1994

*DISPLAY 1 Display 1 Specification

Find(x,A,j,present)

R0(,) = ((1n) and [for all (1in) ‘A[i] ‘A[i+1]])

present’=

j’ |

i[(1in) and (‘A[i]=‘x)] =

true false

‘A[j]=‘x true

true false and NC(x,A)

Display 1 Programprocedure find (...)..........end {find}

Display 1 Specifications of the Invoked Programs

..........


Harness vs. Embedded AssertionsHarness vs. Embedded Assertions

Driver

Stubs

Unit or Subsystem

OracleOracleDriver

Stubs

Unit or Subsystem

Oracle

Oracle

Oracle

Oracle

Oracle

Oracle

Embedded assertions act as oracles within the unit under test


Assertions as OraclesAssertions as Oracles

/* * Alphabetic sort of an array of strings */void sort( char *words[ ], int nwords ) {

. . . assert( is_sorted(words, nwords) ); return; }


Another example: HttpUnitAnother example: HttpUnit

The main class is WebConversationIt is a browser that interacts with a server

With this class we can buid several interactions

WebConversation wc = new WebConversation();

WebResponse resp = wc.getResponse("http://httpunit.sourceforge.net/doc/Cookbook.html");

WebLink link = resp.getLinkWith("response");


An exampleAn example

Check

Check


An exampleAn example

Check

! Check password


AnalysisAnalysis


Software Inspection:Software Inspection:Low tech but effectiveLow tech but effective

Fagan Code InspectionsOne of many “walk-through” and inspection techniques; among the most successful

More formal and well-defined than “structured walk-throughs” etc.

Has been extended to designs, requirements, etc. with similar organizing principlesA completely manual technique for finding and correcting errors


Software Inspection RolesSoftware Inspection Roles

Moderator: Typically borrowed from another project. Chairs meeting, chooses participants, controls process

Readers, Testers:Read code to group, look for flaws

Author:Passive participant; answer questions when asked


Software Inspection ProcessSoftware Inspection Process

PlanningModerator checks entry criteria, choose participants, schedule meeting

OverviewProvide background education, assign roles

PreparationInspection (see ahead)ReworkFollow-up (& possible re-inspection)


In the MeetingIn the Meeting

Goal: Find as many faults as possiblemax 2 x 2 hour sessions per dayapprox. 150 source lines/hour

Approach: Line-by-line paraphrasingReconstruct intent of code from sourceMay also “hand test”

Find and log defects, but don’t fix themModerator responsible for staying on track


Checklists — NASA exampleChecklists — NASA example

About 2.5 pages for C code, 4 for FORTRANDivided into: Functionality, Data Usage, Control, Linkage, Computation, Maintenance, Clarity

Examples:Does each module have a single function?Does the code match the Detailed Design?Are all constant names upper case?Are pointers not typecast (except assignment of NULL)?Are nested “INCLUDE” files avoided? Are non-standard usages isolated in subroutines and well documented?Are there sufficient comments to understand the code?


Inspection AutomationInspection Automation

Although a manual technique, many kinds of automated support are possible:

Automate trivial checks (e.g., formatting)Reference: Checklists, standards w/ examplesFocus (highlight, selection) on relevant partsAnnotation & CommunicationProcess guidance and (partial) enforcement

e.g., InspeQ will not allow check-off until all relevant parts of a document have been observed


Why does inspection work?Why does inspection work?

The evidence says it is cost-effective. Why?Detailed, formal process, with record keepingCheck-lists; self-improving processSocial aspects of process, esp. for authorConsideration of whole input spaceApplies to incomplete programs

LimitationsScale: Inherently a unit-level techniqueNon-incremental; what about evolution?


Data flow analysisData flow analysis

if (a < b)

tmp := a; a := b; b := tmp;

absdiff := a - b;

function absdiff (a, b: integer)

return integer is

tmp: integer;

begin

if (a < b) then

tmp := a;

a := b;

b := tmp;

end if;

return ( a - b );

end absdiff;


Classic data flow analyses to find Classic data flow analyses to find program errorsprogram errors

Uninitialized variable“May” result from classic “avail” analysis

but conservative analysis can be annoying“Must” version is also possible (how?)

Dead assignment (no possible use)Classic “live variables” analysis In FORTRAN, Awk, BASIC, PERL, etc., usually indicates a misspelled variableless useful in languages requiring declarations


Precision & SafetyPrecision & Safety

An analysis is conservative (safe) if it doesn’t miss errors An analysis is precise to the extent that it doesn’t report spurious errorsStatic flow analysis considers all (syntactic) program paths; it can be conservative or precise, but not both

An overly conservative, imprecise analysis may be useless.A well-defined but overly strict property may be preferable to spurious error reports


Analysis of Models: Analysis of Models: State-Space ExplorationState-Space Exploration

Concurrency (multi-threading, distributed programming, ...) makes testing harder

introduces non-determinism; time- and load-dependent bugs escape extensive testing

Finite-state models can be exhaustively verified

accept E do ... ...

-E

?E

!E

Extract Combine Check

E


Automated Finite-State VerificationAutomated Finite-State Verification

Example tool SPIN (one of many)

verifies simple program-like design modelhigh-level design of process interaction, ignoring other aspects of computation (e.g., functional behavior)

used for protocols, OS scheduling, ... useful despite limited capacity; best for verifying high-level design before coding

Domain-specific analysislimited “proof” of simple but critical properties in a limited domain

G. Holzmann, “The model checker SPIN.” IEEE TSE 23(5), May 1997

G. Holzmann, “The model checker SPIN.” IEEE TSE 23(5), May 1997


State explosion is one face of a (provably) hard problem. The same fundamental limits appear in different form for

non-enumerative analyses

State explosion problemState explosion problem

Size of composite state graph is product of individual state graphs.

OK for a simple two-party protocol, but impossibly expensive for systems with many processes

Brute force state enumeration is limited to a few processes


What is static analysis good for?What is static analysis good for?

Not a replacement for testingfocused, (mostly) automated analysis for limited classes of faults

More thorough than testing (within scope)conservative analyses are tantamount to formal verification

Also augments testing, e.g., dependence analysis for data flow testing


Load and performance testLoad and performance test


Robustness and scalabilityRobustness and scalability

Robustness: It is the capability of behaving “decently” even in cases not explicitly considered in the requirements definition document

If we consider Web applications, the number of users is the key factor

Scalability: It is the capability of serving a given load, along with a predefined QoS, and being able to adapt to the evolutions of the load


An example loadAn example load problem problem

Yahoo was attacked by sending thousands of emails and the servers collapsed


ApproachesApproaches

We can analyze the architecture to discover problems in its components or in the way they interact

We can verify their design and interactionsWe can produce models

We can test the application (load test)


Load testLoad test

We simulate a given load on the server by means of virtual users

These users behave like real users and test the capability of the system to support the loadWe must have the application before being able to work on load testing

With this test, we can Study the HW/SW configuration needed to support a given loadDiscover the maximum load we want offer with the current configuration


HowHow

Manual testsWe use employees during the week-endsIt is really expensive and we cannot simulate exceptional situationsWe need millions of users for Web applications

Automatic tests (Mercury, Rational, Empirix):We do not need too many employeesWe can perform what-if simulationsWe can better collect statistics

Available tools record the activities done by real tools, create the scripts that virtual users should execute, and measure the mean time to answerScripts can be parametric and configurable and use a DB to store and retrieve data and generate coherent use cases


Astra load test (Mercury)Astra load test (Mercury)

It can emulate thousands of users and provides a graphical monitor to identify and isolate problemsIt allows users to record browsing/interaction sessionsIt allows users to fill forms in a parametric way

San FranciscoPortlandAcapulco


Astra load testAstra load test

The test monitor allows us to

Define the number of virtual usersIdentify the tests that we want to execute and are already stored in the right formatThe machines on which we want to execute the test

The monitor also displays the results even during the execution of the tests


PerformancePerformance


PerformancePerformance

We should always consider our clients before designing the application

Its weight basically should depend on them

We should carefully plan performance testing at the beginning of our project

Our requirements should clearly state what we wantWe must clearly state pass and fail conditions

A transaction that takes too longs can be seen as an errorWhat is the meaning of “too long”?


What should we test? Where?What should we test? Where?

We must identify those cases are we think can be critical

Trivial problems become important on the server because of the degree of parallelismDifferent applications have different criticalities

In client-server like architectures we should always identify the bottleneck

Is the server slow?Is available bandwidth limited on the client side?Are CPU and memory not sufficient?

Usually, we mainly consider problems on the server sideBefore starting we must clearly know the HW/SW architecture


Three levelsThree levels

Ad hoc performance testingTesters validate if the application answers with a reasonable delay. They may record failures, but they do not locate them

Observational testingWe use the first measures, but we use watches and similar tools

Measured testingWe use objective measures and specific tools


What do we measure?What do we measure?

Measures on the serverMegacycles (MCs)Memory footprint

Measures that consider the networkTime to last byte (TTLB)User-perceived response time

Other measuresBytes over the wire (BoW)


Megacycles (MCs)Megacycles (MCs)

CPU cost = (CPU usage * number of CPUs * CPU speed in MHz) / requests per secondThis measure defines the computation cost on the CPU

If I know this cost and the number of users, I can foresee the number of CPUs I need

When I add a new CPU, at most it gives an improvement of .8 because of the hardware it shares (bus, ram, ...)


ExampleExample

I have two bi-processor Web servers that work at 400 MHzTheir usage is equal to the 60% They must server 30 requests per second

The mean value of each measure is: Available MCs =(1+0.8)+(1+0.8)*400= 1440Used MCs =1440*0.60= 864MCs used for each request =864/30= 28.8Requests that we can serve per second =1440/28.8= 50


Other measuresOther measures

Memory footprintIt is used to study the maximum usage of memoryIt is not interesting if we have only static pages and imagesIt is interesting when we have dynamic pages, accesses to DBs, and code that uses memory (Java and C#)

Time to last byte (TTLB)It measures the time from the instant at which the request leaves the client till the server sends the last byteIt does not consider the time used by the client to execute what receives (to execute a script, render a page)If the “execution” is complex and time-consuming, the user can have significant delays even if the TTLB is low


Other measuresOther measures

User-perceived response timeIt is the time needed to fully load a page on the client and have it ready to interact with the userIt is usually greater than TTLB since we must consider the overhead necessary after the last byte

Bytes over the wire (BoW)It counts the number of bytes moved between client and server. It distinguished between:

First request: the cache is emptyFurther request: the cache contains some data and we must transmit only what changes (dynamic data)


Model-based validationModel-based validation

Can provide quick feedback during the designAllow for the tuning of resources

page

Client

Internal Web

Internal LAN

Internal LAN Internet

ReqReq

Req

new page

page

pageReq


Performance testingPerformance testing

We need testing toolsFor example: E-Test (Empirix)

We create a script that corresponds to a given browsing/interaction session

To measure the performance of the server with respect to the number of usersTo verify the performance of each single component and identify bottlenecks

Kb/sec

Pages/sec

Transactions/secMemory

usage by the DB

CPU usage by the DB


Testing services through the WebTesting services through the Web

We can use some applications that allow us to test our applications without installing anything on our machinesFor example: www.netmechanic.comThey work on the load time for each page and on the “weight” of each page or componentLoad Time 38.71 seconds, height/width

problems

Load Time by Modem Speed Modem Speed Download Time 14.4k 75.42 seconds 28.8k 38.71 seconds 56k 20.68 seconds ISDN (128k) 10.26 seconds T1 (1.44 MB) 2.73 seconds

Size Object URL

78698 HTML http://www.tiscali.it/

6743 IMG http://www.tiscali.it/magazine/Italy/spettacoli/mytv/foto/ gino_maggio_2.jpg

5591 IMG http://www.tiscali.it/magazine/Italy/eventi/eventi/Media/Foto/2002/ aprile/19/junior_spazio.jpg

5246 IMG http://www.tiscali.it/magazine/Italy/eventi/eventi/Media/Foto/2002/ maggio/2/foto_epoca.jpg

4179 SCRIPT http://www.tiscali.it/inc/hp/cookie.js

Original: JPEG Size: 6743 BytesH: 130W: 150DL Time (28.8): 0:02

Type: JPEGQuality: 30Size: 4170 bytesDL Time (28.8): 0:01Savings: 38 %

•You may want to break this Web page into several smaller pages. •Click on an image's file size to reduce it with GIFBot. •Adding HEIGHT and WIDTH attributes to your images will help browsers display your page sooner. •Adding WIDTH attributes to your TABLE tags will help browsers display your page sooner. •Your page's overall rating was lowered one level because of HTML problems. •Correct the HTML problems to get our highest rating.

http://www.netmechanic.com/

http://results.netmechanic.com/aps5/toolbox/load_time.cgi?f=121/1140/53-10366&p=1&vi=0&fp=1&so=&fv=5&s=NetMechanic&t=Load+Time

http://www.netmechanic.com/cgi-bin/nph-job.cgi?url=http://www.tiscali.it/magazine/Italy/spettacoli/mytv/foto/gino_maggio_2.jpg&site_name=NetMechanic&outputtype=1&robot=gifbot

http://www.netmechanic.com/cgi-bin/nph-job.cgi?url=http://www.tiscali.it/magazine/Italy/eventi/eventi/Media/Foto/2002/aprile/19/junior_spazio.jpg&site_name=NetMechanic&outputtype=1&robot=gifbot

http://www.netmechanic.com/cgi-bin/nph-job.cgi?url=http://www.tiscali.it/magazine/Italy/eventi/eventi/Media/Foto/2002/maggio/2/foto_epoca.jpg&site_name=NetMechanic&outputtype=1&robot=gifbot

http://www.tiscali.it/magazine/Italy/spettacoli/mytv/foto/gino_maggio_2.jpg

http://results.netmechanic.com/aps3/gifbot/nph-saveimage.cgi?site_name=NetMechanic&url=http://results.netmechanic.com/aps3/GIFbot/gino_maggio_2.jpg12112071490026_30.jpg

http://www.tiscali.it/magazine/Italy/spettacoli/mytv/foto/gino_maggio_2.jpg

http://results.netmechanic.com/aps3/gifbot/nph-saveimage.cgi?site_name=NetMechanic&url=http://results.netmechanic.com/aps3/GIFbot/gino_maggio_2.jpg12112071490026_30.jpg


Security testSecurity test

????????


ProcessProcess


Software Qualities and ProcessSoftware Qualities and Process

Qualities cannot be added after developmentQuality results from a set of inter-dependent activitiesAnalysis and testing are crucial but far from sufficient.

Testing is not a phase, but a lifestyleTesting and analysis activities occur from early in requirements engineering through delivery and subsequent evolution. Quality depends on every part of the software process

An essential feature of software processes is that software test and analysis is thoroughly integrated and not an afterthought


The Quality ProcessThe Quality Process

Quality process: set of activities and responsibilitiesfocused primarily on ensuring adequate dependability concerned with project schedule or with product usability

The quality process provides a framework for selecting and arranging activities considering interactions and trade-offs with other important goals.


Interactions and tradeoffsInteractions and tradeoffs

Example: high dependability vs. time to marketMass market products:

Better to achieve a reasonably high degree of dependability on a tight schedule Than to achieve ultra-high dependability on a much longer schedule

Critical medical devices:Better to achieve ultra-high dependability on a much longer scheduleThan a reasonably high degree of dependability on a tight schedule


Properties of the Quality ProcessProperties of the Quality Process

Completeness: appropriate activities are planned to detect each important class of faults. Timeliness: faults are detected at a point of high leverage (as early as possible)Cost-effectiveness: activities are chosen depending on cost and effectiveness

cost must be considered over the whole development cycle and product lifethe dominant factor is likely to be the cost of repeating an activity through many change cycles.


Planning and MonitoringPlanning and Monitoring

The quality process balances several activities across the whole development processselects and arranges them to be as cost-effective as possibleimproves early visibility

Quality goals can be achieved only through careful planningPlanning is integral to the quality process


Process VisibilityProcess Visibility

A process is visible to the extent that one can answer the question

How does our progress compare to our plan?‘Are we on schedule? How far ahead or behind?

The quality process has not achieved adequate visibility if one cannot gain strong confidence in the quality of the software system before it reaches final testing

quality activities are usually placed as early as possible

design test cases at the earliest opportunity (not “just in time'') uses analysis techniques on software artifacts produced before actual code


A&T StrategyA&T Strategy

Identifies company- or project-wide standards that must be satisfied

procedures required for obtaining quality certificates techniques and tools that must be useddocuments that must be produced


Quality planQuality plan

a comprehensive description of the quality process that includes:

objectives and scope of quality activitiesdocuments and other items that must be available items to be testedfeatures to be tested and not to be testedanalysis and test activities staff involved in qualityconstraintspass and fail criteriascheduledeliverableshardware and software requirementsrisks and contingencies


Improving the ProcessImproving the Process

Long lasting errors are commonIt is important to structure the process for

Identifying the most critical persistent faultstracking them to frequent errorsadjusting the development and quality processes to eliminate errors

Feedback mechanisms are the main ingredient of the quality process for identifying and removing errors


Organizational factorsOrganizational factors

Different teams for development and quality?separate development and quality teams is common in large organizationsindistinguishable roles is postulated by some methodologies (extreme programming)

Different roles for development and quality?test designer is a specific role in many organizationsmobility of people and roles by rotating engineers over development and testing tasks among different projects is a possible option


Example of Allocation of Example of Allocation of ResponsibilitiesResponsibilities

Allocating tasks and responsibilites is a complex job: we can allocate

Unit testing to the development team (requires detailed knowledge of the code)but the quality team may control the results (structural coverage)

Integration, system and acceptance testing to the quality teambut the development team may produce scaffolding and oracles

Inspection and walk-through to mixed teams

Regression testingto quality and maintenance teams

Process improvement related activities to external specialists interacting with all teams


Product vs. process improvementProduct vs. process improvement

Product improvement: Fault is detected (by inspection, testing, user report, ...) Fault is diagnosed and repaired

Process improvementFaults are detected (and maybe repaired)Fault record is analyzed to tune process


Fault analysisFault analysis

What are the faults?Categorize by kind (Memory leak, interface error, mis-feature, etc.)And by severity

When did they occur? And when found?Coding? Design? Requirements?

Why did they occur? Look for “root causes”

How could they be prevented?


Categorizing faultsCategorizing faults

There is no “right” categorizationMay depend on design style, implementation language, process and documents, ... Should probably be revised occasionally

Goal is enough precision for “Pareto” analysis (80/20 rule) considering severity and cost

Categorization needn’t be perfect or painful, but keeping records is essential


Fault severityFault severity

Typical breakdown (of failures): Critical: Product is unusableSevere: Product feature cannot be used; no workaroundModerate: Product feature can be used only with workaround (loss of efficiency, reliability, or significant loss of convenience)Cosmetic or minor inconvenience

Cost may be distinct from severity


80/20 rule (a.k.a. Pareto analysis)80/20 rule (a.k.a. Pareto analysis)

Identify one or two “dominant” fault categoriesConsidering severity, cost, and frequencyFurther problem analysis is limited to these

Categories may “level” over timeA good time for rethinking the categories


Test DocumentationTest Documentation

Must be an organization standardDepends on

organization (size,turnover,..), type of software (criticality, average life, complexity, number of versions,...)

It must include at least:test suite documentationtest case documentation


Test Documentation (cont....)Test Documentation (cont....)

Documentation of test suites:software testedversiongoaloverall resultsauthor

Documentation of Test Cases:goal“environment” (driver,stub,oracle)inputexpected outputactual outputresultobservations


ToolsTools


ToolsTools

Test process managementValidator/ReqTest DirectorRational RobotMercury WinRunner Segue SilkTest Compuware QA Run

UsabilityWebSATBobbyPage ValetW3C HTML Validator ServiceHTML Authoring ServiceXenuAlert LinkRunnerHTML Link ValidatorHTML Validator 5.0Web Link Validator


ToolsTools

PerformanceWeb Performance TrainerHttploadWebserver Stress ToolHttp/s Load OpenSTAjMeter

ScaffoldingATTOL Unit Test (NOW Rational)Cantata++jUnitCactusHTTPunit

CoverageLogiscope TestCeckerDeep CoverjCoverLDRA TestBedAttol Coverage (Rational)

Memory leaksPurify (Rational)Sentinel

Static analysisLogiscope AuditLDRA TestBed


ReferencesReferences


Web sitesWeb sites

www.w3c.orgwww.useit.comwww.softwareqatest.com/qatweb1.htmlwww.junit.org/index.htmwww.swquality.com/users/pustaver/index.shtmlstandards.ieee.org/catalog/olis/se.htmlwww.stickyminds.comwww.qaforums.comwww.qualitytree.com

www.cs.uoregon.edu/~michal/book/


BooksBooks

UsabilityJakob Nielsen, “Designing Web Usability: The Practice of Simplicity”, New Riders Publishing, Indianapolis, 2000

Software engineeringIan Sommerville, "Software Engineering". Addison-Wesley, 2000Roger S. Pressman, "Software Engineering: A Practitioner's Approach". McGraw-Hill, 2000

Test in generalCem Kaner, Hung Quoc Nguyen, Jack Falk, "Testing Computer Software", John Wiley & Sons, 1999


BooksBooks

Test for object-oriented softwareRobert V. Binder, "Testing Object-Oriented Systems: Models, Patterns, and Tools", Addison-Wesley, 1999

Software metricsNorman E. Fenton, Shari Lawrence Pfleeger, "Software Metrics: A Rigorous and Practical Approach, Revised", Brooks/Cole, 1998Stephen H. Kan, "Metrics and Models in Software Quality Engineering", Addison-Wesley, 1995


Grazie !!!!Grazie !!!!

Luciano BaresiLuciano BaresiDEI - Politecnico di MilanoDEI - Politecnico di Milano

Piazza L. da Vinci, 32 - 20133 Milano (Italia)Piazza L. da Vinci, 32 - 20133 Milano (Italia)tel: 02 2399 3638 tel: 02 2399 3638

email: [email protected]: [email protected]

www.elet.polimi.it/~baresiwww.elet.polimi.it/~baresi


HomeworkHomework

Study and analyze one of the arguments presented in these days

Some particular methodologiesSome interesting examples…

Imagine you are a test manager and try to identify the activities that should be carried out to test a Web application

For example, www.amazon.comConsider what, how, and whenTry to estimate costs in terms of person months

Constraints10 days5 pages (at most)

http://www.amazon.com/

Politecnico di Milano Web quality Luciano Baresi Politecnico di Milano – Dipartimento di...

Documents

Transcript of Politecnico di Milano Web quality Luciano Baresi Politecnico di Milano – Dipartimento di...