Software Tool to Assist Scenario Testing of Website

Student: Siqi Fan

Supervisor: Assoc. Prof. Christopher Johnson

Presented on November 4th, 2010

Agenda

• Problem • Analysis • Idea • Result • Conclusion

Large Scale Business Oriented Website

Website More

Contents

More

Features

More

Languages

www.office.com

www.apple.com

www.honda.com

www.dell.com

…

During the production, there are tremendous amount of defects found…

Tester

Manager

Developer

? Which defect should I fix first?

Severity

Priority Defect

Defect Defect Report

… Release Schedule

When the scheduled releasing date is approaching, there are still many defects not fixed…

Manager

? Could the defect be published to the live website?

Severity

Priority Defect

Defect Defect Report

Release Schedule

Business Value

… Developing and Testing Resources

Must-Fix

Bypass

Problem: Ineffective Prioritization

The critical defects do no receive the higher priority because something wrong with the reference factor Severity: • No difference between bugs with same

severity level • Lack of big picture

More distinguishable

More objectively

Severity is an important factor for prioritization, but what is it about?

IEEE standard 1044-2009: “Severity is the highest failure impact that the defect could (or did) cause, as determined by (from the perspective of) the organization responsible for software engineering”

Severity is an important factor for prioritization, but what is it about to a large scale business oriented website?

IEEE standard 1044-1993: “The impact of an anomaly shall be considered at each step of the anomaly process.”

Using a scenario is probably the best way to describe the process.

functional and non-functional requirements

systematic overview of the website

Scenarios

Case Study: Consider impact on scenario

Considering impact on scenario is not enough for measuring severity.

Website Quality

Two aspects of website quality

Website Quality

Customer Website Provider GAP

How to narrow?

The order relationship between quality factors

Finance Page 1. Currency / Timeliness /

Update 2. Completeness /

Comprehensiveness of Information

3. Navigation 4. Accuracy 5. Readability /

Comprehension / Clarity

Entertainment Page 1. Visual design 2. Engaging 3. Navigation 4. Information representation 5. Site accessibility /

Responsiveness

Case Study: Consider importance of quality factors

Shall we also consider the usage?

YES!

Likelihood = Page Trip Rate × Hit Rate Statistical Analysis on the historical data

How to represent Severity in a more distinguishable form? and How to evaluate Severity objectively in a systematic manner?

Framework

Framework

Framework

Framework

Framework

Classification Scheme and Representation

Impact 1~5 Importance 1~5 Likelihood 1~5

𝐒𝐞𝐯𝐞𝐫𝐢𝐭𝐲 = Impact + Importance + Likelihood

ImpactMax + ImportanceMax + LikelihoodMax

Prototype

Prototype

Conclusion

The prototype is built to prove the feasibility to adopt the framework into the practical scenario testing tools

It enables the severity of defects to be set in a systematic manner and be represented in a more distinguishable form so that severity can provides the greatest reference value for effective defect prioritization.

It lights up a direction for website design and website testing research (for software engineering?) for quality delivery and quality assurance.

Future Work

• Extend the content of Quality Dimension , Quality Feature Clusters, Deliverables, and Testing Tasks

• Prove in the practical projects • Refine the severity classification scheme and

representations

𝐒𝐞𝐯𝐞𝐫𝐢𝐭𝐲 = ω1 × Impact + ω2 × Importance + ω3 × Likelihood

ImpactMax + ImportanceMax + LikelihoodMax

• Refine the severity score aggregation method

Acknowledgement First and foremost I would like to offer my sincerest gratitude to my supervisor, Assoc Professor Chris Johnson, who has supported me throughout of my thesis with his patience and knowledge. In the various research works and laboratories along the entire project, I have been also aided in setting the research direction by him. Without Chris Johnson’s kind support and encouragement this thesis would not have been possible. Secondly, I would like to express my thankfulness to all my previous colleagues in the Office International Publishing Group (OIPG), Microsoft Ireland. It was them who created the heuristic work environment and offered me the opportunity to receive the fundamental knowledge and experience of a large scale business oriented website production. Finally, I would like to thank my beloved parents, wife and friends for their understanding and endless love throughout all my studies.

Appendix

Appendix

Appendix

Appendix

Appendix

Appendix

Appendix

Appendix Impact is to measure how the defect affects the scenario. The following list is suggested for ranking impact: • Level 0: Not set • Level 1: The defect impairs at least the third level scenario and can be skipped

without worries in the entire scenario. There is at least one easy work around. The user can go further with no difference in the scenario. The issue is low and cosmetic.

• Level 2: The defect impairs at least the third level scenario and can be skipped with notice by the user. There is at least one work around. The user can go further but it will be better to have the defect fixed. The issue is enhancement.

• Level 3: The defect impairs the dependent (secondary) scenario and can be skipped with worries in the entire scenario. There is at least one work around. The user can go further with mind in the scenario. The issue is minor.

• Level 4: The defect impairs the key (primary) scenario and can be skipped with a lot of troubles in the entire scenario. There is at least one unexpected work around. The user can hardly go further in the scenario. The issue is major.

• Level 5: The defect blocks the entire scenario. There is no work around available. The user cannot go further in the scenario. The issue is fatal.

Appendix Importance is to measure how important the quality feature cluster or quality dimension in the scenario. The following list is suggested for ranking importance: • Level 0: Not set • Level 1: Negligible. The quality factor or the quality feature cluster or the

quality dimension will not affect the overall website quality. • Level 2: Acceptable. The quality factor or the quality feature cluster or the

quality dimension will slightly affect the overall website quality. • Level 3: Remarkable. The quality factor or the quality feature cluster or the

quality dimension will moderately affect the overall website quality. • Level 4: Significant. The quality factor or the quality feature cluster or the

quality dimension will strongly affect the overall website quality. • Level 5: Critical. The quality factor or the quality feature cluster or the quality

dimension will dominantly affect the overall website quality.

Appendix

Likelihood is to measure the product value of page trip rate and hit rate. The following list is suggested for ranking likelihood: • Level 0: Not set • Level 1: 0% - 20% • Level 2: 20% - 40% • Level 3: 40% - 60% • Level 4: 60% - 80% • Level 5: 80% - 100%