Post on 29-May-2020
Software Tool to Assist Scenario Testing of Website
Student: Siqi Fan
Supervisor: Assoc. Prof. Christopher Johnson
Presented on November 4th, 2010
Agenda
• Problem • Analysis • Idea • Result • Conclusion
Large Scale Business Oriented Website
Website More
Contents
More
Features
More
Languages
www.office.com
www.apple.com
www.honda.com
www.dell.com
…
During the production, there are tremendous amount of defects found…
Tester
Manager
Developer
? Which defect should I fix first?
Severity
Priority Defect
Defect Defect Report
… Release Schedule
When the scheduled releasing date is approaching, there are still many defects not fixed…
Manager
? Could the defect be published to the live website?
Severity
Priority Defect
Defect Defect Report
Release Schedule
Business Value
… Developing and Testing Resources
Must-Fix
Bypass
Problem: Ineffective Prioritization
The critical defects do no receive the higher priority because something wrong with the reference factor Severity: • No difference between bugs with same
severity level • Lack of big picture
More distinguishable
More objectively
Severity is an important factor for prioritization, but what is it about?
IEEE standard 1044-2009: “Severity is the highest failure impact that the defect could (or did) cause, as determined by (from the perspective of) the organization responsible for software engineering”
Severity is an important factor for prioritization, but what is it about to a large scale business oriented website?
IEEE standard 1044-1993: “The impact of an anomaly shall be considered at each step of the anomaly process.”
Using a scenario is probably the best way to describe the process.
functional and non-functional requirements
systematic overview of the website
Scenarios
Case Study: Consider impact on scenario
Considering impact on scenario is not enough for measuring severity.
Website Quality
Two aspects of website quality
Website Quality
Customer Website Provider GAP
How to narrow?
The order relationship between quality factors
Finance Page 1. Currency / Timeliness /
Update 2. Completeness /
Comprehensiveness of Information
3. Navigation 4. Accuracy 5. Readability /
Comprehension / Clarity
Entertainment Page 1. Visual design 2. Engaging 3. Navigation 4. Information representation 5. Site accessibility /
Responsiveness
Case Study: Consider importance of quality factors
Shall we also consider the usage?
YES!
Likelihood = Page Trip Rate × Hit Rate Statistical Analysis on the historical data
How to represent Severity in a more distinguishable form? and How to evaluate Severity objectively in a systematic manner?
Framework
Framework
Framework
Framework
Framework
Classification Scheme and Representation
Impact 1~5 Importance 1~5 Likelihood 1~5
𝐒𝐞𝐯𝐞𝐫𝐢𝐭𝐲 = Impact + Importance + Likelihood
ImpactMax + ImportanceMax + LikelihoodMax
Prototype
Prototype
Conclusion
The prototype is built to prove the feasibility to adopt the framework into the practical scenario testing tools
It enables the severity of defects to be set in a systematic manner and be represented in a more distinguishable form so that severity can provides the greatest reference value for effective defect prioritization.
It lights up a direction for website design and website testing research (for software engineering?) for quality delivery and quality assurance.
Future Work
• Extend the content of Quality Dimension , Quality Feature Clusters, Deliverables, and Testing Tasks
• Prove in the practical projects • Refine the severity classification scheme and
representations
𝐒𝐞𝐯𝐞𝐫𝐢𝐭𝐲 = ω1 × Impact + ω2 × Importance + ω3 × Likelihood
ImpactMax + ImportanceMax + LikelihoodMax
• Refine the severity score aggregation method
Acknowledgement First and foremost I would like to offer my sincerest gratitude to my supervisor, Assoc Professor Chris Johnson, who has supported me throughout of my thesis with his patience and knowledge. In the various research works and laboratories along the entire project, I have been also aided in setting the research direction by him. Without Chris Johnson’s kind support and encouragement this thesis would not have been possible. Secondly, I would like to express my thankfulness to all my previous colleagues in the Office International Publishing Group (OIPG), Microsoft Ireland. It was them who created the heuristic work environment and offered me the opportunity to receive the fundamental knowledge and experience of a large scale business oriented website production. Finally, I would like to thank my beloved parents, wife and friends for their understanding and endless love throughout all my studies.
Appendix
Appendix
Appendix
Appendix
Appendix
Appendix
Appendix
Appendix Impact is to measure how the defect affects the scenario. The following list is suggested for ranking impact: • Level 0: Not set • Level 1: The defect impairs at least the third level scenario and can be skipped
without worries in the entire scenario. There is at least one easy work around. The user can go further with no difference in the scenario. The issue is low and cosmetic.
• Level 2: The defect impairs at least the third level scenario and can be skipped with notice by the user. There is at least one work around. The user can go further but it will be better to have the defect fixed. The issue is enhancement.
• Level 3: The defect impairs the dependent (secondary) scenario and can be skipped with worries in the entire scenario. There is at least one work around. The user can go further with mind in the scenario. The issue is minor.
• Level 4: The defect impairs the key (primary) scenario and can be skipped with a lot of troubles in the entire scenario. There is at least one unexpected work around. The user can hardly go further in the scenario. The issue is major.
• Level 5: The defect blocks the entire scenario. There is no work around available. The user cannot go further in the scenario. The issue is fatal.
Appendix Importance is to measure how important the quality feature cluster or quality dimension in the scenario. The following list is suggested for ranking importance: • Level 0: Not set • Level 1: Negligible. The quality factor or the quality feature cluster or the
quality dimension will not affect the overall website quality. • Level 2: Acceptable. The quality factor or the quality feature cluster or the
quality dimension will slightly affect the overall website quality. • Level 3: Remarkable. The quality factor or the quality feature cluster or the
quality dimension will moderately affect the overall website quality. • Level 4: Significant. The quality factor or the quality feature cluster or the
quality dimension will strongly affect the overall website quality. • Level 5: Critical. The quality factor or the quality feature cluster or the quality
dimension will dominantly affect the overall website quality.
Appendix
Likelihood is to measure the product value of page trip rate and hit rate. The following list is suggested for ranking likelihood: • Level 0: Not set • Level 1: 0% - 20% • Level 2: 20% - 40% • Level 3: 40% - 60% • Level 4: 60% - 80% • Level 5: 80% - 100%