An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing
description
Transcript of An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing
![Page 1: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/1.jpg)
1
An Exploration of Errors in Web-based Applications in the Contextof Web-based Application Testing
PhD Proposal
Kinga Dobolyi
May 2009
![Page 2: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/2.jpg)
2
The shopping cart
![Page 3: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/3.jpg)
3
The shopping cart
![Page 4: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/4.jpg)
4
The shopping cart
![Page 5: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/5.jpg)
5
What is going on
• Problem: faults in web-based applications cause losses of revenue, and they are hard to test
• Approach: study errors in web-based applications in the context of web testing
• Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing
![Page 6: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/6.jpg)
6
Outline
• Introduction and motivation
• Thesis statement
• Background
• Goals and approaches
• Preliminary work
• Expected contributions
![Page 7: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/7.jpg)
7
Motivation
• Testing of web-based applications in particular deserves further examination due to economic considerations:– Monetary throughput: Backbone of e-
commerce and communications businesses– Customers: low customer loyalty– Development: Companies are choosing not to
test due to resource constraints
![Page 8: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/8.jpg)
8
Motivation: e-commerce
• Internet usage: 73% of people in the US in 2008– Browsers are dominant application– $204 billion in Internet retail sales annually
• Global online B2B transactions total several $trillions annually
• One hour of downtime at Amazon.com cost $1.5 million dollars
• 70% of major online sites exhibit user-visible failures
![Page 9: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/9.jpg)
9
Motivation: customers
• Customer loyalty is notoriously low– Determined by the usability of the application
[Offutt 2002]– Freedom and options
![Page 10: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/10.jpg)
10
Motivation: customers
• Lesson learned: web-based applications need to be well-designed and well tested
• Are they?
![Page 11: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/11.jpg)
11
Motivation: development
• Technology
challenges:– Heterogeneous, opaque
components – Dynamic page content generation– Persistent state operated upon by
concurrent, global users around the clock
![Page 12: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/12.jpg)
12
Motivation: development
• Web-based applications are often not tested– Enormous pressure to change
• Short delivery times, high developer turnover rates, and quickly evolving user needs
– No formal process model
![Page 13: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/13.jpg)
13
Motivation: summary
• Problem: faults in web-based applications cause losses of revenue, and they are hard to test
• Approach: study errors in web-based applications in the context of web application testing
• Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing
![Page 14: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/14.jpg)
14
Thesis statement
• Hypothesis: web-based applications have special properties that can be exploited to build tools and models that improve the current state of web application testing and development:– Tend to fail in predictable and similar ways– Human centric definition of acceptability
![Page 15: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/15.jpg)
15
Thesis statement
• Problem: faults in web-based applications cause losses of revenue, and they are hard to test
• Approach: study errors in web-based applications in the context of web testing
• Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing
![Page 16: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/16.jpg)
16
Background: testing techniques
• Non-functional (static) validation– Server load testing– Link testing– HTML/spelling validation
• Modeling approaches
• Capture-replay– User session-based testing
![Page 17: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/17.jpg)
17
Background: oracles
• Oracles (oracle-comparator)
1 < <P>The same table could be indented.
2 < <TABLE border="1">
3 ---
4 > <p>The same table could be indented.</p>
5 > <table border="1" summary="">
– False positives from
diff-like tools– Want precise comparators
![Page 18: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/18.jpg)
18
Background: automation
• Automation– Test case generation: VeriWeb, PHP– Test case replay
• URL + post-data
– Failure detection
![Page 19: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/19.jpg)
19
Background: metrics
• How do we measure success?– Code coverage– Fault seeding
• Human• Automatic
– Cost
• How do we know these are indicative of the real world?
![Page 20: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/20.jpg)
20
Background: fault definition• Defining an error:
– “the inability to obtain and deliver information, such as documents or computational results, requested by webusers.” [Ma & Tian 2003]
• Fault taxonomies– Figure from
[Marchetto et al2007]
![Page 21: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/21.jpg)
21
Background: challenges
• Functional validation remains a challenge– Regression testing should be more precise
and automatic
• We do not know if test suite efficacy metrics are indicative of the real world– We should examine the severity of uncovered
faults
![Page 22: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/22.jpg)
22
Goals and approaches
• Problem: faults in web-based applications cause losses of revenue, and they are hard to test
• Approach: study errors in web-based applications in the context of web testing
• Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing
![Page 23: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/23.jpg)
23
Goals and approaches
• I propose to:– Model errors in web-based applications
• Identify them more accurately • Automate the oracle-comparator process
– Make web testing more cost-effective• Devise a model of fault severity that will guide test
case design, selection, and prioritization• Validate or refute the current underlying
assumption that all faults are equally severe in fault-based testing
![Page 24: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/24.jpg)
24
Goals and approaches: Goals
• Reduce the cost of regression testing web-based applications – Use special structure of web-based application output
to precisely identify errors
• Automate web-based application regression testing– Unrelated web-based applications tend to fail in
similar ways
• Understand customer-perceived severities of web application errors.
![Page 25: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/25.jpg)
25
Goals and approaches: Goals
• Formally ground the current state of industrial practice– Validate or refute fault injection as a standard for
measuring web application test suite quality
• Understand how to avoid high-severity faults during web application design and development
• Reduce the cost of regression testing web applications by exposing high-severity faults– Test case design, selection, and prioritization (test
suite reduction)
![Page 26: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/26.jpg)
26
Goals and approaches: Outline
![Page 27: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/27.jpg)
27
Goals and approaches: Step 1 – oracle-comparator
• Construct a precise oracle-comparator that uses the tree-structured nature of XML/HTML output and other features– Model errors on a per-project basis– Semantic distance metric to reduce false
positives
![Page 28: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/28.jpg)
28
Goals and approaches: Step 2 – automated oracle-comparator
• Exploit the similar way in which web applications fail to avoid the need for human annotations in training a precise oracle-comparator– Train a precise oracle-comparator on data
from other, unrelated web applications– Use fault injection to improve the results when
necessary
![Page 29: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/29.jpg)
29
Goals and approaches: Step 3 – human study
• Conduct a human study of real-world fault severity to identify a model of fault severity– Severities different than self-reported in bug
repositories– Screenshots of current-next idiom– Also survey developers
![Page 30: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/30.jpg)
30
Goals and approaches: Step 4 – fault seeding validation
• Compare the severities of real-world faults to seeded faults using human data (validate fault seeding)– The severities of seeded errors have uniform
distributions?– The severity distribution of seeded errors
matches the distribution of real-world errors, according to the results of the survey from Step 3?
![Page 31: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/31.jpg)
31
Goals and approaches: Step 5 – software engineering guidelines
• Identify underlying technologies and methodologies that correlate with high-severity faults– As an alternative to testing– Tie high severity errors to underlying code,
components, programming languages, and software engineering practices
![Page 32: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/32.jpg)
32
Goals and approaches: Step 6 – testing techniques
• Identify testing techniques to maximize return on investment by targeting high-severity faults– Introduce a new metric for the (web
application) test suite reduction research community
![Page 33: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/33.jpg)
33
Preliminary Work: Step 1
• Step 1: Construct a precise oracle-comparator using tree structured XML/HTML output and other features– Multiple versions of
10 open-source
benchmarks– 7154 pairs of oracle-
testcase output, 919
of which labeled as
“possibly an error”
![Page 34: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/34.jpg)
34
Preliminary Work: Step 1
• Evaluation: F-measure (precision and recall) using our model
![Page 35: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/35.jpg)
35
Preliminary Work: Step 1• Longitudinal study to measure effort saved
– Calculate cost of looking : cost of missing– Low ratio means we are saving effort
![Page 36: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/36.jpg)
36
Preliminary Work: Step 2
• Step 2: Exploit similarities in web application failures to avoid human annotations when training a precise oracle-comparator– Same setup as Step 1– Use existing, annotated pairs of test-oracle
output from unrelated applications to train a comparator for the application at test
![Page 37: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/37.jpg)
37
Preliminary Work: Step 2
• Evaluation: measure precision and recall
![Page 38: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/38.jpg)
38
Preliminary Work: Step 2
• Use fault seeding to introduce project-specific faults into the training data set
![Page 39: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/39.jpg)
39
Preliminary Work: Step 3
• Step 3: Model real-world fault severity based on a human study.– Collect 400 real-world faults
• Evaluation:
have subjects
use the model
to classify
faults
![Page 40: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/40.jpg)
40
Preliminary Work: Step 4
• Step 4: Compare the severities of real-world faults to seeded faults using human data.– Test same human subjects with 200 + 200
manually-injected and automatically-injected faults
– Conduct a survey
of developers for
fault severity
![Page 41: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/41.jpg)
41
Preliminary Work: Step 5
• Step 5: Identify technologies and methodologies that correlate with high-severity faults
• Do high severities correlate with:– Programming Language– Level in three-tier architecture– COTS component– User error (usability issue)– Type of error (business logic, resource allocation, syntax error,
etc)– Fault taxonomies (existing)– Surface features of visible output: white screens, stack traces,
misspellings, formatting errors
• Evaluation: developer survey (time permitting)
![Page 42: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/42.jpg)
42
Preliminary Work: Step 6
• Step 6: Identify testing techniques to target high-severity faults– Targets testing– Assign a testcase a severity rating a priori
• Evaluation: compare the severity of faults exposed with our technique versus other test suite reduction approaches
![Page 43: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/43.jpg)
43
Expected Contributions
• Problem: faults in web-based applications cause losses of revenue, and they are hard to test
• Approach: study errors in web-based applications in the context of web testing
• Solution: improve the state of the art in web testing techniques through guidelines targeted at high severity faults and automation and precision in regression testing
![Page 44: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/44.jpg)
44
Expected Contributions
• Reduce the cost of regression testing by constructing a precise oracle-comparator
• Develop a model of customer-perceived severities of web application faults
• Validate or refute fault injection as a standard for measuring web application test suite quality
• Propose new software engineering guidelines for web application development and testing
![Page 45: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/45.jpg)
45
Questions?
![Page 46: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/46.jpg)
46
Original Contributions
• Fault Severity Model– Severity has not been studied in this domain, and
customers are an integral part of these applications– Providing a new metric to the research community– Validate/refute fault seeding
• Precise-oracle comparators– First to use different versions of benchmarks– Can be completely automated– XML and HTML
![Page 47: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/47.jpg)
47
Expected Impact
• Fault Severity Model– Can be applied to testing techniques in this field to
make them financially feasible for developers– Change the way in which test suite efficacy is
measured– Potentially impact web site design as usability issues
may become more evident• Precise oracle-comparator
– Automation makes it much more feasible for adoption than existing techniques
– Potentially allow companies to conduct regression testing if they were not testing beforehand
![Page 48: An Exploration of Errors in Web-based Applications in the Context of Web-based Application Testing](https://reader033.fdocuments.in/reader033/viewer/2022051623/568157c3550346895dc54736/html5/thumbnails/48.jpg)
48
Timeline
• Steps 1 and 2: precise comparators– Completed, 2 papers under submission
• Steps 3 and 4: human study– Data collection completed, analysis under way for submission of
1 paper– Expected completion by September
• Step 5: software engineering guidelines– Expected completion by October– Expected 0.5 paper
• Step 6: testing according to fault model– Expected completion by February– Expected 1 paper