Economic Perspectives in Test Automation: Balancing Automated and Manual Testing with Opportunity...
-
date post
19-Dec-2015 -
Category
Documents
-
view
214 -
download
0
Transcript of Economic Perspectives in Test Automation: Balancing Automated and Manual Testing with Opportunity...
Economic Perspectives in Test Automation:
Balancing Automated and Manual Testing with Opportunity Cost
Paper By – Rudolf Ramler and Klaus Wolfmaier
Presented By – Pallavi Gulati
Topics
Why Automated Testing? Simplistic Cost Model for deciding
between Automated vs. Manual Testing
Flaws in the Simplistic Cost Model New Model based on Opportunity Cost Benefits offered by the new model Future Plans
Why the need to Automate Tests?
Testing accounts for at least 50% of total project cost
Fast-paced iterative development Shrinking schedules and budgets
Test Automation proposed to reduce testing costs
Advantages - Automated Tests
Reduces overall cost of RUNNING tests Though some initial cost required for writing
automated tests Increases the number of tests run Increases frequency at which tests are
run Incremental cost of test running is less
Major concern in deciding whether to fix a particular bug or not
Disadvantages - Automated Tests
Can not automate every test case – timing-related, appearance, user-friendly, some system & integration test cases, involving different hardware setups
May not catch new bugs, targeted towards regression bugs
Higher latency as compared to manual testing which can start immediately
When to go for automation?
How do managers generally decide when to automate?
Decision is based on factors like: Efforts required to run manual test Frequency of testing Cost of third-party / in-house developed
tool Efforts required to write automated tests
Simplistic Cost Model
Vm = Estimated Cost of designing manual tests
Am = Overall cost of manual testing for some # of times
Va = Estimated Cost of designing automated tests (may also include cost of test tool)
Aa = Overall cost of automated testing for some # of times
Flaws - Simplistic Cost Model
Manual & Automated Testing are not comparable Exploring new functionality vs regression testing
Only costs are analyzed not benefits Manual testing starts early in the process but some
latency in automated testing All test cases & executions are considered equal Project Context (budget, resources) not considered Misses additional cost factors like training on the
testing tool, new process adaptation
Why automated tests fail to provide expected benefits?
Due to flaws in estimation using the Simplistic Cost Model and other factors like:
Failure to understand that automated tests are targeted for regression bugs
Gross underestimation of the effort required to develop automated tests
Practically no allocation of efforts for maintaining automated tests
When to automate? vs. What to automate?
It does not help to automate every test
Need to decide trade-off between automated and manual testing
Alternative - Opportunity Cost Model
Opportunity Cost Model
Consider manual & automated test cost Decide how many manual test cases
should be automated Pick those which will give maximum
benefit / risk mitigation / coverage Project budget & deadlines considered
Looks something like...
Opportunity Cost Model Va = 1 hour to
automate 1 test 0 min to run 1
automated test
0 min to come up with 1 manual test
Dm = 15min to execute 1 manual test
Target Function
Ra = Risk Mitigation for automated testing
Rm = Risk Mitigation for manual testing
This allows us to include the benefit obtained by automating a test
Model can be extended to include Additional Influencing Factors
Tests that rely solely on automation e.g. 100 concurrent transactions
Productivity changes over time Causes an outward curved graph
Growing Test Effort in iterative development Improves risk mitigation of automated test
Maintenance cost Early / Late return on investment
Developing automated test early using data driven technique
Defect detection capability random input invoking crash
Advantages– Opportunity Cost Model
Draws attention to balancing automated and manual testing
Benefits as well as costs Risk mitigation capability of tests Budget and deadlines More realistic comparison between
automated and manual testing
Future Plans
Reduce complexity & make it more practical to use
Evaluate preciseness and completeness
Identify & add more influencing factors
Analyze need to add more restrictions
Summary
Advantages of Automated Test Simplistic Cost Model Flaws Proposed Opportunity Cost Model Benefits offered Future Plans for expanding the
model