Building Trust and Efficiency in Your Test Suites
-
Upload
kksure -
Category
Technology
-
view
139 -
download
0
Transcript of Building Trust and Efficiency in Your Test Suites
www.unicomlearning.com
Next Gen Testing
Summit-201414th Nov. 2014, PUNE
BUILDING TRUST AND EFFICIENCY
IN AUTOMATED TEST SUITES
Krishna Kumar (KK) Sure
Lead Consultant
www.nextgentesting.org
www.unicomlearning.com
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
First things first…
www.unicomlearning.com
Why Do We Automate?
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
Saves Time
and Money
Improves
Accuracy
Increase Test
Coverage
Simulate Huge
Loads Easily
Catches
Problems
Quickly
Enables
Continuous
Delivery
www.unicomlearning.com
More Around Test Automation
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
Don’t consider test
automation a
specialization
Test Automation
should be integral part
of your software
development
Software development
team should share
responsibility of
building and
maintaining a robust
suite of tests
www.unicomlearning.com
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
Part I:
Trustworthiness
www.unicomlearning.com
Non-Idempotent Tests/Suites
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
www.unicomlearning.com
Issues with Non-Deterministic Tests
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
Raises False
Alarms
Brings Huge
Maintenance
Costs
Makes It Easy
To Ignore The
Tests
Lowers
Confidence In
Making
Changes
Lowers Trust
In Test suite
Disrupts
Continuous
Delivery
www.unicomlearning.com
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
How to build trust back in your
suite
www.unicomlearning.com
Quick Fix
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
• Identify non-deterministic tests
• Quarantine them (Comment
them out)
www.unicomlearning.com
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
Reasons for non-determinism in tests
www.unicomlearning.com
Interdependent Tests
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
sceanrio1 ‘verify consumer can login’
consumer = create_consumer(email, name, password)login_as(consumer)verify_login_successful_for(consumer)
end
sceanrio2 ‘verify consumer creation’
consumer = create_consumer(email, name, password)verify_consumer_count(1)
end
www.unicomlearning.com
Interdependent Tests…
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
• One test execution effects the result of another
test
• Sequence of the tests matters
Ways to fix:
• Rebuild your starting state for each test.
• Make sure each test cleans up after itself
www.unicomlearning.com
Interdependent Tests…
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
sceanrio1 ‘verify consumer can login’
consumer = create_consumer(email, name, password)login_as(consumer)verify_login_successful_for(consumer)
end
before_scenariodelete_all(consumers)
end
sceanrio2 ‘verify consumer creation’
consumer = create_consumer(email, name, password)verify_consumer_count(1)
end
www.unicomlearning.com
Asynchrony
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
sceanrio1 ‘verify consumer can login’consumer = create_consumer(email, name, password)login_as(consumer)verify login_successful_for(consumer)
end
login_successful_for(consumer)sleep 2message_present(‘Welcome consumer’)
end
www.unicomlearning.com
Asynchrony…
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
• Slows the tests
• Gives False failures
Ways to fix:
• Poll for a specific change
• Use methods to wait for jQuery property
www.unicomlearning.com
Asynchrony…
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
login_successful_for(consumer)wait_for_text_to_be_present(‘Welcome’)message_present(‘Welcome consumer’)
end
start_time=Time.now
wait_for_text_to_be_present(text)while(!text.present)
if (Time.now – Start_time > timeout)exit
endsleep 1
endend
www.unicomlearning.com
Asynchrony…
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
wait_for_ajax(text)while(!page.evaluate(‘jQuer.active’).zero?)
if (Time.now – Start_time > timeout)return false
endsleep 1
endend
www.unicomlearning.com
Dependency on External Systems
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
Web
Server
App
Q
U
E
U
E
Payment
Gateway
DB
www.unicomlearning.com
Dependency on External Systems…
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
• Services can go down without notice
• Services could be slow and hence increase
test execution time
• Test instances of these dependencies might
not be stable
Ways to fix:
• Use test doubles e.g. Stubs, Mocks, Spies
www.unicomlearning.com
Too Much Automation
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
• Automating complex test scenarios which don’t
provide enough value
• High chances of failure and hence increases
maintenance costs
Ways to fix:
• Evaluate ROI before automating test scenarios
• Avoid automating scenarios that are complex
but provide less value
www.unicomlearning.com
Too Much Automation…
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
Value
Complexity
Quick Wins, High Value, Definitely
Automate
High value, But complex. Priority 2.
High Complexity, low value. Definite No.
Low value, low complexity. Automate,
maybe
www.unicomlearning.com
Retry Mechanism
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
• Collect the failures of the test suite execution
• Rerun these failed tests for double checking
First pass:
bundle exec rake test –collect_failures –f failures.txt
Second pass:
bundle exec rake test failures.txt
www.unicomlearning.com
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
Part II:
Efficiency
www.unicomlearning.com
Issues with Inefficient Test Suites
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
High
Execution
Times
Test
Duplication
Maintenance
Costs
www.unicomlearning.com
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
Building efficiency in your test
suite
www.unicomlearning.com
Useless Tests
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
• You could accumulate obsolete tests over a
period of time
• Some throw away tests stay for ever
• Multiple tests testing same code paths
Ways to fix:
• Use ATDD, BDD techniques
• Have test refactoring cycles to consolidate and
delete tests
• Integrate code coverage tool
www.unicomlearning.com
Code Coverage Tools
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
www.unicomlearning.com
Code Coverage Tools…
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
www.unicomlearning.com
Long Build Times…
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
• Large number of tests
• Huge build times and hence long feedback
cycles
• Non-deterministic failures
www.unicomlearning.com
Long Build Times…
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
Ways to fix:
• Parallelization
• Isolate database
• Isolate external dependencies
• Collect the failures and rerun
• Use headless browsers (phantomjs,
nightmarejs)
• Segregate Sanity and Regression test suites
www.unicomlearning.com
Testing Everything Everywhere
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
• Segregated development and test teams
• Testing same code paths multiple times
• Difficulty in maintenance
www.unicomlearning.com
Testing Everything Everywhere…
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
Test pyramid
Test refactoring
www.unicomlearning.com
Key Takeaways
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
• Test Automation is an integral part of Software
Development
• Share responsibility of test automation as a
team
• It is important to build trust and efficiency in
your test suite to make it effective
www.unicomlearning.com
Key Takeaways…
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
• Isolate tests from each other
• Don’t use explicit wait, instead use wait for
conditions
• Reduce dependencies on external systems
• Don’t automate everything. Evaluate ROI
• Put a retry mechanism to get better results
www.unicomlearning.com
Key Takeaways…
UNICOM Presents
Next Gen Testing Summit-2014www.nextgentesting.org
• Consolidate and delete obsolete tests in a
timely manner
• Follow ATDD, BDD approach
• Integrate Code Coverage tool
• Parallelize the test execution to reduce build
time
• Try to stay close to Test pyramid