Ananya Das
Commercial Test Automation Tools and their real benefits
An Automation Fable Once upon a time in the peaceful land of Testing, the wizard proposes , “ Let there be automation ..” and the King agrees “Yes.. that is the way to be”
Enter Chandan Nilekani “delegate” Reviews commercial tools, selects , buys , costs …. Assigns to Narad Murthy “overworked “ Narad has several other responsibilities , tool is complicated , tries customer care Finally gets an expert – works and works – several days pass … but no automation
Enter Nazim Premji “short timer” Fresh graduate, new ideas, trainings Works full time … library, gets some parts working After 6 months … gets a great offer …..
Enter Shankar Nadar “hardluck” Tries to execute scripts, no documentation Incomplete scripts, poor error messages, gets some bugs … king happy Next version of product.. .Changes …. Changes scripts … long hours … gets working Product shipped Customer very unhappy … lots of bugs … why ?? Automation Programming error had dismissed these bugs …
King – “Off with their Heads !!!!!!!!!“
Some Commercial test Automation tools
•QTP , RFT, •TestPartner•Silktest•IBM Rational Robot
Functional Testing
•Load Runner•QA LoadPerformance
Testing
•Quality Center•Test Director
Test Management
Evaluating a commercial tool How easy is it to use? Learning Curve? Continuous Integration (CI) friendly? Call & Log without manual intervention? Work with
existing systems ? How does it interact? IE (client side)? HTTP layer? What types of tests does it support?
Web? WinForms? Performance? Framework for Integration Test (FIT)? SQL?
Does it support recording? Does it support scripting?
Languages? Easy creation of scripts ? Object Oriented Scripting Support? Interactive Debugging Tools? Easy integration with External libraries?
Ability to identify objects using multiple parameters? Ability to run multiple scripts consistently in a Batch mode? Ability to store test application's objects so that scripts are well maintainable? Reusability of Scripts and Tests? Reporting mechanism? Object parametrization? Ajax? Popup handling? Keyword driven? Automatic Exception
Handling? Need to support IE7? How easy is it to transfer tests to/from platform?
Evaluating a commercial tool
How well is the tool supported? Commercial? Community?
Support for mocking + stubbing? Test case organization?
Organizing tests similar to structured manual test plans? Support cross frame script calls? Web Objects? Cost/Licensing? Integrated with other test components? VPN / Remote Desktop friendly? Active Community of Users?
Forums? Blogs?
Advantages of Commercial Tools
•Many of the open source tools come and go with little to no support •Most commercial tools are constantly being updated as technologies
change
Customer support
•Most commercial tools usually have more functionality (QTP can test various GUI applications: Web, .Net, Java, VB, C/C++ etc)
•Commercial tools are integrated with Test Management tools which makes reporting and execution much simpler.
Functionality
•Commercial tools usually have a large community of users, which translates into better availability of qualified resources
•Commercial tools require less advanced programming
Resources
•More test automation frameworks are available for commercial tools
Framework
Truth or Myth ?Commercial test tools are expensive ?
Myth Under the influence of this myth some companies, especially the small
ones:o Try to develop their own test automation toolso Use scripting lang. like Perl and Ruby,use shareware test toolso Do not consider test automation at all
Reality It has been calculated that if implemented properly, the cost of this
tool is just 3% of the person who uses it, but productivity gain can be very significant
Truth or Myth ?Recorders make test automation easier
Myth
The Truth:They make recording easy …But they also make it easy to create automation that is difficult
and expensive to maintainA recorder can serve as a training device to help you learn a
scripting language
Truth or Myth ?You can’t develop test automation for an application
until it is stable
Myth
The Truth:You should develop test plans as soon as the Requirements are
documented, You can also develop automated tests from wire-frames, mock-ups and prototypes
In the case of wire-frames, you can even test some of your methodsIf tests are structured properly, only a small portion of your work
will have to be maintained when GUI changes are made to an application that has not yet stabilized.
Truth or Myth ?Test automation is most effective for regression testing
Part Truth:
Test automation is very effective for regression testingIt is also effective for first pass testing when a data driven
approach is used Automated tests should be designed to find bugs not just verify
that regression tests don’t break
Truth or Myth ?It is difficult to automate a dynamic application where data
changes constantly
Myth
The Truth:Data can be separated from test code so that only the data needs to be
changedBetter yet, in some cases input data can be captured at runtime to meet
the criteria required for a testOr a database query can be executed to get data at runtimeVerification data can also be captured at runtime so that tests remain
flexible in a dynamic environment
Truth or Myth ?Test automation is a software development task
Truth:Automated tests should be designed, developed and testedYou need to have some kind of a programming background to
implement test automation. Test Automation is not as complex as C++/C#/Java development.
Automated test components are assets that should be treated like application source code
Test automation standards should be developedTest automation should be subject to peer reviews
Truth or Myth ?Productivity gains will be realized from test tools right
away
Myth:
Test automation is an investmentQuality gains will be realized right awayProductivity gains will be realized over time as skills develop and
manual testing tasks are offset by completed automation
Truth or Myth ?Automated tests are difficult to maintain
Myth
The Truth:Separate business rules, test methods and data to promote
maintainabilityIf properly structured, automated tests can be maintained across
many releases of a target applicationIt’s important to run automated tests on every build and make
incremental changes as needed
Why is Automation great Speed up testing to accelerate releasesAllow testing to happen more frequentlyReduce costs of testing by reducing manual laborImprove test coverageEnsure consistencyImprove the reliability of testingAllow testing to be done by staff with less skillDefine the testing process and reduce dependence on the few who
know itMake testing more interestingDevelop programming skillsBUT ………….
Then … why failure ?About 63 % of the Automation projects fail …. Why ?
MisconceptionsTesting tools don’t work and are difficult to use This application is too complex and the schedules are too tight Senior Management will never support itAutomation will eliminate the need for all manual testingManual testing must have taken place before automating a feature
Critical Success Factors
Clearly define Automation Objectives
Get management support for the project
Clearly demonstrate the short-term & well as long-term financial benefits to be accrued by investing quality resources and time in the project.
Ensure that all stakeholders are in agreement on the end objective of the project. Top Management, the IT team, the developers, the tester, and the automation team--may have different
requirements from the project. However, success will be elusive unless they all agree to a common goal.
Treat Automation as Software development
Study the requirements Planning and tracking Set realistic goals , assess existing skill sets and resources Evaluate the testing tools; perform automation feasibility; Identify reusable functions
Critical Success FactorsDesign and develop a framework
Design and document extensible automation architecture, which clearly outlines the foundation and rules for the project.
Incorporate flexibility and scalability into the architecture to be able to cope with growth in the requirements
Also, provide the testing team with a proof-of-concept test suite and make test case maintenance a top priority
Evaluate testing tools Validate the tools and approach at the earliest and evaluate its compatibility with
the organization’s requirement and the AUT
Check the availability of trained/experienced resources
Critical Success factors` Decide what needs to be automated
Include short and simple transactions and tests that are executed regularly. Execution with multiple datasets Do not automate where it is difficult to predict results or are not going to be repeated. Avoid automating long or complex transactions, cross multiple applications
Spend time on Automation Strategy and Plan
List of activities Organization test objective Reusable and Modular test script identification and planning
Run Automation as a full time project
Look at it as a long term investment
Co-operation with functional testers
GM Regression Automation
Test Logs and bugs
Coverage Matrix
Alpha Test Cases
Components and procedures
Automation Framework
ROI
ROI = BENEFIT/COST
Automation Cost = Price of Tool + Dev Cost + Maint. Cost + Execution Cost
Manual Testing Cost = Development Cost + Maintenance Cost + Execution Cost
ROI = (Manual Testing Cost - Automation Cost)/Automation Cost
Looks right, Doesn’t it? NO …..
You can’t compare Automated Testing and Manual Testing. They are not the same and they provide different information about the AUT.
You can’t compare cost of multiple execution of automated tests vs. manual tests. You would never dream of executing that many test cases manually.
Automation ROI
•People available to work on other projects•Higher market share•Higher margins
Reduce Time to Market
•Defects found early•More testing gets done fasterTest
Efficiency
•Effective regression tests•AccuracyTest
Effectiveness
Conclusion … Long Live the King Commercial Automation tools are useful if implemented and
managed correctly
ROI should be managed
CSFs should be considered seriously
And then ... the King said – “ Let there be automation ….” And there were profits, productivity, growth and success all thru the Kingdom
THE END
Top Related