Scalable Test Automation SQA 2014 01 14 v3.pdfProject for a new teller system Large, state of the...
Transcript of Scalable Test Automation SQA 2014 01 14 v3.pdfProject for a new teller system Large, state of the...
© 2014 LogiGear Corporation. All Rights Reserved
Scalable Test Automation SQuAD, Denver, January 14th, 2014
Hans Buwalda LogiGear Corporation
hans @ logigear.com
www.logigear.com www.testarchitect.com www.happytester.com
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
The 5% Rules of Test Automation
No more than 5% of all test cases should be executed manually
No more than 5% of all efforts around testing should involve automating the tests
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Vision on Automation
Have a good method integrated method for testing and automation testers in the driver seat maximum flexibility, best technology minimal technical efforts maximum maintainability
Have a good tool that supports that method specifically designed to provide a framework for testing and
automation does not stand in the way of the engineers encourages testers to follow the method supports maximum manageability of
• products • process
Succeed in the three "success factors for automation" test design automation solution organization
© 2010 LogiGear Corporation. All Rights Reserved © 2014 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Record and Playback
select window "Logon"
enter text "username", "administrator"
enter text "password", "testonly"
push button "Ok"
select window "Main"
push button "New Customer"
expect window "Customer Information"
select field "First Name"
type "Paul"
select field "Last Name"
type "Jones"
select field "Address"
type "54321 Space Drive"
.
.
.
© 2010 LogiGear Corporation. All Rights Reserved © 2014 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Scripting
Test Case
Design
Test Case
Automation
Test Case
Execution
TEST DESIGNER
AUTOMATION ENGINEER
PLAYBACK
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Example scripting
/// <summary> /// AddItems - Use 'AddItemsParams' to pass parameters into this method. /// </summary> public void AddItems() { #region Variable Declarations WinControl uICalculatorDialog = this.UICalculatorWindow.UICalculatorDialog; WinEdit uIItemEdit = this.UICalculatorWindow.UIItemWindow.UIItemEdit; #endregion // Type '{NumPad7}' in 'Calculator' Dialog Keyboard.SendKeys(uICalculatorDialog, this.AddItemsParams.UICalculatorDialogSendKeys, ModifierKeys.None); // Type '{Add}{NumPad2}{Enter}' in 'Unknown Name' text box Keyboard.SendKeys(uIItemEdit, this.AddItemsParams.UIItemEditSendKeys, ModifierKeys.None); }
State of the art, but stuff for coders . . .
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Action Based Testing
Modularized keyword-driven method for automated testing
Particularly suited for large and complex projects
High degree of agility, even when used in traditional projects
High level of automation, 95% or more
Puts the focus on test design, less on technology
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
number name quantity
new product P-9009 Sledge Hammer 5
number quantity
add quantity P-9009 20
add quantity P-9009 3
add quantity P-9009 6
number quantity
check quantity P-9009 34
Organize the test cases into "test modules", each with a clear and differentiated scope
Test modules contain test objectives and test cases
Test cases are written as a sequence of "actions", each with an action keyword and arguments
The automation engineers focus on automating actions, not test cases
Action Based Testing
actions, each with a
keyword and
arguments
"34" is the
expected value
here
read from top
to bottom
fragment from a test module with actions
© 2010 LogiGear Corporation. All Rights Reserved © 2014 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Action Based Testing
Test Development Plan
Test Module 1
Test cases
Test Objectives
Test Module 2 Test Module N
Actions
ACTION AUTOMATION
Break down
Automate
Test cases
Test Objectives
Test cases
Test Objectives
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Test Design
Effective test breakdown (into test modules) make sure every test module has a clear focus keep different kinds and levels of tests separate
Right level of actions as “high level” if possible, hiding as many details
as much as possible but not if the details are relevant for the test
It is my believe that successful automation is not a technical
challenge. It is most of all a test design challenge.
© 2010 LogiGear Corporation. All Rights Reserved © 2014 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
The 3 "Holy Grails" of Test Design
Effective breakdown of tests into test modules each with a focused and differentiated scope
Right approach for each test module which test design technique who to involve, etc
Proper level of detail in the test lines what actions what checks
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
What's the trick...
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
What's the trick...
Have or acquire facilities to store and organize you content
Edit your stuff
Decide where to put what assign and label the shelves
Put it there
If the organization is not sufficient anymore, add to it or change it
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Breakdown Criteria
Straightforward Criteria Functionality (customers, finances, management information, ...)
-> follow the CRUD for items, with tests for UI, interaction, and business rules
Architecture of the system under test (client, server, protocol, sub
systems, components, modules, ...)
Kind of test (navigation flow, negative tests, response time, ...)
Ambition level (smoke test, regression, aggressive, …)
Additional Criteria Stakeholders (like "Accounting", "Compliance", "HR", ...)
Complexity of the test (put complex tests in separate modules)
Technical aspects of execution (special hardware, multi-station, ...)
Overall project planning (availability of information, timelines, sprints, ...)
Risks involved (extra test modules for high risk areas)
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Example of business level test module
Consists of an (1) initial part, (2) test cases and (3) a final part
Focus is on business functionality, with a clear business scope
Navigation details are avoided
TEST MODULE Car Rental Payments
user
start system john
TEST CASE TC 01 Rent some cars
first name last name car
rent car John Doe Ford Escape
rent car John Doe Chevvy Volt
last name amount
check payment Doe 140.4
FINAL
close application
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Example of an interaction level test module
Interaction details that are the target of the test are not hidden
Focus is not on business ("is the payment amount correct"), but on interaction ("can I retrieve the payment amount")
TEST MODULE Screen Flow
user
start system john
TEST CASE TC 01 Order button
window button
click main create order
window
check window exists new order
FINAL
close application
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Case: American Bank
Project for a new teller system
Large, state of the art
Many system releases, many adjustments
Need for very high level of automation
Over 1 million test lines, in over 650 test modules
Initially little attention paid to "holy grails" UI and functional tests in the same modules virtually un-maintainable, came close to killing the project test design forced upon the team by a powerful stakeholder
who did not care much for methods...
Emergency re-organization of the test modules after system changes the tests would run again within a day
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Example of using actions
In this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function
• the "#" means an expression, in this case a variable
• the ">>" assign to a variable for use later on in the test
key
key navigate F7
key navigate 3
page tab
locate page tab Scan Criteria
w indow
wait for controls loaded search
text
check breadcrumb general functions > search
w indow control value
select search scan direction Backward
w indow control value
enter value search business date match # bus date
source control
click search go
w indow
wait for controls loaded search results
w indow control variable
get search results sequence number >> seq num
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
In this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function
• the "#" means an expression, in this case a variable
• the ">>" assign to a variable for use later on in the test
variable
get sequence number >> seq num
Example of using actions
© 2010 LogiGear Corporation. All Rights Reserved © 2014 LogiGear Corporation. All rights reserved.
TestArchitect
© 2010 LogiGear Corporation. All Rights Reserved © 2014 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
TestArchitect for Visual Studio®
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Focus on Test Development
Traditional testing tools are designed around test management and automation
Test Management
Automation
Test Management
Automation Test Development
TestArchitect™ is intended as a tool for test development but has a lot of test management and automation too
© 2010 LogiGear Corporation. All Rights Reserved © 2014 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Automation Framework
Actions
Test Modules
Action
Definitions
Low Level
Actions
Interface
Definitions
(UI, SQL, OLE, ...)
Test Architect
Built-in Additional
Scripting (C#, Java, Python, ...)
90% +
• split up in lower level actions
• provided defaults values for
arguments
• shield (volatile) technical
details of an interface
• provide "logical" names for
controls and their classes
• test objectives
• test cases Test Modules
© 2010 LogiGear Corporation. All Rights Reserved © 2014 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Example application
© 2010 LogiGear Corporation. All Rights Reserved © 2014 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
A Test Module for the application
We click a tree node, and then do a check
The actions here are built-in in the framework
© 2010 LogiGear Corporation. All Rights Reserved © 2014 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Making a new "action"
This action definition uses existing actions to create a new action called "check bitrate"
Argument names can be used in cell expressions, that start with "#", and support the usual string and numeric operators
create a node path
from the first two
arguments the expected value is
given by the 3rd argument
the arguments of
the new action
name of the new action
© 2010 LogiGear Corporation. All Rights Reserved © 2014 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Using the action in a test
These test lines don't care about the navigation in the UI of the application, the focus is functional: verify data
Such functional tests are easier to read with high level actions, and the reduced dependency on navigation makes them (much) easier to maintain in the long term
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Low-level, high-level, mid-level actions
Low-level: detailed interaction with the UI (or API) generic, do not show any functional or business logic
examples: "click", "expand tree node", "select menu"
High-level: represent a business function specific to the
scope of the test hide the interaction
examples: "enter customer", "rent car", "check balance"
Mid-level: auxiliary actions that represent common
sequences of low level actions usually to wrap a form or dialog
greatly enhance maintainability
example: "enter address fields"
enter customer
enter address fields
enter select set . . . . . .
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Example Script for an Action
# get table object, column number and column count
windowName = LIBRARY.NamedArgument("window")
tableName = LIBRARY.NamedArgument("table")
columnName = LIBRARY.NamedArgument("column")
table = ABT.OpenElement(windowName, tableName)
column = table.GetColumnIndex(columnName)
rowCount = table.GetRowCount()
# check the sort order, row by row
previous = table.GetCellText(0, column)
for i in range(1, rowCount):
current = table.GetCellText(i, column)
if current < previous :
LIBRARY.AdministerCheck("order", "sorted", "fails " + str(i+1), 0)
return
previous = current
LIBRARY.AdministerCheck("order", "sorted", "all rows in order", 1)
The following action script will verify whether the rows in a table are sorted:
find the table in the UI
if a value is smaller than before, fail the test
if all rows are ascending, pass the test
get arguments from the test line
def action_checkSortOrder():
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Mapping the interface
An interface mapping will map windows and controls to names
When the interface of an application changes, you only have to
update this in one place
The interface mapping is a key step in your automation success,
allocate time to design it well
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Some Tips to Get Stable Automation
Test your automation develop separate tests for the actions themselves
Let developers supply values for hidden identifying properties of UI element, to ease automated identification like “accessible name” in .Net and Java, or “id” in Web controls a lot of time is spent on identifying and re-identifying UI elements
Use "active" timing wait based on conditions in the system under test, avoid fixed “sleeps” if needed request developers to provide hooks
Use automation to identify differences between versions of the system under test test in a loop if all control can still be found
Keep an eye on the test design unsuccessful automation virtually always can be traced back to poor test
design
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Hidden Identifying Property
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Variables and expressions with keywords
This test does not need an absolute number for the available cars, just wants to see if a stock is updated
As a convention we denote an assignment with ">>"
The "#" indicates an expression
TEST CASE TC 02 Rent some more cars
car available
get quantity Chevvy Volt >> volts
first name last name car
rent car John Doe Chevvy Volt
rent car John Doe Chevvy Volt
car expected
check quantity Chevvy Volt # volts - 2
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Data driven testing with keywords
The test lines will be repeated for each row in the data set
The values represented by "car", "first" and "last" come from the selected row of the data set
DATA SET cars
car first last value
Chevvy Volt John Doe 40000
Ford Escape Mary Kane 22500
Chrysler 300 Jane Collins 29000
Buick Verano Tom Anderson 23000
BMW 750 Henry Smyth 87000
Toyota Corolla Vivian Major 16000
TEST CASE TC 03 Check stocks
data set
use data set /cars
car available
get quantity # car >> quantity
first name last name car
rent car # first # last # car
car expected
check quantity # car # quantity - 1
repeat for data set
© 2010 LogiGear Corporation. All Rights Reserved © 2014 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Multiple System Access
System
Under Test
Action Automation
API
access
protocol
access
UI
access
database
access
Test Modules
© 2010 LogiGear Corporation. All Rights Reserved © 2014 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Device Testing
Software Under Test
Agent ABT
Automation
Interface
Info
Testing Host Device
Android
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Life Cycles
Product life cycles, rather than task life cycles
The project planning and execution largely determines when the
products are created
system
development
test
development
test
automation
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Typical Time Allocation
TEST DEVELOPMENT
AUTOMATION
time
effort
s
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
ABT in Agile
Test Module
Definition
(optional)
Test Module Development
Interface Definition
Action Automation
Test Execution
Sprint Products Product Backlog
Test re-use
Automation re-use
product owner
team prod owner
& team
User stories
Documentation
Domain understanding
Acceptance Criteria
PO Questions
Situations
Relations
Agile life cycle
Test development
Main Level Test Modules
Interaction Test Modules
Cross over Test Modules
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Using ABT in Sprints (1)
Aim for "sprint + zero", meaning: try to get test development and automation "done" in the same sprint, not the next one next one means work clutters up, part of team is not
working on the same sprint, work is done double (manually and automated), ...
Agree on the approach: questions like does "done" include tests developed and
automated? do we see testing and automation as distinguishable tasks
and skillsets? is testability a requirement for the software?
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Using ABT in Sprints (2)
Just like for development, use discussions with the team and product owners deepen understanding, for the whole team help identify items like negative, alternate and unexpected
situations
Start with the main test modules, that address the user stories and acceptance criteria try to keep the main test modules at a similar level as those
stories and criteria test modules can double as modeling device for the sprint
Plan for additional test modules: low-level testing of the interaction with the system under
test (like UI's) crossing over to other parts of the system under test
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Using ABT in Sprints (3)
Create good starting conditions for a sprint: automation technology available (like hooks, calling
functions, etc) how to deal with data and environments understanding of subject matter, testing, automation, etc
Do interface mapping by hand, using developer provided identifications saves time by not having to use the viewer or other spy
tools recording of actions (not tests) will go better
Make testing and automation part of the evaluations and hardening sprints
© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.
Summary
Keywords is one of the techniques for automated testing, in addition to record & playback and scripting
In itself keywords are not a silver bullet. It needs a good approach, careful planning and good organization to be successful, maintainable and scalable
Thanks for your time