The difference that makes the difference. Make your test automation project a success story Magnus...

The difference that makes the difference

Make your test automation project a success story

Magnus NilssonLemontree

Agenda

The Test automation concept Walkthrough of a test automation project

– Execution and maintenance of an automated solution– Test strategy

Development of a Framework– Organizing your code

The test automation concept

Fundamentals behind test; a short review of– Lifecycle activities– Test levels– Test methods– Development models

Test automation and test objectives Test automation success factors

Test Automation

Concept

Requirement

Project

Business process

Test

DefectManageme

nt

Release

Maintenance

Monitoring

Architecture

Design

Development

Test Automation

Concept

Test

Test as part of the lifecycle activities

Connecting the activitiesTest levels Integratio

ntest

Acceptance test

Unit test

Test Automation

Concept

Test

Development

Systemtest

Unit test System test Integration test Acceptance test

Functional Non functional

Test types

Connecting the activities Test methods

Black box

Test

White box

Code

Black box– Testing delivered

functionality– GUI test

White box– Testing internal structures

Code– Testing modules, classes,

libraries, functions on code level

Test Automation

Concept

Connecting the activities Development models

Waterfall

Development

AgileIterative

Scrum

Waterfall– The lifecycle approach

Iterative– Planned repeated cycles

Agile– Iterative and development

adaptive to changes

– Practicing continuous Integration, CI

Scrum– Sprint-based development based

on both on agile and iterative development methods Test

Automation

Concept

The challenge of test

Development

Test

Test Automation

Concept

Code

White box

Black box

Testing methods

Unit test

Integration

test

System test

Acceptance test

Test levels

Development models

Waterfall

Agile

Test automation is all about unifying test.

Environment

Automation

TestHarness

Development

Development– Increase code quality by implementing unit tests– Development methodology TDD– Automated builds, CI

Environment– Test environment creation– Test data management, TDM

Harness– Stubbing interfaces

Test– Scripted tests

• Test interfaces• GUI• Performance

Test Automation

Concept

Development

Test

The challenge of test, connecting…

Unit test

Code

Integration

test

GUI

White box

System test

Acceptance test

Black box

Test Automation

Concept

Testing methods

Test levels

Development models

TDD

TDM

Test automatio

n

Test Harness

CI

AutomationWaterfall

Agile

Example 1 - Test harness

Enables test before interfaces exists or are available, agile development but also project delivery regardless development model

System test

Project 1

Test Automation

Concept

System integration test Acceptance test

Project 2

Project 3

Stable environment

Time

Release

Start SITStart AT

XX

X

Test harness

X

X

Test Harness

Example 1 - Test harness

Each flow have a known expected result Data driven solution Includes test of functions, web services and interfaces

Calling system Test harnessSimulated

system

Calling system Test harnessSimulated

system

1. Call service2. Evaluate simulator rules

3.2.1 Call live test system

3.2.2 Return response to harness

3.1. Fetch simulator response

4.Return response to calling system

Test Harness

Test Automation

Concept

Test automation and test objectives

Test objectives depends on the testing level. But finding bugs is one of the main objectives.

A bug is – Wrong or missing requirement– Bad design or architecture– Coding error– Poor implementation

The objectives of test is to make sure none of the above takes place.

Test Automation

Concept

Test automation and test objectives

The automated solution must fulfill the test objectives

Input is functional requirements, design use cases and test cases.

Output is functional verifications interpreted as check points.

Test objectives

Use cases

Requirement

Test cases

Test Automation

Concept

Test automation success factors

Develop functional components

Decouple in- and output data from the script

Central data storage

Development project – strategy and – center of

excellence

Combine components to test cases by actions as input data

Functionalcomponents

Test automation

In-/outdata

Central data

storage

Project

Test Automation

Concept

Test automation GUI – functional components

Product

Customer

Billing

Account

Order

Configuration

Subscription

Login

Task

Credit check

Product

Inventory

Calling system External systems (with or without test harness)

Test Automation

Concept

Each component have a defined degree of freedom Population of mandatory data Trigger of validations

Test automation GUI – input data

Next process

Start customer form

Private customer?

Populate private

customer form

Address Info

BusinessID

Business Name

Customer Name

Validatecustomer

ValidateType

Complete customer form

and save

CustomerT

ype

Populate business

customer form

yes

no

Test Automation

Concept

Results of validations Identification data

Test automation GUI – output data

Next process

Start customer form

Private customer?

Populate private

customer form

Address Info

BusinessID

Business Name

Customer Name

CustomerID

CustomerStatus

Validatecustomer

ValidateType

Complete customer form

and save

CustomerT

ype

Populate business

customer form

yes

no

Test Automation

Concept

Test automation GUI – output data

[create]

Customer Account[new]

CustomerID

Components are linked by use of identification data. Output data from one component is input data for another

component

Test Automation

Concept

Test automation GUI – test cases

Login

Customer Account[new]

Inventory

Login Credit check

[create]

Subscription Order

[new]

Account [new]

Customer [open]

Billing

Test case 1:

Test case 2:

Test Automation

Concept

Test automation GUI – changes

Login

Customer

Inventory

Login Credit check

[create]

Subscription Order

[new]

Account [new]

Customer [open]

Billing

Test case 1:

Test case 2:

Account [new]

ver 2.0

ver 1.0

Account[new]

ver 1.0

Account [new]

ver 2.0

Account[new]

ver 2.0

Test Automation

Concept

Building up the components

Functionality available in system test – Development of the functionality

as it comes available

Requirement

Design and development

System test

Phase 1 Phase 2

Function 1

Function 2

Function 1

Function 2

Function 1

Func. 2

Each functionality is part of a component (or a component itself)

Func. 3

Func. 4

Test Automation

Concept

Func. 3

Func. 4

Function 1

Func. 2

Func. 5

Func. 6

Account

Customer

Building up the components

Systems ready– Testing relations of functionality

Regression, E2E, Acceptance

System test

Phase 3 Phase 4

Function 5

Func. 6

Regression test

Function 1

Func. 2

Function 3

Func. 4Function 1

Func. 2

Function 5

Func. 6

Func. 3

Func. 4

Function 1

Function 5

Test Automation

Concept

Example 2 - Agile Automation

Test

Development

Requirement

Design

Requirement

A. Product backlog

B. Sprintbacklog

C. Sprint withautomation

Release

Iteration 3-4 weeks

D. Sprint release

Agile

Test Automation

Concept

Example 2 - Agile Automation sprint schema

Requirement

Design and development

System test

Regression test

Sprint 0 Sprint 1 Sprint 2

Function 1

Function 2

Function 1

Function 2

Function 1

Function 2

Function 1

Func. 2

Function 3

Function 4

Function 3

Function 4

Function 3

Func. 4

Function 5

Test Automation

Concept

Lifecycle activities Test levels Test methods Development models Test automation and test objectives Test automation success factorsCOVERED

The test automation concept

Test Automation

Concept

Agenda




Walkthrough of a test automation project

Walkthrough

Project

Project activities Project plan - phased implementation Project organisation Business case and follow up

??

?

?

?

?

Test automation project - activities

Analysis Design Requirement Development Test Documentation Training Execution

Customer

Product

Order

Customer ProductOrderCustomer Product Order

Customer Product Order

ver 2.0

ver 2.0

ver 2.0

ver 1.0

ver 1.0

ver 1.0

Walkthrough

Project

Project plan – phased implementation

Walkthrough

Project

Analysis

Detailed design

Development

System test

Acceptance test

Phase 1

Week

Plan next phase

Phase 2

Detailed design

Development

Automation stakeholder

Subject Matter Expert

Test Automation Center

Project organisation - Use roles to enforce competence

Automation Engineer

Automation Analyst

Automation Solution Architect

Automation Project Lead

Walkthrough

Project

Project A



Project B



Project C



Project organisation - Test automation center

Development

Execution

Maintenence

Walkthrough

Project

CommunicatingReportingPlanned activities

Test automation center

TAC Service Management

Management

Test management

Release management

Project planning

Change request

TAC Execution

TAC Maintenance

TAC Developmen

t

Request

Result

Operations

ST SIT AT

Project AProject B

Project C

Maintenance

Supplier

Defect and Issue Management

Resolution

Business case

Why do we need a business case?– Test automation cost time and money– The automated solution must at least ensure the time spent in development, maintenance

and execution is less than a manual solution would cost to accomplish the same goal.

What is a test automation business case?– List of all costs and savings associated with test that possibly can be covered by test

automation.– Not all drivers are possible to calculate directly as quality. But quality can be expressed as an

estimation of the number of testing hours to ensure the desired level of quality.– Additional test automation drivers as repeated executions is calculated indirectly as increased

quality.

How can a business case be used?– Business case can be used to predict effectiveness as automation deliveries should follow the

calculated schedule.– ROI should be calculated at regular audits based on the business case.

Walkthrough

Project

Business caseCost for manual regression tests

Cost for error handling

External and business costs for errors

ProductionAT

Number of errors found per year

Handling time per error (report, reproduce, administer, test data, close defect)

Number of FTE's for error handling per year

Error handling cost per hour

Annual error handling costs

1000

2

2

100

200000

Error source

500

4

1

100

200000

Total test case executions per cycle

Number of test cycles

Execution time per test, h

Execution cost per hour

10004

0.5

100

Annual cost for execution of regression test

200000

Grand total

Annual error correction costs for errors

Business costs for productions errors1)

0 100000

2

400000

Cost multiplier for Production Errors

1)Goodwill, compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

1100000

Business case

Savings - Manual Test Labour

Savings - Error Reduction

Implementation Cost (Year 1)

Operational costs (running)

Description Reduction ratio

Annual reduction/cost

Acc reduction

100000

70000

70000

Phase 1

Phase 2

Phase 3

Total savings

270000

90000

180000

Year 1

Year 2

Year 3

10%

20%

30% 540000

90000

270000

Year 1 120000

160000

140000Year 2

Year 3

60%

70%

80%

120000

420000

260000

Year 1 proj support, SME

Year 2 TAC OP

Year 3 TAC OP

50000

60000

60000 Acc savings

Year 1

Year 2

Year 3

-80 000

260000

370000

180000

550000

-80 000

Walkthrough

Project

Business case – follow up

Analysis

Detailed design

Development

System test

Acceptance test

Plan next phase

Phase 1

Phase 2

Detailed design

Development

Week

DeliveredPlannedTest cases

Week

Walkthrough

Project

Test automation strategy

Description of the standard approach on how to use the tools Company specific set of rules on how automation should be done Generic solution that is not dependent on individuals Maintenance focused Encapsulated by test automation centre Enforced by framework Ensured by the automation project

Walkthrough

Project

We will now go through the test automation project

Analysis and design Evaluation

– Scope

Analysis– Analysis based on existing documentation, test cases, use cases and requirements (static test review)– Provides the prioritization to a analysis structure template

Disposition– Decision making: if we go live what does and does not work?– What business processes are working and not working or not tested?– Often details on number of test cases passed and failed on integration and system level is not vital– Defect status itself often only aids prioritizing of resources/focus

Structure– Common and clear reporting on a level that enables correct business decisions– Structured test design and coverage analysis– Will control the test automation deliveries– Used in test management to create common priorities for both manual and automated tests

Implementation– After the analysis and approved structure design, the structure may be implemented using

available tools in HP QC/ALMWalkthrough

Project

Analysis and design - evaluation Value

– Part of business case, added value by the test case, frequency and time to execute Stable

– Can the stability be impacted by sources out of control? Test data

– Is all test data needed defined or is a manual process involved? Reusability

– Can the business flow be reused in the automated component solution? Well defined

– Is the business flow well defined and no manual decisions needed? Fully automated flow

– No manual interventions needed? Maturity

– Is the functionality in maintainable where only the test data may be changed? Defect frequency

– Are there a lot of issues in the functional area?Walkthrough

Project

Analysis and design - the evaluation starValue

Stable

Test data

Reusability

Well defined

Defect frequency

Maturity

Fully automated flow

Prio 1 test case

Prio 2 test case

Walkthrough

Project

Analysis structure - disposition

what is the system used for?– how is it done?

• which are the requirements on process level?

» which data conditionsare covered by use cases?

Use case

Process

Requirement

Data condition 1

Data condition 2

Walkthrough

Project

Example

Mobile subscription

Web self care

New

Free surf

Free SMS

Analysis structure - disposition

Requirements

Folder area Process Description Prio

Use cases/product

Customer Account Subscription

Create Terminate New New

Sales Web self care 1

Modify Cancel

3 1 10Web shop

Sales Customer operations

1 3 1 1 3 1 15CRM

Sales Device 2 3App

Sales Device 2 1 1 1Online

1

Product Inventory 1Product configurations

Walkthrough

Project

Requirement and component design - refresh

All processes requires several components to enable E2E automation

Each component mainly automates one business process or interface

Validations may be performed in interfacing systems A component may be developed in iterations Components are used together – keyword driven Components handle unlimited data conditions – data driven Map which processes require which components

Walkthrough

Project

Requirement and component coverage

Requirements

Folder area

Requirement Process

Data conditionTest case Prio

Components

Customer Account Customer

Create Terminate New Create

SalesWeb self care R1.1

1

Modify Cancel

Create Customer

Sales Customer op R2.3.4

1

Sales 2Terminate customer

Sales 2Create bank account

1

CRM

Create Customer

Web Shop

X

Customer op R2.4.1

X

Customer op R4.1.1

X

X

SalesCustomer op R4.1.1

Create invoice account

X

Test case

TC23

TC12

TC1

TC45

TC6

Walkthrough

Project

X

Coverage analysis and status report

Subscription

Customer operations

New

Modify

Cancel

Web self care

New

2 1

0 3

3 1

1 1

No runNot

completedFailed PassedStatus Not covered

1

0

4 1 51

3

Not covered

Subscription requirement not completely covered New subscription requirement covered by web self care

1

1

13

Passed

Failed

Not completed

Coverage analysis and status report, HP ALM

Coverage in ALM Test either

– manual,– automated or– business

componentWalkthrough

Project

Component design template

Coverage– Covered test cases and requirements; Requirement and

component coverage Component description

– Prerequisities• Data• Environment

– Design description• Component type• Flow/use case description• Detailed steps

– Data table• Input/output data

– Verification element Walkthrough

Project

Component data table

M

O Business ID will be validated if provided

Mandatory/optional

DescriptionField name Input/output

Out

N/A

CustomerID_Out

BusinessID

FamilyName

FirstName In

In

In

M

N/A The created CRM customer ID

Family name

First name of customer

MOfficialName In Private or Business

OutScriptstate_Out Status of execution

LinkEntity_In In O Entity linked from previous component

Walkthrough

Project

Generic input data

oBusinessID

1. Public function GetBusinessID(oID)

2. Dim oBusinessID

3.

4.

5.

6.

7.

8.

9.

10.

11.

12. GetBusinessID = oBusinessID

13. End Function

Static

Parameter

Data tableGenerated

Dynamic

Query, UDF, link

DynamicInput data Static

Data table value Assigned value

Parameter Assigned value Table data, link

ManualX Linked

oBusinessID = ”AB46”

oBusinessID = oID

oBusinessID = Parameter(”BusinessID”)

oBusinessID = DataTable.value(”BusinessID”)

oBusinessID = oBusinessID & "_" & cstr(now)

oBusinessID = sqlQuery(”select businessID fr…)

oBusinessID=DataTable.value(”BusinessID_In”)

Walkthrough

Project

Generic input data, ALM

variable

Static

Parameter

Data tableGenerated

Dynamic

Manual Linked

Data table

Development

Development according to design and requirements According to company specific test automation strategy The strategy is enforced by a framework including templates and

best practices The tools included in the framework must be mastered by

automation engineers. Experienced automation engineers not necessarily subject matter

experts of the systems. Automation tool as HP QuickTest Professional/Functional Tester

facilitate script creations and object handling. Configuration management used to maintain the scripts.

Walkthrough

Project

Execution

Test cases created Ordering procedure set up Test data prepared Result reporting arranged Press the button

Walkthrough

Project

Then what? X

Automation Engineer

Functional tester

Analysis

Detailed design

Development

System test

Acceptance test

Plan next phase

Phase 1

Phase 2

Detailed design

Development

Week

COVERED

Walkthrough of a test automation project

Walkthrough

Project

Agenda




Development of a framework

Code test frameworks– NUnit, JUnit– Test-driven development (TDD) i agile development models

GUI test frameworks Automation framework infrastructure to support use of different

tools Data-driven testing (DDT) Keyword-driven testing (KDT) Hybrid testing Component/modular based Model based testing Framework

Development of a

1st generation script– Using automation tool for capture and replay only– Hardcoded data– Migration test

2nd generation script– Code refactoring for maintainable scripts– Uses generic functions for iterations and error handling

3rd generation script– Data driven test where dynamic data is taken from data tables– Use of driver scripts for executing multiple test cases– SME for test designs and requirement– More complex scripting by developers


Framework

Development of a


The Tool– HP QuickTest Professional/Functional tester– Framework itself with

• test management connection ALM, • keyword driven action scripts with • data table,• function libraries,• Repositories, • Business Process Testing and• recovery scenario

– Generates code Code standard Library definitions, template of standard functions Shared repositories Structured data handling, input and output data Error handling Execution flow Reporting

Repository

objects

Application GUI layer

List object

Text object

Table object

Object recognition

engine

Data table

M


Out

N/A

CustomerID_Out

BusinessID

FamilyName

FirstName

In

In

In

M


Family name


MOfficialName

In Private or Business

OutScriptstate_Out

Status of execution

LinkEntity_In

In O Entity linked from previous component

Function library A

Function library B

ALM

Test resources

Test plan

Test lab

Dashboard

Recovery scenario

Test action script

Framework

Development of a

Framework

Development of a


The Tool

Code standard– Templates secures a common use of functionality– Unified naming convention– Keep it simple– Function declaration to extract common documentation

Library definitions, template of standard functions Shared repositories Structured data handling, the use of input and output data Error handling Execution flow Reporting

'*****************************************************

'* Function: [FunctionName]

'*

'* [Description]

'*

'* Parameters:

'*

'* [Param1] - [Description]

'* [Param2] - [Description]

'*

'* Returns:

'*

'* [Returnvalue]

'*

'* Author: [Author]

'*

'* See Also:

'*

'* <[DependentFunction]>

'*

'******************************************************


The Tool Code standard

Library definitions, template of standard functions– Application specific libraries– Standard actions in applications should be in application library

Shared repositories Structured data handling, input and output data Error handling Execution flow Reporting

Framework

Development of a

Framework

Development of a


1. …

2. …

3.

4.

5.

6.

7.

8.

9.

10.

11.

12. …

13. …

result = Browser("Home").Page("Customer").WebElement("LoadingAnimation_left").exist(1)

Do While Not( result)

result = Browser("Home").Page("Customer").WebElement("LoadingAnimation_left").exist(1)

thisWait = thisWait +1

If thisWait > maxWait Then

Err.Raise vberror + 100, " WaitLoading()", "Timeout loading page"

Exit Do

End If

Loop

Repository

objectsExtract from action script CreateCustomer

Framework

Development of a


1. …

2. …

3.

4. …

WaitLoadingObject(Browser("Home").Page("Customer").WebElement("LoadingAnimation_left"), timeout)

Extract from action script CreateCustomer

1. Public Sub WaitLoadingObject(theObject, maxWait)

2.

3.

4.

5.

6.

7.

16.

17. End Sub

Dim maxWait, thisWait

maxWait = theTimeout

thisWait = 0

result = theObject.exist(1)

Do While Not( result)

…

CRM_utility.qfl


The Tool Code standard

Library definitions, template of standard functions– Application specific libraries– Standard actions in applications should be in application library– Component libraries enabling Configuration Management

• Branched Release Management for ST, AT and dev• Multiple checkout and merge• Only driver scripts

– Framework libraries handling framework and best practices Shared repositories Structured data handling, input and output data Error handling Execution flow Reporting

Framework

Development of a

The Tool Code standard Library definitions, template of standard functions

Shared repositories– Objects as well as code needs to be shared in order to avoid redundancy

and maintenance overhead– Local and shared repositories, application area

Structured data handling, input and output data Error handling Execution flow Reporting


ALM

Test resources

Test plan

Test lab

Dashboard

Test action script

CreateCustomer

LocalRepository

objects

Test action script

UpdateCustomer

CRMCustomerRepository

objects

Framework

Development of a

The Tool Code standard Library definitions, template of standard functions Shared repositories

Structured data handling, input and output data– Central data source– Structure and data

Error handling Execution flow Reporting


Data table

Component A sheet

M


Out

N/A

CustomerID_Out

BusinessID

FamilyName

FirstName

In

In

In

M

N/A

The created CRM customer ID

Family name


MOfficialName


OutScriptstate_Out

Status of execution

LinkEntity_In


Component B sheet

M


Out

N/A

CustomerID_Out

BusinessID

FamilyName

FirstName

In

In

In

M

N/A


Family name


MOfficialName


OutScriptstate_Out

Status of execution

LinkEntity_In


CreateCustomer sheet

M


Out

N/A

CustomerID_Out

BusinessID

FamilyName

FirstName

In

In

In

M

N/A


Family name


MOfficialName


OutScriptstate_Out

Status of execution

LinkEntity_In


ALM

Test resources

Test plan

Test lab

Dashboard

Structure

Data

Master DB

Data

Framework

Development of a

Framework

Development of a

The Tool Code standard Library definitions, template of standard functions Shared repositories Structured data handling, input and output data

Error handling– Essential in script execution; defect, automation or script language– Recovery scenarios handle all unexpected events

in execution, calling framework functions– Handle user defined errors in framework template

Execution flow Reporting


Recovery scenario

Test action script Framework

library

Exception

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

On Error Resume Next

Do While row < rowCount + 1

rowFunction

If err.Number <> 0 Then

LogError

Err.Clear

Else

LogRun

End If

row = row + 1

Loop

On error goto 0

CreateCustomer.qfl

56.

57.

69.

70.

71.

72.

73.

74.

75.

76.

77.

85.

Public Function rowFunction

…

WaitLoading timeout

'Wait for customer table

retVal =waitForObjectComplete(Browser("Home")…), timeout)

If not retVal Then

Err.Raise vberror + 1000, "Customer not found", "CreateCustomer"

End If

'Retrieve customerID in customer table

customerID = GetCustomerID(privateCust)

…

end function

Customer.qfl

The Tool Code standard Library definitions, template of standard functions Shared repositories Structured data handling, input and output data Error handling

Execution flow– Combining components to test cases by

• Driver scripts and• Test data

Reporting


Componentlibrary A

Componentlibrary B

ALM

Test resources

Test plan

Test lab

Dashboard

Component B action script

Driver action script

Framework

Development of a

Framework

Development of a

ALM

Test resources

Test plan

Test lab

Dashboard

The Tool Code standard Library definitions, template of standard functions Shared repositories Structured data handling, input and output data Error handling Execution flow

Reporting– Use built in reporting mechanism to HP Run Results– Upload attachments– Report using OTA API to other ALM resources– User logs


Log file

Run Result

Frameworklibrary


Repository

objects

Application GUI layer

List object

Text object

Table object

Object recognition

engine

Data table

M


Out

N/A

CustomerID_Out

BusinessID

FamilyName

FirstName

In

In

In

M


Family name


MOfficialName


OutScriptstate_Out

Status of execution

LinkEntity_In


Function library A

Function library B

ALM

Test resources

Test plan

Test lab

Dashboard

Recovery scenario

Test action script

Framework

Development of a

COVERED

Agenda

Session 1– The Test automation concept– Walkthrough of a test automation project

• Execution and maintenance of an automated solution• Company test strategy

Session 2– Development of a Framework

• Organizing your code

Short about Lemontree

Swedish company established 1999

Awarded Gazelle company and Supercompany 2008-2012

Turnover +98 MSEK 2011

AAA-rated

About 100 employees, growing intensive

Owners are actively working in the company, no external capital

Offices in Stockholm and Oslo

QM trainee program since 2012

Lemontree

Industries

• Telecom

• Bank & Finance

• Insurance

• Public sector

Focus areas

• Quality Management

• eGovernment

• Enterprise Solutions

Bankgirot

http://www.telia.se/

http://www.tre.se/Privat/

https://www.riksgalden.se/

http://sodexo.se/sv/Sodexo-Sverige/

http://www.spinbox.se/

http://www.a-sec.se/page.jsp?node=31

And yes, we do have a framework called LTE, Lemontree Test Enabler!

Download Course Presentation

www.vivit-worldwide.org

Thanks!Magnus Nilsson

Lemontree Solution [email protected]

The difference that makes the difference. Make your test automation project a success story Magnus...

Documents

Transcript of The difference that makes the difference. Make your test automation project a success story Magnus...