A Test Automation Framework

A Test Automation Framework

Gregory SoloveyAlcatel–Lucent

www.qualitymanagementconference.com1

2www.qualitymanagementconference.com

Test related Processes

Provide decomposition/

abstraction

Design Tests

Build Harness

Automate Testing

Include in CI Framework

It is easier to

perceive error than

to find truth

Give me a lever and a

fulcrum and I shall move

the world

Divide and rule

He who lives must be

prepared for changes

Trust but verify

There is nothing new under the Sun


• 10 minutes to automate a test case• 10 minutes to identify an error• 10 minutes to update the testware

10/10/10 Goals for Test Automation Projects


To paraphrase a quote from Tolstoy’s Anna Karenina:

All successful test automation projects are alike; each unsuccessful automation project fails in its own way.

Test Automation


Test Tools ImplementationProgramming language Test language

Data structures

variables, arrays, hashes TS,UC,TC,TA, buffer

Control statements

if, case comparefor, while, until loop for TS,UC,TC,TA

Functions subroutine, procedure TC {set, test, compare, tear down}

Control flow

algorithm unconditional sequence

Users Developers prefer to use open source software which can be adapted

Testers prefer to use commercial tools, with technical support


The Three Pillars of a Framework

• testware hierarchy• testware organization• test language


Application Layer

Abstraction Layer

Middleware Layer

Driver Layer

Behavior Layer User documentsStructural Layer

Architectural documentsFunctional Layer

Requirements documentsInterface Layer

Design documents

System Hierarchies


Test Design Techniques:•Boundary analysis•Path sensitization

Derived Test Design Methods:•Arithmetic expressions •Relational expressions •Logical expressions •Algorithm•State machine•Instruction set•Syntax

Test Design Techniques

From test techniques to test methods, Professional Tester, issue 29, October 2014

http://www.professionaltester.com/files/PT-issue29.pdf







• Test Set (TS) is a part of the testware that verifies a particular architectural unit of the system.

• Use Case (UC) is a collection of end-to-end scenarios to cover the functionality of a subsystem.

• Test Case (TC) is a single verification act that moves the object-to-test from an initial state, executes a test, compares the actual and expected results, and returns it back to its initial state; a UC consists of TCs.

• Test Action (TA) is the simplest act of communication with the object-to-test. A TA supports the interfaces of the object-to-test (CLI, GUI, SNTP, HTTP).

Testware Hierarchy

Testware Hierarchy


TS1

UC1

TC1

TA1

TA2

TC2

TA3

TA4

TA5

UC2

TC3

TA6

TA7

TC4

TA1

TA3

TA4

TS2

UC3

TC7

TA30

TA32

TA36

TC8

TA41

TA42

UC4

TC12

TA50

TA55

TC13

TA90

TA92

UC5

TC21

TA77

TA78

Test Script 2

Testware File Organization


Language Commands interface action words

Test Case callsbusiness action words

Test Script 1

Test Script 3

Config fileTS1

UC1

TC1

TA1

TA2

TC2

TA3

TA4

TA5

UC2

TC3

TA6

TA7

TC4

TA1

TA3

TA4

TS2

UC3

TC7

TA30

TA32

TA36

TC8

TA41

TA42

UC4

TC12

TA50

TA55

TC13

TA90

TA92

UC5

TC21

TA77

TA78

Driver Libraries

Test case Libraries

Testware File Organization


Language Commands interface action words

Test Case callsbusiness action words

Config file

Driver Libraries

Test case Libraries

TS: ‘verifyMidleware’, ‘verifyAudioDrivers’, ‘verifyMapApplication’

UC: ‘vrfBufferManagement’, ‘vrfProgramMonitor’, ‘vrfFaultNotifications’, ‘vrfNetworkManager’

TC: ‘createBuffer’, ‘FillBuffer’, ‘SendMessage’, ‘SendAlarm’’

TA: ‘set’, ‘send’, ‘push button’, ‘compare’, ‘repeat’, ‘connect’

Testware Scale


Number per unit Total Unique

TS 2-50 25 25scripts 1-4 60 60

UC 2-10 250 250TC 2-30 5,000 50TA 2-10 25,000 25

Test Language• A test is presented as an unconditional

sequence of TS/UC/TC/TA executions• The instruction set includes the following

command groups: – run TS/UC/TC commands (one or multiple

times)– connection commands– compare commands– variable manipulation commands – limited control commands-pause/stop/exit


Reliability

• returning the object to its initial state• handling multiple echoes• support for stimulus-response

synchronization• multiple connection attempts


Output Buffer Structure


Text

Text

Text

Text

Text

Match Text

Test Case Structure


Test Case

Tear down phase

Test phase Action

Action

Action

Compare

Action

Action

Verdicts


Test object

Verdict Conditions

TA OKTIMEOUTERROR

‘MATCH’ detected in Output Buffer‘MATCH’ not found after predefined elapsed timePredefined error detected in Output Buffer

TC PASS

FAIL

All TAs are ‘OK’ & expected result is found in Output BufferAt least one TA is not ‘OK’ or a compare failed

UC PASSFAIL

All TCs ‘PASS’edAt least one TC ‘FAIL’ed

TS PASSFAIL

All UCs ‘PASS’edAt least one UC ‘FAIL’ed

userFuncLib.pm

Framework Structure


tesToy.pl

ssh/ telnet/ ksh

Failed CasesResult Log

Debug Log

Capture Log

Config file

connectLib.pm

testCommLib.pm

fullPathib.pm

Test ScriptTest ScriptTest ScriptTest Script

Test ScriptTest ScriptTest ScriptTC Library

Object to test

The test framework is a data driven application that reads test data, provides syntax checking and compiles the data into internal test structures, creates the necessary connections to the object to test, executes the tests and finally creates the log and result files.


Framework Functionality

Debug Log Structure• Supports TS/UC/TC/TA hierarchy and IDs• TA: output buffer, rules to capture it, and

how verdict was determined, based on the buffer content.

• TC: expected and real results and how verdict was determined, based on the comparison constraints.

• UC/ TS: how verdict determined, based on the children verdicts


10/10/10 Goals• Test design in 10 minutes

A framework shifts the testers’ focus from writing new test scripts to reusing existing ones

• Error detection in 10 minutesThe debug file structure allows for easy error identification and creation of “defect tracker”

• Testware maintenance in 10 minutesTestware is organized in a fashion that allows to make unique changes for any single changes in the object-to-test.


Example: Triangle Algorithm


a< b+c and b<a+c and

c<a+b

a = b a = c

b = c b =c

Not a triangle (NT)

Equilateral(ET)

Isosceles(IT)

Regular(RT)

0

2

1

1

1

1

0

0

00

Enter a, b, c

1

53

4

1


TC # description a b c result1-14 less than 3 parameters , symbols, negative and 0 values for

each parameterNT

15-17 a<b+c 6611

233

339

NTNTRT

18-20 b<a+c 233

6611

339

NTNTRT

21-23 c<a+b 233

339

6611

NTNTRT

24-26 2: a=b 11113

31111

999

RTITRT

27-29 3: b=c (a=b) 31111

31111

2113

ITETIT

30-32 4: a=c (a<>b) 31111

1199

12113

RTITRT

33-35 5: b=c (a<>b and a<>c) 11 9 9

31111

2113

RTITRT

Triangle Test Set

tesToy Configuration File


[tt_defaults] testwareDir = "/home/user/testware" swDesc="Triangle Test" prompt_base = "myServer!user> " time_out = 5 logDir = '/home/user/logs' keep_going = 1 expectEchoes = 1 debug_level = 3[test_files]TS "verify triangle application" "telnet" "myServer" run triangle.uc[libraries] /home/user/libs/general.lib[user_vars] myServer = "135.135.135.135"

tesToy Test Script


UC set test environment run general::setDir ("/home/user/bin")

UC verify out-boundary data run general::whichTriangle ( "", "", "", "error") run general::whichTriangle ("5", "", "", "error") run general::whichTriangle ("5", "5", "", "error") run general::whichTriangle ( "", "5", "5", "error") run general::whichTriangle ("5", "5", "a", "error")

UC verify that length of first side is less than other two run general::whichTriangle ( "6", "2", "3", "NT")

UC verify 'a=b' condition run general::whichTriangle ("11", "3", "9", "RT") run general::whichTriangle ("11","11", "9", "IT")

tesToy Library


# execute any command and compare results TC setDir send ("cd $P[0]") send ("pwd") compare ("$P[0]")reset_phaseend_of_case # execute the triangle application and verify output TC whichTriangle send ("bin/triangle.pl $P[0] $P[1] $P[2]") compare ($P[4])reset_phaseend_of_case

Further Reading• Embedding testability, Professional Tester, issue 27, August 2014,

8– 15. – how to build testability for an embedded system • From test techniques to test methods, Professional Tester, issue

29, October 2014, 4 – 14. – how to build the minimum number of tests that covers all implementation errors

• Automating Inside the Lines, Professional Tester, 2015, issue 30. – how to build a test automation framework

• A Continuous Integration Test Framework, Testing Experience, issue 28, December 2014, 30 – 34. – how to build a continuous integration test framework

• QA of Testing, Professional Tester, issue 28, October 2014, 9 – 12. - how to automatically monitor the test completeness


Thank you for attending this session

Send your comments or questions to [email protected]


mailto:[email protected]

A Test Automation Framework

Software

Transcript of A Test Automation Framework