ElfriedeDu

8/16/2019 ElfriedeDu

1/43

1

Automated Software Testing using Open Source

Testing ToolsBy Elfriede Dustin


2/43


3/43

3

http://www.idtus.com

Who is IDT$

• IDT specia"i%es in the design& deve"opment& and

imp"ementation of Automated Software Testing and

Re-Test (ATRT) and 'ua"it Assurance ('A)

so"utions – De"iver turn e automated test suite

• Automated test strateg and identification of highest paoff areas

to app" automation for our pro*ect

• Se"ection of +est automation too"s for our pro*ect• ,sing our eisting test cases or deve"op test cases for ou

• Test resu"ts documented in re.uirements tracea+i"it matri

– Training and pi"ot pro*ect imp"ementation with companies


4/43

4


Who is IDT$

• IDT specia"i%es in the design& deve"opment& and

imp"ementation of Automated Software Testing and

Re-Test (ATRT) and 'ua"it Assurance ('A)

so"utions – De"iver turn e automated test suite

• Automated test strateg and identification of highest paoff areas

to app" automation for our pro*ect

• Se"ection of +est automation too"s for our pro*ect• ,sing our eisting test cases or deve"op test cases for ou

• Test resu"ts documented in re.uirements tracea+i"it matri

– Training and pi"ot pro*ect imp"ementation with companies

•We are currently hiring Java Developers

•Plus we’d like to hire 2 or grad students to

work !or us part time " i.e. 2# hours week $

•must be US citizen


5/43

5


State of Software Testing -

Wh ATRT$

?


6/43

6


State of Software Testing -

Wh ATRT$

Utilize Automated Testing Strategies and

Technology to mpro!e "roducti!ity and #uality

M a n

u a l

A u to ma ted

$eduction inTest Days

DT "ro!ides Automated Software Testing Solutions

ncreased

Test

%o!erage

Manual vs. Automated

Spiral & $elease '

Spiral & $elease (

Spiral & $elease )

Spiral & $elease *

S i z e + % o m p l e , i t y o f S - B

a s e l i n e

Test

Days

TestDays

Test

Days

Test

Days

./0 or 1ore of O!erall De!elopment %ost is

Typically Spent on Testing


7/43

7


What is ATRT$

• ur definition of ATRT is/

Application and implementation of softwaretechnology throughout the entire Software

Testing Life Cycle (STL) and QA lifecycle;

with the goal to improve STL efficiencies andeffectiveness


8/43

8


ATRT Spans the Software Deve"opment

0ife #c"e – 1isua" 2ode"ing

%odeling and test case generation%odeling and test case generation

De!elopment Tools

%omponentsUse

cases

Design Build Assemble

Round-Trip Engineering

TestTest


9/43

9



0ife #c"e – Re.uirements

&rgani'es( tracks( ) controls re*uirements&rgani'es( tracks( ) controls re*uirements

+e*uirements ,racea-ility %atri "+,%$+e*uirements ,racea-ility %atri "+,%$

De!elopment Tools

%omponents

2isual 1odeling

$e3uirements 1anagement and"rocess Automation


10/43


11/43

11



0ife #c"e – Integrated Suite of Too"s

dditionally: dditionally:

%iddleware%iddleware

1n!rastructure1n!rastructure

De!ect ,rackingDe!ect ,racking

on!igurationon!iguration%anagement%anagement

%emory 3eak%emory 3eakDetectorsDetectors

Per!ormancePer!ormance,esting ,ools,esting ,ools

DocumentationDocumentation,ools,ools

othersothers

De!elopment Tools

%omponents

2isual 1odeling

Automated Testing Tools

$e3uirements 1anagement and

"rocess Automation

Software %onfiguration 1anagement

Defect Trac4ing

1iddleware

nfrastructure


12/43

12


Tpes of Software Test 0ife-

cc"e Support Too"s

• Which too"s are ou fami"iar with$


13/43

13


Tpes of Automated Test

Too"s

0ife-#c"e3hase

Tpe of Too" Too" Description

4usinessAna"sis

3hase

Business Modeling Records definitions of user needs and automates rapid

construction of flexible, graphical, client-server applications

ConfigurationManagement

Baselines important data repositories

Defect Tracing Manages s!stem life-c!cle defects

Technical Revie"Management

#acilitates communication, "hile automating thetechnical revie"$inspection process

Documentation

%enerators

Automate document generation

Re.uirementsDefinition

3hase

Re&uirementsManagement

Manages and organi'es re&uirements( allo"s for testprocedure design and test progress reporting

Re&uirements)erifiers

)erif! s!ntax, semantics, and testabilit!

*se Case

%enerators

Create use cases


14/43

14


Tpes of Too"s (#ont5d)

3rogramming3hase

Memor! +ea andRuntime rror

Detection

Detects runtime errors and memor! leas

ource Code Testing )erifies maintainabilit!, portabilit!, complexit!, and standards compliance

tatic and D!namic

Anal!'ers

Depict &ualit! and structure of code

0ife-#c"e3hase

Tpe of Too" Too" Description

Ana"sis andDesign 3hase

Database Design Develops second generation enterprise client-server s!stems

tructure Charts,#lo"charts, and

e&uence Diagrams

Manage processes

Test rocedure%enerators %enerate test procedures from re&uirements, design, or data and ob/ectmodels

!ntax Checers$Debuggers

erform s!ntax checing and have debugging capabilit!( usuall! available"ith built-in programming language compiler

2etrics Too"s Code 0Test1 CoverageAnal!'ers or Code

2nstrumentors

2dentif! untested code and support d!namic testing

*sabilit! Measurements rovide usabilit! testing as conducted in usabilit! labs


15/43


16/43

16


Dep"oment Approach –

Adapted AT02

%ollect&%reate5• +e*uirements• ,est ases• ,est Procedures• 4pected +esults• 1nter!ace 5peci!ications• 5ystem/omponent

on!iguration Description

dentify or De!elop

+ "ilot• ,est %anager

• ,est 5cripts• Data omparators• Per!ormance ,est ,ools• 671 +ecord/Play-ack• 5toring/+etrieving +esults

"opulate Test 1anager

with Test %ases and

$e3uirements

De!elop Test Scripts

from Test "rocedures

Test $esults5• Pass/8ail -y ,est ase• Pass/8ail 5ummary• ,est ,ime +e*uired


17/43

17


Wh pen Source

$


18/43

18


Wh pen Source$

• Advantage of pen Source/

– no "icensing issues6 i7e7 "icense cost6 maintenance6

etc7

– easi" modifia+"e

– adapta+"e

– "ightweight

– f"ei+i"it - not tied to one vendor


19/43

19


ATRTs ID! - !c"ipse

!c"ipse – this site "ists a"" the p"ugins

http/88www7ec"ipse-p"ugins7info8ec"ipse8p"ugins7*sp$categor9S#2:pager7offset9;:firstItem9<

!c"ipse8T3T3 – pen Source - Deve"oped + I42 – !c"ipse8Test and 3erformance Too"s 3"atform (T3T3)

offers a common etensi+"e framewor for the fo""owing

functions/

• Re.uirements 2anagement http/88sourceforge7net8pro*ects8osrmt8• #onfiguration 2anagement - Su+version

• Testing Too"s (http/88www7na++"e7com8!c"ipse-T3T3---Testing-

Too"s-f==>=7htm")

• 3rofi"ing/ defau"t ?ava app"ications& +ut can +e etended to other

apps


20/43

20


ATRTs ID! - !c"ipse

!c"ipse8Test and 3erformance Too"s 3"atform (T3T3)

offers a common etensi+"e framewor for the

fo""owing functions (cont)/

• Static Ana"sis/ defau"t for #@@ and ?ava& +ut can +eetended to other apps

• App"ication monitoring and "og ana"sis

• 4ui"d too" – ant

• m" input and output - using an open source ?ava pacage http/88*aarta7apache7org8ecs8 & which is a ?ava

"i+rar that has an A3I for constructing and outputting

207

• Test 2anager – STAB8STA


21/43

21


#urrent Test Too" Imp"ementation –

!amp"e and #ha""enges

1endor-provided (#apture83"a+ac) Too"/

Automated test tools mimic actions of the test engineer.

During testing, the engineer uses the e!board and mouse to perform

some t!pe of test or action.

Testing tool captures all e!stroes and subse&uent results, "hich are

baselined in an automated test script.

During test pla!bac, scripts compare latest outputs "ith previous

baseline.

Testing tools have built-in, reusable test functions.

Most test tools provide for non-intrusive testing( i.e., the! interact "ith

the 4application-under-test5 as if the test tool "as not involved.


22/43

22


#urrent Testing Too"s – #ha""enges

(cont)

In-Couse Software Deve"opment re.uired/

#apture83"a+ac too"s generate hard-coded va"ues6 testscripts are not reusa+"e& nor do the imp"ement software

deve"opment +est practices right out of the +o6 scripts needto +e modified7

#apture83"a+ac too"s don5t necessari" provide a"" testingfeatures re.uired6 code enhancements are often re.uired tomeet testing needs7

#apture83"a+ac too"s are not necessari" compati+"e withsstem engineering environment& and software testing scriptsneed to +e deve"oped in-house7

Deve"oped testing scripts are not cross-vendor compati+"e&even if the same scripting "anguage is used


23/43


24/43

24


Automated Test and Re-Test

0ATRT1

• ATRT inc"udes a common framewor comprised of

re-usa+"e software services

– pen Source Software Test Automation Bramewor

(STAB)

• 3rovides automated test procedures& test inputs& and

test resu"ts for SW components – Suita+"e for use + rd parties not associated with

deve"opment of the component


25/43

25


STAX is an automation system

With STAX you can automate and monitor your entire test environment ƒ System/product setup

ƒ Testcase distribution

ƒ Testcase execution ƒ Testcase results collection and analysis

STAX consists of

ƒ A programming language designed for automation

–Programs written in this language are called obs

ƒ An execution engine !similar to an interpreter for other programming/scriptinglanguages" which manages the execution and runtime behavior of the ob

ƒ A #$% application& called the STAX 'onitor& which provides a dynamicallyupdated view of your executing obs

ƒ A tool !STAX(oc" that is used to generate documentation for your STAX obs

What is STAX)


26/43


9 Windows ;

9 Windows < "and < 54$

9 Windows %illenium 4dition

9 Windows =, >ersion ?.#

9 Windows 2###

9 Windows @P

9 Windows 5erver 2## "12( 1A?( %DA?$

9 Windows >ista "12( %DA?$

9 3inu "1ntel2/A?( %DA?( PP2/A?$

9 3inu on '5eries "B0-it( A?0-it$

9 1@ >ersion ?...# or higher "20-it( A?0-it$ 9 5olaris "5parc$ 2.A and higher

9 5olaris %D &pteron A?0-it

9 CP07@ BB.## and higher "P0+15( 1A? 20-it and A?0-it$

9 &5/?## >;+2 or higher

9 '/&5 7=1@ >B.? and higher

STAB can run on/


27/43

27


STAB Interna" Services

• D2A% 3rovides diagnostics services Internal (!IA")

• D+A6 3rovides a means to s"eep a specified amount of time Internal (!#LA$)

• C78 !chos +ac a supp"ied message Internal (#C%&)

• #2+ 6TM A""ows ou to get and cop fi"es across the networ Internal ('S)

• 7A3D+ 3rovides information a+out eisting STAB hand"es Internal (%A!L#)

• 7+ 3rovides Ce"p on STAB error codes Internal (%#L)

• M2C Cand"es misce""aneous commands such as disp"aing the version of STAB

• that is current" running Internal (*ISC)

• 23% 3rovides a simp"e is-a"ive message Internal (I")

• R8C A""ows ou to start& stop& and .uer processes Internal (+&C#SS)

• 9** 3rovides a networ-ena+"ed I3# mechanism for STAB 3rograms Internal (Q,#,#)

• MA78R 3rovides networ-ena+"ed named event and mute semaphores Internal (S#*)

• R)2C A""ows ou to "ist services avai"a+"e on a machine and to eamine the• Re.uests that have +een su+mitted on a machine Internal (S#+-IC#)

• 7*TD8:3 3rovides a means to shutdown STAB and register for shutdown

• Eotifications Internal (S%,T!&.)

• TRAC 3rovides tracing information for STAB services Internal (T+AC#)

• TR*T Interfaces with STABFs securit Internal (T+,ST)

• )AR2AB+3rovides a method for maintaining configuration and runtime data

• (varia+"es) Internal (-A+)


28/43

28


STAB !terna" Services

The executable code for external TA# services resides outside of TA#roc, forexample in a ;ava /ar file, a C2 rovides a means to 'ip$un'ip$list$delete ?>ip$:in>ip compatible archives External (C++)


29/43

29


STA*

ƒ STA* provides the infrastructure on which STAX builds+ The full power of STA*and its services is exposed for use within STAX obs+

X',

ƒ The STAX programming language is based on X',+ This provides built-instructure to your obs& as well as providing a set of existing tools for use in

constructing your obs& such as X', !aware" editors and XS,T+

Python

ƒ The STAX programming language builds on Python to provide a rich and

accessible data model ƒ STAX.s Python integration also allows you to access the wealth of existing

Python libraries

ava

ƒ The STAX programming language allows you to access existing ava

classes/libraries& providing another source for reuse

Technologies used in STAX


30/43


31/43

31


4nd0to04nd utomation with 5,8 and5,@

4vent( 4vent%anager( ron

85( Process

Process( %onitor( 3og( >aria-le(ueue( +esPool

3og( Process

Process

4mail( C,,P

5

,

@

J

o

-

E!unctionF

EparallelF

Ese*uenceF

EprocessF

Esta!cmdF

EtestcaseF

EtimerF

E-lockF

EloopF

EmessageF

ElogF

EimportF

EGo-F

EiterateF

EparalleliterateF

EholdF

EreleaseF

EterminateF utomation ompletion

+esults =oti!ication

,estcase leanup

,estcase &utput nalysis

,estcase 4ecution

5ystem 5etup

utomation 5tartup


32/43


33/43

33


Disp"a App"ication Disp"a App"icationAna"sis : Disp"a

App"ication

Simu"ation of App"ication

3rocessing 2essages

Simu"ation of App"ication3rocessing 2essages

System Under Test

Test 2anager

RT2

4ug%i""a

AT$T

!ggp"ant

R83

1E# Ro+ot

R83

STAB8STA

3rototpe – 3roof of #oncept

Windows

Hased

3inu Hased


34/43

34


• A+"e to successfu"" eecute ATRT for H hours without operator

intervention

• During the tests/

1E#Ro+ot - Approimate" =J&;;; operator e stroes were

rep"aed STAB8STA - 2ore than


35/43

35


Additional Re&uirement

• 3ristine test environment re.uired – automated

testing too" cannot +e insta""ed on App"ication-

,nder-Test 3#

!ggp"ant – a""ows for remote capture8p"a+ac via

1E#server and K12 switch

1E#Ro+ot


36/43

36


Automated Software Testing

• Software testing parado

• What is it$


37/43

37


!ffective Automated

Software Testing

#urrent State/Deve"oping software in order to test software

Automate the test support deve"opment/ automated ID0

to code generation6 test data generation (using Bire!efrom EIST)6 compare uti"ities6 "og fi"e ana"%ers6

ur Automated Software Testing Goa"/• 2inimi%e Cuman interaction

– 3ossi+" get on par with Cardware Automated Testing$


38/43

38


Cardware Testing/ current

state

• 2an automated test generators for

manufacturing8 "ogic devices& circuit testing&

etc7& avai"a+"e i7e7 S!I8ATG& u2aster& etc7

• Automated testing without human interaction

is the norm for hardware testing


39/43

39


Cardware vs7 Software

AT

Cow is hardware testing different$

• Software #ha""enge/ – #hanging re.uirements

– G,I interfaces

– #ode access8avai"a+i"it

– Defects uncovered ear"ier in the software

deve"opment "ifecc"e are cheaper to fi• So"ution f"ei+i"it re.uired – pro+"em

dependent


40/43

40


3roposed so"utions

Bocus on Reusa+"e #omponents – Test Data 8 !pected Resu"ts

– Interfaces

– 2idd"eware

Assuming code and mode"s are avai"a+"e/

• 2ode" +ased testing6 automated test case generators +ased on mode"s• !c"ipse – effective ATRT ID!

• 3ro+es – see Apro+e – #ode instrumentation

• 4ui"ding se"f-testa+"e components

• ,sing standard "anguage 2B to Tet to generate automate test cases

G,I Testing too" ideas Eew too"s - other

• Woring with EIST to produce additiona" too"s


41/43

41


G,I Testing

• ?emm - *ava "i+rar to test ,I

• http/88*emm7net+eans7org8

• http/88a++ot7sourceforge7net8doc8overview7shtm"

• 3er" modu"es

• run via 1E#Server$


42/43

42


'uestions

$!mai" edustinLidtus7com with an fo""ow-up .uestions


43/43

43

IDT

We are hiringMM7

3"ease come see me now if interested or sendemai" edustinLidtus7com with an fo""ow-up

ElfriedeDu

Documents

Transcript of ElfriedeDu