Microsoft PowerPoint - GUI Testing-Tutorial(Lab Seminar)

Improving Automation in Improving Automation in GUI Testing

Presented by Gigon Bae

ⓒ KAIST SE LAB 2010

Content

IntroductionAbout GUICharacteristics of GUIs

GUI TestingChallenges and achievementsChallenges and achievements

Better test casesBetter test oraclesBetter test oraclesBetter test suite management

ConclusionConclusion

2ⓒ KAIST SE LAB 2009

Graphical User Interface(GUI)About GUI

p ( )

GUI is a type of user interface item that allows people to interact with programs in more ways than typing [ iki]interact with programs in more ways than typing.[wiki]

GUI software is an Event-driven Software(EDS)GUI provides a simple model of an eventGUI provides a simple model of an event H2

H3User eventsI k

User events

H1

H4

Invokeevent handlers

newDocActionPerformed(org.eclipse.swt.widgets.Event)

H5

Other EDSs

(org.eclipse.swt.widgets.Event)

Other EDSs•Web applications•Middleware•Object-oriented software•Multi-agent based systems


* “Graphical user interface,” in Wikipedia. [Online]. Available: http://en.wikipedia.org/wiki/Graphical_user_interface. [Accessed: Jan 11, 2010]

Multi agent based systems•….

Characteristics of GUIs Characteristics of GUIs

4ⓒ KAIST SE LAB 2009 *Referenced from [Gerrard, 1997]


Access to features of the system is provided via various mechanismsp

menu bars, buttons and keyboard shortcuts





GUI Allows multiple windows to be pdisplayed at the same time





GUI Allows multiple windows to be pdisplayed at the same timeVarious GUI objects are introduced such as radio buttons, scrolling lists, suc as ad o butto s, sc o g sts,and check boxes


GUI TestingCharacteristics of GUIs

The process of testing a product that uses a hi l i t f t it t it

g

graphical user interface, to ensure it meets its written specifications. [wiki]

The term ‘GUI Testing’ has the following meaning g g gin many previous papers.

Exercise the entire app. by generating only GUI inputsExercise the entire app. by generating only GUI inputs with the intent of finding failures that manifest themselves through GUI widgets[B 2010][Bryce, 2010]

*“GUI software testing,” in Wikipedia. [Online]. Available: http://en wikipedia org/wiki/GUI software testing [Accessed: Jan 11 2010]

※Widget: an element of a GUIe.g. window, text box, etc.


Available: http://en.wikipedia.org/wiki/GUI_software_testing. [Accessed: Jan 11, 2010]*R. C. Bryce, S. Sampath, A.M. Memon, “Developing a single model and test prioritization strategies for event-driven software”, TSE 10.

General Software Testing ProcedureCharacteristics of GUIs

g

?

Test Cases

+ =Outputs ExpectedOutputs

?

Test oraclesExecution

Software

Test oracle*:A mechanism for determining weather a test has been passed or failed

•* “Oracle(software testing) ” in Wikipedia [Online] Available:


•* Oracle(software testing), in Wikipedia. [Online]. Available: http://en.wikipedia.org/wiki/Oracle_(software_testing). [Accessed: Jan 25, 2010]

Differences with Conventional SWCharacteristics of GUIs

?

Test Cases


?

Test oraclesExecution

Software

Conventional SW GUI SW

Test cases usually consists of a single set of an event seq. y ginputs

q:effect of event may dependent upon the effects of prev. seq.

Expected outputs

the output results from processing the input:the form of the out can be readily specified

there is no specific output: each action affects the state of the GUI

specified

e.g. values of a certain set of variables

11ⓒ KAIST SE LAB 2009 *Referenced from [Memon, 2000]

Challenges in AutomationChallenges and Achievements

g

+ =Outputs Expected

?

Test Cases

pOutputs

Test oracles

?Ver. 1?

Test Cases


Test oraclesVer. 2

Better test cases: How to sample the space of possible inputs: How to sample the space of possible inputs

Better test oracles: How do we know if the program is in the correct state?

Better test suite management: How do we manage the test suite when making the next iteration?


Challenges for Better Test CasesChallenges and Achievements

g

?

Test Cases


?

Test oracles

Too many permutations of event to test (dealing with domain size) E.g. a very small programs such as Microsoft WordPad has 325 possible GUI operations[Memon 1999]operations[Memon,1999]

There may be complicated interactions among events(dealing with sequences)


Methods of Test Case Generation Challenges and Achievements – Better Test Case Generation

Generation 1 : Totally manual generationBy hand

Generation 2 : Semi-automated generationGeneration 2 : Semi automated generationEvent capture

Generation 3 : Automated generationModel based generationModel-based generation


Totally manual Test Case GenerationChallenges and Achievements – Better Test Case Generation – Totally Manual Generation

y

Labor-intensive and takes much time!

Manually generate

…Test

Test Cases(scripts)

…Software


Test Case Generation using Event Capture

Challenges and Achievements – Better Test Case Generation – Semi-automated Generation

Capture event sequences using toolsAbbot, Winrunner,Quick Test Professional, etc.

Recorded Test Script

16ⓒ KAIST SE LAB 2009*Images are retrieved from http://abbot.sourceforge.net/doc/images/costello.png

Automated GUI Test Case Generation Technique

Challenges and Achievements – Better Test Case Generation – Automated Generation

1. Genetic algorithm based2. FSM model based3. AI planning based4 GUI graph model based4. GUI graph model based


1. GA Based Test Case GenerationChallenges and Achievements – Better Test Case Generation – Automated Generation

Simulate ‘novice’ system usage[Kasik, 1996]

Assumes that experts take a direct path to reach a goal state whereas novice users takes longer, indirect pathsU i i i l iUses an expert to generate an initial concise event sequence manually and then uses genetic algorithm techniques to generate longer sequencestechniques to generate longer sequences.Depend on the paths taken by the expert user


2. FSM Model Based Test Case Generation


State is represented in terms of widgetswidgetsEvent triggers state transition in FSMA test case is a path(sequence ofA test case is a path(sequence of events) in the FSMState space explosion problem canState-space explosion problem can be arose

FSM Model*


Initial StateA State for ‘Open File Dialog’

*The image is retrieved from [White, 2000]

Efforts to improve FSMChallenges and Achievements – Better Test Case Generation – Automated Generation

VFSM(Variable Finite State Machine)[Shehady&Siewiorek 1997]

p

Machine)[Shehady&Siewiorek, 1997]: augment an GUI FSM with global variables which can assume a finite number of values during the execution of a test caseduring the execution of a test case

Regular expression[Belli, 2001]g p: convert a GUI FSM into simplified regular expressions

Regular Expression

VFSM Model

VFSM FSM

Regular Expression


VFSM Model

3. AI Planning Based Test Case Generation


Exploit a well-studied technique in AI to find operation sequences to reach a goal state[M 2001]sequences to reach a goal state[Memon, 2001]

Planning operators should be prepared• Pre/post conditions for each eventp

Test cases are automatically generated from tasks(pairs of initial and goal states) using AI PlannerT t l t ti ll t d b iTest oracles are automatically generated by using

postconditions

Goal State

Initial State


Example of AI Planning


Operator

Example of OperatorExample of State

set-background-colorParameter

w:window; col:colorPrecondition

isCurrent(w)isCurrent(w)background-color(w,oldColor)oldColor!=col

Effects:background-color(w,col) window(w19),window(w29),background-

l ( 19 bl ) i t( 19)

set-background-color(w19,yellow)Huge effort to create/maintain

color(w19,blue),is-current(w19), …

goperations!!

For modified GUIsShould change planning g p goperatorsShould do replanning

Works well for small app.i d ( 19) i d ( 29) b k d


ppwindow(w19),window(w29),background-color(w19,yellow),is-current(w19), …

4. GUI Graph Model Base Test Case Generation


Event Flow Graph(EFG)[Memon, 2001]

Event Interaction Graph(EIG)[Memon,2005]

Event Semantic Interaction Graph(ESIG)p ( )[Yuan, 2007,2010]


Graph Model-based Test Case Generation


Extracts GUI events and interaction using reverse engineeringCreate a graph model(EFG EIG ESIG)Create a graph model(EFG,EIG,ESIG) Generate test cases(event sequences) based on some criteria using graph traversal algorithms

Event sequences start from a event which can be invoked from an initial state S0

Coverage criteria such as ‘N-way interaction coverage’ can be appliedg y g pp

‘Di ’ t b f dGUI event

‘Discover’ event may be performedimmediately after ‘Help’ event

: Can be performed from an initial state S0

Example of EFG(Event Flow Graph)

24ⓒ KAIST SE LAB 2009 *The image is retrieved from [Memon, 2003]

Example of EFG(Event Flow Graph)

GUI Graph Model


Event Flow Graph(EFG)[Memon, 2001]Nodes represent GUI eventsEdge from node nx to ny means that the event represented by ny may be performed immediately after the event represented by node nx

Event Interaction Graph(EIG)[Memon,2005]Based on the observation that the code for certain types of events(e.g. events that open pull-down menus) is straightforward and usually generated automatically by visual GUI-building toolsE.g. ‘Paste’ ‘Edit’;‘Paste’

Event Semantic Interaction Graph(ESIG)[Yuan, 2007,2010]

Use GUI’s states as the feedback to identify important edges• Called Event Semantic Interaction Edges

Generate new longer test cases covering those edgesGenerate new longer test cases covering those edges


Challenges for Better Test OraclesChallenges and Achievements

g

?

Test Cases


?

Test oracles

There is no ‘good’ mechanism to check the behavior of a programprogram

Outputs can be represented by the changes in GUIs such as bitmaps, layouts, fonts, texts, and so on

E1+ =E2… En

Is this correctb h i ?


behavior?

Test Oracle in GUIChallenges and Achievements – Better Test Oracles

Judge a test case to be failed if the execution of the test case leads to a crash(exception)case leads to a crash(exception)Manual inspection by human

See what is suspicious on the screenSee what is suspicious on the screen

Using capture/replay toolsManually select some widgets/properties to store info duringManually select some widgets/properties to store info. during capture session.

Using formal specificationsE.g. pre/post conditions for each GUI eventOracle info. can be automatically derived [Memon, 2001]

Using execution extractionA test case is executed on previous(presumably correct) version of SW and its state is retrieved and stored as oracle infoSW and its state is retrieved and stored as oracle info.[Memon, 2003]


Challenges for Better Test Suite Management

Challenges and Achievements

+ =Outputs Expected

?

Test Cases

pOutputs

Test oracles

?Ver. 1?

Test Cases


Test oraclesVer. 2

Test suite repairing problemA small changes in the layout of GUIs can cause failures in the execution of previous test casesof previous test cases

Test suite reduction problemMany redundant/meaningless test cases are hard to manage and take y g gmuch time to execute

Test suite prioritization problemWe want to detect faults faster if the time is not enough to execute all testWe want to detect faults faster if the time is not enough to execute all test cases


Test Suite RepairingChallenges and Achievements – Better Test Suite Management

p g

Uses repairing transformations to repair broken test cases*Memon, “Automatically repairing event sequence-based GUI test suites for regression testing,” TOSEM 08

Version 1

modified

Version 2Version 1

Test suite for ver.1 Test suite for ver.2

Version 2

… …

Unusable

Usable

Not repairable

Repaired New

RepairSelect Discard


…UsableTest cases

RepairedTest cases+ New

Test cases+ =

Test Suite RepairingChallenges and Achievements – Better Test Suite Management

p g

Uses repairing transformations to repair broken test cases*Memon, “Automatically repairing event sequence-based GUI test suites for regression testing,” TOSEM 08

R i i T f tiRepairing Transformations

For deleted event ei For deleted edge(ei,ej)


Test Suite ReductionChallenges and Achievements – Better Test Suite Management

Uses call-stack coverage to reduce a test suite* McMaster, and Memon, “Call-Stack Coverage for GUI Test Suite Reduction,” TSE 08.

modified

Used in regression testingTest

… …Reduce redundant test cases

…maintaining the original suite’s

Too many test cases…

Original test suite Reduced test suiteg g

fault-detection effectiveness as much as possible


Test Suite ReductionChallenges and Achievements – Better Test Suite Management

Uses call-stack coverage to reduce a test suite* McMaster, and Memon, “Call-Stack Coverage for GUI Test Suite Reduction,” TSE 08.

Program

Execute

Test Case 1(tc1)

Program

Execute

Test Case 2(tc2)

tc tc iff two sets ofProgram

e

f g

a

b c

Program

e

f g

a

b c

tc1 ~ tc2 iff two sets of unique call stacks are equal

f g

c

Thread 1

b c

d d h

Thread 2

f g

c

Thread 1

b c

d d h

Thread 2 Reduce to one test case!

: Set of maximum-depth call stacks

: Set of Unique Call Stacks for a test case


: Set of Unique Call Stacks for a test case

Test Suite PrioritizationChallenges and Achievements – Better Test Suite Management

Evaluates various prioritization criteria

•Bryce, and Memon, “Test suite prioritization by interaction coverage,” DoSTA 07.* Bryce, Sampath, Memon, “Developing a single model and test prioritization strategies for event-driven software”, TSE 10

If there is not enough time to testT1 T2 T3 T4 T5 T6

T

Need prioritization of test cases based on some criteria for effectivenessT1 T2 T3 T4 T5 T6

T’

- N-way interaction- Unique event- Unique window

L t h t- Long to short- Short to long- …


ConclusionConclusion

GUI testing is important and yet costlyN d t tiNeeds automation

Better Test Cases: help generate new better test inputsGenerate event sequencesGenerate event sequences

Better Test Oracles: help generate better test oraclesCheck behavior of test inputsCheck behavior of test inputs

Better Test Suite Management: help manage test suitesTest suite repairingp gTest suite reductionTest suite prioritization


My Interest on GUI Testingy g

Generate better test case generation in GUI t titesting

Using information from the execution, static analysis, dand so on

Take advantage of unit test cases


Reference“The Economic Impacts of Inadequate Infrastructure for Software Testing,” NIST, 2002.G. J. Myers, “The Art of Software Testing,” Wiley, 1979.y g yA. M. Memon, “GUI Testing: Pitfalls and Process,” Computer, vol. 35, no. 8, pp. 87-88, 2002. P. Gerrard, “Testing GUI Applications,” EuroSTAR, 1997.A. M. Memon, I. Banerjee, and A. Nagarajan, "DART: A Framework for Regression Testing Nightly/daily Builds of GUI Applications," Proceedings of the International Conference on Software Maintenance 2003, 2003.F. Belli, “Finite-State Testing and Analysis of Graphical User Interfaces,” in Proceedings of the 12th International Symposium on Software Reliability Engineering, 2001, pp. 34.L. White, and H. Almezen, “Generating Test Cases for GUI Responsibilities Using Complete Interaction Sequences ” in Proceedings of the 11th International SymposiumComplete Interaction Sequences, in Proceedings of the 11th International Symposium on Software Reliability Engineering, 2000, pp. 110.R. K. Shehady, and D. P. Siewiorek, “A Method to Automate User Interface Testing Using Variable Finite State Machines,” in Proceedings of the 27th International Us g a ab e e S a e ac es, oceed gs o e e a o aSymposium on Fault-Tolerant Computing (FTCS '97), 1997, pp. 80.A. M. Memon, M. E. Pollack, and M. L. Soffa, “Hierarchical GUI Test Case Generation Using Automated Planning,” IEEE Trans. Softw. Eng., vol. 27, no. 2, pp. 144-155, 2001.D. J. Kasik, and H. G. George, “Toward automatic generation of novice user test scripts,” in Proceedings of the SIGCHI conference on Human factors in computing systems: common ground, Vancouver, British Columbia, Canada, 1996, pp. 244-251. 37ⓒ KAIST SE LAB 2009

ReferenceX. Yuan, and A. M. Memon, “Generating Event Sequence-Based Test Cases Using GUI Run-Time State Feedback,” IEEE Transactions on Software Engineering, vol. 99, no. 1, 2010.P. A. Brooks, and A. M. Memon, “Automated gui testing guided by usage profiles,” in Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering Atlanta Georgia USA 2007 pp 333 342software engineering, Atlanta, Georgia, USA, 2007, pp. 333-342.A. Memon, A. Nagarajan, and Q. Xie, “Automating regression testing for evolving GUI software,” Journal of Software Maintenance and Evolution, vol. 17, no. 1, pp. 27-64, January, 2005.January, 2005.A. M. Memon, and Q. Xie, “Studying the Fault-Detection Effectiveness of GUI Test Cases for Rapidly Evolving Software,” IEEE Trans. Softw. Eng., vol. 31, no. 10, pp. 884-896, 2005.A. M. Memon, M. L. Soffa, and M. E. Pollack, "Coverage criteria for GUI testing," ESEC/FSE-9: Proceedings of the 8th European software engineering conference held jointly with 9th ACM SIGSOFT international symposium on Foundations of software engineering ACM Press 2001 pp 256 267engineering, ACM Press, 2001, pp. 256-267.X. Yuan, and A. M. Memon, “Using GUI Run-Time State as Feedback to Generate Test Cases,” in Proceedings of the 29th international conference on Software Engineering, 2007, pp. 396-405.2007, pp. 396 405.


S. McMaster, and A. M. Memon, "Call Stack Coverage for Test Suite Reduction," ICSM '05: Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM'05), IEEE Computer Society, 2005, pp. 539-548.S. McMaster, and A. Memon, “Call Stack Coverage for GUI Test-Suite Reduction,” in Proceedings of the 17th International Symposium on Software Reliability Engineering, 2006 pp 33 442006, pp. 33-44.S. McMaster, and A. M. Memon, "Fault Detection Probability Analysis for Coverage-Based Test Suite Reduction," ICSM '07: Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM'07), IEEE Computer Society, 2007.Conference on Software Maintenance (ICSM 07), IEEE Computer Society, 2007.S. McMaster, and A. Memon, “Call-Stack Coverage for GUI Test Suite Reduction,” IEEE Trans. Softw. Eng., vol. 34, no. 1, pp. 99-115, 2008.A. M. Memon, M. E. Pollack, and M. L. Soffa, "Automated test oracles for GUIs," , , , ,SIGSOFT '00/FSE-8: Proceedings of the 8th ACM SIGSOFT international symposium on Foundations of software engineering, ACM Press, 2000, pp. 30-39.A. M. Memon, I. Banerjee, and A. Nagarajan, "What Test Oracle Should I Use for Eff i GUI T i ? " P di f h IEEE I i l C fEffective GUI Testing?," Proceedings of the IEEE International Conference on Automated Software Engineering, IEEE Computer Society, 2003, pp. 164-173.


Benchmark

http://www.cs.umd.edu/~atif/Benchmarks/


Example of Event Handlersp


Accessibility in GUIy

There is a no magical way of accessing GUI statesI l / l tf th i t lIn some languages/platforms, there exist several mechanisms to access GUI states

In Java applicationIn Java application• Accessibility feature in Swing, java.awt.Robot class• FEST: http://fest.easytesting.org• Abbot: http://abbot.sourceforge.net/doc/overview.shtml

In Microsoft GUI objects• MSAA(Microsoft Active Accessibility) or UIA(Microsoft UI• MSAA(Microsoft Active Accessibility) or UIA(Microsoft UI

Automation)[1]

Nowadays most platforms include accessibility technologies since it is mandated by the law[2]

[1] htt // i ft / bl / i ft/hi t


[1] http://www.microsoft.com/enable/microsoft/history.aspx[2] http://www.access-board.gov/508.htm

Event-driven Software(EDS)( )

Execution environment

Event

Can handle an event of type e1

Dispatcher

H1

H2H4

e1

e2 H3

H5H6

Eventse2…e6

Registered event handlers

Affects the system’s state


Test Case Generation for GUI*Challenges and Achievements – Better Test Case Generation

Test suite should satisfy the followings to t b tt t f t tgenerate a better set of test cases

Covers all the functionality of systemFully exercises the GUI itself

The difficulty in accomplishing this task arise from following problemg p

Deal with domain size• E.g. MS WordPad has 325 possible GUI operations

Deal with sequences• Some functionality of the system may only be accomplishable

b f ll i l f GUI tby following some complex sequence of GUI event

44ⓒ KAIST SE LAB 2009* “GUI software testing,” in Wikipedia. [Online]. Available: http://en.wikipedia.org/wiki/GUI_software_testing. [Accessed: Jan 11, 2010]

Modeling a GUI State


The state of a GUI at a particular time t is the set S f i l {( )} h W P Vof triples {(wi, pj, vk)}, where wi∈W, pj∈P, vk∈V

W(set of widget) = {w1,w2,…,wl}• E.g. buttons, panels, text fields

P(set of properties) = {p1,p2,…,pm}E b k d l i f t• E.g. background color, size, font

V(set of values) = {v1,v2,…,vn}• E g red bold 16pt• E.g red, bold, 16pt

Events(‘편집’-‘이동’)


S0S1

Example of a GUI State


Window1Caption 제목없음 -메모장

Font MS Shell Dlg

Minimize Box TRUE

Visible TRUE

… …… …

Window2Caption 줄이동

Bt 1 Font MS Shell Dlg

Minimize Box FALSE

Region (857,488)-(1122,624)

… …

Btn1Caption 이동

Type PUSH_BTN

… …

{(Window1,Caption, “제목없음-메모장”, (Window1,Font, “MS Shell Dlg”),(Window1,MinimizeBox,TRUE),(Window1,Visible,TRUE),

… …

( , , ),( , , ),(Window2,Caption, “줄이동”, (Window2,Font, “MS Shell Dlg”),(Window1,MinimizeBox,FALSE),(Window2,Region,(857,488)-(1122,624),(Btn1,Caption, “이동”), (Btn1,Type,PUSH_BTN), … }

46ⓒ KAIST SE LAB 2009 * The image is retrieved from [Memon, 2003]

GUI Graph Modelp

Event Flow Graph(EFG)[Memon, 2001]Nodes represent GUI events

*Memon&Soffa&Pollack, "Coverage criteria for GUI testing," FSE 2001.

Nodes represent GUI eventsEdge from node nx to ny means that the event represented by ny may be performed immediately after the event represented by node nx


GUI Graph Modelp

Event Interaction Graph(EIG)[Memon,2005]Based on the observation that the code for certain types of events(e g

* Memon&Xie, “Studying the Fault-Detection Effectiveness of GUI Test Cases for Rapidly Evolving Software,” TSE 2005.

Based on the observation that the code for certain types of events(e.g. events that open pull-down menus) is straightforward and usually generated automatically by visual GUI-building toolsThi d d t i t t ith d f th tThis code does not interact with code for other events

• Expect that very few errors are revealed

48ⓒ KAIST SE LAB 2009 *The image is retrieved from [Memon, 2005]

GUI Graph Modelp

Event Semantic Interaction Graph(ESIG)[Yuan, 2007,2010]Use GUI’s states as the feedback to identify important edges

* Yuan&Memon, “Generating Event Sequence-Based Test Cases Using GUI Run-Time State Feedback,” TSE 2010.


Generate new longer test cases covering those edges


*This slide referred to http://www.cs.umd.edu/~atif/Teaching/Spring2008/StudentSlides/Bao_execution_feedback.ppt

GUI Graph Modelp

Event Semantic Interaction Graph(ESIG)[Yuan, 2007,2010]Use GUI’s states as the feedback to identify important edges

* Yuan&Memon, “Generating Event Sequence-Based Test Cases Using GUI Run-Time State Feedback,” TSE 2010.


Generate new longer test cases covering those edges

ESIESI

ESI

ESI

ESI



Event Semantic Interaction

Heuristic: Two events executed together results differently than executed in isolationthan executed in isolation

Should be tested together(semantic interaction)

<e1>Select Eclipse

<e2>Drag

<e1; e2>Drag



Event Semantic Interaction(con’t)( )

Defined 6 predicates and 3 contextsPredicate 1: Predicate 4:Predicate 1:

not affected by e1(S0) and e2(S0)affected by e2(e1(S0))

Predicate 4: affected by e1(S0)affected differently by e2(S0) affected differently by e2(e1(S0))

Predicate 2: not affected by e2(S0)affected by e1(S0) affected differently by e2(e1(S0))

Predicate 5: a new widget is created by event ex(e1 or e2)a new widget is created with a differentaffected differently by e2(e1(S0))

Predicate 3: not affected by e1(S0)

a new widget is created with a different value for some property by the sequence <e1 ;e2>

affected by e2(S0) affected differently by e2(e1(S0))

Predicate 6:a widget is disabled in initial statethe widget is enabled by e1

Context 1 : events in modeless windowContext 2 : events in same modal window


Context 3 : events in parent and child modal window

Regression TestingChallenges and Achievements – Better Test Suite Management

g g

Selective retesting of a system or component to verify that modifications have not caused unintended effects and that themodifications have not caused unintended effects and that the system or component still complies with its specified requirements[IEEE Standard Glossary]

… … ?Test suite for ver.1 Test suite for ver.2

modified

Version 1 Version 2

modified


GUI Challenges for Test Suite Reductiong

Interesting behaviors related to context in which GUI t i k dGUI events are invoked.Multi-language development.Object-orientation

Virtual func. callsVirtual func. callsReflectionCallbacks (used as event handlers)Callbacks (used as event handlers)…

Multi threadingMulti-threadingExtensive use of libraries and frameworks.


Call-stack coveragegS. McMaster, and A. Memon, “Call-Stack Coverage for GUI Test Suite Reduction,” IEEE Trans. Softw. Eng., 2008.

Using call stacks as a coverage criteria to address GUI challengesCall-stack example

java.util.ArrayList.<init>java.util.ArrayList. initjava.util.ArrayList.<init>sudoku.SudokuPanel.<init>sudoku.MainFrame.<init>sudoku.Main$2.runsudoku.Main$2.runjava.awt.event.InvocationEvent.dispatchjava.awt.EventQueue.dispatchEventjava.awt.EventDispatchThread.pumpOneEventForFiltersjava.awt.EventDispatchThread.pumpEventsForFilterj p p pjava.awt.EventDispatchThread.pumpEventsForHierarchyjava.awt.EventDispatchThread.pumpEventsjava.awt.EventDispatchThread.pumpEventsjava.awt.EventDispatchThread.run


j p

Call-stack coverage(con’t)g ( )

java.util.ArrayList.<init>java.util.ArrayList.<init>sudoku.SudokuPanel.<init> This call-stack meanssudoku.MainFrame.<init>sudoku.Main$2.runjava.awt.event.InvocationEvent.dispatchjava awt EventQueue dispatchEvent

This call stack meansthese call-stacks were

once activejava.awt.EventQueue.dispatchEventjava.awt.EventDispatchThread.pumpOneEventForFiltersjava.awt.EventDispatchThread.pumpEventsForFilterjava.awt.EventDispatchThread.pumpEventsForHierarchyjava.awt.EventDispatchThread.pumpEventsjava.awt.EventDispatchThread.pumpEventsjava.awt.EventDispatchThread.run

Needs only record the maximum depth stack


Call stacks and Test Suite Reduction

DefinitionCall Stack c = <m1, m2, … mn>

Superstack cs = <m1,m2, … mn, … mi>, i > n

Set of all unique stacks generated by a C(t)Set of all unique stacks generated by a thread t

C(t)

Set of Maximum-Depth Stacks (incorporates all unique stacks on a

C(t)max = {c ∈ C(t) | cs = ∅}(incorporates all unique stacks on a thread)

Set of Threads T = <t1, t2, … tn>

Set of Unique Call Stacks for a program C (I)= ∪ { C(t) | t ∈ T }Set of Unique Call Stacks for a program input I

Cmax(I) ∪ { C(t)max | t ∈ T }

tc1 ~ tc2 iff C (tc1)=C (tc2)tc1 tc2 iff Cmax(tc1)=Cmax(tc2)

Stacks(TS)= ∪ { Cmax(tc) | tc ∈ TS }, then


Stacks(TSfull)=Stacks(TSreduced)

Microsoft PowerPoint - GUI Testing-Tutorial(Lab Seminar)

Documents

Transcript of Microsoft PowerPoint - GUI Testing-Tutorial(Lab Seminar)