Олег Бунин

Testing JavaFX UITips from JavaFX quality team

Alexandre (Shura) IlineJava and JavaFX quality architectOracle

Agenda

UI testing overviewTest automation approachesJemmy libraryTest automation effectivenessTest library design approaches

UI testing … by Wikipedia

«GUI software testing is the process of testing a product that uses a graphical user

interface, to ensure it meets its written specifications.»

UI testing … most often ...

«Checking whether usage of a product UI leads to results expected by the the person

who performs testing»

Start text editorPush «File/Open»Verify file chooser directorySelect some fileVerify editor area contentVerify application titleVerify buttons availabilities....

UI Testing Manual Automated

Initial step Design test specification. Create tests.

Establish regular runs.

Ongoing Click, click, click Analize results

File bugs

Change the specification Change the tests

Qualification High, low for test executors High

Effectiveness Low High

Other Inexpensive to start Continuous quality monitoring , inexpensive to reuse

Fun Bo-o-o-ring Much like programming

Development

Continuous build

Commit

BuildExecuted

automagicallyafter commit

Success

Analysis.Rollback!!!

Code changes

NoYes.

PromoteCode is compilableTest further

Continuous build with testing

CommitBuild Success

TestingIs it working?

Passed

Analysis.

Rollback!!!

Code changes

Testchanges

No

No

Test furtherBuild is good

Code line is healthyGo on ...

Yes

Yes. = Compilation

successful

Automation approach

Record && Replay Coding

Test creation Inexpensive*Usually just repeating manual test in special environmentMust be accompanied by other means

Expensive*Consists of programming

Test execution Does not depend on approach

Test maintanance

Higher (in most cases)*Very much depends on test format

Lower*Depends on principles of building test library

Test analysis Does not depend on approach

(*) Much more information closer to the end of this presentation

Jemmy

Jemmy v2Started as a tool to tests TeamWare UI (1999)Used for NetBeans extensions (2000)Official test tool for NetBeans (2001)Open-source (2001)

Jemmy v3Started in (2008) as a proof of concept experiemntExtended to support JavaFX (2009)Opensource with support of JavaFX 1.2 (2009)Developed in close-source for 1.3 since then

Jemmy usagesV2 (Swing/AWT)Internal: Swing, NetBeans, GlassfishExternal: Used by: Boeing (x2), agfa.com, compuware.com, attix5.com,

exgate.tek.com, globo.com, gmx.de, non.agilent.comEvaluated by: syntelinc.com, covast.com, visus-tt.com, covast.com,

nokia.com, twt.com, ubs.com

V3 (FX)Internal: JavaFX, Authoring Tool, JavaStore

Demo...Finally!

Jemmy v3JavaFX

AWT/Swing

SWT

Scenegraph

LCDUI

JemmyCore

Interfaces

AWT robot FX robot

Hierarchy

Input

Scene, nodecontainers

Scroller, slider, list, text, ...

Genericinterfaces

Covered in this presentation

UI test

Find Do VerifyPass Pass

Pass

Fail Fail Fail

Failure analysis

Find next controlTo perform operation

On in

Verify that expected

State reached

Perform necessary actions

LookupBy IDEasiest but may not be possible«open_file_btn»

By typeMost commonButton, text, combo box, etc

By indexUnavoidable2nd button with text «browse»

By toString()Sucksprojectmanager.Main$Main$Script$1Scene$ObjLit$20@4d16318b

By text, tooltip, associated labelBest, if possibleButton with text «browse»

UI c

ou

ld b

eve

ry c

om

plic

ate

d

UI c

oul

d b

eve

ry c

om

plic

ate

d

UI c

oul

d b

eve

ry c

om

plic

ate

d

Lookup demo

InterfacesInterface Control types Description

Mouse, Keyboard, Drag

Enything Low level input

Parent Containers, list (for its content) Something you could look within

Selectable Toggle button, radio button, combobox, check box, lists, etc

A control which provides limited number of choices

CaretOwner Text box, scroll bar, slider A control which has a number value which changes within some range

Text Text box Editable text container

Interfaces demo

VerificationsUI feedback Non UI feedback

Dialog displayed

Test changed

Image updated

Progress bar changed position

File created

Database updated

Sunset happened :)

Waiting

UI is a multy-thread environmentThings happen in backgroundTest code is in another thread

Not much could be really verifiedEverything should be waited for

Verification demo

Test automation effectiveness

TD + *T

SN

R

TM * N

RN

C*E

A =N

C*

EA – automation effectiveness

NR and N

C are characteristics for a product.

TM is a characteristic of a test suite

TD and T

S depend on test automation approach

Smaller TD and T

S - higher the E

A.

1 2 3 4 5 6 7 80

5

10

15

20

25

0

1

2

3

4

Nc = 1 Ratio

Auto Manual Effect

Time

1 2 3 4 5 6 7 80

5

10

15

20

25

0

1

2

3

4Nc = 3 Ratio

Auto Manual Effect

Time

Assumptions: TM = 1 engineer*week T

S = 0.1 * T

M T

D = 5 * T

M N

R = 8

Td or T

s – what to minimize

TS - if (N

C * N

R) is big

Multi-platform

Compatibility with external products (servers, browsers, ...)

Long-living

TD - if (N

C * N

R) is small

Proof of concept

Preview

Tests fail every now and then ...

… because the tested UI is changed

Tests fail every now and then ...

Ah! And also because errors are made ...

End2End demo

Ts continuends

Time spent on What to do %% of time

Allocating failures Use test harness 1% - 5%

Anderstandint the failure reason

Use test logging, save images, save UI state

10% - 80%

Fix the tests Move common code to the library (*)

90% - 10%

(*) The only way it is different from programming is that there are a lot of tests. More on this later.

Harness

TestNGTestNG

Understanding failureLogging

UI operationsGoing to push ... buttonSelecting ... menu

UI responcesFound list view: …

UI hierarchyScene «…»

Group «…»Button «…»

UI statesImages

DemoJemmy logging functionality

Fix the testsOne UI, many-many tests.

Do as few test code changes as possible

Ideally … one change in test code for one change in UI code.

Decrease maintenance costImplement reusable operations in a library

openFile(String filename)

Decrease implementation dependencieslookup(Button.class, new ByText<Button>(«Low»))

Increasing stability.wrap(0).mouse().click();Thread.sleep(1000);...

DemoRefining test.

Library design approaches

Remember the formula?

TD + *T

SN

R

TM * N

RN

C*E

A =N

C*

EA – automation effectivenessTo be used for every particular product.

NR and N

C are unique for a product.

TM is a characteristic of a test suite.

Smaller TD and T

S - higher the E

A.

Coefficient depend on the way you write your tests

TS mainly consists of time for test modification.

When product changes, tests need to be changed accordingly.

Many tests! Hundreds.

“Less changes of test code per a change in the product UI.”

Ideally ... no more than one.

But ... how? You are the coders – you know:

Move code to test library.

Application domain modelC

ar

rec

ord

Ca

r re

co

rd

Color

Model Make

Year License plate

VIN

...

That's ... car catalog of some sort

Application UIProduct UI

Coordinatesclick(134,32) //selects some record

click(215,122) //hits “Properties”

sleep(5) //sleeps to let dialog be painted

click(64,182) //expands color combo

click(235,182) //selects Gray

click(235,212) //hit OKTT

dd ~= 1.1 * T ~= 1.1 * T

mm, , TT

ss ~= 1 * T ~= 1 * T

mmNever tried, but ...

WidgetsFind “Car records” frameFind tableSelect “1abc234” cellPush “Properties” buttonWait for “1abc234” dialogSelect “Gray” color in combo boxPush “OK”

Pro

duct

UI

Dom

ain

mod

el

Car recordColorModel Make Year License plateVIN

Test

Widgets or coordinates

TTDD ~= 3 * T ~= 3 * T

MM, , TT

SS ~= .5 * T ~= .5 * T

MM

UI PrimitivesFind car list frameCarListFrame list = new CarListFrame()

Open properties dialog for car “1abc234”CarDialog propDialog = list.carProperties(“1abc234”);

Set color to graypropDialog.setColor(Color.GRAY);

Apply changespropDialog.ok();

Pro

duct

UI

Dom

ain

mod

el Car record

Test library

ColorModel Make Year License plateVIN

CarListFrame CarDialog

TestTT

DD ~= 5 * T ~= 5 * T

MM, , TT

SS ~= .2 * T ~= .2 * T

MM

Domain modelSet color to gray for a car “1abc234”

new CarRecord(“1abc234”).

setColor(Color.GRAY);

Underneath the cover, CarRecord class does all described earlier

Pro

duct

UI

Dom

ain

mod

el Car record

Domain test library

ColorModel Make Year License plateVIN

UI test libraryCarList CarDialog

CarRecord

TestTT

DD ~= 7.5 * T ~= 7.5 * T

MM, , TT

SS ~= .05 * T ~= .05 * T

MM

Coordinates Widgets UI Library Domain library0

1

2

3

4

5

6

7

8

1.1

3

5

7.5

10.5

0.1 0.05

Td/TmTs/Tm

Td and T

s together

Coordinates Widgets UI Library Domain library0

5

10

15

20

25

30

1.13

5

7.5

24

12

2.41.2

25.1

15

7.48.7

TdTsTd+(Ts*Nc*Nr)

TD and T

S for N

C=3, N

R=8, T

M=1

EA for N

C=3, N

R=8

Coordinates Widgets UI Library Domain library0

0.5

1

1.5

2

2.5

3

3.5

0.96

1.6

3.24

2.76

Ea

Олег Бунин

Testing JavaFX UITips from JavaFX quality team

Alexandre (Shura) IlineJava and JavaFX quality architectOracle

Links

http://jemmy.dev.java.net