Post on 05-Jul-2015
Олег Бунин
Testing JavaFX UITips from JavaFX quality team
Alexandre (Shura) IlineJava and JavaFX quality architectOracle
Agenda
UI testing overviewTest automation approachesJemmy libraryTest automation effectivenessTest library design approaches
UI testing … by Wikipedia
«GUI software testing is the process of testing a product that uses a graphical user
interface, to ensure it meets its written specifications.»
UI testing … most often ...
«Checking whether usage of a product UI leads to results expected by the the person
who performs testing»
Start text editorPush «File/Open»Verify file chooser directorySelect some fileVerify editor area contentVerify application titleVerify buttons availabilities....
UI Testing Manual Automated
Initial step Design test specification. Create tests.
Establish regular runs.
Ongoing Click, click, click Analize results
File bugs
Change the specification Change the tests
Qualification High, low for test executors High
Effectiveness Low High
Other Inexpensive to start Continuous quality monitoring , inexpensive to reuse
Fun Bo-o-o-ring Much like programming
Development
Continuous build
Commit
BuildExecuted
automagicallyafter commit
Success
Analysis.Rollback!!!
Code changes
NoYes.
PromoteCode is compilableTest further
Continuous build with testing
CommitBuild Success
TestingIs it working?
Passed
Analysis.
Rollback!!!
Code changes
Testchanges
No
No
Test furtherBuild is good
Code line is healthyGo on ...
Yes
Yes. = Compilation
successful
Automation approach
Record && Replay Coding
Test creation Inexpensive*Usually just repeating manual test in special environmentMust be accompanied by other means
Expensive*Consists of programming
Test execution Does not depend on approach
Test maintanance
Higher (in most cases)*Very much depends on test format
Lower*Depends on principles of building test library
Test analysis Does not depend on approach
(*) Much more information closer to the end of this presentation
Jemmy
Jemmy v2Started as a tool to tests TeamWare UI (1999)Used for NetBeans extensions (2000)Official test tool for NetBeans (2001)Open-source (2001)
Jemmy v3Started in (2008) as a proof of concept experiemntExtended to support JavaFX (2009)Opensource with support of JavaFX 1.2 (2009)Developed in close-source for 1.3 since then
Jemmy usagesV2 (Swing/AWT)Internal: Swing, NetBeans, GlassfishExternal: Used by: Boeing (x2), agfa.com, compuware.com, attix5.com,
exgate.tek.com, globo.com, gmx.de, non.agilent.comEvaluated by: syntelinc.com, covast.com, visus-tt.com, covast.com,
nokia.com, twt.com, ubs.com
V3 (FX)Internal: JavaFX, Authoring Tool, JavaStore
Demo...Finally!
Jemmy v3JavaFX
AWT/Swing
SWT
Scenegraph
LCDUI
JemmyCore
Interfaces
AWT robot FX robot
Hierarchy
Input
Scene, nodecontainers
Scroller, slider, list, text, ...
Genericinterfaces
Covered in this presentation
UI test
Find Do VerifyPass Pass
Pass
Fail Fail Fail
Failure analysis
Find next controlTo perform operation
On in
Verify that expected
State reached
Perform necessary actions
LookupBy IDEasiest but may not be possible«open_file_btn»
By typeMost commonButton, text, combo box, etc
By indexUnavoidable2nd button with text «browse»
By toString()Sucksprojectmanager.Main$Main$Script$1Scene$ObjLit$20@4d16318b
By text, tooltip, associated labelBest, if possibleButton with text «browse»
UI c
ou
ld b
eve
ry c
om
plic
ate
d
UI c
oul
d b
eve
ry c
om
plic
ate
d
UI c
oul
d b
eve
ry c
om
plic
ate
d
Lookup demo
InterfacesInterface Control types Description
Mouse, Keyboard, Drag
Enything Low level input
Parent Containers, list (for its content) Something you could look within
Selectable Toggle button, radio button, combobox, check box, lists, etc
A control which provides limited number of choices
CaretOwner Text box, scroll bar, slider A control which has a number value which changes within some range
Text Text box Editable text container
Interfaces demo
VerificationsUI feedback Non UI feedback
Dialog displayed
Test changed
Image updated
Progress bar changed position
File created
Database updated
Sunset happened :)
Waiting
UI is a multy-thread environmentThings happen in backgroundTest code is in another thread
Not much could be really verifiedEverything should be waited for
Verification demo
Test automation effectiveness
TD + *T
SN
R
TM * N
RN
C*E
A =N
C*
EA – automation effectiveness
NR and N
C are characteristics for a product.
TM is a characteristic of a test suite
TD and T
S depend on test automation approach
Smaller TD and T
S - higher the E
A.
1 2 3 4 5 6 7 80
5
10
15
20
25
0
1
2
3
4
Nc = 1 Ratio
Auto Manual Effect
Time
1 2 3 4 5 6 7 80
5
10
15
20
25
0
1
2
3
4Nc = 3 Ratio
Auto Manual Effect
Time
Assumptions: TM = 1 engineer*week T
S = 0.1 * T
M T
D = 5 * T
M N
R = 8
Td or T
s – what to minimize
TS - if (N
C * N
R) is big
Multi-platform
Compatibility with external products (servers, browsers, ...)
Long-living
TD - if (N
C * N
R) is small
Proof of concept
Preview
Tests fail every now and then ...
… because the tested UI is changed
Tests fail every now and then ...
Ah! And also because errors are made ...
End2End demo
Ts continuends
Time spent on What to do %% of time
Allocating failures Use test harness 1% - 5%
Anderstandint the failure reason
Use test logging, save images, save UI state
10% - 80%
Fix the tests Move common code to the library (*)
90% - 10%
(*) The only way it is different from programming is that there are a lot of tests. More on this later.
Harness
TestNGTestNG
Understanding failureLogging
UI operationsGoing to push ... buttonSelecting ... menu
UI responcesFound list view: …
UI hierarchyScene «…»
Group «…»Button «…»
UI statesImages
DemoJemmy logging functionality
Fix the testsOne UI, many-many tests.
Do as few test code changes as possible
Ideally … one change in test code for one change in UI code.
Decrease maintenance costImplement reusable operations in a library
openFile(String filename)
Decrease implementation dependencieslookup(Button.class, new ByText<Button>(«Low»))
Increasing stability.wrap(0).mouse().click();Thread.sleep(1000);...
DemoRefining test.
Library design approaches
Remember the formula?
TD + *T
SN
R
TM * N
RN
C*E
A =N
C*
EA – automation effectivenessTo be used for every particular product.
NR and N
C are unique for a product.
TM is a characteristic of a test suite.
Smaller TD and T
S - higher the E
A.
Coefficient depend on the way you write your tests
TS mainly consists of time for test modification.
When product changes, tests need to be changed accordingly.
Many tests! Hundreds.
“Less changes of test code per a change in the product UI.”
Ideally ... no more than one.
But ... how? You are the coders – you know:
Move code to test library.
Application domain modelC
ar
rec
ord
Ca
r re
co
rd
Color
Model Make
Year License plate
VIN
...
That's ... car catalog of some sort
Application UIProduct UI
Coordinatesclick(134,32) //selects some record
click(215,122) //hits “Properties”
sleep(5) //sleeps to let dialog be painted
click(64,182) //expands color combo
click(235,182) //selects Gray
click(235,212) //hit OKTT
dd ~= 1.1 * T ~= 1.1 * T
mm, , TT
ss ~= 1 * T ~= 1 * T
mmNever tried, but ...
WidgetsFind “Car records” frameFind tableSelect “1abc234” cellPush “Properties” buttonWait for “1abc234” dialogSelect “Gray” color in combo boxPush “OK”
Pro
duct
UI
Dom
ain
mod
el
Car recordColorModel Make Year License plateVIN
Test
Widgets or coordinates
TTDD ~= 3 * T ~= 3 * T
MM, , TT
SS ~= .5 * T ~= .5 * T
MM
UI PrimitivesFind car list frameCarListFrame list = new CarListFrame()
Open properties dialog for car “1abc234”CarDialog propDialog = list.carProperties(“1abc234”);
Set color to graypropDialog.setColor(Color.GRAY);
Apply changespropDialog.ok();
Pro
duct
UI
Dom
ain
mod
el Car record
Test library
ColorModel Make Year License plateVIN
CarListFrame CarDialog
TestTT
DD ~= 5 * T ~= 5 * T
MM, , TT
SS ~= .2 * T ~= .2 * T
MM
Domain modelSet color to gray for a car “1abc234”
new CarRecord(“1abc234”).
setColor(Color.GRAY);
Underneath the cover, CarRecord class does all described earlier
Pro
duct
UI
Dom
ain
mod
el Car record
Domain test library
ColorModel Make Year License plateVIN
UI test libraryCarList CarDialog
CarRecord
TestTT
DD ~= 7.5 * T ~= 7.5 * T
MM, , TT
SS ~= .05 * T ~= .05 * T
MM
Coordinates Widgets UI Library Domain library0
1
2
3
4
5
6
7
8
1.1
3
5
7.5
10.5
0.1 0.05
Td/TmTs/Tm
Td and T
s together
Coordinates Widgets UI Library Domain library0
5
10
15
20
25
30
1.13
5
7.5
24
12
2.41.2
25.1
15
7.48.7
TdTsTd+(Ts*Nc*Nr)
TD and T
S for N
C=3, N
R=8, T
M=1
EA for N
C=3, N
R=8
Coordinates Widgets UI Library Domain library0
0.5
1
1.5
2
2.5
3
3.5
0.96
1.6
3.24
2.76
Ea
Олег Бунин
Testing JavaFX UITips from JavaFX quality team
Alexandre (Shura) IlineJava and JavaFX quality architectOracle
Links
http://jemmy.dev.java.net