TMF2014 Mobile Testing Workshop Michael Palotas
-
Upload
k-j-ross-associates -
Category
Technology
-
view
338 -
download
0
description
Transcript of TMF2014 Mobile Testing Workshop Michael Palotas
TEST MANAGERS FORUM 2014 SYDNEY, AUSTRALIA MOBILE TESTING WORKSHOP
1
2
WHO AM I? Gridfusion Software Solutions Contact: Michael Palotas Gerbiweg 2 8853 Lachen SWITZERLAND Tel.: +41 79 6690708 Email: [email protected]
Head of Productivity & Test Engineering, eBay
Founder / Principal Consultant Gridfusion Software Solutions
SETTING THE STAGE
Tell me about yourself J
What are your expectations for today?
3
MOBILE TESTING SPECIFICALITIES
4
MOBILE IS A VAST LAND
5
YOU ONLY HAVE ONE SHOT
6
MOBILE TESTING CHALLENGES DEVICE DIVERSITY
7
MOBILE TESTING CHALLENGES NEW OS VERSIONS
New versions usually fix issues
but
May also introduce new ones
8
MOBILE TESTING CHALLENGES NEW OS VERSIONS
9
MOBILE TESTING CHALLENGES NEW OS VERSIONS
10
NATIVE APPS AND MOBILE WEB
11
MOBILE TESTING CHALLENGES LOCALIZATION
12
MOBILE TESTING CHALLENGES NETWORK
13
MOBILE TESTING CHALLENGES OPERATORS
14
MOBILE TESTING CHALLENGES MOBILE ENVIRONMENT
15
16
17
18
19
MOBILE TESTING CHALLENGES DEVICE MANAGEMENT
20
DEVICE OR EMULATOR?
21
RISK ASSESSMENT AND STRATEGIES
22
RISK & STRATEGIES
Hotfixes are difficult and expensive
Agile?
...yes...but...for mobile the goal should be to get it right the first time
23
RISK & STRATEGIES
May shift back to waterfall
(waterfall is not that bad)
Difficult to get (mobile) experienced people
Need to grow capabilities in-house
24
RISK & STRATEGIES
No automation in most places today
No continous integration / continous delivery
25
RISK & STRATEGIES DEVICE SELECTION
26
Be pragmatic
Focus on where your money is coming from
Forget the “Long Tail”
RISK & STRATEGIES
Multi code base
Many devices
Device / OS management
Simulator vs. real devices
27
CLOUD TESTING
28
CLOUD VENDORS - DEVICE ANYWHERE
29
CLOUD VENDORS - SAUCELABS
30
TESTDROID
31
GENERAL EXPERIENCES WITH THE CLOUD
No dedicated devices
Or no devices at all
Device management
Your company firewall...
Trust issues
Slow(er)
No in-house knowledge
Not onsite
It is a bit like outsourcing
$ in the beginning
$$$$$$ when you are scaling
32
CROWD TESTING
33
REALLY?
34
CROWD TESTING – WHAT USUALLY WORKS?
Fresh set of eyes
Access to many regions, devices, OS combinations
(at least theoretically)
Lots of issues reported
Relatively low cost
35
CROWD TESTING – CHALLENGES
Significant ramp up required for testers
Low bug quality
You don’t know who is on the other end...
People in mature markets won’t work for 5$ / bug
Only access to production environment
QA system access very difficult
Legal implications, NDAs etc.
No development resources to fix the bugs
36
MOBILE DEVELOPMENT APPROACHES
37
MOBILE DEVELOPMENT APPROACHES
Mobile First
Responsive Design
38
TEAM ORGANIZATION
39
AGILE AND MOBILE
CI – OK
CD – OK for QA versions
CD – NOK for prod versions
40
SKILLSET - THE PERFECT TESTER
41
GENERAL TEST AUTOMATION
42
FLIP THE AUTOMATION TRIANGLE
Unit Tests
E2E Automation Tests
Manual Tests
Unit Tests
E2E Tests
Manual Tests
Integration Tests
TEST AUTOMATION
WHY?
44
TEST AUTOMATION
WHO?
45
TEST AUTOMATION
WHAT?
46
TEST AUTOMATION
WHEN?
47
INSTANT FEEDBACK
D T D T D T D T D T D T D T D T
Development Test
AUTOMATION IS GOOD…BUT
Tools help, but tools don’t test
Manual testing is equally important
Manual (exploratory) and automated testing complement each other
49
Automation
Manual
WHY IS E2E TESTING SO IMPORTANT?
Because it is the most difficult way to test
Because our users use our products End2End
Because you can verify that the whole system works together
50
PERSONAL THOUGHTS ON AUTOMATION
Manual testing is still very important
E2E tests are expensive, so keep them to the minimum Automation is software development
Be careful with record & playback tools
Automate the repetitive tasks (regression)
Build testability into your product
ROI...
More at http://gridfusion.net/thoughtsonautomation.html
51
MOBILE TEST AUTOMATION
52
MOBILE TEST AUTOMATION
Not as mature as web automation
Immature tools market
Tools are usually platform specific
Multi code base
Need to understand the underlying architecture much more than for the web
Need people who can code well
53
EMULATORS VS. DEVICES
What to test where?
54
WHAT TO TEST WHERE - EMULATOR
functional test
55
WHAT TO TEST WHERE - DEVICE
performance look and feel
memory usage carrier specific tests
56
IOS SIMULATOR SPECIFICS
57
Comes from Apple
Works well & reliable
ANDROID EMULATOR SPECIFIC
58
SLOW
ANDROID VS. IOS
Android: lots of variations, versions, vendors
Android: open source (sort of)
iOS: easy to update
iOS: everything comes from one vendor
iOS: not exactly open source J
59
SAMPLE MOBILE AUTOMATION REQUIREMENTS
Use the same approach and infrastructure as web
Business is borderless à test automation needs to be as well
Aim for REAL E2E test automation
Use the Load Balancer / Resource Management Features of Selenium Grid
Manage multiple applications / versions / languages
No modification of mobile app
No modification of mobile device
Mobile App Runtime Inspection Support
Hybrid app support
Support for emulator + real device
Integration into CI system
60
SAME AUTOMATION CODE BETWEEN PLATFORMS?
Sounds good first
but
Most apps are different between platforms
Different element locator strategy
Do reuse the helper functions
61
MOBILE AUTOMATION IOS / ANDROID DIFFERENCES
iOS has no IDs à harder to automate
iOS relies on xcode
Android is open source
Android: ID support
62
TEST INFRASTRUCTURE
63
Driver Browsers Mobiles
AUT
DB
API
TOOLSETS - OVERVIEW
Ranorex
QTP
TOSCA
SeeTest
Selenium/ ios-driver/Selendroid
Selenium / APPIUM
64
SeeTest
65
WHAT ABOUT BDD / ATDD?
I have seen it working
Required to close the loop
Removed complexity
(while adding it at the same time)
Have you ever seen a product owner test?
66
SAMPLE TOOLS
JBehave
Fitnesse
Cucumber
67
SELENIUM
68
SELENIUM – WHAT IS IT?
Web testing tool Highly successful open source project Stable
Protocol describing the user interactions
Supports most browser implementations Support for most common programming languages
69
SELENIUM – WHAT IT IS NOT
a drag & drop tool
a network testing / monitoring tool
a performance testing tool
a reporting tool
a testcase management tool
70
SELENIUM
i.e. Firefox driver
Server
JSON WIRE PROTOCOL
Client
Java
C#
Ruby
Python
Server
i.e. Selendroid, iOS-Driver
SCALING - SELENIUM GRID
72
SELENIUM GRID
73
SCALING USING SELENIUM GRID
74
Client
server GRID Hub Client
CI
server
server
server
server
server
server
75
Implements the JSON Wire Protocol (so it is fully compatible with Selenium)
Runs on emulators and devices
Can be run as a node in Selenium Grid
Support for localized native apps (1 test for all locales)
No need to change the AUT
No need to jailbreak the device
76
Hybrid support Different locator types are supported to find elements Gestures are supported: Advanced User Interactions API Hot plugging It can be integrated as a node into the Selenium Grid for scaling and parallel testing Android target API support from 2.3.3 to 4.3 (latest) Built in Inspector to simplify test case development.
REPORTING
Invest into proper reporting Easy analysis of the reports is crucial
Selenium does no reporting
TestNG or JUnit are responsible for reporting 100% customizable reports (i.e. different reporting levels,
PDF, HTML, Email, SMS etc...) Hooks into test case management tools
77
REPORT WITH SCREENSHOT
78
SELENIUM SELENDROID IOS-DRIVER
79