12 May 2015
Ville-Veikko HelppiTechnical Product Manager
Parallel Test Runs with Appium on Real Mobile Devices
W E B I N A R
Agenda
• Testdroid Update
• Statistics from Q1'2015
• Testing and Mobile Test Automation
• Appium – Client vs. Server Side
Execution
• Demonstration
• Q&A
TESTDROID UPDATE
Public Device Cloud on-demand devices
(multitenant)
Mobile app testing on thousands of real
Android and iOS devices hosted by
Bitbar
Private Device Cloud Reserved Devices
Hosted by Bitbar in the US and/or Europe
Devices chosen by and reserved only for
the Customer
On-Premise Device Cloud
Automated mobile app testing devices
hosted by the customer, usually 30-500 devices
1 Product – 3 Deployment Options
Testdroid Cloud Testdroid Enterprise Testdroid PrivateCloud
New Pricing Plans for Cloud
*) = Prices for annual commitment
Appium
• Client-Side Execution
• Server-Side Execution
Q1'2015 STATISTICS
Q1'2015 – Test Run Stats
Why Apps Fail?
iOS 8.0
iOS 8.0.2
iOS 8.1
iOS 8.1.1
iOS 8.2
iOS 8 ALL
KitKat 4.4
KitKat 4.4.2
KitKat 4.4.3
KitKat 4.4.4
KitKat ALL
Lollipop 5.0
Lollipop 5.0.1
Lollipop 5.0.2
Lollipop ALL
0 5 10 15 20 25 30 35
Failed test runs. Percentage (%).
Some Observations
1
2
Some Observations3
4
Some Observations
5
6
Android Fragmentation 2014
by Open Signal report July 2014
How Many Devices is Enough?
~90%market coverage can
be achieved with
128devices
~20%market coverage can
be achieved with
12 devices
US Market
25 Android devices= ~2/3 market
Global Market
60 Android devices= ~1/2 market
TESTING AND MOBILE TEST AUTOMATION
OS
versions
Chipsets
CPU +
GPU
Tens of
OEMs
Memory
Displays
(resolutions,
physical hw)
OEM
mods
Other
hardware
(connectivity
calibration)
Relation
to other
software
Where Test Automation Can Help
Correct behaviouracross platforms and browsers
Integration with web back-ends
Typically need to fully utilize HW (CPU+GPU)
Resource (e.g. battery) consumption
OpenGL ES 2/3
Functionality and usability
Screen orientations, connectivity, user profiles
Robustness
Robustness and security!
Brand
Compliances, verification with back-
ends and data
Different Mobile 'App Verticals'
Manual Testing is not Agile
Test Automation is Agile
Why Real Devices Are Must-to-Have
• Emulators cannot help you to test...
• User Experience
• Usability
• Hardware
• Software
• Infrastructure
0 % = the percentage of your app users that use emulator to run your app!
Manual vs. Automation
Manual Automation
Manual vs. Automation
Smaller coverage, More money burnt & time wasted, Error-prone
Manual Automation
Large coverage,
quickly completed,
Less money & time wasted, Exact results.
APPIUM
– CLIENT VS. SERVER SIDE EXECUTION
Family Tree of Android Test
Automation Frameworks
JUnit
Android Instrumentation Framework
Robotium
Espresso and Espresso v2
uiautomator
AppiumExtSolo
Calabash
What Framework Works You The Best?
Robotium uiautomator Espresso Appium Calabash
Android Yes Yes Yes Yes Yes
iOS No No No Yes Yes
Mobile web Yes
(Android)
Limited to x.y
clicks
No Yes (Android &
iOS)
Yes
(Android)
Scripting
Language
Java Java Java Almost any Ruby
Test creation
tools
Testdroid
Recorder
UI Automator
viewer
Hierarchy
Viewer
Appium.app CLI
Supported
API levels
All 16 => 8, 10, 15- All All
Community Contributors Google Google Active Pretty quiet
Selenium & Appium
Client Side Appium at
Testdroid CloudTest Script
Test Case
Desired Capabilities
{
“device”: “Android”,
“app”: “/Users/user/ApiDemos.apk”
“app-package”: “com.example.android.apis”
“app-activity”: “.ApiDemos”
}
Test Script
Test Case
Desired Capabilities
WebDriverhttp://localhost:4723/wd/hub
Appium Server
4723
Device
Localhost
(DesiredCaps)
http://localhost:4723/wd/hub
*Testdroid Caps
http://appium.testdroid.com/wd/hub
(DesiredCaps)
Test Script
Test Case
Desired Capabilities
WebDriver
From Localhost to Testdroid
Client Side Execution
Go tocloud.testdroid.com
Client Side Execution
Get a Device Name
Go tocloud.testdroid.com
Client Side Execution
Add Testdroid Desired Caps to test script
{
“testdroid_username”: “[email protected]”,
“testdroid_password”: “p4s$w0rd”,
“testdroid_project”: “My First Project”,
“testdroid_testrun”: “Test 1”,
“testdroid_device”: “iPad Mini 7.0.4 A1432”,
“testdroid_app”: “http://domain.com/app_v1.ipa”
.
.
“app”: “com.bitbar.testdroid.BitbarIOSSample”
}
Get a Device Name
Go tocloud.testdroid.com
Client Side Execution
driver = webdriver.Remote("http://appium.testdroid.com/wd/hub", desired_caps);
Point the Webdriver to http://appium.testdroid.com/wd/hub
Add Testdroid Desired Caps to test script
Get a Device Name
Go tocloud.testdroid.com
Client Side Execution
Run the Test ScriptGet Results from Testdroid Cloud
Point the Webdriver to http://appium.testdroid.com/wd/hub
Add Testdroid Desired Caps to test script
Get a Device Name
Go tocloud.testdroid.com
Client Side Execution
Pull the Results from the Result URL
driver = webdriver.Remote("http://appium.testdroid.com/wd/hub", desired_caps);
Run the Test ScriptGet Results from Testdroid Cloud
Point the Webdriver to http://appium.testdroid.com/wd/hub
Add Testdroid Desired Caps to test script
Get a Device Name
Go tocloud.testdroid.com
Multiple Devices – Client Side
python testscript.py –device <devicename>1
python testscript.py –device <devicename>2
python testscript.py –device <devicename>n
Test Script
Test Cases
Instigator Script
deviceArray=[
“iPad 4 6.0.1 A1458”,“iPad mini 7.0.4 A1432”,...“iPhone 4S 6.1.3 A1387”,
]
21
3
4
5
6
7
WebDriver Session Request@http://appium.testdroid.com/wd/hub
WebDriver Session Response
Testrun
Configure Project
Appium Ready
Wait for Device toBecome Available
Device 1 Device 2 Device 3
Session
Map
Proxy
Device Cluster
Start Appium
Desired Capabilities, .apk / .ipa
SessionID
SessionID
Sessionid
Test
Script
Appium
Broker
Client Side – Behind the Scenes
Introducing "Server Side Appium"
DemonstrationAppium with Real Devices using Image
Recognition for Hill Climb Racing
Setup
• Using real Android devices at Testdroid Cloud
• Parallel test runs without a need to configure desired
capabilities
• Device groups (= set of devices used for runs) can be
manually created and configured
File Structure• pom.xml (maven)
• testdroid.properties (overwritten after submitted to Cloud)
• run-test.sh (shell script for execution)
• image files
15 seconds
Image Recognition
• Resolution agnostic implementation
• Identifies stretched and rotated images as well
Let's See How it Works!
THANK YOU!More information about frameworks, devices
and testing at www.testdroid.com
Top Related