Speed to Market in

Mobile Development

Finding the right solution with continuous

integration on real devices

Tina Su, Director of Development, Intuit

About me

Nearly 20 years of industry experience in R&D,

quality and engineering management, with a strong

passion for automation.

Engineering Director at Intuit, Developer Productivity

Sr. Engineering Director at Yahoo, Mobile Platforms

Sr. Engineering Manager at Sun, Java Standard and

Mobile Editions

Compiler Developer at MIPS and Tandem

Tina Su, Intuit

Agenda

Mobile trends & developers’ challenges

Speed & quality through continuous integration

- Intuit Virtual Device Lab

- Intuit mobile test automation

Takeaways

Aggressive mobile momentum

“2013 Internet Trends,” KPCB

Will surpass desktop

by 2014

It should just work!

Proliferation of platforms,

OS versions & device types

Developer challenges

Immense configuration matrix

- OS platforms (Android, iOS, Windows, phone vs. tablet, …)

- Inputs (touch/gesture, real buttons, voice, camera, GPS receiver)

- Outputs (portrait/landscape, screen resolution, language)

Application delivery through the markets

- App signing, provision (app & device), submission

Offline experience

- Network, different network bandwidth/latency, no-network

- HTML5 offline app

Media challenges

- Some codecs don’t work on a device

- Streaming, sound

Security & privacy

Intuit mobile app portfolio today

Apple

Google

Amazon

SMS

0 5 10 15 20 490

22

15

11

3

491 mobile apps for financial institutions

Ramping up Intuit’s innovation engine

Pain: Takes days per build to test

combinations of device/OS configurations

physically & manually, one by one …

QA can’t keep up with the changes

Goal: Reduce development iteration cycle

from days to minutes on an ever-growing

number of devices

Strategy: Speed & quality through mobile

continuous integration

• iPhone 4

• iPhone 4S

• iPhone 5

• iPad 2

• iPad 3

• iPad Mini

Devices

OS versions

• iOS 5.0

• iOS 5.1

• iOS 6.0

• iOS 6.1.x

• iOS 7.0

Builds

X X

Daily

builds

• Apple

• Google

• Amazon

Platforms

X

Continuous integration (CI)

What is CI?

Automate build, test & deployment process

Integrate & build early, often, daily, for every check-in

Best practice of Agile development

Jenkins is widely adopted, flexible & extensible

Benefits of CI in a mobile world

Instant feedback on quality, functionality & system impact

Less complex, no integration hell

Detect and address device configuration issues early

Involve stakeholders early

Ultimately achieve shippable release any time

Speed and quality through mobile CI

Key enablers:

• Mobile device lab

• Test automation

Test processor Real

devices

Jenkins

SCM:

source

repository

Developers

/IDE

Automatic

test run

Test

results

Submit the app to

app store if

passed

Check in

Check-in

triggers

automated CI

Install the test app to

real devices in

parallel

Run tests on real

devices in parallel

Analyze

test

results

Trigger testing in

mobile device lab

Process starts here

3rd-party vendor solutions didn’t work for us

Couldn’t handle E2E automation of testing pre-release apps

- Dependency on internal resources & infrastructure

Vendor lock-in: customized CI & test automation solutions

- No flexibility to use our existing tools & frameworks

Disadvantages of vendors’ shared devices

- Security concern on testing on shared devices

- No access to corporate network

- Cost-prohibitive to reserve devices

Private cloud (on-site setup) with local devices not practical

- With sensitive customer data, we need private cloud

- Extensive use of devices for daily testing of hundreds of mobile apps is expensive

Why test on real devices?

Most mobile features can be tested on simulators, but simulators

have limitations:

Device capabilities (GPS, camera, accelerometer, gyros), form

factors & pixel density

Real-world experience over the real connection

(3G, 4G, crappy Wi-Fi, etc.)

True performance, security tests

Testing on non-default Web browsers (e.g., Opera Mini)

Not all simulator/emulators have good quality

(don’t reflect real rendering/behavior, bugs)

Testing SMS & app integration

Intuit Virtual Device Lab

Intuit Virtual Device Lab (VDL) Gives access to real mobile devices using a browser

Intranet

VDL

User machine

User machine

User machine

3-click access

Always on

Integrated with

dev infrastructure

Globally available

iPhone

iPad

Galaxy

Captivate

Optimus Nexus One

Desire

Incredible

Manufacturers/

devices: Droid

Milestone

Moto

VDL architecture – Android

Windows PC

Tomcat

Java image

transferring

application Browser &

Java applet

Internet/intr

anet

Device

interaction

app

Developer’s laptop

VDL

Image transfer

USB connection

VDL architecture – iOS

Mac with Xcode

installed

Tomcat

Java VNC

client

Port

forwarding

Browser &

Java applet

Internet/intr

anet

VNC server

USB connection

Developer’s laptop

VDL

Open source tools & technologies used in VDL

Key capabilities Android iOS

Installing mobile applications into

the real device

adb install <APK file> Fruitstrap

https://github.com/ghughes/fruitstra

p

Transferring device screen images,

controlling device

Screencast tool

http://code.google.com/p/androidsc

reencast

Veency VNC app

Fetching device local files adb pull <filename> sftp command

(jailbroken devices only)

VDL key features & functionality today

Control the device through standard

browser/RESTful APIs

Change orientation to landscape or portrait

Copy local file to the device from your PC

Get file from the device onto your local PC

Execute an ADB command on the device

See current running logs in your browser

Download history logs to your local PC

Continuous integration – how it works

Product

builds

Trigger file launches test

VDL device

inventory Resource

manager

Select a device

Test execution

engine

Results

Monito

r for

new

builds

Mobile continuous integration

Leveraging the Virtual Device Lab for test automation on multiple, real devices simultaneously.

Submit, build, test … in minutes vs. hours

Select devices & tests 1 Tests run automatically across selected

devices with each build

2 See the results! 3

Next step: over-the-air replaces USB connection

Fast onboarding

Scalable

Any device

Internet/intr

anet

App

installation

Test

execution

Results

retrieval

VDL

Mobile Test Automation

Mobile test automation tools

Instrumentation-based

Non-instrumentation

Android iOS

MonkeyRunner

MOET

Sikuli

Android SDK

Robotium

TestDroid (BitBar) Calabash

(LessPainful)

Soasta

iOS SDK

UIAutomation

UISpec

Frank

MonkeyTalk

DA/PerfectoMob

ile

eggPlant Cross-platform solution

iOS Driver Appium

• Assessed as of 2012

Mobile test automation trends

Abstraction with domain-specific language (DSL)

- Common test script over different devices (e.g., Cucumber, MonkeyScript, MOET)

- Support behavior-driven development (BDD)

Client/server-based approach (remotely driven tests via HTTP)

Instrumentation & non-instrumentation solutions complement

each other

Mobile cloud testing environment

- From hardware (cradle) to software approach to drive tests on

real devices

Automated OTA app installation & tests

Intuit automation framework of choice/recommendation

Native apps:

- Cucumber + Calabash for BDD on iOS & Android

Mobile Web:

- Selenium/WebDriver

What is Cucumber?

It lets stakeholders (customers, QE or developers) describe how

software should behave in plain text.

BDD: Written in business domain-specific language (DSL)

- Good abstraction over different device types

One format for multiple purposes

- UI feature specification

- Automated tests

Cucumber itself is NOT a test automation tool

- Add-on to any test automation framework

- Examples: Calabash, Sikuli & Frank mobile test frameworks

What is Calabash?

Open source cross-platform mobile test framework

- iOS + Android, simulator + real device, native + hybrid

Supports Cucumber for BDD

Application needs instrumentation that embeds a Calabash HTTP server

- Tests run from a remote client (in a Jenkins slave)

- Tests can be driven via Wi-Fi (no USB wire)

Ruby clients in API level (Java clients in experiment)

Works great in touch simulation & real device testing

Intuit shared test library – rapid test development

Internal open source repository

Teams focus on

creating

Use shared test lib

… and then contribute to

common test library

Tests

Contribute

Tests

Common test library Product tests

Tests Tests

Globalization test framework

(I18N/L10N)

Performance/load test framework

Security test framework

Native/hybrid test framework

High-level test domain-specific language

(Cucumber steps)

Mobile

back-end

Mobile

front-end

Code

build

Deploy

Test

Results

Connect device

Notify

email

Get

devices

Consolidate

report

Upload

report

From: manual

50*5 = 250

15 30

5

10

20

15

30 10

= 6 hours

Code

build Trigger

Execute tests in parallel

Consolidated

report in archive

Deploy on devices

0 5 10 0

= 15 mins

To: automated CI

TIME SAVED: 95% Release quality increased

*

Stunning results: reduced iteration cycle from hours to minutes

Tested on 5 devices

VDL cost saving over vendors’ private clouds (today)

Annual

cost ($)

# of devices

Intuit Virtual Device Lab

150

3rd-party vendor 1

local device in cradle

Assumptions: 50 native

apps & daily check-in

Note: As vendor pricing

changes, so do the savings

Savings

25

3rd-party vendor 2

local device w/o cradle

(Software approach)

Savings

Private cloud setup behind corp firewall

Key takeaways

Building custom solution vs. 3rd-party vendor solution

Understand your technical/platform/global spread

Many vendor solutions provide a larger set of devices/OS versions

in the global market

Quite a few offer mobile test services – might be tightly coupled with their

device access solution & automation framework

Connecting to your corporate network is a challenge – some provide

private cloud (on-site setup within your network) but it’s cost-prohibitive

Invest in automation and CI

Automate tests with robust automation framework

Enable rapid test development through shared test library

Adopt CI in mobile development for speed & quality

More mobile innovations at Intuit

Mobile crash reporter

Mobile components & design patterns

Mobile analytics SDK & gateway

Visit booth #601 – Accelerate Development

Demos & conversations

Contest

Mobile device charging station

Foosball

We’re hiring ! Win a weekend with a TESLA!

Thank you!!

Tina_Su@intuit.com