iOS Parallel Automation:Run faster than fast!

Viktar Karanevich

self.introduce =>

{ name: "Viktar Karanevich", title: ["SSTAE", "SSDET", "SQAAE"], languages: ["Java", "Ruby"], experience: "iOS Automation 7+ years" }

History Context

Jun 2010UI Automation

Apple

History Context

Jun 2010UI Automation

Sep 2014UI Automationxcrun simctl

Apple

"xcrun simctl" is introduced in xCode 6

It is command line utility to control the Simulator (create, delete, erase, install, uninstall, ...).

History Context

Jun 2010UI Automation

Sep 2015UI Automationxcrun simctlUI Testing

Apple

UI Testing in xCode 7 is introduced as replacementfor UIAutomation

Sep 2014UI Automationxcrun simctl

History Context

Jun 2010UI Automation

Sep 2016UI Testing

xcrun simctl

Apple

Sep 2015UI Automationxcrun simctlUI Testing

Sep 2014UI Automationxcrun simctl

UI Automation removed in xCode 8

History Context: Calabash

Jun 2010UI Automation

Sep 2014xcrun simctl

Sep 2016UI Testing

Sep 2016UI Testing

Apple

Calabash

Nov 2009UI Spec

Calabash

Nov 2009UI Spec

Feb 2011Frank

History Context: Calabash

Calabash

Nov 2009UI Spec

Feb 2011Frank

Jan 2012Calabash

Step DefinitionsRuby Client Library

Features

Developer host

Calabash server

Simulator

Your app

UIQuery

Touch Actions

History Context: Calabash

Calabash

Apr 2013Calabash +

UIAutomationLink: https://github.com/calabash/run_loop

Step DefinitionsRuby Client Library

Features

Developer host

Calabash server

Simulator

Your app

UIQueryRun loop:

UIAutomation

History Context: Calabash

Calabash

Sep 2016Calabash +

DeviceAgentLink: https://github.com/calabash/run_loop/tree/develop/lib/run_loop/device_agent

Developer host Simulator

Step DefinitionsRuby Client Library

XCTest Runner

Features

Calabash server

Your app

UIQuery

Device Agent

History Context: Calabash

iOS Automation at Badoo

Approach

~80flaky tests

80data ~50

fail tests

Before Parallel RunTest Suite RunNightly Cucumber ReportNightmare!

~80flaky tests

80data

15htotal time

~50fail tests

1simulator

129data

80data

~500tests

Before Parallel RunTest Suite RunNightly Cucumber ReportNightmare!

~80flaky tests

80data

15htotal time

~50fail tests

1simulator

129data

80data

~500testsKey Issue: How to speed up Test Run?

Let's Parallel!

Let's Parallel!

Blocker Issue: - Calabash does not support Parallel run on 1 desktop

Let's Parallel with multi-desktops!

iOS Parallel Automation: multi-desktops

https://techblog.badoo.com

Tim's talk at Badoo: "Parallel Calabash Testing on iOS"

Test Result: multi-desktops

5 Desktops: test suite ~4h on Mac Mini

Test Result: multi-desktops

5 Desktops: test suite ~4h on Mac Mini

10 Desktops: test suite ~2h on Mac PRO (24-core)

Test Result: multi-desktops

5 Desktops: test suite ~4h on Mac Mini

10 Desktops: test suite ~2h on Mac PRO (24-core)

X Desktops: host limit reached (can not increase number of desktops)

Clustering Hosts

✦ Parallel run on multi (virtual) hosts

✦ Mac Pro and hosts: test suite - ~1h 30 min

Clustering Hosts

Issues:- multi-desktops start up is Flaky- simulators can stuck

Clustering Hosts

Is it possible to automate on multi-simulators?

Facebook: Web Driver Agent

Developer host Simulator

Selenium DriverWDA Runner Your app

FB UIA WDA

Key Change: Reimplemented UIAutomation dependencyLink: https://github.com/facebook/WebDriverAgent

iOS Parallel Automation: FB UIA WDA

Profit:- 7 simulators on single desktop- More stable! more efficient!

Full Migration to FB UIA WDA

Issues:- all tests have calabash code

Full Migration to FB UIA WDA

Issues:- all tests have calabash code- custom app backdoors

Full Migration to FB UIA WDA

Issues:- all tests have calabash code- custom app backdoors- workarounds and other

Full Migration to FB UIA WDA

Full Migration Costs: ~x hours

Is it possible to do Fast Migration?

Calabash + FacebookFast migration: Facebash!

Step DefinitionsRuby Client Library

Features

Developer host

Calabash server

Simulator

Your app

UIQueryRun loop:

UIAutomation

Developer host Simulator

Step DefinitionsRuby Client Library

WDA Runner

Features

Calabash server

Your app

UIQuery

FB UIA WDA

iOS 10 Update:from UIAutomation WDA to XCTest WDA

Link: https://github.com/facebook/WebDriverAgent

Developer host Simulator

Selenium DriverYour app

FB UIA WDA

Developer host Simulator

Your app

FB XCTest WDA

WDA Runner

Selenium Driver

FB simctl

iOS Hybrid 2.0iOS 10 update

Developer host Simulator

Step DefinitionsRuby Client Library

WDA Runner

Features

Calabash server

Your app

UIQuery

FB UIA WDA

Developer host Simulator

Step DefinitionsRuby Client Library

FB simctl

Features

Calabash server

Your app

UIQuery

FB XCTest WDA

Parallel Run: multi-desktops vs multi-simulators

Test Suite: 500 testsNightly Cucumber ReportHappy morning!

80data

Mac PRO

Test Suite Run Time

0 min

225 min

450 min

675 min

900 min

1 sim 10 sims 16 sims

75 min120 min

900 min

MultiDesktops

MultiSimulators

Thank You!

Questions?