Kristian Karl, Spotify

Experiences Of Test Automation at Spotify

www.eurostarconferences.com

@esconfs#esconfs

Insert speaker picture here, no more than 150x150 pixels

October 16, 2013

Experiences Of Test Automation

@ SpotifyKristian Karl

Scaling Agile @ Spotify with Tribes, Squads, Chapters & Guilds, by Henrik Kniberg & Anders Ivarsson

Why automate?

• We want to be faster

• Shorter release cycles

• Facilitate the testing in

agile dev cycle

From: ”Intelligent Test Automation” by Harry Robinson

What to automate?

Graphical user interface testingUsability testingSoftware performance testingSystem testingFunctional testingLoad testingVolume testingStress testingSecurity testingScalability testingSanity testing

Unit testingSmoke testingComponent testingAPI testingRegression testingInstallation testingMaintenance testingRecovery and failover testing.Accessibility testingMonkey testingIntegration testing

Graphical user interface testing

Our challenges• Hard-to-test SUT

(Experiences of test automation: case study 1, An Agile Team’s Test

Automation Journey: The First Year), Dorothy Graham & Mark

Fewster)

• Maintenance of automation

• Peoples expectations

• Flaky SUTs

• Flaky tests

• Re-prioritizations of non-critical but test hindering bugs

• Test data, test environments

• Supporting services

Our goals• Create automated end-user regression tests on 4 major

platforms1) Desktop – Windows and OSX

2) iOS – iPhone and iPad

3) Android

4) Webplayer

• To facilitate the workload for testers

• To deliver automated regression tests for a feature as a part of

definition of done

• To deliver short feedback loops to teams using Dashboards

Model-based testing

• The models are the abstraction layer

• The testers designs the automation

using models

• The developers implements the code

of the automation

Login view displayed

Main view displayed

Valid credentials

App not running

Start app

Log out

Close

Exit app

Invalid credentialsToggle

’Remember Me’

Start app

public interface SimpleLogin {

public void e_Close();

public void e_Exit();

public void e_Init();

public void e_InvalidCredentials();

public void e_Logout();

public void e_StartClient();

public void e_ToggleRememberMe();

public void e_ValidPremiumCredentials();

public void v_ClientNotRunning();

public void v_LoginPrompted();

public void v_WhatsNew();

}

Dem

o

Developers and developers

• Why not use developers for TA?

• Why use developers for TA?

• Using Test API’s

• Defined by TA

• Implemented by developers

Before

android.view.View seekBarView =

solo.getView(com.spotify.mobile.android.ui.view.CancellableSeekBar.class, 0);

int[] xy = new int[2];

seekBarView.getLocationOnScreen(xy);

solo.clickOnScreen(xy[0] + 9 + (seekBarView.getWidth() - 18) * position, xy[1] +

seekBarView.getHeight() / 2.0f);

After

page().seekTrack(position)

Supporting services

• Continuous testing

• QA Lab

• TDS – Test Data Service

• TRS – Test Result Service

Open source tools that we use

• yEd – editing FSM models [www.yworks.com]

• GraphWalker – generate test sequences from models [graphwalker.org github.com/spotify/python-graphwalker]

• Sikuli – image recognition tool [www.sikuli.org]

• NuRemote - communicating with iOS client [github.com/nevyn/NuRemoting]

• TestNG – group tests into suites and and run them [testng.org]

• Java – code implementation

• Jenkins – [test] job scheduler 24/7 [jenkins-ci.org]

Demo

Q & A