1

Wix AutomationAutomation Manager

Stas Slutsker

2

Introduction

3

What We HaveTeamCity (CI) as test executor

TeamCity (CI) as test executor

4

It Works, But…

5

It Works, But…• Controlling rapidly growing tests amount,

more & better info about them

• Dealing with flakiness in efficient way, technically• Reduce trust and increase annoyance• Poison. How to avoid them and progress

faster? -> Move fast and break nothing

6

TeamCity is NOT test management tool!

It Works, But…

7

So, What’s Missing?

8

• Analyzing tests per configuration, history & statistics for cross-section demands

• Pursue stability

• Risk analysis, better decision making

So, What’s Missing?

9

We need central management of tests for smarter automation

• Progress faster, reaching our release goals• Better development velocity• Quality of our automation service, as a team

Conclusion

10

Automation Manager

11

Automation Manager Goals

• Gathering information about test runs

• Statistics & history, per cross-section demands

• Disabling & approving new or suspected tests, controlling tests statesSmarter Automation

12

States & Stability

13

States & Stability• Each test has a state

• State is changed according to test behavior

• Stable state must be earned!Stable

Beta (new / flaky)

Disabled

14

Beta Tests

15

Statistics

16

Dashboards according to different needs

Statistics

17

Statistics – Ignored Tests

18

Statistics – Test Runs

19

Decision Making

20

Risk analysis & behavior patterns

• Skipping new or non-stable tests

• Discovering non-valuable tests

• Vulnerable product areas (error prune)

Better Decision Making

21

Project Structure

22

Project Structure

API Core Web DB

Client

REST API

Test Observer@ Before@ Failure@ Success@ Skip

GAE

23

Objects Modeling

Test

RunResult

RunResult

RunResult

24

Objects Modeling

RunConfiguration

- browser- experiments- sanity- environment- platform- configurationNa

me- numOfThreads

Test- date- name- state- appearanc

e

RunResult- testName- buildId- runConfigurati

on- passStatus

1

n

25

Thank you!Questions?