Thank you for attending Automated Testing Boston Meetup

Sponsored by

Follow us:

@SiteSpect

Running Regression

Tests in Parallel

using Jenkins

Presented by Josh Shapiro & Dustin Masterson

SiteSpect Inc.

Outline

Background

Description of Parallelized System

Jenkins Performance

Conclusions

Q&A

Part I : Introduction

What is SiteSpect?

SiteSpect is website testing and optimization

It is a reverse proxy

Uses regex to apply changes to page

Control panel for creating tests, analyzing data

SiteSpect Example

vs

What is the Test Harness?

Sets up SiteSpect and a website

Verifies:

Using the control panel

Visitor assignment

Data collection

Diagram of the Test Harness

Harness VM

Headless Browser

Harness

Test Web Site

Control Panel

SiteSpect

Traffic Processing

1. Configure Test

Objects/Features

2. Stream Traffic

a. User

b. SiteSpect

c. Website

d. SiteSpect

e. User

Why change what works?

350 tests

550,000 subtests

Full run takes 7.5 hours

Requires a specific system configuration

Part II : Jenkins and the Harness Lab

What is Jenkins?

Distributes and manages external jobs

Works across multiple machines

1000+ plugins, good community

Runs in JRE, uses Groovy for internal scripting

Jenkins Components/Terms

● Jobso Series of steps, each of which can fail or succeed

o shell scripts, groovy scripts, email notifications,

start other jobs, etc.

● Buildo A single invocation of a given job

● Downstream/Upstreamo Jobs & builds that are called by another jobs or that call

another are downstream or upstream respectively

Summary of Harness Lab

1 master VM, 24+ worker nodes (~75 VMs)

Master hosts:

Jenkins’ Control Panel

MySQL

Custom Test Results Viewers

Worker hosts:

SiteSpect, Harness, Jenkins worker Agent

Diagram of Jenkins

Jenkins Workers

24 Harness Clusters

SVN

Are there new commits?

Configure Machines, Distribute Tests

Test Results to MySQL, Test Result TAP

General Outline of How it All Works

1. Find SiteSpect version and Harness version

2. Configures a single worker

3. Configures the rest of the workersa. Cannot configure multiple boxes in parallel without

some customizations

b. Matrix Project Plugin—It’s a TRAP!

c. NodeLabel Parameter Plugin

General Outline of How it All Works

5. Finds list of all tests in Harness brancha. Automatically creates Jobs for new tests

6. Queues all the tests

7. Adds results to the database

8. Creates a result page

9. Emails the responsible developer

Other Uses for Jenkins

● Building and testing our GeoIP and

Mobile Device databases (WURFL)

● iOS Testing

o OS X worker node

o appium testing with iOS simulator

● VM setup tool

Part III: Performance

At first, performance was a nightmare!

We could not:

Load Control Panel

Display Test Results

Go a single day without a reboot

Performance Issues and Solutions

Culprits

Heap size/Machine Power

A runaway restart worker call

Front-end issues

Using the file system

Views

Build History

Aggregating Downstream Results

Test Results Searching

Harness API Viewer - Hackathon Project

Searching test results DB

See relationships between runs that are not

tracked by Jenkins

Much faster than Jenkins

Avoid the Jenkins UI

For us:

The Jenkins UI is slow

It doesn’t look good

It hurts performance

Still popular, and they are redesigning UI

Part IV : Conclusions

Takeaways

Parallelizing your tests makes everyone happy

Now takes 30 minutes instead of 8 hours

Jenkins is powerful and flexible

It is a very quirky program

Always check the plugins

List of Useful Plugins

Parameterized Trigger

Groovy/Groovy Post-Build

Purge Build Queue

JobConfigHistory

Priority Sorter

Build Timeout

Build User Vars

Conclusions

Thank you!

Questions?

Answers?