Automated performance testing

using Maven & JMeter

George Barnett, Atlassian Software Systems@georgebarnett

• Create controllable JMeter tests

• Configure Maven to create a repeatable cycle

• Run this “build” in your CI server every 8 hours

• Find performance regressions faster

WHY?

• Catch regressions quickly

• Stop hunting for bugs

• Code with confidence

• Know the performance cost of features

• Fearless external dependency upgrades

• Go home on time

ATLASSIAN

• 15,000+ customers

• 100+ developers

• 8 software streams

• 1 performance engineer

ATLASSIAN

• JIRA - Issue management

• Confluence - Enterprise Wiki

• Bamboo - Continuous Integration

• FishEye, Crucible & Clover - Dev Tools

• Crowd - Single Sign-On.

BEFORE WE BEGIN

• Continuous Integration is critical

• Keep code working with Unit and Functional tests

• Broken or irregular builds means no code to performance test

• Performance testing is NOT a replacement for other testing (but it helps)

PROCESS

• Write code (1-6 months)

• Compile code

• Send it to QA (or Production?!)

• Performance Regression

• PANIC!

No performance testing!

A BETTER WAY

• Write code

• CI compiles code and runs tests

• Send it to QA (or Production?!)

• Performance Regression

• PANIC!

No performance testing!

A BETTER WAY 2.0

• Performance test artifacts as soon as they pass Unit and Functional testing

• Fast feedback for developers

• No Surprises

TOOLS

• JMeter

• Maven & plugins

• Bamboo & plugins

• A Profiler

JMETER

• Apache Project

• Java

• Supports HTTP, JDBC, SOAP, XML-RPC, FTP, SMTP, LDAP, JUnit & more.

MAVEN

• Apache Project

• Java

• Build Tool

• Dependency management

• Documentation

MAVEN CONCEPTS

• Project Object Model (POM)

• Standardised directory structure

• Build Lifecycle

• Reporting Lifecycle

• Plugins

MAVEN POM<project> <groupid>com.atlassian.com.performance.jira</groupid> <artifactid>automated-performance</artifactid>

<properties> <test.dataset>/opt/dataset/dataset1.zip</test.dataset> </properties>

<build> <!--Main Lifecycle plugins configured here--> </build>

<reporting> <!--Reporting Lifecycle here--> </reporting></project>

MAVEN PLUGINS

• Chronos

• Ant-run

• Cargo

• Maven Assembly plugin

• Maven Dependency plugin

BAMBOO

• Continuous Integration

• Can run builds on Amazon’s EC2

• JMeter Aggregator Plugin

A PROFILER

• Java Profiler

• CPU Snapshots

• GC & Memory information

• Thread usage and contention

• Triggers & offline sampling

STEP 1

• Maven Assembly plugin

• Create a .zip from project directories

• `mvn deploy` to repository as a SNAPSHOT

Package the JMeter test

• Package application state into zip files

• Ant-run - application pre-setup

• Cargo - deploy container

• Chronos - configure the JMeter tests

STEP 2Configure the Maven automated build

MAVEN PROFILES<project> .. <properties> <test.dataset>/opt/dataset/dataset1.zip</test.dataset> </properties>

<profile> <id>dataset1</id> <properties> <test.dataset>/opt/dataset/dataset1.zip</test.dataset> ..

<profile> <id>dataset2</id> <properties> <test.dataset>/opt/dataset/dataset2.zip</test.dataset> ..

MAVEN PROFILES• Invoked on command line

• eg: mvn verify -Pmysql, tomcat5, dataset1

• Configuration in profile is added to build

• Use for a wider variation in test environments

• Use profiles for individual “work units”, eg JMeter

ADDING A PROFILER

<global.jvm.args>-Xms2000m -Xloggc:${project.build.directory}/gc.log</global.jvm.args><jvm.args>${global.jvm.args}</jvm.args>

<profile> <id>jprofiler</id> <properties>

<jvm.args>-agentlib:jprofilerti=offline,id=${jprofiler.profileid},config=${jprofiler.configfile} -Xbootclasspath/a:${jprofiler.home}/bin/agent.jar ${global.jvm.args}</jvm.args>

</properties></profile>

$ export LD_LIBRARY_PATH=”$JPHOME/bin/linux-x64/

REPORTING

• Available in maven

• eg: mvn site / mvn chronos:report

• Generates graphs and a numerical report

• Can compare to historical data

• Static :(

CHRONOS

REPORTINGCHRONOS

REPORTINGCHRONOS

REPORTINGCHRONOS

REPORTINGCHRONOS

REPORTINGCHRONOS - HISTORY

REPORTINGCHRONOS - HISTORY

REPORTING

• Open source Bamboo plugin (supported!)

• James Roper (Atlassian) 20% time project

• Can graph JMeter .JTL and CSV files

• Dynamic

BAMBOO JMETER AGGREGATOR

REPORTINGBAMBOO JMETER AGGREGATOR

TIPS

• Allocate time for fixing tests

• Your tests WILL break

• Work out a process for fixing bugs

• Gather as many artifacts from each test run as possible

• React sooner rather than later

Q&Agbarnett @ atlassian . com

@georgebarnett

Code samples online soon!blogs.atlassian.com/developer

Meet Atlassian @ The Summit:May 31 - June 2 in SF

www.atlassian.com/summit