Are You Failing Fast Enough?

Tips & tricks for a speedy build system

Sarah Goff-Dupont Atlassian

1

Greetings! If you’ve come to hear a few tips for making your build system run faster then you’re in the right place!

Before we begin... this talk is designed for people who have a basic continuous integration system in place. If you and your team are already doing full continuous delivery, this talk might be too basic for you. So if you’d like to go find another session to attend, I won’t be offended.

My name is Sarah Goff-Dupont, and I have a confession to make.

2

• I work at Atlassian. And as you may have heard, we make a build server called Bamboo. Not only that, but I am the marketing manager for Bamboo. Which means my credibility with you just took a nose-dive.

• But before that, I did spend about 4 years as a test automation engineer, working mostly in Java. So maybe that gives me a bit of my nerd-cred back.

• You didn't come here to listen to a sales pitch. So for the next 20 minutes we're going to take a look at things from the conceptual level.

• Most of the screenshots you'll see are indeed from Bamboo. And I’ll also show examples from Hudson, Jenkins and TeamCity since those are also very popular tools.

• But the purpose of this talk is to share with you some tips and practices that we at Atlassian have found useful --and that you can implement as well, no matter if you use Bamboo, Hudson, Jenkins, or one of the other CI products out there.

StartIssueB Co

de B

8am 4pm12pm 8am 4pm12pm 8am 4pm12pm 8am 4pm12pm 8am 12pm

StartIssueA Pu

shGet

Build

Resu

lts

Code

APu

shGet

Build

Resu

lts

Code

APu

shGet

Build

Resu

ltsCloseIssueA

Push

Get Bu

ild

Resu

lts

Code

B

StartIssueC

Push

Get Bu

ild

Resu

lts

Code

B

Push

Code

C

StartIssueD

Get Bu

ild

Resu

ltsPu

shCo

de C

Push

Code

B

W.I.P. = 2

W.I.P. = 3 W.I.P. = 2

W.I.P. = 3

Why Do We Care?

Day 1 Day 2 Day 3 Day 4 Day 5

3

• First, let's talk about why you even care how fast your builds run.

• IMO, it's all about getting feedback to the developers --like the results of automated tests or whether there were any compile errors.

• If it takes hours for a developer to find out whether their latest changes broke anything, they're going to make that commit, then move on to something else for a while, then find out there are more changes needed on the previous commit, then spend (waste) some time getting their head back into it... and repeat that cycle several times.

• By the time the original task is complete, a developer may have 2 or 3 additional tasks in motion.

• Multi-tasking is great for computers. But let's face it: we humans do our best, and most efficient work, when we are focused.

• So the goal of an efficient build system is to get feedback to developers fast enough that they don't have time to get wrapped up in a new task --and end up in a confusing mess like this.

Fast Feedback Promotes Efficiency

What’s in it for managers, product owners, and other stakeholders?

Efficiency!

Oh yeah, how?

Limiting the amount of work in progress......allows developers to focus...decreases chances of locked files and time wasted on merges...increases chances of finding bugs while they’re still cheap to fix

4

• Some of you are managers, product owners, etc. You might say "Well, I'm not a developer, so if I get my devs faster feedback, what's in it for me?"

• In a word: efficiency.

• We write better code, and write it faster, when we're not switching back and forth between 3 or 4 different tasks.

• We don’t drive safely while talking on the phone, and we don’t code safely while working on 3 user stories simultaneously. Why? Because we’re distracted. This leads to more bugs being introduced and more features implemented incorrectly --and then we have to waste time cleaning up the mess.

• Another consequence of juggling 3 or 4 tasks is that each task will extend out over many days. And depending on what your source control system looks like, you may have one developer locking up files that another developer needs to work on and creating a bottleneck. Even if files aren't locked down when you're working on them, if another developer needs to work in the same code, we'll have to spend additional time working through merging our changes.

• The third reason fast feedback increases efficiency is that code changes might introduce defects in the application. Those are easiest and cheapest to address when they are discovered right away. The work is still fresh in the developer's mind, and more code hasn't yet been layered on top of it.

• So as much as we were all inspired by the Cirque d’Eclipse performers last night, leave the juggling to them.

Push

Fast Feedback Promotes Efficiency

StartIssueB

8am 4pm12pm 8am 4pm12pm 8am 4pm12pm 8am 4pm12pm 8am 12pm

StartIssueA Get

Build

Resu

ltsCloseIssueA

StartIssueC

W.I.P. = 2

Day 1 Day 2 Day 3 Day 4 Day 5

Push

Get Bu

ild

Resu

ltsPu

shGet

Build

Resu

lts

Push

Get B

uild

Resul

tsPu

shGet

Build

Resu

ltsCloseIssueB

Push

Get Bu

ild

Resu

lts Push

Get Bu

ild

Resu

lts CloseIssueC

5

• If we can get our build systems to give feedback so quickly that devs have only 1 or 2 tasks in progress, we simply waste less time.

• Ultimately, the more we can streamline our development workflows, the more features we can build in an iteration. And the faster we can deliver value to our customers.

• Or to put it another way, when our workflows are IN-efficient, we loose time --and when we loose time, we loose customers.

If you’re going to fail, fail fast!#ece2012

6

• All of this is a really long way of saying that if your code is going to fail for any reason, we want to discover that failure as fast as possible so we can address it right away.

Terminate Hung Builds

7

So now that you care, let's get on with it!

• Probably the easiest and cheapest improvements you can make is to ensure hung builds don't sit there hanging for hours and hours, locking up build agents while your queue gets all clogged up.

Terminate Hung Builds

8

• Most of the popular build tools handle this in basically the same way.

• You set a time limit for how long the build should be allowed to run, and if that threshold is exceeded, the build is stopped.

• With the commercial tools, this is a standard feature.

• WIth Jenkins and Hudson, you just need to install the Build Timeout plugin.

• If you take only one action based on this talk, set your system up to terminate hung builds. It doesn’t cost anything extra, it's easy, it's effective.

Terminate Hung Builds

Repo

CI Server

“Give me free-flowing builds, or give me death!”

9

• So if we visualize a build system, it might look something like this.

Automate Deploys

10

• So that was the easiest thing you can do. • Next, let’s jump right to probably the hardest thing: automate your deploys.

Impact from automating deploys goes beyond fast feedback

Automate Deploys

Encourages standardization across environments and discourages server “drift”

Ops engineers can focus on improvements and innovations

Prioritize low-level environments that receive frequent deploys

11

• Why? Because to get fast feedback you should build with every commit, and test each build so any problems are found right away when they’re easiest to fix.

• Now, unit tests alone won’t find all your bugs because most bugs live in the integration points between different parts of your application. So you need to run some integration, API, and possibly UI-level tests. This usually means deploying the build to an environment.

• But to deploy each build by hand would require at least one person dedicated to it full time. So you automate it.

• Automating deploys IS an up-front investment in staff time, but it pays off.

• See, automated deploys themselves aren't all that hard to accomplish. It's the prerequisites: standardizing your environments is the biggest thing.

• You can't do auto deploys effectively if your servers are like snowflakes --if each one is a little different.

• Or from the “glass-half-full” perspective, automating deploys sort of forces you to eat your vegetables and keep your environments from drifting.

• And the more deploys you automate, the more your Ops engineers have bandwidth to work on more interesting things like scalability projects, production monitoring, increasing uptime, etc.

• Now, in my opinion, you get the most impact by automating deploys to your lowest-level environment, probably your dev or QA environment. This is where you’re going to deploy most frequently.

• So make that those the priority, then replicate it upwards through your environments.

Automate Deploys

Repo

CI Server

12

• So with automated deploys, now our build system is lookin’ something like this.

Pass The Artifact, Please!

13

If and when you have deploys automated, the next step is to pass a pre-packaged build to that deploy step rather than re-building a second (or third) time.

Pass Artifacts

artifact passing for the win!

What is an “artifact”?

JARs, WARs, TARs, ZIPsjavadocs, rdocscobertura.ser, checkstyle.xml, etc.

Artifact passing saves time

Compile only onceCheck out code only once

Store artifacts in Archiva, NexusPro,Artifactory, temp file server, or in the CI tool

14

So, when I refer to an “artifact”, that of course includes the JARs, WARs and TARs that we’re accustomed to with compiled languages.

But an artifact is really anything that you need manipulate or operate on in your builds: a ZIP file containing your source code is an artifact - a Checkstyle or Cobertura report is an artifact... for purposes of this discussion, we can keep the definition very broad.

Now, when we started passing artifacts at Atlassian instead of rebuilding for each environment, we cut about 40 minutes off the time it takes to run a full test cycle.

• Not only do you save time on the compile step, but you skip the additional checkout step too. That can be a huge win if your code base is large and/or your network is prone to slowness.

• You can upload artifacts to external repositories like the ones listed here, if you need to. For most use cases, your CI tool will just handle it for you.

Pass Artifacts

15

• Here’s how it works: In the configs of the job that produces the artifact, you tell the tool where on the server to find it, and the file name.

• Then in the consumer job, you declare the artifact as a dependency, and tell the tool where to place it so the deploy step can find it.

• There are slight differences from tool to tool, but they all work in basically this way. The tool’s documentation is your friend.

Pass Artifacts

What else is in it for me?

Deploy the build you testedNo un-tested code sneaks inNo file corruption

Confidence!

Oh yeah, how?

16

• There are other benefits to artifact passing as well: confidence being the biggest one.

• When you build an artifact once, then pass it up through all your environments, you know with certainty that the build you are about to release into the wild is the exact same build you worked so hard to vet in your pre-production environments.

• No worries about whether it was built from the right revision number, and no opportunities for a network hiccup to corrupt files as they're being checked out from the repo.

• Give your build one last check on Staging, hit the 'deploy to prod' button, and head out for a beer.

Repo

CI Server

Pass Artifacts

17

• Let’s check in on our build system... lookin’ pretty good.

• But I think we can do even more.

• So, what if you don't have automated deploys? Is killing hung builds the only thing you can do to tighten the feedback loop?

Run Tests In Parallel

18

• Thankfully, no.

• Running your automated tests in parallel can make a huge difference. In fact, it’s probably the most effective way to speed up your builds.

Parallelize Tests

Prioritize parallelization of longer-running tests

Break the test suite into batches, and run each as a separate job

You are limited only by the number of build agents you have

19

• First, let me say that this is most effective when applied to integration-, API-, and UI-level tests. Unit tests run really fast anyway, so your biggest return on investment is working with tests that involve calls across different modules of the application or across a network.

• Parallelizing tests is done by dividing your test suite into batches and running each batch as it's own job.

• All the major CI servers are capable of running jobs in parallel, so long as you have build agents available to do so.

Parallelize Tests

20

• I want to share my personal favorite way to split tests into batches: with groups.

• Groups are a feature of several testing frameworks: TestNG (java & Ruby), Proboscis (python), and PHP Unit to name a few

• I’ll use Java & TestNG as my example

• Inside each @Test annotation, you can specify one or more groups that the test or class belongs to.

• The groups are arbitrary, so you can slice n' dice your collection based on functional area of the application, level of the technology stack, or even whether or not the test is currently expected to pass.

Parallelize Tests

21

• So back in your CI tool, you set up jobs based on groups.

• When it's time to run tests, you pass in the group name as a parameter, and only the tests belonging to that group will run in that job. Doesn't matter if the tests in that group are distributed across classes or packages.

• If you're executing tests via Maven, Gradle or Ant, this is pretty trivial.

Parallelize Tests

22

• You can take this idea of parallel test jobs about as far as you want.

• As you can see here, some of our CI builds for JIRA have over 20 jobs running in parallel

• The next thing you’ll want to do is aggregate the results from your testing batches so you have one nice easy number to look at.

• Bamboo, TeamCity, and most of the other commercial CI tools do it for you automatically.

• Jenkins and Hudson do not.... get what you pay for?

Repo

CI Server

Parallelize Tests

23

• So back in our build system...

• Hey, now we’re getting somewhere!!

Create Multiple Feedback Loops

24

• Last on our journey today is the idea of dividing your build & testing cycles into tiers, or loops, that run at strategic intervals.

• Now, what the heck are loops and what do I mean by "strategic intervals"?

Create Multiple Feedback Loops

25

• Let's say that your entire build & test phase takes 3 hours to run through. That's a long time for a developer to wait before getting any results back about their commit.

• And let's say that you have 20 committers , and between them, your repository sees an average of 5 commits per hour.

• Unless you have an army of build agents, your queue is going to clog up FAST, making the wait time even longer.

Repo

CI Server

Create Multiple Feedback Loops

26

• So try this:

• First, create one build/job/pipeline that gets triggered by each commit, and does just 4 things: source code checkout, unit tests, artifact packaging, and storing the artifact somewhere where downstream loops can get at it.

• That loop might take 15-20 minutes to execute. Long enough to document the changes made and get a cup of coffee before you get the results –but not long enough to get going on another user story.

• That becomes your inner loop. It's a quick sanity check for your developers.

Repo

Test Env.CI Server

Create Multiple Feedback Loops

27

• Then, create an outer-loop that deploys your build to a test environment and executes the longer-running integration and UI-level tests.

• Ideally, you'll pass this job the artifact you packaged up in the inner-loop. Otherwise, you can recompile as your first step.

• Run that a few times each day, maybe 3 times during working hours. That keeps the overall load on your build system low and keeps your queue from getting backed up, and it's still frequent enough for there to be a small number of changes between each run. So when new failures pop up, you can still trace them to the offending commit pretty quickly.

Repo

CI Server Test Env.

Create Multiple Feedback Loops

28

• For extra credit, some projects will benefit from a nightly build that runs your entire test suite against multiple platforms.

• Atlassian does this with most of it's products. We deploy builds of JIRA, for example, to a linux box that uses Oracle as the DB, a linux box that uses MySQL, a windows box using Postgress, etc, etc... there are about 8 combinations of OS, DB and browser that our products get tested against.

• Platform-specific bugs are fairly rare, and are almost never show-stoppers. So running this gauntlet once a day is probably enough. And it's strategic to run it at night when your devs aren't making commits and your build agents would otherwise be sitting idle.

• Now, none of these ideas are silver bullets by themselves. Each of you will cherry-pick the combination of concepts that are best suited for your projects, and apply them. And you won't get it perfect the first time. So it's important to measure and track your progress so you know exactly what's working and what not.

Know Where You’re Starting From!

29

• Fortunately, any CI tool can generate reports on build duration.

• Bamboo also has native support for reporting how long builds are waiting in the queue before they're picked up.

• Get to know these reports before you start making any changes so you know where your baseline is.

Know Where You’re Starting From!

30

• There are also plugins that can help you out.

• Jenkins offers the Global Build Stats plugin which lets you refine your reports to look at, say, average build time for only passing builds, or only failed builds.

• Jenkins also has the Wasted Minutes plugin which tallies the total amount of time wasted by all builds in the queue

• Similarly, TeamCity has the Queue Statistics Report, and Bamboo offers a queue report out of the box.

• Bamboo also has the Build Times plugin that graphs the duration of each step and identifies bottlenecks in your pipeline

• These are all useful plugins, and they don’t cost anything extra. So take advantage.

x3

x6

Know Where You’re Starting From!

31

• Because you might feel like it's good enough to have a gut feeling that your build times are improving.

• But when you present all your hard work to management and tell them you deserve a raise, they're gonna want to see some stats.

Q & A

Sarah Goff-Dupont | sgoffdupont@atlassian.com | @DevToolSuperFan

#ece2012

32

• At this point, I'm going to stop so we have time for Q&A.

• I've had a great time. I hope you've enjoyed this talk, and gotten a little bit of inspiration for improvements to try out in your build systems. (CLICK)

• Here is my contact information. If you’d like a copy of the slide deck from today, I’m happy to share it with you. And if you enjoyed this talk, feel free to send a tweet about it with the #ece2012 hash tag --that’ll help me persuade Christian and the selection committee that I should come back next year ;o)

• And with that, let’s open it up for questions.