TLB - rocket booster for your build

MotivationTLB

Dev AdrenalineThank you

TLBrocket booster for your build

Pavan KS1 Janmejay Singh2

1mail: [email protected]: http://itspanzi.blogspot.com

2mail: [email protected]: http://codehunk.wordpress.com

Pavan, Janmejay TLB - http://test-load-balancer.github.com

MotivationTLB

TLB what?

A Free and Open Source software: BSD licensedCuts your build time by executing tests parallely on a gridSupports multiple build toolsSupports multiple testing toolsSupports multiple languages

MotivationTLB

Whats in it for me?

What can I expect out of this session?

A small, hopefully interesting, story that describes aproblemHow we solved that problemHow we can help you solve itHow you can help us, help others solve it!

MotivationTLB

Outline

1 MotivationProblem that we solvedDream: Fast Builds

2 TLBIntroducing TLBConcepts in TLBShow me the code honey!Hooking TLB up with your build process

3 Dev AdrenalinePolynomial Time Set-Partitioning

MotivationTLB

Problem that we solvedDream: Fast Builds

Outline

MotivationTLB

The Story

MotivationTLB

This is the story of how we went from...

What is it doing that takes more than 2 hours and 7 minutes?

MotivationTLB

This is the story of how we went from...

What is it doing that takes more than 2 hours and 7 minutes?

MotivationTLB

With just a few lines changed in the build script & executingtests parallely.The longish build ’firefox’ split into 16 partitions to executedparallely on 16 virtual machines.

MotivationTLB

With just a few lines changed in the build script & executingtests parallely.The longish build ’firefox’ split into 16 partitions to executedparallely on 16 virtual machines.

MotivationTLB

Outline

MotivationTLB

Fast build = Rapid developmentDevs spend less time waiting to checkin

Need not be limited to unit/integration testsFunctional/Acceptance tests

Easier for devs to run precommit builds - Results in pullingupstream changes often and running builds frequently

MotivationTLB

Also we know that...

Major part of build time is spent in running testsSpeeding up builds is non trivialMost teams have a dev task for this don’t they??

MotivationTLB

Common solutions...

Split applications into modulesDifficult to model (typically end up in diamonddependencies)Pipelines (unit -> integration -> smoke -> functional) (serialprocess)BUT, if downstream dependencies fail, turn around time tofix is huge

Throw more hardware at it - Slice and diceHand written partitioning using directories/tags etc (unequalpartitions)

Logical! but Suboptimal :-(Nither too efficient nor effective.

MotivationTLB

Common solutions...

MotivationTLB

Common solutions...

MotivationTLB

Common solutions...

MotivationTLB

Common solutions...

MotivationTLB

Common solutions...

MotivationTLB

Common solutions...

MotivationTLB

Common solutions...

MotivationTLB

Optimal solution

Optimal solution can be obtained by minimizing, the followingexpression.

D(A1,A2, ...An) = max

∑x∈Ai

−min

∑x∈Aj

Its ideal when global minima for this function is reached.

MotivationTLB

Introducing TLBConcepts in TLBShow me the code honey!Hooking TLB up with your build process

Outline

MotivationTLB

What if partitioning can be off-loaded?

What does TLB do?Makes n partitionsUnderstands which partition the current test runnerprocess isEach partition runs only one of the n mutually exclusive &collectively exhaustive sets

MotivationTLB

For example:

Some potential balancing strategies could bePartition tests to make every set have equal number oftestsOr to have every set take about the same time to finish

Some of these strategies require a central place to store andretrive test-data(running-time, result etc)

MotivationTLB

For example:

MotivationTLB

For example:

MotivationTLB

For example:

MotivationTLB

SetupTypical Setup

MotivationTLB

SetupAlien Environment Setup

MotivationTLB

CommunicationServer-Client-Runner Talk

Step 1Receive list of to-be-run

tests from thetesting-framework

MotivationTLB

Step 2Fetch historical test data

from TLB server(tests thatfailed in the previous

run/time taken by each test)

MotivationTLB

Step 3Prune the list ofto-be-run tests to gettests to be actuallyexecuted(otherpartitions take care ofpruned items)Re-order the prunedlist,for instance pulltests that failed in theprevious run to executefirst

MotivationTLB

Step 4Let the party begin,execute the prunedtestsContinue capturingtest-result/test-time asthe suites run

MotivationTLB

Step 5Some feedback to the

server; posttest-run-time/test-resultsback, seeding data for

future runs

MotivationTLB

Step 6Terminate gracefully; build

task returns

MotivationTLB

Ok enough of handwaving!

Well, that was all too abstract, lets see what TLB has to offer interms of concrete features.

MotivationTLB

Outline

MotivationTLB

TLB Clientthe workhorse

TLB client has two major sub-units.

Balancer - the pruner guy (chosen by setting an environmentvariable TLB_CRITERIA)Orderer - the shuffler guy (chosen by setting an environmentvariable TLB_ORDERER)

both environment variables require fully qualified java class-names

MotivationTLB

TLB ClientBalancer

Count-based Balancing

20 tests / 4 splits = 5 tests on each

Conceptually straight-forwardInefficient in practiceTLB uses this as a fallback, not recomended as prefferedalgorithm

MotivationTLB

TLB ClientBalancer

Time-based Balancing(inspired by Amdahl‘s law)

N tests / 4 splits ≈ 4 splits that take equal time

Much better, yields fairly close to ideal solutionOne slow machine can not only slow down the current run,but skew balancing on the next one too

MotivationTLB

TLB ClientBalancer

Smoothened Time-based Balancing(Ensures no outliers, builds on top of time based balancing)

N tests / 4 splits ≈ 4 splits which take equal time based on historyover past few/several runs

While exponential smoothing, every test-time entry St ∀t > 0 isrecorded as:

S1 = xo (2)

St = αxt−1 + (1− α)St−1 ∀

{t > 10 < α < 1

Where α is the factor of smoothing, which can be tuned externally for every partition and x is unsmoothed reading.

MotivationTLB

TLB ClientBalancer

Default-chain Balancing

Allows users to define criteria chain, which is a COLON(:)seperated list of algorithmsThe chain may include some custom balancer recepies ofyour ownUsed to ensure build doesn’t fail when no data available todo advanced algorithms like Time-balanceAllows defaulting to simpler algorithms likeCount-balancing

MotivationTLB

TLB ClientBalancer

*XYZ* Balancing(This is not a canned algorithm, its something you can create)

You can create your custom balancing algorithm, and use itwith TLBThe contract is enforced by a java abstract class calledTestSplitterCriteriaIf it can potentially fail in some situations and you want afallback, you can use DefaultingTestSplitterCriteria withyour balancer in chainNote: Algorithm need to be repeatable, since its executedon every partition. Mutual-exclusion &Collective-exhausion are imparative.

MotivationTLB

TLB ClientOrderer

Failed First Orderer(Runs tests that failed last time around, first)

Perfect for fixing builds that have a tendency to break after6 in the eveningYou don’t need to wait for the entire build, just watch theconsole log for a few minutes, as you see the test you fixedpass and scroll by

MotivationTLB

TLB ClientOrderer

*ABC* Orderer(This is not a canned algorithm, its something you can create)

You can create your custom ordering algorithm, and use itwith TLBThe contract is enforced by a java abstract class calledTestOrdererOrdering tests to ensure execution-order/side-effects is aslippery slope and is considered an ANTI PATTERN, so westrongly recomend not abusing Ordering facility

MotivationTLB

TLB ClientCorrectness Checking

Correctness Checking(Optional extension that allows you to check correctness of partitioning algorithm and/or your build configuration.)

Asserts correctness of...

new partitioning/ordering algoritms, ensures mutualexclusion/collective exhaustionbuild configuration, testing framework configurationand TLB configuration in your environment

MotivationTLB

Outline

MotivationTLB

TLB Client needs to embed in build environment and interact with atesting-framework

Balancing JUnit test-suite using Apache-AntBalancing Cucumber test-suite using Rake

MotivationTLB

Outline

MotivationTLB

Leverage parallel execution capabilities oftools like...

MotivationTLB

or for that matter

Hudson, Bamboo, TeamCity, AntHill Pro(if you are rich enough),or even

Capistrano/Shell script fork(if you are apoor dev like us).

MotivationTLB

Supported Setup(s)

When we say Ruby, we mean both 1.9 and 1.8

Junit using Ant or Buildr for JavaRspec 1.x & 2.x using Rake on MRI and JRubyTest::Unit using Rake on MRI and JRubyCucumber using Rake on MRI and JRubyTwist using Ant or Buildr for Java

MotivationTLB

Supported Setup(s)

MotivationTLB

Supported Setup(s)

MotivationTLB

Supported Setup(s)

MotivationTLB

Supported Setup(s)

MotivationTLB

Working on...

Junit using Maven on Java

Waiting on http://jira.codehaus.org/browse/SUREFIRE-726

Check out http://tinyurl.com/ceal2uy (google groups thread andpatchset)

MotivationTLB

We plan to support...

TestNG on Java

JBehave on Java

NUnit on .Net

MS Test on .Net

PyUnit on Python

CPPUnit on C++

5am (fiveam) onCommonLisp... on ...

NAnt on .NetMS Build on .Net... on ...

MotivationTLB

While thats our wish-list

Bad news is, we haven’t started work on most of these yet.Good news is, we have good hackers, like yourself,listening to us here, who can help!

We’d love to support anything else that you can make time tocontribute :-)

MotivationTLB

Polynomial Time Set-Partitioning

Outline

MotivationTLB

Partitioning Approaches

Greedy Algorithm (time-balancing-criteria uses this one)Differencing Algorithm (half done, not upstrem yet)GA spike, terrifingly good (upstream as a spike, written inCommonLisp, needs some more tuning)

MotivationTLB

GA spike for set-partitioning

TLB doesn’t have it yet; coming soon!

Check details @ http://github.com/test-load-balancer/set-part

MotivationTLB

Keep children when..fitness function

The new generation

A′x = x1...xi + yj ...yj+δ′ + xj+δ+1...xn (4)A′y = y1...yj + xi ...xi+δ + yj+δ′+1...ym (5)

Can be considered better than parents

Ax and Ay iff :

SD(Ax ,Ay ) > SD(A′x ,A′y )

∑Xi=1 Uxi

k : k = no. of bags(6)

This is just one way to mesure the evolution of the newgeneration.

MotivationTLB

Random numbers0 < n < 10000; 10 generations

MotivationTLB

Band distribution of numbers0 < 60%X’es < 20%; 60% < 20%X’es < 80%; 80% < 20%X’es; 10000 generations

MotivationTLB

Thank you

We are patch hungry*.Please Contribute.

(its BSD 2 clause)

Thank you.References:

http://test-load-balancer.github.comhttp://goo.gl/39t20 [TLB how to YouTube playlist]

* http://code.google.com/p/tlb/issues/list

TLB - rocket booster for your build

Technology

Transcript of TLB - rocket booster for your build