Massive Continuous Integration and Light-speed Iterations

54
BT3 Concurrent Session 11/8/2012 10:15 AM "Massive Continuous Integration and Light-speed Iterations" Presented by: Jesse Dowdle AtTask, Inc. Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 8882688770 9042780524 [email protected] www.sqe.com

description

Continuous integration (CI) has become a buzzword, with most engineering organizations claiming they've adopted the practice. However, the sad truth is that unreliable tests, long feedback loops, and poor configuration management block their efforts and minimize CI's potential benefits. Jesse Dowdle shares how AtTask radically redesigned its engineering pipeline and, through massive CI scaling, drove three days of testing to just minutes. Learn the pros and cons of different CI systems and how to integrate them with the cloud. Watch a live demo of AtTask's internal test and CI systems, which they’ve designed to make "Every commit a potential release candidate"-meaning that every commit is an iteration. Arm yourself with the talking points to sell massive CI to executives. Learn how, by leveraging publicly available tools, you can deliver hyper-scalability to your continuous integration framework and drive cycle time, defect counts, and technical debt to new lows.

Transcript of Massive Continuous Integration and Light-speed Iterations

  • 1. BT3 ConcurrentSession 11/8/201210:15AM "Massive Continuous Integration and Light-speed Iterations" Presented by: Jesse Dowdle AtTask, Inc. Broughttoyouby: 340CorporateWay,Suite300,OrangePark,FL32073 [email protected]

2. Jesse Dowdle AtTask, Inc. Senior manager of development for AtTask Inc. Jesse Dowdle directs the engineering efforts of nearly a dozen agile teams in the U.S. and overseas. Successful engineering in a fast-paced industry requires constant research in emerging technologies. Jesse drives thought leadership on continuous delivery, development best practices, and process evolution to help AtTask ship software at lightning speed. Happiest when hes building great software, Jesse sees technology, process, and people as equally important prerequisites for success. A native of Utah, where he enjoys fast cars in the summer and snowboarding in winter, Jesse tweets at @dowdlemj and blogs. 3. [email protected] Continuous Integration and Light-speed IterationsJesseDowdle, Director of Development@dowdlemj linkedin.com/in/jessedowdle www.continuousdeliverist.com 4. SaltLake City, UtahSoftware Projectas a ServiceManagementCollaborativeSaltWork ManagementLake City, UtahSoftware Projectas a ServiceManagementCollaborativeWork Management 5. SaltLake City, UtahSoftware Projectas a ServiceManagementCollaborativeWork ManagementAgile at AtTask 5 years Wall to wall 11 Dev Teams 6. How we did ScrumSprintSprintSprintHow we did Scrum 4 WEEKS SprintSprintSprint 7. How we did Scrum 2 days Demo Retrospective Plan4 WEEKS SprintSprintSprintHow we did Scrum 2 days Demo Retrospective Plan4 WEEKS SprintSprintPublic ReleaseSprintPublic ReleasePublic Release 8. How we did Scrum 2 days Demo Retrospective Plan4 WEEKS SprintInternalInternalSprintInternalPublic ReleaseInternalInternalSprintInternalPublic ReleaseInternalInternalInternalPublic ReleaseInternalPublic ReleaseHow we did Scrum 2 days Demo Retrospective Plan4 WEEKS SprintInternalInternalSprintInternalPublic ReleaseInternalInternalSprintInternalPublic ReleaseInternalInternal 9. Then Scripted scenarios Slow, brittle Monthly Releases 3-5 days acceptance Manual administrationNow Tight unit-like UI tests Fast, Robust Daily releases 30-45 minutes Sign-off and go 10. What is Continuous Integration? Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is veried by an automated build (including test) to detect integration errors as quickly as possible. Many teams nd that this approach leads to signicantly reduced integration problems and allows a team to develop cohesive software more rapidly. -- Martin FowlerComponents Version Build TestControl SystemServerRunnerTestSuiteNoticationMechanism 11. Basic Integration Version Build TestControl triggers new actionServer retrieves latest version, compiles, starts Application StackRunner executes tests against Application StackResults Buildare collected and sent by noticationServer is cleaned and restored.Basic Integration 12. BASIC INTEGRATIONBUILD SERVER VERSION CONTROLTEST RUNNERNOTIFICATION SYSTEMBASIC INTEGRATION 13. BUILD SERVER APPLICATION STACKVERSION CONTROLTEST RUNNERNOTIFICATION SYSTEMBASIC INTEGRATIONBUILD SERVER APPLICATION STACKVERSION CONTROLTEST RUNNERNOTIFICATION SYSTEMBASIC INTEGRATION 14. BUILD SERVER APPLICATION STACKVERSION CONTROLTEST RUNNERNOTIFICATION SYSTEMBASIC INTEGRATIONBUILD SERVER VERSION CONTROLTEST RUNNERNOTIFICATION SYSTEMBASIC INTEGRATION 15. Massive Test Automation 800Unit tests3300API tests2000UI tests55,000hours per monthReleaseAcceptance + CITrue Continuous Integration Howoften do I integrate?Whattests tell me Im integrated?Whatmust I know to release? 16. Pipeline Priorities Performance(Cycle Time)Scalability Visibility AccountabilityATTASKS PIPE BUILD -- DEPLOY -- TEST -- DESTROY 17. CommitATTASKS PIPE BUILD -- DEPLOY -- TEST -- DESTROYCommit Build Installer Build TestsATTASKS PIPE BUILD -- DEPLOY -- TEST -- DESTROY 18. Selenium Grid Grid + AtTask Commit Build Installer Build TestsATTASKS PIPE BUILD -- DEPLOY -- TEST -- DESTROYSelenium Grid Grid + AtTask CommitInstall Build Installer Build TestsATTASKS PIPE BUILD -- DEPLOY -- TEST -- DESTROY 19. Selenium Grid Grid + AtTask CommitInstall Build Installer Build TestsUnitIntegration 1/43/42/44/4ATTASKS PIPE BUILD -- DEPLOY -- TEST -- DESTROYSelenium GridSelenium IEGrid + AtTask CommitInstallSelenium FFBuild Installer Build TestsOther testsUnitIntegration 1/4 2/4ATTASKS PIPE BUILD -- DEPLOY -- TEST -- DESTROY3/4 4/4 20. Selenium GridSelenium IEGrid + AtTask CommitInstallSelenium FFBuild Installer Build TestsOther testsUnitIntegration 1/43/42/44/4ATTASKS PIPE BUILD -- DEPLOY -- TEST -- DESTROYHOW TO MAKE A PAPER AIRPLANE OR, QUEUEING THEORY AS EXPRESSED BY A FOLDED PROJECTILETear Down 21. Selenium GridSelenium IEGrid + AtTask CommitInstallSelenium FFTear DownBuild Installer Build TestsOther testsUnitIntegration 1/43/42/44/4SERIAL VS PARALLELSelenium GridSelenium IEGrid + AtTask CommitInstallSelenium FFBuild Installer Build TestsOther testsUnitIntegration 1/4 2/4SERIAL VS PARALLEL3/4 4/4Tear Down 22. CYCLE TIMECYCLE TIME 23. CYCLE TIMECI Vendors CruiseControl Jenkins (Hudson) TeamCity Go Team Foundation Server 24. Cloud Vendors Amazon Web Services OpenStack SauceLabsJENKINS, EC2, AND ATTASK A MATCH MADE IN HEAVEN 25. JENKINS, EC2, AND ATTASK A MATCH MADE IN HEAVENJENKINS DYNAMIC SLAVE ALLOCATION 26. JENKINS DYNAMIC SLAVE ALLOCATIONSLAVEJENKINS DYNAMIC SLAVE ALLOCATION 27. SLAVE SLAVE SLAVE SLAVEJENKINS DYNAMIC SLAVE ALLOCATIONJENKINS DYNAMIC SLAVE ALLOCATION 28. Scalability Taking Tothe Test Suite to massive scalethe cloud with Amazon Cloud FormationsDynamicSelenium GridsDynamicAtTask EnvironmentsDivide and Conquer Moduleseparation will allow for scaleTestSuites should support shardingRuntests with different dependencies in parallel 29. Divide and Conquer Selenium GridSelenium IEGrid + AtTask CommitSelenium FFInstall Build Installer Build TestsOther testsUnitIntegration 1/43/42/44/4Divide and Conquer UnitIntegration 1/43/42/44/4Tear Down 30. PARALLEL BY DEPENDENCIESIn MemoryDatabaseEmailPARALLEL BY DEPENDENCIESUI OnlyCustomer Data 31. In Memory Database Email UI Only Customer DataPARALLEL BY DEPENDENCIESFrom 3 days to 30 minutes with Selenium 2 32. SELENIUM 1 (1800 TESTS)SOMETIMES THE TESTS NEED TO CHANGESELENIUM 1 (1800 TESTS) 4 HOURSSLOWSOMETIMES THE TESTS NEED TO CHANGE 33. TRANSITIONING TO SELENIUM 2TRANSITIONING TO SELENIUM 2 34. TRANSITIONING TO SELENIUM 2SELENIUM 1 (1800 TESTS) 4 HOURSTRANSITIONING TO SELENIUM 2 35. SELENIUM 1 (1800 TESTS) 4 HOURSSELENIUM 2 (750 TESTS) 30 MINUTESTRANSITIONING TO SELENIUM 2TRANSITIONING TO SELENIUM 2WHERE DID THE TESTS GO? 36. 2.1 MINUTESTRANSITIONING TO SELENIUM 22.1 MINUTESTRANSITIONING TO SELENIUM 234 SECONDS 37. 2.1 MINUTES34 SECONDS WITH GRID...TRANSITIONING TO SELENIUM 22.1 MINUTES34 SECONDS WITH GRID...8 SECONDSTRANSITIONING TO SELENIUM 2 38. 2.1 MINUTES34 SECONDS WITH GRID...8 SECONDS2.4 SECONDSTRANSITIONING TO SELENIUM 2JSON PARTLY CLOUDY WITH A CHANCE OF RAIN 39. JSON PARTLY CLOUDY WITH A CHANCE OF RAINScalability Selenium GridSelenium IEGrid + AtTask CommitInstallSelenium FFBuild Installer Build TestsOther testsUnitIntegration 1/43/42/44/4Tear Down 40. Scalability Selenium GridGrid + AtTaskCommit Build InstallerBuild TestsSLAVEITS RAINING GRIDS 41. SLAVEITS RAINING GRIDSSLAVESELENIUM GRIDSELENIUM GRIDITS RAINING GRIDS 42. SLAVESELENIUM GRIDSELENIUM GRIDITS RAINING GRIDSSLAVESELENIUM GRIDSELENIUM GRIDITS RAINING GRIDS 43. SLAVESELENIUM GRIDSLAVESLAVE SELENIUM GRIDSLAVESLAVEITS RAINING GRIDSVisibility Testresults in JenkinsDescription TheresView Allpluginmy commit!pluginthe jobs on one line, oh the humanity!Screen-shotting TestUI TestsAge, Project Health 44. Visibility Testresults in JenkinsDescription TheresView Allpluginmy commit!pluginthe jobs on one line, oh the humanity!Screen-shotting TestUI TestsAge, Project HealthVisibility Testresults in JenkinsDescription TheresView Allpluginmy commit!pluginthe jobs on one line, oh the humanity!Screen-shotting TestUI TestsAge, Project Health 45. Visibility Testresults in JenkinsDescription TheresView Allpluginmy commit!pluginthe jobs on one line, oh the humanity!Screen-shotting TestUI TestsAge, Project HealthVisibility Testresults in JenkinsDescription TheresView Allpluginmy commit!pluginthe jobs on one line, oh the humanity!Screen-shotting TestUI TestsAge, Project Health 46. Accountability Claim/ Blame test results using AtTaskAccountability Claim/ Blame test results using AtTask 47. Accountability Claim/ Blame test results using AtTaskCI as a Platform On-Demand Easydemo environmentsbutton for version controlVisibility Otherinto production systemsquality measurementsStaticAnalysisCodeCoverage 48. Stafng for Massive CI Getthe right skill setWatchyour integration pointsTechnological OrganizationalDevOpsAdoption Process/ Tool alignmentInformation Integration ScrumRadiators with existing toolsboardBacklogIntegrationwith Release Management 49. CI EVOLUTION ITS A CONTINUOUS INVESTMENTWhat does it cost? 3FTETwohuman, one machine(s)Keeping Efcientcosts down machine sizesRock-solid Elasticitytear down 50. Return on investment 5days (40 hours) to certify one release...10engineers$40per hour$20,00030minutes to certify one release...10engineers$40per hour$250To release 12 times in a year... $240,000 TO CERTIFY$3000 TO TEST $160,000 FOR TWO FTE $77,000 FOR CITo release 24 times in a year... $480,000 TO CERTIFY$6000 TO TEST $160,000 FOR TWO FTE $77,000 FOR CITo release 100 times in a year... $2,000,000 TO CERTIFY$25000 TO TEST $160,000 FOR TWO FTE $77,000 FOR CI 51. 100 deployments annually$2,000,000100 deployments annually$265,000$2,000,000 52. 100 deployments annually$265,000Where to go from here ContinuousDeliveryApplicationLifecycle ManagementContinuousIntegration of InfrastructureOpenSource 53. Tools Manifest AtTask EC2-contributions... wiki.jenkins-ci.org/display/JENKINS/Amazon+EC2+PluginCloudFormation PipelineView -Description GIT--wiki.jenkins-ci.org/display/JENKINS/AWS+Cloudformation+Plugingithub.com/JoelJ/PipelineDashboardSetter -github.com/JoelJ/DescriptionSetter/wiki.jenkins-ci.org/display/JENKINS/Git+PluginAtTaskfor Jenkins -Coming soon...Further Reading ContinuousIntegration -- Martin Fowler - http://www.martinfowler.com/articles/continuousIntegration.html ExtremeProgramming Explained -- Kent Beck - http://amzn.to/c0VvTBContinuousDeployment in 5 Easy Steps -- Eric Ries - http://radar.oreilly.com/2009/03/continuous-deployment-5-eas.html FourPrinciples of Low Risk Software Releases -- Jez Humble - http://www.informit.com/articles/article.aspx?p=1833567 ContinuousDelivery -- Jez Humble & David Farley - http://continuousdelivery.com/ TheLean Startup -- Eric Ries - http://theleanstartup.com/ContinuousIntegration -- Paul Duvall - http://amzn.to/9Zx1l1 54. Thank you! JesseDowdle, Sr Manager of Development@dowdlemj linkedin.com/in/jessedowdle www.continuousdeliverist.comDavidTolley, Sr QA Automation Engineer@davidtolley linkedin.com/prole/view?id=81007074