PuppetConf 2016: Turning Pain Into Gain: A Unit Testing Story – Nadeem Ahmad & Jordan Moldow, Box

Jordan Moldow and Nadeem Ahmad PuppetConf 2016

FromPainToGain:APuppetUnitTes4ngStory

Boxisamoderncontentmanagementpla=ormthattransformshoworganiza4onsworkand

collaboratetoachieveresultsfaster.

WhoWeAre

NadeemProduc'vityEngineering

JordanDesktopEngineering

Unittes4ngmonolithicPuppetrepositories

Orperhapsspaghe6repos

hFps://www.flickr.com/photos/oeuildebarzo/3026845290/MaksBarzo

Unittes4ngmonolithicPuppetrepositories

Characteris8csofspaghe6monoliths

•  Inter-connectedmodules

•  Nospecifica4ons•  Organicgrowth•  Businesslogiceverywhere

Thistalkisnotaunittes8ngtutorial

Puppet @ Box

Agenda

Why unit tesBng?

GeFng started

Making it work

ConBnuous integraBon

1

2

3

4

5

Rolling it out 6

Puppet@BoxImportance and scale

Puppet@Box

8,000+NODES

300+MODULES

200+CHANGESPERWEEK

115AVERAGECOMMITTERSPER

MONTH

EnsuringPuppetcorrectnessiscri8cal

Whyunittes4ng?Our current tesBng process and how we wanted to improve

ManualPuppettes8ng@Box

~/puppet $ git commit

~/puppet $ git push origin $BRANCH

$ ssh test.server.com

user@test $ sudo puppet agent –-test --environment=$BRANCH


1.  Waitforcatalogtocompile

2.  Waitforno-opruntocomplete

3.  Reviewandacceptno-opresults

4.  WaitforPuppetruntocomplete

5.  Reviewstateofmachine

Running Puppet on a test machine


3:00:00

Time spent by engineers

Otherissueswithmanualmonolithtes8ng

•  Cannotmanuallytesteveryaffectedrole

•  Testsarenoteasilyrepeatable

ModuleM

ModuleCModuleBModuleA

RoleR3RoleR2RoleR1

Cluster1 Cluster2 Cluster3

ApproachestoPuppettes8ng

1.  Manual

2.  Acceptance(Beaker)

3.  Unit/Integra4on

Projectgoals

1.   TestscanbewriPenformostfiles

2.   Testsrunautoma8cally

3.   Testsrunquickly

4.   Testsfailifsomethingisbroken

5.   Failingtestsarereportedtocodereview

6.   Developerswritetestswiththeircode

GefngstartedFirst experiment with rspec-puppet

Toolsweused

•  puppet(3.6.2)

•  puppetlabs_spec_helper(0.10.2)

•  rspec(3.1.0)–  Rubyspectes4ngframework

•  rspec-puppet(2.3.0)–  Extensiontorspecfortes4ngPuppetmodules

Basictes8ng Rakefile – Command definiBons

Basictes8ng.fixtures.yml –Pointerstoyourmodules

Basictes8ngspec/spec_helper.rb – Configure rspec-puppet

Basictes8ngspec/classes/foo/bar_spec.rb

Runningtests

~/puppet $ rake spec

Runalltestsinadirectory

~/puppet $ rake spec SPEC=spec/classes/foo

Runalltestsinafile

~/puppet $ rake spec SPEC=spec/classes/foo/bar_spec.rb

Runalltests

Runningtests

Basictes8ng

•  Thisworks!•  …onsomesimplemanifests

•  Fortechnicalbenefits,thisneedstoworkonmostmanifests

•  Forculturalchange,itneedstoworkANDbefric4onless

MakingitworkProblems we encountered and our soluBons to them

box_spec_helper hSps://github.com/jmoldow/box_spec_helper

Problem#1:Undefinedfacts

•  rspec-puppetinvokesPuppetwithabareminimumoffactsprovided

•  Factvaluesdefaulttoundefinmanifests

•  Manifestswon’tcompileasexpected

Solu8on#1:Defaultfactsspec/spec_helper.rb

Problem#2:DefaultrequiredaPributes

•  ResourcecollectorssefngdefaultaFributes

•  Inproduc4on,execcommandsdon'tneedabsolutepath

•  Buttestswillfailwithoutit

Solu8on#2:site.ppsymlink

•  Allofourglobaldefaultsareinmanifests/site.pp •  Createdsymlinkfromspec/fixtures/manifests/site.pp

Problem#3:Produc8on-onlyfunc8ons

•  Somecustomfunc4onsmayonlyworkinproduc4on,e.g.‒  Queryingaproduc4ondatastore‒  AccessingfilesthatonlyexistonPuppetmasters

•  Anymanifestthatusessuchafunc4oncannotbetested‒  Norcananymanifeststhatincludessuchamanifest

Solu8on#3:Globally-stubbedfunc8ons

•  Produc4on-onlyfunc4onsshouldalwaysbestubbed•  AnytestcanbewriFenwithouttest-specificstubs•  Allotherfunc4ons,classes,typesshouldactastheywouldinprod

Solu8on#3:Globally-stubbedfunc8ons

•  module_pathcanincludemul4pledirectories

•  Customfunc4onscannotberedefined;firstdefini4onwins

•  Addedextratreetofrontofmodule_path •  Addednewmoduleswithredefinedfunc4ons

Stubbing via module paths

Globally-stubbedfunc8on-Example

Otherbigenhancements

•  Simpletestauto-generator

•  Accumula4ngfact,paramdefini4onsfrommul4pleblocks‒  Includingsharedcontexts

•  Emula4ngotheropera4ngsystems‒ Wewererestrictedonwhathardwarewecouldruntests

Con4nuousIntegra4on

FirstItera8onofCI Define small beta group of Puppet developers

FirstItera8onofCI Run unit tests for their code reviews, report feedback

FirstItera8onofCI

•  Iterateonframework

•  Encouragebetatesterstoalsowritenewtests

FirstItera8onofCI

•  Testsuitetakes6-8minutestorun

•  Goal:orderofseconds

UsingClusterRunnertoRunTestswww.clusterrunner.com

ClusterRunnerPuppetModule

54

UnlockingParalleliza8onFind individual test unit

~/puppet $ find spec -name "*_spec.rb"

spec/classes/os_hardening/init_spec.rb spec/classes/rsyslog/init_spec.rb spec/classes/rsyslog/package_spec.rb spec/classes/dashboard/sync_hive_query_runner_spec.rb spec/classes/pod/delivery_pipeline_spec.rb spec/classes/yum/centos_extras_spec.rb spec/classes/dev_env/repo/devtools_for_windows_spec.rb spec/classes/dev_env/repo/devtools_spec.rb spec/classes/snmp/server_spec.rb spec/classes/bind/staging_spec.rb spec/classes/bind/base_spec.rb …

Runningtests

~/puppet $ rake spec

Runalltestsinadirectory

~/puppet $ rake spec SPEC=spec/classes/foo

Runalltestsinafile

~/puppet $ rake spec SPEC=spec/classes/foo/init_spec.rb

Runalltests

UnlockingParalleliza8onrake spec is actually a combinaBon of three subcommands:

~/puppet $ rake spec_prep

Createsfixturesneededbyalltests


~/puppet $ rake spec_clean

Deletesalltestfixtures


~/puppet $ rake spec_standalone

Runstheactualtest(assumestestfixturesarepresent)

clusterrunner.yaml

Ini8alTime(Serial)FinalTime(Parallel)

420

20TotalTestT

ime(secon

ds)

TestRun8mefor~400PuppetUnitTests

Workflow

$ git push

post-receive

Git hook

GitServer

ClusterRunnerMaster

JenkinsMaster

BuildStateinJenkins

RollingItOut

Roadblocks

•  Testresultsneedtobepostedtocodereviewforalldevelopers•  Post-mergetestresultsneedtobeeasilyavailable

Conclusion

ProjectGoalsChecklist

ü TestscanbewriPenformostfiles

ü Testsrunautoma8cally

ü Testsrunquicklyü Testsfailifsomethingisbroken

q Failingtestsarereportedtocodereviewq Developerswritetestswiththeircode

Possiblefuturework

•  Contributetorspec-puppet,puppetlabs_spec_helper•  Createstandalonerubygem

hSps://github.com/jmoldow/box_spec_helper

Ques8ons?

•  box_spec_helper ‒  hFps://github.com/jmoldow/box_spec_helper

•  Contact‒  Box

‒  [email protected]‒  opensource.box.com‒  www.box.com/careers/

‒  Nadeem‒  [email protected]‒  github.com/nadeemahmad

‒  Jordan‒  [email protected]‒  github.com/jmoldow

UsingtheTemplateandGuidelines

•  Refertothisdocumentforrulesandrecommenda4onsforthetreatmentofgraphicelements

•  Alwaysstartanewpresenta4onusingtheBoxcorporatetemplateandmakeeditsfromthere;thiswillensureyoualwayshavethecorrectstyles,colorpaleFeandfontsinuse

•  Forlayoutinspira4on,pleaserefertothekeyslidespresenta4on

ColorandFontThemesHow to Save Themes…

StartbyopeningtheBoxCorporateTemplateandclickingontheDesigntab.

1

Savethecolorsfromthetemplatetoyoursystemby

clickingColorsandCreateNewThemeColorsandsaveasBox.

2

3

Dothesamethingforthetemplatefonts.CreateNewTheme

FontsandsaveasBox.

BulletSlideTreatment(CalibriRegular24pt)

•  Bulletsaresentencecase(CalibriRegular20pt)‒  Second-linebulletsareCalibri16pt

‒  Third-linebulletsareCalibri14pt

•  Limitthenumberofbulletsonaslide

•  TexthighlightsareCalibriBold,butnotunderlined•  Trynottogobelowtherecommendedfontsizes

BulletSlideWithSub-8tleTreatment

•  Trytokeepyouruseofbulletslidestoaminimum

•  Becrea4veandthinkvisually•  IfyouneedtosourcesomethingcopyandpastethetextboxattheboFomlesontoyourpage

SubBtle Text Placeholder Title Case (Calibri 20pt)

Source:GrayCalibriRegular10pt

CenteredBulletSlideTreatment

•  Bulletsaresentencecase(CalibriRegular20pt)‒  Second-linebulletsareCalibri16pt

‒  Third-linebulletsareCalibri14pt

•  Limitthenumberofbulletsonaslide

•  TexthighlightsareCalibriBold,butnotunderlined•  Trynottogobelowtherecommendedfontsizes

CenteredBulletSlideWithSub-8tleTreatment

•  Trytokeepyouruseofbulletslidestoaminimum

•  Becrea4veandthinkvisually•  IfyouneedtosourcesomethingcopyandpastethetextboxattheboFomlesontoyourpage

SubBtle Text Placeholder Title Case (Calibri 22pt)

Source:GrayCalibriRegular10pt

SegueSlideTitleGoesHereandCanBeaFewLinesinLengthSubBtle goes here in sentence case

Enter Btle for secBon one here and use sentence case

AgendaSlideDesign

Enter Btle for secBon two here and use sentence case

Enter Btle for secBon three here and use sentence case

Enter Btle for secBon four here and use sentence case

Enter Btle for secBon five here and use sentence case

1

2

3

4

5

Enter Btle for secBon one here and use sentence case

AgendaSlideDesignWithHighlightforEachSec8on

Enter Btle for secBon two here and use sentence case

Enter Btle for secBon three here and use sentence case

Enter Btle for secBon four here and use sentence case

Enter Btle for secBon five here and use sentence case

1

2

3

4

5

Enter4tleforsec4ononehereandusesentencecase

AgendaSlideColorCodingOp8onUse different colors to color code each secBon/segue slide

Enter4tleforsec4ontwohereandusesentencecase

Enter4tleforsec4onthreehereandusesentencecase

Enter4tleforsec4onfourhereandusesentencecase

Enter4tleforsec4onfivehereandusesentencecase

1

2

3

4

5

ChartSlideWithMul8pleColorsSub-Btle or Chart Title Here in Title Case

76

60

4457

65

86

72

0

20

40

60

80

100

Category1

Axis Title Goes Here in Title Case Calibri 16pt

Series 1

Series 2

Series 3

Series 4

Series 5

Series 6

Series 7

ChartSlideWithMul8pleDataBarsWithinEachCategorySub-Btle or Chart Title Here in Title Case

76 7569

88

60 6557

63

4450

6272

0

20

40

60

80

100

Category1 Category2 Category3 Category4

Axis Title Goes Here in Title Case Calibri 16pt

Series1

Series2

Series3

“

Placeaquotefromsomeonereally,reallyimportantanditcanbeuptothreelines…

AuthorNameHereAuthorTitleand/orDateHere

HEADERSuppor4ngtextgoeshereunderthenumber



2M 15M 7B

BigNumberTreatment

ProcessDiagramTreatmentSee Style Page for More Color OpBons

96

1 2 3 4Suppor4ngtextgoeshereunderthenumber

Suppor4ngtextgoeshereunderthenumber



ProcessDiagramTreatmentYou Can Use the Blue Bar to House Heavy Content

TitleGoesHere•  Onebullet•  Twobullet•  Threebullet




1 2 3 4

BoxWithBulletTreatment

•  Onebullet•  Twobullet•  Threebullet



TitleHere TitleHere TitleHere

TableLayoutTreatmentSubBtle Text Placeholder Title Case (Calibri 20pt)

Header Header Header Header

TitleOne Informa4on Informa4on Informa4on

TitleTwo Informa4on Informa4on Informa4on

TitleThree Informa4on Informa4on Informa4on

TitleFour Informa4on Informa4on Informa4on

TitleFive Informa4on Informa4on Informa4on

TitleSix Informa4on Informa4on Informa4on

TitleSeven Informa4on Informa4on Informa4on

BigIdeaGoesHereWithHighlightinCalibriBold(40pt)

Usethisareaforabigidea,makesureyourimagehasenoughclearspace+contrastsoyoucanensurereadability

Youcanusethisareaforatexttreatmentthatsupportsyourchosenimagery

ImageTreatmentWithCap8onLayoutHow to Add Your Own Photos and Crop Properly…

InsertoneofthePicturewithCap4onlayouts.Clickontheiconatthecentertoaddyourimage.

1Yourimagewillpopulatethecontainerbutyouwilllikelyneedtoadjustthecrop.

2Clickontheimageandtheformattabandthenselectcrop.

3

ImageTreatmentWithCap8onLayout(Con8nued)How to Add Your Own Photos and Crop Properly…

Clickonthegrayedoutpor4onoftheimageanddragtothelesorrightun4lyouarehappywiththecrop.

4Clickoutofthecroppedimagetosetitandyourimageisreadytogo.

5

ImageWithBulletsTreatmentInsert Your Own Photo, See Slide 39 for InstrucBons

•  Bullettexthere

•  Somemorebullettext

•  Anothershortbullet

•  Lastbulletgoeshere

LogoWithBulletsTreatmentInsert Your Chosen Logo to the Lek of the Line

•  Bullettexthere

•  Somemorebullettext

•  Anothershortbullet

•  Lastbulletgoeshere

ImageTreatmentforTwoImagesSub-Btle or Secondary InformaBon Here

ImageTreatmentforThreeImagesSub-Btle or Secondary InformaBon Here

Mul8pleImageTreatmentSub-Btle or Secondary InformaBon Here

Mul8pleImageTreatmentHow to Insert MulBple Images…

Thislayoutwillappear.Next,clicktheicontoinsertanimage.

Navigatetoyourimageandinsertit.Repeatforalltheimageboxes.Forinstruc4onsonhowtocropyourimageseepage39.

ClicktheNewSlidetabandselectMul4pleImageLayout.

3

2

1

ScreenShotTreatmentWithBrowserWindow

ScreenShotTreatmentWithBrowserWindowHow to Drop in Your Screen Shot…

Thebrowserwindowislikeaframesoanythingyoudropbehinditwillshowthrough.

1

Dropinyourscreenshot,gototheformatmenuandcropittoshowonlywhatyouwant.

2

Last,besuretorightclickonyourscreenshotandsendtobacksoittucksbehindthebrowserwindowframe.

3

ColorPalePe

Hex# 22A7F0 Hex# 2A62B9 Hex# 2D3D4F Hex# BCC1C7

Hex# 3FB87F Hex# 955CA5 Hex# F7931D Hex# E33D55

Hex# 74808C

Primary

Secondary

StylesandTreatments

Clickontheselogos/iconsandyoucanrecolorthemwithinPowerPointasneeded.Besuretofollowbrandguidelinesforusageandcolors.

Varioustemplatecolorscanbeusedforshapes.Shapesshouldhavea4ptlinestroke.Gradientlinecanbeusedtoseparatecontentandshouldhavea1ptlinestroke.

Primary

Secondary

VideoinYourPresenta8onHow to Insert Video and Set Your Playback SeFngs…

Toinsertavideoclickinsertonthemenubar

Nextselectvideoandnavigatetoyourvideo.

Onceyourvideoisontheslideandyouclickonit,youwillgetanewmenutab.Youcannowmakechangestoboththeformatandplaybackofyourvideo.

Usethesesefngsforvideoplayback:StartAutoma4cally,PlayFullScreen,andHideWhileNotPlaying.ThiswillensureaconsistentlookwithinallBoxpresenta4ons.

1

2

3

4

PuppetConf 2016: Turning Pain Into Gain: A Unit Testing Story – Nadeem Ahmad & Jordan Moldow, Box

Technology

Transcript of PuppetConf 2016: Turning Pain Into Gain: A Unit Testing Story – Nadeem Ahmad & Jordan Moldow, Box