Tony Wildish

Continuous Integration with Gitlab

-1-

Feb6th2017

Today’s session…

•  h,p://bit.ly/2kAuhFo

-2-

Today’s session…

•  Introduc=ontoGitlab•  GitlabforCon=nuousIntegra=on•  Hands-onsession

–  A‘helloworld’tourofthebasics•  Aside-thatincident,andwhatyoucandoaboutit

–  ThankyouOnur,Chris,Mario,Patrick,Michael,Joel,Alex,Andrew…–  hCps://about.gitlab.com/2017/02/01/gitlab-dot-com-database-incident/

•  Pre-requisites:–  Youwillneedbasicknowledgeofgit,dockerisusefultoo(e.g.seethegit+dockertraining)

–  ThispresentaPon,andotherGenepooltrainingmaterial:hCps://www.nersc.gov/users/computaPonal-systems/genepool/genepool-training-and-tutorials/

-3-

Why should you care? •  Safeguardyourcodeagainstaccidentalloss

–  LikewithanygitplaUorm,distributedreplicas•  Automatecheckingthatyourcodecompiles

–  …andworks.Canbenchmarkittoo•  Automatedeployingyourcode

–  IncludingDockercontainers->usefulforShiZer/cloud•  Reproducibility!

–  Knowhowthatdataorplotwasproduced•  UsefuloneyearfromnowwhentherefereestartsaskingawkwardquesPonsaboutyourdraZpaper

•  Whygitlab,whynotbitbucket,Travis,Jenkins…?–  LotsofacPveplayersintheCIworld,gitlabseemtobeaheadofthepack,haveveryflexibleoffering,easytouse

–  Thatsaid,ifyoupreferanotheropPon,giveitatry!

-4-

Gitlab is…

•  Agit-basedcodehos=ngservice–  Likegithub.com,bitbucket.com,andmanyothers–  SCM,Wiki,issue-tracking,project/team-management…

•  Acon=nuousintegra=on(CI)plaVorm–  LikeTravis,Jenkins,andothers–  Youcommit/tagcode,gitlabbuilds,tests,packagesanddeploysit•  (youtellithow!That’swhattodayisabout)

–  Distributedbuilds,canusemanyplaUorms•  Laptop/desktop,Cori/Edison/Genepool,cloud(AWS,GCP)•  CanevenusemulPpleplaUormsinthesamebuild

-5-

Gitlab components

•  Gitlabserver–  ThehosPngservice–  Projectmanagementcomponents–  CIbuildsystemmanagement(how‘runners’areused)

•  Gitlabrunners–  User-spacedaemonsthatexecutebuilds–  Drivenbytheserveronpushingtotherepository–  Highlyconfigurable,canhavemulPplerunnersperrepowithdifferentcompilers,runPmes,OS…

–  Canrunanywhere:laptop,NERSCmachines,cloud

-6-

Gitlab server

•  Twoedi=ons,threeop=ons–  CE:CommunityEdiPon(free,self-hosted)–  EE:EnterpriseEdiPon(paid,self-hostedorcloud-hosted)–  *Gitlab.com(EE,free)

•  Unlimitedrepositories,privateorpublic•  10GBdiskspaceperproject•  Cannotmirrorexternalprivaterepositories(update:seeappendix)•  Mirroringexternalpublicrepositorieshas0-1hourslatency

–  FullcomparisonathCps://about.gitlab.com/products/•  Whichop=onworksbestforus?–  Notclear,nordoweneedtochooseonlyone–  Comeanddiscussyourneedsatofficehours

-7-

Gitlab runner •  CanrunonanyplaVorm

–  Laptop,Cori/Edison/Genepool/Denovo,AWS/GCP/SPIN–  Configurerunnersperproject

•  Cansharerunnersbetweenprojects,orbeproject-specific•  *Gitlab.comprovidessharedrunners,allreadytouse!

–  SpecifyrunnerscapabiliPeswithtags•  E.g.gcc/python/perlversion,systemcapabiliPes(RAM,cores)

–  Atbuild-Pme•  Serverchoosesrunnersbasedontagsinconfigfile–perstep!•  Serverlaunchesasmanybuildprocessesasrequired•  Canstoreproductsfromeachstepbacktoserver,forinspecPon/use

–  Eachrunnercanrunacustomworkflow•  E.g.‘build’onCori,‘build/test/deploy’onGenepool•  Infinitelyconfigurable,perproject•  Workflowconvenientlyspecifiedinconfigfileintheprojectrepository

-8-

Gitlab and Docker

•  Manypossiblecombina=ons…–  Q:CanIdoXwithDockerandGitlab?A:Yes,forallX!

•  RunGitlabRunnerinaDockercontainer–  AvoidslocalinstallaPon

•  Pull/runDockercontainerstoexecuteyourCIjob–  Getexactlythebuildenvironmentyouwant–  *Usedifferentdockercontainersperstep

•  BuildDockercontainersinsideyourCIjob–  *PushthemtoGitlabContainerRegistryorelsewhere

•  GitlabContainerRegistry–  IntegratedDockerregistry,uploadacontainerfromyourCIjob–  CanautomaPcallytagwithbranchname/versionetc

-9-

The CI configuration file

•  StandardYAML–  YetAnotherMarkupLanguage.Veryhuman-friendly–  .gitlab-ci.yml,inthetopdirectoryofyourgitrepository–  Describespipelineswhichconsistofstages–  EachstagehasaspecificfuncPon:build,test,deploy…–  Eachstagecanhaveitsowntags(requiredenvironment)–  Eachstagecanproducear=facts/re-usefromotherstages–  Stagescanruninparallel–  Check/debugyourYAMLfileathCps://gitlab.com/ci/lint

•  Similartomakefilesinsomeways–  Specifydependencies&acPons,notexplicitlycodingworkflows

-10-

-11-

Defineenvironmentvariablesforuseinthebuild

Executedbeforeeverystage

Definethestagesofthisbuildpipeline

-12-

Compilestep,executesthe‘build’stage

Tellgitlabtokeeptheintermediatebuildproductsforoneweek

Thebuildcommands:eitherinline,orascriptinyourgitrepository

Runstepexecutesthe‘test’stage.Dependsonthe‘compile’stage,getsitsarPfactsautomaPcally

Onlyrunsforgit-taggedversions

-13-

Installsteprunsthe‘deploy’stage.Runsadockercontainertobuilda

dockerimageofourcode

-14-

-15-

-16-

Clonesrepository,downloadsarPfactsfromcompilestep

-17-

Hands-on, exercise 1, part 1 •  GotoGitlab.com,createanaccount•  UploadyourSSHpublickey(notyourprivatekey!)

–  Avatartop-right->pull-downmenu->Sevngs->SSH-keys•  Createanewproject

–  ‘Hamburger’icontop-leZ->Projects->NewProject(top-right)–  Followthestepstosetitupfromscratch

•  EnabletheContainerRegistryforthisproject–  Gearicontop-right->EditProject->scrolldown

•  Gotoh,p://bit.ly/2kAuhFo,download=ny-test.tar–  Untarit,moveallthefilesintoyourproject(including‘.git*’)–  Edit.gitlab-ci.yml,changeREGISTRY_USERandAPPLICATIONtoyourusernameand

yourprojectname,allinlowercase•  Add/commit/pushthiscodetoyourproject

–  gitadd.;gitcommit–m‘blah...’;gitpush•  Gotoyourproject‘Pipelines’page

–  Watchtheprogressofyourbuild!

-18-

Hands-on, exercise 1, part 2

•  Gotoyourproject‘Registry’page–  YoushouldseeaDockerimagelisted,withversion‘latest’

•  Logintothegitlabdockerregistry–  Fromaterminalwindow,type:

•  dockerloginregistry.gitlab.com

–  GiveyourGitlabusername/passwordwhenprompted

•  Runyourdockerimage!–  dockerrunregistry.gitlab.com/$USER/$PROJECT

•  $USERisyourgitlabusername•  $PROJECTisthenameofyourproject•  Youshouldseethe‘HelloWorld’messageonyourterminal!

-19-

Hands-on, exercise 2

•  Nowaddagittag:–  gittagv1.0–  gitpush–tags

•  That’stwo‘-’sthere,dash-dash-tags

•  WatchthePipelinespage–  Youshouldseeathree-stepbuild,withthe‘test’stage

•  ChecktheRepositorypage–  Youshouldseeav1.0dockerimagetheretoo–  Checkyoucanrunitwith:

•  dockerrunregistry.gitlab.com/$USER/$PROJECT:v1.0

-20-

Hands-on – offline, for bonus points… •  Ex.3.Changethepipelinetodothefollowing:

–  Fortaggedcode,dotheteststageaZerthedeploy,notbefore•  Hint:

–  Wheredoyouspecifytheorderofstages?–  Wheredoyouspecifythedependencies?

•  Ex.4.ThenaddanothertesttoruntheDockerimage,nottheliveexecutable–  Hint:

•  Pickauniquenameforthetest,specifyitrunstheteststage•  SeehowtheDockerimageisbuilt,copy/modifytorunitinstead

•  Ex.5.Installagitlab-runnerlocallyonyourmachine–  Makeitproject-specific,notshared–  See‘Crea=ngandRegisteringaRunner’inthedocs(

hCps://docs.gitlab.com/ee/ci/runners/README.html)

-21-

Further steps…

•  Install/runrunnersonCori/Genepool?–  Can’tbuilddockerimagesthere,dockernotsupported– WillhaveaccesstothefullNERSCbuildenvironment–  Gotchaw.r.t.installaPon,cometalktousfirst

•  InstallrunnersonSPIN(NERSCinternalcloud)–  Underdevelopment,watchthisspace…–  ShouldbeabletobuilddockerimagesfrombuildsonCori

•  Installrunnersonyourlaptop/desktop?–  Goodwaytogetexperience/pracPceunPlwehaverunnerssupportedonSPIN

-22-

That incident…

•  OnFeb1st,Gitlabaccidentally‘rm–rf’edinthewrongdirectory–  Theylost6hoursofdata–  5backupmethodsallfailed–  Laughonlyifyou’veneverscrewedupyourselfJ

•  Whatwaslost?–  Issues,mergerequests,anythingdonethroughtheweb–  AnycodecommitsfromrepositorieswhichwerethenremovedfromdiskduringthatPme-window•  IfyousPllhaveyourrepoondisk,‘gitpush’andnothingislost!

-23-

What could you do to be even safer? •  Dual-remotegitrepositories

–  Storeyourcodein2ormoreofgitlab,github,bitbucket...•  How?

–  Createarepository,R1,ononeservice,populateasusual–  Createasecondrepository,R2,somewhereelse,leaveitempty–  CloneR1toyourlocaldisk–  SetR2asasecondremotepushdesPnaPon–  Thenhack,commit,push,pushR2;updatebothremotes!

•  Gotchas?–  R1andR2knownothingabouteachother

•  Ifthey’rebothmodifiedindependently,youcangetintotrouble–  However,fineifR2isonlyusedforspecificpurposes,likeCI–  …andit’saverygoodwaytogetstartedwithgitlab!

-24-

Using dual git-remotes for CI •  Problem:youwanttouseGitlabCI,but…

–  YouhavecodeinaprivaterepositoryinBitbucket–  Gitlab.comcan’teasilymirrorexternalprivaterepositories

•  SeeappendixtothispresentaPon–  Youdon’twanttomoveyourrepositorytoGitlab–(yet!)

•  Solu=on:usedualgitremotes–  CreateanemptyGitlabrepository–  CloneyourBitbucketrepositorysomewhere–  ConfigureyourclonetopushtoGitlab

•  ButtopullonlyfromBitbucket!–  ConPnueworkingexactlyasbefore,evenonsharedprojects

•  CanpullchangescommiCedtoBitbucketbyotherpeople•  Thenpushthem,tosendthemtoGitlab

•  Thisisadvancedgit,amazeyourfriendsJ

-25-

Using dual git-remotes for CI

-26-

Bitbucket Gitlab

Fetch,Push

Using dual git-remotes for CI

-27-

Bitbucket Gitlab

Fetch Push

Using dual git-remotes for CI

-28-

Bitbucket Gitlab

Fetch,Push

Push

Hands-on, exercise 6 •  Gotoh,ps://bitbucket.org/TWildish/gitlab-ci-demo•  Followtheinstruc=onsintheREADME.md

–  Forktherepository,soyouhaveyourowncopyinbitbucket,–  Cloneittoyourlocaldisk–  Createanemptyrepositoryingitlab–  SetthepushdesPnaPonofyourclonetopointtogitlab–  Gitpush,andwatchthecodebuild!

–  Inanotherdirectory,clonethebitbucketrepositoryagain,asnormal–  Modifyitinsomeway(addafile)andcommitthosechanges–  Gobacktoyour‘bitbucket+gitlab’clone–  Pullthechanges,andpushthemtogitlab!

•  Nottheonlywaytodoit–  CanhavemulPplepushdesPnaPonsinthesameclone–  WhichyoudoisamaCerofpersonalchoice,noclearadvantage–  MoreinfoonmulPpleremotes:‘ProGit’,hCps://git-scm.com/book/en/v2,

freeontheweb.Oraskus

-29-

Best practices, gotchas… •  Becarefulwithenvironmentvariables

–  Gitlabsetssomesecretenvironmentvariables(APIkeysetc)–  Ifyouechothemtoyourlogfiles,theywillbevisibleontheweb–  Theonlywaytodeleteoldlogfilesfromgitlab.comistodeletethebuild!

•  CheckyourYAMLconfigura=onfileforerrors–  Use‘CILint’,athCps://gitlab.com/ci/lint,caneditliveandvalidate

•  Setyourar=factstoexpire–  Stuffyouwanttokeepshouldbeproperlydeployed

•  e.g.inaDockerimage

•  Keepyourbuildenvironmentsclean,simple–  Unixconfigure,make,make-test,make-installisade-factostandard–  Tagrunnerstospecifyrequirements,avoidcomplexrunPmescripts

•  E.g.runnerwithtag‘genepool’,usethattaginYAMLconfigfileJ•  Scriptswith“if$NERSC_HOST==‘genepool’”L

-30-

National Energy Research Scientific Computing Center

-31-

Mirroring private bitbucket repositories •  Itispossibletomirrorprivatebitbucketorgithub

repositories,buttherearerisks–  Yougiveyourbitbucketusername&passwordintheURLofthe

repositoryyouwanttomirror–  Thisisvisibletoanyonewiththerightstomanageyourproject–  Anyonewhogetsaccesscanmodifyordeleteyourprivaterepositories

•  Here’stherecipe:–  Createanewaccountonbitbucket,callit‘YourNameRO’–  GrantitReadOnlyaccesstoyourprivatebitbucketrepositories–  Givetheusername&passwordofthataccounttogitlab,insteadof

yourrealaccount–  OnlyeverusetheYourNameROaccountforread-onlyaccess

•  Nevercreaterepositoriesorforks,it’sjustagatewayaccount–  Nowifyourgitlabaccountiscompromisedyouleakfarlessaccess

•  Someonecanreadyourprivatebitbucketcode,butnotchangeit•  ChangeyourYourNameROaccountpasswordandyou’resafeagain!

-32-

Mirroring private bitbucket repositories 2

•  Bitbucket(andotherservices)requireauniqueemailaddressforaccountregistra=on

•  Howdoyouregisterforanewaccountwithoutanaliasforyourlbl.govemailaddress?–  Lbl.govismanagedbyGoogle,it’sGmailunderthehood–  AnyGmailaddresscanhavearbitrary‘extensions’totheusernameasaliasesfortheprimaryaccount•  Justadd‘+’followedbymoretext

–  E.g.,theseareallequivalenttoyourprimaryaddress•  user@lbl.gov•  user+bitbucket_ro@lbl.gov•  user+other_service@lbl.gov

–  Youdon’tneedtoregistertheseemailaliasesanywhere,youcanjustusethem.Goahead,tryit!

-33-