Leveraging version control and continuous integration to ... · Leveraging version control and...
Transcript of Leveraging version control and continuous integration to ... · Leveraging version control and...
![Page 1: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/1.jpg)
Leveraging version control and continuous integration toimprove a Linux HPC eco system.
Yorick PoelsKU LeuvenJune 15 2017
![Page 2: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/2.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
CONTENTINTROTOOLSEVOLUTION OF SETUPWORKFLOWCONCLUSION
![Page 3: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/3.jpg)
![Page 4: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/4.jpg)
![Page 5: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/5.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
Tier1: BrENIACI 580 nodesI CentOS 7.xI Puppet
Tier2: ThinKing - CerebroI 352 nodes - shared memory systemI RedHat 6.xI Bash
Service nodes (tier1 & 2)I 25+I Foreman + Puppet
![Page 6: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/6.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
![Page 7: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/7.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
GIT
![Page 8: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/8.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
REDMINE
![Page 9: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/9.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
JENKINS
![Page 10: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/10.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
DOCKER
I CentOS 6I CentOS 7
![Page 11: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/11.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
PUPPET
![Page 12: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/12.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
FOREMAN
![Page 13: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/13.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
MATTERMOST
![Page 14: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/14.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
EVOLUTION OF SETUP
1. Bash scripts2. Puppet code + version control3. Puppet code rpm in Jenkins4. Puppet code rpm in Jenkins, multiple git branches5. Lots of tests (lint,rspec, catalog,...)
![Page 15: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/15.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
![Page 16: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/16.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
![Page 17: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/17.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
![Page 18: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/18.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
GIT HOOKS
![Page 19: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/19.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
JENKINS TESTS
![Page 20: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/20.jpg)
![Page 21: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/21.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
PUPPET-LINT
![Page 22: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/22.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
JENKINS - PUPPET-CATALOG-TEST
![Page 23: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/23.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
JENKINS BUILD
![Page 24: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/24.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
JENKINS - MATTERMOST
![Page 25: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/25.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
DEPLOY
![Page 26: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/26.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
PUPPET-HPC SCRIPT
![Page 27: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/27.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
DEPLOY MULTIPLE
![Page 28: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/28.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
CONCLUSION
DisadvantagesI Slower than file based setupI Logical mistakes still happenI Learning curve
![Page 29: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/29.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
AdvantagesI Lot of errors caught during testingI Versioning (rpms)I Reproducible builds / codeI (More) predictable outcomeI Puppet code as documentation
![Page 30: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/30.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
FutureI Spec tests (serverspec,...)I Production branchI Code reviewI ...
![Page 31: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/31.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
I Step by stepI Fail early & often (before breaking too much ;) )I Small improvementsI Version controlI LintI Suitable solutionI ...
![Page 32: Leveraging version control and continuous integration to ... · Leveraging version control and continuous integration to improve a Linux HPC eco system. Author: Yorick Poels Subject:](https://reader033.fdocuments.in/reader033/viewer/2022060513/5f81a0aebd35f34c460a510f/html5/thumbnails/32.jpg)
ToC INTRO TOOLS EVOLUTION OF SETUP WORKFLOW CONCLUSION
REFSI http://puppet-lint.comI http://serverspec.orgI http://www.kuwata-lab.com/kwalifyI https://about.mattermost.comI https://git-scm.comI https://github.com/42wim/matterstuffI https://github.com/cea-hpc/clustershell:I https://github.com/drwahl/puppet-git-hooksI https://github.com/github/octocatalog-diffI https://github.com/grondo/pdshI https://github.com/invadersmustdie/puppet-catalog-testI https://github.com/voxpupuli/librarian-puppetI https://jenkins.ioI https://puppet.comI https://www.theforeman.org