Fully fault tolerant real time data pipeline with docker and mesos
using Docker with Pipeline because it's good.
Transcript of using Docker with Pipeline because it's good.
![Page 1: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/1.jpg)
using Dockerwith Pipelinebecause it's good.
![Page 2: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/2.jpg)
using Docker
![Page 3: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/3.jpg)
Useful Docker Images
![Page 4: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/4.jpg)
Useful Docker Images
● OpenJDK○ docker pull openjdk:7-jdk
○ docker pull openjdk:8-jdk
● Maven○ docker pull maven:3-jdk-7
○ docker pull maven:3-jdk-8
● Golang○ docker pull golang:1.7
● Ruby○ docker pull ruby:2.3
● Python○ docker pull python:2
○ docker pull python:3
![Page 5: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/5.jpg)
with Pipeline
![Page 6: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/6.jpg)
Building a simple Java app
![Page 7: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/7.jpg)
Iteration #0: A basic Jenkinsfile
node {
checkout scm
sh 'mvn clean install'
junit 'target/surefire-reports/**/*.xml'
}
![Page 8: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/8.jpg)
Iteration #0: Requirements
● JDK on the node● Maven in the PATH for the Jenkins agent executing on the node● Other requirements for build/test execution?
![Page 9: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/9.jpg)
Iteration #1: Tool Installers
node {
checkout scm
withEnv(["JAVA_HOME=${tool 'jdk8' }",
"PATH+MAVEN=${tool 'maven3'}/bin:${env.JAVA_HOME}/bin"]) {
sh 'mvn clean install'
}
junit 'target/surefire-reports/**/*.xml'
}
![Page 10: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/10.jpg)
Iteration #1: Requirements
● Tool installers configured by Jenkins administrator for:○ JDK8○ Maven
● Developer creating Jenkinsfile must know "names" of tools configured.● New tools require Jenkins administrator involvement
![Page 11: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/11.jpg)
Iteration #2: Docker
node {
checkout scm
docker.image('maven:3-jdk-8').inside {
sh 'mvn clean install'
}
junit 'target/surefire-reports/**/*.xml'
}
![Page 12: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/12.jpg)
Iteration #2: Requirements
● Node has running Docker daemon
![Page 13: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/13.jpg)
because it's good.
![Page 14: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/14.jpg)
![Page 15: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/15.jpg)
using Docker
![Page 16: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/16.jpg)
Useful Docker Images
![Page 17: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/17.jpg)
Useful Docker Images
● Redis○ docker pull redis:3
● PostgreSQL○ docker pull postgres:9
● MySQL○ docker pull mysql:5.7
● Cassandra○ docker pull cassandra:3
![Page 18: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/18.jpg)
with Pipeline
![Page 19: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/19.jpg)
Testing a simple Java app
![Page 20: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/20.jpg)
Iteration #0: A basic Jenkinsfile
node {
checkout scm
sh 'redis-server & ; PID=$!; mvn test && kill $PID'
junit 'target/surefire-reports/**/*.xml'
}
![Page 21: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/21.jpg)
Iteration #0: Requirements
● JDK on the node● Maven in the PATH for the Jenkins agent executing on the node● Redis installed on the node
○ What happens when two teams need different versions?○ How are upgrades handled?
● A desire to make sysadmins cry with reckless disregard for sane process management
![Page 22: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/22.jpg)
Iteration #1: Dockernode {
checkout scm
docker.image('redis:3').withRun { c ->
docker.image('maven').inside("--link ${c.id}:redis") {
sh 'mvn test'
}
}
junit 'target/surefire-reports/**/*.xml'
}
![Page 23: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/23.jpg)
Iteration #1: Requirements
● Node has running Docker daemon
![Page 24: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/24.jpg)
because it's good.
![Page 25: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/25.jpg)
![Page 26: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/26.jpg)
using Docker
![Page 27: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/27.jpg)
Useful Docker Images
![Page 28: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/28.jpg)
Useful Docker Images
● Your frontend app● Your backend app
![Page 29: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/29.jpg)
with Pipeline
![Page 30: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/30.jpg)
Deploying a simple Java app
![Page 31: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/31.jpg)
Iteration #0: A basic Jenkinsfile
node {
checkout scm
sh 'mvn release && ./trigger-production-deploy.sh'
mail to: '[email protected]',
subject: "I think we deployed ${env.BUILD_ID}?"
}
![Page 32: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/32.jpg)
Iteration #0: Requirements
● Application "stack" (JVM, Ruby, Golang, etc) known ahead of time● Actual deployment orchestration done outside of Jenkins
![Page 33: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/33.jpg)
Iteration #1: Docker
node {
checkout scm
docker.build("initech/app:${env.BUILD_ID}").push()
sh './trigger-production-deploy.sh'
mail to: '[email protected]',
subject: "I think we deployed ${env.BUILD_ID}?"
}
![Page 34: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/34.jpg)
Iteration #2: Dockernode {
def image = docker.build("initech/app:${env.BUILD_ID}")
image.push()
image.inside {
sh 'mvn acceptance-test'
}
image.push('latest')
sh './trigger-production-deploy.sh'
}
![Page 35: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/35.jpg)
because it's good.
![Page 36: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/36.jpg)
![Page 37: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/37.jpg)
thank you
![Page 38: using Docker with Pipeline because it's good.](https://reader034.fdocuments.in/reader034/viewer/2022051522/58a2cf771a28ab724d8b6452/html5/thumbnails/38.jpg)
https://jenkins.io/s/docker