Zero to Jenkins: Tips and Tricks for Using Jenkins in...

Zero to Jenkins: Tips and Tricks for Using Jenkins in AWS Kolby Allen

1

Zero to Jenkins: Tips and Tricks for Using Jenkins in AWS Kolby Allen

AWS Architect WatchGuard Technologies

2

A WatchGuard Firebox is

deployed every 4 minutes

around the world

WatchGuard appliances conduct more than

1 BILLION security scans

every hour

WatchGuard protected

our customers 22+

BILLION times in 2016

WatchGuard has saved customers more

than

16 years of labor with

RapidDeploy

Founded in 1996

Headquarters: Seattle, WA

4 operations centers and direct presence in 19 countries

520+ Employees

100+ Distributors 9,000+ Active VARs

Mission: To bring widely deployable, enterprise-grade security to small and midsize organizations and distributed enterprises.

80K+ Customers

3

WatchGuard Cloud

4

ThreatSync

Threat IntelFeeds

Endpoint

Multi-Tier, Multi-Tenant Public Cloud The WatchGuard Cloud

Network

Wi-Fi

VisibilityCollect, Analyze, Respond.

ProtectionDetect, Correlate, Respond.

ManagementConfigure, Deploy, Control.

AaaS

Me

• Started life as a chemist (M.S. in Physical Chemistry) but enjoyed the software and computing side much more

• Working with AWS 5+ Years • 4 years as IT Consultant helping take companies to the Cloud • 1.5 Years with WatchGuard helping build DevOps practice,

automation best practices, and architect core cloud infrastructure.

• Twitter: @kolbyallen • LinkedIn: https://www.linkedin.com/in/kolbyallen/

5

Agenda

• Automated Agent Creation • Shared Groovy Libraries • Hybrid Jenkins files • Jenkins & AWS IAM • Pipelines & Multi-branch Pipelines • QA

6

Agenda


7

Automated Agent Creation

8


• Desire to minimize dependence on plugins • Automated creation of ECS Cluster • Automated updates of Docker slave • Controlled fully by QA team

– Slave deployment subnet – Slave docker configuration – Testing script execution

9


@NonCPS def removeSlave() { for (aSlave in hudson.model.Hudson.instance.slaves) { if (aSlave.name == ‘qa-test-slave‘) { aSlave.getComputer().setTemporarilyOffline(true, null); aSlave.getComputer().doDoDelete(); } } }

@NonCPS def addSlave() { Slave slave = new DumbSlave( "qa-test-slave","qa-test-slave description", "/", "1", Node.Mode.EXCLUSIVE, “qa-test-slave", new JNLPLauncher(), new RetentionStrategy.Always(), new LinkedList())

Jenkins.instance.addNode(slave) }

10


# Base image FROM openjdk:8

... Setup all your tooling ...

# Connecting as slave ENTRYPOINT java -jar slave.jar -jnlpUrl https://<jenkins_url>/<jenkins_slave_name>/slave-agent.jnlp -secret <SECRET>

11

Agenda


12

Shared Groovy Libraries

• Why? – Share common commands (DRY principle) – Allows for complex commands to be maintained in one place – Simplify Jenkinsfile – Easier to onboard — less Groovy/Declarative Pipeline to

learn – Easier to update functionality across all files

– Currently we run 642+ jobs (including multibranch pipelines)

13

Shared Groovy Libraries(root) +- src # Groovy source files | +- org | +- foo | +- Bar.groovy # for org.foo.Bar class +- vars | +- foo.groovy # for global 'foo' variable | +- foo.txt # help for 'foo' variable +- resources # resource files (external libraries only) | +- org | +- foo | +- bar.json # static helper data for org.foo.Bar

├── README.md ├── resources ├── src │ └── org │ └── watchguard │ ├── basefunctions │ │ └── Microservices.groovy │ ├── helpers │ │ └── AWSCLI.groovy │ ├── notifications │ │ └── Hipchat.groovy │ ├── npm │ │ └── Npm.groovy │ ├── unittest │ │ └── Python.groovy │ ├── userfunctions │ │ └── SubmittedUserInfo.groovy │ └── validators │ └── pylint.groovy └── vars

Folder Structure

14


Writing Library Files • Declare information about the library

• Create methods, structures, etc that are needed

// src/org/watchguard/basefunctions/Microservices.groovy package org.watchguard.basefunctions;

/* * pythonUnitTestResults() * Description: Obtains all JUnit outputs * @return: None */ def pythonUnitTestResults(){ step([$class: 'JUnitResultArchiver', testResults: '*.xml']) sh "rm -rf *xml" }

15

Shared Groovy Librariesmicroservicebuild.checkOut(deploygithuburl, deploybranch, githubuser)

/* * checkOut(repo=Null, branch=Null, creds="github_user") * Description: Performs Git Checkout with default credentialsId * @param: repo - Github Repo URL * @param: branch - Git Branch to build * @param: creds - Github Credentials - default github_user * @return: None */ def checkOut(repo, branch, creds=“github_user”){ checkout( [ $class: 'GitSCM', branches: [ [ name: '*/'+branch ] ], doGenerateSubmoduleConfigurations: false, userRemoteConfigs: [ [ credentialsId: creds, url: repo ] ] ] ) }

16


17


// Import Shared libraries @Library('P-Funk') // Import needed classes import org.jenkinsdemo.basefunctions.* import org.jenkinsdemo.notifications.* import org.jenkinsdemo.unittest.* import org.jenkinsdemo.validators.* import org.jenkinsdemo.userfunctions.*

// Initialize all objects def microservicebuild = new Microservices() // org.jenkinsdemo.basefunctions.Microservices def notifications = new Hipchat() // org.jenkinsdemo.notifications.Hipchat def pythontests = new Python() // org.jenkinsdemo.unittest.Python def pylinttests = new pylint() // org.jenkinsdemo.validators.pylint def user = new SubmittedUserInfo() // org.jenkinsdemo.userfunctions.SubmittedUserInfo

18

Agenda


19

Hybrid Jenkinsfiles

• We struggled with our dev not knowing Java — thus finding groovy hard to use

• Known languages – Golang – Python – Ruby – etc

• Mixing Groovy & External files allow for more control

20

Hybrid Jenkinsfiles

Groovy to build software and Python to perform AWS deployments

node{ // Initial cleanup sh "rm -rf *" try { // Check out the frontend code from Github stage('Checkout') { microservicebuild.checkOut(deploygithuburl, deploybranch, githubuser) } // Build Docker Image stage('Build') { sh "docker build -t artifactory.server.com/test-docker-image:${BUILD_ID} -f Dockerfile ." } // Push Docker Image stage('Build') { sh "docker push artifactory.server.com/test-docker-image:${BUILD_ID}" } // Deploy to AWS stage("Deployment"){ sh "python run.py" } } catch (e) { throw e } }

21

Hybrid JenkinsfilesTip: Turn on debug logging inside of external scripts

[Test-Stack] Running shell script+ python run.py2017-08-25 15:51:52,612 : INFO : Loading STS Client2017-08-25 15:51:52,653 : INFO : Starting new HTTP connection (1): 169.254.169.2542017-08-25 15:51:52,663 : INFO : Starting new HTTP connection (1): 169.254.169.2542017-08-25 15:51:52,693 : INFO : Generating STS credentials for account using Role arn:aws:iam::123456789101:role/role-name2017-08-25 15:51:52,696 : INFO : Starting new HTTPS connection (1): sts.amazonaws.com2017-08-25 15:51:53,222 : INFO : Creating client using remote role2017-08-25 15:51:53,237 : INFO : Creating resource using remote role2017-08-25 15:51:53,255 : INFO : Creating client using remote role2017-08-25 15:51:53,264 : INFO : Executing Lambda to get AMI ID2017-08-25 15:51:53,266 : INFO : Starting new HTTPS connection (1): lambda.us-west-2.amazonaws.com2017-08-25 15:51:54,793 : INFO : Starting new HTTPS connection (1): cloudformation.us-west-2.amazonaws.com2017-08-25 15:51:54,873 : INFO : Stack exists. Checking if update is needed.2017-08-25 15:51:54,873 : INFO : Starting stack update2017-08-25 15:51:54,873 : INFO : Updating stack with IAM: Test-Stack2017-08-25 15:51:55,755 : INFO : Stack Updated: Test-Stack2017-08-25 15:51:55,755 : INFO : Creating Waiter: stack_update_complete2017-08-25 15:52:25,903 : INFO : Resetting dropped connection: cloudformation.us-west-2.amazonaws.com2017-08-25 15:52:55,987 : INFO : Resetting dropped connection: cloudformation.us-west-2.amazonaws.com2017-08-25 15:53:26,077 : INFO : Resetting dropped connection: cloudformation.us-west-2.amazonaws.com2017-08-25 15:53:56,196 : INFO : Resetting dropped connection: cloudformation.us-west-2.amazonaws.com2017-08-25 15:54:26,286 : INFO : Resetting dropped connection: cloudformation.us-west-2.amazonaws.com2017-08-25 15:54:56,383 : INFO : Resetting dropped connection: cloudformation.us-west-2.amazonaws.com2017-08-25 15:55:26,484 : INFO : Resetting dropped connection: cloudformation.us-west-2.amazonaws.com2017-08-25 15:55:56,565 : INFO : Resetting dropped connection: cloudformation.us-west-2.amazonaws.com2017-08-25 15:56:26,637 : INFO : Resetting dropped connection: cloudformation.us-west-2.amazonaws.com2017-08-25 15:56:56,783 : INFO : Resetting dropped connection: cloudformation.us-west-2.amazonaws.com2017-08-25 15:57:26,865 : INFO : Resetting dropped connection: cloudformation.us-west-2.amazonaws.com2017-08-25 15:57:56,937 : INFO : Resetting dropped connection: cloudformation.us-west-2.amazonaws.com2017-08-25 15:58:27,033 : INFO : Resetting dropped connection: cloudformation.us-west-2.amazonaws.com2017-08-25 15:58:57,104 : INFO : Resetting dropped connection: cloudformation.us-west-2.amazonaws.com2017-08-25 15:59:27,352 : INFO : Resetting dropped connection: cloudformation.us-west-2.amazonaws.com2017-08-25 15:59:27,416 : INFO : Waiter is done waiting.2017-08-25 15:59:27,416 : INFO : Template updated.

22

Hybrid Jenkinsfiles

• Deployment scripts can be generalized and have Jenkins parameters passed to them.

• Can parse output from scripts inside of Groovy • Allows for Sec/Build team to control Jenkinsfiles, but then

give ability for developers to customize their deployment scripts

• More SDKs!

23

Agenda


24

Jenkins & AWS IAM

• AWS recommends using server roles instead of API keys in order to interact with their API’s. – Jenkins server should have a role that allows you to perform

some actions – In order to make this deliberate we will leverage assumed roles —

Jenkins will have minimum permissions • Why?

– Auto-rotate of API keys – Secure method to request and use keys – Tracking through CloudTrail on API calls

25

Jenkins & AWS IAM Single Account

26

IAM Role that only allow STS

Assume Role with more

rights

Return temporary credentials

Jenkins & AWS IAM

27

Multi Account

Jenkins & AWS IAM

28

• IAM Creds to perform actions – Cloudformation – Terraform – Ansible – AWS API

• Credentials are timed and will expire — there not just sitting around

Jenkins & AWS IAM

29

Jenkins IAM Policy

--- AWSTemplateFormatVersion: "2010-09-09" Description: "Jenkins IAM Role" Resources: JenkinsRole: Type: AWS::IAM::Role Properties: RoleName: JenkinsRole AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: "Allow" Principal: Service: - "ec2.amazonaws.com" Action: - "sts:AssumeRole" Path: "/" Policies: - PolicyName: jenkins_policy PolicyDocument: Version: "2012-10-17" Statement: - Action: - sts:* Resource: "*" Effect: Allow JenkinsProfile: Type: AWS::IAM::InstanceProfile Properties: InstanceProfileName: JenkinsProfile Path: "/" Roles: - !Ref JenkinsRole

Jenkins & AWS IAM

30

Jenkins Assumed Role - Limits access to JenkinsRole

- Limits access from specific IP

- Allows full IAM control only

--- AWSTemplateFormatVersion: "2010-09-09" Description: "Assumed IAM Role to deploy IAM Policies" Resources: JenkinsRole: Type: AWS::IAM::Role Properties: RoleName: JenkinsECSAssumedRole AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: "Allow" Principal: AWS: - arn::aws::iam::123456789012:role/JenkinsRole Action: - "sts:AssumeRole" Condition: IpAddress: aws:SourceIp: "1.1.1.1/32" Path: "/" Policies: - PolicyName: jenkins_policy PolicyDocument: Version: "2012-10-17" Statement: - Action: - ecs:CreateCluster - ecs:CreateService - ecs:RunTask Resource: "*" Effect: Allow

Jenkins & AWS IAMGroovy assume role

31

def stscreds(role){

// Setup Request Command sh "aws sts assume-role --role-arn ${role} --role-session-name \"RoleSession1\" --output text | awk 'END {print \$2 \"\\n\" \$4 \"\\n\" \$5}' > temp.txt"

def key = sh(returnStdout: true, script: 'head -1 temp.txt | tail -1 | tr -d "\n" ') def secret = sh(returnStdout: true, script: 'head -2 temp.txt | tail -1 | tr -d "\n" ') def token = sh(returnStdout: true, script: 'head -3 temp.txt | tail -1 | tr -d "\n" ')

sh(returnStdout: false, script: 'rm -f temp.txt')

return [key, secret, token] }

node { // Assume Role with rights def sts = stscreds("arn::aws::iam::123456789012:role/JenkinsS3AssumedRole")

// Using those creds upload to S3 withEnv(["AWS_ACCESS_KEY_ID=${sts[0]}", "AWS_SECRET_ACCESS_KEY=${sts[1]}", "AWS_SESSION_TOKEN=${sts[2]}"]) { sh "aws s3 sync . s3://${bucketName}/ --acl public-read --delete --region ${region} --storage-class ${storageClass}" } }

Jenkins & AWS IAM

• Allows for full separation of policies for tracking • Fine grained permissions • Different Jenkins masters can assume different roles • Infrastructure master that can build ECS Clusters • Microservice deployment master that can create running

applications • Assumed role’s can be access via deployment scripts

32

Agenda


33

Pipelines & Multibranch Pipelines

• Pipeline provides the ability to run more complex and interrelated build processes

• Compartmentalization of build steps in order to easily identify errors in build

• Separation of slaves for different steps • Parallelization • (Multibranch) - One configuration that controls all builds

within a repository

34


35

stage('Checkout') { microservicebuild.checkOut(url, branch, githubuser) }

stage("Build Docker"){ sh "docker build --no-cache -t watchguard/build:${BUILD_NUMBER} -f Dockerfile ." }

stage("Push to ECR"){ def dockerlogin = sh(script: "aws ecr get-login --no-include-email --region ${region}", returnStdout: true).trim() sh dockerlogin

sh "docker tag watchguard/build:${BUILD_NUMBER} 12345679012.dkr.ecr.us-west-2.amazonaws.com/build:${BUILD_NUMBER}" sh "docker push 12345679012.dkr.ecr.us-west-2.amazonaws.com/build:${BUILD_NUMBER}" sh "docker rmi 12345679012.dkr.ecr.us-west-2.amazonaws.com/build:${BUILD_NUMBER}" }

stage("Deploy"){ sh "python35 run.py" }


36


37


38

This also allows us to generalize - feed in region to scripts from Jenkins


39

• All builds are controlled via the same Pipeline file

Agenda


40

Zero to Jenkins: Tips and Tricks for Using Jenkins in...

Documents

Transcript of Zero to Jenkins: Tips and Tricks for Using Jenkins in...