JUC Europe 2015: Hey! What Did We Just Release?
-
Upload
cloudbees -
Category
Technology
-
view
67 -
download
0
Transcript of JUC Europe 2015: Hey! What Did We Just Release?
Jenkins World Tour 2015London, UK, June 2015
1
Footer
#jenkinsconf
Hey! What Did We Just Release?
Robert McNulty
Experian Marketing Services
http://www.experian.com/marketing-services
2
Footer
#jenkinsconf
Agenda
● Introduction● Things As They Are● Things As They Should Be● Demonstration● Q & A
3
Footer
#jenkinsconf
About Me
• 20+ years in software development and database design
• Lead Technical Developer and Architect at Experian Marketing Services
• Lead and mentor a global team of developers➢ United States➢ Costa Rica
4
Footer
#jenkinsconf
About Me
• Not a dedicated build engineer. It’s just one of the many hats I wear at work.
• Started using Continuous Integration with CruiseControl
• Moved to Hudson after viewing a session at JavaOne by the creator, Kohsuke
• Migrated to Jenkins after the Hudson split
5
Footer
#jenkinsconf
Problem Statement
• In an out-of-the-box Jenkins parameterized build scenario, user input such as notes, Git branches, and bug tracking issues are lost once a build has completed.
• The only historical reference, to most Project Managers is in a report.
• This data cannot easily be reused for future builds. • Using Jenkins, Groovy and Neo4j, the data can be
persisted for the life of a project.
6
Footer
#jenkinsconf
Standard Email Notification
• The standard post-build email notification only runs on failure and subsequent success
• The failure message contains a portion of the console output
• The success message is not much to look at either
7
Subject: Jenkins build is back to normal : hello-world #6
See <http://localhost:8080/jenkins/job/hello-world/6/changes>
Footer
#jenkinsconf
Standard Email-ext Success Email
• Sample body content of unmodified email-ext plugin email
• This is what is contained in the body of the email message sent by an unmodified project.
• Not exactly user-friendly.
8
juc-hello-world - Build # 5 - Successful: Check console outputat http://localhost:8180/jenkins/job/juc-hello-world/5/ to view the results.
#jenkinsconf
Footer
Standard Success Email – Multiple Builds
Clicking on the link, the information that the user entered for this build can be viewed:
Easy enough, but not very helpful.
Footer
#jenkinsconf
Standard Success Email – Multiple Builds
• The information is not so easily available to subsequent builds. The user must open each build individually to see what information had been entered.
• For example, a user has run three builds and included notes for each build. If the email has just been received for build #7, clicking on the link will take the user to the build page where the user notes can be viewed.
10
Footer
#jenkinsconf
Standard Success Email – Multiple Builds
However, if the user wants to see the notes of all the builds to date, each build must be opened one at a time to view the notes for that build. What a hassle!
11
Footer
#jenkinsconf
Standard Success Email – Multiple Builds
And if the build administrator has chosen to discard old builds and limit the number of builds to keep – good luck with that.
12
Footer
#jenkinsconf
#jenkinsconf
There Is A Better Way!!
13
#jenkinsconf
Footer
Enhanced Success Email
Wouldn’t it be nice to receive an email with this formatted content instead?
#jenkinsconf
Footer
Enhanced Success Email
That looks just as good on a mobile device!
#jenkinsconf
Footer
User Interface Possibilities
Wouldn’t it be nice to save, review, reconstruct and print reports from any and all builds via a spiffy user interface?
Footer
#jenkinsconf
User Interface Possibilities
17
Footer
#jenkinsconf
#jenkinsconf
And this…
Is how… it's done!
18
#jenkinsconf
Footer
Components Used for this Jenkins Job
Main Components Jenkins Plugins
Git Git plugin Build User Vars plugin
Groovy Groovy plugin EnvInject plugin
Gradle Gradle plugin Workspace Cleanup plugin
Artifactory Artifactory plugin Export Parameters plugin
Database (Neo4j) Email-ext plugin Job Exporter plugin
Credentials plugin Deploy plugin
Multiple SCMs plugin
#jenkinsconf
Footer
Database – Why Neo4j?
● Previous versions used MySQL● Adding projects with different build data requirements resulted in a non-
cohesive database, sparse cells● In Neo4j, a node can contain an arbitrary number of properties● The need for join tables is eliminated
#jenkinsconf
Footer
Database – Graph Model
● Neo4j is an open-source, NoSQL graph database implemented mainly in Java and Scala.
● The fundamental units that form a graph are nodes and relationships.
● Nodes are often used to represent entities
● Relationships organize the nodes by connecting them
#jenkinsconf
Footer
Database - Schema
Nodes Relationships
Build
SubProject OF_PROJECT
User BUILT_BY
Server DEPLOYED_TO
Property HAS_PROPERTY
Tag TAGGED_AS
● This is the base graph model the Jenkins application uses:
#jenkinsconf
Footer
Database – Node Properties
SubProject Server User Tag Property
name host name name name
title port email version value
artifactId userId
artifactExt
artifactDir
gitRepoName
buildTasks
context
Nodes and relationships can hold any number of individual attributes (key-value pairs)
Footer
#jenkinsconf
Cypher Query Language
• Neo4j uses Cypher as a query language.• Queries can be execute interactively via the built-in UI
OR via the REST API• Cypher borrows its structure from SQL — queries are
built up using various clauses such as: MATCH (This is the most common way to get data from the graph) WHERE ORDER BY RETURN
24
#jenkinsconf
Footer
Database – Create SubProject Node
MATCH (p1:Project) WHERE p1.name = "juc"CREATE (s1:SubProject { name: "juc-hello-world", title: "Jenkins User Conference Test", artifactId: "juc-hello-world", artifactExt: "war", artifactDir: "build", gitRepoName: "juc-hello-world", buildTasks: "clean war", context: "/hello-world"}),(p1)-[:HAS_SUBPROJECT]->(s1)
1.Get the Project node (p1)
2.Create lookup node SubProject (s1)
3.Add properties to node
4.Create a relationship between the Project and the SubProject
#jenkinsconf
Footer
Database – Query SubProject Node
• Neo4j provides a web interface on port 7474
• Queries can be executed interactively and the resulting graphs displayed
• For the SubProject node previously created, the following graph is displayed
• MATCH (s:SubProject)<-[HAS_SUBPROJECT]-(n) RETURN s,n
#jenkinsconf
Footer
Database – SubProject Node
• The node data can also be viewed in the web interface by toggling the view button
#jenkinsconf
Footer
Database – 'Build' Node
• Query the Build node inserted by the Jenkins job
MATCH (b:Build)-[]-(n)RETURN b,n
#jenkinsconf
Footer
Database – 'Build' Node
• Running the same query after running a second build yields both builds and associated nodes
MATCH (b:Build)-[]-(n)RETURN b,n
#jenkinsconf
Footer
Database – Query with Groovy
def neo4jQuery = “”” MATCH (u:User) WHERE u.id = '$userId' MATCH (s:SubProject) WHERE s.name = '$subProjectName' MATCH (v:Server) WHERE v.host = '$host' CREATE (t:Tag {name: '$tag', version: '$releaseVersion'}) CREATE (p1:Property {name: 'NOTE', seq: 1, value: '$note'}) CREATE (b:Build { number: '$buildNo', started: '${new Date()}', gitBranch: '$gitBranch' }), (b)-[:BUILT_BY]->(u), (b)-[:OF_PROJECT]->(s), (b)-[:DEPLOYED_TO]->(v), (b)-[:TAGGED_AS]->(t), (b)-[:HAS_PROPERTY]→(p1) RETURN ID(b)“””
1.Create the CYPHER query as a regular string
2.Variable substitution is supported
3.Return the unique ID of the newly created Build node
#jenkinsconf
Footer
Database – Query with Groovy
import groovy.json.JsonBuilderimport groovy.json.JsonSlurperimport groovyx.net.http.*
def server = new RESTClient("http://localhost:7474/db/data/cypher")server.headers['Accept'] = 'application/json; charset=utf-8'server.headers['Content-Type'] = 'application/json'server.contentType = JSON
def json = new JsonBuilder()json query: neo4jQuery
def results = server.post(body: json.toString())return results
1.Create a RESTClient to the Neo4j server
2.Set http headers to send and accept JSON
3.JSON is POSTed to the server
Footer
#jenkinsconf
#jenkinsconf
Putting it All Together
32
#jenkinsconf
Footer
Configuring the Build Job
Enable Parameterized Build• This example will only define 2
parameters:➢ GIT_BRANCH
➢ NOTES
#jenkinsconf
Footer
Configuring the Build Job
Git Setup
Source Code Management
• In ‘Branches to build’, enter the parameter➢ $GIT_BRANCH
• The variable will be replaced by the value entered by the user, defaulting to ‘master’. This is the branch that will be checked out and built.
#jenkinsconf
Footer
Configuring the Build Job
Git Setup – cont.
Check out to a sub-directory
• Since we will also be checking out the Groovy code from a different repository, it is important to keep the projects separated.
#jenkinsconf
Footer
Configuring the Build Job
Environment Variables
Inject environment variables to the build process
• Define global properties, used in both the Jenkins job and the Groovy script, in the ‘Property content’ section.
#jenkinsconf
Footer
Configuring the Build Job
Inject Passwords
Inject passwords into the build as environment variables
• Passwords are not included in the regular environment variable section because they would then be visible to end-users via the logs.
• Here, we are defining the database password that will be passed to the Groovy script.
#jenkinsconf
Footer
Configuring the Build Job
Build Steps
Execute Shell script
• The first step is to execute inline Bash commands to save the build properties to an external file.
• This makes it easier to pass properties back and forth between the Jenkins job and the Groovy script.
• Especially useful when there are many properties
#jenkinsconf
Footer
Configuring the Build Job
Build StepsExecute Shell script
#!/bin/bashcd $WORKSPACEecho "APPLICATION=juc-hello-world" >> $PROP_FILEecho "GIT_BRANCH=$GIT_BRANCH" >> $PROP_FILEecho "PROP_FILE=$PROP_FILE" >> $PROP_FILEecho "SCRIPT_DIR=$SCRIPT_DIR" >> $PROP_FILEecho "WORKSPACE=$WORKSPACE" >> $PROP_FILE# -----------------------------------------USER_NOTES=`echo "$NOTES" | sed -e '$ ! s|\(.*\)$|\1~|' | tr '\n' ' '`echo "NOTES=$USER_NOTES" >> $PROP_FILE
#jenkinsconf
Footer
Configuring the Build Job
Build Stepsexport job runtime parameters
● This build step exports various build job and Jenkins parameters into a property file named hudsonBuild.properties in the project workspace.
● The external Groovy script will read this file and load several of the properties into the master properties file
Property Value
build.user.id ID of user that started this job
build.user.name User that started this job
build.user.fullName Full name of user that started this job
build.user.emailAddress Email address of user that started this job
#jenkinsconf
Footer
Configuring the Build Job
Workspace
View of the job workspace after the Git repositories have been cloned and the property files created
#jenkinsconf
Footer
Configuring the Build Job
Execute Groovy Script● The Groovy script is called twice, first , using the ‘preBuild’
action and finally using the ‘postBuild’
• ‘preBuild’ is executed prior to running the Gradle build script
• ‘postBuild’ is executed after the successful completion of the Gradle build
#jenkinsconf
Footer
Configuring the Build Job
Execute Groovy Script - preBuild
This script will:1. Load the properties files ($PROP_FILE)2. Query the database for the project and server information3. Append database values as new properties back to the file
#jenkinsconf
Footer
Configuring the Build Job
Execute Groovy Script - preBuildRC_PROPERTIES_FILE=juc.propertiesGIT_BRANCH=origin/masterARTIFACT_DIR=/data/jenkins/ci/jobs/juc-hello-world/workspace/juc-hello-world//build/libARTIFACT_ID=hello-worldARTIFACT_TYPE=warBUILD_DIR=/data/jenkins/ci/jobs/juc-hello-world/workspace/juc-hello-worldBUILD_NUMBER=90BUILD_USER_FULLNAME=Robert McNultyCONTEXT_PATH=/hello-worldEMAIL_SUBJECT=JUC Hello World AppGIT_REPOSITORY_DIR=/data/jenkins/ci/jobs/juc-hello-world/workspace/juc-hello-worldGIT_REPO_NAME=juc-hello-worldPROJECT_DIR=/data/jenkins/ci/jobs/juc-hello-world/workspace/juc-hello-world/SERVER_HOST=localhostSERVER_PORT=8080ARTIFACTORY_VERSION=1.0.0-20140519.034403-8RC_TAG=JUC_HELLO_WORLD_2_0_0_SNAPSHOT_0071USER_NOTES_HTML=<li>BUILD 5 : Initial build of hello-world project.</li><li>BUILD 6 : Fixed a problem in the doohickey.</li><li>BUILD 7 : Working on thingamajig refactoring.</li>
#jenkinsconf
Footer
Configuring the Build Job
Inject environment variables● After the properties have been written to the file, it is necessary
to reload the file into Jenkins so the build can use the properties
● The ‘Inject environment variables’ plugin accomplishes this task
● The properties that were retrieved from the database are now available to the Jenkins job
#jenkinsconf
Footer
Configuring the Build Job
Run the Gradle Build● Build the project using the Gradle plugin
➢ Note: Maven can be used in place of Gradle
● The $BUILD_TASKS and $PROJECT_DIR variables were loaded from the properties file
#jenkinsconf
Footer
Configuring the Build Job
Execute Groovy Script - postBuild
This script will:1. Load the properties files ($PROP_FILE)2. Get artifact information (groupId, version)3. Create unique tag name4. Aggregate notes from previous builds
5. Create HTML elements to use in report6. Create and push new branch to Git
(using tag name)7. Append new properties back to
properties file
Footer
#jenkinsconf
#jenkinsconf
Post-build Actions …and more!
48
#jenkinsconf
Footer
Configuring the Build Job
Post-build actions
Once the Gradle build has completed, the artifacts deployed to Artifactory and the Groovy scripts finished processing, there are still tasks that are required to successfully complete the Jenkins job
• Deploy artifact to Tomcat (if webapp)• Send the Email
#jenkinsconf
Footer
Configuring the Build Job
Execute Groovy Script - postBuild
● For this demo, I used the ‘Deploy Plugin’ for simplicity● Unfortunately, this plugin does not recognize the environment variables imported into Jenkins
#jenkinsconf
Footer
Configuring the Build Job
Deploy to Tomcat● In our production instance
of Jenkins, I deploy to Tomcat via the postBuild Groovy script
● Since our Jenkins instance runs on Linux, it is a simple matter to use ‘curl’ to access the Tomcat manager and list, deploy or undeploy applications
static boolean deploy(context, path) { def curl = new StringBuilder().with { append "curl -X PUT -F file=@$path “ append ‘http://jenkins@localhost:8180’ append “/manager/text/deploy?path=$context" }.toString() def result = curl.execute().text return result.startsWith('OK')}
#jenkinsconf
Footer
Configuring the Build Job
Editable Email Notification● The ‘piece de resistance’ is the email report delivered to a list of
interested recipients• The ‘email-ext plugin’ does have the ability to use the imported
environment variables• Using the HTML Content Type enables the use of basic HTML and
CSS to format the page• The ‘dynamic’ portions of the email are populated using the
properties generated by the Groovy script and imported into Jenkins
#jenkinsconf
Footer
Configuring the Build Job
Editable Email NotificationUse HTML and CSS in the report template
#jenkinsconf
Footer
Configuring the Build Job
Editable Email Notification
Use HTML and CSS in the report template
● If the build is successful, configure the ‘Success’ trigger
Footer
#jenkinsconf
#jenkinsconf
Demonstration
55
Footer
#jenkinsconf
#jenkinsconf
Questions? Answers!
56
#jenkinsconf
Please Share Your Feedback
• Did you find this session valuable?• Please share your thoughts in the
Jenkins User Conference Mobile App.• Find the session in the app and click
on the feedback area.
57