DTO #ChefConf2012
-
Upload
anthonyshortland -
Category
Business
-
view
25.065 -
download
1
description
Transcript of DTO #ChefConf2012
Using Rundeck and Chef to build DevOps
ToolchainsAnthony Shortland - [email protected]
Moses Lei - [email protected]
DTO Solutions
Process Improvement
Design
Business
Release Engineering
Development
QA
Operations
Program Management
L/T Lead timeP/T Process timeH/C Head countS/R Scrap rate
D DefectsEP Extra processesM Motion
PD Partially doneTS Task switchingW Waiting
Service Delivery Process Current state value stream map
DevelopmentEnvironments
ReleaseBuild and deploy
QAEnvironments
PerformanceEnvironment
StagingEnvironment
ProductionEnvironment
Merge project branch
Deploy to Performance
Web Assets, flows and mockups
Features and fixes
Set of service
branches
Documented test plans and cases
Request QA build
Daily Scrum meeting
Daily Scrum meeting
Deployment Meeting Business Systems
Support
Stan Walters
Change Control
Project Management
Eric Switzer
Change request
BusinessAnalysis
Luke Hanson
Full Projects / Expedites
Estimates and
requirements
Technical Specification
Deployment Document
Change Control Forms
Bugzilla issues
Jira issues
Jira issues
Tested packages
Development
Joe Harbaugh
L/T 10 weeksP/TH/CS/R
6 weeks438%
Developweb assets
Bob McNulty
L/T 8 weeksP/TH/CS/R
6 weeks158%
DevelopmentBuild and deploy
Joe Harbaugh
L/T 1 daysP/TH/CS/R
2 hours220%
Development Integration testing
Joe Harbaugh
L/T 2 dayP/TH/CS/R
2 day1196%
Performance testing
Deploy toProduction
Build test plan
Raj Lee
L/T 8 daysP/TH/CS/R
4 days230%
Build test cases
Raj Lee
L/T 2 weeksP/TH/CS/R
8 days220%
Fernando Gomez
L/T 11 hoursP/TH/CS/R
8 hours25%
Acceptance testing
Raj Lee
L/T 5 daysP/TH/CS/R
8 hours210%
Feature testing
Raj Lee
L/T 5 weeksP/TH/CS/R
3.5 weeks520%
Allen Cannata
L/T 1 weekP/TH/CS/R
3 days166%
RT tickets
Cut project branch
Chris Holmes
L/T 20 minutesP/TH/CS/R
8 minutes20%
Deploy to Staging
Frances Middleton
L/T 7 daysP/TH/CS/R
110 minutes470%
Staging testing
Frances Middleton
L/T 4 daysP/TH/CS/R
3.5 days566%
Allen Cannata
L/T 10 hoursP/TH/CS/R
7 hours550%
Frances Middleton
L/T 6 daysP/TH/CS/R
90 minutes366%
Fernando Gomez
L/T 1 hourP/TH/CS/R
30 minutes150%
Program Committee
Eric Switzer
5 weeks
3 weeks
5 weeks
5 weeks
5 hours
2.5 days
3 day 8 hours 6 hours
2 hours 5 days
2 weeks
2.5 weeks
110 minutes
6.75 days
4.5 days
2.5 days
9 hours
2 hour
Process Time = 59 days
Lead Time = 136 days
D
EP
DEPx2
Mx3W
PDD
Dx3Mx3
TSx5 Dx2 Wx2 PDx3
W
DPDx4 Mx7
WDx2
Mx3PD
WW
Estimates and
requirements
TechnicalAnalysis
Maya Highsmith
Full Projects / Expedites
Test Requirements
Luke Hanson
L/T 21 daysP/TH/CS/R
3 days250%
EP
EP
Value Stream
65% +5%
7 Days 3h, -5h
Delivery Cycle Time
Value-Creating Time (VCT)
Metrics: 1 week, 1 week historical
Scott, Erica, Reigner,…
2 versions4 environments8 15m2 Hours
-30m
Provision Lead Time
85%, -5%
Stability
Environment: Production
5% 1/20
5Delivery Support
Deployments
+2 environments +1 versionProvision Scrap Rate
20% 2/10
Delivery Scrap Rate
-3+1
RetailTrader - v133 "Add logic updates " 4 Days Remain 10d total
4/16 Open
Work Tasks
15 Closed
Build Success
OK retailtr-v133.war
80% 4/5-5%
-5%
10% Support
Team Utilization
34m avgLead Time
MilestoneProduct
Features
Process Automation
• End-to-end automation• Supports business goals• Optimize the system, not the parts
Take a toolchain approach
Unix Tool Chain
Measurement Instrumentation
System
Charting, History, SPC
Eventing, Alarm Mgmt
Control
Runbook Automation
Industrial Control Automation
Introduction
E-commerce and software-as-a-service business models have matured quickly, but the
quality of the web operations that support these businesses has lagged behind.
Outages are all too common. High variability and defect rates are bemoaned but have
become an accepted reality. Key engineers spend all day (and sometimes all night)
mired in deployment issues and bottlenecks. And topping it all off, what tooling that
does exist are usually a custom one-offs that are brittle and expensive to maintain.
Today’s business of operating software over the Web as a revenue producing service is a
dramatic departure from the days when software was primarily produced for delivery on
physical mediums and IT Operations was considered a back-of-the-house support
function. Shouldn't we be completely rethinking our tooling and operational capabilities
to match these new innovations?
In short, we need to get out of Web Operations 1.0 -- mired in legacy tools, outdated
approaches, and low expectations -- and into Web Operations 2.0 where tools and
procedures are built from the ground up for highly efficient, reliable, and agile
operations.
There are multiple factors that go into achieving excellence in Web Operations, but the
linchpin that holds it all together is a fully automated provisioning system.
In this paper we will be:
1. Defining what we mean by "fully automated provisioning"
2. Explaining why virtualization and cloud computing efforts fail without fully
automated provisioning capabilities
3. Proposing a reference open source tool chain for fully automated provisioning
4. Describing a live implementation where a leading online retailer is actively rolling
out a fully automated provisioning system using all open source tools
2 Web Ops 2.0: Achieving Fully Automated Provisioning
” Today’s business of
operating software over the
Web as a revenue producing
service is a dramatic
departure from the days
when software was primarily
produced for delivery on
physical mediums...”
+
Service Delivery Platform Elements
Design points1. Everything is code
2. Everything is packaged
3. Separate code and configuration
4. Separate environment-dependent attributes
5. Balance distributed vs local orchestration
6. Resolve directed vs convergent orchestration
7. Integrate application and infrastructure provisioning
8. Design for flow not the organization
Example Service Delivery Platform
Integration Plugins • Jenkins/GitHub - https://wiki.jenkins-ci.org/display/
JENKINS/Github+Plugin
• Jenkins/Rundeck - https://wiki.jenkins-ci.org/display/JENKINS/RunDeck+Plugin - Vincent Behar
• Nexus/Rundeck - https://github.com/vbehar/nexus-rundeck-plugin - Vincent Behar
• Nexus/Yum - http://code.google.com/p/nexus-yum-plugin/
• Chef/Rundeck - https://github.com/opscode/chef-rundeck - Adam Jacob
Example Service Delivery Platform Layout
Development ServerDeploy ServerBuild Server
Source CodeBuild ArtifactRepository ClientServer
Deployment ConsoleBuild Console
Application Server Database
Develop
Build - Projects by artifact
Build - Trace by build number
Build - Template RPM SPEC
[ec2-user@centos-ami bank]$ cat SPECS/dukesbank-ear.spec.template Summary: Duke's Bank ear fileName: dukesbank-earVersion: ${version}Release: ${release}
License: LGPLGroup: Applications/System
Prefix: /opt/jboss-4.0.3SP1BuildArch: noarchRequires: jboss = 4.0.3SP1
.
.
.
Build - Register packages
.
.
. <target name="rpmdeploy" depends="resolve,rpmbuild"> <ivy:makepom ivyfile="${basedir}/ivy.xml" pomfile="RPMS/noarch/dukesbank-ear-${version}-${release}.noarch.pom"/> <ivy:publish resolver="nexus-deploy" pubrevision="${version}-${release}" overwrite="true" publishivy="true"> <artifacts pattern="RPMS/noarch/[artifact]-[revision].noarch(-[classifier]).[ext]"/> <!--artifact name="dukesbank-ear" type="rpm" url="${build.dir}/rpm/RPMS/noarch/dukesbank-1.0-2.noarch.rpm" /--> </ivy:publish> </target>...
</project>
Repository - Build view
Repository - Deploy view
Deploy - Package version list[root@centos-ami yum.repos.d]# cat /etc/yum.repos.d/dukesbank.repo [dukesbank]name=DukesBank demo repobaseurl=http://ec2-75-101-135-205.compute-1.amazonaws.com:8081/nexus/content/repositories/releasesenabled=1gpgcheck=0metadata_expire=1s[root@centos-ami yum.repos.d]# repoquery --repoid=dukesbank --search dukesbank-eardukesbank-ear-0:1.0-14.noarchdukesbank-ear-0:1.0-15.noarchdukesbank-ear-0:1.0-16.noarchdukesbank-ear-0:1.0-18.noarchdukesbank-ear-0:1.0-19.noarchdukesbank-ear-0:1.0-20.noarchdukesbank-ear-0:1.0-21.noarchdukesbank-ear-0:1.0-22.noarchdukesbank-ear-0:1.0-23.noarchdukesbank-ear-0:1.0-24.noarchdukesbank-ear-0:1.0-25.noarchdukesbank-ear-0:1.0-26.noarch[root@centos-ami yum.repos.d]# cat ~jenkins/jobs/dukesbank-ear.rpm/workspace/versions.json [ "1.0-14", "1.0-15", "1.0-16", "1.0-18", "1.0-19", "1.0-20", "1.0-21", "1.0-22", "1.0-23", "1.0-24", "1.0-25", "1.0-26", "1.0-27", "1.0-28", "1.0-29", "1.0-30", "1.0-31", "1.0-32", "1.0-33", "1.0-34" ][root@centos-ami yum.repos.d]#
Deploy - Nodes & roles
Deploy - Node provider[ec2-user@ip-10-36-119-128 ~]$ curl http://ec2-75-101-135-190.compute-1.amazonaws.com:9980/<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE project PUBLIC "-//DTO Labs Inc.//DTD Resources Document 1.0//EN" "project.dtd"><project><node name="ip-10-195-11-159.ec2.internal" type="Node" description="ip-10-195-11-159.ec2.internal" osArch="x86_64" osFamily="unix" osName="centos" osVersion="6.2" tags="_default,base,chef_server" username="ec2-user" hostname="ip-10-195-11-159.ec2.internal" editUrl="http://ip-10-195-11-159.ec2.internal:4040/nodes/ip-10-195-11-159.ec2.internal/edit"/><node name="ip-10-36-119-128.ec2.internal" type="Node" description="ip-10-36-119-128.ec2.internal" osArch="x86_64" osFamily="unix" osName="centos" osVersion="6.2" tags="dev,base,dukesbank_database_master,dukesbank" username="ec2-user" hostname="ip-10-36-119-128.ec2.internal" editUrl="http://ip-10-195-11-159.ec2.internal:4040/nodes/ip-10-36-119-128.ec2.internal/edit"/></project>
Deploy - Application attributes[ec2-user@centos-ami chef-repo]$ cat data_bags/apps/dukesbank.json { "id": "dukesbank", "server_roles": [ "dukesbank" ], "type": { "dukesbank": [ "jboss", "jboss_webapp" ] }, "database_master_role": [ "dukesbank_database_master" ], "packages": { "dev": { "dukesbank-ear": "1.0-23" } }, "jboss_state": { "dev": "start" },...
Links
• DTO Solutions - http://dtosolutions.com• DTO Labs - http://dtolabs.com• DTO Labs Github - https://github.com/dtolabs • Rundeck - http://rundeck.org• Devops-Toolchain -
http://groups.google.com/group/devops-toolchain