Case Study: Green Field Implementation of Puppet 3.0 at ESPN
-
Upload
puppet-labs -
Category
Technology
-
view
1.905 -
download
0
description
Transcript of Case Study: Green Field Implementation of Puppet 3.0 at ESPN
PuppetConf 2013
CHALLENGES
• Rapid growth– Number of servers being provisioned– New services being provided
• Manual configurations
• Drift
• Silos
• Traditional clouds not reasonable
Objectives
• Push button configuration
• Self service for customers
• Visibility into state of configurations– Reports
• Simplify knowledge transfer
• Data driven
• Elastic
• Build/rebuild a node at anytime
• Support workstations/dev/test/prod
• Allow adhoc manual changes*
• Do not slow down development teams
• Include legacy servers
Application
Middleware
OS
Server
Storage
Network
IAAS
PAAS
SAAS
Puppet
Puppet Versions
• Puppet 3.0.0
• Puppet Dashboard 1.2.12
• PuppetDB 1.0.1
• MCollective 2.2.1
• ActiveMQ 5.5
The Data
• Inherent Facts – Facter– Exists simply because the node exists.
• Derived Facts – Business rules applied to inherent facts.– Puppet custom facts
• Assigned Facts – Exist because we deem it to be true.– Top or node scoped variables
Assigned Facts• espn_role
– Identifes the role class to be applied• espn_cluster
– Identifies nodes in the cluster• espn_env
– workstation, dev, test, qa, prod• espn_owner
– change notifications– security delegation – licensing
What to manage? Everything!
Role
• Aligns with business and or IT needs
• Defines everything on a node
• Used for classification of nodes
• Exactly one role per node
• Includes profiles and releases
• espn_role
Group everything by roles
AA BB CC
Release Environment
• Not puppet environments
• Determined by application development and release cycle
• espn_env
Group by release environment
AA BB CC
ProdProd
QAQA
DevDev
WorkWork
Clusters
• Always identical
• Configured to interact with one another
• espn_cluster
Group by clusters
AA BB CC
ProdProd
QAQA
DevDev
WorkWork
Hiera hierarchy in theoryhostname/%{hostname}
cluster/%{espn_cluster_name}
role_env/%{espn_role}_%{espn_env}
role/%{espn_role}
env/%{espn_env}
network/%{espn_network}
os/%{operationsystem}
default
Hiera hierarchy in practice
role_env/%{espn_role}_%{espn_env}
role/%{espn_role}
env/%{espn_env}
network/%{espn_network}
defaults
Classification
Application
Middleware
OS
Server
Storage
Network
IAAS
PAAS
SAAS
Role
Profiles
Releases
Role
Profiles
Releases Resources
Resources
Classes Resources
Resource Hierarchy
Resources
• The building blocks
• Everything managed is a resource
• Defined in modules, used by profiles
• Two resources never manage the same configuration
Modules
• Isolate resources within the module
• Never reference another module
• No organizational specific logic
• init.pp is a minimal installer
• Reusability is key
R RR RR RR R
R RR RR RR R
R RR RR RR R
R RR RR RR R
R RR RR RR R
Resources by module
R RR RR RR R
R RR RR RR R
R RR RR RR R
R RR RR RR R
R RR RR RR R
Never cross modules
Role
Profiles
Releases Resources
Resources
Classes Resources
Resource Hierarchy
Profile
• Defines the platform
• Cross module references
• Enforces dependencies between modules
• Class parameters prevent hiera overrides
Example Profileclass profile::jboss_eap_6(
$java_version=‘jdk_1_7_u10’,
$mod_cluster_version=‘1.2.3’,
$httpd_version=‘2.2.22’,
){
include “java::${java_version}”
include directories::middleware
class{‘mod_cluster’:
version => $mod_cluster_version,}
class{‘httpd’:
version => $httpd_version,}
class{‘jboss_eap_6’:
java_home => getvar(“java::${java_version}::home”),}
}
Role
Profiles
Releases Resources
Resources
Classes Resources
Resource Hierarchy
Release
• Special type of profile
• Knows how to install on top of a profile
• Deploys resources from an “artifact hash”
• Cleans up removed artifacts
• Driven by versioned release id– espn_release_id
Artifact Hash
• Defines abstract resources in a release
• Contract between developers and operational groups
• Profile agnostic
Example Artifact Hashapplications
‘jee-app.ear’:
version: 1.2.3
espn.war:
version: 1.0.0
datasources:
datasource1:
driver_name: ”oracle”
libraries:
utility.jar:
version: 1.0.0
Example ReleaseClass release::studio_record{
artifact_hash = hiera(‘artifacts’,undef,”release/${espn_release_id}”)
#modify the artifact hash so it can be used with create_resources
#set organization specific parameters such URL’s to the artifact repo
resources {‘jboss7_datasource: purge => true,}
create_resources(jboss7_datasource, $artifact_hash[‘datasources’])
resources {‘jboss7_deployment: purge => true,}
create_resources(jboss7_deployment, $artifact_hash[‘applications’])
}
Role
Profiles
Releases Resources
Resources
Classes Resources
Resource Hierarchy
Example Role
class role::studio_record{
include profile::base
include profile::jboss_eap_6
include profile::mysql_5
include release::studio_record
}
site.pp
# assigned facts retrieved from external datasources
$espn_cluster_nodes = espn_cluster_nodes()
$espn_release_id = hiera(‘espn_release_id’)
# single assigned fact drives 100% of classification
node default {
include “role::${espn_role}”
}
Defining a Node
MCollective
Colonol John Boyd• Military Strategist
• OODA Loop
ObservationsDecision
(Hypothesis)Action(Test)
Observe Orient Decide Act
Feedback
Feedback
UnfoldingCircumstances
OutsideInformation
UnfoldingInteraction
WithEnvironment
CulturalTraditions
Analysis &Synthesis
PreviousExperience
NewInformation
GeneticHeritage
Feedback
ImplicitGuidance& Control
ImplicitGuidance& Control
OODA Loop
"Time is the dominant parameter. The pilot who goes through the OODA cycle in the shortest time prevails because his opponent is caught responding to situations that have already changed.“
Harry Hillaker (chief designer of the F-16)
ObservationsDecision
(Hypothesis)Action(Test)
Observe Orient Decide Act
Feedback: Puppet runs on dev/test puppet environments
Feedback
UnfoldingCircumstances
OutsideInformation
UnfoldingInteraction
WithEnvironment
CulturalTraditions
Analysis &Synthesis
PreviousExperience
NewInformation
GeneticHeritage
Feedback
ImplicitGuidance& Control
ImplicitGuidance& Control
ObservationsDecision
(Hypothesis)Action(Test)
Observe Orient Decide Act
Feedback: Puppet runs on dev/test puppet environments
Feedback: (Test) Noop puppet run on production
UnfoldingCircumstances
OutsideInformation
UnfoldingInteraction
WithEnvironment
CulturalTraditions
Analysis &Synthesis
PreviousExperience
NewInformation
GeneticHeritage
Feedback
ImplicitGuidance& Control
ImplicitGuidance& Control
Noop Puppet Runs
• Hourly cron job– “mco puppet runall 20 –noop”
• Dashboard displays unresponsive nodes– no_longer_reporting_cutoff
ObservationsDecision
(Hypothesis)Action(Test)
Observe Orient Decide Act
Feedback: Puppet runs on dev/test puppet environments
Feedback: (Test) Noop puppet run on production
UnfoldingCircumstances
OutsideInformation
UnfoldingInteraction
WithEnvironment
PuppetDashboard
CulturalTraditions
Analysis &Synthesis
PreviousExperience
NewInformation
GeneticHeritage
Feedback: (Action) No-noop puppet run on production
ImplicitGuidance& Control
ImplicitGuidance& Control
Configuration Changes• Always noop first
• Always target nodes with filters
• Always use tags
• Validate changes then apply –no-noop
mco puppet runonce --no-noop -C jboss-eap-6 --tag initscript --tag jboss-eap-6
ObservationsDecision
(Hypothesis)Action(Test)
Observe Orient Decide Act
Feedback: Puppet runs on dev/test puppet environments
Feedback: (Test) Noop puppet run on production
UnfoldingCircumstances
OutsideInformation
CulturalTraditions
Analysis &Synthesis
PreviousExperience
NewInformation
GeneticHeritage
Feedback: (Action) No-noop puppet run on production
Implicit Guidance & Control:MCollective Agent Plugins
ImplicitGuidance& Control
UnfoldingInteraction
WithEnvironment
PuppetDashboard
20/20 Hindsight
Do Differently?
• Implement MCollective first– Security– Sub-collectives– Availability
• Plan for developer dashboards
• Implement Custom ENC
Nice to have?
• Puppet runs that span multiple nodes– Allocate disk on node A, create shared
filesystem on node B
• Resources automatically tagged with catalog unique identifiers
• Role based access control for dashboard
Questions?