Puppet for Production in WebEx - PuppetConf 2013
-
Upload
puppet-labs -
Category
Technology
-
view
20.879 -
download
0
description
Transcript of Puppet for Production in WebEx - PuppetConf 2013
![Page 1: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/1.jpg)
Cisco Confidential© 2011 Cisco and/or its affiliates. All rights reserved. 1
Puppet at Cisco CCATG
Aug 23, 2013
Reinhardt Quelle, Cloud Services Architect
![Page 2: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/2.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
CCATG Cloud Services Hosts and Manages SaaS Applications
Millions of Meetings
for
10s of Millions of Users
totaling
Billions of Minutes
each month
7x24x365
Cisco Social
WebEx
Connect
![Page 3: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/3.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3
US
UK
India
Australia
China
Hong Kong
Amsterdam
Japan
Global Footprint
~ 7K Hosts~ 8 Data Centers> 12 iPOPsPrivate Backbone
![Page 4: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/4.jpg)
Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 4Cisco Confidential 4© 2011 Cisco and/or its affiliates. All rights reserved.
Deployment Automation: Keeping these screens green
while evolving the service
![Page 5: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/5.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
Systems (or services) may look complicated
![Page 6: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/6.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
…but they are made of simple parts
![Page 7: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/7.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
…composed into modules
![Page 8: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/8.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
…and assembled into a system
![Page 9: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/9.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
Our systems are similar:Puppet manages “Resources”
Files
Packages
=
Users
Services
…Etc.
![Page 10: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/10.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
Which are composed into systems
Manifests - nodes.pp - site.pp
Classes, Modules
=
![Page 11: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/11.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11
So what’s missing?
![Page 12: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/12.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
TIME
![Page 13: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/13.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
Software versions, and often configuration versions, evolve as time passes:
BaseOS_Hardening v1.1ElasticSearch v0.20.6
JRE v1.7.0_25
BaseOS_Hardening v1.1ElasticSearch v0.90.2-1
JRE v1.7.0_25
• Some systems can simply be knocked over the head and recreated with fresh versions
• Others – notably most database servers – cannot; updates are performed in-place
• “Big Bang” upgrades don’t often happen; we step methodically through groups of machines
![Page 14: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/14.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14
In other words, SEQUENCE
![Page 15: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/15.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15
We never run one of anything
DC1 DC2
Multiple DC PairsMultiple Clusters of each Service type• By Customer Class• By Lifecycle Stage• By Special Needs
![Page 16: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/16.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
We must manage versions across these clusters, through time
By DC
Or by Node
v1v1 v2
v2
v1 v1v2v2
![Page 17: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/17.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
Cluster management includes a layer of definition and control above Puppet
“Blueprints” or “Models”
• JSON/YAML• TOSCA• CMDB *
Orchestration
• Fabric• SLiM• Mcollective
![Page 18: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/18.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
One Puppet Master?
Puppet Master
Manifests & Modules
DC1 DC2
![Page 19: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/19.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19
One Puppet Master?
Puppet Master
Manifests & Modules
DC1 DC2
Guess when you’ll need to push infrastructure changes the most!
![Page 20: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/20.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20
So, maybe one per DC?
Puppet Master
Manifests & Modules
DC1 DC2
Puppet Master
Manifests & Modules
![Page 21: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/21.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21
So, maybe one per DC?
Puppet Master
Manifests & Modules
DC1 DC2
Puppet Master
Manifests & Modules
![Page 22: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/22.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22
Perhaps Many…
WebEx Meetings
WebEx Connect
![Page 23: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/23.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23
Or, how about masterless?
puppet apply \ -–modulepath=/opt/puppet_local \ --execute “include servertype::front-end”
Manifests & Modules copy [/etc/puppet/*] to each node
![Page 24: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/24.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24
Our OS’s have robust packaging systems…
Manifests & Modules .rpm or .deb
yum install app_pp_v1 && puppet apply …’
privatepackage repository
![Page 25: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/25.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25
Orchestration tools provide a means of applying the changes
ssh node81 ‘yum install app_pp_v1 && puppet apply …’
fab dfw-frontends pp_apply:latest’
FabricMcollectiveSaltAnsible
![Page 26: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/26.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26
Cisco/WebEx uses multiple solutions
• Application Stacks/Deployment are NOT Homogenous
• The “right” solution for one stack not always right for another
• Share as much as possible, but don’t force it
• Tightly coupled systems are often rigid, brittle
• Solving big, general problems is hard; small bites are easily digested
“A foolish consistency is the hobgoblin of little minds” – Emerson
![Page 27: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/27.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
Whether one, many, or no servers,Puppet is CODE
• Every artifact (module, manifest, Hiera file) is checked into version control
• Versions are packaged and released and should go through same promotion process as application code
• All good coding practices applyModular
Well defined interfaces
Tested
Shared
![Page 28: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/28.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28
Design Pattern: ServerType/Profile/Module• Modules are the atomic packages of configuration
• “Profiles” bundle modules into commonly used sets for ease of consumption:
BaseOS
JavaApp
Tomcat App
• A given machine has exactly one “ServerType”
• Inspired by Chef’s “roles”, and similar to Craig Dunn’s Role/Profile/Modules
• At the code level, these are actually all just modules
![Page 29: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/29.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29
Design Pattern: Cooperative Modules• Build loosely coupled modules that can work together if installed together,
but that can stand on own, too
• Example: standard monit config includes /etc/monit/conf.d/*Application that wants to be monitored just drops file in this location
• logrotate, collectd, apache, nginx, etc all support
class elasticsarch {
…
if $monit::include_dir != undef { validate_absolute_path($monit::include_dir) file { "${monit::include_dir}/${monit_config_file_name}": ensure => present, content => template("elasticsearch/${monit_config_file_name}.erb"), notify => Service['monit'], } }}
![Page 30: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/30.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30
Design Pattern: Assembling Configuration• The singleton ServerType defines which profiles and modules are
included; structure and order
• Puppet Librarian and its Puppetfile describe which version of a module is used, and where it comes from
• After Puppet Librarian has run and downloaded all required assets, FPM is called upon to build the package
• Dev cycle includes doing local builds/tests against Vagrant in both develpers personal machines as well as the Jenkin’s build farm
• Upon successful build, packages are uploaded to repositories
![Page 31: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/31.jpg)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31
Miscellaneous Tidbits
• Dependencies in Puppet, not RPM/DEB
• Packaging works for deploying to Puppetmasters, too.
• Modules are designed for transparency, simplicity: “4AM-proofing”
• Composition usually trumps inheritance
• Tim Bell and the CERN folks talk of “Pets” and “Cattle”You can only shoot a system in the head if you can create another at will
• “Fried” or “Baked”? YES.
![Page 32: Puppet for Production in WebEx - PuppetConf 2013](https://reader033.fdocuments.in/reader033/viewer/2022061221/54bcfe7a4a7959dc2d8b45c3/html5/thumbnails/32.jpg)
Cisco Confidential© 2010 Cisco and/or its affiliates. All rights reserved. 36Cisco Confidential© 2011 Cisco and/or its affiliates. All rights reserved. 36
Photo placeholder
Thank you
Please tell your friends we’re hiring DevOps Engineers!