Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)
-
Upload
oholiab -
Category
Technology
-
view
597 -
download
3
Transcript of Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)
![Page 1: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/1.jpg)
Keeping AWS in check with Puppetor
“Managing cloud networking with Puppet with a healthy dose of trolling yourself on the side”
Matt Carroll
![Page 2: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/2.jpg)
HELLO.
![Page 3: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/3.jpg)
![Page 4: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/4.jpg)
What’s all this about then?
● We use AWS extensively, especially EC2● We needed a way to organise our AWS networking (VPCs)● It needs to be:
○ Centrally managed○ Reproducible○ Declarative (idempotent)○ Ideally not another solution on top of a stack of solutions
![Page 5: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/5.jpg)
We (Tom) decided to use Puppet
We wrote a set of types and providers to use with the AWS API
Spoiler
![Page 6: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/6.jpg)
What I’m going to cover
1. Intro2. AWS networking basics3. Why we chose Puppet4. How it actually works5. Some of the “interesting” things we learned6. Summary7. Questions
YOU ARE HERE
![Page 7: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/7.jpg)
What I hope you’ll get from this
● A sense of Puppet as an extensible framework that can manage dependencies in external APIs
● Nodes are just units of computing capacity - an operation does not have to be a subset of a node
● Some insight into the types and providers system● Some of the strange things we learned along the way
… and as always...
![Page 8: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/8.jpg)
To serve as a lesson to others.
![Page 9: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/9.jpg)
SOME AWS BASICS.
![Page 10: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/10.jpg)
Some of this
isn’t exactly
retina resolution
![Page 11: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/11.jpg)
The (simplified) Hierarchy
Account
Region
VPC
Routetable
Subnet
![Page 12: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/12.jpg)
The (needlessly complicated simplified) Hierarchy
Account:• Region
• dopt: DHCP Options• vpc: Virtual Private Cloud
• igw: Internet Gateway• cgw: Customer Gateway• vgw: Virtual Gateway
• vpn: VPN• routetable
• subnet
![Page 13: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/13.jpg)
How do we make all these objects?
Just make them in the console!
![Page 14: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/14.jpg)
How do we make all these objects?
Just make them in the console!
IF YOU HATE YOURSELF
![Page 15: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/15.jpg)
How do we actually make all of these objects?
• AWS::SDK!
• We’ll have to ensure that the resources get created (and
purged)...
• Idempotently…
• With all their dependencies…
• Remind you of something?
![Page 16: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/16.jpg)
USE PUPPET!
![Page 17: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/17.jpg)
![Page 18: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/18.jpg)
Create Puppet types for VPC objects!● Resources for each of the objects
○ All API calls made on this level○ Will contain all code for reading and creating individual
resources○ No dependencies other than autorequires in the types -
those in the hierarchy earlier● Business logic in manifests
○ For your site-specific dependencies and network structure○ e.g. we have a separate VPC for each environment
● Data in hiera
![Page 19: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/19.jpg)
(Actually though) Why Puppet?
• With the ability to query and create and modify objects through the API we can state them declaratively
• We can thus create resources which can be included or purged idempotently
• Rather than specify order, we can state dependencies and allow Puppet to figure order out
![Page 20: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/20.jpg)
Why Puppet?Using this model you can even collect dependencies and resources:• Make all resources of a type with a parameter evaluate after
another
• Control purge and no-op of all aws resources
![Page 21: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/21.jpg)
Why Puppet?Admittedly it seems a little asymmetrical:• Puppet runs on nodes which creates
resources in AWS• AWS networking is not a subset of a node• Nodes just act as executors for creating AWS
resource
But actually this plays to our advantage (more on this later)
![Page 22: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/22.jpg)
HOW IT WORKS.
![Page 23: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/23.jpg)
• The bit that you specify in the manifest
• Really just a DDL for the resource’s metadata
• The “front end” for the pluggable “back end” (the provider)
• Interface to specify all properties and parameters
Puppet Types
![Page 24: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/24.jpg)
• The business end• This is where the API
code lives• Quite a lot longer• Handles all application
and querying of the resource including prefetch
Puppet Providers
![Page 25: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/25.jpg)
• Some resources are expensive to read, so you read them all once when you first come across one
• This is done before the catalog is completely compiled
What’s a “prefetch”?
![Page 26: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/26.jpg)
Manifest examples
![Page 27: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/27.jpg)
Associated Infrastructure
• How do you handle multiple accounts?• Dedicated AWS admin box within each
account to apply the resources on• IAM roles to handle credentials• Logging resource changes separately• Why not do it in the code...?
![Page 28: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/28.jpg)
LESSONS LEARNED.(the hard way)
![Page 29: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/29.jpg)
Don’tdo
whatDonny
Don’tdoes
![Page 30: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/30.jpg)
Why not put credentials in the code?
![Page 31: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/31.jpg)
Why not put credentials in the code?• We (I) tried to make an
aws_credentials type• This requires access to the
catalog in the prefetch phase so other resources can query it
• You ALSO need to guarantee type evaluation order and access to credentials in the catalog so that you can use prefetch
![Page 32: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/32.jpg)
Why not put credentials in the code?
• The instances method suddenly requires an argument
• In fact, so does anything that isn’t an instance method
• Actually ended up copying and pasting the resources resource to aws_resources and adding in a “credentials” parameter
![Page 33: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/33.jpg)
On second thoughts...
![Page 34: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/34.jpg)
Prefetch isn’t… Exceptional
• https://tickets.puppetlabs.com/browse/PUP-3656• This means that if anything goes awry in prefetch, puppet will
swallow it• In our case, we hit the API limit occasionally, meaning we got
duplicate resources
![Page 35: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/35.jpg)
So we did a bad thing...
![Page 36: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/36.jpg)
Asynchronous APIs tell lies• When you create an object you get
a 200 OK• This doesn’t mean “I’ve done it”,
this means “I’ll do it”• In a dependency chain, this can
mean that a resource is about to be created, but when it’s checked by another resource it’s not there yet
• In the Puppet paradigm it’s best just to run until convergence.
![Page 37: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/37.jpg)
SO IN SUMMARY...
![Page 38: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/38.jpg)
What I learned (the hard way)
• The “resources” resource• Is hard to understand unless
you read the code (type only) because Googling it is impossible.
• Could do with being able to apply other arbitrary properties and parameters?*
*this may be an awful idea
![Page 39: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/39.jpg)
What I learned (the hard way)
• Instances vs. class methods paradigm gets very confusing because it is tied to applying catalogue vs. prefetch
• Prefetch is generally pretty confusing when you add in dependencies TO the prefetch
![Page 40: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/40.jpg)
What really cool things I learned• You can reproduce AWS networking (and other objects
thanks to other contributors!)• A really cool insight into the types and providers system and
how it could grow in the future• Learning to treat servers not like pets OR cattle but as a
medium by which you do useful stuff
… and you could do this with a whole bunch of APIs
![Page 41: Keeping AWS in check with Puppet (Puppetcamp London 2014-11-17)](https://reader033.fdocuments.in/reader033/viewer/2022060206/55a23da21a28ab256e8b461d/html5/thumbnails/41.jpg)
github.com/bobtfish/puppet-aws_apiMatt CarrollSRE at Yelp
yelp.com/careers (we’re hiring)
THANKS FOR LISTENING!