PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler...
-
Upload
puppet -
Category
Technology
-
view
107 -
download
1
Transcript of PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler...
![Page 1: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/1.jpg)
Tyler Langlois, October 12th 2017 Software Engineer, Elastic @leothrix, github: tylerjl
Custom Types and Providers: Modeling Modern REST Interfaces and Beyond
![Page 2: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/2.jpg)
2
Obligatory “About Me” Slide
• Been with company since 2014
• Co-maintainer of Elastic Puppet modules (primarily Elasticsearch and Kibana)
• Puppet-ing in one way or another over my whole professional career
• Brought too many Elastic stickers that need to be given away (please partake)
• Talk to me about Elasticsearch/Logstash/Kibana/Beats!
Infrastructure/Operations/Software Engineer @ Elastic
![Page 3: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/3.jpg)
3
Who is This Presentation For?
Developers who work with Puppet modules
Puppet users who want to dip into native type/provider development
“What in the %@#$ is the Elasticsearch module doing”
Operators who want to automate against APIs
Hopefully empowers you to implement custom resources on your own
![Page 4: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/4.jpg)
4
WHAT DOES THIS TALK’S TITLE EVEN MEAN?
![Page 5: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/5.jpg)
5
![Page 6: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/6.jpg)
6
Managing Resources with Raw APIsExample: CloudFormation
• Pro:
• Infrastructure resources are data
• Extensible
• Con:
• Managing changes
• Grokking huge chunks of JSON
![Page 7: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/7.jpg)
7
Modeling Resources with Raw APIsExample: Terraform
• Pro:
• Readable
• Manageable
• Lifecycle + changes
• Interoperability between other systems
![Page 8: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/8.jpg)
8
Modeling Resources in Puppet
A DSL to Model Disparate Resources
A Graph to Manage Relationships
A Concept of Changes to Manage Lifecycles
ls, stat, chmod, chown
sysv, systemd, upstart
deb, rpm, pkg }
![Page 9: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/9.jpg)
9
Modeling Resources in PuppetAbstraction is Powerful
file { “/tmp/foo”:source => “puppet:///foo”,
} ->package { “foo”:
source => “/tmp/foo”} ~>service { “foo”:
ensure => “running”,}
![Page 10: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/10.jpg)
10
Modeling Resources in Puppet
?Elasticsearch
Logstash
Other REST APIs }
![Page 11: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/11.jpg)
11
Modeling Resources in PuppetExtending the idea to APIs
elasticsearch::template { “logstash”:content => {
“template” => “*”,“settings” => {
“number_of_replicas” => 0}
}} ->service { “es-app”:
ensure => “running”}
![Page 12: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/12.jpg)
12
Modeling Resources in Puppet
• State Changes
• Instead of comparing changes with GET responses and template files, compare during a no-op
• A change in state can form dependencies and refresh events into other resources
• Trickling changes up via reports lends better visibility
Benefits
![Page 13: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/13.jpg)
13
Modeling Resources in Puppet
• State Changes
• More finely-grained control
• Most resources can be represented as Puppet hashes, so Hiera can be fully leveraged
• Communicating via full Ruby HTTP libraries means CA files, auth, and more are easier to control
• TESTS!
Benefits
![Page 14: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/14.jpg)
14
Modeling Resources in Puppet
• State Changes
• More finely-grained control
• Some existing API-based resources:
• Kubernetes module (swagger-generated)
• Google Cloud
• Following examples will be low-level (i.e. with just native Ruby HTTP libraries)
• …hopefully, will help you write your own for $system
![Page 15: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/15.jpg)
15
Let’s (briefly) talk about Puppet Types and Providers
![Page 16: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/16.jpg)
16
Types, Providers, and their Resources
Underlying Resource
Puppet Provider
Puppet Type
• Has some way to change a property
• Its state is introspectable and discoverable
• Uniquely identified
• How Ruby interacts with actual commands/system properties
• Knows how to discover the properties of resources
• Normalized provider API to Puppet DSL
• Somewhat typed, catalog compilation
• Abstraction over providers
![Page 17: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/17.jpg)
17
Types, Providers, and their Resources: service
• systemctl/service/rc commands
• Startup visibility with enable/chkconfig/etc.
• Primarily shell-based for state
• One provider for each init system
• Ruby knows which shell commands to invoke to start, stop, enable, etc.
• Unified API to start, enable, and restart a general service resource
• Abstraction over provider-specific implementations
• What we see in a manifest
Underlying Resource
Puppet Provider
Puppet Type
![Page 18: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/18.jpg)
18
Types, Providers, and their Resources: elasticsearch
• REST API endpoints
• Objects modeled in JSON
• Individual endpoints via _template, _ingest, etc.
• One provider base class, one provider per resource type
• Using native Ruby HTTP APIs are high-level enough
• Better alternative than `exec { “curl”:`
• Resource properties expressed in Puppet DSL hashes
• We don’t make API calls, we declare desired state
Underlying Resource
Puppet Provider
Puppet Type
![Page 19: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/19.jpg)
19
Then:
![Page 20: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/20.jpg)
20
Now:
![Page 21: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/21.jpg)
Case Study: Elasticsearch Pipelinescurl vs. Puppet
![Page 22: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/22.jpg)
22
Ingest Pipelines
![Page 23: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/23.jpg)
23
Ingest Pipelines
![Page 24: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/24.jpg)
24
Ingest Pipelines
• All pipelines are uniquely identified by a name (like defined or native types!)
• Endpoints to manage pipelines:
• GET to retrieve JSON object enumerating all pipelines
• Note: can also retrieved based by name alone
• PUT to create with JSON body
• Note that we’re using unauthenticated APIs right now
Key observations
![Page 25: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/25.jpg)
25
Ingest Pipelines: Puppet Type
![Page 26: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/26.jpg)
26
Ingest Pipelines: Puppet Type (Implementation)
![Page 27: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/27.jpg)
27
Ingest Pipelines: Puppet Type (Implementation)…what the included abstraction does
![Page 28: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/28.jpg)
28
Ingest Pipelines: Puppet Provider (Implementation)
![Page 29: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/29.jpg)
29
Ingest Pipelines: Puppet Provider (details)…what the parent class does
![Page 30: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/30.jpg)
30
Ingest Pipelines: Puppet Provider (details)…what the parent class does
![Page 31: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/31.jpg)
31
Ingest Pipelines: Puppet Tests
![Page 32: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/32.jpg)
32
Ingest Pipelines
• That’s most of it!
• Test-driven development + rspec makes it smooth
• Bulk is abstracted; the beefy parts are in parent classes and reused by templates, indices, etc.
• Native types and providers ≠ scary
Summary
![Page 33: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/33.jpg)
33
Fitting REST Resources Into PuppetConsiderations
`exists?` versus `prefetch`
Leveraging type-level tools
HTTP
API availability
1
2
3
4
![Page 34: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/34.jpg)
34
![Page 35: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/35.jpg)
35
An Example of Returning a Hash to PrefetchAutomatically Gathering Resources
uri = URI(“http://localhost:9200/_template”) http = Net::HTTP.new uri.host, uri.port req = Net::HTTP::Get.new uri.request_uri response = http.request req JSON.parse(response.body).map do |object_name, api_object| { :name => object_name, :ensure => :present, :content => api_object, :provider => name } end
![Page 36: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/36.jpg)
36
Advantages
• puppet resource functionality
• Minimizes chatter with API endpoints
• i.e., checking for existence versus properties, etc.
• Call flush only when necessary
• Additional API freebies (i.e., centralized access in flush(), etc.)
Prefetching resources versus vanilla exists?
![Page 37: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/37.jpg)
37
![Page 38: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/38.jpg)
38
Fitting REST Resources Into PuppetConsiderations
`exists?` versus `prefetch`
Leveraging type-level tools
HTTP
API availability
1
2
3
4
![Page 39: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/39.jpg)
39
Response Content vs. Request ContentUsually never 1:1 mappings
{ "logstash": { "order": 0, "version": 60001, "index_patterns": [ "logstash-*" ],
. . .
elasticsearch::template { 'logstash': content => { 'template' => '*', 'settings' => {
. . .
vs.
![Page 40: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/40.jpg)
40
Types To the Rescue
• A resource’s desired state is almost never the plain response for a query against the resource
• Example: kubernetes Deployment versus the state of a Deployment
• munge can help unify the resource versus JSON for comparability
• insync? can be enhanced to understand which fields are being explicitly controlled by a user
• e.g., I want {“foo”: “bar”} set, I don’t care about what’s in {“another”: “field”}
• Used pretty heavily in puppet-elasticsearch
Managing response data
![Page 41: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/41.jpg)
41
Example: Setting Default FieldsElasticsearch template
# Set default values for templatesmunge do |value|
{'order' => 0,'aliases' => {},'mappings' => {}
}.merge(value)end
![Page 42: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/42.jpg)
42
Example: Unifying FormattingElasticsearch template
# Normalize then compare the Puppet hash and jsondef insync?(is)
Puppet_X::Elastic.deep_implode(is) == \Puppet_X::Elastic.deep_implode(should)
end
{ “foo”:{ “bar”: “value” }
}
{“foo.bar”: “value”
}
![Page 43: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/43.jpg)
43
Fitting REST Resources Into PuppetConsiderations
`exists?` versus `prefetch`
Leveraging type-level tools
HTTP
API availability
1
2
3
4
![Page 44: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/44.jpg)
44
HTTP In Providers
![Page 45: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/45.jpg)
45
HTTP In Providers
• Native HTTP libraries let us more easily control and pass:
• TLS certificate authorities and verification booleans
• HTTP basic auth credentials
• Failure cases (timeouts, 4xx/5xx response codes, etc.)
• In this case with Elasticsearch, error responses can return JSON messages for more helpful Puppet failures
![Page 46: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/46.jpg)
46
Fitting REST Resources Into PuppetConsiderations
`exists?` versus `prefetch`
Leveraging type-level tools
HTTP
API availability
1
2
3
4
![Page 47: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/47.jpg)
47
API Availability
• What happens if:
• An API-based REST resource requires an API to be up, not just a daemon?
• A resource should block until one is available?
• An unrelated resource needs that API as well?
Weird edge cases when controlling APIs as opposed to hosts
![Page 48: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/48.jpg)
48
API Availability
• es_instance_conn_validator doesn’t resolve until a connection can be made
![Page 49: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/49.jpg)
Some observations after a couple years…
![Page 50: PuppetConf 2017: Custom Types & Providers: Modeling Modern REST Interfaces and Beyond- Tyler Langlois, Elastic](https://reader033.fdocuments.in/reader033/viewer/2022052706/5a64dabd7f8b9ac21c8b6edf/html5/thumbnails/50.jpg)
50
Results From the Field
• One parent class makes creating more easy
• Supported REST-based resources include:
• indices • templates • pipelines • + more
Extensibility
• rspec + webmock for great testing
• ES docs + specs first have made some implementations first try successes
• Good mocks make some acceptance tests unnecessary (faster CI!)
Reliability
• Much easier to extend to new OS’s (i.e., Windows)
• Greater control has made some tasks (like 3.x → 4.x module update) smooth
+ more