Facilitating Continuous Delivery at Scale
Dan Woods
/danveloper
Senior Software Engineer: Delivery Engineering
Learning Ratpack
What is this talk about?
• Continuous Delivery is tremendously important for unprecedented scale
!• The “Rube Goldberg” machine of a microservice
architecture needs to keep moving !• The culture needs to be amenable to continuous delivery !• Understand that continuous delivery is an organic
concept within an organization
Culture
Freedom and Responsibility!!• One of the greatest aspects of the culture !• Means that if something isn’t working for you, you are
empowered to change it !• We can’t think about things in traditional Ops (or even
DevOps) terms
Culture
Supporting Teams!!• Need to really understand what the various problem
domains are throughout the org !• Try to design for the overlap in use-cases !• From a tooling standpoint, we need to have versatile
infrastructure that can adapt
Facilitation
Facilitating Continuous Delivery!!• Can’t be locked in to a single path or way of doing things !• Need to be able to pivot as quickly as the product teams !• Recent emergence of new technology stacks has made
this ever-apparent
Facilitation
Integrating with Netflix!!• Netflix has a large insight infrastructure, built on JVM
technologies !• Solutions for the various problem domains might not fit
into that structure !• Need to have an answer for getting these apps
integrated into the platform
Facilitation
Platform Sidecar!!• Decouples the application from the platform !• Provides a standard RESTful API for utilizing the platform !• Comes with some reduced insight, but also some
valuable functionality !• Very easy internally to build this into an application’s
deployable artifact
Facilitation
Platform Sidecar!!!!!!
https://github.com/netflix/prana!!
!!!!
Facilitation
This is about enabling the “Freedom and Responsibility” culture by having answers that are not
“no”.
Facilitation
How Does This Fit Into Continuous Delivery?!!• Continuous Delivery is the full show — “code to cloud” !• Having decoupled tooling available means that we can
enable teams to move quickly !• They can be more productive, push code more rapidly !• Less time spent on “integration”, more time spent
developing products
Facilitation
Designing Toward Extensibility!!• No one-size fits all for Continuous Delivery !• Designing workflows based around configuration !• Version all the things! !• Have “best practices”, not shoehorns
Facilitation
Being flexible and driving Continuous Delivery through configuration means that teams can iterate more
rapidly.
Facilitation
Designing for Availability and Durability!!• Infrastructure supporting CD pipelines needs to be highly
available !• Need to be durable to outages — if we’re having an
outage, a production team certainly is as well !
• Design for failure and hope for the best; continuous delivery is not limited to a zone or region, need recovery built-in
Facilitation
Designing for Availability and Durability!!• Getting into the realm now of self-healing systems !• Some teams at Netflix already have this capability !• We’re designing the tooling to be flexible enough to
facilitate most teams
Facilitation
Designing for Availability and Durability!!• Active/Active deployment model !• Non-core functionality can be gracefully degraded out of
the stack !• Ensure that continuous delivery tooling is always
available; this is always the first stop in the event of an outage
Facilitation
Designing for Availability and Durability!!• Netflix Continuous Delivery tooling of the future is a
distributed platform !• Making use of service discovery makes use more resilient
than ever !• For availability purposes, we can leverage Eureka for
load balancing and routing to send requests to the right region/zone/service combination
Facilitation
Build Extensibility - Immutable Infrastructure!!• The “Golden Nugget” of deployable artifact !• Has been advocated since forever, often ruled out !• Netflix has created extensive tooling around designing
and building immutable infrastructure !
• This is one of the only requirements to run in the Netflix cloud (it is the only way to get there)
Facilitation
Immutable Infrastructure - Resources:!!
http://nebula-plugins.github.io/!!
https://github.com/netflix/aminator!
Facilitation
Other Extensible Aspects!!• Extensibility in the build system: Gradle/Nebula !• Extensibility in the configuration model: Archaius !• Extensibility in the data store: Cassandra - Astyanax;
Redis - Dynomite !• Extensibility in the deployment systems: “designing”
continuous delivery pipelines !
Facilitation
Documentation as a First Class Citizen!!• Often the first thing sacrificed when moving quickly !• Is of paramount importance in a distributed platform !• Need to make it appealing for developers to write
documentation !• Need to make it easy for consumers to find/parse your
documentation !
Facilitation
Documentation as a First Class Citizen!!• Build the documentation with the code (literally make it a
module of the project) !• We’ve landed on using Asciidoctor for documenting all of
our processes and APIs !• Generate a manual to HTML, which can be served
directly inline with the service (keeping the documentation as close to the code as possible)
!• Avoid auto-generated docs!! !
Facilitation
Rules for Good Documentation!!• Be sure to include the context for what you’re doing !• Speak at a layman’s level !• Always approach documentation as though the reader
has no knowledge of what you’re doing !• Remember that terminology may need to be described
as well !
Facilitation
DONT DO THIS
Facilitation
DONT DO THIS
Facilitation
YES! DO THIS!
Facilitation
We’re Hiring.
Top Related