Nelsonautomated, multi-region container deployment

Hello.!

verizon.github.io/nelson/

Problem• Provisioning applications is still too slow (bare metal or cloud).

• Runtime traffic control systems are medieval at best.

• Coupling CI and CD creates monolithic operational systems.

• These systems do everything. This is a distinct problem.

• Current market solutions limited or hard to adopt.

• Most teams have brittle, painful automation nobody wants to use.

• Many teams attempt CD ignorant of the side-effects.

Lessons• Automate every part of the system.

• Testing a distributed system locally is a fable.

• Emergent properties. Scaling issues etc.

• Uniformity is highly desirable and wildly advantageous.

• Beautiful, unique snowflakes are however, inevitable.

• Automated lifecycle management is required.

Goals• Use the minimally powerful components.

• System elements should be awesome at just one thing.

• Reduce overall platform complexity.

• Increase responsibility of engineering teams. Break it, you bought it.

• Decentralize process gatekeepers.

• No build team. No ticket filing for deployments. No configuration management.

Goals• All application specifications are checked in.

• Build. Deployment. Alerting etc.

• Reduce deployment time 2 minutes or less.

• Support multi-DC topologies from the get-go.

• Automatic credential management and secure-introduction

• Transparent, strong encryption for application I/O on the wire.

Build Better.

Nomad.

Nomad• Use a farm of servers as a single resource pool: RAM, CPU, etc

• Typically used at larger scale, becoming more common.

• Blazing fast: only placement without provisioning.

• Integration with Vault, so secure-introduction works OOTB.

• Monolithic resource manager & scheduler [1]

• Several open-source & commercial alternatives: Mesos, k8s etc[1] https://research.google.com/pubs/pub43438.html

Envoy.

Envoy• Fast L4 and L7 proxy solving many practical ops concerns.

• Open-sourced end of 2016; blossomed since.

• Lyft, Google, IBM et al all actively contributing.

• Make applications dumb; invest in a single element of routing infra

• Retries, Circuit Breaking, TLS Encryption etc

• Integrate horizontally, not vertically

• Integrate with whatever discovery system you want via APIs.

Nelson.

– Vice Admiral Horatio Nelson, 1758-1805

“Desperate affairs require desperate remedies.”

– Vice Admiral Horatio Nelson, 1758-1805

“Desperate affairs require desperate remedies.”

#opslife

Overview• Github driven developer workflow (.com or enterprise).

• Choose whatever build / CI system you want.

• State of the art runtime routing via Envoy.

• Secure introduction for safe distribution of credentials from Vault.

• Integrated with Nomad; target any datacenter running a scheduler.

• Integrated alert definition with Prometheus.

Lifecycle.Deployment is the easy part.

based on consul

typical state

user activated

pending GC

pluggable

borrowed time

garbage collection

Graph Pruning

XX

Upgraded!

last two major revsXX X

last two featuresXX X

Namespaces.

machines

scheduler

namespaces

namespaces

entirely virtual!

root namespace

qa/unstable

qa/staging/tim

Discovery & Routing.

Discovery.• Discovery protocol written to Consul KV for every stack

• We call this Lighthouse protocol

• Application dependencies are declared a-priori.

• You cannot route to that which you do not tell Nelson about.

• Makes for awesome auditing and security.

• Language implementations need only consume the protocol.

Routing.• Non-prescriptive approach to routing tier implementation.

• Provides a control plane protocol describe routing actions.

• Typically implemented with Envoy, but you can choose.

• Minor application changes required.

• Incentivized these with tracing and context propagation.

• Models traffic shifting as a time vs traffic policy curve.

http://timperrett.com/2017/05/13/nomad-with-envoy-and-consul

embeded envoy

http://timperrett.com/2017/05/13/nomad-with-envoy-and-consul

sidecar envoy

http://timperrett.com/2017/05/13/nomad-with-envoy-and-consul

host-based envoy

Challenges• Non-trivial level of investment and execution.

• Tight integration with Hashistack is both pro or con.

• Containerizing legacy applications can be “interesting”.

• Migration can be a challenge if not collocated with “the new world”.

• Small organizations better served by existing solutions.

Future Work• Aim to open-source supporting and complimentary tools.

• Consul / Envoy integration. Cost analysis subsystem.

• Make Nelson easier to extend for third-parties

• eDSL for workflows, externalize policy algebra

• General “plugin” system is a possibility

• Listen to the community feedback.

Summary• Fully automated application lifecycle: no manual housekeeping.

• Choose whatever CI setup best fits your team.

• Secure your deployments.

• Transparent mTLS and rotating credentials.

• Automatic Vault policy management.

• Provide rigor to your application Death Star.

EOFtimperrett

verizon.github.io/nelson/