Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime!...

44
Immutable Infrastructure Rethinking Configuration in the Age of Easy Redeployment RackN, Inc February, 2018 Note: Graphics mainly from http://pexels.com <<< Shift Left <<<

Transcript of Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime!...

Page 1: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

Immutable Infrastructure

Rethinking Configuration in the Age of Easy Redeployment

RackN, Inc

February, 2018Note: Graphics mainly from http://pexels.com

<<< Shift Left <<<

Page 2: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

2

@zehicle #immutable

Involved in Open Ops Software:

Digital Rebar Project

Kubernetes ClusterOps SIG

OpenStack Board

Your Humble Presenter

I’m all about automating infrastructure.

Rob Hirschfeld (aka @zehicle)

Co-Founder of RackN

[email protected]

Page 3: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

3

@zehicle #immutable

Storytime! “Self-Bootstrapping Kubernetes”

Kubecon in Nov 2017 we created this demoSimple “immutable” Idea:

1) In Memory Boot Machines2) Install Docker3) Elect Leader4) Run Kubeadm on Leader5) Run Kubeadm on Remainder

But….it’s shockingly hard to maintain.

Dependencies breaks the installation

And they are constantly changing.

Page 4: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

4

@zehicle #immutable

Storytime! “Self-Bootstrapping Kubernetes”

So, while it’s pretty cool,

it’s not “real” immutability

Presentation & Demo

https://youtu.be/OowxF6GqK4I

sa !

Page 5: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

Why is configuration fragile?

Page 6: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

Why is configuration fragile?mu on

V

Page 7: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

7

@zehicle #immutable

But… I Infrastructure as Code?!Sorry. Mutability adds complexity

Traditional “build-in place” approaches

● Have hidden dependency graphs● Create variation between environments● Are harder to “lock down” due to config

AND OMG… updates and patches are even harder● Idempotent operations are difficult● Roll backward is next to impossible!● Creating indeterminate state

Page 8: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

8

@zehicle #immutable

Traditional “build-in place” approaches

● Have hidden dependency graphs● Create variation between environments● Are harder to “lock down” due to config

AND OMG… updates and patches are even harder● Idempotent operations are difficult● Roll backward is impossible● Creating indeterminate state

But… I Infrastructure as code?!Sorry. Mutability adds complexity

Let’s o k w !

Page 9: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

What is Immutable Infrastructure?

Page 10: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

What is Immutable Infrastructure?Pre-de y fi dV

Page 11: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

11

@zehicle #immutable

Traditional Deploy and Configure

System is configured in situ from a least common denominator baseline.

This can be “immutable-like” under the right conditions.

We’ll come back to that...

Delivery Pipeline

Dep

loym

ent

Co

de

Bu

ild

Inte

grat

e

Ru

n

Co

nfi

gure

Page 12: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

12

@zehicle #immutable

Shifting Configuration BEFORE Deployment

In our ideal delivery pipeline,

configuration is before deployment.

Running systems are delivered as a complete runnable unit for deployment.

Delivery Pipeline

Dep

loym

ent

Co

de

Bu

ild

Inte

grat

e

Ru

n

Co

nfi

gure

Page 13: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

13

@zehicle #immutable

Shifting Configuration BEFORE Deployment

In reality, it’s very hard to create a distinct artifact for every running instance; instead, we create incremental versions.

So we do some initialization of the reusable versioned instance.

Cloud init is the most commonly known pattern for this.

Delivery Pipeline

Dep

loym

ent

Co

de

Bu

ild

Inte

grat

e

Ru

n

Co

nfi

gure

Ini iz !

V

Page 14: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

Cloud NativeInfrastructureCNIbook.infoJustin Garrison & Kris Nova

“Infrastructure as software”

Page 15: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

15

@zehicle #immutable

Which Enables… Delegating Operations

If you can make your artifacts immutable then you can delegate management of them to a platform like Kubernetes.

Kubernetes does not configure infrastructure. It maintains state based on a manifest. St

ate

Man

ager

(e.g

. Ku

ber

net

es)

Co

de

Bu

ild

Inte

grat

e

Ru

n

Co

nfi

gure

Delivery Pipeline

Page 16: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

16

@zehicle #immutable

Which Enables… Delegating Operations

If you can make your artifacts immutable then you can delegate management of them to a platform like Kubernetes.

Kubernetes does not configure infrastructure. It maintains state based on a manifest. St

ate

Man

ager

(e.g

. Ku

ber

net

es)

Co

de

Bu

ild

Inte

grat

e

Ru

n

Co

nfi

gure

Delivery Pipeline

Kub te ? W !

Is ut > K8s?

Page 17: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

Immutable is a DevOps Pattern

<<< Shift Left & Create/Delete

Page 18: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

18

@zehicle #immutable

The Problem

Immutability <<< Shifting Left

package server image

provision server

initial config

Page 19: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

19

@zehicle #immutable

The Problem

Immutability <<< Shifting Left

patch 1package server image

provision server

initial config

Page 20: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

20

@zehicle #immutable

The Problem

Immutability <<< Shifting Left

patch 1 patch 2package server image

provision server

initial config

Page 21: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

21

@zehicle #immutable

The Problem

Immutability <<< Shifting Left

patch 1 patch 2

t e d s o n't

s o t c 2!

package server image

provision server

initial config

Page 22: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

22

@zehicle #immutable

The Problem

Immutability <<< Shifting Left

patch 1 patch 2

t e d s o n't

s o t c 2!What Madness?

● We have to maintain root access● Patches assume system state● Patches create dependency graphs● Coordination? Should we halt work?● Drift is inevitable!

package server image

provision server

initial config

Page 23: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

23

@zehicle #immutable

The Problem

Immutability <<< Shifting Left

patch 1 patch 2SA !!package server image

provision server

initial config

Page 24: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

24

@zehicle #immutable

Apply cloud and container lessons to our Bare Metal …

Immutability <<< Shifting Left

package server image

provision server

initial config

destroy!!

Page 25: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

25

@zehicle #immutable

Apply cloud and container lessons to our Bare Metal …

Immutability <<< Shifting Left

destroy!!

destroy!!patch 1

package server image

provision server

initial config

package server image

provision server

initial config

Page 26: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

26

@zehicle #immutable

Apply cloud and container lessons to our Bare Metal …

Immutability <<< Shifting Left

destroy!!

destroy!!patch 1

destroy!!patch 2

package server image

provision server

initial config

package server image

provision server

initial config

package server image

provision server

initial config

Page 27: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

27

@zehicle #immutable

Apply cloud and container lessons to our Bare Metal …

Immutability <<< Shifting Left

destroy!!

destroy!!patch 1

patch N

destroy!!patch 2

package server image

provision server

initial config

package server image

provision server

initial config

package server image

provision server

initial config

package server image

provision server

initial config

Page 28: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

Cloud like behavior …

Page 29: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

29

Immutable Provisioning systems treat infrastructure as a black box

Cloud-like Integration and Staged Workflow

ProvisioningSystem

Requested State

Returned State

REST API

EventHook

Page 30: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

30

Cloud-like Integration and Staged Workflow

Immutable Provisioning systems treat infrastructure as a black box

Provision requests are for a system state with optional parameters.

The intermediate changes to achieve the state are not exposed to the requester.

ProvisioningSystem

Reset Join

Inst

all

Co

nfi

g

Tes

t

Requested State

Returned State

REST API

EventHook

Page 31: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

31

Cloud-like Integration and Staged Workflow

Immutable Provisioning systems treat infrastructure as a black box

Provision requests are for a system state with optional parameters.

The intermediate changes to achieve the state are not exposed to the requester.

REMEMBER: Operators of the provisioning system require high transparency, stages and control.

ProvisioningSystem

Reset Join

Inst

all

Co

nfi

g

Tes

t

Requested State

Returned State

REST API

EventHook No h n

op i n !

Page 32: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

Immutable Patterns1) Baseline + Configuration2) Live Boot + Configuration3) Image Deploy

Page 33: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

33

Provision

1: Baseline + Configuration

Benefit: Easiest to achieve with current tools, Safer than Patching

Challenge: Lots of Post-Configuration, Not Really “Immutable”, Slow

Instead of relying on patches, rely on starting from a pristine image

ResetBaseline Configure Run

Additional Reference https://thenewstack.io/immutable-hardware-ops-hygiene-security-efficiency/

Page 34: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

34

Benefit: Fast reset times, forces good behavior

Challenge: Provisioning becomes critical path, still have dependency graph

Like #1 but clean-up is simply a reboot. Favors smaller footprint O/S.

2: Live Boot + Configuration

Provision RebootBaseline Configure Run

Page 35: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

35

3: Image Deploy

Benefit: Shorter time to ready, highly controlled (“shift left”), rollback

Challenge: Harder to create and deploy images

Image is deployed from source instead of Baseline + Configure

ProvisionDeploy Image

Run ProvisionDeploy Image

Run

Page 36: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

36

3: Image Deploy

Benefit: Shorter time to ready, highly controlled (“shift left”), rollback

Challenge: Harder to create and deploy images

Image is deployed from source instead of Baseline + Configure

ProvisionDeploy Image

Run ProvisionDeploy Image

Run

Ini iz !

V

Ini iz !

V

Page 37: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

37

So… Let’s talk Image Creation

Ideally in an automation build process.

You DO THE CONFIGURATION on a live system (so you still need configuration tools) and then capture the image into a portable format.

Tools like Hashicorp Packer, Image Builder, WBIC or raw images are used to create source files (e.g. AMI, OVS).

Page 38: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

38

So… Let’s talk Image Creation

Ideally in an automation build process.

You DO THE CONFIGURATION on a live system (so you still need configuration tools) and then capture the image into a portable format.

Tools like Hasicorp Packer, Image Builder, WBIC or raw images are used to create source files (e.g. AMI, OVS).

Tha n li ot

of k & re y ow!

Page 39: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

Yes, But…

It’s faster, safer & more scalable.

Page 40: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

40

Build Pipeline

Immutable Demo

Prep: Image is pre-created from reference system.

ReferenceSystem

Image

Read

Page 41: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

41

Build Pipeline Deploy

Immutable Demo

Prep: Image is pre-created from reference system.

Stage: Boot RAM image and write image to disk(s)

ReferenceSystem

Image

Target SystemRAM BOOT

WriteRead

Page 42: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

42

Build Pipeline Deploy

Immutable Demo

Prep: Image is pre-created from reference system.

Stage: Boot RAM image and write image to disk(s)

Deploy: Reboot and run

ReferenceSystem

Image

Target SystemRAM BOOT

Target SystemRUNNING

Reb

oo

t

Write RunRead

Page 43: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

43

Build Pipeline Deploy

Immutable Demo

Prep: Image is pre-created from reference system.

Stage: Boot RAM image and write image to disk(s)

Deploy: Reboot and run

ReferenceSystem

Image

Target SystemRAM BOOT

Target SystemRUNNING

Reb

oo

t

Write RunReadIni iz !

V

Page 44: Immutable Infrastructure - USENIX · 2019-12-18 · 3 @zehicle #immutable Storytime! “Self-Bootstrapping Kubernetes” Kubecon in Nov 2017 we created this demo Simple “immutable”

Thank you!Questions?

Interested in IMMUTABLE METAL?

It’s complicated, but we can get you there.

Start at http://portal.rackn.io

• Quickstart takes about 30 minutes

• Use your own hardware, VirtualBox or Packet.net account

– use “RACKN100” on Packet.net for credit