StackEngine Demo - Boston

Managing Docker Applications in Production

Boyd E. Hemphill - @behemphi Mike Raab - @stackengine

Background

Oracle Developer, MySQL DBA, PHP Developer, System Architect, DevOps Director (snickering encouraged), Evangelist

Background

Oracle Developer, MySQL DBA, PHP Developer, System Architect, DevOps Director (snickering encouraged)

Started Austin DevOps In 2012

Background



Co-Organizer of DevOps Days Austin

Background




Became involved in Docker Austin in August.

BackgroundOracle Developer, MySQL DBA, PHP Developer, System Architect, DevOps Director (snickering encouraged)



Became involved in Docker Austin in August.

Launching the First Ever Container Days in Austin - http://bit.ly/1CBppRV

http://bit.ly/1CBppRV

History -StackEngine

Started in May 2014 to address challenges facing Docker adopters



Funding announcement October



Funding announcement October

http:/stackengine.com/product/

http://stackengine.com/product/

Common Ground

Common Ground Business

What is The Goal of Your

Company?

What is The Goal of Your

Company?Make Money!

So … What is DevOps?

Common Ground Technology

Is DevOps a Process?

Is it an intersection of overlapping concerns?

Is DevOps a Culture?

DevOps:

DevOps is the way in which a technology organization embeds itself in a business to the benefit of that business.

The Ideal Containerized Application

NetflixService Oriented Architecture

UnicornHuge, scary, difficult to reason about

Pantheon A Grokkable Unicorn

Article on Container Use: http://bit.ly/15a2UIW

http://bit.ly/15a2UIW



Note the date is October 2013





Presentation Containers v. VMs: http://slidesha.re/1kMsSY9


http://slidesha.re/1kMsSY9




Presentation Containers v. VMs: http://slidesha.re/1kMsSY9

Video: http://bit.ly/1L3jjA5


http://slidesha.re/1kMsSY9

http://bit.ly/1L3jjA5

Benefits

Benefits

Feature velocity - Increase developer productivity using containers to model application topology

Benefits


Innovation velocity - Decreased build and test times make more time for trying that next great idea!

BenefitsFeature velocity - Increase developer productivity using containers to model application topology


Density - Reduce Costs

Benefits



Density - Reduce Costs

Separation of Concerns - “Code to an interface” moves to the infrastructure level

Benefits

Taken together a company that can get to the end state has a significant and lasting competitive advantage

Today’s Docker Reality

Concerns

Concerns

This is still very early days for Docker

Concerns


It is even earlier for Rocket

Concerns



Tools to support containerized applications are immature or do not exist

Concerns




Service oriented architecture is hard

ConcernsThis is still very early days for Docker



Service oriented architecture is hard

Increasing developer throughput is only a local optima, thus a false economy

Concerns

Docker adoption is the first major DevOps initiative not driven by Ops.

ConcernsDocker adoption is the first major DevOps initiative not driven by Ops.

Laundry Ops: http://bit.ly/1E2gMUS

http://bit.ly/1E2gMUS

Software Delivery Life Cycle

SDLC

Developers are moving forward

SDLC


QA sees production modeling, large test grids, more effective integration testing

SDLC


QA sees production modeling, large test grids, more effective integration testing

Build engineers see parallelized build grids for faster feedback and better build quality

xkcd … er … SDLCOperations panics - no tools or instrumentation

SysAdminDon’t want this guy panicking

Information Architecture

Problem Description


Problem Description

StackEngine’s work on that problem today


Problem Description

StackEngine’s work on that problem today

StackEngine’s planned work on that problem for the future

Problem #1 The Final Mile

The Final Mile

Developer uses containers

The Final Mile


System is built and images are produced

The Final Mile



Images stored in a private registry

The Final Mile



Images stored in a private registry

Now we need to deploy to a QA environment

The Final Mile

So we build a manual solution depending on human behavior

The Final Mile

So we build a manual solution depending on human behavior

Someone builds a tool

Crossing the Final Mile - Today

A Simple Solution to Deployment

Deploy My Stuff - TodayCreate a component to represent your registry

Deploy My Stuff - TodayCreate a component to represent the image you would like to run

Deploy My Stuff - TodayDistribute that image and run it on the hosts in your Mesh.

Deploy My Stuff - TodayMagic happens.

Deploy My Stuff - TodayLife is good

Crossing the Final Mile - Future

Heterogeneous, Policy Driven Deployment

Deploy My Stuff - Future

Support other container types

Deploy My Stuff - Future

Support other container types

Policy Based Deployment

Deploy My Stuff - FutureWe should put another bunny cloud over there …

Problem #2 Container Sprawl

Ants not Cattle

First there was VMware and suddenly a large corps departments could easily spin up environments

Ants not Cattle

First there was VMware and suddenly a large corps departments could easily spin up environments

Then there was the cloud and developers could completely circumvent ops

Ants not CattleFirst there was VMware and suddenly a large corps departments could easily spin up environments

Then there was the cloud and developers could complete circumvent Ops

Now there is Docker and Developers are creating numerous abstractions that are not managed like traditional VMs

Ants not CattleFirst there was VMware and suddenly a large corps departments could easily spin up environments

Then there was the cloud and developers could complete circumvent Ops

Now there is Docker and Developers are creating numerous abstractions that are not managed like traditional VMs

Rocket potentially segments the Market and means different performance, security, runtime and management characteristics

Container Sprawl

Containers, Dockerfiles, and Images on VMs

Container Sprawl


VMs on physical hardware located in availability zones

Container Sprawl


VMs on physical hardware hardware located in availability zones

AZs in data centers in geographic regions.

My Head HurtsContainer Sprawl

Component Inventory - TodayA drillable and searchable view of

infrastructure

Component Inventory - Today

Hosts


Images


Containers


Filters and Build Actions


Search

Component Inventory - Future

Geographically aware physical topology


Geographically drillable physical topologies


Geographically drillable physical topologies

Policy based inventory levels (pull not push)


Hey, why are there two purple stripes?

Problem #3 Telemetry

Telemetry

What does monitoring a containerized app look like?

Telemetry


http://goo.gl/VkAQo4


Telemetry



What does log aggregation look like?

Telemetry




http://goo.gl/uNPe9P


TelemetryWhat does monitoring a containerized app look like?




Good telemetry (monitoring and tests) eliminates the need for discussion about the content of an image!


Decision Dimensions - TodayHost level metrics to help you manage

running containers.

What’s my stuff doing - Today

I have this much stuff.


My hosts are not yet stuffed.


The stuffing on a host is as expected.


A bit of the stuffing is doing what it should.


What is the stuff in this image?


How do I visualize the stuff in an image?

YOU DID WHAT? WHEN?

Stuff is broken? What just changed?

Enable the Important Conversation

What were you thinking??!!

Decision Dimensions - Future

There is never too much information about run time.

Component Performance - Future

Improve Host Metrics



Improve Image Metrics



Improve Image Metrics

Add Container Metrics


Ahh … look … the bunny on the left is happy!

Problem #4 Centralized Control Plane

Centralized Control Plane

The ability to act upon a system from a single point of entry



Actions should happen next to the information causing the action




Actions should happen within the context of inventory




Actions should happen within the context of inventory

UI, API, CLI

Actions - TodayPreventative, automatic, deterministic

Make Stuff do Stuff - Today

Manage Images

Make Stuff do Stuff - Today

Manage Containers

Actions - FutureInitialization, Taxonomy, Corrective

Make Stuff do Stuff - Future

Policy based actions by the Controller


Policy based actions by the Controller

Powerful Control Plane Taxonomy (Tagging)


Blow softly and make the bunnies kiss.

Problem #5 Heterogeneous Service

Providers

Heterogeneous Service Providers

Containers live on services (IaaS, PaaS)

Heterogeneous Service Providers

Containers live on services (IaaS, PaaS)

Images come from different places (Docker Hub, Quay.io)

Heterogeneous Service ProvidersContainers live on services (IaaS, PaaS)


Within an IaaS service provider are other services (Databases, load balancers, queues)

Heterogeneous Service ProvidersContainers live on services (IaaS, PaaS)


Within an IaaS service provider are other services (Databases, load balancers, queues)

Persistent Data stores present their own unique problem as a service

Components - Today

Abstractions for Control Plane Operation

Building Apps with My StuffComponents are infrastructure service abstractions

Building Apps with My StuffComponents can be arranged in logical ways to represent

physical infrastructure.

Building Apps with My Stuff

Start the db


Start the web server


Run the application server


Run the Cache

Components - Future

More and varied components

Building Apps with My Stuff - Future

More component types is like having more varied Lego bricks

Building Apps with My Stuff - Future

More component types is like having more varied Lego bricks

Pluggable component architecture

Building Apps with My StuffWhat else could we make with bunnies and rainbows?

Roadmap

Problem #6 Service Discovery

Service Discovery

Allows multiple deployments of the same service on the same host.

Service Discovery


Containers register themselves upon startup and are found by tags rather than URI’s, fixed ports, etc.

Service Discovery



We favor something like consul with a search interface like the chef server.

Service DiscoveryAllows multiple deployments of the same service on the same host.



A step towards cattle containers instead of pet containers

Service DiscoveryAllows multiple deployments of the same service on the same host.



A step towards cattle containers instead of pet containers

Ultimate goal is “ants not cattle”

Roadmap

User Management

Roadmap

User Management

Addition of new Components

Roadmap

User Management


Better telemetry

Roadmap

User Management


Better telemetry

Scheduling

Roadmap

User Management


Better telemetry

Scheduling

A/B Deployment support

StackEngine is the StuffIt’s easier to dream with StackEngine.

Live Product DemoAl Gore and his gremlins willing

Boyd E. Hemphill - @behemphi Mike Raab - @stackengine

Thank You for Your Time and Comments.

StackEngine Demo - Boston

Technology

Transcript of StackEngine Demo - Boston