Patrick Chanezon

@chanezon, Docker Inc.

Docker from development to production

London, October 2016

French

Polyglot

Platforms

Software Plumber

San Francisco

Developer Relations

@chanezon

1995 2015

PublicHybridPrivate

Ops Devops Developers

Linux Container Ecosystem

flockerglusterfs

weavecalicomidokuracisconuage

Cloud

OS

Plugins

Orchestration

Agility

Agile methodologies (circa 1999)

Low MTBIAMSH

MTBIAMSH (Mean Time Between Idea And Making Stuff Happen)

Agility == $$

Devops

25

Mainframe

Client-Server

26

27

Web

28

Cloud - Devops

Devops

• Cultural movement

• Inspired by agile methods

• People, Processes & Tools

• Continuous delivery

• Infrastructure as code

• Cross silo collaboration

• Small iterations

• Feedback loop, measurement

Image from Patrick Debois

http://www.slideshare.net/jedi4ever/devops-the-war-is-over-if-you-want-it

http://www.slideshare.net/jedi4ever/devopsdays-downundervfinal

Devops: singing Kumbaya?

28

http://highscalability.com/blog/2013/11/19/we-finally-cracked-the-10k-problem-this-time-for-managing-se.html

Server/Sysadmin

1999: 5(Windows) - 50 (Linux)

2015: 10k-20k

x2000

28

https://blog.docker.com/2014/12/dockercon-europe-keynote-continuous-delivery-in-the-enterprise-by-henk-kolk-ing/

Henk Kolk, ING, DockerCon EU 2014

People, Processes, Products

deployment time: 9 months -> 15 min1500 deployments/week

Docker Platform

The world needs

tools of mass innovation

A programmable Internet would be the ultimate

tool of mass innovation

A commercial product,

built on

a development platform,

built on

infrastructure,

built on

standards.

Docker is building a stack to program the Internet

Isolation using Linux kernel features

namespaces

pid

mnt

net

uts

ipc

user

cgroups

memory

cpu

blkio

devices

Image layers

What’s New?

1.

Developer experience

1. Get out of the way

The best tools…

2. Adapt to you

3. Make the

powerful simple

Docker for Mac Docker for Windows

2.

Orchestration

Introducing the best way to orchestrate Docker: Docker.

Docker 1.12: now with orchestration built-in.

Swarm mode

Service API

Cryptographic node identity

Built-in routing mesh

Docker 1.12: now with orchestration built-in.

Using the beta? You already have 1.12 installed.

> docker swarm init

> docker service create

3.

Ops experience

Deep integration with native load-balancers, templates,

SSH keys, ACLs, scaling groups, firewall rules…

beta.docker.com

Distributed Application Bundle

www.docker.com/dab

A portable format for multi-container applications

Docker CaaS

Goals

+ +

Agility Portability Control

Cloud Zone 1

Cloud Zone 2 Data Center

Development

Center

Headquarters

Containerization: standard containers

on a standardized container engine

Orchestration: build and deploy

complex systems easily

Enterprise: Enable delivery and time

to value across a large, complex,

rapidly evolving enterprise

The Global Enterprise Supply Chain

BUILDDevelopment Environments

SHIPRegistry: Secure Content &

Collaboration

RUNControl Plane: Deploy,

Orchestrate, Manage, Scale

Networking Volumes MonitoringLoggingConfig MgtCI/CD

IT Operations

Developers IT Operations

Docker CaaS Workflow

Docker Containers as a Service platform

44

BUILDDeveloper Workflows

SHIPRegistry Services

RUNManagement

Docker for Mac and Windows Docker Trusted Registry Docker Universal Control Plane

Docker Cloud

Docker Container Engine

Ecosystem Plugins and Integrations

UCP Permission Model

Docker UCP 1.1 - DTR 2.0• HA

• Unified Auth

• Compose deployment

• UI to add nodes

Security scanning in Docker Cloud

Plumbing

20

13

-0

5

20

13

-0

6

20

13

-0

7

20

13

-0

8

20

13

-0

9

20

13

-1

0

20

13

-1

1

20

13

-1

2

20

14

-0

1

1,000,000

0

20

14

-0

2

20

14

-0

3

20

14

-0

4

20

14

-0

5

20

14

-0

6

20

14

-0

7

20

14

-0

8

20

14

-0

9

20

14

-1

0

20

14

-1

1

20

14

-1

2

20

15

-0

1

20

15

-0

2

20

15

-0

3

20

15

-0

4

20

15

-0

5

20

15

-0

6

20

15

-0

7

20

15

-0

8

20

15

-0

9

20

15

-1

0

20

15

-1

1

20

15

-1

2

20

16

-

01

1,000,000,000

~

10,000,000

9,000,000

8,000,000

7,000,000

6,000,000

5,000,000

4,000,000

3,000,000

2,000,000

6,000,000,000

5,750,000,000

5,500,000,000

5,250,000,000

5,000,000,000

4,750,000,000

4.500,000,000

4,250,000,000

4,000,000,000

3,750,000,000

3,500,000,000

3,250,000,000

3,000,000,000

2,750,000,000

2,500,000,000

2,250,000,000

2,000,000,000

1,750,000,000

1,500,000,000

1,250,000,000

20

13

-0

5

20

13

-0

6

20

13

-0

7

20

13

-0

8

20

13

-0

9

20

13

-1

0

20

13

-1

1

20

13

-1

2

20

14

-0

1

1,000,000

0

20

14

-0

2

20

14

-0

3

20

14

-0

4

20

14

-0

5

20

14

-0

6

20

14

-0

7

20

14

-0

8

20

14

-0

9

20

14

-1

0

20

14

-1

1

20

14

-1

2

20

15

-0

1

20

15

-0

2

20

15

-0

3

20

15

-0

4

20

15

-0

5

20

15

-0

6

20

15

-0

7

20

15

-0

8

20

15

-0

9

20

15

-1

0

20

15

-1

1

20

15

-1

2

20

16

-

01

~

20

16

-0

9

1,000,000,000

~

10,000,000

9,000,000

8,000,000

7,000,000

6,000,000

5,000,000

4,000,000

3,000,000

2,000,000

6,000,000,000

5,750,000,000

5,500,000,000

5,250,000,000

5,000,000,000

4,750,000,000

4.500,000,000

4,250,000,000

4,000,000,000

3,750,000,000

3,500,000,000

3,250,000,000

3,000,000,000

2,750,000,000

2,500,000,000

2,250,000,000

2,000,000,000

1,750,000,000

1,500,000,000

1,250,000,000

Notary

runC •

containerd •

HyperKit , VPNKit, DataKit •

SwarmKit •

libcontainer •

libnetwork • • Docker 1.8 : Docker Content Trust

• Docker for Mac

Docker for Windows

• Docker 1.12

with built-in

orchestration

• Docker 0.9 : Pluggable execution

• Docker 1.7 : Multi-Host Networking

• Docker 1.11:

OCI support

Notary

“Let’s stop using curl|sh”

Trusted collections for any content

Transport-agnostic

Reliable updates, proof of origin, resistant to untrusted

transport, survivable key compromise

Build on industry-leading standards and research

RunCThe universal container runtime

https://runc.io

containerdA daemon to control runC

built for performance and density

http://containerd.tools/

containerd

Docker 1.11

Docker for Mac architecture

(simplified)

Hypervisor Framework

vmnet Framework

Docker Container Engine

Hypervisor

Linux

VPN

Data Service

Interface

Client Libraries

Admin GUI

CLI

Security Sandbox

Docker for Mac internals

Unikernels

http://unikernel.org/

Hypervisor Framework

vmnet Framework

Docker Container Engine

Hyperkit

Linux

VPNKit

DataKit

Client Libraries

Admin GUI

CLI

Security Sandbox

Improving Docker with unikernel tech

InfraKit

Problem:

Managing Docker on different infrastructure isdifficult and not portable.

Consistent User Experience

62

How do we handle updates to a cluster??

Docker for AWS

EBS ELB

Container Engine

Storage plugin

Infrastructure Management

Network plugin Orchestration

IAM

CloudFormation

EC2VPC

Admin interface

Linux

User Applications / Services

Docker for AWS

EBS ELB

Container Engine

Storage plugin

InfraKit

Network plugin Orchestration

IAM

CloudFormation

EC2VPC

Admin interface

Linux

User Applications / Services

InfraKit

A toolkit for building declarative, self-healing infrastructure.

Declarative

• JSON configuration for desired infrastructure state:

• Specification of instances — vm image, instance type, etc.

• Group properties — size, logical identifiers, etc.

• Design patterns encourage

• encapsulation

• composition

• Config is input to all operations — system figures out what to do66

Self-healing

• Composed of a set of active components / processes that

• monitor infrastructure state

• detect state divergence

• take actions

• Continuous monitoring and reconciliation — always on

• No downtime — rolling update

67

Toolkit

• Primitives for managing collections of resources

• create, scale, destroy

• rolling update

• Abstractions & Developer SPI

• Group - manages collection of resources

• Instance - describes the physical resource

• Flavor - extra semantics for handling instances

• A collection of executable, active components — plugins

• Initially, Go daemons in the toolkit

• Soon, easy management via Docker Plugins (runc)

Architecture

Instance Plugin

• Spec: specification / model of an instance (e.g. vagrant, EC2):

• Logical ID, Init, Tags, and attachment

• Platform-specific properties

• Methods:

• /Instance.Validate

• /Instance.Provision

• /Instance.Destroy

• /Instance.DescribeInstances

• Examples: instance plugins for EC2, Azure VM, Vagrant, … 70

Flavor Plugin

• Gives more context about the group members:

• Size, or list of Logical ID’s (e.g. IP addresses for ‘pets’)

• Application-specific notions of ‘health’

Is the node not only present but also joined a swarm?

• Methods:

• /Flavor.Validate

• /Flavor.Prepare

• /Flavor.Healthy

• Examples: flavor for Zookeeper members, Docker swarm nodes71

Group Plugin

• Main entry point for user interaction:

• Create, describe update, update, destroy

• Config JSON is always the input

• Composed of Instance and Flavor — mix and match to

manage cattle (fungible) or pets (special)

• Methods:

• /Group.Watch

• /Group.Unwatch

• /Group.Inspect72

• /Group.DescribeUpdate

• /Group.Update

• /Group.StopUpdate

• /Group.Destroy

Configuration

Example config file (zk.conf): Group configuration = Instance + Flavor

{"Properties": {

/* raw configuration */

}}

{"groups" : {

"my_zookeeper_nodes" : {"Properties" : {

"Instance" : {"Plugin": "instance-vagrant","Properties": {

"Box": "bento/ubuntu-16.04"}

},"Flavor" : {

"Plugin": "flavor-zookeeper","Properties": {

"type": "member","IPs": ["192.168.1.200", "192.168.1.201", "192.168.1.202"]

}}

}}

}}

Operations

• Make sure the plugins are running:• infrakit/group &; infrakit/zookeeper &; infrakit/vagrant &;

• “Watch” the group starts management:• infrakit/cli group watch zk.conf

• Update the config, e.g. change size or add IP address• Describe changes before committing —

infrakit/cli group describe zk.conf

• Begin update —

infrakit/cli group update zk.conf74

Today

75

• InfraKit is just getting started… only

primitives for working with groups like

clusters of hosts

• But we have big plans

• Improve group management strategies

• More resource types — networking, load

balancers, storage…

• A cohesive framework for active

management of infrastructure — physical,

virtual, or containers

Get Involved

• Help define and implement new and interesting plugins

• Instance plugins for different infrastructure providers

• Flavor plugins for systems like etcd or mysql clusters

• Group controller plugins — metrics-driven auto scaling

and more

• Help define interfaces and implement new infrastructure

resource types — load balancers, networks and storage

volume provisioners76

More Info

• Github:

https://github.com/docker/infrakit

• A quick tutorial:

https://github.com/docker/infrakit/blob/master/docs/tutorial.m

d

77

Demo

THANK YOU