March 2017 Update:Will it blend? Joint OpenStack Kubernetes Environment

A pragmatic operational assessment of if and when Kubernetes can become an underlay for OpenStack.

TL;DR: Yesand then Kubernetes wins as the platform.

Rob Hirschfeld (aka Zehicle online)

In Community: OpenStack Board Member (4 years)

Co-Chair of Kubernetes Cluster Ops SIG

Founder of Digital Rebar & Crowbar Projects

Professional: CEO of RackN - hybrid automation software

Executive at Dell - scale data center ops

Cloud Data Center Ops going back to 1999

Addressing Operators Needs

Operational Success is Essential to Project Success

Operators are not developers!

Simple, Transparent and Stable are key concerns

Becoming a super-user of the platform should not be required to run it

Scale & Upgradability has both internal and external drivers

Generally, Kubernetes has good operational fundamentals

We’re Talking Underlay, not Overlay

We’re talking about installing Kubernetes first (aka underlay) and using it to manage the OpenStack control plane.

Objectives for Kubernetes Underlay:

● Must Work with Kubernetes Primatives● Not a Dedicated Kubernetes● Limited Outside Management

Physical Infrastructure

KubernetesUnderlay

OpenStack

KubernetesOverlay

This Talk

Simplest conception of the K8s OpenStack Sandwich

What is Kubernetes?

Container Scheduler (no, it’s not really Orchestration)

API driven to provide restart, placement, network routing and life-cycle

For Applications designed for Kubernetes

Key Design Elements: Immutable Infrastructure (stateless ops)

12 Factor Configuration

Service Oriented

What is Kubernetes: A Three Tier Application

Client

0Ready

1Prereq

2Control

3Nodes

etcd(cluster)etcd

(cluster)etcd

(cluster)

API(cluster)

API(cluster)

API(cluster)

Kubelet

KubeCtl

Container Manager

5Apps

Network CNI

HostNetwork

HostStorage

HostInit

Pod Pod Pod Pod

4Add-Ons

CertificateAuthority

Scheduler(leader)

Heapster

InfrastructureAPIs

Routers,Storage,

LBs...

Proxy

...

Controller(leader)

DNS Watcher ...

Together 4ever: API server + Kubelet

Client

0Ready

1Prereq

2Control

3Nodes

etcd(cluster)etcd

(cluster)etcd

(cluster)

API(cluster)

API(cluster)

API(cluster)

Kubelet

KubeCtl

Container Manager

5Apps

Network CNI

HostNetwork

HostStorage

HostInit

Pod Pod Pod Pod

4Add-Ons

CertificateAuthority

Scheduler(leader)

Heapster

InfrastructureAPIs

Routers,Storage,

LBs...

Proxy

...

Controller(leader)

DNS Watcher ...

Kubernetes = Rainbows?!

Why do we want Kubernetes as Underlay?

Community Perception Accuracy

1 OpenStack Operations is still not “solved” True (no change)

2 We already do most new deploys in containers True (was partially)

3 Kubernetes is awesome at containers True (was partially)

4 Kubernetes is simple, stable and secure (for operators) Partially (was false)

5 Kubernetes means easy Upgrades and High Availability Partially (was false)

There are REAL Potential Benefits

● Leverage Docker packaging efforts and reduce Python & O/S dependencies● Upgrades would benefit from Kubernetes built-in processes● Use of the Kubernetes job scheduler for maintenance● “Free” fault tolerance of key components● Easier install if Kubernetes already running on-site● More constrained options for configuration and operation

BUT REALLY, IT’S ABOUT COMMUNITY SIZE...

I expect more people will understand Kubernetes operations than OpenStack operations because Kubernetes is cloud and physical.

Kubernetes Underlay is coming,So let’s get pragmatic about it.

Leadership Kudos toSAP, ATT Comummity Dev, & Port Direct

Issue 1: Marketing Message is Confusing

Marketing around Kubernetes under OpenStack is a “hot mess”

● People hear “Kubernetes is stable, OpenStack is not”● Further confuses “OpenStack one platform message”● Encourages Kubernetes as target instead of OpenStack

Confusion with the Plain Old Container Install (“POCI”) message

● Canonical (Ubuntu Cloud Install), ● Rackspace (OpenStack Ansible)● Cisco (Kolla)● Triple O

Key Principle: Containerization vs Kubernetes

Containers can be treated as a) lightweight vms or 2) packaged daemon sets.

● Canonical builds their containers like persistent vms and configures with Juju● Kolla & OSA treats containers as packaging and configures with Ansible

Kubernetes accepts neither approach – they expect containers to be immutable and 12 factor configured

● Kubernetes manages the full container life-cycle● Containers need to be able to handle being added, removed ● Services need to be able to handle IP address changes (or use DNS names)

This work is progressing quickly!

Using Kubernetes v1.5+ Primatives

● Using Kubernetes Helm Charts● Services are tagged to nodes● Agents become Daemon sets● Databases using Stateful sets● Multiple container sources

Hard work remains….

● Networking, Configuraton & Storage● OpenStack Projects must handle immutable

container requirements

Physical Infrastructure

Kubernetes + HelmUnderlay

OpenStackMgmt

OpenStackNodes

OtherApps

KubernetesWorkers

More Detail: Kubernetes Underlay of OpenStack

Physical Infrastructure

Kub

erne

tes

Con

trolle

rs

OpenStackMgmt

OpenStackNodesD

ata

base

If you to really want to build this, give me a call - RackN has all the components

Msg

B

usSoftware Defined NetworkingCeph Distributed Storage

OtherWorkloads

Hel

m

Issue 2: Technical Challenges Remain

This discussion keep kicking the operations & install problems down the field

Kubernetes is much newer than OpenStack, so even less understood

Yet more complexity and some very basic questions:

● Now we have a both a Kubernetes and OpenStack upgrade problem● We still need tooling to manage OpenStack in Kubernetes● We still need someone to package the containers● Relies on Docker to keep systems running● Storage and Networking are still being worked out

In summary,

OpenStack operability is not solved via the underlay platform alone.

Technical Leadership motivation required for OpenStack adopting Kubernetes architecture requirements.

Serious messaging confusion in effort has to be resolved.

However, this collaboration is required for OpenStack

Because Kubernetes will have a larger footprint in Operations