Kubernetes Operators for VMware Enterprise PKS and VMware ...

#vmworld

CODE1360U

Kubernetes Operatorsfor VMware Enterprise PKS and VMware Cloud PKS

Tom Schwaller, VMware, Inc.Michael Gasch, VMware, Inc.

#CODE1360U

VMworld 2019 Content: Not for publication or distribution

©2019 VMware, Inc. #CODE1360U

Disclaimer

This presentation may contain product features or functionality that are currently under development.

This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for any new features/functionality/technology discussed or presented, have not been determined.

2

The information in this presentation is for informational purposes only and may not be incorporated into any contract. There is no commitment or obligation to deliver any items presented herein. VMworld 2019 Content: Not for publication or distribution

©2019 VMware, Inc. #CODE1360U

Agenda

3

Kubernetes Operators

Introduction

Custom Resource Definitions (CRDs)

CRD Examples (CronTab, PKS Sink Resources)

Operators and Operator Frameworks

Controller Fundamentals

Architecture

Writing Controllers

Kopf Demo

Kopf (Kubernetes Operator Pythonic Framework)

VM-Operator using Kopf


©2019 VMware, Inc. #CODE1360U 5

November 03, 2016 - CoreOS Blog Post by Brandon Philips Introduced and described the Concept of a Kubernetes Operator

Published two Examples: etcd and Prometheus Operator

Introduction


An Operator is an application-specific controller thatextends the Kubernetes API to create, configure, andmanage instances of complex stateful applications on behalf of a Kubernetes user.

Custom Resource Definition Custom ControllerOperator


https://coreos.com/blog/introducing-the-etcd-operator.html


Kubernetes Objects defining Custom Object Types

New API Endpoints

Example: Crontabs

/apis/stable.example.com/v1/namespaces/*/crontabs/...

Introduced in Kubernetes 1.7

Scope: Namespace or Cluster

Once registered you can create Objects of that Type

OpenAPI 3.0 Validation Schema

Beta in Kubernetes 1.15

Allows CRD Pruning: Deletion of unknown fieldsspec.preserveUnknownFields: false

Custom Resource Definitions (CRDs)

https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/

Extending the Kubernetes API


https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/


Example: CronTab CRD + Custom Resource CreationapiVersion: apiextensions.k8s.io/v1beta1

kind: CustomResourceDefinition

metadata:

name: crontabs.stable.example.com

spec:

group: stable.example.com

versions:

- name: v1

served: true

storage: true

scope: Namespaced

names:

plural: crontabs

singular: crontab

kind: CronTab

shortNames:

- ct

preserveUnknownFields: false

validation:

openAPIV3Schema:

type: object

properties:

spec:

type: object

properties:

cronSpec:

type: string

pattern: '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'

image:

type: string

replicas:

type: integer

default: 1

$ cat my-crontab.yaml

apiVersion: "stable.example.com/v1“

kind: CronTab

metadata:

name: my-cron-object

spec:

cronSpec: "* * * * */5"

image: my-cron-image

replicas: 1

$ kubectl apply -f my-crontab.yaml

$ kubectl get crontabs

NAME AGE

my-cron-object 6s

using

CR

CRD



Enterprise PKS - Sink Resources (CRDs)

https://docs.pivotal.io/runtimes/pks/1-4/create-sinks.html

Global Logging

Cluster Logging

Namespace Logging

overwrites

overwrites

Enable PKS users to configure Log-Destinations

Supported Protocols: Syslog (RFC 5424)

Log-Forwarding per Cluster or per Namespace

Using fluentbit Log Aggregator

Cluster & Namespace Sinks implemented as K8s CRDs

sinks.apps.pivotal.io

clustersinks.apps.pivotal.io

Enabled/disabled globally (i.e. for all Clusters) in PKS Tile

Specified by YAML file

kubectl create -f sink.yaml


https://docs.pivotal.io/runtimes/pks/1-4/create-sinks.html

https://tools.ietf.org/html/rfc5424


Sink Resources - Cluster View

K8s Cluster

Worker

sink-controller

Worker

fluentbit

Master Worker

fluentbit

Worker

fluentbitfluentbit

elastic stackLog Insight splunk>

The sink-controller and fluentbit Pods belong to the Namespace pks-system.VMworld 2019 Content: Not for publication or distribution


New Custom Ressource Definitions, sink-contoller Deployment and fluentbit Daemonset

Sink Resources in PKS Cluster

root@cli-vm:~# kubectl get crd

clustersinks.apps.pivotal.io 2018-11-27T15:57:03Z

sinks.apps.pivotal.io 2018-11-27T15:57:03Z

kubectl get all –n pks-system



Sink Resources - Worker Node View

Worker Node

Namespace A

Pod 2Pod 1

Namespace B

Pod 2Pod 1

Namespace C

Pod 2Pod 1

Pod stdout & stderr

Pod stdout & stderrnamespace

Pod stdout & stderr

elastic stack

Log Insight

pks-system

fluentbit

The fluentbit Pod on each Worker Node aggregates Logs from

all Namespaces and forwards them to the corresponding endpoint.VMworld 2019 Content: Not for publication or distribution



https://kubedex.com/operators/https://operatorhub.io/

A Selection

Operator Domain URL

Velero Backup https://velero.io/

Kanister Backup https://github.com/kanisterio/kanister

Tensorflow Big Data https://github.com/kubeflow/tf-operator

Jaeger Tracing https://github.com/jaegertracing/jaeger-operator

MongoDB Database https://github.com/mongodb/mongodb-enterprise-kubernetes

Spinnaker CI/CD https://operatorhub.io/operator/spinnaker-operator

Minio Storage https://github.com/minio/minio-operator

OpenFaaS FaaS https://github.com/openfaas-incubator/openfaas-operator

Kafka Streaming https://github.com/strimzi/strimzi-kafka-operator

Aqua Security https://github.com/aquasecurity/aqua-operator

Istio Service Mesh https://github.com/banzaicloud/istio-operator


https://kubedex.com/operators/

https://operatorhub.io/

https://velero.io/

https://github.com/kanisterio/kanister

https://github.com/kubeflow/tf-operator

https://github.com/jaegertracing/jaeger-operator

https://github.com/mongodb/mongodb-enterprise-kubernetes

https://operatorhub.io/operator/spinnaker-operator

https://github.com/minio/minio-operator

https://github.com/openfaas-incubator/openfaas-operator

https://github.com/strimzi/strimzi-kafka-operator

https://github.com/aquasecurity/aqua-operator

https://github.com/banzaicloud/istio-operator


Operator Framework - https://github.com/operator-framework

Operator SDK (released Mid 2018 by Red Hat)

Operator Lifecycle Manager

Operator Metering

Supported Languanges: Golang

• You can also use Helm Charts or Ansible Playbooks

Kubebuilder - https://github.com/kubernetes-sigs/kubebuilder

Supported Languages: Golang

Kopf - https://github.com/zalando-incubator/kopf

Zalando Open Source Project

Supported Languages: Python

Google Metacontroller - https://metacontroller.app

Google Open Source Project

Supported Languages: Python, JavaScript, etc.

Kubernetes Operator Toolkits


https://github.com/operator-framework

https://github.com/kubernetes-sigs/kubebuilder

https://github.com/zalando-incubator/kopf

https://metacontroller.app/


Kubernetes ArchitectureFrom the View of a Controller Developer



Overview

Kubernetes Architecture

Co

ntr

ol P

lan

e

API Serveretcd

ControllerManager

Scheduler …

Acce

ss

REST API SDKs Web UI kubectl

Wo

rke

rs

Kubelet Kubelet Kubelet

Pod Pod Pod



In Simple Terms…


Co

ntr

ol P

lan

e

API Server

Wo

rke

rs

= Control LoopVMworld 2019 Content: Not for publication or distribution


Request Flow


API ServerREST

REST

DecodingConversion &

DefaultingAdmission

Persistency (etcd)

…

WA

TC

HP

OS

T

$ kubectl create –f my_replicaset.yaml

apiVersion: extensions/v1beta1

kind: ReplicaSet

spec:

replicas: 2

Commands

Events



Commands and Events


Commands Events

• Requests (intent) to do something

• Named in the imperative, e.g. “CREATE”

• Can be rejected

• Typically used in synchronous 1-to-1 request/responsecommunication

• Higher coupling between sender and receiver

• Something that has happened (a fact)

• Named in past tense, e.g. “CREATED”

• Cannot (semantically) be rejected by receiver

• Asynchronous 1-to-many communication, e.g. publish/subscribe

• Low coupling between sender and receiverVMworld 2019 Content: Not for publication or distribution


Choreography, not Orchestration


AP

I S

erv

er

CREATE


kind: ReplicaSet

spec:

replicas: 2

CREATEPod

ReplicaSetCREATED

ReplicaSetController

BINDPod

PodCREATED

Scheduler Kubelet

PodBOUND

UPDATEPod

(“running”)

Time

Command

EventVMworld 2019 Content: Not for publication or distribution


Control Loop (Reconciliation)


Observe

Analyze

Act


kind: ReplicaSet

spec:

replicas: 2

desired := getDesiredState() current := getCurrentState()

diff := desired – current

if diff < 0 {

deletePods()

}

if diff > 0 {

createPods()

}

Command

Event



Controllers, oh my…




Further Reading




Writing ControllersGuidelines and Principles



Kubernetes comes with several features to make the life of a (controller) developer easier

Scheduling and Supervision (self-healing*)

Configuration and Secret Management

Service Discovery and Networking

Storage Management

(Cloud) Portability

Declarative API Stability and Extensibility (CRDs)

AuthN and AuthZ (RBAC)

SDKs

(Probably a lot more than we could fit on this slide)

* Self-healing (eventual recovery) != HA

Net Benefits Working With Kubernetes

Writing Controllers



Depending on the complexity of your controller, there might be a steep learning curve

Lots of primitives and objects to learn

client-go (de facto SDK) “is not for mere mortals” (Bryan Liles)

Optimistic concurrency in an asynchronous eventual consistent system

There is No Now

The (global) state is always behind you (distributed, delayed and unknown to the local observer)

Fast moving project

BUT!

Writing Controllers


https://queue.acm.org/detail.cfm?id=2745385


Writing ControllersRequired Mindset

Single Responsible Principle

Decoupling via event-driven messaging

No central coordinator

Eventual consistent by

design

Don’t rely on (assume) order

API server (etcd) is the source of

truth*

In-memory cache via reconciliation

Things will go wrong (crash)

No shared (wall) clock

Anticipate effects on the rest of the

system

Delivery and processing

guarantees only within Kubernetes

AutonomousProcesses

Concurrency&

Asynchrony

Statelessover

Stateful

Side EffectsDefensive

Programming



Kopf Demo



Easy Way to write Kubernetes Operators

Kopf - Kubernetes Operator Pythonic Framework

https://kopf.readthedocs.io/en/latest/https://bit.ly/2KoLvmP

$ kopf run example.py$ kubectl apply -f obj.yaml

$ kubectl get KopfExample kopf-example-1 -o yaml

apiVersion: zalando.org/v1

kind: KopfExample

metadata: ...

spec:

duration: 1m

field: value

items:

- item1

- item2

status:

message: hello world

# Custom Resource for Kopf Example Operator

apiVersion: zalando.org/v1

kind: KopfExample

metadata:

name: kopf-example-1

labels:

somelabel: somevalue

annotations:

someannotation: somevalue

spec:

duration: 1m

field: value

items:

- item1

- item2

obj.yaml example.pyimport kopf

@kopf.on.create('zalando.org', 'v1', 'kopfexamples')

def create_fn(spec, **kwargs):

print(f"And here we are! Creating: {spec}")

return {'message': 'hello world'} # will be new status

1 2

3


https://kopf.readthedocs.io/en/latest/

https://bit.ly/2KoLvmP


Conceptual Overview

Kopf vSphere VM Operator

vSphere

VMTemplate

Kubernetes

Folder:Example-1

VM VM

VM VM

Folder:Example-2

VM VM

ESXi Cluster

CRExample-2

CRExample-1

API ServerKopf

VM Operator

LIST/WATCH()

vCenter

CREATE()CLONE()

DESTROY()



DEMO

https://github.com/embano1/kopf-operator-vmworld



Example: etcd Operator (1)Installation, Load Balancer Configuration

$ git clone https://github.com/coreos/etcd-operator.git

$ cd etcd-operator/

$ ./example/rbac/create_role.sh Create RoleBinding for the etcd Operator$ kubectl apply -f ./example/deployment.yaml Create etcd Operator$ kubectl apply -f ./example/example-etcd-cluster.yaml Create 3-node etcd Cluster

$ cat example/example-etcd-client-service-lb.yaml Create LoadBalancer for etcd-ClusterapiVersion: v1

kind: Service

metadata:

name: example-etcd-client-service-lb

namespace: default

spec:

ports:

- name: client

port: 2379

protocol: TCP

targetPort: 2379

selector:

etcd_cluster: example-etcd-cluster

type: LoadBalancer

apiVersion: "etcd.database.coreos.com/v1beta2"

kind: "EtcdCluster"

metadata:

name: "example-etcd-cluster"

spec:

size: 5

version: "3.3.13"

To resize/upgrade the etcd Cluster change its properties and re-apply


https://github.com/coreos/etcd-operator.git


Example: etcd Operator (2)Connnecting to the etcd Cluster

$ kubectl get svc | grep example-etcd-client-service-lb | awk '{print $4;}‘

10.40.14.41,100.64.48.5

$ export ETCDCTL_API=3

$ export ETCDCTL_ENDPOINTS=http://10.40.14.41:2379

$ wget https://github.com/etcd-io/etcd/releases/download/v3.3.13/etcd-v3.3.13-linux-amd64.tar.gz

$ tar xvfz etcd-v3.3.13-linux-amd64.tar.gz

$ sudo mv etcd-v3.3.13-linux-amd64/etcdctl /usr/local/bin

$ kubectl get pods

NAME READY STATUS RESTARTS AGE

etcd-operator-85f7494fcf-4xql5 1/1 Running 0 69m

example-etcd-cluster-5kkp8lxrsn 1/1 Running 1 51m

example-etcd-cluster-6bbnlbp7kb 1/1 Running 1 67m

example-etcd-cluster-f2gzqtmh57 1/1 Running 1 51m

$ etcdctl put /message "Hello World"

$ etcdctl get /message

/message

Hello WorldVMworld 2019 Content: Not for publication or distribution

https://github.com/etcd-io/etcd/releases/download/v3.3.13/etcd-v3.3.13-linux-amd64.tar.gz


Resources



Kopf vSphere VM Operator Github Repository

• https://github.com/embano1/kopf-operator-vmworld

Zalando Kopf Github Repository:

• https://github.com/zalando-incubator/kopf

Awesome Operators

• https://github.com/operator-framework/awesome-operators

Programming Kubernetes (Book)

• https://learning.oreilly.com/library/view/programming-kubernetes/9781492047094/

Links and Resources (1)


https://github.com/embano1/kopf-operator-vmworld

https://github.com/zalando-incubator/kopf

https://github.com/operator-framework/awesome-operators

https://learning.oreilly.com/library/view/programming-kubernetes/9781492047094/


Kubernetes API Reference (v1.15)

• https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/

Kubernetes Sample Controller (CRD)

• https://github.com/kubernetes/sample-controller

Writing Controllers

• https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/controllers.md

Internals of how Controllers are implemented under the Covers

• https://medium.com/@cloudark/kubernetes-custom-controllers-b6c7d0668fdf

Kubernetes Operator Best Practices

• https://blog.openshift.com/kubernetes-operators-best-practices/



https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/

https://github.com/kubernetes/sample-controller

https://github.com/kubernetes/community/blob/master/contributors/devel/sig-api-machinery/controllers.md

https://medium.com/@cloudark/kubernetes-custom-controllers-b6c7d0668fdf

https://blog.openshift.com/kubernetes-operators-best-practices/


KubeCon Sessions (Controllers and Operators)

• Writing Kube Controllers for Everyone

– https://www.youtube.com/watch?v=AUNPLQVxvmw

• Writing Kubernetes Controllers for CRDs: Challenges, Approaches and Solutions

– https://www.youtube.com/watch?v=7wdUa4Ulwxg

• How We Built Contour, and What You Can Learn From Our Experience

– https://www.youtube.com/watch?v=4usXJE0EwHo

• Keep the Space Shuttle Flying: Writing Robust Operators

– https://www.youtube.com/watch?v=uf97lOApOv8

• Controllers: Lambda Functions for Extending your Infrastructure

– https://www.youtube.com/watch?v=TM-2GgQ6Q2A



https://www.youtube.com/watch?v=AUNPLQVxvmw

https://www.youtube.com/watch?v=7wdUa4Ulwxg

https://www.youtube.com/watch?v=4usXJE0EwHo

https://www.youtube.com/watch?v=uf97lOApOv8

https://www.youtube.com/watch?v=TM-2GgQ6Q2A


KubeCon Sessions (Architecture)

• The Kubernetes Control Plane for Busy People Who Like Pictures

– https://www.youtube.com/watch?v=zCXiXKMqnuE

• The Magic of Kubernetes Self-Healing Capabilities

– https://www.youtube.com/watch?v=91dgNqma7-Q&t=2s

• Kubernetes Design Principles: Understand the Why

– https://www.youtube.com/watch?v=ZuIQurh_kDk&t=1s



https://www.youtube.com/watch?v=zCXiXKMqnuE

https://www.youtube.com/watch?v=91dgNqma7-Q&t=2s

https://www.youtube.com/watch?v=ZuIQurh_kDk&t=1s

TPM, CNABU Platform Architect (OCTO)@tom_schwaller @embano1

Tom Schwaller Michael Gasch


Kubernetes Operators for VMware Enterprise PKS and VMware ...

Documents

Transcript of Kubernetes Operators for VMware Enterprise PKS and VMware ...