Clusternaut: Orchestrating Percona XtraDB Cluster

with Kubernetes.Raghavendra Prabhu

FOSDEM ’16rprabhu@yelp.com/me@rdprabhu.com/@randomsurfer

Yelp’s Mission:Connecting people with great

local businesses.

Yelp Stats:As of Q3 2015

89M 3271%90M

MeRaghavendra Prabhu

Software Engineer, Distributed Systems @ Yelp rprabhu@yelp.com / me@rdprabhu.com

Galera - “The Oar boat”

κυβερνήτης “The Helmsman”

DCOS

➔ Mesosphere ➔ Paasta➔ GCE

◆ Reference➔ ECS ➔ Smartcloud➔ Tectonic*

Rationale➔ Nodes v/s Hosts➔ Reusable components

◆ Monitoring◆ DNS◆ Logging◆ Metrics◆ Scheduler

➔ Agnostic ➔ Roles

The Fit➔ Layered

◆ Client - Server◆ Multi-layered

➔ Scaling◆ Horizontal and Vertical

● Preferred?➔ Statelessness in databases

◆ Planes of logic: Control/Data➔ Elasticity

◆ Elastic Scalability

Declarative vs Imperative➔ Configurable mgmt

◆ Puppet, Nix, Terraform ➔ Microservices

◆ What runs on my laptop ● What runs on server● Reproducibility

➔ 12-factor app➔ Composability ➔ Immutable deployment artifact

Containers ● Wth is a container and why should I care

○ Operating system virtualization● Isolation

○ Hierarchies of isolation - application, cgroups, namespaces, seccomp…

● Unikernels and VMs○ Role?

● Some - LXC, Docker*, Rocket*, runc, jails, solaris zones, lmcty, systemd-nspawn

Galera - really short intro! ➔ MySQL and WSREP api➔ Galera plugin➔ Group communication➔ Synchronous replication

◆ ‘Virtually’➔ EVS ➔ Certification-based

◆ Optimistic Concurrency➔ Automatic Node Provisioning

Galera - really short intro! ➔ CAP theorem and Galera

◆ CP➔ How does it fit

◆ Others◆ Idempotency

➔ Stateless? ◆ Symmetric◆ Replicas - Cassandra et.al.◆ MySQL Cluster

➔ Maintenance of Quorum

Orchestration● SOA def

○ ‘Stitching’○ ‘Composing’

● Automation?● Choreography● Best of both worlds

Kubernetes➔ Constituents:

◆ Kubelet◆ Pods

● Main service & helpers◆ Services

● The gcomm:// URL.◆ Replication Controller◆ Labels and Selectors

● MCollective

Kubernetes➔ Others:

◆ Mesos● Aurora, Chronos, Marathon

◆ Docker Swarm◆ Fleet

➔ Key Differences

Pods

Pods● Herd.. ● What should they contain - containers! ● How is the grouping done ● Pods and nodes

○ Colocation ● Pod communication● Labels

Services

Services● Don’t commingle with `microservices`● Think of endpoints. ● Layering architecture

○ Logical address of subset of pods● Communication

○ Environment■ Ordering requirement■ Discovery

○ DNS ■ Issues with DNS

Replication Controller● “Herd Management”● ASG● Pod template

○ Pattern○ Anti-pattern

● Role ○ Init for cluster○ Rolling updates ○ Multi-version

Networking● Docker-style linking● Proxy for Pods● Types

○ Pod to Pod○ Pod to Service○ Intra-Pod○ External to Service

● Providers:○ OpenVSwitch / Flannel / Calico / Weave / Google

External components● Flannel / Others● Etcd● Fluentd● Skydns● Container Registry● Scheduler● REST server● Proxy● cAdvisor

PAAS: PaaSTA● Docker● Mesos

○ Chronos○ Marathon

● Sensu● Smartstack

○ Zookeeper● Jenkins● Splunk / Signalfx● Why

Deployment● Declare and build individual Galera/PXC nodes.

○ Keep it minimal and simple○ No assumptions

● Without Kubernetes ○ Docker-compose

■ Possible issues ● Galera node ⇔ Pod

○ Haproxy○ xinetd

Deployment● Basic Steps:

○ Create a ‘flat’ network - 10.0.0.0/24○ Create a ‘cluster’ - zone○ Create a service endpoint.

■ Internal service - 3306/4567/4568.■ External service - 3306/3306(?).■ Expose the external.■ Session affinities.

Deployment● Next:

○ Bootstrap a node Pod from a template.■ Query existing with selector.

○ Start rest of nodes from template.■ Point to Service with selector.■ Replication controller

○ Volumes

Deployment - Implications● Load balancing in state transfers● Respawning of nodes on timeout

○ May not be same nodes.● kubectl to manage● Separation of client and cluster traffic

Service Definition

Dockerfile

```

... | kubectl create -f -

kubectl expose service eclient --port=3306 --target-port=3306 --name=loadbl --type='LoadBalancer’

```

``` .. | kubectl create -f -

kubectl scale --replicas=8 replicationcontrollers controller

```

```kubectl get --no-headers pods -l 'name=pxc' | wc -l2 kubectl stop …kubectl get --no-headers pods -l 'name=pxc' | wc -l2 ```

Credits! ● https://www.pinterest.com/duanejohnson851/star-trek-tng/● https://upload.wikimedia.org/wikipedia/commons/a/a5/CubeSpace.jpg● https://upload.wikimedia.org/wikipedia/commons/thumb/b/ba/IUB_Arboretum_-_lotus_pond_-

_dry_seed_pod_-_P1100172.JPG/1280px-IUB_Arboretum_-_lotus_pond_-_dry_seed_pod_-_P1100172.JPG● https://raw.githubusercontent.com/kubernetes/kubernetes/master/docs/design/architecture.png● https://pbs.twimg.com/profile_images/511909265720614913/21_d3cvM.png● https://camo.githubusercontent.

com/96468330aba188dbd7d7eeae0caca32d9a6329df/687474703a2f2f656e67696e656572696e67626c6f672e79656c702e636f6d2f696d616765732f70726576696577732f7061617374615f707265766965772e706e67

● http://galeracluster.com/documentation-webpages/_images/replicationapi.png

Further reading! ● http://kubernetes.io ● https://open.mesosphere.com/frameworks/ ● https://coreos.com/kubernetes/docs/latest/kubernetes-networking.

html● http://paasta.readthedocs.org/en/latest/about/paasta_principles.html● http://12factor.net/ ● http://kubernetes.io/v1.1/docs/api-reference/v1/definitions.html

ContactRaghavendra Prabhu

rprabhu@yelp.com / me@rdprabhu.com Twitter: @randomsurfer

Linkedin: rdprabhuGithub: ronin13

http://rdprabhu.com http://about.me/raghavendra.prabhu

http://rprabhu.yelp.com

@YelpEngineering

YelpEngineers

engineeringblog.yelp.com

github.com/yelp