KubeCon EU 2016: Full Automatic Database: PostgreSQL HA with Kubernetes

FullAuto

DatabaseJosh Berkus

Red Hat Project Atomic

KubeCon.EU 2016

WIP: waiting for1.2/1.3

Single Master DBs:Problem

low availabilityunidirectional replicationvery manual HA solutions

Why not multimasterDBs?

just moving the problem around

"eventual" consistencynetwork lagmaturity issuesfeature povertyapp compatibility

But PG Replicaton isAwesome!

Easy to set upGuaranteedCorruptionfreeAntifootgunCombines with DR

Y U No Failover?

"Automatedfailover is toocomplicated.You don't want

it."

Hard != Impossible

Hard != Impossiblegeneral autofailover is prohibitive

but ... we can implement commonuse cases

The 80% Solution1. Pool of async replicas2. Cheap/replacable nodes Containers

3. Watchdog service4. Autopromote one replica5. Other nodes remaster6. Update routing

Now, a little history ...

Handyrepmastercontroller architecturebased on Python Fabric + SSHworked in productionworked with any Postgres configpluggable

www.handyrep.org

Handyrep: too generalDifficult to installDifficult to debugOver 100 configuration optionsScaled poorlyHR server was SPoF

Zalandono1 European online fashion15m customers150 databases24/7/365 operation

... needed automated,decentralized HA

False failoverMisfiresRaceconditions

Failover Failure

Split Brain and SMDBs

worst possible outcomeautomated recovery impossiblemanual recovery painful

Patroni

1. Postgres is a poor store of its ownreplication state

2. Smart agents > topdowncontrollers

Compose GovernorContainersEtcdbased consensusSimple PostgreSQL controller

... so we forked it.

How it works

failover est omnis divisa inpartes tres

The Patroni Controller

Patroni controllerPython daemonRuns in each container as PID 1Controls Postgresstartup/shutdown/configProvides external REST APIEnforces opinionated config

Patroni Failover

What about splitbrain?

Etcddistributed consensus HTTP datastoreRaft algoritmimplements CAgreat for config + metadata

not for data data

Etcd AlternativesZookeeper

larger scalesupported

Consulintegrates discoverynot (yet) suppported

What's AtomicDB?WIP project

PostgreSQLPatroniAtomic HostKubernetesDynamic proxy (dev)Cockpit UI (dev)

Let's see thatagain

The Proxy Problemdifferentiate master and readonlyconnectionsmaster service needs to followfailoverfailover logic too complex forkubernetes (1.1)

pgbouncer?current implementation inpgbouncermaster, read slaves separateservices/portsdepends on flannel LB

not good enough. Waiting for1.2/1.3!

More featurespg_rewind support (9.4+)configurable node imaging

WALEPITR

synchronous replicationnonfailover replicas

More Stuff Underdevelopment

cascading replicationintegrated proxyBDR support?

fork us on Github!

ResourcesThis Presentation: jberkus.github.io/full_auto_dbPatroni Project: github.com/zalando/patroniAtomicDB Project: github.com/jberkus/atomicdb

morejberkus:

projectatomic:

@fuzzychefwww.databasesoup.com

www.projectatomic.ioRedHat booth forCockpit Kube demo

¿questions?

KubeCon EU 2016: Full Automatic Database: PostgreSQL HA with Kubernetes

Technology

Transcript of KubeCon EU 2016: Full Automatic Database: PostgreSQL HA with Kubernetes