ACIDic ClustersReview of contemporary ACID-compliant databases with

synchronous replicationFossetcon 2014

Raghavendra Prabhuraghavendra.prabhu@percona.com

Percona

12 September, 2014

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 1 / 42

Outline

1 Introduction

2 ReviewArchitecture

3 Epilogue

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 1 / 42

Outline

1 Introduction

2 ReviewArchitecture

3 Epilogue

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 1 / 42

Outline

1 Introduction

2 ReviewArchitecture

3 Epilogue

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 1 / 42

Introduction

Seed quotes..

“’Network is reliable’ - a fallacy of the distributedsystem.”

“A distributed system is one in which the failure of acomputer you didn’t even know existed can render your owncomputer unusable.” - Leslie Lamport

“Those who would give up essential correctness, topurchase a little temporary scalability, deserve neithercorrectness nor scalability.” - Leif Walsh

“A given row can’t be modified more than once perRTT." - Alexey Yurchenko

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 2 / 42

Introduction

Introduction

I A twist on CAP

Pick any Two?

� ACID

� SQL

� Synchronous replication

I Solution: Don’t have to pick!I CAP and latency - eventual consistencyI Lambda architecture

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 4 / 42

Introduction

Introduction

I A twist on CAP

Pick any Two?

� ACID

� SQL

� Synchronous replication

I Solution: Don’t have to pick!

I CAP and latency - eventual consistencyI Lambda architecture

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 4 / 42

Introduction

Introduction

I A twist on CAP

Pick any Two?

� ACID

� SQL

� Synchronous replication

I Solution: Don’t have to pick!I CAP and latency - eventual consistencyI Lambda architecture

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 4 / 42

Introduction

RAS

I Why Synchronous

� Symmetry∅ Easier to manage and conceive∅ Build once, deploy everywhere!

� Latency!

I Do we need ACID?

� Depends!� Concurrent workload

I Relational database� Can it be a KV/document store?� Everyone wants SQL though.� Beats every other API!

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 6 / 42

Introduction

RAS

I Why Synchronous� Symmetry

∅ Easier to manage and conceive∅ Build once, deploy everywhere!

� Latency!

I Do we need ACID?

� Depends!� Concurrent workload

I Relational database� Can it be a KV/document store?� Everyone wants SQL though.� Beats every other API!

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 6 / 42

Introduction

RAS

I Why Synchronous� Symmetry

∅ Easier to manage and conceive∅ Build once, deploy everywhere!

� Latency!

I Do we need ACID?

� Depends!� Concurrent workload

I Relational database� Can it be a KV/document store?� Everyone wants SQL though.� Beats every other API!

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 6 / 42

Introduction

RAS

I Why Synchronous� Symmetry

∅ Easier to manage and conceive∅ Build once, deploy everywhere!

� Latency!

I Do we need ACID?

� Depends!� Concurrent workload

I Relational database� Can it be a KV/document store?� Everyone wants SQL though.� Beats every other API!

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 6 / 42

Introduction

RAS

I Why Synchronous� Symmetry

∅ Easier to manage and conceive∅ Build once, deploy everywhere!

� Latency!

I Do we need ACID?

� Depends!� Concurrent workload

I Relational database� Can it be a KV/document store?

� Everyone wants SQL though.� Beats every other API!

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 6 / 42

Introduction

RAS

I Why Synchronous� Symmetry

∅ Easier to manage and conceive∅ Build once, deploy everywhere!

� Latency!

I Do we need ACID?

� Depends!� Concurrent workload

I Relational database� Can it be a KV/document store?� Everyone wants SQL though.

� Beats every other API!

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 6 / 42

Introduction

RAS

I Why Synchronous� Symmetry

∅ Easier to manage and conceive∅ Build once, deploy everywhere!

� Latency!

I Do we need ACID?

� Depends!� Concurrent workload

I Relational database� Can it be a KV/document store?� Everyone wants SQL though.� Beats every other API!

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 6 / 42

Review

Contemporary systems

I PXC/GaleraI NDB ClusterI Google F1I Others - FoundationDB, CockroachDBI Not talking of comparisons/benchmarks.I Apples and Pineapples!

(Since they are all clusters)

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 7 / 42

Review

Family of Synchronous Systems

I Transactional replication� Overhead and workarounds� One-copy equivalence� NDB: 2 PC

I Virtual Synchrony� Extended Virtual Synchrony: Galera

I Multi-phase� Paxos: Google F1� Also 2PC

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 8 / 42

Review Architecture

Layers

I Necessity� Strengths� Degrees of freedom

I Monolithic v/s Layered� Monolithic - Galera� Layered - F1/Spanner, NDB(?)

I Implications� Failures� Debugging

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 9 / 42

Review Architecture

Storage

I Unbundling of translation and dataI Different strategies

� Spanner∅ CFS underneath∅ Provides guarantees in replication

� Hierarchical storage� NDB

∅ Separatation of data and management nodes∅ Hybrid storage

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 11 / 42

Review Architecture

Storage

I WSREP� Plugin system for Galera Provider

∅ InnoDB∅ More tightly bound∅ In-memory and ring-buffers

� Transaction awareness

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 12 / 42

Review Architecture

Storage

I Statelessness� Virtual Synchrony

∅ Causality

� Loose/Tight Binding� Consequences

∅ Co-ordination issue∅ Flexibility∅ Transitivity∅ Uncoupled relation

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 13 / 42

Review Architecture

Relational Database

I NoSQL / NewSQL / OldSQL� New wine in old bottle?� Approach towards ACIDity.

∅ Pros and Cons∅ The Fit

� Does NoSQL meet RAS∅ Eventual (in)Consistency∅ Strict definitions∅ Instrumentation

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 14 / 42

Review Architecture

Relational Database

I Role of API� Presence of Layer� Nature of storage� API defines data model?

I NoSQL access to SQLI ORM

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 16 / 42

Review Architecture

Degree of Synchronicity

I Latency is a killer!� Amortization

I Synchronicity imply Causality?I Commutative writesetsI Role of Applications/Architecture

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 18 / 42

Review Architecture

ACIDity

I ACIDity test?

I BASE and NoSQLI MVCC and Synchronous Replication

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 19 / 42

Review Architecture

ACIDity

I Does ACID require synchronous ORI Does Synchronous demand ACIDity?

� Reconciliation requires it.∅ Expensive without∅ Rollbacks

� ACID possible without Relational semantics?� Thought: Filesystems with replication

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 21 / 42

Review Architecture

Locking

I Optimistic Concurrency� Reduce communication

∅ WWW/HTTP� Software Transactional Memory

∅ Lock elision∅ Natural to Databases

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 23 / 42

Review Architecture

Locking

I Conflicts and Deadlocks� Limiting factor: Amdahl’s� Scale: number of nodes and size of transactions

=> Quadratic or Quinary=> Not strictly bound=> Only a strict lower bound

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 24 / 42

Review Architecture

Locking

I Pessimistic Locking� Performance� Network Overhead

∅ More roundtrips∅ Pipeline/Batching

I Hybrid approach� Switching

I Granularity of Locking

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 25 / 42

Review Architecture

Locking

I Externalities� GPS clock - Google F1

Controlled conditionsLimitations

� Lock Managers: Zookeeper, ChubbyI Lamport timestamps!

� A Zeitgeber!

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 27 / 42

Review Architecture

Integration

I Scaling the system upwardsI Composable TransactionsI XA SupportI Extensibility

� Adding other storage engines: PSA

I Geographic Replication

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 28 / 42

Review Architecture

Sharding

I Systems that support itI Issues of QuorumI Adding support externally

� Spider Engine� Cluster of clusters� MySQL Fabric?

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 29 / 42

Review Architecture

Transition

I From single node to a clusterI Idempotency of transactionsI Integration with Async replicationI Loss of key parts

� Degree of ACIDity

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 30 / 42

Review Architecture

Operational usage

I Resource Manager� Moving parts� Reduces the confusion

I Backups� Impact on cluster operations� Logical and Physical backups� NoSQL interface

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 31 / 42

Review Architecture

Operational usage

I Integration with other HA� Load balancers and Proxies� Fencing / STONITH� Resource Managers

∅ Pacemaker

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 32 / 42

Review Architecture

Eventual Consistency

“not consistent right now, maybe come back later andthings will be better”

“Full transactional consistency is one of the mostimportant properties of F1.”

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 34 / 42

Review Architecture

EC: Qualitative Analysis

I Active analysis� System-centric� Convergence time

I Passive analysis� Distributed tracing� Dangling reads� Client-centric

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 35 / 42

Review Architecture

EC: Qualitative Analysis

I Guarantees� Latency and staleness� Monotonic reads� Causality� k|delta - atomicity

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 36 / 42

Epilogue

Limitations

I None!

I Silver bullet?I Where you may want Async / delayI Hot updates/insertsI Schema changesI Compromise on strictness/features/performance

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 38 / 42

Epilogue

Limitations

I None!I Silver bullet?I Where you may want Async / delayI Hot updates/insertsI Schema changesI Compromise on strictness/features/performance

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 38 / 42

Epilogue

Further Reading

I Is this a solved problem?

May be not.I The Dangers of Replication and a SolutionI Replication Using Group Communication Over a Partitioned NetworkI Transaction Processing: Concepts and TechniquesI F1: A Distributed SQL Database That ScalesI Eventually Consistent: Not What You Were Expecting?I The Layer ConceptI Don’t Settle for Eventual ConsistencyI Lambda Architecture

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 39 / 42

Epilogue

Further Reading

I Is this a solved problem? May be not.I The Dangers of Replication and a SolutionI Replication Using Group Communication Over a Partitioned NetworkI Transaction Processing: Concepts and TechniquesI F1: A Distributed SQL Database That ScalesI Eventually Consistent: Not What You Were Expecting?I The Layer ConceptI Don’t Settle for Eventual ConsistencyI Lambda Architecture

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 39 / 42

Epilogue

Image Credits

I http://upload.wikimedia.org/wikipedia/commons/e/e8/Latency_map_world.png

I http://blog.wikimedia.org/2014/07/09/how-ripe-atlas-helped-wikipedia-users/

I https://secure.flickr.com/photos/alexbrn/5584251627I https://secure.flickr.com/photos/russmorris/407778776I https://secure.flickr.com/photos/carlosluzz/561920999I https://secure.flickr.com/photos/eiriknewth/282268782I https://secure.flickr.com/photos/shaireproductions/5509387767I https://secure.flickr.com/photos/grantmac/3049823203I http://guide.couchdb.org/draft/consistency/01.pngI http://www.yeeach.com/post/583

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 40 / 42

Epilogue

Summary

I QuestionsI Theories

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 41 / 42

Epilogue

About

I /me: Raghavendra Prabhu, Product Lead, Percona XtraDBCluster, Percona.

I Slides will be at http://www.slideshare.net/slidunderI Twitter: randomsurferI LinkedIn: rdprabhuI Github: ronin13I Presentation under CC BY-SA 4.0

Raghavendra Prabhu (Percona) ACIDic Clusters 12 September, 2014 42 / 42