MINIMIZING TRANSACTION LATENCY IN GEO...
Transcript of MINIMIZING TRANSACTION LATENCY IN GEO...
![Page 1: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/1.jpg)
MINIMIZING TRANSACTION LATENCY IN
GEO-REPLICATED DATA STORES
Divy Agrawal
Department of Computer Science
University of California at Santa Barbara
Joint work with: Amr El Abbadi, Hatem Mahmoud, Faisal Nawab, and Vaibhav Arora
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 2: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/2.jpg)
Motivation
• Cloud computing and big data infrastructures:
• Move away from traditional database solutions.
• Infrastructure technologies: Bigtable (for data management), and
MapReduce (for data analysis).
• Last decade: proliferation of numerous solutions.
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 3: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/3.jpg)
Data-centric Synchronization: A Renaissance
• Low-level Storage Architectures:
• Eventual consistency higher-level guarantees
• Reason: Lack of clear semantics source of confusion to
developers
• Data Store Architectures:
• Single row atomicity multi-row transactional guarantees
• Reason: Primitive data API Proliferation of duplicate code
• Data Replication:
• Non-zero failure probabilities Multi-datacenter architectures
(fault-tolerance)
• Datacenter migration Multi-datacenter solutions (availability)
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 4: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/4.jpg)
Data-centric Synchronization: A Renaissance
• Low-level Storage Architectures:
• Eventual consistency higher-level guarantees
• Lack of clear semantics source of confusion to developers
• Data Store Architecture:
• Single row atomicity multi-row transactional guarantees
• Primitive data API Proliferation of duplicate code
• Data Replication:
• Non-zero failure probabilities Multi-datacenter architectures
(fault-tolerance)
• Datacenter migration Multi-datacenter solutions (availability)
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 5: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/5.jpg)
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 6: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/6.jpg)
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 7: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/7.jpg)
Multi-datacenter Architectures
A
B
C
Datacenter CaliforniaA
B
C
Datacenter VirginiaA
B
C
Datacenter Oregon
10/25/2016 SBBD'2016 @ Salvador, Bahia
• Localized access
• Fault-tolerance
• Planned Outages
![Page 8: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/8.jpg)
Multi-datacenter Synchronization
Wide-area latency
Coordination is expensive
Consistency
guarantees
Transaction
support
10/25/2016 SBBD'2016 @ Salvador, Bahia
Focus on cross-datacenter messages
![Page 9: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/9.jpg)
PHOTON [SIGMOD’12]
Multi-datacenter Deployment in
Production
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 10: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/10.jpg)
Data-processing Pipelines @ Google
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 11: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/11.jpg)
Search Queries & Ad Clicks
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 12: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/12.jpg)
Engineering Challenges
• Exactly-once semantics:
• used for billing & internal monitoring
• Reliability
• continuous monitoring in spite of datacenter outage
• Scalability
• millions of joins per second
• Latency
• real-time
• Delayed primary stream
• click and query logs shipped independently – must be joined
whenever query event becomes available
SBBD'2016 @ Salvador, Bahia10/25/2016
![Page 13: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/13.jpg)
Photon: Unique Event-ID
• Thousands of servers across the world
• all events (e.g. query and click) recorded to a persistent storage
• and replicate them to multiple logs datacenters
• An event id consists of three fields:
• ServerIP - Uniquely identify the server
• ProcessID - Uniquely identify process that generated an event
• Timestamp - Time the event was generated
• A quick generation of event ids is critical for a fast
response to user actions
SBBD'2016 @ Salvador, Bahia10/25/2016
![Page 14: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/14.jpg)
Single Datacenter Architecture
SBBD'201
6 @
Salvador,
Bahia
10/25/2016
![Page 15: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/15.jpg)
Multi-datacenter Architecture
SBBD'201
6 @
Salvador,
Bahia
10/25/2016
![Page 16: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/16.jpg)
SPANNER [OSDI’12]
2PC/PAXOS
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 17: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/17.jpg)
Spanner [OSDI’12]
• Google’s solution for multi-datacenter replication
• Each partition has a leader
• Read operations are synchronized (locked) at Paxos Master copy
• Writes are deferred.
• Commit protocol (2PC/Paxos)
• Two-Phase Commit (2PC) across partition leaders
• Paxos to replicate each step of 2PC
A B
C
A B
C
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 18: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/18.jpg)
Optimistic variant: F1 on Spanner
• Transaction Execution:
• Optimistic reads
• Deferred writes
• At commit:
• Validate reads (using timestamps)
• Obtain read and write locks at the master
• Spanner commit: 2PC/Paxos
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 19: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/19.jpg)
Transaction latency
A B
Datacenter California
C
A B
Datacenter Virginia
C
Read requests
2PC message
Paxos message
Read (2)Send
prepare (1)
Replicate
(2)
Receive
prepare (1)
Commit
(0)
Replicate
(2)
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 20: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/20.jpg)
Effective multi-datacenter replication
• Spanner: An effective approach for multi-datacenter
operations
• (Acceptable) throughput
• Fault-tolerance
• Transactional consistency: multi-objects + replication
• Major challenge with geo-replication:
• Wide-area communication delays adversely impacts latency
budgets for interactive transactions (e.g., mobile E-commerce)
Engineered solutions (leverage datacenter proximity)
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 21: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/21.jpg)
REPLICATED COMMIT
[VLDB’13]
PAXOS/2PC
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 22: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/22.jpg)
Wide-area latency awareness
• Inter-datacenter latency is much higher than intra-
datacenter latency
• Intra-datacenter latency: ~ 1-2 milliseconds
• Inter-datacenter latency: 10s to 100s milliseconds
Minimize cross-datacenter communication
• Replicated Commit [VLDB’13: UCSB]
• Majority voting algorithm for the geo-replication framework
• Metaphor: “Datacenter as a computer” paradigm
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 23: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/23.jpg)
Replicated Commit
• Transaction execution:
• Read operations synchronized (locking) at majority data-centers.
• Write operations deferred until commit.
• At commit:
• User application initiates a Paxos protocol
• One of the shards at each data-center acts as a 2PC coordinator
• 2PC within each datacenter (prepare phase of 2PC is the voting phase
of Paxos)
• Commit phase of 2PC is integrated with the accept phase of Paxos
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 24: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/24.jpg)
Replicated Commit [VLDB’13]
A B
Datacenter California
C
B A
Datacenter Virginia
C
Read requests
Voting messages
Locking messages
Read (2)Voting
request (1)Locks (0) Voting (1)
Commit
(0)
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 25: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/25.jpg)
Latency Analysis
• Latency depends on the network topology
• Read phase
• Replicated commit: majority (subject to stragglers)
• Spanner: read from the leader
• Commit phase
• Replicated commit: 1 round to majority (2)
• Spanner: 1 round to leaders (2) + majority round from leaders (4)
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 26: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/26.jpg)
Performance (Amazon Datacenters)
337
167403
Replicated Commit
2PC/Paxos
NOTE: Our implementation
• Five data centers
• Data into 3 partitions
• YCSB clients at
each data center
• Average commit
latency (ms)
92
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 27: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/27.jpg)
REQUEST-RESPONSE
PARADIGM RTT BARRIER
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 28: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/28.jpg)
Datacenter A Datacenter B
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 29: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/29.jpg)
Datacenter A Datacenter B
Latency
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 30: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/30.jpg)
MESSAGE LOGGING
[CIDR’13]
Log-based Replication
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 31: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/31.jpg)
Event Log Propagation
• Message Futures [CIDR’13]
• Leverages event logs and log propagation that ensures causality
[FiMi82,Wuu84,Liskov86,…]
• Decouples consistency and fault-tolerance
• Protocols to ensure consistency only
• Augment with fault-tolerance [skipped]
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 32: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/32.jpg)
Message Futures
A
B
Simple case: Ping ponging log propagations
Commit rule: (1) wait until next log is received
(2) detect conflicts with coming log.
txnCommit
Latency
less than
RTT
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 33: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/33.jpg)
Arbitrary Propagation Rates
A
B
General case: Continuous log propagations
Commit rule: (1) wait until previous log transmission is acknowledged
(2) detect conflicts with coming log
txn
Commit
10/25/2016 SBBD'2016 @ Salvador, Bahia
txn
![Page 34: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/34.jpg)
[SIGMOD’15]
“IS THERE A LOWER-BOUND
ON TRANSACTION LATENCY?”
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 35: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/35.jpg)
What is the relationship between the latencies
of transaction T1 and T2?
T1 requests to
commit T1 commits
A
B Events can affect
outcome of T1
Events can be
affected by T1Transaction T2
T1 latency
T2 latency
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 36: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/36.jpg)
10/25/2016 SBBD'2016 @ Salvador, Bahia
• L[T] = c(T) – q(T)
• Awareness zone > q(T) + RTT/2
• Influence zone < c(T) – RTT/2
• Critical zone: Awareness Zone – Influence Zone = RTT - LA
• Duration of the critical zone: RTT – LA
• But that is the latency of T’ at B, i.e., LB > RTT – LA
• Equivalently: LA + LB > RTT
B
q(T) c(T)
Influence Awareness
T latency
Critical Zone:
T’ latency
A
![Page 37: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/37.jpg)
Optimal latency
• Lower bound:
• Latency(x) + Latency(y) > RTT(x,y)
Minimize (sum of latencies)
Subject to (1) Latency(A) + Latency (B) > RTT(A,B), for all A,B
(2) Latency (A) >= 0, for all A
A
B C
30 20
40
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 38: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/38.jpg)
Optimal latency
A
B C
30 20
40
Protocol Latency(A) Latency(B) Latency(C) Average
Leader-based (Leader A) 0 30 20 16.67
Leader-based (Leader C) 20 40 0 20
Majority 20 30 20 23.33
Optimal 5 25 15 15
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 39: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/39.jpg)
Lower-bound result: What is the point?
• What is the significance of this result – especially since
we have no control of the physical reality (RTTs)?
• No physical system can guarantee pre-determined values
of RTTs between datacenters, right?
• What happens when message communication is subject
to random aberrations, e.g., lost messages?
• Does this approach require perfect clock synchronization?
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 40: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/40.jpg)
Philosophy of the Design
• Use the physical reality (i.e., RTTs) to develop the right abstraction.
Physical RTTs Logical Notion of RTTs
• Develop a protocol in the transformed domain to ensure that the resulting protocol guarantees correctness.
• Deployment in the physical world should only impact efficiency not correctness.
• Proximity of the physical reality to the logical abstraction will yield the desired result, i.e., optimal latency.
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 41: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/41.jpg)
[SIGMOD’15]
ACHIEVING THE
LOWER-BOUND
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 42: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/42.jpg)
Helios: The Logical
10/25/2016 SBBD'2016 @ Salvador, Bahia
A
B C
30 20
40
Protocol LA LB LC Average
Optimal 5 25 15 15
A
B
C
100105
90
95
115
110
![Page 43: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/43.jpg)
Impact of the Physical Reality: RTT
• What happens when:
RTTreal ≠ RTT
• Does this jeopardize correctness?
• The answer is NO.
• RTTreal > RTT:
• The commit latency realized at Datacenter A will be larger than
what was expected.
• RTTreal < RTT:
• The commit latency realized at Datacenter A will be smaller than
what was expected.
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 44: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/44.jpg)
RTTreal > RTT
10/25/2016 SBBD'2016 @ Salvador, Bahia
A
B C
30 20
40
Protocol LA LB LC Average
Optimal 5 25 15 15
A
B
C
100105
90
95
115
110
110
120
40
![Page 45: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/45.jpg)
RTTreal < RTT
10/25/2016 SBBD'2016 @ Salvador, Bahia
A
B C
30 20
40
Protocol LA LB LC Average
Optimal 5 25 15 15
A
B
C
100105
90
95
115
110
110
20
![Page 46: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/46.jpg)
Impact of Physical Reality: Clock skews
• What happens when Datacenters clocks are not perfectly
synchronized?
• Correctness is preserved.
• If a Datacenter A’s clock drifts ahead of others
• The commit latency at Datacenter A will be larger than what was
expected.
• If a Datacenter A’s clock drifts behind others
• The commit latency at Datacenter A will be smaller than what was
expected
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 47: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/47.jpg)
Helios: Datacenter A’s clock is 5 units
ahead
10/25/2016 SBBD'2016 @ Salvador, Bahia
A
B C
30 20
40
Protocol LA LB LC Average
Optimal 5 25 15 15
A
B
C
100105
90
95
115
110
95
115105
110
![Page 48: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/48.jpg)
Helios: Datacenter A’s clock is 5 units
behind
10/25/2016 SBBD'2016 @ Salvador, Bahia
A
B C
30 20
40
Protocol LA LB LC Average
Optimal 5 25 15 15
A
B
C
100105
90
95
115
110
85
95100
![Page 49: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/49.jpg)
PERFORMANCE EVALUATION
IN A MULTI-DATACENTER
ENVIRONMENT
10/25/2016 SBBD'2016 @ Salvador, Bahia
![Page 50: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/50.jpg)
Performance Evaluation: Latency
10/25/2016 SBBD'2016 @ Salvador, Bahia
58ms
211ms
Helios
Fault-tolerance
f = 2
Helios 2PC/Paxos
107ms
221ms
217ms
226ms
132 152 235
Average
Replicated
Commit
110ms
167ms
131
43% 35% 44%
![Page 51: MINIMIZING TRANSACTION LATENCY IN GEO ...sbbd2016.fpc.ufba.br/.../slides/talk01_GeoReplication.pdfMINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department](https://reader036.fdocuments.in/reader036/viewer/2022081404/5f05e45d7e708231d4153d5d/html5/thumbnails/51.jpg)
Concluding Remarks
• Protocol design based on traditional request-response
paradigm:
• Replicated Commit with optimistic reads
• Protocol design based on log propagation and causality of
message communication:
• Helios (with appropriate level of fault-tolerance)
• Research and Engineering Challenge:
• Data stores Data Processing Pipelines
• Database Operator and Message Communication framework (e.g.,
Apache Flink)
Geo-replicated for scalability and fault-tolerance
10/25/2016 SBBD'2016 @ Salvador, Bahia