EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y...
Transcript of EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y...
![Page 1: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/1.jpg)
EECS 491Introduction to Distributed
Systems
Fall 2019
Harsha V. Madhyastha
![Page 2: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/2.jpg)
October 22, 2019 EECS 491 – Lecture 14 2
● Added to SOSP “Hall of Fame”● Rumored to be underpinning of Amazon S3’s
architecture
![Page 3: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/3.jpg)
Dynamo
● Setting:◆ Tens of millions of customers◆ Data spread across tens of thousands of servers
● Example use case: Store shopping carts● Goals:
◆ High availability◆ Low latency
October 22, 2019 EECS 491 – Lecture 14 3
![Page 4: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/4.jpg)
Consistent Hashing in Dynamo
● Recall: Consistent hashing maps value for key to successor in hash space
● Replicate value for every key at N nodes◆ N clockwise successors of key
● Execution of writes◆ Write received by coordinator (successor of key)◆ Coordinator forwards to successors
October 22, 2019 EECS 491 – Lecture 14 4
![Page 5: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/5.jpg)
Replication in Dynamo
5
N32
N90
N105N10
N60
N120
October 22, 2019 EECS 491 – Lecture 14
K21
![Page 6: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/6.jpg)
Using Consistent Hashing
Client
Front-end
Server
Server
Server
October 22, 2019 EECS 491 – Lecture 14 6
Front-end
![Page 7: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/7.jpg)
Consistent Hashing in Dynamo
Client
ServerServer
Server
Server
October 22, 2019 EECS 491 – Lecture 14 7
Server
What would it take to make this work?
1-hop DHT
![Page 8: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/8.jpg)
Gossip● Once per second, each server contacts a
randomly chosen other server● Servers exchange their lists of known servers
◆ Including virtual node IDs
● How long for everyone to discover server addition/removal?
● Won’t mismatch in view lead to consistency violations?
October 22, 2019 EECS 491 – Lecture 14 8
![Page 9: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/9.jpg)
Sloppy quorums● N replicas for every key
◆ Higher durability with greater N● Serving reads and writes:
◆ Coordinator forwards request to N-1 successors◆ Waits for response from R or W replicas
● How to pick values of R and W ...◆ to maximize availability/minimize latency?
» Low R and/or low W
◆ to ensure read sees last committed write?» R+W > N
October 22, 2019 EECS 491 – Lecture 14 9
![Page 10: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/10.jpg)
Latency/availability over consistency
October 22, 2019 EECS 491 – Lecture 14 10
Client1 Client2
k: x k: x k: x
Put(k, y)
k: y
Get(k)
N = 3, W = 1, R = 1
A B C
![Page 11: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/11.jpg)
Consistency over latency/availability
October 22, 2019 EECS 491 – Lecture 14 11
Client1 Client2
k: x k: x
Put(k, y)
k: y
Get(k)
k: y
How to tell which of R copies read is latest version?N = 3, W = 2, R = 2
A B C
![Page 12: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/12.jpg)
Vector clocks
● Store a vector clock with each key-value pair● What we have discussed previously:
◆ Vector with # of components = # of servers◆ Not scalable
● Dynamo’s adaptation of vector clocks:◆ List of (coordinator node, counter) pairs◆ Example: [(A, 1), (B, 3), …]
October 22, 2019 EECS 491 – Lecture 14 12
![Page 13: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/13.jpg)
Vector clocks in Dynamo
October 22, 2019 EECS 491 – Lecture 14 13
Client1 Client2
k: x k: x
Put(k, x)
N = 3, W = 2, R = 2
AB
C
Put(k, y)
([A, 1]) ([A, 1])
k: y k: y
([A, 1], [B, 1])([A, 1], [B, 1])
Get(k)
([A, 1], [B, 1]) > ([A, 1])
![Page 14: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/14.jpg)
Project 3● Focus first on correctness, then efficiency
● A peer may be acting as proposer, acceptor, and learner on same instance concurrently◆ Keep roles as separate as possible
● Okay to move on to part B if passing all handout test cases for part A
● Reminder about Honor Code
October 22, 2019 EECS 491 – Lecture 14 14
![Page 15: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/15.jpg)
Vector clocks in Dynamo
October 22, 2019 EECS 491 – Lecture 14 15
Client1
k: x
Put(k, x)
N = 3, W = 2, R = 2
AB
C
Put(k, y)
([A, 1])
k: y k: x
([A, 1])([B, 1])
Get(k) ([B, 1]) and ([A, 1])are in conflict!
How to recognize thatPut(k, x) à Put(k, y)?
Client2
![Page 16: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/16.jpg)
● How can vector clocks help in differentiating◆ Conflict?◆ Okay to garbage collect?
● When responding to a GET, Dynamo returns the vector clock for value returned
● Client includes vector clock in subsequent PUT
16
Vector clocks in Dynamo
October 22, 2019 EECS 491 – Lecture 14
![Page 17: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/17.jpg)
17
Automatic conflict resolution
v1 [(A,1)]
v2 [(A,1), (B,1)]
put handled by node Bwritten to B and C
put handled by node Awritten to A and C
v2 > v1, so Dynamo automatically drops v1 at COctober 22, 2019 EECS 491 – Lecture 14
![Page 18: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/18.jpg)
18
App-specific conflict resolution
v1 [(A,1)]
v3 [(A,1), (C,1)]
put handled by node C
put handled by node A
put handled by node B
v2 [(A,1), (B,1)]
v4 [(A,2), (B,1), (C,1)]
Client reads v2, v3; writes with [(A,1), (B,1), (C,1)]
v2 || v3, so client must perform reconciliation
October 22, 2019 EECS 491 – Lecture 14
![Page 19: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/19.jpg)
Dynamo’s client interface● Client interface in projects:
◆ Get(key) à value◆ Put(key, value)
● Get(key) à List of <value, context> pairs◆ Returns one value or multiple conflicting values◆ Context describes version(s) of value(s)
● Put(key, value, context)◆ Context indicates which versions this version
supersedes or mergesOctober 22, 2019 EECS 491 – Lecture 14 19
![Page 20: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/20.jpg)
● Many nodes may process Puts to same key◆ Version vectors may grow arbitrarily long
● Dynamo’s clock truncation scheme◆ Dynamo stores time of modification with each
version vector entry◆ When version vector > 10 nodes long, Dynamo
drops node that least recently processed key
● Problems with truncation?◆ False concurrency
20
Trimming version vectors
October 22, 2019 EECS 491 – Lecture 14
![Page 21: EECS 491 Introduction to Distributed Systems · Client1 Client2 k: x k: x k: x Put(k, y) k: y Get(k) N = 3, W = 1, R = 1 A B C. Consistency over latency/availability October 22, 2019](https://reader035.fdocuments.in/reader035/viewer/2022062219/5fb179396562be4fb82e2709/html5/thumbnails/21.jpg)
21
Impact of clock truncation
v1 [(A,1)]
v2 [(A,1), (B,1)]
put handled by node B
put handled by node A
October 22, 2019 EECS 491 – Lecture 14