Symmetric Replication in Structured Peer-to-Peer Systems Ali Ghodsi, Luc Onana Alima, Seif Haridi.

Symmetric Replication in Structured Peer-to-Peer Systems

Ali Ghodsi, Luc Onana Alima, Seif Haridi

29 July 2005 [email protected] 2

Scope of the Talk

• Where to place replicas in a DHT?


Background

• DKS System (http://dks.sics.se/)

– Were initially replicating on successor-lists (Similar to Chord/Koorde)

– Turned out to have several problems…

http://dks.sics.se/


Outline

• Background• Proposed scheme: symmetric

replication• Extensions• Related Work & Summary


What is Replication on the Successor-List?

• In one sentence:

– Replicate node n’s data to its f closest

neighbors for replication degree f


Successor-list Example: Chord

A Chord-alike system, with nodes 0, 3, 5, 10, 11

0 12

1514

13 3

12

11

4

5

6

9 87

10



With a replication factor of f=1

0 12

1514

13 3

12

11

4

5

6

9 87

10

Data: 12, 13, 14, 15, 0

Data: 1, 2, 3

Data: 4, 5

Data: 11

Data: 6, 7, 8, 9, 10




0 12

1514

13 3

12

11

4

5

6

9 87

10

Data: 12, 13, 14, 15, 0

Data: 1, 2, 3

Data: 4, 5

Data: 11

Data: 6, 7, 8, 9, 10

Node 0: 12, 13, 14, 15, 0

Node 3: 1, 2, 3

Node 5: 4, 5

Node 10: 6, 7, 8, 9, 10

Node 11: 11




0 12

1514

13 3

12

11

4

5

6

9 87

10

Data: 12, 13, 14, 15, 0

Data: 1, 2, 3

Data: 4, 5

Data: 11

Data: 6, 7, 8, 9, 10

Node 0: 12, 13, 14, 15, 0

Node 3: 1, 2, 3

Node 5: 4, 5

Node 10: 6, 7, 8, 9, 10

Node 11: 11

Node 11: 11

Node 0: 12, 13, 14, 15, 0

Node 3: 1, 2, 3

Node 5: 4, 5

Node 10: 6, 7, 8, 9, 10




0 12

1514

13 3

12

11

4

5

6

9 87

10

Data: 12, 13, 14, 15, 0

Data: 1, 2, 3

Data: 4, 5

Data: 11

Data: 6, 7, 8, 9, 10

Node 0: 12, 13, 14, 15, 0

Node 3: 1, 2, 3

Node 5: 4, 5

Node 10: 6, 7, 8, 9, 10

Node 11: 11

Node 11: 11

Node 0: 12, 13, 14, 15, 0

Node 3: 1, 2, 3

Node 5: 4, 5

Node 10: 6, 7, 8, 9, 10

Node 11: 11

Node 0: 12, 13, 14, 15, 0

Node 3: 1, 2, 3

Node 5: 4, 5

Node 10: 6, 7, 8, 9, 10


Problem: Dynamism

What happens in case of leaves or failures?f=4

0 12

1514

13 3

12

11

4

5

6

9 87

10

Data: 12, 13, 14, 15, 0

Data: 1, 2, 3

Data: 4, 5

Data: 11, 10, 9, 8, 7, 6

Node 0: 12, 13, 14, 15, 0

Node 3: 1, 2, 3

Node 10: 6, 7, 8, 9, 10

Node 11: 11

Node 11: 11

Node 0: 12, 13, 14, 15, 0

Node 5: 4, 5

Node 10: 6, 7, 8, 9, 10

Node 11: 11

Node 3: 1, 2, 3

Node 5: 4, 5

Node 10: 6, 7, 8, 9, 10


Problem: Dynamism

What happens in case of leaves or failures?f=4

0 12

1514

13 3

12

11

4

5

6

9 87

10

Data: 12, 13, 14, 15, 0

Data: 1, 2, 3

Data: 4, 5

Data: 11, 10, 9, 8, 7, 6

Node 0: 12, 13, 14, 15, 0

Node 3: 1, 2, 3

Node 11: 11,10,9,8,7,6

Node 11: 11,10, 9, 8, 7, 6

Node 0: 12, 13, 14, 15, 0

Node 5: 4, 5

Node 11: 11Node 3: 1, 2, 3

Node 5: 4, 5


Problem: Dynamism

Need to move items to restore replication degree!

0 12

1514

13 3

12

11

4

5

6

9 87

10

Data: 12, 13, 14, 15, 0

Data: 1, 2, 3

Data: 4, 5

Data: 11, 10, 9, 8, 7, 6

Node 0: 12, 13, 14, 15, 0

Node 3: 1, 2, 3

Node 11: 11,10,9,8,7,6

Node 11: 11,10, 9, 8, 7, 6

Node 0: 12, 13, 14, 15, 0

Node 5: 4, 5

Node 11: 11,10, 9, 8, 7, 6

Node 3: 1, 2, 3

Node 5: 4, 5

Get items

Data: 4, 5

Node 5: 4, 5


Problem: Dynamism


0 12

1514

13 3

12

11

4

5

6

9 87

10

Data: 12, 13, 14, 15, 0

Data: 1, 2, 3

Data: 4, 5

Data: 11, 10, 9, 8, 7, 6

Node 0: 12, 13, 14, 15, 0

Node 3: 1, 2, 3

Node 11: 11,10,9,8,7,6

Node 11: 11,10, 9, 8, 7, 6

Node 0: 12, 13, 14, 15, 0

Node 5: 4, 5

Node 11: 11,10, 9, 8, 7, 6

Node 3: 1, 2, 3

Node 5: 4, 5

Get items

Data: 1, 2, 3

Node 5: 4, 5

Node 3: 1, 2, 3


Problem: Dynamism


0 12

1514

13 3

12

11

4

5

6

9 87

10

Data: 12, 13, 14, 15, 0

Data: 1, 2, 3

Data: 4, 5

Data: 11, 10, 9, 8, 7, 6

Node 0: 12, 13, 14, 15, 0

Node 3: 1, 2, 3

Node 11: 11,10,9,8,7,6

Node 11: 11,10, 9, 8, 7, 6

Node 0: 12, 13, 14, 15, 0

Node 5: 4, 5

Node 11: 11,10, 9, 8, 7, 6

Node 3: 1, 2, 3

Node 5: 4, 5

Get itemsData: 12, 13, 14, 15, 0

Node 5: 4, 5

Node 3: 1, 2, 3

Node 0: 12,13,14,15,0


Problem: Dynamism


0 12

1514

13 3

12

11

4

5

6

9 87

10

Data: 12, 13, 14, 15, 0

Data: 1, 2, 3

Data: 4, 5

Data: 11, 10, 9, 8, 7, 6

Node 0: 12, 13, 14, 15, 0

Node 3: 1, 2, 3

Node 11: 11,10,9,8,7,6

Node 11: 11,10, 9, 8, 7, 6

Node 0: 12, 13, 14, 15, 0

Node 5: 4, 5

Node 11: 11,10, 9, 8, 7, 6

Node 3: 1, 2, 3

Node 5: 4, 5

Get items

Data: 10, 9, 8, 7, 6


Problem: Dynamism

• Every join/leave/failure requires: (f) messages, to restore a replication

degree of f

– Need coordination of the restoration• What if the coordinator fails?


Problem: Performance & Security

• How do we contact 2nd, 3rd, 4th… replica? – Have to contact the 1st replica!

• The 1st replica is– Bottleneck – Can fail– Can be malicious (distributed voting

useless)


Outline




Symmetric Replication

Goal of Symmetric Replication:– Simple algorithms

• Each join/leave should require O(1) messages

– Enable concurrent requests• Load-balancing without a bottleneck

– Increase security• Use distributed voting

– Add proximity information• Route through closest replica


Symmetric Replication

Idea:– For replication degree f, partition the

identifier space into m equivalence classes:

• Use the congruence classes modulo f

– The cardinality of each class is f

– Each node replicates the equivalence class of every identifier it is responsible for


Symmetric replication

Replication degree f=4, Id={0,…,15}• 4 equivalence classes:

0 12

1514

13 3

12

11

4

5

6

9 87

10

Data: 15, 0

Data: 1, 2, 3

Data: 4, 5

Data: 14, 13, 12, 11

Data: 6, 7, 8, 9, 10

0, 4, 8, 12

1, 5, 9, 13

2, 6, 10, 14

3, 7, 11, 15


Symmetric replication


0 12

1514

13 3

12

11

4

5

6

9 87

10

Data: 15, 0

Data: 1, 2, 3

Data: 4, 5

Data: 14, 13, 12, 11

Data: 6, 7, 8, 9, 10

0, 4, 8, 12

1, 5, 9, 13

2, 6, 10, 14

3, 7, 11, 15

Data: 10, 9, 8, 7

Data: 11, 12

Data: 13, 14, 15

Data: 0, 1

Data: 2, 3, 4, 5, 6

Data: 6, 5, 4, 3

Data: 2, 1, 0, 15Data: 7, 8

Data: 3, 4

Data: 9, 10, 11

Data: 5, 6, 7

Data: 12, 13

Data: 8, 9

Data: 14, 15, 0, 1, 2

Data: 10, 11, 12, 13, 14


Symmetric replication: joins!


0 12

1514

13 3

12

11

4

5

6

9 87

10

Data: 15, 0

Data: 1, 2, 3

Data: 4, 5

Data: 14, 13,

Data: 6, 7, 8, 9, 10

0, 4, 8, 12

1, 5, 9, 13

2, 6, 10, 14

3, 7, 11, 15

Data: 10, 9

Data: 11, 12

Data: 13, 14, 15

Data: 0, 1

Data: 2, 3, 4, 5, 6

Data: 6, 5

Data: 2, 1Data: 7, 8

Data: 3, 4

Data: 9, 10, 11

Data: 5, 6, 7

Data: 12, 13

Data: 8, 9

Data: 14, 15, 0, 1, 2

Data: 10, 11, 12, 13, 14

Data: 11, 12

Data: 7, 8

Data: 3, 4

Data: 0, 15

Get items

Data: 11, 12, 7, 8, 3, 4, 0, 15


Symmetric replication: failures

0 12

1514

13 3

12

11

4

5

6

9 87

10

Data: 15, 0

Data: 1, 2, 3

Data: 4, 5

Data: 14, 13,

Data: 6, 7, 8, 9, 10

0, 4, 8, 12

1, 5, 9, 13

2, 6, 10, 14

3, 7, 11, 15

Data: 10, 9

Data: 11, 12

Data: 13, 14, 15

Data: 0, 1

Data: 2, 3, 4, 5, 6

Data: 6, 5

Data: 2, 1Data: 7, 8

Data: 3, 4

Data: 9, 10, 11

Data: 5, 6, 7

Data: 12, 13

Data: 8, 9

Data: 14, 15, 0, 1, 2

Data: 10, 11, 12, 13, 14

Successor has to fetch [11], [12] from closest

replica


Restoring Replication Degee: failures

• Assume a node responsible for (10, 14) fails– Classes [10]={1, 10, 19} [14]={5, 14, 23}

• We decide to fetch replicas from (1, 5)

• Use restricted broadcast to fetch (1, 5)– [ansary-iptps02, ghodsi-pdcs02]– Reliable, O(log R) time complexity to reach R nodes

• Several nodes might be responsible for (1, 5)– Given uniform distribution of node identifiers

E[peers in range (1,5)] = 1


Outline




Extensions to Symmetric Replication

•Augment the routing information to:–Add proximity awareness

•Example: node 2, f=3 [2]={2, 5, 8}

Routing Table: node 2

Finger 1 Succ(2+20)

Finger 2 Succ(2+21)

Finger 3 Succ(2+22)

Finger 4 Succ(2+23)


Extensions to Symmetric Replication

•Augment the routing information to:–Add proximity awareness

•Example: node 2, f=3 [2]={2, 5, 8}

Routing Table: node 2

Finger 1 Succ(2+20

)Succ(5+20

)Succ(8+20

)

Finger 2 Succ(2+21

)Succ(5+21

)Succ(8+21

)

Finger 3 Succ(2+22

)Succ(5+22

)Succ(8+22

)

Finger 4 Succ(2+23

)Succ(5+23

)Succ(8+23

)


Extensions: Erasure Codes

– Symmetric Replication with Erasure Codes

– Encode data into n blocks, only k needed to reconstruct original data

• Example: n=10, k=5

– Replication degree=n (# encoded blocks)

fetch any k blocks to reconstruct data


Outline




Related Work

• Several systems use multiple hash functions– Content Addressable Networks (CAN)– Tapestry – Simple Load Balancing in Chord


Problem with Multiple Hash Functions

• Hash1(“music.mp3”)=11

• Hash2(“music.mp3”)=2

0 12

1514

13 3

12

11

4

5

6

9 87

10

Data: 15, 0

Data: 1, 2, 3

Data: 4, 5

Data: 14, 13,

Data: 6, 7, 8, 9, 10

Node 12 crashed,We need the inverse of

Hash1-

1(11)=“music.mp3”


Summary

• A general replica placement scheme– Algorithms for join, leave, fail, insert in paper

• O(1) messages to restore replication degree f instead of (f)

• Can access any replica without a bottleneck:– Concurrent Requests– Load balance queries– Distributed Voting

• Current Use:– Implemented in DKS, http://dks.sics.se/

http://dks.sics.se/

http://dks.sics.se/

http://dks.sics.se/


Thank you for listening!

Symmetric Replication in Structured Peer-to-Peer Systems Ali Ghodsi, Luc Onana Alima, Seif Haridi.

Documents

Transcript of Symmetric Replication in Structured Peer-to-Peer Systems Ali Ghodsi, Luc Onana Alima, Seif Haridi.