LAND:Locality Aware Networks for Distributed Hash Tables

Dahlia MalkhiThe Hebrew University of Jerusalem

Joint work with:Ittai Abraham and Oren Dobzinski

2

Motivation

Today’s Internet:– Many lightweight clients (web browsers)– Relatively simple servers (HTTP)

Client-Server paradigm is suited for a world of thin clients which do not have a lot of bandwidth and computational power.

Tomorrow’s Internet ?– Most devices will have enough bandwidth and

CPU to become both a client and a server (peer).– Users will have an active network presence.

3

Challenges

New Distributed Storage and Retrieval Services will face many challenges:

Scalability – number of users constantly increases. Dynamism – unlike today’s web servers, peers will be

constantly joining and leaving. Congestion – Access to data is non-uniform and may

cause hot spots. Fault tolerance – availability and reliability of data. Efficiency – systems should be resource efficient and

provide the best performance (more on this later).

4

Overlay networks and distributed data structures

Hash tables: store and lookup by object id

Quorum systems: global search Prefix lookup SQL Google ?

6

Viceroy Overview

Viceroy, the first constant-degree distributed hash table [Malkhi, Naor, Ratacjzak, PODC 02]

LAND, the first peer-to-peer network and lookup algorithm that has worst case constant distortion [Abraham, Malkhi, Dobzinski SODA 2004]

A generic overlay network approach with implicit load balancing [Abraham, Awerbuch, Azar, Bartal, Malkhi, Pavlov, IPDPS 03]

A publish-subscribe mechanisms for scale-free graphs based on probabilistic quorums [Abraham, Malkhi, DISC 03]

Small-world DHTs on planar metrics [Abraham, Malkhi, 2003] An optimal asynchronous resource discovery scheme [Abraham,

Dolev, PODC 03] Investigation of user privacy and anonymity [Bickson, Malkhi, 2003] An efficient, localized scheme for estimating the number of

nodes in a dynamic network [Horowitz, Malkhi, IPL 2003]

7

Overlay Networks for Finding Nearest Copy of Data

Nodes construct an overlay layer that allows to use new network architectures and services.

A Content Addressable Network allows to route to the target node by examining the object’s id.

If multiple copies of the same object exist then the closest copy should be accessed.

Complexity measures for Overlay Networks:

– Number of hops from source node to target node. – Degree of the overlay network. – Amount of additional memory needed per object. – Adaptability: number of nodes that change their state each

time a peer joins/leaves the system.– Load on nodes related to the locating task

Previous worksNode-degree

Dilation Congestion Topology

Chord(MIT)

log(n) log(n) log(n)/n Hypercube

Tapestry(Berkeley)

log(n) log(n) log(n)/n Hypercube

CAN(ICSI)

d dn(1/d) dn(1/d)/n d-dim. tourus

Smallworlds

3 log2(n) log2(n)/n Cube connected cycle

Viceroy 3 log(n) log(n)/n Butterfly

9

Locality

Suppose a new DHT ensures each object will be found in 4 hops.

So a lookup could begin in Boston and go to Brazil, New Zealand, France and finally New York.

For some applications this is not a desired outcome.

10

Network models

The Internet– Fully connected weighted graph– Weight = ping latency (?)

Internet with geometric coordinates– Distance = geographic distance

Mobile network– Geometric space with limited transmission range

Arbitrary graphs

11

Distortion

Let c(s, t) be the distance from s to t Let s=x1 x2 …xk=t the route from s to t

Distortion is the ratio betweenc(x1, x2) + … + c(xk-1, xk)andc(s, t)

12

The Model Cost function c that forms a metric

– c(x,y) ≥ 0 (positive), c(x,x)=0 (reflexive), c(x,y)=c(y,x) (symmetric)

– Triangle inequality: c(x,y) + c(y,z) ≥ c(x,z)

Minimal distance between peers is 1. N(x,r) denote the set of nodes at distance <r

from x. Growth Bounded Metric:

Actually, assume uniform density first

r2r rxNrxN ,2,

13

LAND origins and related work

Based on the scheme of Plaxton, Rajaraman, Richa. “Accessing nearby copies of replicated objects in a distributed environment.” Theory of Computing Systems, 1999.

PRR ensures that the expected distortion is constant.

Tapestry and Pastry DHT’s are both based on the basic static PRR scheme. They enhance PRR by handling dynamic changes in the network.

More..

14

LAND architecture

A set of objects A. Objects can be stored on any

node.– Multiple nodes can keep a replica

of each object. Uniformly distributed hash

function h(A). Uniformly selected node

identifiers Nodes keep

transient routing information about objects.

Resides hereH

ashe

d-ho

me

here

?

15

Identifiers and links

Each node has n=log(N) identifier 2-bit digits Node a1a2…an has n links

– Link k `fixes’ k’th digit– Connects to closest node with identifier

a1a2…ak-1[0-3]*

Link k property:– Found with probably 1/4k

– Expected to be found within a ball with 4k nodes– Goes to distance 2k

16

Publish and lookup

Prefix-routing: fix one digit at a time Publish object A at node t:

– Leave reference “A;t” at each node en-route

Lookup A: route until reference to A found

Route distance: At most network diameter = 21 + 22 + … + 2n

17

Example of Publish:

t w1 w2

w3

node t publishes Objh(Obj)=101t=***w1=1**w2=10*w3=101

18

Problem 1:

Link distance could be more than expected

Solution: Emulate a shadow node– A shadow node that fixes digit k has links

fixing digit k+1

– If a1a2..ak-1d* not found at distance 2k then look for a1a2..ak-1d[0-3]* and within 2(k+1)

19

Expected O(n) number of shadow nodes

– Probability of emulating link k at most (1-1/4k) 4^k < e-1

– Bk+i – expected number of shadow nodes with prefix k+i

– E[Bk+i] = E[Bk+i|Bk+i-1] = 2e-1E[Bk+i-1] = (2/e) i

– Total expected number of shadow nodes starting with link k is constant

20

Problem 2:

Unbounded distortion; if target is very close, price is too high

Solution: `publish’ links– Step k in a publish route places reference

in appropriate nodes in bigger neighborhood

21

Publish links

Denote 2k ≥ c(s,t)– k is the first such index

Step k of lookup route is at most 2(k+1) away (denote xk)

Step k of publish route is at most 2(k+1) away (denote wk)

c(xk , wk) ≤ 3*2(k+1)

Publish a k’th step to all nodes within distance 2(k+1)+2 with identifer matching k-prefix , so xk will contain a reference to wk

22

Example of Publish:

t w1 w2

w3

node t publishes Objh(Obj)=101t=***w1=1**w2=10*w3=101

For example:w1 publishes to nodes with id=1* within a distance proportional to distance(t, w1)

23

x0

Lookup Algorithm

t w1 w2

w3

x x1 x2 x3

x4

24

Distortion

Route from s to xk

Plus distance from xk to wk

Plus route from wk to t

constant factor over distance from s to t

Distortion can be made close to 1 by increasing range of publish links

25

Summary of LAND properties

Guaranteed (small) constant distortion Expected logarithmic node-degree Simple analysis Amenable to dynamic deployment

26

Dynamic maintenance

Find closest node to x:– From any node y, let Sn = {y}– Recursively, set Sk-1 = closest node among

incoming prefix-(k-1) links into Sk

– Closest is S1

Correctness:– Let s be closest to x; route from s to y– Step k is within 2(k+1) distance– If Sk is closest to x with k-prefix match to Sn is within

2(k+1) distance, then so is Sk-1

27

Neighbor finding

From closest, route k steps, then back-track incoming links two steps, to find prefix-(k-2) links

Correctness:– Node k is 2(k+1) distance away– Incoming links are 2k+2 away, covering 2(k-2)

away

28

Back to growth-bound model N(x,r) denote the set of nodes at distance <r

from x. Growth Bounded Metric:

r2r

rxNrxN ,2,

29

Setting node Identifier and Level.

Set a radix B Let M such that BM = N. Id has M digits of radix B. Denote A i (x) as the ball around x with α

Bi nodes (e-α B<1)– A i (x) has constant number of expected

nodes with any specific length-i identifier.

30

Network links

Each node has M initial routers Router in level k has links to routers in level

k+1 :– router u with id a1,a2,…,ak,ak+1…aM and level k,

maintains three types of links:

1. Neighbor – for each digit b in [0…B-1] a link to the closest node with id

beginning with a1,a2,a3,…,ak,b and level k+1 inside the ball A k+1 (u).

2. Publish - a link to all nodes with id beginning with a1,a2,a3,…,ak and

level k+1 inside the ball A k+5 (u).

31

Example of Publish:

t w1 w2

w3

Super-node t publishes Objh(Obj)=abct=***w1=a**w2=ab*w3=abc

For example:w1 publishes to routers with level 2 and id=ab* inside theball A6(w1)

32

Enforcing Locality withShadow Nodes Recall: Neighbor – for each digit b in B a link to the

closest node with id a1,a2,a3,…,ak,b and level k+1. Want all neighbor links of a level k node u to be

inside Ak+1(u). For any b, if no b’th neighbor is in Ak+1(u) then u

emulates a shsadow node v with id a1,a2,a3,…,ak,b and level k+1.

Node u establishes all of this shadow router’s network links. Including v’s neighbor links.

Recursively this process continues until all shadow nodes have all their links either close enough or emulated.

33

Variations and extensions

Two-tier architecture, constant expected node degree

Content Addressable Networks Fault Tolerance

34

Two-hop stretch-3 DHT

Each node v has identifier h(v) – h() has sqrt(N) different values

Node v has links to:– log(N)*sqrt(N) closest nodes, so one of each value w.h.p– All nodes u with h(u)=h(v)

Routing from s to t in two hops:– Find node w with h(w)=h(t)– Find t

Stretch:– c(s, w) + c(w, t) ≤ c(s, t) + 2c(s, t)

35

Analysis:Balls

Recall Ai(x) is the smallest ball around x with α Bi M nodes (e-αB<1).

Suppose y in Ai(x) then:

– Ai(y) in Ai+1(x)

– Ai(x) in Ai+1(y)y

xAi(x)

Ai+1(x)

Ai(y)

Ai+1(y)

36

Proof of Ai(y) in Ai+1(x)

Ai(y) is less than N(2ai(x),x) because it contains the ball Ai(x)

N(2ai(x),y) is less than N(4ai(x),x) by simple distances

N(4ai(x),x) is less than Ai+1(x) due to the growth restriction and the way we chose B

Proving Ai(x) in Ai+1(y) is very similar

37

Growth of Balls

Recall Ai(x) is the smallest ball around x with α Bi nodes (e-αB<1).

Let ai(x) denote the radius of Ai(x)

ai+1(x) ≤ maxgrow ai(x)

ai+1(x) ≥ mingrow ai(x)

Ai(x)2log

2log

Bmingrow

Bmaxgrow

Ai+1(x)

ai+1(x) ai(x)

38

Analysis:Distortion The initial node is x looking for Obj. x0=s is the closest super node. w0=t is the closest super node holding Obj. w0,w1,w2,.. is the sequence of nodes used to publish

OBj. x0,x1,x2,.. is the set of nodes fixing the bits to reach

h(Obj), node xi has level i. Xk is the first node that has a reference to OBj

published by node wk-1. Need to find bound on path x,x0,x1,x2,…,xk,wk-1,wk-2,

…,w2,w1,w0=t compared to c(x,t).

39

Analysis:Distortion

For every i:– xi in Ai+1(s) and wi in Ai+1(t)

– The path from s=x0 to xi is at most

– Similarly for the path from t=w0 to wi

If t in Ak(s) then xk contains a reference to Obj

Distortion is

)(1

21 sa

mingrow

mingrowi

22

31

41 mingrow

mingrow

maxgrowmingrow

40

Analysis:Expected Degree

Expected number of virtual nodes emulated by a node is constant.

Expected number of publish links is constant. Expected degree of regular nodes is

constant. Expected degree of super-nodes is

logarithmic.

41

Expected number of emulated nodes is

The probability that a random node will be a neighbor link is 1/(Bl+1M).

The probability that a neighbor link will be found inside Al+1(u) is

1

1 Be

1

1

11 1 1

kB M

ke

B M

42

Expected number of emulated nodes is

Let bl+i be the number of virtual nodes of level l+i. So bl=1.

E(bl+i | bl+i-1) = bl+i-1 B e-α.

Thus E(bl+i) = E(bl+i-1) B e-α.

By induction: E(bl+i) = (B e-α)i

Number of virtual nodes:

BeBebE

i

i

lMiil 1

1

00

1

1 Be

53

A Generic Scheme for Building Overlay Networks in Adversarial Scenarios

Ittai Abraham (HUJI), Baruch Awerbuch (JHU),Yossi Azar (TAU), Yair Bartal (HUJI),Dahlia Malkhi (HUJI), Elan Pavlov (HUJI)

55

Dynamic Model Suppose the set of node in the network is

dynamically evolving. Peers in the DHT are constantly leaving and

joining the system. Join: a new node wants to join the system, it

initially has access to an existing node. Leave: a node departures from the system,

this departure can either be graceful (performing any necessary cleanup operation) or sudden.

Low degree overlay network helps reduce overhead in the event of join/leave.

This process may cause network imbalance.

56

Coping with Imbalance Solution 1 [FS 01, FSGKS 02]: Assume population is

always in between n and ½n (censorship). Solution 2 [Chord 01, CAN 01]: Execute periodic global

overhaul operations for rebalancing. Problem: global operations are costly and may totally

shut down the service. Impractical for large systems. Solution 3 [Pastry 01, Tapestry 01]: Assume population

change is a random process that maintains the initial randomness.

Problems:– DHT systems may have hot spots, and many nodes entering

may use the same access node.– Failures tend to be correlated.– A malicious adversary may try to disrupt the network by causing

imbalance.

57

Load Balancing Against an Adversary In this work we allow an adversary to

adaptively choose:– The order of join and leave events.– For leave events: which node to remove.– For join events: what is the access node of the

newly added node. Against such an adversary we employ load

balancing upon arrival and departure. After each event the overlay network

executes protocols for rebalancing the network.

58

Problem Statement

Devise an overlay network and join leave protocols, with the following properties:– Efficient decentralized routing.– Low cost for rebalancing join and leave

events against an adversary.

59

Generic Solution for Child-Neighbor Commutative Families

Consider a set of graphs G1, G2, G3, … With mapping pi from Gi+1 on to Gi. Denote child function ci(u)={v| pi(v)=u} Denote neighbor function n(u)={v|

(u,v)E} Child-Neighbor commutative property:

For every u: n(c(u))=c(n(u))

60

The Hypercube as an example The hypercube Gi :has 2^i nodes each node’s

id is a binary string of length i. A node in Gi :has links to the i nodes that

have only one bit different in their id. The child function is c(x)={x0,x1} Example of Child-Neighbor commutative

property for node 10 in G2. n(10)={11,00}, c(n(10))={111,110,001,000} c(10)={100,101},

n(c(10))={000,110,101,001,110,100}

61

The Dynamic Graph For this talk we focus on the Dynamic

Hypercube (in paper de Bruijn, Butterfly). Start with two nodes with id 1, 0. Split: change node with id x into nodes x0, x1. Merge: change two twin nodes x0, x1 into node

with id x. Not all of the nodes n(x) exist in the network. Edges: A node x connects to all the nodes

whose id is a prefix of n(x) or n(x) is a prefix of their id.

For example: Node 110 would link to all nodes in the dynamic graph whose id is a prefix of {010, 100, 111} for instance: 01, 100101, 1000, and 111.

62

Example of split Operations on the Dynamic Hypercube

Start with 0,1. Split 1 to 10,11 Split 10 to 100,101 Split 0 to 00,01 Split 11 to 110,111

1010

11

101

10000

01 110

111

63

Tree View of Dynamic Graphs

Leafs of the tree represent current nodes

Inner nodes in the tree represent nodes that were split

000

111

110

101

00

110 111

001

100

011

010

000 001 010 011 100 101

00

Example: merge of 000, 001 into 00

64

Dynamic Graphs

Goal: efficient routing (logarithmic in number if nodes).

Level of a node – number of bit in identifier (distance from the root).

Global gap – difference between smallest level and biggest level.

Local gap of a node– difference in the levels of the nodes neighbors.

We show that with a logarithmic global gap efficient (logarithmic) routing is possible.

Constant local gap implies a logarithmic global gap.

65

Deterministic Balancing Strategy

Goal: maintain a local gap of 1. Node addition: starting from the access node

using the network links, reach a node with the smallest level and split it.

Node removal: starting form the removed node, reach a node with the biggest level and merge it with its twin.

If the local gap was 1, it will remain 1. Since the global gap is log n, requires to

examine log n nodes at most.

66

Randomized Balancing Strategy Goal: maintain a global gap of O(1) w.h.p. Node addition: Choose log n different locations reach

them using the network routing, and split the node with the smallest level.

Node removal: Choose log n different locations and merge the node with the biggest level.

This process is similar to throwing balls into bins, by choosing log n different bins, adding into the least full bin, and removing from the most full.

Requires to examine log n nodes per operation. Analysis: M. Mitzenmacher, A. Richa and R.

Sitaraman. The power of two random choices: a survey of techniques and results.

67

In the paper

Dynamic de Bruijn, and butterfly networks with logarithmic routing and constant degree.

Logarithmic number of messages sent and constant number of nodes that change state per leave/join event.

Generalize to any Child-Neighbor commutative family of graphs (like grids).

68

Related Work M. Naor and U. Weilder present a generic

way to emulate continuous graphs by discrete graphs [MW - SPAA 03].

Use rapidly mixing random walks on a dynamic de Bruijn network in order to establish probabilistic quorums in dynamic settings [A,Malkhi – DISC 03].

Independently, other de Bruijn based P2P networks: [MW], [D2B: Fraigniaud, Gauron], [Koorde: Kaashoek, Karger]

69

Open Questions Deal with multiple, parallel joins and

leaves without waiting till the network rebalances. Use online analysis to find the rate in which the network remains balanced.

Find ways to avoid network partitions. Find other graphs that are applicable to

the generic construction (random graphs).

70

Questions ?

Email: ittaia@cs.huji.ac.il

71

x0

Lookup Algorithm

t w1 w2

w3

x s=x1 x2 x3

x4