Post on 21-Dec-2015
Routing Classification
Centralized Distributed
A Main controller updates all node’s routing tables.
Fault Tollerent.
Suitable for small networks.
Route computation shared among nodes by exchanging.
Widley used.
Routing Classification…
Static Adaptive
Routing based only on source and destination.
Current network state - Ignored
Adapt policy to time and trafic.
More attractive.
Ossilations in path.
Short falls of Static Routing
Dynamic networks are subjected to the following changes.Topologies changes, as nodes are added
and removedTraffic patterns change cyclicallyOverall network load changes
So, routing algorithms that assume that the network is static don’t work in this setting
Tackling Dynamic Networks
Periodic Updates? Routing traffic? When to update?
Adaptive Routing’s the Answer?
Reinforcement Learning
Agent Playing against a player- Chess and Tic-Tac-ToeLearning a Value Function
Learning Value Function
For K = 0.4 We have V(e) = 0.5 + 0.2 = 0.7
1
0.5
Temporal DifferenceV(e) = V(e) + K [ V(g) – V(e) ]
Exploration Vs Exploitatione and e*
Q-Routing Qx(d, y) is the time that node x estimates it will take to
deliver a packet to node d through its neighbor y
When y receives the packet, it sends back a message (to node x), containing its (i.e. y’s) best estimate of the time remaining to get the packet to d, i.e. t = min(Qy(d, z)) over all z neighbors( y )
x then updates Qx(d, y) by: [Qx(d, y)]NEW = [Qx(d, y)]OLD + K.(s+q+t - [Qx(d,y)]OLD )
Where • s = RTT from x to y• q = Time spent in queue at x• T = new estimate by y
DQ
Q-Routing…
x
y
to d
min(Qy(d, zi)) = 13;RTT = s = 11
messageQy(d, z1) = 25
21wd
…
…
d y 20
Dest Next Qx(d, y)
Qy(d, z2) = 17
Qy(d, ze) = 70
[Qx(d, y)] += (0.25).[(11+17) - 20 ]22
w
estimated RTT = 3
message to d
Short falls
Shortest path algorithm – better than Q Routing under low load.
Failure to converge back to shortest paths when network load decreases again.
Failure to explore new shortcuts
Short falls…
x
y
to dmessageQy(d, z1) = 25
Dest Next Qx(d, y)
…
d y 22
d w 21
…
Qy(d, z2) = 17
Qy(d, ze) = 70
w
message to d
20 Even if route via y reduces later,It never gets used untill route viaW gets cunjusted
Predictive Q-Routing
DQ = s+q+t - [Qx(d,y)]OLD
[Qx(d, y)]NEW = [Qx(d, y)]OLD + K.DQ Bx(d,y) = MIN[Bx(d,y), Qx(d,y)]
If(DQ < 0) //Path is improving DR = DQ/(currentTime – lastUpdatedTime) Rx(d,y) = Rx(d,y) + B.DR //Decrease in R
Else Rx(d,y) = G.Rx(d,y) //Increase of R
End If
lastUpdatedTime = currentTime
PQ-Routing Policy…
Finding neighbour y
For each neighbour y of x Dt = currentTime – lastUpdatedTime Qx-pred(d,y) = Qx(d,y) + Dt.Rx(d,y) Choose y with MIN[Qx-pred(d,y)]
PQ-Routing Results
Performs better than Q-Routing under low, high and varying network loads.
Adapts faster if “probing inactive paths” for shortcuts introduced.
Under high loads, behaves like Q-Routing. Uses more memory than Q-Routing.
Ant- Routing Stigmergy - Inspirations From Nature…
Sorts brood and food items Explore particular areas for food, and preferentially
exploits the richest available food source Cooperates in carrying large items Leaves pheromones on their way back Always finds the shortest paths to their nests or food
source Are blind, can not foresee future, and has very limited
memory
Ants Each router x in the network maintains for each
destination node d a list of the form: <d, <y1, p1>, <y2, p2>, …, <ye, pe>>, where y1, y2, …, ye are the neighbors of x, and p1 + p2 + …+ pe = 1
This is a parallel (multi-path) routing scheme
This also multiplies the number of degrees of freedom the system has by a factor of |E|
Ants…
Every destination host hd periodically generates an “ant” to a random source host hs
An “ant” is a 3-tuple of the form: < hd, hs, cost>
cost is a counter of the cost of the path the ant has covered so far
Ant Routing Example
1
3
2
4
0
Next Node 0 2 3 0 0.33 0.33 0.33 2 0.33 0.33 0.33 3 0.26 0.48 0.26
Dest. Node
4 0.33 0.33 0.33
Routing Table for 1
< 4,0,cost >
Ants: Updation
When a router x receives an ant < hd, hs, cost> from neighbor yi, it:
1. Updates cost by the cost of traversing the link from x to yi (i.e. the cost of the link in reverse)
2. Updates entry for host (<hd, <y1, p1>, <y2, p2>, …, <ye, pe>>)
pi = pi + p
p = k / cost, for some k
for j i, pj = pj
1 + p 1 + p
normalizing sum of probabilities to 1
Ants: Propagation
Two sub-species of ant: Regular Ants:
P( ant sent to yi ) = pi
Uniform Ants:
P( ant sent to yi ) = 1 / e
Regular ants use learned tables to route ants
Uniform ants explore randomly
Ants: Comparision
Regular Ants Uniform AntsExplore best paths very thoroughly; others hardly at all
Explore all paths equally
Propagate “bad news” extremely quickly, “good news” extremely slowly
Propagate “good” and “bad” news equally fast
Tends to find shortest paths Natural parallel (multi-path) routers
Converges to Q-Routing in a static network
Does not converge to Q-Routing
Q-Routing vs. Ants
Q-Routing only changes its currently selected route when the cost of that route increases, not when the cost of an alternate route decreases
Q-Routing involves overhead linear in the volume of traffic in the network; ants are effectively free in moderate traffic
Q-Routing cannot route messages by parallel paths; uniform ants can
Ants with Evoperation
Evaporation is a real life scenario - Where pheromone laid by real ants evaporates.
Link usage statistics are used to evaporate (E(x)).
It is the proportion of number of ants from node x over the total ants received by the current node.
)()(_
)(_1)(
0
xxPisendant
xsendantxE N
i
xixEiPiP ),()()(
xiN
xEiPiP
,
1
)()()(
Summary
Routing algorithms that assume a static network don’t work well in real-world networks, which are dynamic
Adaptive routing algorithms avoid these problems, at the cost of a linear increase in the size of the routing tables
Q-Routing is a straightforward application of Q-Learning to the routing problem
Routing with ants is more flexible than Q-Routing
Reference Boyan, J., & Littman, M. (1994). Packet routing in dinamically changing networks: A rein-
forcement learning approach. In Advances in Neural Information Processing Systems 6 (NIPS6), pp. 671-678. San Francisco, CA:Morgan Kaufmann.
Di Caro, G., & Dorigo, M. (1998). Two ant colony algorithms for best-eort routing in datagram networks. In Proceedings of the Tenth IASTED International Conference on Parallel and Distributed Computing and Systems (PDCS'98), pp. 541-546. IASTED/ACTA Press.
Choi, S., & Yeung, D.-Y. (1996). Predictive Q-routing: A memory-based reinforcement learning approach to adaptive trac control. In Advances in Neural Information Processing Systems 8 (NIPS8), pp. 945-951. MIT Press.
Dorigo, M., Maniezzo, V., & Colorni, A. (1996). The ant system: Optimization by a colony of cooperating agents. IEEE Transactions on Systems, Man, and Cybernetics-Part B, 26 (1), 29-41.