Tomography-based Overlay Network Monitoring and its

Applications

Joint work with David Bindel, Brian Chavez, Hanhee Song, and Randy H. Katz

UC Berkeley

Yan Chen

Motivation• Applications of end-to-end distance

monitoring– Overlay routing/location – Peer-to-peer systems – VPN management/provisioning– Service redirection/placement – Cache-infrastructure configuration

• Requirements for E2E monitoring system– Scalable & efficient: small amount of probing traffic– Accurate: capture congestion/failures– Incrementally deployable– Easy to use

Existing Work• Static estimation:

– Global Network Positioning (GNP)• Dynamic monitoring

– Loss rates: RON (n2 measurement)– Latency: IDMaps, Dynamic Distance Maps, Isobar

• Latency similarity under normal conditions doesn’t imply similar losses !

• Network tomography– Focusing on inferring the characteristics of

physical links rather than E2E paths– Limited measurements -> under-constrained

system, unidentifiable links

Problem FormulationGiven n end hosts on an overlay network and

O(n2) paths, how to select a minimal subset of paths to monitor so that the loss rates/latency of all other paths can be inferred.

• Key idea: select a basis set of k paths that completely describe all O(n2) paths (k «O(n2)) – Select and monitor k linearly independent paths to compute

the loss rates of basis set– Infer the loss rates of all other paths– Applicable for any additive metrics, like latency

End hosts

Overlay Network Operation Center

topology

measurements

Path Matrix and Path Space

• Path loss rate p, link loss rate l

• Totally s links, path vector v

1..

)1(1jvtsj

jlp

xvxvlvp Ts

j

jj

s

j

jj

11

)1log()1log(

1,}1|0{, rsr bGbGx• Path matrix G

– Put all r = O(n2) paths together

– Path loss rate vector b

A

p

B

lj

Sample Path Matrix

• x1 - x2 unknown => cannot compute x1, x2

• Set of vectorsform null space

• To separate identifiable vs. unidentifiable components: x = xG + xN

• All E2E paths are in path space, i.e., GxN = 0

01

1

2)(

2/2/

100

011

2)(

21

2

1

1

321

xxx

bbb

xxxx

N

G

GNG GxGxGxGxb

111100011

G

3

2

1

3

2

1

bbb

xxx

G

A

D

C

B

1

2

3b1

b2

b3

(1,-1,0)

link 2

link 1

link 3

(1,1,0)

row space(measured)

null space(unmeasured)

T]011[

1 2

1

2 3

1’

Real links (solid) and overlaypaths (dotted) going through them

Virtualization

Virtual links

1’ 2’

1

23

1’2’

4

Rank(G)=1

Rank(G)=2

Rank(G)=3

3’

4’

Intuition through Topology Virtualization

1

1 2

12

3

• Virtual links: minimal path segments whose loss rates uniquely identified

• Can fully describe all paths

• xG : similar forms as virtual links 5

Algorithms• Select k = rank(G) linearly independent paths

to monitor– Use rank revealing decomposition, e.g., QR with

column pivoting– Leverage sparse matrix: time O(rk2) and memory

O(k2)• E.g., 10 minutes for n = 350 (r = 61075) and k = 2958

• Compute the loss rates of other paths

– Time O(k2) and memory O(k2)

GGG GxbbxGx then,with Solve

How much measurement saved ?

• k « O(n2) ?• For power-law Internet topology, M nodes, N end

hosts– There are O(M) links and N > M/2 (with proof)– If n = O(N), overlay network has O(n) IP links, k = O(n)

When a Small Portion of End Hosts on Overlay

• Internet has moderate hierarchical structure [TGJ+02]– If a pure hierarchical structure (tree): k = O(n)– If no hierarchy at all (worst case, clique): k = O(n2)– Internet should fall in between …

For reasonably large n, (e.g., 100), k = O(nlogn)BRITE 20K-node hierarchical topology Mercator 284K-node real router topology

Practical Issues• Topology measurement errors tolerance

– Care about path loss rates than any interior links– Poor router alias resolution present show multiple

links for one => assign similar loss rates to all the links

– Unidentifiable routers => ignore them as virtualization

• Topology changes– Add/remove/change one path incurs O(k2) time– Topology relatively stable in order of a day =>

incremental detection

Evaluation• Simulation

– Topology• BRITE: Barabasi-Albert, Waxman, hierarchical: 1K – 20K nodes• Real router topology from Mercator: 284K nodes

– Fraction of end hosts on the overlay: 1 - 10%– Loss rate distribution (90% links are good)

• Good link: 0-1% loss rate; bad link: 5-10% loss rates• Good link: 0-1% loss rate; bad link: 1-100% loss rates

– Loss model: • Bernouli: independent drop of packet• Gilbert: busty drop of packet

– Path loss rate simulated via transmission of 10K pkts• Metric: path loss rate estimation accuracy

– Absolute/relative errors– Lossy path inference

Areas and Domains # of hosts

US (40)

.edu 33

.org 3

.net 2

.gov 1

.us 1

Interna-tional (11)

Europe (6)

France 1

Sweden 1

Denmark 1

Germany 1

UK 2

Asia (2)Taiwan 1

Hong Kong 1

Canada 2

Australia 1

Experiments on Planet Lab• 51 hosts, each from

different organizations– 51 × 50 = 2,550 paths

• Simultaneous loss rate measurement– 300 trials, 300 msec each– In each trial, send a 40-byte

UDP pkt to every other host• Simultaneous topology

measurement– Traceroute

• Experiments: 6/24 – 6/27– 100 experiments in peak

hours

• Loss rate distribution

• Accuracy– On average k = 872 out of 2550– Absolute error |p – p’|:

• Average 0.0027 for all paths, 0.0058 for lossy paths– Relative error

lossrate [0, 0.05)

lossy path [0.05, 1.0] (4.1%)

[0.05, 0.1) [0.1, 0.3) [0.3, 0.5) [0.5, 1.0) 1.0

% 95.9% 15.2% 31.0% 23.9% 4.3% 25.6%

Tomography-based Overlay Monitoring Results

)',max()('),,max()()()(',

)(')(max)',(

ppppwherepp

ppppF

Absolute and Relative Errors

• For each experiment, get its 95 percentile absolute and relative errors for estimation of 2,550 paths

Lossy Path Inference Accuracy

• 90 out of 100 runs have coverage over 85% and false positive less than 10%

• Many caused by the 5% threshold boundary effects

Topology Measurement Error Tolerance

• Out of 13 sets of pair-wise traceroute …• On average 248 out of 2550 paths have no

or incomplete routing information• No router aliases resolved

Conclusion: robust against topology measurement errors

Performance Improvement with Overlay

• With single-node relay• Loss rate improvement

– Among 10,980 lossy paths:– 5,705 paths (52.0%) have loss rate reduced by 0.05 or more– 3,084 paths (28.1%) change from lossy to non-lossy

• Throughput improvement– Estimated with

– 60,320 paths (24%) with non-zero loss rate, throughput computable

– Among them, 32,939 (54.6%) paths have throughput improved, 13,734 (22.8%) paths have throughput doubled or more

• Implications: use overlay path to bypass congestion or failures

lossraterttthroughput 5.1

SERVER

OVERLAY RELAYNODE

OVERLAY NETWORKOPERATION CENTER

CLIENT

3. Network congestion /failure

4. Detect congestion /failure

2. Register trigger

7. Skip-free streamingmedia recovery

6. Setup New Path

1. Setupconnection

5. Alert +New Overlay Path

XUC Berkeley

UC San Diego

Stanford

HP Labs

Adaptive Overlay Streaming Media

• Implemented with Winamp client and SHOUTcast server• Congestion introduced with a Packet Shaper• Skip-free playback: server buffering and rewinding• Total adaptation time < 4 seconds

Adaptive Streaming Media Architecture

Client 1

MEDIASOURCE

SERVERSHOUTcast

Server

Buffering Layer

Clie

nt 1

Clie

nt 2

Clie

nt 3

Clie

nt 4

FromSHOUTcast

server

Calculatedconcatenation

point

BU

FFE

R

ByteCounter

Client 2

Client 3

Client 4

INTERNET

Triggering /alert + new path

OVERLAY RELAY NODE

RELAYOverlay Layer

Path Management

TCP/IP Layer

RELAY

CLIENTWinamp client

TCP/IP Layer

Overlay Layer

Internet

Path Management

Winamp Video/Audio Filter

Byte Counter

TCP/IP Layer

OVERLAY NETWORKOPERATION CENTER

Conclusions• A tomography-based overlay network

monitoring system– Given n end hosts, characterize O(n2) paths with

a basis set of O(nlogn) paths– Selectively monitor O(nlogn) paths to compute

the loss rates of the basis set, then infer the loss rates of all other paths

• Both simulation and real Internet experiments promising

• Built adaptive overlay streaming media system on top of monitoring services– Bypass congestion/failures for smooth playback

within seconds

Backup Slides