End-to-end Bandwidth EstimationEnd-to-end Bandwidth Estimationin the Wide Internetin the Wide Internet

Daniele CrocePhD dissertation, April 16, 2010

• “Breakfast Can Wait. The Day’s First Stop Is Online.” [NYTimes‘09]

but is our connection performing well?

Internet is wonderful

2

• Inter-connected networks– Different technologies, many operators– No global view

The Internet

3

Net1

Net2Net3

Net4

Objective: characterize the E2E performance

• Simple metrics– Packet loss– Delay (One-Way, RTT), jitter– (TCP) throughput

• Advanced metrics– End-to-end capacity

C=min(Ci)– End-to-end available bandwidth (AB)

• i.e., the unused capacityA=min(Ai)

Performance metrics

4

• On a generic link i :Available Bandwidth

5T

TBCTA iii

),0(),0(

• An example:Narrow link and tight link

6

Narrow Link Tight Link

100 Mbps90 Mbps

1000 Mbps400 Mbps

155 Mbps20 Mbps

Available bandwidth

C =AB =

Capacity

• Tools require access to both end hosts– Impossible between different organizations!

Three single-ended tools

Contribution 1

7

Net1

Net2Net3

Net4

• Large-scale deployments of active AB tools– Routing, P2P optimization, improve TCP

Performance evaluation of AB techniquesin large-scale measurement systems

Contribution 2

8

Net1

Net2Net3

Net4

• Three AB measurement paradigms exist:– PRM (Probe Rate Model)

• “Is rate higher than the AB?”– PGM (Probe Gap Model)

• “Has the Inter-Packet Gap increased?”– PDM (Probe Delay Model)

• “Has the packet queued?”• Only analytical or simulative studies• Better than PRM or PGM?

Real implementation and comparison with other classic PRM and PGM tools

Contribution 3

9

NEW!!!

SINGLE-ENDED TECHNIQUES

10

Non-cooperative estimation

• RTT = OWDf

DSLAM

ACK probes

TCP RSTsSender Receiver

Can we separate the effects of the two paths?

Sender Receiver

ACKs

Sender

RSTs

+ OWDr

11

Where is the tight link?

12

Sender

ACKs

Sender

RSTs

• RSTs are always 40 Bytes• No matter the size of the ACK probes

• By varying the ACK size We can load the two paths equally (SACK = SRST) We can load the downlink more than the uplink (SACK > SRST) We can NOT load the uplink more than the downlink (SACK <

SRST)

ABw-Probe (ABP)

13

• Measuring the downlink (no uplink traffic)

• Impact of “cross”-traffic on the uplink

cooperative

non-coop.

Uplink cross-traffic

14

Filtering uplink cross-traffic

• Cross-traffic is not just MTU packets– Use DT to remove

large packets– Then use RR for

refining15

FAB-probe (large-scale)

16

Do we really need a 40 kbps precision?

Real-world experience• Tested on 1244 ADSL hosts, 10 different ISPs

– Participating in Kademlia DHT (eMule)• Used KAD crawler (ACM IMC 2007)• Selected ADSL using Maxmind

1. Capacity of the ADSL link2. A snapshot of the available bandwidth3. Average AB on over 10 days

– 82 hosts online for over one month– Static IP address– Measured every 5 minutes

• On average 6 seconds per measurement17

Capacity estimation• Comparison of 2 large ISPs

The policy used by Free is quite uncommon (see IMC07)

0.7Mbps

2.5Mbps 0.3Mbps

1Mbps

Downlink

Uplink

18

Available bandwidth (I)• Snapshot of 1244 (eMule) hosts

Hosts are divided in congested or idle

19

Available bandwidth (II)• 82 hosts, 10 days average

– Each point is an average of one user over 10 days

• 30% congested, 30-40% frequently idle20

ANALYSIS OF LARGE-SCALEAB MEASUREMENT SYSTEMS

21

Motivation• We have a dream: measure AB everywhere

– Route selection, server selection – Overlay performance optimization– Improve TCP– ...

• Naïve approach:– pick one of the existing techniques!

• BUT what if we all do the same simultaneously?

Interference between measurements22

In brief• Existing techniques

– Pathload, Spruce, pathChirp• Experimental testbed

– All tools suffer from mutual interference• But not in the same way!!!

– High intrusiveness and overhead• Analytical models

– Probability of interference– Measurement bias

• What can we do?23

Pathload – Packet Trains• Probing strategy:

– Iteratively send N trains at different rates– Binary search to converge to the AB

• Inference:– Detect One-Way Delay increase (rate > AB)

24

Spruce – Packet Pairs• Probing strategy:

– Two packets with specific inter-packet gap• Inference:

– Measure dispersion (gap increase) of the pair

– Accuracy is debated, out of our scope

∆in ∆in Bottlene

ck

∆out

∆out

25

Interference in Spruce• One pair interfering…

• What is the probability that this happens?– Hint: similar to ALOHA protocol

0

26

∆in ∆out

100% error!

pathChirp – Packet “chirps”

• Probing strategy:– One train with exponentially increasing rate

• Inference:– Detect One-Way Delay increase

27

ABwLimit

Testbed results• 62 hosts running linux

– Half are senders, half receivers• Single bottleneck (10 Mbps), CBR traffic

– Ideal conditions for ABw tools– Errors are due to mutual interference only

28

Pathload

29

Spruce

True?? How much OVERHEAD?

30

Results are biased

pathChirpResults seem better

31

True?? High OVERHEAD!

Intrusiveness

x10 x100

32

Possible Solutions• Mutual interference

– Direct probing more promising• Simple, Spruce-like algorithms. No binary search

– Identify interference (and correct it)??

• Overhead– “In-band” measurements (piggy-backing)

• Best, no overhead at all• Complex! (SIGCOMM09) + delay constraints

– “Out-of-band” measurements• At least, make the overhead scale with the ABw!

• Lets help each other! Network Tomography33

Conclusions• Non-cooperative estimation

– Three highly optimized tools– No need to install software or buy new equipment

• An Italian ISP already interested!

• Analysis of large-scale AB measurements– Tools can not be used off-the-shelf

• Mutual interference, Intrusiveness, Overhead

– Interference can be predicted and modeled– Discussed possible solutions

• Future work includes– Technologies different from ADSL (cable, FTTH)– New, lightweight techniques (passive?), tomography

34

BACKUP

35

Collision with ON-OFF meas.

36

Few hosts cause > 10%

collisions!

Non-cooperative estimation• Who is answering to what (Monarch, IMC’06)

Measurement bias: Spruce• Measurement error in Spruce

– Depends on the # of interfering pairs n :

• The average number of interf. pairs is

• This explains why Spruce bias is proportional to

38

Pathload interference, two trains

39

PathChirp, two chirps

With only two trains, errors up to

80%!!!

40

Measurement Overhead• Spruce

– Overhead = min(240kbps, 5% of Bneck Capacity)– Few hosts can consume a LOT of Bw!

• Pathload– Overhead ≈ ABw– Cons: measurements consume all the ABw– Pro: overhead “scales” with the ABw

• pathChirp– Overhead = 300kbps (tunable parameter)– What if 10 hosts are measuring? If 100?

41

With traffic load• 20 hosts running, ABw=6 Mbps

42

All tools together• 9 hosts per type (27 senders)

43

Delay-based tools• Consider a single server queue

– The utilization can be computed as

• 0 is the probability of the queue being empty

– Probe-Delay-Model (PDM) tools estimate 0

• PDM tools– Make no assumptions on cross-traffic– Inject very little overhead

• no need for high probing rates44

Forecaster Model

45

The AB is estimated by “projecting” the

utilization

Threshold problem

46

-100 0 100 200 300 400 500 600 700 800 9000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

One Way Delay(usec)

CD

F

CDF of OWD with 20% cross traffic

delay in this area is considered not suffered from queueing

Time Threshold

In our experiments, must allow ~100us for inaccuracies!