Mobile Network Performance from User Devices:A Longitudinal, Multidimensional Analysis

Ashkan Nikravesh, David R. Choffnes, Ethan Katz-BassettZ. Morley Mao, Matt Welsh

Problem

Mobile Network Performance:

8 Poor visibility into user perceived performance.

8 It is difficult to capture a view of network performance.

Why is that difficult?

• Performance depends on many factors

How to improve the visibility?

• Pervasive network monitoring is needed:

◦ Continuous◦ Large-scale

• Sampling performance of devices across:

◦ Carriers◦ Access Technologies◦ Location◦ Time

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 2 / 16

Problem

Mobile Network Performance:

8 Poor visibility into user perceived performance.

8 It is difficult to capture a view of network performance.

Why is that difficult?

• Performance depends on many factors

How to improve the visibility?

• Pervasive network monitoring is needed:

◦ Continuous◦ Large-scale

• Sampling performance of devices across:

◦ Carriers◦ Access Technologies◦ Location◦ Time

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 2 / 16

Problem

Mobile Network Performance:

8 Poor visibility into user perceived performance.

8 It is difficult to capture a view of network performance.

Why is that difficult?

• Performance depends on many factors

How to improve the visibility?

• Pervasive network monitoring is needed:

◦ Continuous◦ Large-scale

• Sampling performance of devices across:

◦ Carriers◦ Access Technologies◦ Location◦ Time

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 2 / 16

Problem

Mobile Network Performance:

8 Poor visibility into user perceived performance.

8 It is difficult to capture a view of network performance.

Why is that difficult?

• Performance depends on many factors

How to improve the visibility?

• Pervasive network monitoring is needed:

◦ Continuous◦ Large-scale

• Sampling performance of devices across:

◦ Carriers◦ Access Technologies◦ Location◦ Time

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 2 / 16

Problem

Mobile Network Performance:

8 Poor visibility into user perceived performance.

8 It is difficult to capture a view of network performance.

Why is that difficult?

• Performance depends on many factors

How to improve the visibility?

• Pervasive network monitoring is needed:

◦ Continuous◦ Large-scale

• Sampling performance of devices across:

◦ Carriers◦ Access Technologies◦ Location◦ Time

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 2 / 16

Previous Works

Their Limitations:

• Passively collected from cellular network infrastructure (e.g.GGSN)

• One month of data• Limited to a single carrier

• Collected from mobile devices, but not continuously

Our work differs from previous related work:

• Longitudinal

• Continuous

• Gathered from mobile devices using controlled experiments.

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 3 / 16

Previous Works

Their Limitations:

• Passively collected from cellular network infrastructure (e.g.GGSN)

• One month of data• Limited to a single carrier

• Collected from mobile devices, but not continuously

Our work differs from previous related work:

• Longitudinal

• Continuous

• Gathered from mobile devices using controlled experiments.

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 3 / 16

Previous Works

Their Limitations:

• Passively collected from cellular network infrastructure (e.g.GGSN)

• One month of data• Limited to a single carrier

• Collected from mobile devices, but not continuously

Our work differs from previous related work:

• Longitudinal

• Continuous

• Gathered from mobile devices using controlled experiments.

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 3 / 16

Data Analysis

4 Analyzing the data collected from:

4 144 carriers4 17 months4 11 cellular networks

4 Identify patterns, trends, anomalies, and evolution of cellularnetworks’ performance.

We find:

• Significant variance in end-to-end performance for all carriers.

• Part of the high variability is due to the geographic and temporalproperties of network.

• Routing and signal strength are potential sources of performancevariability.

• Performance is inherently unstable.

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 4 / 16

Data Analysis

4 Analyzing the data collected from:

4 144 carriers4 17 months4 11 cellular networks

4 Identify patterns, trends, anomalies, and evolution of cellularnetworks’ performance.

We find:

• Significant variance in end-to-end performance for all carriers.

• Part of the high variability is due to the geographic and temporalproperties of network.

• Routing and signal strength are potential sources of performancevariability.

• Performance is inherently unstable.

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 4 / 16

Data Analysis

4 Analyzing the data collected from:

4 144 carriers4 17 months4 11 cellular networks

4 Identify patterns, trends, anomalies, and evolution of cellularnetworks’ performance.

We find:

• Significant variance in end-to-end performance for all carriers.

• Part of the high variability is due to the geographic and temporalproperties of network.

• Routing and signal strength are potential sources of performancevariability.

• Performance is inherently unstable.

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 4 / 16

Data Analysis

4 Analyzing the data collected from:

4 144 carriers4 17 months4 11 cellular networks

4 Identify patterns, trends, anomalies, and evolution of cellularnetworks’ performance.

We find:

• Significant variance in end-to-end performance for all carriers.

• Part of the high variability is due to the geographic and temporalproperties of network.

• Routing and signal strength are potential sources of performancevariability.

• Performance is inherently unstable.

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 4 / 16

Data Analysis

4 Analyzing the data collected from:

4 144 carriers4 17 months4 11 cellular networks

4 Identify patterns, trends, anomalies, and evolution of cellularnetworks’ performance.

We find:

• Significant variance in end-to-end performance for all carriers.

• Part of the high variability is due to the geographic and temporalproperties of network.

• Routing and signal strength are potential sources of performancevariability.

• Performance is inherently unstable.

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 4 / 16

Data Analysis

4 Analyzing the data collected from:

4 144 carriers4 17 months4 11 cellular networks

4 Identify patterns, trends, anomalies, and evolution of cellularnetworks’ performance.

We find:

• Significant variance in end-to-end performance for all carriers.

• Part of the high variability is due to the geographic and temporalproperties of network.

• Routing and signal strength are potential sources of performancevariability.

• Performance is inherently unstable.

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 4 / 16

Methodology

• User perceived performance:

• HTTP GET Throughput• Ping RTT• DNS Lookup time

4 Traceroute

4 Carrier + Cellular network technology

4 Signal strength

4 Location

4 Timestamp

To identify and isolate the performanceimpact of each factor

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 5 / 16

Methodology

• User perceived performance:• HTTP GET Throughput• Ping RTT• DNS Lookup time

4 Traceroute

4 Carrier + Cellular network technology

4 Signal strength

4 Location

4 Timestamp

To identify and isolate the performanceimpact of each factor

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 5 / 16

Methodology

• User perceived performance:• HTTP GET Throughput• Ping RTT• DNS Lookup time

4 Traceroute

4 Carrier + Cellular network technology

4 Signal strength

4 Location

4 Timestamp

To identify and isolate the performanceimpact of each factor

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 5 / 16

Methodology

• User perceived performance:• HTTP GET Throughput• Ping RTT• DNS Lookup time

4 Traceroute

4 Carrier + Cellular network technology

4 Signal strength

4 Location

4 Timestamp

To identify and isolate the performanceimpact of each factor

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 5 / 16

Methodology

• User perceived performance:• HTTP GET Throughput• Ping RTT• DNS Lookup time

4 Traceroute

4 Carrier + Cellular network technology

4 Signal strength

4 Location

4 Timestamp

To identify and isolate the performanceimpact of each factor

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 5 / 16

Dataset

• Mainly collected by Speedometer:

• 2011-10 to 2013-2 (17 months)• Internal android app developed by Google• Anonymized data

• Mobiperf• 11 months• Only used for our signal strength analysis

• Controlled experimentsSpeedometer dataset: 4-5 measurements per minute

• Code is open source and data is publicly available

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 6 / 16

Dataset

• Mainly collected by Speedometer:

• 2011-10 to 2013-2 (17 months)• Internal android app developed by Google• Anonymized data

• Mobiperf• 11 months• Only used for our signal strength analysis

• Controlled experimentsSpeedometer dataset: 4-5 measurements per minute

• Code is open source and data is publicly available

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 6 / 16

Dataset

• Mainly collected by Speedometer:

• 2011-10 to 2013-2 (17 months)• Internal android app developed by Google• Anonymized data

• Mobiperf• 11 months• Only used for our signal strength analysis

• Controlled experimentsSpeedometer dataset: 4-5 measurements per minute

• Code is open source and data is publicly available

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 6 / 16

Dataset

• Mainly collected by Speedometer:

• 2011-10 to 2013-2 (17 months)• Internal android app developed by Google• Anonymized data

• Mobiperf• 11 months• Only used for our signal strength analysis

• Controlled experimentsSpeedometer dataset: 4-5 measurements per minute

• Code is open source and data is publicly available

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 6 / 16

Performance across Carriers

How observed performance matches with the expectations across accesstechnologies?

• Ping RTT Latency

1 Latency varies significantly across carriers and access technologies2 Same performance for different access technologies

100

1000

T-Mobile

AT&T

YesOptus

Swisscom

Vodafone(DE)

Vodafone(NL)

Vodafone(IE)

Vodafone(UK)

O2(U

K)

Airtel

Telkomsel

Rogers

SingTel

NTT D

oCoM

o

Telstra

SFRSK Telecom

Emobile

Pin

g R

TT

(m

s)

GPRSEDGEUMTS

HSDPAHSPA

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 7 / 16

Performance across Carriers

How observed performance matches with the expectations across accesstechnologies?

• Ping RTT Latency

1 Latency varies significantly across carriers and access technologies2 Same performance for different access technologies

100

1000

T-Mobile

AT&T

YesOptus

Swisscom

Vodafone(DE)

Vodafone(NL)

Vodafone(IE)

Vodafone(UK)

O2(U

K)

Airtel

Telkomsel

Rogers

SingTel

NTT D

oCoM

o

Telstra

SFRSK Telecom

Emobile

Pin

g R

TT

(m

s)

GPRSEDGEUMTS

HSDPAHSPA

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 7 / 16

Performance across Carriers

How observed performance matches with the expectations across accesstechnologies?

• Ping RTT Latency1 Latency varies significantly across carriers and access technologies

2 Same performance for different access technologies

100

1000

T-Mobile

AT&T

YesOptus

Swisscom

Vodafone(DE)

Vodafone(NL)

Vodafone(IE)

Vodafone(UK)

O2(U

K)

Airtel

Telkomsel

Rogers

SingTel

NTT D

oCoM

o

Telstra

SFRSK Telecom

Emobile

Pin

g R

TT

(m

s)

GPRSEDGEUMTS

HSDPAHSPA

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 7 / 16

Performance across Carriers

How observed performance matches with the expectations across accesstechnologies?

• Ping RTT Latency1 Latency varies significantly across carriers and access technologies2 Same performance for different access technologies

100

1000

T-Mobile

AT&T

YesOptus

Swisscom

Vodafone(DE)

Vodafone(NL)

Vodafone(IE)

Vodafone(UK)

O2(U

K)

Airtel

Telkomsel

Rogers

SingTel

NTT D

oCoM

o

Telstra

SFRSK Telecom

Emobile

Pin

g R

TT

(m

s)

GPRSEDGEUMTS

HSDPAHSPA

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 7 / 16

Performance across Carriers

• HTTP GET Throughput

1 Relatively smaller difference between the carriers2 Download size (224KB) in not sufficiently large3 Lower latency is generally correlated with higher throughput

10

100

1000

T-Mobile

AT&T

YesOptus

Swisscom

Vodafone(DE)

Vodafone(NL)

Vodafone(IE)

Vodafone(UK)

O2(U

K)

Airtel

Telkomsel

Rogers

SingTel

NTT D

oCoM

o

Telstra

SFRSK Telecom

Emobile

HT

TP

Th

rou

gh

pu

t (K

bp

s)

GPRSEDGEUMTS

HSDPAHSPA

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 8 / 16

Performance across Carriers

• HTTP GET Throughput1 Relatively smaller difference between the carriers

2 Download size (224KB) in not sufficiently large3 Lower latency is generally correlated with higher throughput

10

100

1000

T-Mobile

AT&T

YesOptus

Swisscom

Vodafone(DE)

Vodafone(NL)

Vodafone(IE)

Vodafone(UK)

O2(U

K)

Airtel

Telkomsel

Rogers

SingTel

NTT D

oCoM

o

Telstra

SFRSK Telecom

Emobile

HT

TP

Th

rou

gh

pu

t (K

bp

s)

GPRSEDGEUMTS

HSDPAHSPA

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 8 / 16

Performance across Carriers

• HTTP GET Throughput1 Relatively smaller difference between the carriers2 Download size (224KB) in not sufficiently large

3 Lower latency is generally correlated with higher throughput

10

100

1000

T-Mobile

AT&T

YesOptus

Swisscom

Vodafone(DE)

Vodafone(NL)

Vodafone(IE)

Vodafone(UK)

O2(U

K)

Airtel

Telkomsel

Rogers

SingTel

NTT D

oCoM

o

Telstra

SFRSK Telecom

Emobile

HT

TP

Th

rou

gh

pu

t (K

bp

s)

GPRSEDGEUMTS

HSDPAHSPA

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 8 / 16

Performance across Carriers

• HTTP GET Throughput1 Relatively smaller difference between the carriers2 Download size (224KB) in not sufficiently large

3 Lower latency is generally correlated with higher throughput

10

100

1000

T-Mobile

AT&T

YesOptus

Swisscom

Vodafone(DE)

Vodafone(NL)

Vodafone(IE)

Vodafone(UK)

O2(U

K)

Airtel

Telkomsel

Rogers

SingTel

NTT D

oCoM

o

Telstra

SFRSK Telecom

Emobile

HT

TP

Th

rou

gh

pu

t (K

bp

s)

GPRSEDGEUMTS

HSDPAHSPA

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 8 / 16

Performance across Carriers

• HTTP GET Throughput1 Relatively smaller difference between the carriers2 Download size (224KB) in not sufficiently large3 Lower latency is generally correlated with higher throughput

10

100

1000

T-Mobile

AT&T

YesOptus

Swisscom

Vodafone(DE)

Vodafone(NL)

Vodafone(IE)

Vodafone(UK)

O2(U

K)

Airtel

Telkomsel

Rogers

SingTel

NTT D

oCoM

o

Telstra

SFRSK Telecom

Emobile

HT

TP

Th

rou

gh

pu

t (K

bp

s)

GPRSEDGEUMTS

HSDPAHSPA

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 8 / 16

Performance across different Locations

• Different topologies in different regions

• New York, Bay Area, and Seattle

20

30

40

50

60

70

80

90

100

110

Oct 30 Nov 19 Dec 09 Dec 29 Jan 18 Feb 07 Feb 27 Mar 18 Apr 07 Apr 27 May 17 Jun 06

Pin

g R

TT

(m

s)

Mean

and

Sta

ndard

Err

or

Time (Days) 2011-2012

Bay AreaSeattle

New York

Verizon LTE

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 9 / 16

Performance across different Locations

• Different topologies in different regions

• New York, Bay Area, and Seattle

20

30

40

50

60

70

80

90

100

110

Oct 30 Nov 19 Dec 09 Dec 29 Jan 18 Feb 07 Feb 27 Mar 18 Apr 07 Apr 27 May 17 Jun 06

Pin

g R

TT

(m

s)

Mean

and

Sta

ndard

Err

or

Time (Days) 2011-2012

Bay AreaSeattle

New York

Verizon LTE

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 9 / 16

Performance across different Locations

• Different topologies in different regions

• New York, Bay Area, and Seattle

20

30

40

50

60

70

80

90

100

110

Oct 30 Nov 19 Dec 09 Dec 29 Jan 18 Feb 07 Feb 27 Mar 18 Apr 07 Apr 27 May 17 Jun 06

Pin

g R

TT

(m

s)

Mean a

nd S

tandard

Err

or

Time (Days) 2011-2012

Bay AreaSeattle

New York

Verizon LTE

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 9 / 16

Performance over Time

• How much performance depends on time?• time of the day• stability

• When to measure the network?

Time of the day

600

700

800

900

1000

1100

1200

1300

1400

0 2 4 6 8 10 12 14 16 18 20 22 24

HT

TP

Th

rou

ghput (K

bps),

Mean a

nd S

tan

dard

Err

or

Local Time from 0 (00:00) to 24 (24:00)

AT&T HSDPASprint EVDO_A

T-Mobile HSDPAVerizon EVDO_A

1 Throughput decreases duringthe busy hours of usage

2 Carriers experience minimumthroughput at different times

3 Carriers experience differentvariation in performance duringthe busy hours

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 10 / 16

Performance over Time

• How much performance depends on time?• time of the day• stability

• When to measure the network?

Time of the day

600

700

800

900

1000

1100

1200

1300

1400

0 2 4 6 8 10 12 14 16 18 20 22 24

HT

TP

Th

rou

ghput (K

bps),

Mean a

nd S

tan

dard

Err

or

Local Time from 0 (00:00) to 24 (24:00)

AT&T HSDPASprint EVDO_A

T-Mobile HSDPAVerizon EVDO_A

1 Throughput decreases duringthe busy hours of usage

2 Carriers experience minimumthroughput at different times

3 Carriers experience differentvariation in performance duringthe busy hours

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 10 / 16

Performance over Time

• How much performance depends on time?• time of the day• stability

• When to measure the network?

Time of the day

600

700

800

900

1000

1100

1200

1300

1400

0 2 4 6 8 10 12 14 16 18 20 22 24

HT

TP

Th

roughput (K

bps),

Mean a

nd S

tandard

Err

or

Local Time from 0 (00:00) to 24 (24:00)

AT&T HSDPASprint EVDO_A

T-Mobile HSDPAVerizon EVDO_A

1 Throughput decreases duringthe busy hours of usage

2 Carriers experience minimumthroughput at different times

3 Carriers experience differentvariation in performance duringthe busy hours

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 10 / 16

Performance over Time

• How much performance depends on time?• time of the day• stability

• When to measure the network?

Time of the day

600

700

800

900

1000

1100

1200

1300

1400

0 2 4 6 8 10 12 14 16 18 20 22 24

HT

TP

Th

roughput (K

bps),

Mean a

nd S

tandard

Err

or

Local Time from 0 (00:00) to 24 (24:00)

AT&T HSDPASprint EVDO_A

T-Mobile HSDPAVerizon EVDO_A

1 Throughput decreases duringthe busy hours of usage

2 Carriers experience minimumthroughput at different times

3 Carriers experience differentvariation in performance duringthe busy hours

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 10 / 16

Performance over Time

• How much performance depends on time?• time of the day• stability

• When to measure the network?

Time of the day

600

700

800

900

1000

1100

1200

1300

1400

0 2 4 6 8 10 12 14 16 18 20 22 24

HT

TP

Th

roughput (K

bps),

Mean a

nd S

tandard

Err

or

Local Time from 0 (00:00) to 24 (24:00)

AT&T HSDPASprint EVDO_A

T-Mobile HSDPAVerizon EVDO_A

1 Throughput decreases duringthe busy hours of usage

2 Carriers experience minimumthroughput at different times

3 Carriers experience differentvariation in performance duringthe busy hours

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 10 / 16

Performance over Time

• How much performance depends on time?• time of the day• stability

• When to measure the network?

Time of the day

600

700

800

900

1000

1100

1200

1300

1400

0 2 4 6 8 10 12 14 16 18 20 22 24

HT

TP

Th

roughput (K

bps),

Mean a

nd S

tandard

Err

or

Local Time from 0 (00:00) to 24 (24:00)

AT&T HSDPASprint EVDO_A

T-Mobile HSDPAVerizon EVDO_A

1 Throughput decreases duringthe busy hours of usage

2 Carriers experience minimumthroughput at different times

3 Carriers experience differentvariation in performance duringthe busy hours

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 10 / 16

Performance over Time

Stability of Performance• Users want a stable network!• Measurements are expensive!

Two metrics:

• Auto-Correlation• Weighted moving average

1PM 2PM 3PM 4PM 5PM

5PMWindow size: 2Sampling Period: 2hrs

w1

Error

w2

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 11 / 16

Performance over Time

Stability of Performance• Users want a stable network!• Measurements are expensive!

Two metrics:

• Auto-Correlation• Weighted moving average

1PM 2PM 3PM 4PM 5PM

5PMWindow size: 2Sampling Period: 2hrs

w1

Error

w2

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 11 / 16

Performance over Time

Stability of Performance• Users want a stable network!• Measurements are expensive!

Two metrics:

• Auto-Correlation• Weighted moving average

1PM 2PM 3PM 4PM 5PM

5PMWindow size: 2Sampling Period: 2hrs

w1

Error

w2

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 11 / 16

Performance over Time

Stability of Performance• Users want a stable network!• Measurements are expensive!

Two metrics:

• Auto-Correlation• Weighted moving average

1PM 2PM 3PM 4PM 5PM

5PMWindow size: 2Sampling Period: 2hrs

w1

Error

w2

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 11 / 16

Performance over Time

Stability of Performance (Weighted Moving Average)

0.1

0.2

0.3

0.4

0.5

0.6

0.7 0.8 0.9

1 3 6 9 12 15 18 21 24 27 30 33 36

Err

or

(%)

Sampling Period (hour)

Verizon, LTE, BayArea

Sprint, EVDOA, Seattle

Sprint, EVDOA, BayArea

T-Mobile, HSDPA, BayArea

1 Prediction accuracy varies

significantly by carriers

2 Prediction error increases with

longer sampling periods with

the exception of 24hr

sampling periods

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 12 / 16

Performance over Time

Stability of Performance (Weighted Moving Average)

0.1

0.2

0.3

0.4

0.5

0.6

0.7 0.8 0.9

1 3 6 9 12 15 18 21 24 27 30 33 36

Err

or

(%)

Sampling Period (hour)

Verizon, LTE, BayArea

Sprint, EVDOA, Seattle

Sprint, EVDOA, BayArea

T-Mobile, HSDPA, BayArea

1 Prediction accuracy varies

significantly by carriers

2 Prediction error increases with

longer sampling periods

with

the exception of 24hr

sampling periods

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 12 / 16

Performance over Time

Stability of Performance (Weighted Moving Average)

0.1

0.2

0.3

0.4

0.5

0.6

0.7 0.8 0.9

1 3 6 9 12 15 18 21 24 27 30 33 36

Err

or

(%)

Sampling Period (hour)

Verizon, LTE, BayArea

Sprint, EVDOA, Seattle

Sprint, EVDOA, BayArea

T-Mobile, HSDPA, BayArea

1 Prediction accuracy varies

significantly by carriers

2 Prediction error increases with

longer sampling periods

with

the exception of 24hr

sampling periods

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 12 / 16

Performance over Time

Stability of Performance (Weighted Moving Average)

0.1

0.2

0.3

0.4

0.5

0.6

0.7 0.8 0.9

1 3 6 9 12 15 18 21 24 27 30 33 36

Err

or

(%)

Sampling Period (hour)

Verizon, LTE, BayArea

Sprint, EVDOA, Seattle

Sprint, EVDOA, BayArea

T-Mobile, HSDPA, BayArea

1 Prediction accuracy varies

significantly by carriers

2 Prediction error increases with

longer sampling periods with

the exception of 24hr

sampling periods

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 12 / 16

Performance Degradation: Root Causes

Focus on the cases where persistent performance degradations were :

• observed in consecutive days• affects both latency and throughput

Inefficient Paths• Time evolution• Impact on the performance

40

60

80

100

120

140

160

11 12 13 14 16 17 18 19 20 21 22

Me

dia

n P

ing

RT

T (

ms)

Time (day) - Feb 2012

Seattle (25-50-75)

Los Angeles (25-50-75)

(a) T-Mobile HSDPA (Seattle)

11

12

13

14

15

16

17

18

05Nov

12Nov

19Nov

26Nov

03Dec

10Dec

17Dec

24Dec

31Dec

07Jan

30

40

50

60

70

80

# o

f H

op

s

Me

dia

n P

ing

RT

T (

ms)

Time (day) - 2011

Ping RTTHops

(b) Verizon LTE (Bay Area)

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 13 / 16

Performance Degradation: Root Causes

Focus on the cases where persistent performance degradations were :

• observed in consecutive days• affects both latency and throughput

Inefficient Paths• Time evolution• Impact on the performance

40

60

80

100

120

140

160

11 12 13 14 16 17 18 19 20 21 22

Me

dia

n P

ing

RT

T (

ms)

Time (day) - Feb 2012

Seattle (25-50-75)

Los Angeles (25-50-75)

(a) T-Mobile HSDPA (Seattle)

11

12

13

14

15

16

17

18

05Nov

12Nov

19Nov

26Nov

03Dec

10Dec

17Dec

24Dec

31Dec

07Jan

30

40

50

60

70

80

# o

f H

op

s

Me

dia

n P

ing

RT

T (

ms)

Time (day) - 2011

Ping RTTHops

(b) Verizon LTE (Bay Area)

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 13 / 16

Performance Degradation: Root Causes

Focus on the cases where persistent performance degradations were :

• observed in consecutive days• affects both latency and throughput

Inefficient Paths• Time evolution• Impact on the performance

40

60

80

100

120

140

160

11 12 13 14 16 17 18 19 20 21 22

Me

dia

n P

ing

RT

T (

ms)

Time (day) - Feb 2012

Seattle (25-50-75)

Los Angeles (25-50-75)

(a) T-Mobile HSDPA (Seattle)

11

12

13

14

15

16

17

18

05Nov

12Nov

19Nov

26Nov

03Dec

10Dec

17Dec

24Dec

31Dec

07Jan

30

40

50

60

70

80

# o

f H

op

s

Me

dia

n P

ing

RT

T (

ms)

Time (day) - 2011

Ping RTTHops

(b) Verizon LTE (Bay Area)

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 13 / 16

Performance Degradation: Root Causes

Focus on the cases where persistent performance degradations were :

• observed in consecutive days• affects both latency and throughput

Inefficient Paths• Time evolution• Impact on the performance

40

60

80

100

120

140

160

11 12 13 14 16 17 18 19 20 21 22

Me

dia

n P

ing

RT

T (

ms)

Time (day) - Feb 2012

Seattle (25-50-75)

Los Angeles (25-50-75)

(a) T-Mobile HSDPA (Seattle)

11

12

13

14

15

16

17

18

05Nov

12Nov

19Nov

26Nov

03Dec

10Dec

17Dec

24Dec

31Dec

07Jan

30

40

50

60

70

80

# o

f H

op

s

Me

dia

n P

ing

RT

T (

ms)

Time (day) - 2011

Ping RTTHops

(b) Verizon LTE (Bay Area)

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 13 / 16

Performance Degradation: Root Causes

Signal StrengthHow much it can affect performance?

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

0.22

0.24

0.26

0 5 10 15 20 25 30 400

600

800

1000

1200

1400

1600

1800

Mea

n P

acket

Loss (

%)

Mean P

ing R

TT

(m

s)

ASU

Ping RTTPacket Loss

(a) Ping RTT and Packet Loss

150

200

250

300

350

400

450

500

550

0 5 10 15 20 25 30

Mean

HT

TP

Thro

ughp

ut (K

bps)

ASU

HTTP Throughput

(b) HTTP Throughput

AT&T HSDPA (Seattle), Arbitrary Strength Units (ASU)

I Accounting for signal strength is important for interpretingmeasurement results.

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 14 / 16

Performance Degradation: Root Causes

Signal StrengthHow much it can affect performance?

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

0.22

0.24

0.26

0 5 10 15 20 25 30 400

600

800

1000

1200

1400

1600

1800

Mean P

acket Loss (

%)

Mean P

ing

RT

T (

ms)

ASU

Ping RTTPacket Loss

(a) Ping RTT and Packet Loss

150

200

250

300

350

400

450

500

550

0 5 10 15 20 25 30

Mean H

TT

P T

hro

ug

hp

ut

(Kbp

s)

ASU

HTTP Throughput

(b) HTTP Throughput

AT&T HSDPA (Seattle), Arbitrary Strength Units (ASU)

I Accounting for signal strength is important for interpretingmeasurement results.

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 14 / 16

Performance Degradation: Root Causes

Signal StrengthHow much it can affect performance?

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

0.22

0.24

0.26

0 5 10 15 20 25 30 400

600

800

1000

1200

1400

1600

1800

Mean P

acket Loss (

%)

Mean P

ing

RT

T (

ms)

ASU

Ping RTTPacket Loss

(a) Ping RTT and Packet Loss

150

200

250

300

350

400

450

500

550

0 5 10 15 20 25 30

Mean H

TT

P T

hro

ug

hp

ut

(Kbp

s)

ASU

HTTP Throughput

(b) HTTP Throughput

AT&T HSDPA (Seattle), Arbitrary Strength Units (ASU)

I Accounting for signal strength is important for interpretingmeasurement results.

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 14 / 16

Performance Degradation: Root Causes

Signal StrengthHow much it can affect performance?

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0.2

0.22

0.24

0.26

0 5 10 15 20 25 30 400

600

800

1000

1200

1400

1600

1800

Mean P

acket Loss (

%)

Mean P

ing

RT

T (

ms)

ASU

Ping RTTPacket Loss

(a) Ping RTT and Packet Loss

150

200

250

300

350

400

450

500

550

0 5 10 15 20 25 30

Mean H

TT

P T

hro

ug

hp

ut

(Kbp

s)

ASU

HTTP Throughput

(b) HTTP Throughput

AT&T HSDPA (Seattle), Arbitrary Strength Units (ASU)

I Accounting for signal strength is important for interpretingmeasurement results.

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 14 / 16

Future Work

• We need for more monitoring and diagnosis.

• Data is difficult to get.

MobilyzerAn Open Platform for Mobile Network Measurement

A comprehensive codebase for issuing measurementsfor researchers and developers

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 15 / 16

Future Work

• We need for more monitoring and diagnosis.

• Data is difficult to get.

MobilyzerAn Open Platform for Mobile Network Measurement

A comprehensive codebase for issuing measurementsfor researchers and developers

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 15 / 16

Any Questions?

Thank You!

A. Nikravesh, D. R. Choffnes, E. Katz-Bassett, Z. M. Mao, M. Welsh PAM 2014 16 / 16