research

Roaming Dynamics and Implications in the Cellular

Ecosystem

Andra Lutu, Telefonica Research

whoami• Associate Researcher at Telefónica Research since

March 2018• Based in Barcelona, Spain• Diverse group working on networks, security and

privacy, ML/DL and HCI• Network Measurements (focus on interactions between

entities in the mobile ecosystem), Testbeds for Mobile Networks, Inter-domain routing, ML/AI for Networks

• H2020 Marie Curie Individual Fellowship (DICE: Dynamic Interconnections for the Cellular Ecosytem)

• Postdoctoral Fellow at Simula Research Laboratory from 2015 to 2018• Building the MONROE platform

• Graduated PhD in November 2014 from IMDEA Networks Institute and UC3M

2

Illustrations from @butlikemaybehttp://www.butlikemaybe.com/

whoamiDiego Perino (Telefonica Research)Alessandro Finamore (Telefonica UK) Anna Brunstrom (Karlstad Universitet)Özgu Alay (Simula Metropolitan)Cristel Pelsser (University of Strasbourg) Marcelo Bagnulo (University Carlos III of Madrid)Anna Maria Mandalari (Imperial College London)Foivos Michelinakis (Simula Metropolitan)Gorry Fairhurst (University of Aberdeen)Ana Custura (University of Aberdeen)Ali Safari Khatouni (Politecnico di Torino)Marco Mellia (Politecnico di Torino)Vaibhav Bajpai (Technische Universität München)Jörg Ott (Technische Universität München)Mohammad Rajiullah (Karlstad Universitet)Hossein Doroud (IMDEA Networks)Narseo Vallina-Rodriguez (ICSI/IMDEA Networks)Joerg Widmer (IMDEA Networks)Abbas Razaghpanah (Stony Brook University)Phillipa Gill (UMass Amherst)Fabian Bustamante (Northwestern University)Byungjin Jun (Northwestern University)

..and many others…

3

today• Motivation: Why study roaming?• Background: Roaming in the cellular ecosystem• The bigger picture: Roaming of Things• Methodology: How can we measure the

implications of roaming?• Results: Performance, content implications and

quality of experience• Reproducibility• What’s next?

4

today• Motivation: Why study roaming?• Background: Roaming in the cellular ecosystem• The bigger picture: Roaming of Things• Methodology: How can we measure the

implications of roaming?• Results: Performance, content implications and

quality of experience• Reproducibility• What’s next?

5

Roaming in the Cellular Ecosystem• Cellular networks have the built-

in feature of supporting user mobility across operators (national or international roaming)• International Roaming is driven

by new regulation (Roam like at Home in the EU), the increased mobility of users and new technologies (e.g., VoLTE) and other industry trends (IoT, M2M)

6

Why Study International Roaming? • Understand the operational reality, the business of roaming and the

ecosystem around it• Which are the technical solutions to enable international roaming that are

actually being used today?• How does roaming impact the performance and the quality of experience

for end users?

7

today• Motivation: Why study roaming?• Background: Roaming in the cellular ecosystem• The bigger picture: Roaming of Things• Methodology: How can we measure the

implications of roaming?• Results: Performance, content implications and

quality of experience• Reproducibility• What’s next?

8

The roaming interconnection ecosystem

Interconnection Network

IPX Provider 2

IPX Provider 1

Mobile Network

Operator 2

Mobile Network

Operator 1

Mobile Network

Operator 3

Fixed Network Operators

Content Provider

Internet Service

Providers

Enterprise and Financial Services

Application Service

Providers 9

The business of roaming

• Each operator has a contract with an IPX provider (for the roaming hub function)• Roaming partners no longer need to have bilateral contracts with each other

• Operators pay a service charge to the roaming hub• Inter-operator tariff (IOT) between the Home Operator (HMNO) and Visited Operator (VMNO)

IPXProvider

VMNOIPXProvider

HMNO Roaming hub Roaming hub

Service ChargeService Charge

Inter-operator tariff

10

View from an Operational Mobile Network• UK MNO with large population of devices (~30mil) • How many are roaming?

11

Inbound Roamers: Things or People?

• Using the GSMA TAC database and the APN used, we classify the devices we identify as inbound roamers

12

Majority of IoT devices are inbound roamers!

Out of all inbound roamers, only 27% are people!

Where are these devices roaming from?

13

today• Motivation: Why study roaming?• Background: Roaming in the cellular ecosystem• The bigger picture: Roaming of Things• Methodology: How can we measure the

implications of roaming?• Results: Performance, content implications and

quality of experience• Reproducibility• What’s next?

14

Why are so many things roaming?

• International Carriers (e.g., Telefónica, Orange, Syniverse, Tata Communications) leverage their extensive infrastructure and deploy M2M/IoT platforms to support IoT verticals (smart meters, connected cars, health etc.)• Roaming is a major component guaranteeing the success of M2M/IoT

platforms• SIM for Things • Single home MNO supports connected things globally

15

The roaming interconnection ecosystem

Interconnection Network

IPX Provider 2

IPX Provider 1

Mobile Network

Operator 2

Mobile Network

Operator 1

Mobile Network

Operator 3

Fixed Network Operators

Content Provider

Internet Service

Providers

Enterprise and Financial Services

Application Service

Providers 16

Operations of an M2M/IoT platform

• A view from one of the largest M2M platforms in the world enabling 4G/LTE devices world-wide

17

today• Motivation: Why study roaming?• Background: Roaming in the cellular ecosystem• The bigger picture: Roaming of Things• Methodology: How can we measure the

implications of roaming?• Results: Performance, content implications and

quality of experience• Reproducibility• What’s next?

18

Roaming People vs. Roaming Things

• Things are active for longer times, consume little data but generate a lot of signaling traffic and occupy resources meant for people

19

On average, IoT devices are active for a period 4 times longer than smartphones

IoT devices generate little to no traffic thus the VMNO does not profit from billing data communications

IoT devices generate significant signaling traffic

Roaming People vs. Roaming Things

• IoT devices are not moving as much and they depend mostly on 2G connectivity

20

Configurations for (Data) Roaming

21

MME

HSS

AAA

S-GW

P-GW

S-GW MME

HSS

AAA

P-GW

eNodeB

HMNO VMNOIPX Network

Other MNOs, FNOs, ISPs, ASPs

eNodeB

MobileDevice

Internet

Evol

ved

Uni

vers

alTe

rres

tria

l Rad

ioA

cces

s N

etw

ork

Evol

ved

Pack

etCo

re (E

PC)

Local breakoutHome-routed

R

R

R

R

IPX breakout

(1) Home-routed (2) Local breakout (3) IPX breakout

Let’s build a testbed!

Operator 1

Operator 2

Operator 3

User Access & Scheduling

Experiment Deployment

INTERNETExperimentation on Commercial Mobile

Networks

[MOBICOM17]Ö. Alay, A. Lutu, M. Peón-Quirós, V. Mancuso, T. Hirsch, K. Evensen, A. Hansen, S. Alfredsson, J. Karlsson, A. Brunstrom, A. S. Khatouni, M. Mellia and M. A. Marsan, Experience: An Open Platform for Experimentation with Commercial Mobile Broadband Networks, in Proceedings of ACM International Conference on Mobile Computing and Networking (MOBICOM), 2017. ACM.

22

Experimental Setup: MONROE-Roaming• Joint effort between UC3M, SRL, KAU,

POLITO, TUM, UoA, TEF to run synchronized measurements in 6 countries (ES, NO, SE, IT, DE, UK)

• 6 countries, 16 operators, each country has two MONROE nodes

• Norway: Telia, Telenor• Sweden: Telenor, Tre, Telia• Spain: Orange, Vodafone, Movistar• Germany: Vodafone, O2, T-Mobile• UK: EE, O2• Italy: Vodafone, TIM, WIND

23

Experimental Setup:MONROE-Roaming• 6 countries, 16 operators, each

country has one MONROE node• One measurement server per

country• Measure one operator at a time

(all nodes have the SAME SIM in the same time)

• Also, measure the visited network natively, where possible

24

25

SpainItalyUK Norway

MONROE-Roaming Experiment Scheduler

Measurement Servers and Targets

Experiment Deployment

Results

Home Country

MONROE-Roaming Nodes

Backend Servers

Vodafone DE

Germany Sweden

Hom

eSI

M

26

Visited Countries

Vodafone DE

Roam

ing

SIM

Vodafone DE Vodafone DE Vodafone DE Vodafone DE

Telia NO Vodafone UK Vodafone IT Vodafone ESVi

site

dSI

MTelenor SE

today• Motivation: Why study roaming?• Background: Roaming in the cellular ecosystem• The bigger picture: Roaming of Things• Methodology: How can we measure the

implications of roaming?• Results: Performance, content implications and

quality of experience• Reproducibility• What’s next?

27

What do we measure?

• Roaming setup and performance• Metadata for tracking the roaming partner• traceroute to discover roaming setup • Dig queries against third party service providers (e.g., ad providers)• Monroe-browsertime for web browsing quality of experience

• Performance of Cloud Service Providers• VoIP and Content Discrimination • ooniprobe web connectivity test• Traffic differentiation measurements using three applications (FaceTime,

Facebook Messenger, Whatsapp) to determine potential traffic differentiation in roaming

28

What do we measure?

• Roaming setup and performance• Metadata for tracking the roaming partner• traceroute to discover roaming setup • Dig queries against third party service providers (e.g., ad providers)• Monroe-browsertime for web browsing quality of experience

• Performance of Cloud Service Providers• VoIP and Content Discrimination • ooniprobe web connectivity test• Traffic differentiation measurements using three applications (FaceTime,

Facebook Messenger, Whatsapp) to determine potential traffic differentiation in roaming

29

Roaming Setup

• Context metadata (RAT, signal strength, visited MNO etc.) from each SIM card• Traceroute: path towards each of the measurement servers

in each country to discover roaming setup

• Result: all 16 MNOs we measure use Home Routed Roaming configuration

30

31

MME

HSS

AAA

S-GW

P-GW

S-GW MME

HSS

AAA

P-GW

eNodeB

HMNO VMNOIPX Network

Other MNOs, FNOs, ISPs, ASPs

eNodeB

MobileDevice

Internet

Evol

ved

Uni

vers

alTe

rres

tria

l Rad

ioA

cces

s N

etw

ork

Evol

ved

Pack

etCo

re (E

PC)

Local breakoutHome-routed

R

R

R

R

IPX breakout

(1) Home-routed (2) Local breakout (3) IPX breakout

Roaming Setup and Performance: Home routed roaming is the usual configuration

Roaming Setup and Performance: Delay implicationsUK User roaming in Germany

32

Roaming Setup and Performance: Delay implicationsUK User roaming in Germany

33

Roaming Setup and Performance: Delay implicationsUK User roaming in Germany

GTP tunnel is slower than native Internet

34

Delay penalty to go back to home

country

35

Roaming Setup and Performance: Delay implicationsUK User roaming in Germany

Roaming Setup and Performance: DNS implications• dig queries against third party

providers FQDNs (e.g., using ABP filter lists)• For the home user the query time

is significantly lower in average than for the other five roaming users

[MOBICOM18]A. Mandalari, A. Lutu, A. Custura, A. Safari, O. Alay, M. Bagnulo, V. Bajpai, A. Brunstrom, J. Ott, M. Mellia, G. Fairhurst (2018). Experience: Implications of Roaming in Europe. Proceedings of ACM International Conference on Mobile Computing and Networking (MOBICOM), 2018.

36

Web Browsing QoE

• 100 target webpages, 10 visits per page• 8 MNOs from SE, ES, DE, NO in roaming,

same parallel configuration as above• MONROE-Browsertime using Chrome,

mobile device, H1s – H2• Collect radio context • QoE metrics: Page Load Time (PLT),

Speed Index (SI), First Paint (FP)

• [WWW19]M. Rajiullah, A. Lutu, A. Safari Khatouni, M. Fida, M. Mellia, A. Brunstrom, O. Alay, S. Alfredsson and V. Mancuso (2019). Web Experience in Operational Mobile Networks: Lessons from Two Million Page Visits. The Web Conference 2019 (WWW’19), 13-17 May 2019, San Francisco, CA, USA

37

What do we measure?

• Roaming setup and performance• Radio metadata for tracking the roaming partner• Traceroute to discover roaming setup • dig: DNS against third party service providers (ad providers)• Monroe-browsertime for web browsing quality of experience

• Performance of Cloud Service Providers (CSPs)• VoIP and Content Discrimination • ooniprobe web connectivity test• Traffic differentiation measurements using three applications (FaceTime,

Facebook Messenger, Whatsapp) to determine potential traffic differentiation in roaming

40

How do mobile apps work?

api.facebook

.comcss.

washingt

onpost.c

om

s.amazon-adsystem.com

config.uca.cloud.unity3d.comsvastx.moatads.com

Fully Qualified Domain Name (FQDN)41

Methodology outline

api.facebook.comcss.w

ashingtonp

ost.com

s.amazon-adsystem.comconfig.uca.cloud.unity3d.comsvastx.moatads.com

Accurate App Traffic Logs

1

CSP

2

PTR ASDetection by:

Commercial MNOs

Active Measurements

• DNS lookups• Traceroute • TCP connection

time• TLS session

establishment time

Empirical Performance Analysis

3

42

TCP Performance towards CSPs (Telia SE)

• Measure performance against Cloud Service Providers (e.g., Akamai, Amazon, Google) to capture the impact of roaming• Metric: Time to First Byte (TTFB)

[INFOCOM18]F. Mickelinakis, H. Doroud, A. Razaghpanah, A. Lutu, N. Vallina-Rodriguez, P. Gill, J. Widmer (March, 2018). The Cloud that Runs the Mobile Internet: A Measurement Study of Mobile Cloud Services. IEEE INFOCOM 2018 – IEEE Conference on Computer Communications. IEEE.

43

TCP Performance towards CSPs (Telia SE)GoogleAmazon AWSAkamai

[INFOCOM18]F. Mickelinakis, H. Doroud, A. Razaghpanah, A. Lutu, N. Vallina-Rodriguez, P. Gill, J. Widmer (March, 2018). The Cloud that Runs the Mobile Internet: A Measurement Study of Mobile Cloud Services. IEEE INFOCOM 2018 – IEEE Conference on Computer Communications. IEEE. 44

today• Motivation: Why study roaming?• Background: Roaming in the cellular ecosystem• The bigger picture: Roaming of Things• Methodology: How can we measure the

implications of roaming?• Results: Performance, content implications and

quality of experience• Reproducibility• What’s next?

48

Reproducible Research

1. [MOBICOM18]A. Mandalari, A. Lutu, A. Custura, A. Safari, O. Alay, M. Bagnulo, V. Bajpai, A. Brunstrom, J. Ott, M. Mellia, G. Fairhurst (2018). Experience: Implications of Roaming in Europe. Proceedings of ACM International Conference on Mobile Computing and Networking (MOBICOM), 2018.https://www.it.uc3m.es/amandala/roaming.html

2. [INFOCOM18]F. Mickelinakis, H. Doroud, A. Razaghpanah, A. Lutu, N. Vallina-Rodriguez, P. Gill, J. Widmer (March, 2018). The Cloud that Runs the Mobile Internet: A Measurement Study of Mobile Cloud Services. IEEE Conference on Computer Communications (INFOCOM), 2018.http://wireless.networks.imdea.org/cloudmap-project

3. [WWW19]M. Rajiullah, A. Lutu, A. Safari Khatouni, M. Fida, M. Mellia, A. Brunstrom, O. Alay, S. Alfredsson and V. Mancuso (2019). Web Experience in Operational Mobile Networks: Lessons from Two Million Page Visits. The Web Conference 2019 (WWW’19), 13-17 May 2019, San Francisco, CA, USA.https://zenodo.org/record/2574157

49

Reproducible Research

Experience: The Implications of Roaming in Europe• Hyperpaper version:

https://bitbucket.org/andralutu/roam_like_at_home_hyperpaper/src/master/

• What is a hyperpaper?Alberto Dainotti, Ralph Holz, Mirja Kühlewind, Andra Lutu, Joel Sommers, and Brian Trammell. "Open collaborative hyperpapers: a call to action." ACM SIGCOMM Computer Communication Review 49, no. 1 (2019): 31-33.

https://ccronline.sigcomm.org/wp-content/uploads/2019/02/sigcomm-ccr-final285.pdf

• Check other examples of hyperpapers: On the Suitability of RTT Measurements for Geolocationhttps://github.com/britram/trilateration

50

Reproducible Research

MONROE Resources• Datasets and papers:

https://zenodo.org/communities/h2020_monroe/

• The MONROE testbeds are open, join the MONROE Alliance• https://www.monroe-project.eu/

• Marvin MONROE – a scheduling daemon https://github.com/MONROE-PROJECT/Scheduler• Experiments code available: • https://github.com/MONROE-PROJECT/Experiments

51

today• Motivation: Why study roaming?• Background: Roaming in the cellular ecosystem• The bigger picture: Roaming of Things• Methodology: How can we measure the

implications of roaming?• Results: Performance, content implications and

quality of experience• Reproducibility• What’s next?

53

• Home routed roaming comes with a delay penalty, that translates into decreased QoE for the end user • Need for enabling local breakout (or home like at

the destination)• Industry trends such as M2M/IoT will put more

stress on the roaming dynamics• IPX providers get more appealing

• Security and privacy aspects of the roaming interconnect need further investigation (3G/4G) to inform the next generation (5G) solutions

54

What’s next?

Thank you!

57

Illustrations from @butlikemaybehttp://www.butlikemaybe.com/