FLEX-MONROE: A Unified Platform for Experimentsunder Controlled and Operational LTE Se�ings

Mah-Rukh Fida1, Konstantinos Kousias2, Andra Lutu2, Mohammad Rajiullah3,

Özgü Alay2, Anna Brunstrom3, Antonios Argyriou4

1 University of Edinburgh, UK 2 Simula Research Laboratory, Norway3 Karlstad University, Sweden 4 University of Thessaly, Greece

ABSTRACTThis paper presents FLEX-MONROE, a unique platform thatfacilitates achieving a thorough understanding of LTE net-works, one that captures the status of current operationalMBB networks and that also enables LTE performance im-provements by allowing experimentation in an environmentwith controllable LTE parameters. Using this platform, wepropose to investigate how variations in the LTE networkparameters in�uence the network characteristics, which, inturn, translate to application performance metrics that rep-resent the end-user experience. We argue that the FLEX-MONROE platform is crucial to understand, validate andultimately improve how current operational MBB networksperform, towards providing guidelines for designing future5G architectures. Furthermore, understanding the e�ects oflow-level tweaks in network parameters in the LTE infras-tructure on the application performance is critical to provideguidelines on how to improve the application performancein the current but also future MBB networks.

1 INTRODUCTIONOpen experimentation with operational mobile broadband(MBB) networks is currently a fundamental requirementof the research community in its strive to innovate mobilecommunications. At the time of writing, despite many indi-vidual e�orts to o�er access to infrastructure and technolo-gies [7, 14], there is still a compelling need for building a trulyopen MBB testbed that can o�er the service and experienceof an operational commercial network with the unique perkPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for pro�t or commercial advantage and that copies bearthis notice and the full citation on the �rst page. Copyrights for componentsof this work owned by others than ACM must be honored. Abstracting withcredit is permitted. To copy otherwise, or republish, to post on servers or toredistribute to lists, requires prior speci�c permission and/or a fee. Requestpermissions from permissions@acm.org.WiNTECH’17, October 20, 2017, Snowbird, UT, USA.© 2017 ACM. ISBN 978-1-4503-5147-8/17/10. . . $15.00DOI: http://dx.doi.org/10.1145/3131473.3131477

of controlling di�erent elements in the network. In this paper,we attempt to address this very ambitious goal. We introduceFLEX-MONROE, a unique platform for open experimenta-tion with LTE networks both in the wild with commercialoperational networks with MONROE [3] and in controlledsettings with con�gurable equipment with FLEX[17].

We present our approach for the integration of two mea-surement platforms (i.e., FLEX and MONROE) to build a one-of-a-kind testbed, FLEX-MONROE. We designed MONROEto advance our understanding of today’s operational MBBecosystem from the end-user’s perspective. Due to the lackof control of the experimenter over commercial networks,MONROE cannot extract and, more importantly, validatethe root-cause of MBB networks performance problems, es-pecially the ones related to the network con�guration andcharacteristics. In order to shed more light on the e�ectof di�erent network-speci�c parameters on the applicationperformance, there is a dire need for complementing the mea-surements we run in the wild with similar measurementsin an experimental LTE network. FLEX o�ers the ideal solu-tion for experimentation with LTE networks by providinga combination of con�gurable commercial equipment, corenetwork software and open source components. The FLEXtestbeds enable the identi�cation of how di�erent LTE net-work parameters impact the performance of services andapplications over a LTE infrastructure. The FLEX-MONROEintegration results in a platform that captures the status ofcurrent operational MBB networks and that also enablesLTE performance exploration through experimentation inan environment with controllable LTE parameters.

To demonstrate the bene�ts of FLEX-MONROE, we showthe impact of twisting the LTE network parameters on thenetwork quality of service characteristics, which, in turn,translate to application performance metrics that impactthe end-user experience. We use the particular case of webbrowsing to showcase the potential of performing such anal-ysis in FLEX-MONROE. We explore the mapping betweenweb browsing performance metrics and the set of featuresthat capture the network characteristics of operational LTE

networks we measure in MONROE. We also check a simi-lar mapping between the LTE network parameters and thenetworks characteristics using FLEX. The merged analysiscaptures how the underlying network parameters in the LTEinfrastructure ripple through all the layers until the end-userand impact the web browsing performance.

2 PLATFORMS OVERVIEWAll major mobile network operators are in the process ofadopting LTE/4G cellular networks. These are expected torule the cellular landscape at least for the current decade,while also forming the starting point for further progress,ushering in the 5G era. The lack of open or at least openlycon�gurable cellular equipment and core network softwareis a major limiting factor for applied research in this �eld out-side of the boundaries of vendor and operator R&D groups.MONROE and FLEX are two approaches that aim to addressthis gap. In the following, we give a brief description of thesetwo measurement platforms.

2.1 FLEXThe FIRE initiative targets the creation of a multidisciplinaryresearch platform for investigation and experimental evalua-tion of innovative ideas in networking and services. FLEX(FIRE LTE Testbeds for Open Experimentation) contributesa crucial missing piece in FIRE’s infrastructure puzzle: con-�gurable cellular access technologies and LTE in particu-lar. FLEX builds programmable LTE components as exten-sions to existing European testbeds, thus providing an openand remotely accessible platform for experimentation withLTE. The deployment of LTE components took place onthree existing and well-established FIRE wireless testbeds,thus enabling these facilities with novel capabilities for LTEexperimentation-driven research. Among di�erent facillities,in this paper we focus on the NITOS [17] indoor testbed. Theintegration with other facilities is part of our ongoing e�ort.

NITOS indoor tesbed is a fully controllable RF-isolated en-vironment consisting of over 60 operational wireless nodeso�ering for experimentation on various technologies, includ-ing Wi-Fi and LTE. To this extent, an LTE core infrastructureresides in NITOS with two LTE Base Stations (eNBs) and theSIRRAN’s LTEnet packet core network. As for now, a thirdof the available nodes are equipped with LTE dongles andcan be scheduled for experimentation with the LTE network.

Limitations: Multiple users can experiment with NITOS,at the same time, as long as they access di�erent resources(i.e., one single user can reserve a node in the platform ata time). For a single experimenter, this might translate intounplanned congestion, making his experimental results un-expected.

2.2 MONROEThe MONROE (Measuring Mobile Broadband Networks inEurope) system is the �rst open access platform for inde-pendent, multihomed and large-scale experimentation oncommercial cellular operators. MONROE integrates 150 hard-ware devices (MONROE nodes) and a software frameworkthat enables the orchestration of experiments and the col-lection, analysis, visualization and sharing of measurementsthat run on each hardware device. MONROE allows authenti-cated external users to access the platform, reserve resourcesand deploy their own experiments, which run in isolationto minimize bias. Aside from this, MONROE o�ers a seriesof ready-to-use experiments1 o�ered as Experimentation asa Service (EaaS). Researchers, and other external users, canrun measurements according to a pre-de�ned data quotato guarantee fairness. The software components and thearchitecture design are open sourced to foster communitymaintenance and deployability [1, 3].

Each MONROE node is a Linux-based programmable de-vice that is multihomed to three Mobile Broadband operatorsthanks to a dedicated 3G/4G modem (LTE CAT6) for eachcarrier [24]. The nodes are deployed in heterogeneous envi-ronments including mobile (e.g., public transport vehicles)and stationary ones (e.g., volunteers hosting nodes in theirhomes). The nodes upload experimental results, togetherwith the node metadata (e.g., geo-location, carrier, low-levelaspects of the radio link) to a central back-end for publicrelease. MONROE enables repeatability of measurements insimilar conditions (speci�c uniform hardware and the sameconditions/context for the stationary nodes).Limitations: Though o�ering access to experiment with 12mobile carriers operating in 4 European countries, MONROEo�ers a limited number of measurement vantage points peroperator and per country. Thus, experimenters can performcomparisons of operators’ services and analysis of measure-ment results only within the constraints of the geographicfootprint of the platform. Moreover, whenever measuringa MBB operator, MONROE uses commercial-grade mobilesubscriptions that are compatible with the ones customerscan purchase. The di�erences in commercial o�ers thus re-�ect in the di�erent data plans we activate in MONROE and,subsequently, in the datasets we can collect. For example,the number of samples we can collect from measurementson Orange (ES) (10GB) is much smaller than the number ofsamples we can collect on Telia (SE) (200GB).

3 FLEX-MONROE INTEGRATIONIn this section, we give a detailed description of the FLEX-MONROE platform and the con�guration involved for theintegration of the two testbeds together. We also outline1https://github.com/MONROE-PROJECT/Experiments

CommercialOperators

ExperimentalLTENetwork

UserAccess&Scheduling

MONROEStorage

ExperimentDeployment

MONROENodes(MobileorStationary)

ReservationSystem

FLEX-MONROENodes

ExperimentalNetworkReservation

INTERNET

ExperimentalResults

Figure 1: High level design of the FLEX-MONROE plat-form. The end user needs to connect to both the MONROEUser Access and Scheduling and the FLEX NITOS Reser-vation System. After reserving the LTE experimental net-work via NITOS, the end-user can deploy experiments onthe FLEX-MONROE nodes using the MONROE user inter-face in the same manner as deploying on normal MONROEnodes connected to commercial operators. The Experimen-tal Results are transferred to the MONROE Storage solution,fromwhere they are linked back to the experimenter via theMONROE User Access web interface.

the work�ow an experimenter needs to follow for runningmeasurements in FLEX-MONROE.

3.1 FLEX-MONROE PlatformThe base idea for achieving the FLEX-MONROE integrationis to have a MONROE node operate in the FLEX premises2

and run the available MONROE measurements while attach-ing to the experimental LTE networks. In other words, ouraim is to measure the experimental LTE networks in FLEXin the same manner as we measure any other commercialnetwork in MONROE. This allows for a uni�ed approachfor running experiments both in commercial and experi-mental networks. We show in Figure 1 the high-level FLEX-MONROE system architecture.

The FLEX-MONROE node can be either the hardware na-tive MONROE node running the MONROE software (hard-ware (HW) integration) or can be a hardware native FLEXnode running the MONROE node software (software (SW) in-tegration). For HW integration, we installed within the FLEXNITOS testbed two physical MONROE nodes equipped withcustom FLEX SIM cards that are con�gured with the NITOSPLMNs. For SW integration, the MONROE node softwareimage is installed in any compatible NITOS native node, fol-lowing the normal steps for experimenting with the NITOStestbed3. This integration required support in the MONROEsystem, of the UEs currently deployed in the FLEX NITOStestbed, e.g Huawei E392 and ZTE MF831. Once activated, the2We further refer to these measurement nodes as FLEX-MONROE nodes3http://nitlab.inf.uth.gr/doc/load_saveOMF.html

FLEX node running the MONROE software image becomesavailable in the MONROE system just as any other nativeMONROE node. In both options for integration, the resultingFLEX-MONROE nodes connect to the NITOS experimentalLTE network as they would connect to any other commer-cial LTE network within the MONROE system. We havetested extensively the connection of the nodes with NITOSas a service provider (PLMN ID 46099) and found everythingworking normally. The nodes run the native MONROE soft-ware without any customization and communicate with theback-end infrastructure in the MONROE system, similar toany native MONROE node. The FLEX-MONROE nodes (onceconnected to the NITOS LTE network) are thus capable torun any experiments MONROE o�ers as a service to thecommunity and run these experiments in isolation whilecontrolling the FLEX LTE network (we further exemplifythis in Section 4). Apart from the existing MONROE EaaS,the experimenter has the freedom to design, implement andrun its custom measurements on FLEX-MONROE.

In order to interact with the platform, the FLEX-MONROEuser needs to access both the standalone MONROE system4,as well as the standalone FLEX NITOS system5. Then, anormal work�ow in order to deploy experiments on FLEX-MONROE requires the synchronization of the reservationsin both systems and follows a number of steps, which weexplain below.

3.2 Resource ReservationRunning an experiment in FLEX-MONROE entails a pre-liminary phase where the user reserves the required re-sources. This is a two-step process as resources from bothFLEX and MONROE testbeds need to be accessed (Figure 1).However, the only legal combination in terms of reserva-tion steps requires the experimenter to �rst access FLEXhttp://nitos.inf.uth.gr/reservation and ensure that the LTEnetwork is exclusively under his/her control. To ensure fairaccess to the FLEX resources, there is a maximum limit forthe reservation period of 4 hours per day and per resource.Although theoretically possible, deploying experiments ona FLEX-MONROE nodes without previously verifying thestatus of the eNodeB is not recommended since the networkresources can be shared with other users or the LTE net-work might not even be active. Within the FLEX reservationperiod, the experimenter can use the MONROE User Inter-face and Scheduling system6 for deploying measurements inFLEX-MONROE (further discussed in Section 3.4). The exper-imenter then inputs the duration of the experiment he/shemeans to deploy, which implicitly delimits the reservationperiod for the FLEX-MORNOE node. The experimenter can4https://www.monroe-system.eu/5http://nitos.inf.uth.gr/6https://www.monroe-system.eu

Table 1: LTE parameters that the experimenter can modifyin the NITOS experimental network.

Parameter Description RangeDL BW Downlink bandwidth 5/10MHzUL BW Uplink bandwidth 5/10MHzPower Signal transmit power -15dbm to -26dBmTx Mode Enabled antennas 1/2MCS DL Downlink MCS pro�le 0-28MCS UL Uplink MCS pro�le 0-26FQ Band LTE band 7/13

further coordinate these experiments with similar ones inthe MONROE platform, for comparison with operations incommercial networks.

3.3 NITOS Con�gurationAssuming the experimenter successfully completed the reser-vation phase in FLEX-MONROE, the next step is to accessand con�gure the NITOS LTE experimental network forfurther experimentation. To access the LTE network, theexperimenter �rst connects to the NITOS server using acerti�ed NITOS slice7. Using the OMF Aggregate Managerservice LTErf, the experimenter can then control the LTEbase station parameters. A dedicated LTErf server runs onLTE base station that allows for con�guration of the LTEparameters. The experimenter can set both the eNB and EPCto their default values to ensure that they are on a ’clean’state or recovery from possible changes that took place fromprior users. After having the above steps completed, the ex-perimenter can check the state of the SIRRAN’s LTE corenetwork. This entails verifying the connection status of thecomponents comprising the architecture of the LTE core net-work (i.e., the Home Subscriber Server, the Packet GateWay,the Serving GateWay and the Mobility Management Entity).When all elements report the connected state, LTE networkis ready for experimentation. We summarize in Table 1 a listof con�gurable parameters with a short description and therange of possible values. The experimenter can both queryand replace the value of each parameter using the LTErfservice. Also, one can both infer and repeat the con�gura-tion settings of commercial network in FLEX-MONROE un-der controlled settings. The value �eld should be restrainedwithin the boundaries speci�ed in the LTErf documentation.Restarting NITOS base station is required for the changes totake place. After having con�gured all the parameters, theexperimenter can save the con�guration for future use.

3.4 Experiment DeploymentAs a �nal act, the FLEX-MONROE experimenter can usethe MONROE User Access and Scheduling system to de-ploy a measurement campaign. MONROE uses Linux Docker7http://nitlab.inf.uth.gr/doc/lte.html

containers to host and deploy a variety of user-speci�c ex-periments. As part of the EaaS initiative, MONROE users canalready con�gure and deploy measurements on any MON-ROE node, both native MORNOE nodes or FLEX-MONROEnodes. The experimenter can also prepare its own customexperiments. All experimental results are stored within theMONROE back-end, using the same solution we o�er to allMONROE authenticated users. From there, results link backto the user, who can download them by accessing at any timethe MONROE user access and scheduling system.

4 WEBWORKS IN FLEX-MONROEIn this section, we demonstrate how to use the FLEX-MONROEplatform to run measurement campaigns using MONROEEaaS. In particular, we direct our attention to web browsing.The end-user experience depends on many factors dictatedby the MBB ecosystem, including radio network parame-ters (e.g., MCS and transmission mode settings), wirelesschannel conditions (e.g., congestion and interference), trans-port settings (e.g., protocol parameter and congestion controlalgorithms) and application layer settings (e.g., caching, dis-tance to CDNs, size, type and number of page objects). FLEX-MONROE o�ers the ideal setup for such research, since itprovides all the pieces in this chain. We further leverageWebWorks8, the EaaS MONROE o�ers for testing web perfor-mance by monitoring the Page Load Time (PLT) metric to-wards particular target websites. In the following subsections,we describe the web-performance evaluation experimentswe run in FLEX-MONROE.

4.1 WebWorks DesignWebWorks enables the collection of multiple web performancemetrics while visiting a target webpage using Firefox inheadless mode. We leverage the Selenium web automationframework [21] to simulate web sur�ng and collect webperformance metrics in MONROE. Among the several toolsthat the framework provides, the Selenium webdriver o�ersa large set of APIs to interact with a given web browserin the same way as a regular user would. For example, wecan use the APIs to click on links, buttons or to enter textin input forms. We enable WebWorks to use Selenium withFirefox: upon invoking the webdriver, WebWorks launches thenative Firefox browser in the MONROE nodes to visit anytarget input webpage. We set the user agent string in Firefoxas to retrieve mobile versions of the pages from the webservers. MONROE nodes are not equipped with displays thatGUI-based programs like Firefox require to render the output.We thus use the X virtual framebu�er (Xvfb [2]) to mimic themissing display and enable the browser to behave normally.

8https://github.com/MONROE-PROJECT/Experiments/tree/master/experiments/WebWorks

InternetWebWorks

Targetwebsites:1. bbc.co.uk2. ebay.co.uk3. wikipedia.org

Datasets

• WebWorks• Metadata• FLEXnetworkstate• QoS

FLEX-MONROEStorage

OfflineDataAnalysis

ObjectiveQoE

PageLoadTime(PLT)

Figure 2: The analysis of web performance in FLEX-MONROE using MONROE WebWorks.

During each experiment run we use the HAR export triggeradd-on [28] to log Firefox’s interactions with the visitedpages in a JSON-formatted archive �le called HAR (HTTPArchive). We then use the HAR �le to derive a number ofparameters and metrics, including PLT, size of web pagesin bytes, number of objects, size of each objects, number ofdomains. WebWorks tracks as main performance metric thePLT, which is primarily based on OnLoad event triggeredby the browser. This event is �red when all objects on apage are loaded. Additional parameters such as object types(javascript, css, image etc.), object load time including DNSresolution time, TCP connection time, object receive timingsare also available from the HAR �les. Apart from the HAR,we collect RTT statistics during each experiment run.

4.2 Methodology and Experiment SetupIn Figure 2, we depict the work�ow for the FLEX-MONROEweb experiments. The methodology we put forward con-sists of a hybrid measurement approach in FLEX-MONROE,where we deploy the WebWorksmodule �rst in operational net-works through MONROE platform and then in a controlledexperimental environment through the FLEX-MONROE nodesin the NITOS testbed. This approach enables us to achieve across-layer empirical understanding of how varying the LTEnetwork con�guration in�uences the end-user sessions.Target Websites: Aside from the impact of the networkconditions, the content in the website is an important factorimpacting an end-user. The category of the website hintstowards its complexity in terms of content [10]. Thus, toidentify the factors that a�ect PLT, we analyze completeweb page loads of three popular target web pages fromdi�erent categories that capture the di�erent interests ofend-users, namely wiki portal (wikipedia.org), news portal(www.bbc.co.uk) and on-line shopping (ebay.co.uk). We limitthe number of websites to only three in order to complywith the data quotas available for each experimenter on theMONROE platform. All nodes resolve the target websitesusing Google’s public DNS resolver; not the mobile carrier’sdefault resolver.

Table 2: Distribution of samples per website and operator.

Operator bbc.co.uk ebay.co.uk wikipedia.orgOrange (ES) 63 113 44Yoigo (ES) 45 32 33Wind (IT) 33 20 32TIM (IT) 42 45 39Vodafone (IT) 50 46 393 (SE) 102 196 77Telenor (SE) 251 88 93Telia (SE) 150 169 67Telia (NO) 60 121 12Telenor (NO) 131 183 60

MONROE Campaign: We ran WebWorks against the threetarget web pages using HTTP1.1 [12] in 39 MONROE nodesover a period of 2 months. The content of popular websitessuch as the ones we mention above changes dynamically.Also, the location of the server(s) delivering the web contentis not �xed. It is bene�cial, in this case, to run the experimentover a longer period in time and analyze web performancewhen the web content varies. In total, the MONROE nodesreported web-browsing performance from 10 di�erent oper-ators9 (Table 2). The purpose of running web-based experi-ments with di�erent operators is to determine the variationof PLTs over di�erent con�gurations in the wild.FLEX-MONROE Campaign: To validate the impact ofchanges in the LTE network, we exploit the opportunitiesprovided by the experimental FLEX platform for tweakingthe values of the network-side parameters, such as the Modu-lation and Coding Scheme (MCS) or the transmission powerlevel from the eNodeB. In particular, we aim to capture hownetwork con�gurations impact the experience of the enduser through the PLT metric. We investigate whether a raisein transmission power level from eNodeB impacts the PLTpositively. This is expected since the corresponding raise insignal quality results in higher Channel Quality Indicator(CQI) reporting from LTE UE to its serving eNodeB. HigherCQI means selection of higher MCS for downlink transmis-sion, which is the deciding factor for the data rate.Datasets: The metrics we retrieve in the WebWorks datasetinclude PLT, size of web-page in bytes, number of objects,types of objects, size of each object, per-object downloadtime (including DNS resolution, TCP connect, send, waitand request timings) and di�erent numbers of domains thatare accessed during the experiment. In the same time asrunning WebWorks, we retrieve network-side context features(Metadata) from all the MONROE node(s) we instrument,both native MONROE and FLEX-MONROE nodes. The listof parameters is extensive and includes the radio accesstechnology (RAT), connection mode, signal strength metrics,

9Roaming operators were excluded from the measurement campaign.

RAT-speci�c parameters, sensor data (e.g., node tempera-ture), RTT and number of hops to the primary web-pageserver. While experimenting with FLEX-MONROE, we regis-ter the state of the LTE network, which we log in the FLEXnetwork state dataset. The FLEX network state dataset col-lects the state of various wireless parameters (e.g., frequencyband indicator, downlink bandwidth, uplink bandwidth, mod-ulation and coding pro�les, etc.), the power control param-eters (e.g., PUCCH SINR Target, PDCCH Power Control,etc.), and CQI report. We also leverage other MORNOE EaaS(i.e., HTTP download) and collect QoS parameters in FLEX-MONROE. This allows us to observe how variations in theQoS relate with the network context data and further impactthe web performance. We store all this data in the MONROEstorage solution, where we de�ne a special repository forFLEX-MONROE (Figure 2). We then use this data for furthero�ine analysis (Section 5).

5 WEB PERFORMANCE EVALUATIONAs mentioned in the previous section, we run a series of web-browsing experiments in FLEX-MONROE using three targetweb-pages: bbc.co.uk, ebay.co.uk and wikipedia.org. We �rstinvestigate the commercial networks, examining the perfor-mance of di�erent operators. Figure 3 reveals that the opera-tors di�er in their PLT performance both across websites andcountries. We observe that Telia (SE) has signi�cantly betterperformance across all websites with, 3 (SE) and Telenor (SE)falling equally behind. To the contrary, Wind (IT) seems tounderperform while showing large variance in PLT betweenmultiple experiments. When it comes to Norway and Spain,there is no clear pattern and web performance varies acrossdi�erent web pages.

All three websites are dynamic with most of the PLT raisedby JavaScript objects and images. The wikipedia.org webpagehas the lowest number of objects, with almost all objectsretrieved from the same domain (Table 3). In our analysisof the MONROE web results, regarding factors impactingPLT of a web-page we found that PLT is highly impactedby the websites’ own features, namely number of objectsdownloaded, number of di�erent domains that a page access,and statistics related to objects’ sizes.

We �nd that all the WebWorks experiments ran in LTE net-works with frequencies of 800, 900, 1800 and 2600 MHz andbands from 508, 1551, 430, 46 and 344. Among the networkparameters we monitor in the metadata, we �nd that thenumber of times the RSRP value changes during an exper-iment has the highest impact on the PLT, followed by theLTE band and the average latency values we measured withping.

While analyzing closer the impact of RSRP variation pernode on the webpage PLT, we observed that for signal strengths

5

10

15

20

25

PLT

(s)

bbc.co.ukebay.co.ukwikipedia.org

16

18

20

22

24

PLT

(s)

Ora

nge

(ES

)

Yoig

o (E

S)

Win

d (I

T)

TIM

(IT

)

Vod

afon

e (I

T)

3 (S

E)

Tele

nor

(SE

)

Telia

(S

E)

Telia

(N

O)

Tele

nor

(NO

)

0.5

1.0

1.5

PLT

(s)

Figure 3: Di�erences in PLT of target web pages across 10commercial operators.

Table 3: Features of target web-pages. For each feature weshow the average value and in parenthesis the standard de-viation. For the JS objects, images, CSS andHTML objects weshow the load time reported to the total PLT.

Featuresmean(std)

bbc.co.uk ebay.co.uk wikipedia.org

PLT 19s (6s) 22s (5s) 0.8s (0.7s)# objects 93 (10) 178 (6) 6 (1)# domains 32 (2) 24 (1.6) 1.07 (0.2)JS / PLT 0.41 (0.08) 0.13 (0.08) 0.31 (0.11)Images / PLT 0.42 (0.08) 0.78 (0.11) 0.49 (0.14)CSS / PLT 0.06 (0.03) 0.009 (0.005) 0 (0)HTML / PLT 0.07 (0.03) 0.05 (0.03) 0.19 (0.18)

that are close in values – the PLT does not show direct im-pact. However, by generating bins of signal strength valuesthat are within a distance of 10 units to each other, the barcharts show a tendency of slightly decreasing PLT and lessvariability with better signal quality. Due to space limita-tion we have shown results (RSRP vs PLT) from a singleMONROE node in Figure 4.

Observing the impact of signal quality on the PLT in theoperational networks, we next con�gure the LTE network inFLEX-MONROE to analyze the same phenomena under thecontrolled settings. To achieve this, we run the same Web-Works experiments for di�erent transmission power levelsat eNodeB. Note that the power level changes at eNodeB inreturn a�ects the received signal strength values. We observethat PLT improves with raise in RFSignalPower (Figure 5);

−110 −100 −90

1416

1820

22

RSRP

PLT

(s)

bbc.co.uk

−110 −100 −90

1820

2224

RSRP

ebay.co.uk

−110 −100 −90

0.2

0.4

0.6

0.8

1.0

RSRP

wikipedia.org

Figure 4: PLT at di�erent signal quality levels for a partic-ular MONROE node.

−23 −20 −17

14.5

15.0

15.5

16.0

16.5

17.0

17.5

Power

PLT

(s)

bbc.co.uk

−23 −20 −17

1820

2224

Power

ebay.co.uk

−23 −20 −17

0.5

1.0

1.5

2.0

2.5

Power

wikipedia.org

Figure 5: PLT at di�erent power levels

−97 −95 −93 −91 −89 −87 −85

−26

−24

−22

−20

−18

−16

RSRP

Pow

er

Figure 6: RSRP at di�erent power levels

the change in PLT is small though even after a gap of 2 to 3power levels.

In order to understand whether the signal strength varia-tion is large enough to bring greater change in correspondingPLTs of webpages, we ran further experiments. In Figure 6,we observe that though the quality of RSRP reduces withreduction in the RFSignalPower, the overall range of RSRPis quite small, varying only from -85 to -97 dBm. This wholerange is considered Good for transmission where RSRP val-ues in general can have best quality (around -44 dBm) toworst (around -140 dBm). We conjecture that by varying thedistance between the FLEX-MONROE node and the eNodeB,we may observe more notable changes in RSRP and its prop-agating e�ects on performance metrics. We leave this forfuture work.

We further quantify the impact of RFSignalPower changeson the throughput. For throughput analysis we leverageanother MONROE EaaS (the HTTP download, in this case)

−26 −25 −24 −23 −22 −21 −20 −19 −18 −17 −16 −15

12

34

56

Power [dBm]

Thro

ughp

ut(M

B/s)

Figure 7: Throughput at di�erent power levelsand download 100MBytes of data from the same source ineach run. Turning to impact of RFSignalPower changes onthroughput, we observe that due to the propagated e�ect ofRSRP to CQI and to MCS, the data rate gradually changes.We show this e�ect in Figure 7, where the throughput dropsfrom 5 MB/s to 1 MB/s with the fall in power level.

6 RELATEDWORKEvaluation of network performance and assessing the qualityexperienced by end users require thorough systematic end-to-end measurements. Operators and independent agenciessometimes perform drive tests to identify coverage holes orperformance problems in cellular networks. These tests are,however, prohibitively expensive and do not scale well [27].An alternative common approach is to rely on end users andtheir devices to run tests [16, 18–20, 25]. The main advantageof crowdsourcing is scalability [13]. However, repeatabilityis challenging and one can only collect measurements atusers’ own will, with no possibility of either monitoring orcontrolling the measurement process. Thus, using dedicatedinfrastructure [6, 15, 22] tackles many of the limitations ofcrowdsourcing, at the cost of con�ning the analysis to asmaller geographical footprint.

The MONROE platform [3, 4] is the �rst open accesshardware-based platform for custom experimentation withcommercial mobile carriers in Europe. Even more, the MON-ROE project o�ers open datasets on network performanceto the community. Several regulators responded to this needwith ongoing nationwide e�orts [11]. Often, the solutionsthey device are not open to the research community to allowfor custom experimentation, nor do they grant free access tothe measurement results and methodology. Despite its nu-merous bene�ts and exclusive features, MONROE does notallow for experimentation with LTE network con�gurationchanges. Many other individual e�orts o�er access to infras-tructure and technologies [7, 14, 17]. However, no existingtestbed can o�er the service and experience of an operationalcommercial network with the unique perk of controlling dif-ferent elements in the network. FLEX-MONROE �lls this gapby unifying the capabilities of MONROE with the advantageof control over the con�guration of the LTE experimentalnetworks within FLEX [17].

We exemplify the research opportunities FLEX-MONROEo�ers by investigating the interaction between web brows-ing performance and network con�guration. PLT (Section 4)

has become the key determinant of user’s web browsingexperiences [29]. Numerous studies correlate PLT with theend-user QoE [8, 9]. The factors that impact the PLT rangefrom the features of the web-page and the browser to client-device characteristics to the network and physical layerQoS [5, 23, 26, 29]. We focus our study on capturing andvalidating the impact of LTE network side features on web-browsing performance. For example, with FLEX-MONROEwe vary the values for MCS in the NITOS LTE network from0-28, where 28 indicate maximum data rate. We validate thatthe MCS impacts the download throughput, which providesan estimate of expected application level performance. Otherthan the MCS, web-browsing performance is e�ected by loadin terms of number of users as it raises both RTT delay andmaximum achieved throughput.

7 CONCLUSIONSIn this paper, we proposed FLEX-MONROE, a unique testbedthat o�ers uni�ed experimentation capabilities in LTE net-works both in operational commercial scenarios (MONROE)and in controlled experimental scenarios (FLEX NITOS). Weleveraged MONROE EaaS and investigated web-performanceQoE features with respect to network side performance pa-rameters in LTE broadband network. This study allowedus to emphasize the potential of FLEX-MONROE and theexperimentation opportunities it o�ers. The dataset we col-lected will be made openly available upon publication. Also,the testbed is open to authenticated users both in the FLEXand MONROE systems. Users can deploy their own customexperiments or use any of the existing MORNOE EaaS. Ourcurrent e�orts in terms of testbed integration focus on ex-panding the FLEX-MONROE platform to also include otherFLEX facilities, such are the iMINDS wireless testbed. Thislatter testbed is particularly interesting for the possibilityof experimenting with LTE under mobility scenarios in con-trolled environments.

REFERENCES[1] 2017. MONROE. (2017). https://github.com/MONROE-PROJECT.[2] 2017. Xvfb manula page. (2017). https://goo.gl/EPBtpt.[3] Özgü Alay et al. 2016. Measuring and assessing mobile broadband

networks with MONROE. In World of Wireless, Mobile and MultimediaNetworks (WoWMoM), 2016 IEEE 17th International Symposium on A.IEEE, 1–3.

[4] Özgü Alay et al. 2017. Experience: An Open Platformfor Experimentation with Commercial Mobile Broadband Net-works. In (To appear) Proc. of ACM Mobicom. URL for re-view purposes: https://www.dropbox.com/s/ce9ql3rglcgjrl0/MobiCom-Prepress.pdf?dl=0.

[5] Athula Balachandran et al. 2014. Modeling web-quality of experienceon cellular networks. In Proceedings of the 20th annual internationalconference on Mobile computing and networking, MobiCom’14. ACM,213–224.

[6] Džiugas Baltrunas et al. 2014. Measuring the Reliability of MobileBroadband Networks. In Proc. of IMC.

[7] Gautam Bhanage et al. 2010. SplitAP: leveraging wireless networkvirtualization for �exible sharing of WLANs. In Global Telecommuni-cations Conference (GLOBECOM 2010), 2010 IEEE. IEEE, 1–6.

[8] Nina Bhatti et al. 2000. Integrating user-perceived quality into webserver design. Computer Networks 33, 1 (2000), 1–16.

[9] Anna Bouch et al. 2000. Quality is in the eye of the beholder: meetingusers’ requirements for Internet quality of service. In Proceedings ofthe SIGCHI conference on Human Factors in Computing Systems. ACM,297–304.

[10] Michael Butkiewicz et al. 2011. Understanding website complexity:measurements, metrics, and implications. In Proceedings of the 2011ACM SIGCOMM conference on Internet measurement conference. ACM,313–328.

[11] FCC. 2013. 2013Measuring Broadband America February Report. Techni-cal Report. FCC’s O�ce of Engineering and Technology and Consumerand Governmental A�airs Bureau.

[12] R. Fielding et al. 1999. Hypertext Transfer Protocol – HTTP/1.1. RFC2616 (Draft Standard). (June 1999).

[13] Matthias Hirth et al. 2015. Crowdsourced network measurements:Bene�ts and best practices. Computer Networks 90 (2015), 85–98.

[14] Mahmoud I Kamel et al. 2014. LTE wireless network virtualization:Dynamic slicing via �exible scheduling. In Vehicular Technology Con-ference (VTC Fall), 2014 IEEE 80th. IEEE, 1–5.

[15] Zahir Koradia et al. 2013. First Impressions on the State of CellularData Connectivity in India. In Procs. of ACM DEV-4 (ACM DEV-4 ’13).

[16] Christian Kreibich et al. 2010. Netalyzr: illuminating the edge net-work. In Proceedings of the 10th ACM SIGCOMM conference on Internetmeasurement. ACM, 246–259.

[17] N. Makris et al. 2015. Enabling open access to LTE network com-ponents; the NITOS testbed paradigm. In Proceedings of the 20151st IEEE Conference on Network Softwarization (NetSoft). 1–6. https://doi.org/10.1109/NETSOFT.2015.7116191

[18] Ashkan Nikravesh et al. 2015. Mobilyzer: An open platform for con-trollable mobile network measurements. In Proceedings of the 13thAnnual International Conference on Mobile Systems, Applications, andServices. ACM, 389–404.

[19] OOKLA. 2017. http://www.speedtest.net/. (2017).[20] John P Rula et al. 2014. No one-size �ts all: Towards a principled

approach for incentives in mobile crowdsourcing. In Proceedings of the15th Workshop on Mobile Computing Systems and Applications. ACM,3.

[21] Selenium. 2017. (2017). http://www.seleniumhq.org.[22] S. Sen et al. 2011. Can they hear me now?: A case for a client-assisted

approach to monitoring wide-area wireless networks. In Proc. of IMC.[23] Junaid Shaikh et al. 2010. Quality of Experience from user and network

perspectives. Annals of Telecommunications 65 (2010), 47–57. Issue 1.[24] Sierra-Wireless. 1997. https://www.sierrawireless.com/products-and-

solutions/embedded-solutions/products/mc7455/. (1997).[25] Joel Sommers and Paul Barford. 2012. Cell vs. WiFi: on the performance

of metro area mobile connections. In Proceedings of the 2012 ACMconference on Internet measurement conference. ACM, 301–314.

[26] Srikanth Sundaresan et al. 2013. Measuring and migrating web-performance bottleneck in broadband access network. In IMC’13.ACM.

[27] Tektronix. 2009. Reduce Drive Test Costs and Increase E�ectiveness of 3GNetwork Optimization. Technical Report. Tektronix Communications.

[28] HAR Export Trigger. 2017. (2017). http://www.softwareishard.com/blog/har-export-trigger.

[29] Xiao Sophia Wang et al. 2013. Demystifying Page Load Performancewith WProf.. In NSDI. 473–485.