Agile Management and Interoperability Testing of SDN/NFV ... · reference points, leveraging...
Transcript of Agile Management and Interoperability Testing of SDN/NFV ... · reference points, leveraging...
Agile Management and Interoperability Testing of
SDN/NFV-Enriched 5G Core Networks
Taesang Choi, TaeYeon Kim, Wouter Tavernier, Aki Korvala, and Jussi Pajunp€a€a
In the fifth generation (5G) era, the radio internetprotocol capacity is expected to reach 20 Gb/s persector, and ultralarge content traffic will travel across afaster wireless/wireline access network and packet corenetwork. Moreover, the massive and mission-criticalInternet of Things is the main differentiator of 5Gservices. These types of real-time and large-bandwidth-consuming services require a radio latency of less than1 ms and an end-to-end latency of less than a fewmilliseconds. By distributing 5G core nodes closer to cellsites, the backhaul traffic volume and latency can besignificantly reduced by having mobile devices downloadcontent immediately from a closer content server. In thispaper, we propose a novel solution based on software-defined network and network function virtualizationtechnologies in order to achieve agile management of 5Gcore network functionalities with a proof-of-conceptimplementation targeted for the PyeongChang WinterOlympics and describe the results of interoperabilitytesting experiences between two core networks.
Keywords: 5G core network (CN), Agilemanagement, Interoperability between CNs, Networkfunction virtualization (NFV), Software-definednetwork (SDN).
I. Introduction
In the fifth generation (5G) era, the radio internet protocol(IP) capacity is expected to reach 20 Gb/s per sector(mobile speeds up to 20 Gb/s), and ultralarge content traffic(for example, ultrahigh definition video streaming,augmented reality (AR), and virtual reality) will travelacross a faster wireless/wireline access network. All 5Gmobile/fixed traffic has to travel via the packet corenetwork (CN). Currently, in the fourth generation (4G),most mobile operators (even large-scale ones) have only afew sites with packet gateways (PGWs) across their entirenetworks. The software-defined network (SDN) paradigmprovides a new capability for faster service provisioning ofthe 5G CN through standard programmable interfaces.Moreover, with cloud computing, datacenters promote theon-demand provisioning of computing resources andservices [1]. If the 5G core nodes are distributed closer tocell sites, content servers (or caching servers) can be placedon the rack right next to the distributed 5G core withnetwork function virtualization (NFV) technologies. Thiscan help significantly reduce backhaul traffic by havingmobile devices download content immediately from thecontent server. Thus, it is desirable to distribute packet corefunctionality to a number of local sites near end users in thecoming 5G era. The 5G core functionality and applicationscan then run on virtualized servers at the local networksites. Other important 5G services—massive and mission-critical Internet of Things (IoT) services—are the maindifferentiator from 4G services. Mission-critical IoT(ultrareliable and low-latency communications) applicationsinclude remote-controlled machines, autonomous driving,and others. These types of ultra-real-time services require aradio latency of less than 1 ms and an end-to-end latency ofless than a few milliseconds [2].To address such challenges, we present a novel agile
management and orchestration (MANO) architecture
Manuscript received Oct. 13, 2017; revised Dec. 4, 2017; accepted Dec. 18,2017.Taesang Choi (corresponding author, [email protected]) and TaeYeon Kim
([email protected]) are with the Hyper-connected Communication ResearchLaboratory, ETRI, Daejeon, Rep. of Korea.Wouter Tavernier ([email protected]) is with the Department of
Information Technology, Gent University, Belgium.Aki Korvala ([email protected]) and Jussi Pajunpaa (jussi.pajunpaa@
nokia.com) are with Nokia, Oulu, Finland.
This is an Open Access article distributed under the term of Korea OpenGovernment License (KOGL) Type 4: Source Indication + Commercial UseProhibition + Change Prohibition (http://www.kogl.or.kr/info/licenseTypeEn.do).
https://doi.org/10.4218/etrij.2017-0236 © 2018 pISSN: 1225-6463, eISSN: 2233-7326
72ETRI Journal, Volume 40, Number 1, February 2018
http://onlinelibrary.wiley.com/journal/10.4218/(ISSN)2233-7326
based on enabling key technologies for 5G corefunctionalities, a proof-of-concept (PoC) implementationtargeted for PyeongChang Winter Olympics, anddeployment and interoperability testing experiences. Theproposed solution is an interim result of a collaborationproject between the Republic of Korea (KR) and theEuropean Union (EU) [3]. The rest of the paper isorganized as follows. We describe the enabling keytechnologies in Section II. We present the agile MANOarchitecture in Section III. Our prototype implementationand deployment experiences are described in Section IV.A performance evaluation of the proposed system,including the interoperability testing results, are providedin Section V. Finally, we conclude our paper with theplans for potential future work in Section VI.
II. Enabling Key Technologies
This section examines the key technologies for theSDN, NFV, MANO, mobile edge computing (MEC),mobility management, and control plane (CP) security andtheir associated design principles for the support of theproposed CN functionalities and their agile management.
1. Software-Defined Networking and Orchestration
Standardization efforts for SDNs were mainly carried outby the Open Networking Forum [4] and the InternationalTelecommunications Union – Telecommunications StudyGroup 13 (ITU-T SG13) [5] by defining the requirements,reference architecture, protocols, and use cases. Open-sourceprojects such as Open Daylight [6] and Open NetworkingOS [7] have played important roles in realizing the SDNconcept in real life. The SDN started with a limitednetworking environment such as cloud data centers andenterprise networks and has widened its coverage to wide-area transport networks and wireless/wireline integratedmultidomain networks. Instead of applying it as a standalonenetwork control tool, it is now used with NFV and as acomponent of an end-to-end orchestration solution. Itprovides an intelligent knowledge plane for making controldecisions via traffic steering, traffic engineering, and flexibleservice chaining for latency-sensitive and reliability-seekingapplications. It can be used in efficient communicationsamong distributed core functional components.
2. Network Function Virtualization
The virtualization of core and radio access networkfunctions will optimize the use of network resources andadd scalability and agility. To this end, the European
Telecommunications Standards Institute (ETSI) NFVIndustry Specification Group has defined the architecture,open application programming interfaces (APIs), andreference points, leveraging open-source PoC projects andcommunities to drive open standards of NFV. In 2016, itpublished Release 2 specifications and reports, includingthe functional requirements, interface, and informationmodel for the reference points for the MANO functionblock called NFV-MANO [8]. These open standardsare intended to enable third-party vendors to developframework components that can collaborate with variousvendor components so that content service providers) arenot restricted in selecting functional and managementcomponents. The main appeal of the use of NFV to deploynetwork elements and virtual network functions (VNFs) isthat services can be launched more quickly by installingsoftware on a standard hardware platform. This is akin tothe way software applications could be developed andlaunched for the personal computer (PC) platform whenit first emerged. Another advantage is lower capitalexpenditures because standardized hardware platformstend to drive costs down. Such advantages can be directlyapplied to the distributed core functional components inthe communications environment.
3. Mobile Edge Computing
In order to support the requirements for the market’sexpected throughput, latency, scalability, and programma-bility, ETSI established the Industry Specification Groupon Mobile Edge Computing in 2014 [9]. It develops astandardized and open environment that offers distributedcloud-computing capabilities and an IT service environ-ment for application developers and content providers. ByFebruary 2016, the group finalized three specifications:the terminology, the technical requirements, and theframework and reference architecture. This group alsoworks on specifications for MEC platform applicationenablement, the API principles and guidelines, the serviceAPIs for radio network information and location, userequipment (UE) identity and bandwidth management,system/host/platform management, lifecycle and policymanagement, the UE application interface, the deploymentof MEC in an NFV environment, and the end-to-endmobility.By offering distributed cloud-computing capabilities
and exposure to real-time radio network and contextinformation, MEC provides the following characteristics:
• Ultralow latency: Mobile edge services can be run closeto end-user devices to provide the lowest possiblelatency,
73Taesang Choi et al.
http://onlinelibrary.wiley.com/journal/10.4218/(ISSN)2233-7326
• Proximity: Being close to the source of information,MEC is particularly useful for capturing key informationfor analytics and big data,
• High Bandwidth: The mobile edge location at the edgeof the network combined with the use of real-time radionetwork information can be used to optimize thebandwidth for applications,
• Location awareness: A mobile edge can leverage thelow-level signaling information to determine thelocation of each connected device,
• Real-time insight into radio network and contextinformation: Real-time network data can be used by theapplications and services to offer context-related services.MEC can provide a significant improvement in a mobile
user’s quality of experience for latency- or quality ofservice (QoS)-sensitive services such as edge videoorchestration, mobile video throughput guidance, AR,intelligent video analytics, and others. Most importantly,MEC enables the implementation of mobile edgeapplications as software-only entities that run on top of avirtualization infrastructure, which is located in or close tothe network edge.
4. Distributed Mobility Management
It is essential to support distributed mobilitymanagement to enable agile management of the CNfunctionality. Currently, the Internet Engineering TaskForce is conducting standardization efforts to define adistributed mobility management architecture andmechanism in a layer 3 IP network environment. The 3rdGeneration Partnership Project (3GPP) also initiated workon defining layer 2 distributed mobility managementrequirements for a mobile communications environment.The functional decomposition and distribution of globalservice management will span multiple points of presence(PoPs) over the network, including network slices in a5G environment. It would be better to determine theanchoring and mobility management tailored to such anetwork environment at the central node, unlike exitinghierarchical and IP mobility. Composition functions andresources will be orchestrated for dynamic mobilitymanagement. Various experiments and simulations areunder study by the research community, and extensivetesting and verification of the concepts of distributedmobility management are needed.
5. Security of the 5G Core Network Control Plane
A software-defined mobile network (SDMN) controllerwill provide the necessary services to the CN functions by
working as an intermediary between the access and corefunctions. The network control functions of the coreelements, for example, the mobility management entity(MME), serving/packet data network gateways (S/P-GWs),and others, will reside in a centralized cloud in the form ofSDN applications that will leverage NFV technology to beinstantiated on different hardware or even at differentnetwork perimeters for a higher scalability and availability.Hence, the main security concern in such architectures willbe the SDN controller since it can become a potentialbottleneck for the overall network.To mitigate the risks of controller failure due to
scalability or the chances of denial of service (DoS)attacks due to its centralized role, controller resiliencestrategies have been proposed. These strategies includecontroller resilience through redundancy, maximizing thestorage and processing capabilities of the controller, anddistributing controller functionalities among multiplecontrol points in the network. The OpenFlow variant of anSDN supports wildcard rules so that the controller sendsan aggregate of client requests to server replicas. Bydefault, microflow requests are handled by the controllerthat can create potential scalability challenges, increasingthe chances of failures due to DoS attacks. Normally,reactive controllers that act on a flow request when itarrives at the controller are used. Proactive controllersinstall flow rules in advance, thus minimizing the flowrequest queue in the controller. Similarly, various load-balancing techniques that would balance the load amongmultiple controllers in a network have been suggested. Wehave worked on a novel communication architecture basedon the host identity protocol (HIP) to secure both controland data channels in SDMNs.
III. System Architecture
This section describes the proposed CN and agilemanagement system architecture based on a combinationof the key technologies described in Section II.
1. Core Network Architecture
We designed our CN architecture (Fig. 1) [10] tosupport CN functionalities and agile management on thebasis of the various key technologies described inSection II. Specifically, the CN functionality is realized byleveraging an SDN and NFV in order to facilitate thedynamic provisioning of CN functions. By using SDNcapabilities, traffic flows can be dynamically controlled,redirecting the traffic to gateways according to theworkloads. Simultaneously, the introduction of NFV
74 ETRI Journal, Vol. 40, No. 1, February 2018
https://doi.org/10.4218/etrij.2017-0236
permits the separation of service functionalities from thecapacity-constrained specific network entities and allowdynamic instantiation in commodity and powerful servers.Starting from late 1990s, the 3GPP has been taking stepstowards a clear separation of the data and control planesand the respective elements in the architecture. Wepropose to take this concept to the next level following theSDN paradigm. Figure 1 also presents the 5G networkcontrol as a group of SDN applications. They are the BaseStation App, Backhaul App, Mobility Management App(MM App), Monitoring App, Access App, and SecureService Delivery App. The network applications areorchestrated via the Controller Northbound API. MultipleSDN applications operate without conflicts.The Base Station App runs the control software that is
now vertically integrated with the evolved Node B (eNB).The physical base stations under its control consist of anantenna, a band-pass filter, and an Ethernet card forbackhaul connectivity [11]. The MM App implementsmobility as a service and incorporates the MME. Inaddition, it needs to manage the QoS for each user,balance the load among alternative paths across theaggregation network, and route the user to a cache whenpossible. The MM App also chooses the path for a device.The load-balancing decision is made on the basis of theinput from the Network Monitoring App. In any case, it isdesirable that the point of attachment of a mobile device tothe Internet is fixed while it remains within the coverageof the current mobile network [11].In one physical mobile network, there may be many
Access Apps. In this case, an Access App is owned andoperated by a particular mobile virtual network operator.Putting mobility aside, the Access App is responsible forproviding data services to mobile users. The keyproperties of the Access App include providing Internetaccess, firewalling unwanted traffic, and providing accessto premium content [11].
The main CN functions are designed and implementedin the form of virtual functions, namely, virtual evolvedpacket cores (vEPCs). Both the EU and KR provide theirown implementations of vEPCs based on this architecture.They are described as follows.
2. European vEPC Architecture (5GTN)
The EU vEPC consists of the following VNFs:
• Mobile gateway: The cloud mobile gateway providesthe service provider-gateway (SP-GW), gateway generalpacket radio service (GPRS) support node, and trafficdetention functions (TDFs), evolved packet datagateway, and trusted wireless access gateway.
• Mobility management: The cloud mobility managerprovides the MME and servicing GPRS support nodefunctions.
• Policy control and charging: The dynamic servicescontroller built on patented agile rules technologyengine provides the policy and charging rules function(PCRF) and wireline radius/change of authorization.Element and network management: The service-aware
manager provides end-to-end network managementvisibility across the entire mobile network.To support the scalability required to meet the expected
5G and IoT service requirements, the packet core VNFsprovide three key design innovations:
• The packet core VNFs are decomposed into separate CPand data-plane virtual machine (VM) instances. Thisenables a distributed architecture where data-planeresources can be deployed in edge data centers closer tothe device, while CP resources can be centralized.
• State-efficient VNF processing unpins the subscriber/device state information from the VMs, freeing up theunderlying computing resources to be reused to processother subscribers/devices.
• The remote cloud database synchronizes the subscriber/device state information into a real-time data store.
• The 5GTN functional architecture [10] is given inFig. 2.
3. Korean vEPC Architecture
The CN of 4G Long Term Evolution (LTE) is in chargeof mobility, authentication, and charging, allowing allmobile traffic to pass through the CN to access servicesincurring traffic congestion in the CNs. Our architecturaldecision for 5G is to distribute mobile core functions tothe edge nodes. A 5G core is generally divided into a 5Gcore user plane (UP) in charge of bearer delivery and a 5GCore CP in charge of signaling and control of the 5G CN.
Applications
InfrastructureCache DPI FW
Internet@ccess
SDN controller NorthboundSouthbound
PCRFHSS
Topologydiscovery
Link provisioning
Load balancingSLA
Service recovery
MME Cache Policy Firewall CES
DNSDHCPBackhaul provisioning
Mobility management
Internet access
Base station Network monitoring
Mobility management
Access control
ChargingSecure service delivery
Fig. 1. CN architecture.
75Taesang Choi et al.
http://onlinelibrary.wiley.com/journal/10.4218/(ISSN)2233-7326
The key CN architectural design principle is a centralizedCP with a distributed UP over the edge nodes.If the CN where bearers are terminated is located closer
to the cell sites, the application servers follow naturally, andthe backhaul traffic will significantly decrease, resulting ina cost reduction for continual backhaul enhancement.A 5G network is supposed to be able to provide
ultra-real-time services such as highly sensitive remotecontrol and automatic driving vehicles. These types ofservices may generate much lesser traffic than videostreaming applications but require an ultralow latency.Figure 3 illustrates the high-level architecture of aKorean vEPC. It is realized as a highly scalable vEPC(HSvEPC) [12]. Its functionality and architecture aredescribed below.
4. HSvEPC Network Architecture
It is possible to deploy different types of virtual mobilepacket cores depending on the demand or network accessenvironment in an HSvEPC network architecture. Twotypes of vEPCs are designed (shown in Figs. 4 and 5):
• Split vEPC (S-vEPC): The first type is an expansion of avEPC by separating conventional consolidated functionsinto UP and CP functions for dynamic scaling operations.
• Mobile hotspot network vEPC (MHN-vEPC): The othertype is an optimized case for a hotspot area to enhancethe agility of the network. For faster and more dynamicmobility management in a mobile hotspot area, the S1(single interface between the LTE radio access networkand the EPC) interface of the virtual EPC has beenmodified in terms of the UP and CP.
5. Management and Orchestration Architecture
Figure 6 shows overall MANO architecture on the EUside based on NFV MANO and an SDN. The architecturehas two management entities:
• The VNF manager is in charge of instantiating andcontrolling EPC functions. It is responsible for
Central data centerCloud packet core
Scale in & out
OAM Q&M
CP DPScale in &
out
OAM-VM
Control plane VM
Load balancer VM
Load balancer VM
Data plane VM
Shared data layer
Edge data center
CP DP
Scale in & outLB LB
LB
LBLB
Fig. 2. 5GTN vEPC functional architecture.
5G service managing Health SecurityEnergy Media Transport
IFA007
IFA005
IFA008 IFA008
VNF managerNFV VIM
Mobility management
Traffic monitoring
Traffic optimization
Service function chaining
High-availability control
SDN controller
Internet(KOREN/KREONET)
Radioaccess
wlan
MHNDistributed vEPC Distributed vEPC
MHN
wlan
RadioaccessApplications
5G core(CP)5G core(UP)
Applications5G core(CP)5G core(UP)
Orchestrator Inventory management
VNF on-boarding Provisioning MonitoringVNF/NS lifecycle management
Fig. 3. High-level architecture of Korea’s distributed vEPC.
UE eNodeB
eNodeB
HSS PCRF SPR
GTP-C(S11)
Diameter(S6a)
*SDP GTP-C(S5)
GTP-C(S5)
GTP-U(S1-U)
Diameter(Gx)
Diameter(Sp)
*SDP SGiPDN
X2-AP/GTP-U(X2)
S1-AP(S1-MME)
vEPCvMME
vSGW-CU
vSGW-DU
vPGW-CU
vPGW-DU
Fig. 4. HSvEPC functional architecture: S-vEPC.
mDUmTE
mRUmGW
(HEvEPC)
AP
AP
SwitchmRU
User
User
mNB
CPRI GbEmmWaveGigaWiFiaccess
User
······
GbE
mRU Access link(inside vehicle)
mRU
mDU mDU
mGW
mRUmRU
mTE
Intra-DUmobility controller
Inter-DUmobility support 5G core
network
···Backhaul link
: mmWavemGW:MHN gatewaymNB: MHN node BmDU: MHN digital unitmRU: MHN radio unit mTE: MHN terminal
equipment
e
IP
Fig. 5. HSvEPC functional architecture: MHN-vEPC.
MME HHS SGW/PGW-S
SDN-C (local)
vCompute vNetworkvStorage
Virtualization
Compute NetworkStorage
vCompute vNetworkvStorage
Virtualization
Compute NetworkStorage
VNF-1 VNF-2 VNF-3 VNF-1 VNF-2 VNF-3
Firewall DPI Video optimization
SDN-C(global)
OF N/W
Internet
MANO
OpenFlow
SDN-C (local)
Orchestrator
Infra manager
VNFM
Fig. 6. Overall EU MANO architecture with an SDN.
76 ETRI Journal, Vol. 40, No. 1, February 2018
https://doi.org/10.4218/etrij.2017-0236
interacting with VNFs, chaining VNFs, and handlingtheir lifecycle—instantiation, maintenance, and others. Itis in charge of the operation and configuration of VNFsthrough the operations support system (OSS)/basestation subsystem (BSS). It will handle multifunctionalEPC components such as the MME and homesubscriber server (HSS) as well as specific-functionalityVNFs such as firewalls and deep packet inspectors.
• The infrastructure manager interacts with (or incorporatesthe capability of) the SDN controller in the servicestratum when deploying VNFs for configuring thecomputing and storage resources for the VNF of interest.It also supports the attachment of the VNFs to the borderof the underlying transport network for the networkingpart to make them reachable from outside the data center.This is only for the service-layer part. It also has todetermine a path for the transport-layer VNFs.The KR CN MANO are also based on NFV and SDN
components. Figure 7 shows the MANO architecture. Itcomprises three different entities: the NFV orchestrator(NFVO), VNF manager (VNFM), and virtualinfrastructure manager (VIM).The NFVO is responsible for managing functions such as
network service (NS) lifecycle management and overallresource management. Service management or orchestrationdeals with the creation and end-to-end management ofservices by composing different VNFs. Resourcemanagement helps to ensure that the NFV infrastructure(NFVI) resources are abstracted cleanly (independent of theVIM) to support the services that access these resources.The VNFM oversees the lifecycle (which typically
involves provisioning, scaling, and terminating)management of instances of a VNF. In this case, each VNFis associated with a VNFM that will manage that particularVNF’s lifecycle. A VNFM may manage multiple instancesof the same type of VNF or different types of VNFs.The VIM controls and manages the NFVI computing,
storage, and network resources. The VIM component hasbeen the focus of a large amount of research and variousopen-source solutions such as OpenStack and has beenused to realize the virtualized infrastructure managementfunctionality of MANO.
6. Autoscaling Based on Performance/FaultManagement
In our M&O, autoscaling functionality is provided asshown in Fig. 8. After instantiation of a 5G mobile CNservice, the NFVO sends a supervision request to thesupervisor, which performs performance monitoring andfault notification over virtualized resources and functions.Scaling is conducted autonomously by the orchestrator onthe basis of the information provided by the supervisor.
7. Automation by Event Chaining
An event chaining process is another importantfunctionality that is supported, which is defined as asequence of event units occurring from inside or outsidethe target VNF and virtual data unit (VDU). It enables theautomation of 5G mobile CN management. Acombination of internal events that are significant in asingle VNF or VDU and external events between VNFsand VDUs enables the automated management of alifecycle of a mobile CN service (see Fig. 9).
Designer(NFV graphic
designer)
LCM(life cycle manager)
For VNF and other SW
VM(VNFC)
MONITORING(OMW)
ETSI ISG NFV
Or-V
i
Vi-Vnfm
Or-Vnfm
Ve-Vnfm-vnf
Ve-Vnfm-em
Os-Ma-nfvo
Nf-Vi
Generic VNFM
VIM
TACKER based
Designer
Or-
Vi
RESTfulJSON structure
Public VNFMmade by INSOFT
RESTfulstructure
JSON
RESTfulstructureYAML
*LCA
JSONhttp
*MAxml
socket
NFVO
VNFM
VIM
RESTful
VIM
OPENSTACK
NFVO
VIM DRIVER
Fig. 7. Overall KR M&O architecture with MANO and an SDN.
NFVO
Interface Monitoring
OpenStack infra.
VM &VNF
Monitoringagent
Monitoringagent
Monitoringcollector
VIMOpenStack
2) Instantiation 3) ProvisioningPre.) set aggregate PM/FM and each indicates
1) Send aggregated alarm to NFVO
VDUprocess
OS
2) L
ook
up
NS /
VN
F
3) VNF/VNFC
Performanceinterface
FaultinterfaceIndicateinterfacePM/FM alarm
interface
Performanceinterface
FaultinterfaceIndicateinterface
Alarminterface
Fig. 8. MANO autoscaling process.
Event chain(for automation)
OrderInternal
Internal
VNF1
Event chain(for automation)
VNF2
Order
External(like a
function)
Internal
Internal
Reference
Event(for setting internal)
Event(for setting outside
information)
The event needs information to set
internal config
External(like a
function)
External(like a
function)
Reverseexternal
Fig. 9. MANO automation process.
77Taesang Choi et al.
http://onlinelibrary.wiley.com/journal/10.4218/(ISSN)2233-7326
8. Security Management Architecture
The security of the CN can be grouped into two parts:the security of the CN elements and the security of thecommunication channels in the CN. In SDNs, controllingthe behavior and interworking of different heterogeneousnetworks is carried out with a logically centralized controlarchitecture that has a global view of all forwardingelements. An operating system maps the entire network toservices and applications that are implemented on top ofthe control plane. Hence, security services will beimplemented as security applications using the networkstats provided either proactively or reactively by thenetwork control platform. Centralized control, which canbe either logically or physically centralized, enables theprogrammability of the network and will thus providefine-grained network security control, remote monitoring,and dynamic security service insertion. The securitymanagement architecture is presented in Fig. 10.
IV. Implementation and Deployment Experience
Both the EU and KR edge and CN functions are underdevelopment. The development of some components hasbeen completed, such as EU’s edge and core functions in a5GTN solution. KR’s vEPC development is underwaywith the core functionality completed. The KR vEPCcurrently supports up to 100 UEs and a channelthroughput of 20 Gbps toward an eNB. To meet the 5Gkey performance indicator (KPIs), we are trying to fill thegaps in both vEPC systems. We are targeting thecompletion of our system development by October 2017.We are also developing our agile CN MANO systems
based on the architectures described in Section II. Initialprototypes are available, and their functionality as separatesystems and their interoperability are being tested as well
across the EU and KR over interconnected research anddevelopment networks between the EU and KR via theKorea Research and Education Network (KOREN)–TransEurasia Information Network (TEIN)–Nordic CountriesNetwork (NORDUNET)–Finnish University and ResearchNetwork (FUNET).
1. vEPC Implementation
5GTN vEPC VNF functions have been implemented,deployed, and tested on CloudBand’s NFVI and itsMANO solution. CloudBand is a hardened, production-ready NFV solution based on OpenStack and other open-source technologies. This open approach allows serviceproviders to benefit from a vast community of engineersand supports investments in a mainstream solution withopen interfaces.The HSvEPC implementation consists of a vMME,
vSGW-CU, vPGW-CU, vSGW-DU, and vPGW-DU. TheCU is a CP that controls the device management, and thedata unit (DU) is a UP that controls the data transferbetween devices. The main reason why we separatedfunctions by each plane is to support scalabilitydepending on the demand situation. Since the functionsin the HSvEPC are implemented as VNFs, they can bemodified on demand and controlled per VNF level. Oneimportant use case of such flexibility is network slicingsupport.Figure 11 shows the access point name (APN)-based CN
slicing use case. An IoT device may have a different APNagainst a UE, and discrimination of each device at the MMEis required. The above use case illustrates ourimplementation of an MME that can classify differentdevices by categorization based on their APNs and mapappropriate resources in the SGW and PGW. Moreover, theHS-vEPC can be scaled in or out depending on the demand,which can reduce the cost, and other unused parts of networkfunctions can be relocated to only the necessary parts.
Network administrator
Security policy
Application plane
Control plane
Data plane
Network users Data path elements
OF switches Sec.
MiddleBox
Controller 1 Controller 2 Controller 3App-specific
API
SDN (OpenFlow) controllersNorth-bound API
South-bound API (OpenFlow protocol)
Security analytics Security
monitoring
Authen. & author.
Access control
DPI
SDN (OpenFlow) applications
Fig. 10. SDN architecture showing the security services andtheir deployment.
IoTdevice
PCRF/SPR
eNodeB
MME
SGW PGW Internet
HSS
IMS
eNodeB IoT-SGW IoT-PGW NB IoTUE
IoT-MME
APN: iot.xxx.com
APN: lte.xxx.com, ims.xxx.com
Fig. 11. HSvEPC core slicing use case.
78 ETRI Journal, Vol. 40, No. 1, February 2018
https://doi.org/10.4218/etrij.2017-0236
2. Management and Orchestration Implementation
MANO in 5GTN has been implemented and deployed.It consists of CloudBand infrastructure software, aCloudBand application manager, and a CloudBandnetwork director that have been optimized to fit the keyNFV MANO shown in Fig. 12.
• CloudBand Infrastructure SoftwareThe CloudBand infrastructure software is a multipurposeNFVI and VIM. It virtualizes and manages computing,storage, and network resources.
• CloudBand Application ManagerThe CloudBand application manager is a VNFM thatautomates lifecycle management actions by managingresources and applying associated workflows.
• CloudBand Network DirectorThe CloudBand network director is an NFV resource andNS orchestrator. It manages virtual resources acrossgeodistributed NFV infrastructure nodes. It visualizes andautomates the lifecycle of NSs, such as virtual customerpremise equipment (CPE), including their forwardinggraphs and service chains.The KR MANO implementation is shown in Fig. 13.
We have implemented it in a rack of servers consisting ofa VIM built and extended over OpenStack, a VNF
manger, and an orchestrator. The management target is, ofcourse, a set of virtual functions implementing CNfunctionality and networks that interconnect those virtualcore functions.
3. Deployment and EU–KR Interoperability Testing
First, phase field deployment and interoperability testingbetween the EU and KR was conducted in July 2017 [13].Both the EU and KR vEPC and MANO have beendeployed. Figure 14 shows the EU’s 5GTN deploymentnetwork environment. The 5GTN network elements inOulu are physically located at two different sites. TheeNBs and Juniper SRX240 router are located at Site 1.Juniper SRX240 connects external entities to 5GTN. Datacenter Tampere in Oulu is connected via a layer 2 virtualprivate network (L2VPN) connection. KR entities are alsoconnected using an L2VPN connection. The EPC isphysically located at Site 2 and connected throughUniversity of Oulu (UOulu) switches to a radio accessnetwork.Figure 15 shows the 5GTN vEPC AirFrame hardware.
The European testbed vEPC has 11 servers, of which threeservers are used as controller nodes and eight servers areused as compute nodes; one hardware (HW) managementswitch; and two leaf switches.
Serv
ices
Element/network managementNetAct 3rd party5620 SAM
VNFsNokia VNFs 3rd party VNF
Service management/BSSNokia software portfolio 3rd party
Other cloud stacks
3rd party
Hardware infrastructureAirFrame 3rd party
CloudBand network director
CloudBand application manager
3rd partySDN
Nuage networks
OpenStack
CloudBand infrastructure
software
Network service orchestration resource orchestration and optimization aggregated view, distributed infrastructures
Automated life cycle execution, rapid onboarding, 40+ VNFs on boarded
Carrier-grade, pre-configured, highly available software on different hardware configurations, blueprinting, automated operations
Fig. 12. 5GTN MANO implementation.
OSS interface Integrated management GUIAccess control
NS configuration management NS
performance & fault
managementInfrastructure
integrated management
NS control
WAN infrastructuremanagement
Infrastructure resource management
VNFM interface VIM interface WIM interface
NFVO
Generic VNFM
Specific VNFM
VNFM
OpenStackVIM
Or-Vnfm
Vi-Vnfm
VNF
vBYOD VNFvScreen
VNFsvIDS vIPS vADC
Or-Vi
Nf-ViHypervisor (KVM)
Compute Storage NetworkNFVI
VIM1
VIM2
VIM/VNFM
Fig. 13. KR MANO implementation. Fig. 15. 5GTN EPC AirFrame hardware.
HSS PCFRDNS
DC tampere (Nokia)
UOulunetwork S-GW
P-GWMME
Nokia AirFrame CORE
JuniperQFX5100
S1-MMES1-US6aSGi
UOulu UOulu/CWCUOulu site 2
Internet
5GTN VLAN
UOulu site 1
JuniperSRX240
CWC
1 Gbit
1 Gbit 1 Gbit
1 Gbit 10Gbit
40Gbit
Dedicated connection Access, cloud core, services…FUNET → NORDUNET→ Geant
→ TEIN → KOREN Korean testbed
1 Gbit
Fig. 14. EU 5GTN deployment environment.
79Taesang Choi et al.
http://onlinelibrary.wiley.com/journal/10.4218/(ISSN)2233-7326
• Server type: 11 9 Quanta B51BP-1U, manufactured byQuanta Computer Inc.
• HW management switch type: 1 9 Quanta LB9,manufactured by Quanta Computer Inc.
• Leaf switch type: 2 9 Juniper QFX5100-24Q switches,each having 2 9 QFX-EM-4Q expansion modules,manufactured by Juniper Networks Inc.The three uppermost servers are controller nodes. The
remaining eight servers are computing nodes. Themanagement switch is below the servers, and the leafswitches are below the management switch.Figure 16 shows the Korean vEPC and MANO
deployment environment. There are three possible PoPs inKR interconnected over KOREN: Seoul, Daejeon, andGangneung, where the PyeongChang Winter OlympicGames take place in 2018. We plan to deploy mobilecore infrastructure for 5G networks at these three sitesfor service deployment. Currently, we deployed oneset of a vEPC and MANO at the Electronics andTelecommunications Research Institute (ETRI) in Daejeonfor interoperability testing with the EU.Our vEPC supports NS provisioning and monitoring
functionality as follows:
• The vEPC NS consists of an MME, a virtual S-GWcontrol unit (S-GW-CU), a virtual S-GW-DU, a virtualP-GW-CU, and a virtual P-GW-DU.
• In our deployment, the vEPC NS does not cover theremaining functionalities for the vEPC (that is, the HSSand PCRF).
• The virtual S-GW-DU and virtual P-GW-DU must havesingle-root input/output virtualization and sharing (SR-IOV)-enabled ports in order to enhance theirperformance.The NFVO, VNFM, and VIM closely interwork with
each other to create and manage NSs. Figure 17 shows theprocedures to provision an NS in the NFV-enabledinfrastructure.
1. An OSS/BSS (or administrators) requests to create anNS at the NFVO by defining a new NS descriptor orselecting one.2. The NFVO requests the allocation of network resourcesat the VIM, which connects the VNFs composing therequested NS. In this step, management network resourcesare also created for management access.3. Once the network resources are allocated, the NFVOrequests the VNFM to instantiate VNFs. Since our NFVIis in the indirect mode, the VNFM indirectly requests theVNF resource allocation at the NFVO, and the thenrequest is sent to the VIM.4. When VNF resources are allocated, the VNFMconfigures VNFs with any VNF-specific parameters.The states of the NSs are monitored with two metrics: a
service utilization metric and a metric for monitoringresource utilization. The NFVO receives the monitoringresults from the VNFM and VIM and exploits the resultsto perform other management operations such as a scalingoperation.Figure 18 shows the two types of monitoring.
• VNF monitoring: VNF providers can specify someindication of VNF behavior, and they include thisinformation as a parameter (that is, VnfIndicator) of the
PTN node
OSS/BSS
EMS EMS EMSVNF VNF VNF
NFVO
VNFM
VIM
PTN node PTN
node
VIMNFVI
NFVIPTN tunnel
Gangneung
ETRI
Daejeon PoP
(@Daejeon)
KOREN NOC(@Seoul)
Seoul/Gangneung PoP
Computing resources
Networking resources
Storage resources
Computing resources
Networking resources
Storage resources
Fig. 16. KR vEPC and MANO deployment environment.
OSS/BSS
EMS
NFVO
VNFM
VIMNFVI
EMS EMS EMS
NS
Usertraffic
4. Configure VNFs
1. Instantiate NS
3. Allocate VNFresources
2. Allocate network resource
Computing resources
Networking resources
Storage resources
Fig. 17. NS provisioning procedures.
OSS/BSS
EMS
NFVO
VNFM
VIMNFVI
EMS EMS EMS
NS
Usertraffic
VNF monitoring
Virtual resources
monitoringComputing resources
Networking resources
Storage resources
Fig. 18. VNF and virtual resource monitoring.
80 ETRI Journal, Vol. 40, No. 1, February 2018
https://doi.org/10.4218/etrij.2017-0236
VNF descriptor. On the basis of this parameter, the VNFMrequests the actual value of a given indicator from theVNFs.
• Virtual resource monitoring: The VIM continuouslymonitors the allocated virtualized resources such as virtualcomputing, virtual storage, and virtual networking.For the preparation of an end-to-end 5G service
demonstration between the EU and KR, we performedinteroperability testing between two CNs as a first step.We are planning to conduct an end-to-end interoperabilitytest including mobile access networks on both sides byNovember 2017, and the results will be described in afuture version of this paper.For CN interoperability testing, we defined the
following two scenarios:
• Scenario 1: There are two users—one connected to theEU vEPC and the other to the KR vEPC. Content isshared between the two users, which is a latency criticalapplication such as shared gaming.
• Scenario 2: In this scenario, a mobile UE on the KR sideis the content provider and is streaming 4K three-dimensional videos to a receiving UE on the EU side. Theaim is to achieve very high data rates across the twovEPCs.For the first phase, we conducted a loose interoperability
test, defined as follows:
• Standard PDN interconnection via IP.
• A dedicated tunnel between the EU and KR test bed,which will provide guaranteed bandwidth and latency.This will ensure that the QoS requirements of the twouse cases are guaranteed.
• A model similar to the DiffServ model to guarantee theQoS. This model must be capable of providing 5Gstandard QoS. The details of such a model need to beworked out further.
• A reachable fixed IP- or DNS-based system, dependingon the actual applications for both use-case scenariosdefined.
• Support for dual stacks. Both IP version 4 and version 6will be supported.
• Dynamic routing protocols (open shortest path first(OSPF), border gateway protocol (BGP)) for advertisingthe PGW IP address to the external network.
• An application server placed strategically between thetwo cores, which will enable the execution of commonapplications such as games with low latencies. Theconnections to and from these servers will also have aguaranteed QoS.The EU–KR interconnectivity is shown in Fig. 19. The
EU–KR dedicated interconnectivity is implemented usingan L2VPN. A dedicated L2VPN connection path is UOulu
, FUNET , NORDUNET , Geant Open , TEIN ,KOREN, ETRI.
4. Dynamic Interoperability Provisioning
The key benefit of an SDN/NFV-enabled mobile corearchitecture is its ability to dynamically adapt requiredresources to the changing context and environment. To takefull advantage of such capabilities, the NFVI on which theEU and KR vEPCs are deployed are (partly) under thecontrol of the same NFVO. Figure 20 illustrates the resultingnetwork architecture, where the common NFVO overseesone or more PoPs, each managed by their own VIM, as wellas the interconnecting wide area network (WAN) managedby its WAN infrastructure manager (WIM).In the static scenario, the NFVO receives an NS request
to deploy the EU and KR vEPCs on their respective PoPsas well as their interconnection via the WAN. As a result,the NFVO will instruct the VIMs to instantiate therequired network function instances as well as the WIM toset up the interconnection.A more advanced and dynamic scenario involves the
dynamic reprovisioning of the interconnected bearers aswell as that of the underlying VNF resources to fulfill thenecessary QoS requirements. This scenario is depicted inFig. 20. In this scenario, the NFVO is used for staticprovisioning of different parts of the mobile coreinteroperability setup and for the dynamic reprovisioning ofthis NS based on monitoring components (see previoussection) as well as other external triggering systems (OSS/BSS or services). These components might, for example,trigger the scale-out of the P-GW-U (1a) or the migration ofthe S-GW (1b) VNFs. Note that the monitoring componentsare not necessarily directly interacting with the NFVO butare usually relying on the interaction of the managementfunctionality of the associated VNF (VNFM) or services. Asa result, the NFVO will (re-)instruct the corresponding VIMsand WIM(s) to instantiate new VNFs (indicated in dark
Interconnection networkCore network Core networkRAN RAN
PCPCBearer Bearer
5GTN PoP Korean PoP
Public internet
jPerf GUI (client)
iPerf server/ 4K video feed
Gangneung city(IoT street)
Daejeon city (ETRI)Oulu (UOULU)
S-GWU
P-GWU
S-GWC
P-GWC
P-GWU
P-GWC
S-GWU
S-GWC
MMEHSSPCRFPCRFHSSMME
FUNET
NORDUNET
Geant
TEIN
KOREN
MPLS L2VPN
Fig. 19. EU-KR interconnectivity.
81Taesang Choi et al.
http://onlinelibrary.wiley.com/journal/10.4218/(ISSN)2233-7326
blue) and rewire the associated network connectivity viathe WIM(s). Future work will refine this process anddetermine the degree of dynamics that will be implementedfor the considered project scenarios and associateddemonstrations.
5. Monitoring
In Sections IV.3 and 4, we described the differentinteroperability scenarios and the necessary steps toinitiate a new connection. We also noted that it is notsufficient to create a connection based on availableresources, but the monitoring of allocated resources isnecessary. By obtaining real information about, forexample, the latency or bandwidth, one can tune the QoSparameters to better align with the application session’srequirements.One of the most trivial metrics to measure is the end-to-
end delay and bandwidth of the newly created path.Depending on the chosen application architecture, one hasto monitor the links between the UEs and the applicationserver or between the two UEs. In case of a client–serverarchitecture, the application server can initiate activemeasurements, or it can passively capture the behavior ofan underlying protocol such as the transmission controlprotocol (TCP) window size and round-trip time (RTT).We have the same possibilities with the point-to-pointarchitecture, except that the UE executes the monitoringapplication. Such an application is the easiest solution, butit would place an unnecessary load on the UE. Moreover,we cannot infer the causes of any quality degradation bymeasuring end-to-end metrics.To overcome these difficulties, we can extend the
vEPCs with monitoring functions or use the existing onesif there are any. One can also install dedicated switchesbetween the serving and PDN gateways in both vEPCs tomonitor the application flows passively. As the traffic isIP-based, we can use OpenFlow switches or even
NetFlow-supporting ones. Moreover, accurate delaymeasurements require clock synchronization between themonitoring nodes. The same synchronization is necessaryfor active measurements, where probes instead of switchesperform the monitoring. These probes have to be aware ofthe properties of the flows in order to inject traffic into thebearers. Therefore, the application server must informthem about the newly created connection.Besides the monitoring of UP traffic, one can also
capture the control traffic (green marker in Fig. 21).Observing the connection setup messages between theMME and the eNB, we can derive the time required forthe initial attachment or a handover. The control messagesbetween the gateways provide information about theduration of the network-initiated (that is, on the request ofan application function) connection setup. Most likely,these setup times have no or little effect on the overall userexperience, but they can inform us about potentialslowdowns. In case of a burst in the number of users, forexample, the application server should refuse some of theconnection requests when it experiences increased setuptimes at one or both of the vEPCs.In addition to the UP and CP traffic as a good indicator of
the performance, we can use also the central processing unit(CPU) usage and memory consumption of the networkelements. Monitoring the CPU usage of the MME, we canforecast system slowdowns, as discussed in the previousparagraph, from the control traffic. A high resource usage atthe gateways in one of the vEPCs indicates failing QoSrequirements, and one can proactively redistribute theresources between the two vEPCs.The monitoring procedures presented so far handle the
functional blocks of the vEPCs as a black box software.They do not require any domain-specific knowledge aboutthe inner workings of the mobile cores, nor do they useany API possible provided by the vendor of the systems(see Fig. 22). Such an API could give us informationabout the number of active connections and the number of
1a. Scale-out trigger 1b. Migrate trigger
2a. Instantiate new P-GW U 2b. Migrate S-GW U
2c. R
e-co
nfig
ure
WA
N c
onne
ctio
n
Dedicated interconnection
Application function
NFVO
MME
S-GW U
S-GW C
P-GW U
P-GW C
HSSPCRF
AP1
S-GW C
P-GW U
P-GW C
PCRF
WAN MGT/CTRL
VIM 2VIM 1
S-GW U
P-GW U
MMEHSS
AP4
AP3
AP2
AP2
eNB
eNB
EU vEPC on PoP 1 KR vEPC on PoP 2
SDN-switch-based user plane
SDN-switch-based user plane
Fig. 20. Dynamic reprovisioning and NW-initiated bearer setup.
MME
S-GW U
S-GW C
P-GW U
P-GW C
HSS PCRF
Bearer
MMEHSSPCRF
Bearer
Korean core networkEuropean core network
AFP-GW
U
P-GW C
S-GW U
S-GW C
Dedicatedinterconnection
Public internet
eNB
Passive monitoring switch or active probe for user planePassive monitoring of the control plane
eNB
Fig. 21. Traffic monitoring locations at the two interworkingsites.
82 ETRI Journal, Vol. 40, No. 1, February 2018
https://doi.org/10.4218/etrij.2017-0236
bearers or even indicate if some of the QoS requirementsare failing. Information from the MME could reveal thephysical location of the UE or at least the cell to which itconnects, which can help us to determine the initiallatency ratio.
V. Performance Evaluation and InteroperabilityTesting Results
During the deployment and interoperability testing, weobserved several important performance measures: thevEPC system performance, the end-to-end networkperformance between the two core systems, and theapplication performance.The HSvEPC provides a total channel throughput of
20 Gbps toward an eNB and can accommodate 100simultaneous UEs. The 5GTN vEPC also supports a totalchannel throughput of 20 Gbps toward an eNB and canaccommodate over 500 simultaneous UEs.The network that connects the two core systems
currently supports up to 1 Gbps, and there are plans toupgrade it to 10 Gbps by November 2017. We performedbandwidth throughput and delay tests on both the EU andKR sides, and the end-to-end context and the resultsobtained over this interconnection link are described asfollows [13].
1. EU CORE Integration and System Testing Results
A. EU’s UOulu Site Testing
Testing was performed with LTE access. A PC with anLTE Universal Serial Bus (USB) stick and with jPerf/iPerftools was used. The simplified test scenario is shown inFig. 23. The LTE band used is Band 7, and the bandwidthis 5 MHz.
• UOulu iPerf Testing with TCPWe performed iPerf testing with TCP traffic. The uplinkbandwidth was about 6.2 Mb/s. The performance was as
much as expected with the 5-MHz bandwidth. The iPerftool does not support downlink measurement when thereis network address translation (NAT) between the endpoints (NAT is carried out at the LTE USB stick).
• UOulu iPerf Testing with the user datagram protocol(UDP)
We also performed iPerf testing with UDP traffic. Theuplink bandwidth was about 11 Mb/s. The jitter was about1.6 ms. The performance was as much as expected withthe 5-MHz bandwidth. The iPerf tool does not supportdownlink measurement when there is NAT between theend points (NAT is carried out at the LTE USB stick).The RTT was measured using a ping test, as shown in
Fig. 24. The average was about 44 ms.
2. KR Core Integration and System Testing Results
A. KR’s ETRI Site Testing
The maximum achievable bandwidth during tested onthe 5G mobile core (5GMC) network was measured byiPerf. In this test scenario, the iPerf client was connectedto the iPerf server running on the PDN GW in the 5GMC.The simplified test scenario is shown in Fig. 25.
• iPerf Testing with TCPWe performed iPerf testing with TCP data streams.Figure 26 shows a screenshot from the iPerf client using
MME
S-GWU
S-GW C
P-GW U
P-GW C
HSS PCRF
Bearer
MMEHSSPCRF
Bearer
Korean core networkEuropean core network
AFP-GW
U
P-GW C
S-GW U
S-GW C
Dedicatedinterconnection
Public internet
eNB
CPU and memory usage monitor Vendor API or extension
eNB
Fig. 22. CPU and memory monitoring and vendor APIexposures.
PC/jPerf clienteNB
5GTN core and
EPC
iPerf server at 5GTN
cloud
Fig. 23. UOulu testing scenario with LTE access.
Fig. 24. UOulu ping test.
iPerf client
5GMC
iPerf server
Fig. 25. Testing without LTE connectivity.
83Taesang Choi et al.
http://onlinelibrary.wiley.com/journal/10.4218/(ISSN)2233-7326
jPerf, a graphical user interface front end for iPerf. Thedownlink bandwidth is about 941 Mb/s.
• iPerf Testing with UDPWe performed iPerf testing with UDP streams. Figure 27shows a screenshot of jPerf on the client side. The uplinkbandwidth is about 812 Mb/s. The downlink bandwidth isabout 910 Mb/s.
• Ping TestThe RTT was measured using a ping test, as shown inFig. 28. It was about 0.31 ms.
3. EU–KR Interconnection Integration and SystemTesting Results
A. EU ? KR Testing
Testing was performed with LTE access on the EU sideand on the ETRI side with LTE. A PC with an LTE USBstick with jPerf/iPerf tools was used. The simplified testscenario is shown in Fig. 29. The LTE band used is Band7, and the bandwidth is 5 MHz.
• EU–KR iPerf Testing with TCPWe performed iPerf testing with LTE access with TCPtraffic. The uplink bandwidth was about 9.31 Mb/s. The
performance was as much as expected with the 5-MHzbandwidth. The iPerf tool does not support downlinkmeasurement when there is NAT between the end points(NAT is carried out at the LTE USB stick). The bandwidthbetween the EU and KR at the CN is sufficient to obtainthis uplink bandwidth with LTE access.
• EU–KR iPerf Testing with UDPWe performed iPerf testing with LTE access with UDPtraffic. The uplink bandwidth was about 11.1 Mb/s. Thejitter was about 1.5 ms. The performance was as much asexpected with the 5-MHz bandwidth. The iPerf tool doesnot support downlink measurement when there is NATbetween the end points (NAT is carried out at the LTEUSB stick). The bandwidth between the EU and KR at theCN is sufficient to obtain this uplink bandwidth with LTEaccess.
• EU–KR Ping TestingThe RTT was measured using a ping test, as shown inFig. 30. The average was about 413 ms.
B. KR ? EU Testing
In order to test the performance of the interconnectionlink between the mobile CNs in KR and Europe, a client PC
Fig. 26. iPerf testing with TCP traffic: iPerf client view.
Fig. 27. iPerf testing with UDP traffic: client view.
Fig. 28. Ping testing with the ETRI server.
PC/jPerfclient
eNB5GTN
core and EPC
ETRI coreDedicated connection
iPerf server at ETRI(Korea)
FUNET → NORDUNET → Geant → TEIN → KOREN
Fig. 29. EU–KR testing scenario with LTE access.
Fig. 30. EU–KR ping test.
84 ETRI Journal, Vol. 40, No. 1, February 2018
https://doi.org/10.4218/etrij.2017-0236
with jPerf/iPerf tools for initiating the test on the Koreanside was connected to the 5GMC, as shown in Fig. 31.
• KR-EU iPerf testing with TCPFigure 32 shows the results for iPerf testing with TCPtraffic. The green dotted line shows the uplink direction,and the blue line shows the downlink direction. Theuplink bandwidth was about 59.8 Mb/s. The downlinkbandwidth was about 37.8 Mb/s. A long RTT (over300 ms) affected the bandwidth with TCP traffic, whichhas delicate flow control and an error control mechanismas a connection-oriented transport protocol.
• KR–EU iPerf Testing with UDPWith iPerf testing with UDP traffic, both the uplink anddownlink bandwidths were measured to be about 812 Mb/s,similarly to the 1,470-b-sized UDP datagram traffic. Thejitter was 0.023 ms on average. The results encouraginglyshow almost full bandwidth considering that the maximumbandwidth between KR and the EU is 1 Gbps.
• KR–EU Ping TestingThe RTT was measured using a ping test. The average wasabout 304 ms.
C. 4K Video Demo
4K video streaming was demonstrated via our L2VPNdedicated connection and public internet access tocompare the quality. Video servers were located in KR.Video streaming used UDP transfer. The downlinkbandwidth could be verified by the computer’sperformance tools available from the “Task Manager.”With dedicated access, 4K video streaming showed very
good performance, and the downlink bandwidth used wasabout 60 Mb/s to 65 Mb/s. The bandwidth was less than10% of the total available bandwidth. However, 4K videostreaming via the public internet exhibited very badquality. The downlink bandwidth via the public internetwas about 8 Mb/s, which was not sufficient to obtain agood end-user experience.
4. Performance Evaluation of the 5G CN SecurityManagement Mechanism
In present mobile networks, IPsec tunneling and securitygateways are widely used to secure backhaulcommunication. We have worked on a novelcommunication architecture based on the HIP to secureboth the control and data channels in SDMNs. We aimedto analyze the added security features as well as theperformance penalty on both the control and datachannels inherent to the proposed simplified architecture(shown in Fig. 33). These performance penalties areconsidered in terms of the throughput, jitter, and latency.The key performance indicators in our performanceanalysis are [10].
• The performance penalty of security on the TCPthroughput
• The performance penalty of security on the UDPthroughput
• The latency introduced
• The performance penalty of security on the jitter
A. Performance Analysis of the Control Channel
In the first set of experiments, we analyze theperformance penalty of security on the SDMN controlchannel due to the proposed architecture.iPerf client
5GMC(Korea)
5GTN core
iPerf server
Dedicated connection
KOREN → TEIN → Geant → NORDUNET → FUNET
Fig. 31. EU–KR testing scenario.
Fig. 32. EU–KR iPerf testing with TCP traffic in the uplink.
POX controller
SecGW
Data channelControl channel
AttackerAttacker
LSALSA
Open vSwitch
Open vSwitch
Host 1 Host 2 Host 3 Host 4
Fig. 33. Testbed for the IPsec tunneling architecture for SDMNcommunication channels.
85Taesang Choi et al.
http://onlinelibrary.wiley.com/journal/10.4218/(ISSN)2233-7326
• Connection Establishment DelayIn the first experiment, we measure the connectionestablishment delay between Open vSwitch 1 and thePOX SDN controller under different scenarios. Here, weattempt to send a ping request from Host 1 to Host 2 andmeasure the connection establishment delay. Theexperimental results in Fig. 34 reveal that the proposedsecure architecture significantly increases (136%) thetunnel establishment delay. HIP tunnel establishmentbetween the local security authority (LSA) and the securegateway (SecGW) adds an extra delay to tunnelestablishment. However, the impact of this delay can beminimized by maintaining the established HIP tunnels fora long period. It is possible to maintain established HIPtunnels for long periods (that is, 15 min).
• Flow Table Update DelayIn the second experiment, we measure the delay to updatethe flow tables for a new packet flow during steady-stateoperation. In steady-state operation, the HIP tunnelsbetween the LSAs and the SecGW are already establishedand operational. Here, we ping from Host 1 to Host 2 andmeasure the RTT. The experimental results in Fig. 35reveal that the performance penalty of the proposed securearchitecture is less significant in steady-state operation.The extra IPsec encryption increases the flow update delayby only 2%. However, this delay can be further minimizedby using IPsec accelerators. IPsec acceleration is possibleby using external accelerators and/or using new AdvancedEncryption Standard instruction sets for processors.
B. Performance Analysis of the Data Channel
In the second set of experiments, we measure the TCPand UDP throughput performance of the data channel indifferent scenarios.
• Impact on the TCP ThroughputIn third experiment, we establish a TCP connectionbetween Host 1 and Host 3 to measure the TCPthroughput performance of data channel by using the iPerftool. The experimental results in Fig. 36 reveal thatthe proposed secure architecture decreases the TCPthroughput by only 2.3% compared to that of thenonsecure data channel. The extra layer of encryptiondecreases the TCP throughput.
• Impact on the UDP ThroughputIn fourth experiment, we establish a UDP connectionbetween Host 1 and Host 3 to measure the UDPthroughput performance of the data channel. Theexperimental results in Fig. 37 reveal that the proposedsecure architecture decreases the UDP throughput by only2.2% compared to that of the nonsecure data channel. Theextra layer of encryption decreases the UDP throughput.Moreover, the performance penalty of security on thethroughput is around 2% for both the UDP and TCPsessions compared with that of the nonsecure scenario.Thus, we can conclude that the performance penalty of
Number of attempts10 20 30 40 50 60 70 80 90 100
020
40
6080
120
140
160180
100
200
Rou
nd tr
ip ti
me
(ms)
OpenFlow with TLSv1Proposed control channelOpenFlow with TLSv1 (average with CI)Proposed control channel (average with CI)
Fig. 34. Connection establishment delay.
Number of attempts20 40 60 80 100
05
10
1520
30
35
4045
25
50
Rou
nd tr
ip ti
me
(ms)
OpenFlow with TLSv1Proposed control channelOpenFlow with TLSv1 (average with CI)Proposed control channel (average with CI)
0
Fig. 35. Flow table update delay.
Time (s)20 40 70 90 100
60
65
70
75
80
90
95
85
100
TCP
thro
ughp
ut (M
bps)
Without secure channelProposed secure channelWithout secure channel (average with CI)Proposed secure channel (average with CI)
0 10 30 60 8050
Fig. 36. Performance penalty on the TCP throughput.
86 ETRI Journal, Vol. 40, No. 1, February 2018
https://doi.org/10.4218/etrij.2017-0236
security on the throughput is independent of the transportlayer protocol.
• Impact on the JitterIn fifth experiment, the jitter performance of a UDPsession between Host 1 and Host 3 is measured by usingthe iPerf tool. The experimental results in Fig. 38 revealthat the performance penalty of the secured architecture is41% relative to the nonsecure data channel. However, thejitter is still well below 500 ls (voice over IP (VoIP)requires a jitter below 4 ms), and the impact of jitter forreal-time applications such as VoIP and video streaming isless significant in a short-range network.
VI. Conclusion and Future Work
In this paper, we proposed an SDN/NFV-enrichedintelligent 5G CN and its agile MANO system to address5G KPIs. As details of the proposed system, we presentedthe architecture of the virtualized CN capabilities andits agile management. Furthermore, we shared ourimplementation, its deployment, and our interoperability
testing experiences with PoC use cases. As described, weare currently in the second phase of conformance andinteroperability testing of the proposed systemfunctionality. Our future work includes a performanceevaluation of the proposed solution in an end-to-end scope(UE-5G access-5G core–data center with applicationservers) in a PoC testing environment, which is thePyeongChang Olympic venue.
Acknowledgements
This work was supported by a grant from the Institutefor Information & Communications TechnologyPromotion (IITP) funded by the Korean government(MSIT) (No. B0115-16-0001, 5G Communication with aHeterogeneous, Agile Mobile Network in thePyeongChang Winter Olympic Competition) and theEuropean Union H2020 5GPPP under grant number723247.
References
[1] H. Shokri, C. Fischione, G. Fodor, P. Popovski, and M.
Zorzi, “Millimeter Wave Cellular Networks: A MAC LayerPerspective,” IEEE Trans. Commun., vol. 63, no. 10, Oct.
2015, pp. 3437–3458.[2] V. Desai, L. Krzymien, P. Sartori, W. Xiao, Z. Soong, and
A. Alkhateeb, “Initial Beamforming for mmWaveCommunications,” Asilomar Conf. Signals, Syst. Comput.,
Pacific Grove, CA, USA, Nov. 2–5, 2014, pp. 1926–1930.[3] E.C. Strinati and H.K. Chung, 5G CHAMPION, 2017. Accessed
Dec. 31, 2017. http://www.5g-champion.eu/
[4] G. Cross, Open Networking Forum SDN Standards, 2017.Accessed Nov. 15, 2017. https://www.opennetworking.
org/software-defined-standards/overview/[5] ITU-T SG13 TSB, ITU-T SG13 WP1 Q.21, 2017, Accessed
Nov 15, 2017. http://www.itu.int/en/ITU-T/studygroups/2017-2020/13/Pages/default.asp
[6] Linux Foundation Projects, ODL Open Source Project,2017, Accessed Nov. 15, 2017. https://www.opendaylight.
org/what-we-do/odl-platform-overview[7] G. Cross, ONOS Open Source Project, 2017, Accessed
Nov. 15, 2017. https://www.opennetworking.org/platforms/
onos/[8] J. Quittek et al., “Network Functions Virtualisation (NFV)
– Management and Orchestration V.1.1.1,” ETSI NFV ISG,Dec. 2014.
[9] R. Schuster et al., “An Introduction to the New ETSIIndustry Specification Group (ISG) for Mobile Edge
Computing (MEC),” ETSI MEC ISG, Oct. 2015.
Time (s) 20 40 70 90 100
60
65
70
75
80
90
95
85
100
UD
P th
roug
hput
(Mbp
s)
Without secure channel Proposed secure channel Without secure channel (average with CI)Proposed secure channel (average with CI)
0 10 30 60 80 50
Fig. 37. Performance penalty on the UDP throughput.
Number of attempts20 40 70 90 100
0
0.1
0.2
0.3
0.5
0.4
Jitte
r (s)
Without secure channelProposed secure channelWithout secure channel (average with CI)Proposed secure channel (average with CI)
0 10 30 60 8050
Fig. 38. Performance penalty on the jitter.
87Taesang Choi et al.
http://onlinelibrary.wiley.com/journal/10.4218/(ISSN)2233-7326
[10] J. Moilanen et al., “Operator Grade NFV-Based and SDN-Enriched EPC Environment at 5GTN (D4.1),” 5GCHAMPION Project, May 2017.
[11] M. Liyanage, A. Gurtov, and M. Ylianttila, “Software
Defined Mobile Networks (SDMN): Beyond LTE Network
Architecture,” Chichester, UK: John Wiley & Sons,2015.
[12] R. Banerjee et al., “5G CHAMPION Architecture, API-andInterface Document (D2.1),” 5GCHAMPION Project, Oct.
2016.[13] J. Moilanen et al., “VNF/SDN/EPC: Integration and System
Testing (D6.2),” 5GCHAMPION Project, June 2017.
Taesang Choi received his MS and PhDdegrees in computer science and
telecommunications from the University ofKansas City, MO, USA in 1988 and 1995,respectively. He joined the ETRI, Daejeon,
Rep. of Korea in 1996 and is currentlyworking as principal research staff. He has
been actively involved in the research and development of trafficengineering, traffic measurement and analysis, SDN/NFV
management, and 5G network slice management. He has alsoactively contributed to various SDOs and open-source activities
such as IETF, ITU-T, ONF, ONOS, and others. He is currentlyacting as an ITU-T SG13 Question 6 Rapporteur and InternationalIT Standardization Expert representing the Rep. of Korea.
TaeYeon Kim received his PhD degree in
computer science from Chungbuk NationalUniversity, Chungju, Rep. of Korea in2007. He also received BS and MS
degrees from Chung-Ang University,Seoul, Rep. of Korea in 1990 and 1992,
respectively. He joined the ETRI, Daejeon,Rep. of Korea in 1992. His current research includes network
and computing convergence platforms and SDN and NFVtechnologies for future networks.
Wouter Tavernier received his BS andMS degrees in computer science in 2002
from Ghent University, Belgium. Hejoined the Internet-Based Communications
Networks group (which became part ofIDLab in October 2016) of Ghent
University in 2006 as researcher on CarrierEthernet. In 2012, he obtained a PhD degree from the sameuniversity on reliable routing and switching. Currently he is
employed as a professor at Ghent University. His currentresearch interests focus on the performance aspects of software-
defined networks and network function virtualization. This workis performed in the context of European projects such as H2020
5G-CHAMPION, SONATA-NFV, and 5G TANGO. Thisresearch has been published in more than 50 scientific
publications.
Aki Korvala received his BS degree from
the Technical Institute of Oulu,Department of Electrical Engineering, in
1997. He has worked at Nokia for18 years in various R&D positions withinmobile phones and network business lines.
Currently, he is working in the 5G area asa program manager. This work is performed in the context of
European projects such as H2020 5G-CHAMPION.
Jussi Pajunp€a€a has worked at Nokia
Networks, Espoo, Finland for 19 years invarious R&D positions in software and
systems engineering in the core networkdomain. Currently, he is working with the
Telco Cloud and virtual network functionarchitecture as a chief architect and R&D
manager and contributing to the 5G test network activities inOulu.
88 ETRI Journal, Vol. 40, No. 1, February 2018