BNV: Enabling Scalable Network Experimentation throughBare-metal Network Virtualization

Pravein Govindan Kannan Ahmad Soltani Mun Choon Chan Ee-Chien ChangSchool of Computing, National University of Singapore

Abstract

New paradigms and architectures, such as Software De-fined Networking (SDN), have added an unprecedentedincrease in the rate of research and development con-ducted in the field of computer networks. With this in-crease, there is a rising need for platforms that can en-able researchers and operators to experiment with vari-ous scenarios involving performance testing, SDN, secu-rity research, topology designs, etc. However, the avail-able emulators fail to address fundamental needs of theexperiments requiring diverse and scalable set of topolo-gies.

In this work, we propose a novel approach to em-bed arbitrary topologies on a substrate network of pro-grammable ToR switches using our network virtualiza-tion technique, called Bare-metal Network Virtualization(BNV). BNV is entirely software configurable and hasbeen implemented on open source software and unmod-ified OpenFlow-enabled switches. The system has beendeployed and currently running in a production testbedin National Cybersecurity Laboratory (NCL) for a year.Our evaluations show that BNV can support various data-center topologies with less number of switches which canfacilitate building a high fidelity, repeatable and isolatedexperimentation platform for data-center, SDN and secu-rity research in computer networks.

1 Introduction

Network experimentation is an integral part of Net-work and Systems research. With new paradigms likeSoftware Defined Networking, networks are becomingmore programmable and customizable according to ap-plication requirements. Data-center architects and net-work operators, constantly work on optimizing networktopologies and algorithms under various scenarios and tomaintain the production network in a steady state [18].

Fidelity and repeatability are two of the essential re-quirements for any experimentation platform. Addition-

ally, to mimic real-life production network scenarios,flexibility and scalability in emulation of topologies areessential at both the control-plane and data-plane. Un-fortunately, none of the available tools can satisfy bothfidelity and flexibility in the data-plane.

In most cases, experiments rely on network emulationtools like Mininet [13], NS, etc. These tools allow exper-imenters to create topologies very quickly on their hostmachines. However, since they run on the CPU of themachines they are hosted on, the network performancedepends on the deployment environment. Hence, it ishard to achieve fidelity and repeatability in the experi-ments. Additionally, these tools cannot scale to emulateproduction networks [18, 26].

Network testbeds like CloudLab [21], DeterLab [19]provide an environment for network and security experi-mentation. However, since the network is considered asan infrastructure [1] and managed using VLANs, exper-imenters cannot program their network to perform SDN-based experiments.

In this work, we propose BNV, a network virtualiza-tion technique to bridge the gap between production net-works and emulation tools by providing the followingfeatures:

Flexible network topology. BNV provides each ten-ant with their desired topology in a bare-metal fashion(virtualization is done at the switch hardware) while theunderlying topology is entirely different. Hence, multi-ple arbitrary topologies can be provisioned over the hard-ware switches using BNV as a network hypervisor. Ourtechnique makes emulating large topologies more acces-sible and cheaper compared to the actual cost of buildingone. The experimenters can design custom algorithms orprotocols to program their allocated network.

Fidelity. BNV provides consistent performance athardware line-rate. Such fidelity requirement is essen-tial for network experimentation involving applicationswhereby the network configurations need fine-tuning foroptimal performance and regulatory verification.

Switch1 Switch2 Switch3

BNV

OpenFlow

Flow Translation

Topology Abstraction

Tenant 1 Tenant 2 Tenant 3

OpenDaylightController

ONOS Controller

POX Controller

BNV Mapper

Virtual Topology

Tenant Mapping

FatTree, JellyFish, HyperCube,

HyperX, Xpander, Arbitrary

topologies…etc

Tenant n

…

Any SDN Controller

Figure 1: BNV System Overview

Isolation and performance guarantee for multipletenants. BNV allows different tenants or experimenterswith different application requirements and topologies toshare the same network infrastructure while guaranteeingcomplete isolation and repeatable performance by imple-menting buffer and queue management.

To implement BNV, we address the following chal-lenges:1. What is the right substrate network for BNV with

maximum flexibility and minimum wiring changes?2. What is the algorithm needed to find the most efficient

network embedding?3. How to implement BNV using existing tools?

To address (1), we propose a novel topology wiringusing loop-back links in switches, to emulate multiplearbitrarily connected virtual switches. To address (2),we formulate the problem as an integer linear program-ming (ILP) to maximize the scalability of experimen-tal support and fidelity. For (3), we implement BNVover an existing network hypervisor OpenVirteX [7] andalso integrated the platform with a bare-metal provision-ing system DeterLab[19]. We have deployed BNV ina Cyber-Security testbed called National CybersecurityLab (NCL) [4] for over a year for experimenters to runSDN and other experiments.

Our evaluations have shown that BNV can supporthigh fidelity virtualization of network topologies, andcan virtualize networks consisting of more than 100 net-work devices using only five top-of-rack (ToR) switches,while supporting line-rate for high-performance testingof production workloads.

2 BNV ArchitectureThe main component of BNV which differentiates itfrom other network hypervisors is the one-to-many map-ping which virtually slices the switches in the underly-ing physical infrastructure into multiple arbitrarily con-nected virtual switches (collection of physical switchports), without losing the queuing behavior and provid-ing fidelity in mapping. We use the terms physical orsubstrate topology interchangeably.

Figure 1 shows the architecture of BNV. BNV con-nects to the SDN switches via OpenFlow. Users submit

CoreN/W

ToR SDN Switch 1 ToR SDN Switch N

…

Rack 1 Rack N

CoreN/W

L2 Switch 1 L2 Switch N

…

Rack 1

ToR SDN Switch 1 ToR SDN Switch N

Rack N

a. Static Loopback b. Software-ConfigurableLoopback

Figure 2: Loopback Configuration in Network

their virtual topologies to the BNV Mapper which em-beds the resources and finally creates the network for thetenant with the reserved physical resources.

BNV can create multiple tenants and supports use ofany SDN controllers to control slice of the network in-dependently, without interfering with the other tenantsco-located on the same physical switch. We explain howBNV performs topology abstraction and flow translationin the following two sub-sections. We present the ILPbased embedding of BNV mapper module in section 3.

2.1 Topology Abstraction

In a generic topology, switches are connected to hostsusing relatively low-bandwidth links, and other switches(or core-network) using high-bandwidth links. We in-troduce additional link type to perform topology ab-straction.The switches, apart from connecting to hostsor other switches, have special links which are loop-back links, i.e., a link between two ports on the sameswitch. If a switch has X ports, H ports are connectedto the hosts, C ports are connected to other switches,and L ports are loop-ports which form L

2 loopback links.The number of loopback links can be pre-configured asshown in Figure 2.a or dynamically configured by plac-ing an L2-Switch or circuit switch between the host andthe SDN Switch (ToR) as shown in Figure 2.b. With thisconfiguration, we can dynamically configure the num-ber of ports allocated to host and loopback at run-timethrough VLAN or circuit switch reconfiguration.

With the loopback links any two connected virtualswitches can map to a single physical switch using theloopback link as their inter-switch link. Consider a sub-strate network as in Figure 3. The hosts are connectedto the switches using 1Gbps links, and the inter-switchlink has a bandwidth of 10Gbps. The virtual topology tobe implemented is a triangular network (3 switches and3 hosts) with 1Gbps links. The virtual topology mappingto the substrate is shown in Figure 3 where the links vL1and vL2 are mapped to the CLink1, which is a backbonelink in substrate network. vL3 in the virtual topology ismapped to the substrate network using loopLink1. Loop-back links provide flexibility in abstraction while main-tain the fidelity of the virtual links.

5 51 2 3 4 1 2 3 4

H1

H2

H5

H6

pSwitch1 pSwitch2CLink1

H1

H2

12

3 2 3 1vL3 -> loopLink1

vL1 -> CLink1 vL2 -> CLink1

vS1

vS2

vS3

H

H

H

H5

2 31

Physical Topology

Virtual Topology Mapping

vS1

vS3

vS2

loopLink1 loopLink2

vL3

vL2vL1

1G1G

1G

10G

1G

Figure 3: Mapping of Virtual Topology with Loopbacks

Table 1: Flow Translation using loopback linkEntity FlowMod added to Switch

vS1pSwitch1 : {in port=1,ip dest:y.y.y.y,

actions=meter:1,lTag:2,output:3}

vS2pSwitch1 : {in port=4,ip dest:y.y.y.y,lTag:2,

actions=RemoveLTag,output:2}

2.2 Flow TranslationBNV performs flow-table translations to provide flows-pace and dataplane isolation for virtualizing multiple-links to support arbitrary topologies. Packet taggingand metering are used to achieve topology virtualization.We maintain the virtual link information encoded in thepacket header. We call it lTag for simplicity.

Refering to the virtual topology in Figure 3 with threelinks vL1, vL2 and vL3, the corresponding lTags being1,2 and 3, We have two scenarios for mapping of the vir-tual links:

Case 1: Two connected virtual switches maps to dif-ferent physical switches with a (in)direct link. (Example: vL1) In that case, any packet that travels vL1 is taggedwith lTag 1 on the outgoing port of CLink1. Similarly,Flows are translated to attach the lTag in the match basedon which virtual switch and virtual port the flowmod ismeant for. Also, a drop-meter m is created to rate-limitthe flows to the max-rate of 1G as in the virtual topol-ogy. For instance, assume a tenant adds a flow entry{in port : 1, ip dest : x.x.x.x,actions= out put : 2} to vS1for routing a packet via link vL1. The translated flowmodis {in port : 1, ip dest : x.x.x.x,actions=meter : 1, lTag :1,out put : 2}.

Case 2: Two connected virtual switches maps to thesame physical switches (vL3). Consider a FlowMod toenable ip-destination based forwarding from H1 to H2are below:vS1 : {in port : 1, ip dest : y.y.y.y,actions = out put : 3}vS2 : {in port : 2, ip dest : y.y.y.y,actions = out put : 1}Loopback link: LoopLink1 emulates vL3. The Flow-Mods that were pushed earlier can be implemented byperforming the translation of the output ports to physi-cal ports, and adding the appropriate lTag in the physicaltopology as in Table 1. Typically Loopback links can bededicated to a single tenant and they can naturally emu-late inter-switch links.

Metering: We employ two kinds of data-plane isolationusing meters:1. Shared links (cLink1): When multiple virtual links

share a single physical link, they share the samequeues. We mitigate the interference by restrictinga single virtual link from exhausting the port buffersby partitioning the maximum burst size of a physicallink (measured using techniques in [15]) to multiplevirtual links.

2. Shared Switches (pSwitch1): When a single phys-ical switch is sliced into multiple virtual switches,it’s important to slice the buffer such that the virtualswitches do not run out of buffer. In this case, weapply a meter to the flows from all ports of a virtualswitch. We explain the implications and observationsof metering in Section 4.3.

3 BNV MapperIn this section, we present the formulation of the networkembedder (BNV Mapper) as an ILP to maximize the fi-delity of the embedding of the virtual topology and tominimize the overheads. The (user) input to the BNVMapper is the virtual network topology that consists of aset of : 1) hosts V , 2) (virtual) switches S and 3) (virtual)links L. Each link is bi-directional and has a requiredbandwidth b. A link can be associated with either (1) twoswitches (corelink Lc) or (2) a switch and a host (hostlinkLh).

The user input is next converted into a form where theswitches are broken down into (1) Core-link. A link be-tween two switches is represented as a link between twoswitch-ports. (2) Host-link. A link between a switch-portand a host. In this way, each switch s can be representedas a set of switch-links or host-links. For a link that is as-sociated with a switch, the required TCAM size t(s) canbe specified.

The physical topology consists of a set of: 1) phys-ical server machines H, 2) physical switches R and 3)physical links Q. Similar to the virtual topology, Q canbe categorized into corelinks Qc and hostlinks Qh. Theavailable bandwidth of a link i is represented as Bi. Notethat, Qc comprises of both loopback links and backbonelinks (links between the switches).

We define a binary decision variable xiv, which is setto one if a virtual corelink i is mapped to the physicalcorelink v, and zero otherwise. Similarly, y jw, whichindicates if a virtual hostlink j is mapped to a physicalhostlink w. Mapping of hostlink automatically impliesmapping of virtual host to physical server. The objectivefunction is given below by Equation (1).

min : ∑i∈Lcv∈Y

xivbi + ∑m∈S{p,q}∈R

Mmpq (1)

CLink1

Core Network

1 2 3

H1

H2

H3

pSwitch11 2 3

H4

H5

H6

pSwitch21 2 3

H7

H8

H9

pSwitch3

CLink2 CLink3

vSwitch1H1

H2

H3

H4

H8

Physical Topology

Virtual Topology

To B/W

H4 1G

H7+H8 2GTo B/W

* 1G

To B/W

H1+H2+H3 2G

H4 1G

H7

Figure 4: Allocation for big-switch abstraction

The objective function has two terms. The first termrepresents the amount of resources to support mappingof corelinks (xivbi) and the second term represents theresources needed to map a single virtual switch over mul-tiple physical switches (Mm

pq). The overall goal is to min-imize the usage of substrate backbone (core) links, sincethese links are (generally) under-provisioned relative tothe hostlinks. This indirectly maximizes the usage ofloopback links which provide higher fidelity in emula-tion. The constraints are:

∑v∈Qc

xiv = 1,∀i ∈ Lc (2)

∑w∈Qh

y jw = 1,∀ j ∈ Lh (3)

∑v∈Qh

y jvb j ≤ Bv,∀ j ∈ Lh (4)

∑v∈Qh

y jvc j ≤Cv,∀ j ∈ Lh (5)

∑i∈Lc

∑v∈Qr

c

xivt(i)+ ∑j∈Lh

∑w∈Qr

h

y jwt( j)≤ Nr,∀r ∈ R (6)

zmn = ∑v∈Qc

xivbi + ∑w∈Qh

y jwb j,

∀i ∈ Lmc , j ∈ Lm

h ,m ∈ S,n ∈ R(7)

Mmpq = min(zmp.zmq),∀m ∈ S,{p,q} ∈ R (8)

∑v∈Qc

xivbi +Mmpvqv ≤ Bv,

∀i ∈ Lc,∀m ∈ S,v ∈ Y,{pv,qv} ∈ R(9)

Constraint (2) and (3) mandates each virtual corelinkto be mapped to only one substrate corelink and similarlyfor hostlinks. Constraint (4) ensures that each physicalhostlink is provisioned within it’s capacity. Constraint(5) ensures that the physical host is not allocated beyondit’s core capacity. The notation uses hostlink instead ofhost, however, in the model hostlink is synonymous tohost. Also, Constraint (4) and (5) are needed only forVM-based provisioning like OpenStack. They are not

BNV Hypervisor

1 x 10G 1 x 10G SDN SwitchHP3810

24 x 1G 48 X 1G

Core Switch   HP7904)

X 4

12 Loopback      links

Control SwitchHP5900

24 Blades

Figure 5: BNV testbed setup environment

needed for bare-metal provisioning methods since eachvirtual host is allocated an entire server blade. Constraint(6) makes sure we do not exceed the TCAM capacitybounds of the switch. TCAM specified for every virtualswitch is split equally among it’s corelinks and hostlinksfor simplicity in allocation.

Big-Switch Abstraction. Constraint (7) defines avariable zm,n which depicts the total bandwidth of vir-tual links of virtual switch m that are mapped to physi-cal switch n. Constraint (8) creates a R x R matrix foreach virtual switch indicating the amount of intra-switchbandwidth between two physical switches mapped to asingle virtual switch. It allocates the minimum link band-width generated by each physical switch for a given pairof physical switches mapping to same big virtual switch.Figure 4 illustrates an example of the embedding of avirtual switch with 5 hosts onto three substrate switcheseach mapping a different number of hosts. In Figure 4,each virtual switch is associated with a table containingthe bandwdith to be limited for traffic to a particular setof hosts belonging to a physical switch. Finally, Con-straint (9) makes sure the physical corelinks are provi-sioned within their capacity accounting both inter-switchlinks and intra-switch link utilization.

4 Testbed Implementation and EvaluationWe have implemented BNV over OpenVirtex [7] usingOpenFlow 1.3. We have also integrated BNV with De-terLab [19] which provides bare-metal provisioning. Weuse Gurobi [3] as the ILP solver. The users submit theirtopology as a NS file1. The SDN switches are specifiedby defining the switch-type as ”ofswitch”.

BNV is deployed and currently functional on the pro-duction testbed of the National Cybersecurity Lab2 fora year to provision SDN-based experiments with arbi-trary topologies. The experimenters can completely usethe functionalities supported by OpenFlow 1.3 (Meters,groups, etc.), and can develop and use custom networkapplications (BGP, congestion control, etc.). The net-work testbed layout is shown in Figure 5. The testbed

1Refer https://ncl.sg/bnvirtusage.pdf2https://ncl.sg

Dynamic Loopback Wiring using L2 intermediate Switch

Static loopback wiring 24-ports of SDN Switch

Figure 6: Loopbacks in Switches

a. Star Topology. b. Emulated Star Topology

c. Clos Topology d. Emulated Clos Topology

Figure 7: Topologies used for application fidelity

has four HP3800 SDN Switches, each connecting to acluster of 24 Lenovo X3550 servers. The SDN switchesare connected using a core-switch which is used only forL2 connectivity. The server blades are also connectedto a control-switch, which is used for out-of-band man-agement (IPMI, PXE booting). Each SDN switch has 12loopback links and 1 10G uplink to the core-switch. Eachloopback link is formed by connecting two ports using ashort cable. Figure 6 shows the testbed with the staticloopback wiring of 12 links per switch. We additionallyperform software-configurable loopback for one clusterin our staging platform. In the following sub-sections weevaluate BNV’s fidelity, isolation and embedding capa-bilities.

4.1 Performance FidelityIn this section, we evaluate the fidelity of the virtualtopology created by BNV by making use of one-to-manyabstraction and many-to-one abstraction. The evaluationconsiders variations of two topologies: (1) a star topol-ogy with 1 switch and 16 hosts (Figure 7(a) and Figure7(b)) and (2) a Clos topology with 4 switches and 16hosts (Figure 7(c) and Figure 7(d)). We change the phys-ical wiring of the testbed to implement the physical clostopology.

We run an Apache Spark application (wordcount on a50GB file) on all 4 topologies and we plot the CDFs ofshuffle read times on the four topologies in Figure 8.

We observe that the CDFs of the physical and virtualnetworks are very similar. The use of loopback links(Figure 7(d) vs Figure 7(c)) has the same application be-haviour compared to actual topology. The use of longer

0

0.2

0.4

0.6

0.8

1

0 20 40 60 80 100

CD

F

Shuffle Read Completion Time

Star (Single Switch)Star (Multi Switch)

Clos (Single Switch)Clos (Multi Switch)

Figure 8: Performance comparison on emulation

Figure 9: Mapping of FatTree4 on testbed

physical hops and multiple physical switch to supportone virtual switch (Figure 7(b) vs Figure 7(a)) do notaffect the performance.

4.2 Topology FlexibilityRecent works on topology convertibility [28] demon-strate tangible gains in network performance by chang-ing the network topology dynamically. BNV can lever-age its inherent ability to map arbitrary network topologyto allow researchers to experiment with different topolo-gies quickly. This makes it possible to perform fine-grainoptimization of topology for a work-load and ability toemulate traffic patterns reliably in a topology.

BNV can virtualize the network to support varioustopologies like FatTree [6], Clos, JellyFish[25], Hyper-X etc in a matter of a few seconds. We illustrate how aFatTree is mapped on to substrate topology in Figure 9.

We create four experiments with various topologies:Binary Tree (16 hosts, 15 switches), Star (16 hosts, 1switch), FatTree with degree 4 (16 hosts, 20 switches),and JellyFish [25](16 hosts, 20 switches) and perform awordcount application for a 50GB file in Apache Spark.We use custom partitioning in order to increase the inter-rack traffic. We perform 10 runs and plot the averageshuffle read time for the four topologies in Figure 10.

We observe that tree topology has the longest shuf-fle read time due to its very limited bisection bandwidth.The star finishes fastest since it has full bisection band-width. FatTree and JellyFish perform somewhere in be-tween as expected. ECMP [14] was used to split the traf-fic equally based on link utilization calculated at the con-troller in the case of FatTree and JellyFish.

0

20

40

60

80

100

120

140

Tree Fat Tree JellyFish Star

Avg

Shu

ffle

Rea

d tim

e (s

ec)

Figure 10: Apache Spark performance over variousTopologies generated using BNV

0

0.2

0.4

0.6

0.8

1

0 10 20 30 40 50

CD

F

Shuffle Read Time(s)

FatTreeJellyfish

Figure 11: Comparison of FatTree and JellyFish for highintra-pod locality traffic

Now, taking a closer look at only two topologies: Fat-Tree and JellyFish. FatTree and Jellyfish observe reallyclose shuffle time. On an average, the shuffle time inJellyFish is about 7-8% lower. This is due to high inter-rack traffic. Then we vary the data placement in Sparkto increase the intra-pod locality. We plot the CDF ofthe shuffle read times in Figure 11 for FatTree and Jel-lyfish. Interestingly, we observe an increased shuffletime (∼7.5%) in case of JellyFish compared to FatTree.To summarize our observation, JellyFish performs muchbetter than FatTree when there are lot of inter-rack net-work traffic and FatTree performs better than JellyFishwhen there are lot of intra-rack exchanges. This perfor-mance behaviour observed is consistent with the obser-vations made by recent work on convertible topologies[28]. Thus, BNV can achieve fidelity and flexibility interms of its topology implementation.

4.3 IsolationWe spawn three experiments on BNV with the follow-ing topologies: 1) Expt-1: FatTree4 (16 hosts and 20switches) running wordcount of 1GB file using Sparkcontinuously with heavy inter-rack traffic, 2) Expt-2: Jel-lyFish (random topology) topology with 16 hosts and20 switches running iperf among all pairs and 3) Expt-3: Random topology using up the rest of the availableswitch-ports(8) and hosts(8) running ping between thepair of nodes with the longest hop-count. Initially, allexperiments are idle with applications not running. Theexperimentation scenario is as follows :1) Between 0-60 minutes, we run the application on eachof the experiments for 20 minutes without interference.

Figure 12: Concurrent experiments to check isolation

0

0.2

0.4

0.6

0.8

1

0 30 60 90 120 150 180

Thro

ughp

ut p

erce

ntag

e

Time (in sec)

Figure 13: Observation of TCP traffic completely unaf-fected by UDP bursts in the same switch over other ports.

2) Between 60-80 minutes (shaded region), all three ap-plications are run in parallel.3) We repeat step (1) between 80-140 minutes.

We present the findings in Figure 12, where we plotdifferent performance metrics such as average shuf-fle time for expt-1(top), aggregate throughput for expt-2(middle) and ping latency for expt-3(bottom). Theshaded region (comprising 20 minutes) represents theportion of time the experiments were run in parallel, andthe non-shaded regions depict the performance for indi-vidual runs. Observe that the applications see no per-ceivable difference in performance when the other ex-periments which share the same switches and backbonelinks are run.

Buffer partition: When two tenants share a physicallink (typically backbone links), a burst of traffic from asingle tenant could fill up the buffers and TX Queue, thusimpacting the other tenants. Fortunately, we can achievean approximate buffer partition by allocating the max-imum burst for a particular virtual link or set of ports(using OpenFlow meters) during provisioning by con-sidering the switch-port buffer-size to be one of the con-straints in the BNV mapper. In the following, we performtwo experiments to measure the impact of bursty traffic.

Bursty transmission on a specific output port doesnot affect the traffic on other switch-ports: We per-form an experiment with two tenants allocated differentslices of the same switch: one tenant performing UDPbursts (sending at the maximum rate) of 11 hosts to 1

0

0.2

0.4

0.6

0.8

1

0 30 60 90 120 150 180 210 240

Thro

ughp

ut p

erce

ntag

e

Time (in sec)

MeteringEnabled

MeteringDisabled

Figure 14: Observation of TCP traffic completely unaf-fected by UDP bursts in the same egress (shared)ports.

host, and the other generating a TCP traffic at line-rateon hosts running over two other ports. In Figure 13, theshaded region is the period where the UDP bursts occur.Clearly, the UDP traffic has no impact on the TCP traf-fic due to the buffer isolation provided. Although switcharchitectures [5, 11] perform buffer isolation inherently,in order to be generic and not rely on specific switch ar-chitectures, we allocate a burst-rate for all the ports be-longing to each virtual switch, and apply the same meterto all the flows belonging to the virtual switch’s ports.In this way, we guarantee that the amount of buffer usedby a single (or a set of) flow(s) is bounded. Although,this implies that the switch-buffer may be under-utilizedin certain circumstances, we reckon that this measure isnecessary to guarantee isolation, fidelity and repeatabil-ity of experiments.

Burst transmission on shared ports (and links)minimally affects the traffic: We perform an experi-ment, where the backbone link is shared by two virtuallinks (either same or different tenants). vlink1 is allo-cated 1/10th of the bandwidth of the physical link, andthe rest of the bandwidth is allocated to virtual-link2. Wetransmit TCP flow on virtual-link1 and 10 different UDPflows on vlink2 to saturate the physical link. We observethe impact of UDP traffic-bursts of vlink2 on vlink1. Theresults are shown in Figure 14. We can observe that theTCP throughput is not affected even during the presenceof burst transmission. This is primarily because we limitthe maximum burst per port. At the 180th second, weturn off metering-based isolation. We notice that the TCPflow on vlink1 is completely starved due to bursty UDPflows on vlink2 in the absence of burst/ rate limiting.

4.4 Network Mapping SimulationIn this section, we evaluate through simulation the net-work embedding efficiency of BNV by considering a net-work consisting of 5 switches consisting of 48 down-linkports (1G each) of which, 24 ports to be connected tophysical servers, and the 24 remaining ports are usedform loopback links (12 loopback links per switch).

We embed random topologies with increasing numberof switches and links to the physical topology. We con-sider two sets of random topologies with a fixed number

0

20

40

60

80

100

20 40 60 80 100 120 140

Per

cent

age

Cor

e B

andw

idth

Utli

zatio

n

Number of Virt Switches(n)

n.sqrt(n) Links (no loops)n.sqrt(n) Links (loops)

2n Links (no loops)2n Links (loops)

Figure 15: Mapping of Random Topologiesof n switches: 1) Number of links = 2n and, 2) Num-ber of links = n

√n. We perform the network mapping

with 10 set of topologies for each size of random topol-ogy. We plot the percentage of backbone link’s band-width utilization against the topology size (number ofswitches) in Figure 15. For topologies with 2n links, thegain is clearly evident, as we are able to fit bigger topolo-gies (up to 130 nodes and 260 links) using loopback linkscompared to only 60 nodes (and 120 links) and withoutloopback links, leading to approximately 2× gain. Sim-ilarly, for topologies with n

√n links, we are able to fit

topologies with 40 switches (252 links) with loopbacklinks, compared to 25 switches (125 links) leading to2× gain. Additionally, we embedded topologies fromTopology Zoo [16] and observed successful mapping for96.5% of topologies.

Take-away: BNV can emulate a network topologywith high fidelity and repeatability by maintaining thekey characteristics of each topology while guaranteeingisolation for each experiment. BNV Mapper can embeda large collection of topology efficiently. The addition ofloopback links increases the scalability significantly.

5 Related WorkNetwork Hypervisors: Network hypervisors likeFlowVisor [24], OpenVirteX [7] and FSFW [2] are usedin major testbeds like GENI [9]. However, these workscan only provide either the corresponding physical topol-ogy or a subset of it. [23] explains ways to provide SDNexperimentation in testbeds. However, supporting em-bedding of arbitrary network topologies is not exploredby existing hypervisors or testbeds.

Switch Abstractions: NVP[17] performs networkvirtualization using virtual switches in the hypervisor.Since, it is based on hypervisor, it is hard to pro-duce arbitrary topologies using software switches dueto packet processing CPU overhead at the VMs orservers. Trellis [10], VINI [8] propose creation of arbi-trary network topologies using containerization and tun-nel abstractions. However, they cannot achieve line-rateneeded for heavy workload testing. Similarly, Mininet[13] and Maxinet [27] are popular network emulationtools used to construct SDN-based virtual topologies onserver/clusters. These are excellent tools for functional

testing, however they cannot scale to emulate large net-works carrying traffic at line-rate. CrystalNet [18] pro-vides an extensible testing environment to mimic pro-duction networks using a large number of virtual ma-chines from public cloud infrastructures. However, itdoes not claim fidelity of network dataplane. BNV canbridge this gap by providing fidelity of the network dataplane. Recently, MiniReal [26] has proposed an ap-proach to achieve multi-switch abstraction using a sin-gle bare-metal switch by modifying the software of theswitches and also loopback link. BNV is different in thatit performs virtualization functioning as a network hyper-visor and does not need to modify the switch software.Additionally, it can dynamically modify topologies atrun-time, while providing repeatable performance.

Network Embedders: There exists a variety of vir-tual network embedding techniques [12, 22]. Other ex-perimental setups like DeterLab [19] use Assign [20].Assign [20] uses simulated annealing to get an embed-ding for a particular network topology. BNV Mapper,since it tries to embed arbitrary topologies, needs exactport mapping. Additionally, BNV Mapper has a differentobjective from the works in [12] to reduce the burden onthe backbone links, and use loopback links instead.

6 ConclusionWe developed BNV (Bare-metal Network Virtualiza-tion), which provides high-fidelity network experimenta-tion at data-plane and control-plane using programmableswitches. BNV can support arbitrary network topologiesusing a unique method of creating loopbacks in switchesin order to provide high fidelity. We propose an ILP-based formulation for efficient embedding of complextopologies to the substrate. We have built BNV on top ofOpenVirtex, and deployed on a production testbed. BNVaccurately emulates the characteristics of the topologiesimplemented over the substrate network while isolatingmultiple experiments.

Acknowledgement: This research is supported bythe National Research Foundation, Prime Ministers Of-fice, Singapore under its National Cybersecurity RD Pro-gramme (grant no. NRF2015NCR-NCR002-001).

References

[1] Cloudlab. http://docs.cloudlab.us/planned.html.[2] Flowspace Firewall. http://globalnoc.iu.edu/sdn/fsfw.htm.[3] Gurobi. http://www.gurobi.com/.[4] National Cybersecurity Lab. https://ncl.sg/.[5] Intel Ethernet Switch Family Memory Efficiency. White

Paper, 2009.[6] M. Al-Fares, A. Loukissas, and A. Vahdat. A scalable,

commodity data center network architecture. In SIG-COMM, 2008.

[7] A. Al-Shabibi, M. De Leenheer, M. Gerola, A. Koshibe,G. Parulkar, E. Salvadori, and B. Snow. Openvirtex:Make your virtual sdns programmable. In HotSDN, 2014.

[8] A. Bavier, N. Feamster, M. Huang, L. Peterson, andJ. Rexford. In vini veritas: Realistic and controlled net-work experimentation. In SIGCOMM, 2006.

[9] M. Berman, J. S. Chase, L. Landweber, A. Nakao, M. Ott,D. Raychaudhuri, R. Ricci, and I. Seskar. GENI: A fed-erated testbed for innovative network experiments . Com-puter Networks, 2014.

[10] S. Bhatia et al. Trellis: A platform for building flexible,fast virtual networks on commodity hardware. CoNEXT,2008.

[11] S. Das. Broadcom smart-buffer technology. 2012.[12] A. Fischer, J. F. Botero, M. T. Beck, H. de Meer, and

X. Hesselbach. Virtual network embedding: A survey.IEEE Communications Surveys Tutorials, 2013.

[13] N. Handigol, B. Heller, V. Jeyakumar, B. Lantz, andN. McKeown. Reproducible network experiments usingcontainer-based emulation. In CoNEXT, 2012.

[14] C. Hopps. Analysis of an equal-cost multi-path algorithm,2000.

[15] G. Jin and B. Tierney. Netest: a tool to measure themaximum burst size, available bandwidth and achievablethroughput. In ITRE, 2003.

[16] S. Knight, H. X. Nguyen, N. Falkner, R. Bowden, andM. Roughan. The internet topology zoo. IEEE Journalon Selected Areas in Communications, 2011.

[17] T. Koponen et al. Network virtualization in multi-tenantdatacenters. In NSDI, 2014.

[18] H. Liu et al. Crystalnet: Faithfully emulating large pro-duction networks. In SOSP, 2017.

[19] J. Mirkovic, T. V. Benzel, T. Faber, R. Braden, J. T. Wro-clawski, and S. Schwab. The deter project: Advancingthe science of cyber security experimentation and test. InIEEE HST, 2010.

[20] R. Ricci, C. Alfeld, and J. Lepreau. A solver for the net-work testbed mapping problem. SIGCOMM CCR, 2003.

[21] R. Ricci and E. e. a. Eide. Introducing CloudLab: Scien-tific infrastructure for advancing cloud architectures andapplications. ;login, 2014.

[22] R. Riggio, F. D. Pellegrini, E. Salvadori, M. Gerola, andR. D. Corin. Progressive virtual topology embedding inopenflow networks. 2013 IFIP/IEEE IM, 2013.

[23] S. S R, J. Mikovic, P. G. Kannan, C. Mun Choon, andK. Sklower. Enabling sdn experimentation in networktestbeds. SDN-NFVSec, 2017.

[24] R. Sherwood, G. Gibb, K. Yap, G. Appenzeller,M. Casado, N. McKeown, and G. Parulkar. Can the pro-duction network be the testbed? In OSDI, 2010.

[25] A. Singla, C.-Y. Hong, L. Popa, and P. B. Godfrey. Jelly-fish: Networking data centers randomly. In NSDI, 2012.

[26] S. Y. Wang and I. Y. Lee. Minireal: A real sdn networktestbed built over an sdn bare metal commodity switch.In ICC, 2017.

[27] P. Wette, M. Drxler, and A. Schwabe. Maxinet: Dis-tributed emulation of software-defined networks. In IFIPNetworking, 2014.

[28] Y. Xia, X. S. Sun, S. Dzinamarira, X. S. Wu, Dingmingan Huang, and T. E. Ng. A tale of two topologies: Ex-ploring convertible data center network architectures withflat-tree. In ACM SIGCOMM, 2017.