1

A Softwarized Internet of Touch

Fernando KuipersLab on Internet Science, Delft University of Technologyhttps://fernandokuipers.nl/

Joint work with:- Niels van Adrichem- Jorik Oostenbrink- Mohammad Riftadi- Belma Turkovic

2

Do IT initiative

• Delft on Internet-of-Thingshttps://www.tudelft.nl/iot/

• Mission: – Connect researchers & industry– Combine technology and creativity

(within and beyond TU Delft)– Create a 5G+IoT field lab

3

30 years ago…

4

What if you could remotely control real or virtual objects in real time?

“The Tactile Internet”

5

6

7

8

1 ms challenge• Ultra-responsive (round-trip latency of 1 ms)

• Ultra-reliable (outage of about 1 ms/day)

5G

Source: https://www.etsi.org/images/articles/Future-IMT.png

9

Software-Defined Networking (SDN) to the rescue?

10

Traditional routing

Source: https://commons.wikimedia.org/wiki/File:Cisco-rs1.jpg

• Control Plane:

– Maintains routing table

– OSPF, BGP, …

• Data Plane:

– Forwards packets

11

Disadvantages of traditional routing

• Difficult to add new functionality (proprietary software)

• Built on fixed-function hardware

• Complex network management

• Constant communication between routers

12

Software-Defined Networking

Traditional Network Software Defined Network

SDN Controller Forwarding device with decoupled control

Forwarding device with embedded control

Decouple control plane from data plane

Traditional Network Software Defined Network

13

Advantages of SDN

• APPs: monitoring, security, TE, …

• Controller a.k.a. Network Operating System– Centralized decision making– Programmable

• Switches– Only need to worry about forwarding– Reduced CapEx

Benefit from open source solutions

Even open hardware:https://www.opencompute.org

14

OpenFlow

Source: OpenFlow Switch Specification v1.5.1

OpenFlow: Enabling Innovation in Campus Networks

Nick McKeownStanford University

Tom AndersonUniversity of Washington

Hari BalakrishnanMIT

Guru ParulkarStanford University

Larry PetersonPrinceton University

Jennifer RexfordPrinceton University

Scott ShenkerUniversity of California,

Berkeley

Jonathan TurnerWashington University in

St. Louis

This article is an editorial note submitted to CCR. It has NOT been peer reviewed.Authors take full responsibility for this article’s technical content.

Comments can be posted through CCR Online.

ABSTRACTThis whitepaper proposes OpenFlow: a way for researchersto run experimental protocols in the networks they use ev-ery day. OpenFlow is based on an Ethernet switch, withan internal flow-table, and a standardized interface to addand remove flow entries. Our goal is to encourage network-ing vendors to add OpenFlow to their switch products fordeployment in college campus backbones and wiring closets.We believe that OpenFlow is a pragmatic compromise: onone hand, it allows researchers to run experiments on hetero-geneous switches in a uniform way at line-rate and with highport-density; while on the other hand, vendors do not needto expose the internal workings of their switches. In additionto allowing researchers to evaluate their ideas in real-worldtraffic settings, OpenFlow could serve as a useful campuscomponent in proposed large-scale testbeds like GENI. Twobuildings at Stanford University will soon run OpenFlownetworks, using commercial Ethernet switches and routers.We will work to encourage deployment at other schools; andWe encourage you to consider deploying OpenFlow in youruniversity network too.

Categories and Subject DescriptorsC.2 [Internetworking]: Routers

General TermsExperimentation, Design

KeywordsEthernet switch, virtualization, flow-based

1. THE NEED FOR PROGRAMMABLENETWORKS

Networks have become part of the critical infrastructureof our businesses, homes and schools. This success has beenboth a blessing and a curse for networking researchers; theirwork is more relevant, but their chance of making an im-pact is more remote. The reduction in real-world impact ofany given network innovation is because the enormous in-stalled base of equipment and protocols, and the reluctance

to experiment with production traffic, which have created anexceedingly high barrier to entry for new ideas. Today, thereis almost no practical way to experiment with new networkprotocols (e.g., new routing protocols, or alternatives to IP)in sufficiently realistic settings (e.g., at scale carrying realtraffic) to gain the confidence needed for their widespreaddeployment. The result is that most new ideas from the net-working research community go untried and untested; hencethe commonly held belief that the network infrastructure has“ossified”.

Having recognized the problem, the networking commu-nity is hard at work developing programmable networks,such as GENI [1] a proposed nationwide research facilityfor experimenting with new network architectures and dis-tributed systems. These programmable networks call forprogrammable switches and routers that (using virtualiza-tion) can process packets for multiple isolated experimen-tal networks simultaneously. For example, in GENI it isenvisaged that a researcher will be allocated a slice of re-sources across the whole network, consisting of a portionof network links, packet processing elements (e.g. routers)and end-hosts; researchers program their slices to behave asthey wish. A slice could extend across the backbone, intoaccess networks, into college campuses, industrial researchlabs, and include wiring closets, wireless networks, and sen-sor networks.

Virtualized programmable networks could lower the bar-rier to entry for new ideas, increasing the rate of innovationin the network infrastructure. But the plans for nationwidefacilities are ambitious (and costly), and it will take yearsfor them to be deployed.

This whitepaper focuses on a shorter-term question closerto home: As researchers, how can we run experiments inour campus networks? If we can figure out how, we canstart soon and extend the technique to other campuses tobenefit the whole community.

To meet this challenge, several questions need answering,including: In the early days, how will college network admin-istrators get comfortable putting experimental equipment(switches, routers, access points, etc.) into their network?How will researchers control a portion of their local net-work in a way that does not disrupt others who depend onit? And exactly what functionality is needed in network

ACM SIGCOMM Computer Communication Review 69 Volume 38, Number 2, April 2008

15

Installing flow rules

16

OpenNetMon: network telemetry

• Per-flow counters

– Packet counters (PC)

– Byte counter (BC)

– Flow duration (FD)

• Throughput: ∆"#∆$%

• Packet loss: PCstart - PCend

• Delay: timed probe packets

N. van Adrichem, C. Doerr, and F.A. Kuipers, “OpenNetMon: Network Monitoring in OpenFlow Software-Defined Networks,”

Proc. of IEEE/IFIP NOMS, 2014.

17

Failures are bound to happen

Waiting for the controller to install new rules is too time consuming!

18

Fast Failover

1. Fast recovery: Pre-install backup paths2. Fast detection: Couple per-link BFD to Fast Failover buckets

The switch executes the actions of the first bucket with a life watch port3. Slow optimality: Rely on controller to reconfigure

- N. van Adrichem, B. van Asten, and F.A. Kuipers, “Fast Recovery in Software-Defined Networks, Proc. of EWSDN 2014.- N.L.M. van Adrichem, F. Iqbal, and F.A. Kuipers, “Backup rules in Software-Defined Networks,” Proc. of IEEE NFV-SDN 2016.

19Purple: back-up entries

20

SD-WANWAN: A network spanning a large geographical area

B4: Experience with a Globally-DeployedSoftware Defined WAN

Sushant Jain, Alok Kumar, Subhasree Mandal, Joon Ong, Leon Poutievski, Arjun Singh,Subbaiah Venkata, Jim Wanderer, Junlan Zhou, Min Zhu, Jonathan Zolla,

Urs Hölzle, Stephen Stuart and Amin VahdatGoogle, Inc.

b4-sigcomm@google.com

ABSTRACTWe present the design, implementation, and evaluation of B�, a pri-vate WAN connecting Google’s data centers across the planet. B�has a number of unique characteristics: i) massive bandwidth re-quirements deployed to a modest number of sites, ii) elastic traf-�c demand that seeks to maximize average bandwidth, and iii) fullcontrol over the edge servers and network, which enables rate limit-ing and demand measurement at the edge. �ese characteristics ledto a So�ware De�ned Networking architecture using OpenFlow tocontrol relatively simple switches built from merchant silicon. B�’scentralized tra�c engineering service drives links to near ���� uti-lization, while splitting application �ows among multiple paths tobalance capacity against application priority/demands. We describeexperience with three years of B� production deployment, lessonslearned, and areas for future work.

Categories and Subject DescriptorsC.�.� [Network Protocols]: Routing Protocols

KeywordsCentralized Tra�c Engineering; Wide-Area Networks; So�ware-De�ned Networking; Routing; OpenFlow

1. INTRODUCTIONModern wide area networks (WANs) are critical to Internet per-

formance and reliability, delivering terabits/sec of aggregate band-width across thousands of individual links. Because individualWAN links are expensive and because WAN packet loss is typicallythought unacceptable,WANrouters consist of high-end, specializedequipment that place a premium on high availability. Finally, WANstypically treat all bits the same. While this has many bene�ts, whenthe inevitable failure does take place, all applications are typicallytreated equally, despite their highly variable sensitivity to availablecapacity.

Given these considerations, WAN links are typically provisionedto ��-��� average utilization. �is allows the network serviceprovider to mask virtually all link or router failures from clients.

Permission to make digital or hard copies of all or part of this work for personal or

classroom use is granted without fee provided that copies are not made or distributed

for profit or commercial advantage and that copies bear this notice and the full cita-

tion on the first page. Copyrights for components of this work owned by others than

ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-

publish, to post on servers or to redistribute to lists, requires prior specific permission

and/or a fee. Request permissions from permissions@acm.org.

SIGCOMM’13, August 12–16, 2013, Hong Kong, China.

Copyright 2013 ACM 978-1-4503-2056-6/13/08 ...$15.00.

Such overprovisioning delivers admirable reliability at the very realcosts of �-�x bandwidth over-provisioning and high-end routinggear.Wewere facedwith these overheads for building aWAN connect-

ing multiple data centers with substantial bandwidth requirements.However, Google’s data center WAN exhibits a number of uniquecharacteristics. First, we control the applications, servers, and theLANs all the way to the edge of the network. Second, our mostbandwidth-intensive applications perform large-scale data copiesfrom one site to another. �ese applications bene�t most from highlevels of average bandwidth and can adapt their transmission ratebased on available capacity.�ey could similarly defer to higher pri-ority interactive applications during periods of failure or resourceconstraint. �ird, we anticipated no more than a few dozen datacenter deployments, making central control of bandwidth feasible.We exploited these properties to adopt a so�ware de�ned net-

working (SDN) architecture for our data center WAN interconnect.We were most motivated by deploying routing and tra�c engineer-ing protocols customized to our unique requirements. Our de-sign centers around: i) accepting failures as inevitable and com-mon events, whose e�ects should be exposed to end applications,and ii) switch hardware that exports a simple interface to programforwarding table entries under central control. Network protocolscould then run on servers housing a variety of standard and customprotocols. Our hope was that deploying novel routing, scheduling,monitoring, andmanagement functionality and protocols would beboth simpler and result in a more e�cient network.We present our experience deploying Google’s WAN, B�, using

So�ware De�ned Networking (SDN) principles and OpenFlow [��]to manage individual switches. In particular, we discuss how wesimultaneously support standard routing protocols and centralizedTra�c Engineering (TE) as our �rst SDN application. With TE, we:i) leverage control at our network edge to adjudicate among compet-ing demands during resource constraint, ii) use multipath forward-ing/tunneling to leverage available network capacity according toapplication priority, and iii) dynamically reallocate bandwidth in theface of link/switch failures or shi�ing application demands. �esefeatures allow many B� links to run at near ���� utilization and alllinks to average ��� utilization over long time periods, correspond-ing to �-�x e�ciency improvements relative to standard practice.B� has been in deployment for three years, now carries more traf-

�c than Google’s public facing WAN, and has a higher growth rate.It is among the �rst and largest SDN/OpenFlow deployments. B�scales tomeet application bandwidth demandsmore e�ciently thanwould otherwise be possible, supports rapid deployment and iter-ation of novel control functionality such as TE, and enables tightintegration with end applications for adaptive behavior in responseto failures or changing communication patterns. SDN is of course

21

IXP + SDN = SDXSDX: A Software Defined Internet Exchange

Arpit Gupta†, Laurent Vanbever?, Muhammad Shahbaz†, Sean P. Donovan†, Brandon Schlinker‡

Nick Feamster†, Jennifer Rexford?, Scott Shenker⇧, Russ Clark†, Ethan Katz-Bassett‡

†Georgia Tech ?Princeton University ⇧UC Berkeley ‡Univ. of Southern California

Abstract

BGP severely constrains how networks can deliver traffic over theInternet. Today’s networks can only forward traffic based on thedestination IP prefix, by selecting among routes offered by theirimmediate neighbors. We believe Software Defined Networking(SDN) could revolutionize wide-area traffic delivery, by offeringdirect control over packet-processing rules that match on multipleheader fields and perform a variety of actions. Internet exchangepoints (IXPs) are a compelling place to start, given their central rolein interconnecting many networks and their growing importance inbringing popular content closer to end users.

To realize a Software Defined IXP (an “SDX”), we must createcompelling applications, such as “application-specific peering”—where two networks peer only for (say) streaming video traffic. Wealso need new programming abstractions that allow participatingnetworks to create and run these applications and a runtime thatboth behaves correctly when interacting with BGP and ensures thatapplications do not interfere with each other. Finally, we must ensurethat the system scales, both in rule-table size and computationaloverhead. In this paper, we tackle these challenges and demonstratethe flexibility and scalability of our solutions through controlled andin-the-wild experiments. Our experiments demonstrate that our SDXimplementation can implement representative policies for hundredsof participants who advertise full routing tables while achievingsub-second convergence in response to configuration changes androuting updates.

Categories and Subject Descriptors: C.2.1 [Computer-Communication Networks] Network Architecture and Design:Network CommunicationsGeneral Terms: Algorithms, Design, ExperimentationKeywords: software defined networking (SDN); Internet exchangepoint (IXP); BGP

1 Introduction

Internet routing is unreliable, inflexible, and difficult to manage.Network operators must rely on arcane mechanisms to performtraffic engineering, prevent attacks, and realize peering agreements.Internet routing’s problems result from three characteristics of theBorder Gateway Protocol (BGP), the Internet’s interdomain routingprotocol:

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.SIGCOMM’14, August 17–22, 2014, Chicago, IL, USA.Copyright 2014 ACM 978-1-4503-2836-4/14/08 ...$15.00.http://dx.doi.org/10.1145/2619239.2626300

• Routing only on destination IP prefix. BGP selects and exportsroutes for destination prefixes. Networks cannot make more fine-grained decisions based on the type of application or the sender.

• Influence only over direct neighbors. A network selects amongBGP routes learned from its direct neighbors, and exports selectedroutes to these neighbors. Networks have little control over end-to-end paths.

• Indirect expression of policy. Networks rely on indirect, obscuremechanisms (e.g., “local preference”, “AS Path Prepending”) to in-fluence path selection. Networks cannot directly express preferredinbound and outbound paths.

These problems are well-known, yet incremental deployment ofalternative solutions is a perennial problem in a global Internetwith more than 50,000 independently operated networks and a hugeinstalled base of BGP-speaking routers.

In this paper, we develop a way forward that improves our existingrouting system by allowing a network to execute a far wider rangeof decisions concerning end-to-end traffic delivery. Our approachbuilds on recent technology trends and also recognizes the need forincremental deployment. First, we believe that Software DefinedNetworking (SDN) shows great promise for simplifying networkmanagement and enabling new networked services. SDN switchesmatch on a variety of header fields (not just destination prefix),perform a range of actions (not just forwarding), and offer directcontrol over the data plane. Yet, SDN currently only applies tointradomain settings, such as individual data-center, enterprise, orbackbone networks. By design, a conventional SDN controller haspurview over the switches within a single administrative (and trust)domain.

Second, we recognize the recent resurgence of interest in Internetexchange points (IXPs), which are physical locations where multiplenetworks meet to exchange traffic and BGP routes. An IXP is alayer-two network that, in the simplest case, consists of a singleswitch. Each participating network exchanges BGP routes (oftenwith a BGP route server) and directs traffic to other participantsover the layer-two fabric. The Internet has more than 300 IXPsworldwide—with more than 80 in North America alone—and someIXPs carry as much traffic as the tier-1 ISPs [1, 4]. For example,the Open IX effort seeks to develop new North American IXPs withopen peering and governance, similar to the models already takingroot in Europe. As video traffic continues to increase, tensions growbetween content providers and access networks, and IXPs are on thefront line of today’s peering disputes. In short, not only are IXPs theright place to begin a revolution in wide-area traffic delivery, but theorganizations running these IXPs have strong incentives to innovate.

We aim to change wide-area traffic delivery by designing, proto-typing, and deploying a software defined exchange (SDX). Contraryto how it may seem, however, merely operating SDN switches and acontroller at an IXP does not automatically present a turnkey solu-tion. SDN is merely a tool for solving problems, not the solution.

22

Network programmability

Source: https://commons.wikimedia.org/wiki/File:Cisco-rs1.jpg

• SDN controller: programmable control plane

• What about the data plane?

23

Fixed function ASIC

User programmable device!

coded in

The limitation was in the hardware...

24

OpenFlow limitations

For every new protocol new headers need to be added to the OF specification and implemented by switch vendors

Version Date #Headers

OF 1.0 Dec 2009 12

OF 1.1 Feb 2011 15

OF 1.2 Dec 2011 36

OF 1.3 Jun 2012 40

OF 1.4 Oct 2013 41

OF 1.5 Mar 2015 45

25

OpenFlow can be a P4 program!

26

Why data-plane programmability? • The device behaves exacly as you want

• Easy to update functionality

• Remove unused features

• See inside the data plane: – Telemetry information (queue occupancy, latency,

time-stamps) can be used while forwarding the packets

27

#include <core.p4>#include <v1model.p4>struct metadata {}struct headers {}

parser MyParser(packet_in packet, out headers hdr,inout metadata meta,inout standard_metadata_t standard_metadata) {

state start { transition accept; }

}

control MyVerifyChecksum(inout headers hdr, inoutmetadata meta){

apply { }}

control MyIngress(inout headers hdr,inout metadata meta,

inout std_mtdt_t std_mtdt) {apply {

if (std_mtdt.ingress_port == 1) {std_mtdt.egress_spec = 2;

} else if (std_mtdt.ingress_port == 2) {std_mtdt.egress_spec = 1;

}}

}

control MyEgress(inout headers hdr,inout metadata meta,inout standard_metadata_t standard_metadata) {apply { }

}

control MyComputeChecksum(inout headers hdr, inoutmetadata meta) {

apply { }}

control MyDeparser(packet_out packet, in headers hdr) {

apply { }}

V1Switch(MyParser(),MyVerifyChecksum(), MyIngress(),

MyEgress(), MyComputeChecksum(), MyDeparser()

) main;

import drop_heavy_hitters

import drop_ddos

define intent dropHeavyHitters:

to any

for traffic('any')

apply drop_heavy_hitters

with threshold('more', 20)

define intent dropDDoS:

to any

for traffic('any')

apply drop_ddos

with threshold('more', 5)

Similar-to-English network intent language

The complexity of a P4 program

Reproduced from P4.org, “P4 Language Tutorial”, 2018.

Intent-Based Networking needed

28combined.p4

Intent 1 Intent 2 Intent N

Intent Parser

Policy Builder

Templates Repository

P4 Code Generator

Topology Manager

Switch Controller

combined.p4

gRPC

States Manager

HH

DDoS

P4I/OHigh-Level Diagram

import drop_heavy_hittersimport drop_ddos

define intent dropHeavyHitters:to anyfor traffic('any')apply drop_heavy_hitterswith threshold('more', 20)

define intent dropDDoS:to anyfor traffic('any')apply drop_ddoswith threshold('more', 5)

...

M. Riftadi and F.A. Kuipers, “P4I/O: Intent-Based Networking with P4,” under submisison.

29

And why is this relevant for the Tactile Internet?

30

Low-latency challenge

• Congestion – when a network node is trying to forward more data than the outgoing link can process – causes queuing delay

• Existing congestion control approaches are slow to react:– Transport layer reacts to RTT and/or loss– SDN is affected by controller-switch delay

31

Central + local control

• Central controller – Configuresand monitors paths for different service classes

• Local controller – Congestiondetection and reaction in the data plane

B. Turkovic, F.A. Kuipers, N. van Adrichem, and K. Langendoen, “Fast network congestion detection and avoidance using P4”, Proc. of NEAT 2018.

32

Network slicing

33

That’s all folks!

34

Related work1. K. Polachan, T.V. Prabhakar, C. Singh, and F.A. Kuipers, Towards an Open Testbed for Tactile Cyber Physical Systems, Proc. of

COMSNETS 2019.2. A. Indra Basuki and F.A. Kuipers, Localizing link failures in legacy and SDN networks, Proc. of RNDM 2018.3. B. Turkovic, F.A. Kuipers, N. van Adrichem, and K. Langendoen, Fast network congestion detection and avoidance using P4, Proc. of

NEAT 2018.4. D. van den Berg, R. Glans, D. de Koning, F.A. Kuipers, J. Lugtenburg, K. Polachan, T.V. Prabhakar, C. Singh, B. Turkovic, and B. van

Wijk, Challenges in Haptic Communications over the Tactile Internet, IEEE Access, Dec. 2017.5. N. Bizanis and F.A. Kuipers, SDN and virtualization solutions for the Internet of Things: A survey, IEEE Access, Sep. 2016.6. H. Krishna, N.L.M. van Adrichem, and F.A. Kuipers, Providing Bandwidth Guarantees with OpenFlow, Proc. of IEEE SCVT 2016.7. N.L.M. van Adrichem, F. Iqbal, and F.A. Kuipers, Backup rules in Software-Defined Networks, Proc. of IEEE NFV-SDN 2016.8. C. Mas Machuca, S. Secci, P. Vizarreta, F.A. Kuipers, A. Gouglidis, D. Hutchison, S. Jouet, D. Pezaros, A. Elmokashfi, P. Heegaard, and

S. Ristov, Technology-related Disasters: A Survey towards Disaster-resilient Software Defined Networks, Proc. of RNDM 2016.9. N. van Adrichem and F.A. Kuipers, NDNFlow: Software-Defined Named Data Networking, Proc. of IEEE NetSoft 2015.10. M.P.V. Manthena, N. van Adrichem, C. van den Broek, and F.A. Kuipers, An SDN-based Architecture for Network-as-a-Service, Proc. of

IEEE NetSoft 2015.11. N. van Adrichem, B. van Asten, and F.A. Kuipers, Fast Recovery in Software-Defined Networks, Proc. of EWSDN 2014.12. N. van Adrichem, C. Doerr, and F.A. Kuipers, OpenNetMon: Network Monitoring in OpenFlow Software-Defined Networks, Proc. of

IEEE/IFIP NOMS 2014.13. R. van der Pol, M. Bredel, A. Barczyk, B. Overeinder, N. van Adrichem, and F.A. Kuipers, Experiences with MPTCP in an

intercontinental OpenFlow network, Proc. of TNC 2013.