1

Real-World Insights from an

SDN Lab

Ron MilfordManager, InCNTRE SDN Lab

Indiana University

A bit about IU, the GlobalNOC, and

InCNTRE....

Indiana University’s Network History

• 1998 University Corporation for Advanced Internet Development (aka

Internet2) selects IU to build and operate the premier network

supporting US research and education. The GlobalNOC is created.

• 1998 National Science Foundation funds IU to provide high-

performance networking between Asia Pacific and US.

• 2005 National Lambda Rail (a facilities-based R&E network) selects IU

to provide NOC services

• 2011 IU announces the formation of Indiana Center for Network

Translational Research and Education (InCNTRE)

• 2013 InCNTRE under contract with ONF to assist with the

development of OpenFlow conformance certification program

The IU GlobalNOC operates this 15,000+ mile network

and....

...these networks internationally...plus

many more domestically

The IU GlobalNOC

• 27 Research and Education Networks

o Architecture, Engineering, Monitoring, Installation

o Exchanges, Regional, National, International

o Optical, Layer2, Layer3

• 3 SDN/OpenFlow Networks

o GENI - 2010, Deployed 1st Nationwide R&E Testbed OpenFlow

Network

o NDDI Prototype - 2011

� Nationwide R&E Production Prototype OpenFlow Network

� IBM G8264

o NDDI AL2S - 2012

� I2 100Gb Advanced Layer 2 Services Production OpenFlow

Network

� Juniper MX-80, Brocade MLXe-4

InCNTRE Mission

• Seeks to advance development, increase knowledge, and encourage

adoption of OpenFlow and other standards-based Software-Defined

Networking (SDN) technologies through:

• A Neutral Testing Facility

• Educational programs

• Training

• Internships

• Direct networking research

• Collaboration between faculty, students, and networking

professionals

InCNTRE Initiatives

• Summer of Networking Internship Program

• Network Professional Training

• SDN Network Research

o FlowScale, RouteFlow, XSP, GENI...

• SDN Lab

o Interoperability Testing

o OpenFlow Conformance Testing

o Internships

o Open Networking Foundation Research Associates

InCNTRE SDN Lab

• Research/Demo Networks

o InteropNet OpenFlow Lab

� Interop Las Vegas 2011

� Interop Las Vegas 2012

o SuperComputing 2011 SCinet Research Sandbox

• ONF Collaboration

o Co - Chair Testing & Interoperability Working Group

o Conformance Testing Program Development

o Conformance Test Specifications

� OpenFlow Switch 1.0 - In Review Phase

� OpenFlow Switch 1.3 - Under development

ONF PlugFests

ONF PlugFests

• March 2012• Controllers - Big Switch, NEC, NTT Data, NOX

• Switches - Big Switch(Indigo), Broadcom, HP, IBM, Intel/WindRiver, Juniper, NEC

• https://www.opennetworking.org/images/stories/downloads/white-papers/onf-testing-

interop-march-2012-whitepaper-v1.0.pdf

• https://www.opennetworking.org/images/stories/downloads/technical-reports/onf-testing-

interop-march-2012-tech-doc-v1.0.pdf

• October 2012• 19 Companies, ~40 Switches, 4 testbeds (OF 1.0, 1.2, Test Frameworks)

• Controllers – Big Switch, HP, Huawei, Ixia, NEC, NTT Data, NTT MCL, NOX

• https://www.opennetworking.org/images/stories/downloads/technical-reports/onf-testing-

interop-oct-2012-tech-doc-v0-4.pdf

• June 2013• Adding OpenFlow 1.3 & OF-Config Testing

What is OpenFlow and Software Defined

Networking in 60 seconds...

OpenFlow is SDN, but SDN is not alway OpenFlow

Software Defined Networking (SDN) is an architecture that

separates and abstracts elements of the network. SDN

centralizes the control plane from the high-speed data plane in

network switching equipment. SDN allows programmatic control

of a network instead of static configuration.

OpenFlow is a protocol that allows the centralized control plane

to control the behavior of distributed switching equipment and

the API that exposes and abstracts the capabilities of the

OpenFlow controlled equipment.

Ethernet Switch

Table-based (e.g., TCAM/CAM) high-speed forwarding engine

Embedded Operating System

Data Plane

Control Plane

Features Value AddCLI, SNMP, TFTP

OpenFlow Controller

Table-based (e.g., TCAM/CAM) high-speed forwarding engine

Embedded Operating System implements OpenFlow

Data Plane

Control Plane

FeaturesValue Add

OpenFlow Protocol

OpenFlow ControllerOpenFlow Controller

OpenFlow Controller

FeaturesValue Add

OpenFlow ProtocolEach switch

connects directly with OF Controller

OpenFlow Abstraction Permits "Slicing" of

the Network into virtual resources

• Allows "Slicing" network into virtual administrative domains

• Each Slice controlled by it's own controller and admin

• Network can be "Sliced" based on many criteriao Switch or Data Path Identifier (DPID)

o Port

o Src or Dst MAC Addresses

o Src or Dst IP Addresses

o Ethertype

o VLAN ID

o TOS Bits

o Rest of 12-tuple (OpenFlow 1.0)

• Flowvisor

• Coming soon in controllers near you

InCNTRE's experience with some interesting

demo networks...

InteropNet LV 2011 OpenFlow Lab

• 14 controller and switch vendors• ~40 hardware and software Switches

• 5 Demos, each with their own controller and administrative “slice” of the network.

InteropNet LV 2011 OpenFlow Lab

• Worked with Stanford's Open Labs

• 3 Months planning, 3 weeks staging, 5 days in production

• OpenFlow 1.0 Switch Vendorso Broadcom, Brocade, Dell, Extreme, Fulcrum, HP, IBM, Juniper, NEC, Netgear, Pronto

o Open vSwitch on XenServer

• OpenFlow 1.0 Controllerso Big Switch, NEC, NOX

• Flowvisor Virtual Slicingo 6 Slices

� Switch & Port

� Source & Destination IP Subnet

� ~60,000 flowspace rules

• With right IP Subnet configured, could connect to almost any port and be in your slice

Supercomputing 2011

• Distributed through show floor

• IBM, HP, NEC, Pronto Switches

• Beacon, NOX, NEC Controllers

• Inter-Connected to GENI and NDDI Prototype OpenFlow Networks

• Inter-Connected to ESnet, Internet 2, NLR, SCinet non-OpenFlow

Networks

• 9 Slices/Administrative Domainso Switch & Port

o Source & Destination MAC Address

o Source & Destination IP Subnet

o VLAN ID

o Ethertype

• 216 Flowspace Ruleso Strict Port Assignments

Supercomputing 2011

IP Subnet: 192.168.140.1/24

MAC Address: 00:02:C9:10:F1:AC

Ethertype: 0x8902

IP Subnets: GENI Subnet, 10.0.0.0/24, 20.0.0.0/24 30.0.0.0/24, 40.0.0.0/24,

50.0.0.0/24

IP Subnets: GENI Subnet, 10.0.0.0/24, 20.0.0.0/24 30.0.0.0/24, 40.0.0.0/24,

50.0.0.0/24

VLAN IDs: 1327,2908-2912

IP Subnet: GENI Subnet

Ethertype: 0x27A0

VLAN: 233, 3715, 3716

Issues from SC11

• Switch CPU limitations• Too many messages to/from the controller

• Traffic flowing before controller can prepopulate tables

• No isolation of switch resources• One heavily active or misbehaving slice can overrun the switch CPU, TCAM or memory

• Switches failing “Open”• Floods traffic everywhere

• Can easily create loops & broadcast storms

• LLDP Behavior Different Across Vendors• Controllers use LLDP for topology discovery

• Hybrid switches send LLDP

• FlowVisor• Can’t change controller IP without restarting FlowVisor

• Rule generation script unable to insert/remove individual flowspace rules

• Some switches not reconnecting correctly through FlowVisor. Required OpenFlow Process

Restarts

Deep Dive into a our nationwide

SDN/OpenFlow Network...

Network Development and Deployment

Initiative (NDDI)

• Partnership among Internet2, Indiana University and the Clean Slate

Program at Stanford University

• Building an OpenFlow-based network substrate to support Internet2’s

new 100Gb/s services

• Services known as I2’s Advanced Layer 2 Service

• Software suite, developed by the GlobalNOC @ IU, known as Open

Exchange Software Suite (OE-SS)

– , © 2013 Internet2

Internet2 Network – Layer 2 Services

A bit about the software IU developed to

provision services on AL2S...

OS3E Service

• Open Science, Scholarship, and Services Exchange

• First service to be built on top of the NDDI substrate.

• A nationwide distributed Layer-2 exchange

• persistent, on-demand & scheduled L2 VLANs

• Supports open inter-domain standards (IDCP)

OS3E Use Cases

• High bandwidth paths for large file transfers

• Layer-2 connectivity between testbeds

• Distributed exchange for establishing IP peering

OESS Key Features

• Develop once, implement on many different types of

switches

• Uses D-Bus to communicate with the OpenFlow

controller allowing easy migration to other OpenFlow

Controllers in the future.

• VLAN Provisioning time: <1 Second typically

• Automated fail-over to backup path

• Controller redundancy

• Automated measurement

• Easy management

Inter-domain Support

• Inter-domain support enables peering with other

dynamic circuit networks.

• Integraded OSCARS 0.6 with OESS for inter-domain

component. Uses IDC (Interdomain Controller) protocol

• See a demo on the real network! http://www.youtube.com/watch?v=C6Nfg6DZqvI&list=UUuFMyVNxNVeTQrKg

km2M_4Q&index=16

• Initial peering with Internet2 at Los Angeles, Chicago, and

Seattle

• Interconnecting with GEANT in New York

• Many more to come

You too can play with OESS

• Apache 2.0 license

• 3 deployment packages• RPMs for RedHat/CentOS 6

• “.tar.gz” distribution

• VM image (virtualbox)

• http://code.google.com/p/nddi

A bit about a production load-balancer

built using OpenFlow...

• Traffic load balancer as a service using OpenFlow

• Use existing switch hardware instead of a dedicated

load balancer

• Built on Beacon

• Integration with other OpenFlow tools• L2 Provisioning, VM Migration

• Hash based on OpenFlow fields (IP src/dst)

• Divides IU Address Space into IP prefixes

• Uses Round Robin to Pre-Distribute Flows across sensors

• Monitors sensor utilization and dynamically adjusts

• Redirects specific types of traffic to specialized

sensors (e.g. http or tcp/udp port)

• Application layer monitoring and failover

• Web UI for admin

Deployment

• IDS Cluster

• 2 Campuses

• 12 Sensor hosts per campus

• 10Gb/sec on all links

• 480Gb/sec total capacity

• https://github.com/InCNTRE/FlowScale

A bit about Internet2's "Innovation Platform"

(i.e., 100G/s connection, SDN support, and

Science DMZ)...

100 GigE Layer 2 Connection

50 – 4/11/2013, © 2012 Internet2

www.internet2.edu

SDN Control Server

Performance Node

Dedicated scienceswitches/servers in

labswith high-speed

storageand network access

Traditional L3 Campus Border Security

High-PerformanceLayer 2/3

Switch/Router

TraditionalCampus

Border Router

Traditionalregional andcommodityproviders

CampusEnterpriseNetwork

Internet2innovationbackbonedelivered

as 100G L1Nx100G

Nx100G

Nx10G

Nx10G

Nx10G

Nx10G

Science DMZ

fasterdata.es.net

innovation@internet2.edu

Software Defined NetworkingGENI

Experiments

Dark Fiber

Optical System

GENI ?Dynamic Layer

2

IP NetworkLayer 3

StaticLayer 2

R&E IP

TR-CPS

Innovation ServicesTraditional Services

Software Defined NetworkingSubstrate

Traditional SwitchSubstrate

Your Research

Innovation Platform

51 – 4/11/2013, © 2012 Internet2

Long-term Architectural Aspiration

52 – 4/11/2013, © 2012 Internet2

Software

Stack —

Today

Vendor Switch

Vendor Software

Flowvisor

Controller(Currently NOX)

OESS

OSCARS

OESS UI

OpenFlow

OpenFlow

IDC

IDC

OSCARS API

OSCARS UI

53 – 4/11/2013, © 2012 Internet2

Software

Stack —

Future

Vendor Switch

Vendor Software

Flowvisor

Controller(Currently NOX)

OESS

OSCARS

OESS UI

OpenFlow

OpenFlow

IDC

IDC

OSCARS API

NSI Extension

OSCARS API

OSCARS UI

54 – 4/11/2013, © 2012 Internet2

Software

Stack —

Not fully

baked

Vendor Switch

Vendor Software

Flowvisor

Controller(Currently NOX)

OESS

OSCARS

OESS UI

Aggregate Manager

Aggregate Manager (FOAM)

OpenFlow

OpenFlow

IDC

IDC

OSCARS API

NSI Extension

Experimenter OpenFlow Controller

OSCARS UI

• http://globalnoc.iu.edu/

• http://www.incntre.iu.edu/

• http://internationalnetworking.iu.edu/