1 Rethinking Network Control & Management The Case for a New 4D Architecture David A. Maltz Carnegie...

1

Rethinking Network Control & ManagementThe Case for a New 4D Architecture

David A. Maltz

Carnegie Mellon University/Microsoft Research

Joint work with

Albert Greenberg, Gisli Hjalmtysson

Andy Myers, Jennifer Rexford, Geoffrey Xie,

Hong Yan, Jibin Zhan, Hui Zhang

222

The Role of Network Control and Management

Many different network environments Access, backbone networks

Data-center networks, enterprise/campus

Sizes: 10-10,000 routers/switches

Many different technologies Longest-prefix routing (IP), fixed-width routing (Ethernet),

label switching (MPLS, ATM), circuit switching (optical, TDM)

Many different policies Routing, reachability, transit, traffic engineering, robustness

The control plane software binds these elements together and defines the network

333

We Can Change the Control Plane!

Pre-existing industry trend towards separating router hardware from software

IETF: FORCES, GSMP, GMPLS

SoftRouter [Lakshman, HotNets’04]

Incremental deployment path exists

Individual networks can upgrade their control planes and gain benefits

Small enterprise networks have most to gain

No changes to end-systems required

444

A Clean-slate Design

What are the fundamental causes of network problems?

How to secure the network and protect the infrastructure?

How to provide flexibility in defining management logic?

What functionality needs to be distributed – what can be centralized?

How to reduce/simplify the software in networks?

What would a “RISC” router look like?

How to leverage technology trends?

CPU and link-speed growing faster than # of switches

555

Three Principles forNetwork Control & Management

Network-level Objectives:

Express goals explicitly

Security policies, QoS, egress point selection

Do not bury goals in box-specific configuration

ManagementLogic

Reachability matrixTraffic engineering rules

666


Network-wide Views:

Design network to provide timely, accurate info

Topology, traffic, resource limitations

Give logic the inputs it needs

ManagementLogic


Read state info

777


Direct Control:

Allow logic to directly set forwarding state

FIB entries, packet filters, queuing parameters

Logic computes desired network state, let it implement it

ManagementLogic


Read state info

Write state

888

Overview of the 4D Architecture

Decision Plane:

All management logic implemented on centralized servers making all decisions

Decision Elements use views to compute data plane state that meets objectives, then directly writes this state to routers

Decision

Dissemination

Discovery

Data

Network-level objectives

Direct control

Network-wide views

999


Dissemination Plane:

Provides a robust communication channel to each router – and robustness is the only goal!

May run over same links as user data, but logically separate and independently controlled

Decision

Dissemination

Discovery

Data


Direct control

Network-wide views

101010


Discovery Plane:

Each router discovers its own resources and its local environment

E.g., the identity of its immediate neighbors

Decision

Dissemination

Discovery

Data


Direct control

Network-wide views

111111


Data Plane:

Spatially distributed routers/switches

Can deploy with today’s technology

Looking at ways to unify forwarding paradigms across technologies

Decision

Dissemination

Discovery

Data


Direct control

Network-wide views

121212

Concerns and Challenges

Distributed Systems issues

How will communication between routers and DEs survive failures in the network?

Latency means DE’s view of network is behind reality. Will the control loop be stable?

What is the overhead to/from the DEs?

What happens in a network partition?

Networking issues

Does the 4D simplify control and management?

Can we create logic to meet multiple objectives?

131313

The Feasibility of the 4D Architecture

We designed and built a prototype of the 4D Architecture

4D Architecture permits many designs – prototype is a single, simple design point

Decision plane

Contains logic to simultaneously compute routes and enforce reachability matrix

Multiple Decision Elements per network, using simple election protocol to pick master

Dissemination plane

Uses source routes to direct control messages

Extremely simple, but can route around failed data links

141414

Evaluation of the 4D Prototype

Evaluated using Emulab (www.emulab.net)

Linux PCs used as routers (650 – 800MHz)

Tested on 9 enterprise network topologies (10-100 routers each)

Example network with 49 switches and 5 DEs

151515

Performance of the 4D Prototype

Trivial prototype has performance comparable to well-tuned production networks

Recovers from single link failure in < 300 ms

< 1 s response considered “excellent”

Faster forwarding reconvergence possible

Survives failure of master Decision Element

New DE takes control within 1 s

No disruption unless second fault occurs

Gracefully handles complete network partitions

Less than 1.5 s of outage

161616

Fundamental Problem: Wrong Abstractions

Management Plane• Figure out what is happening in

network• Decide how to change it

Shell scripts Traffic Eng

DatabasesPlanning tools

OSPFSNMP netflow modemsConfigs

OSPFBGP

Link metrics

OSPFBGP

OSPFBGP

Control Plane• Multiple routing processes on

each router• Each router with different

configuration program• Huge number of control knobs:

metrics, ACLs, policy

FIB

FIB

FIB

Routing policies

Packet filters

Data Plane• Distributed routers• Forwarding, filtering, queueing• Based on FIB or labels

171717

Good Abstractions Reduce Complexity

All decision making logic lifted out of control plane

Eliminates duplicate logic in management plane

Dissemination plane provides robust communication to/from data plane switches

ManagementPlane

Control Plane

Data Plane

DecisionPlane

Dissemination

Data Plane

Configs

FIBs, ACLs FIBs, ACLs

181818

Today: Simple Things are Hard to Do

D

Access Networks

Inter-POP Links

191919

Fundamental Problem: Configurations Allow Too Many Degrees of Freedom

Computing configuration files that cause control plane to compute desired forwarding states is intractable NP-hard in many cases

Requires predictive model of control plane behavior

Configurations files form a program that defines a set of forwarding states Very hard to create program that permits only desired states, and

doesn’t transit through bad ones

Forwarding states allowed by configs

Auto-adaptation leads to/thru bad states

Direct Control avoids bad states

202020

Fundamental Problem: Conflation of Issues

Ideal case: all routing information flooded to all routers inside network

Robustness achieved via flooding

Reality: routing information filtered and aggregated extensively

Route filtering used to implement security and resource policies

Route aggregation used to achieve scalability

212121

4D Separates Distributed Computing Issues from Networking Issues

Distributed computing issues ! protocols and network architecture Overhead

Resiliency

Scalability

Networking issues ! management logic Traffic engineering and service provisioning

Egress point selection

Reachability control (VPNs)

Precomputation of backup paths

222222

Future Work

Scalability Evaluate over 1-10K switches, 10-100K routes

Networks with backbone-like propagation delays

Structuring decision logic Arbitrate among multiple, potentially competing objectives

Unify control when some logic takes longer than others

Protocol improvements Better dissemination and discovery planes

Deployment in today’s networks Data center, enterprise, campus, backbone (RCP)

232323

Future Work

Experiment with network appliances

Traffic shapers, traffic scrubbers

Expand relationships with security

Using 4D as mechanism for monitoring/quarantine

Formulate models that establish bounds of 4D

Scale, latency, stability, failure models, objectives

Generate evidence to support/refute principles

24

Questions?

252525

Direct Control Provides Complete Control

Zero device-specific configuration

Supports many models for “pushing” routes

Trivial push – convergence requires time for all updates to be receive and applied – same as today

Synchronized update – updates propagated, but not applied till agreed time in the future – clock skew defines convergence time

Controlled state trajectory – DE serializes updates to avoid all incorrect transient states

262626


interface Ethernet0 ip address 6.2.5.14 255.255.255.128interface Serial1/0.5 point-to-point ip address 6.2.2.85 255.255.255.252 ip access-group 143 in frame-relay interface-dlci 28

router ospf 64 redistribute connected subnets redistribute bgp 64780 metric 1 subnets network 66.251.75.128 0.0.0.127 area 0router bgp 64780 redistribute ospf 64 match route-map 8aTzlvBrbaW neighbor 66.253.160.68 remote-as 12762 neighbor 66.253.160.68 distribute-list 4 in

access-list 143 deny 1.1.0.0/16access-list 143 permit anyroute-map 8aTzlvBrbaW deny 10 match ip address 4route-map 8aTzlvBrbaW permit 20 match ip address 7ip route 10.2.2.1/16 10.2.1.7

272727


Router ID (sorted by file size)8810

Lines in

config file

2000

1000

0

Size of configuration files in a single enterprise network (881 routers)

282828

292929

303030

Fundamental Problem: Conflating Distributed Systems Issues with Networking Issues

Distributed Systems Concern: resiliency to link failures

Solution: multiple paths through routing process graph

D

D left

RoutingProcess

D left

RoutingProcess

D left

RoutingProcess

D

D

D

313131

Distributed Systems Concern: resiliency to link failures

Solution: multiple paths through routing process graph

D right

RoutingProcess

D left

RoutingProcess

D left

RoutingProcess

D

D

D


323232


Networking Concern: implement resource or security policy

Solution: restrict flow of routing information, filter routes, summarize/aggregate routes

D

D left

RoutingProcess

D left

RoutingProcess

D left

RoutingProcess

D

D

D

Filter routes to D

333333

4D Supports Network Evolution & Expansion

Decision logic can be upgraded as needed

No need for update of distributed protocols implemented in software distributed on every switch

Decision Elements can be upgraded as needed

Network expansion requires upgrades only to DEs, not every switch

343434

Reachability Example

Two locations, each with data center & front office

All routers exchange routes over all links

R1 R2

R5

R4R3

Chicago (chi)

New York (nyc)Data Center Front Office

353535


R1 R2

R5

R4R3

Chicago (chi)

New York (nyc)Data Center

chi-DC

chi-FO

nyc-DC

nyc-FO

chi-DC

chi-FO

nyc-DC

nyc-FO

Front Office

363636


R1 R2

R5

R4R3

Data Center

chi-DC

chi-FO

nyc-DC

nyc-FO

chi-DC

chi-FO

nyc-DC

nyc-FO

Packet filter:Drop nyc-FO -> *Permit *

Packet filter:Drop chi-FO -> *Permit *

Front Office

chi

nyc

373737


A new short-cut link added between data centers

Intended for backup traffic between centers

R1 R2

R5

R4R3

Data Center



Front Office

chi

nyc

383838


Oops – new link lets packets violate security policy!

Routing changed, but

Packet filters don’t update automatically

R1 R2

R5

R4R3

Data Center



Front Office

chi

nyc

393939

Prohibiting Packets from chi-FO to nyc-DC

404040


Typical response – add more packet filters to plug the holes in security policy

R1 R2

R5

R4R3

Data Center Front Office

chi

nyc



414141


Packet filters have surprising consequences

Consider a link failure

chi-FO and nyc-FO still connected

R1 R2

R5

R4R3

Data Center

Drop nyc-FO -> *

Front Office

chi

nycDrop chi-FO -> *

424242


Network has less survivability than topology suggests

chi-FO and nyc-FO still connected

But packet filter means no data can flow!

Probing the network won’t predict this problem

R1 R2

R5

R4R3

Data Center

Drop nyc-FO -> *

Front Office

chi

nycDrop chi-FO -> *

434343

Allowing Packets from chi-FO to nyc-FO

444444

Multiple Interacting Routing Processes

OSPF BGP OSPF

FIBFIB

OSPF

FIB

OSPF

FIB

OSPF

FIB

OSPF

EB

GPPolicy1 Policy2

Internet

ClientServer

454545

The Routing Instance Graph of a 881 Router Network

464646

Reconvergence Time UnderSingle Link Failure

474747

Reconvergence Time When Master DE Crashes

484848

Reconvergence Time WhenNetwork Partitions

494949

Reconvergence Time WhenNetwork Partitions

505050

Many Implementations Possible

Multiple decision engines• Hot stand-by• Divide network & load share

Distributed decision engines• Up to one per router

Choice can be based on reliability requirements• Dessim. Plane can be in-band, or leverage OOB links

Less need for distributed solutions (harder to reason about)• More focus on network issues, less on distributed protocols

Single redundant decision engine

515151

Direct Expression Enables New Algorithms

OSPF normally calculates a single path to each destination D

OSPF allows load-balancing only for equal-cost paths to avoid loops

Using ECMP requires careful engineering of link weights

D

D

Decision Plane with network-wide view can compute multiple paths• “Backup paths” installed for free!• Bounded stretch, bounded fan-in

525252

Systems of Systems

Systems are designed as components to be used in larger systems in different contexts, for different purposes, interacting with different components Example: OSPF and BGP are complex systems in its own right,

they are components in a routing system of a network, interacting with each other and packet filters, interacting with management tools …

Complex configuration to enable flexibility The glue has tremendous impact on network performance

State of art: multiple interactive distributed programs written in assembly language

Lack of intellectual framework to understand global behavior

535353

Supporting Network Evolution

Logic for controlling the network needs to change over time Traffic engineering rules

Interactions with other networks

Service characteristics

Upgrades to field-deployed network equipment must be avoided Very high cost

Software upgrades often require hardware upgrades (more CPU or memory)

545454

Supporting Network EvolutionToday

Today’s “Solution”

Vendors stuff their routers with software implementing all possible “features”

– Multiple routing protocols

– Multiple signaling protocols (RSVP, CR-LDP)

– Each feature controlled by parameters set at configuration time to achieve late binding

Feature-creep creates configuration nightmare

– Tremendous complexity for syntax & semantics

– Mis-interactions between features is common

Our Goal: Separate decision making logic from the field-deployed devices

555555

Supporting Network Expansion

Networks are constantly growing

New routers/switches/links added

Old equipment rarely removed

Adding a new switch can cause old equipment to become overloaded

CPU/Memory demands on each device should not scale up with network size

565656

Supporting Network ExpansionToday

Routers run a link-state routing protocol

Size of link-state database scales with # of routers

Expanding network can exceed memory limits of old routers


Monitor resources on all routers

Predict approach of exhaustion and then:

– Global upgrade

– Rearchitecture of routing design to add summarization, route aggregation, information hiding

Our Goal: make demands scale with hardware (e.g., # of interfaces)

575757

Supporting Remote Devices

Maintaining communication with all network devices is critical for network management

Diagnosis of problems

Monitoring status and network health

Updating configuration or software

“the chicken or the egg….”

Cannot send device configuration/management information until it can communicate

Device cannot communicate until it is correctly configured

585858

Supporting Remote DevicesToday


Use PSTN as management network of last resort

Connect console of remote routers to phone modem

Can’t be used for customer premise equipment (CPE): DSL/cable modems, integrated access devices (IADs)

In a converged network, PSTN is decommissioned

Our Goal: Preserve management communication to any device that is not physically partitioned, regardless of configuration state

595959

Recent Publications

G. Xie, J. Zhan, D. A. Maltz, H. Zhang, A. Greenberg, G. Hjalmtysson, J. Rexford, “On Static Reachability Analysis of IP Networks,” IEEE INFOCOM 2005, Orlando, FL, March 2005.

J. Rexford, A. Greenberg, G. Hjalmtysson, D. A. Maltz, A. Myers, G. Xie, J. Zhan, H. Zhang, “Network-Wide Decision Making: Toward a Wafer-Thin Control Plane,” Proceedings of ACM HotNets-III, San Diego, CA, November 2004.

D. A. Maltz, J. Zhan, G. Xie, G. Hjalmtysson, A. Greenberg, H. Zhang, “Routing Design in Operational Networks: A Look from the Inside,” Proceedings of the 2004 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications (ACM SIGCOMM 2004), Portland, Oregon, 2004.

D. A. Maltz, J. Zhan, G. Xie, H. Zhang, G. Hjalmtysson, A. Greenberg, J. Rexford, “Structure Preserving Anonymization of Router Configuration Data,” Proceedings of ACM/Usenix Internet Measurement Conference (IMC 2004), Sicily, Italy, 2004.

1 Rethinking Network Control & Management The Case for a New 4D Architecture David A. Maltz Carnegie...

Documents

Transcript of 1 Rethinking Network Control & Management The Case for a New 4D Architecture David A. Maltz Carnegie...