Remote Network Labs: An On-Demand Network Cloud for Configuration Testing Huan Liu, Dan Orban...

Remote Network Labs: An On-Demand Network Cloud for Configuration Testing

Huan Liu, Dan Orban

Accenture Technology Labs

Configuration is hard

• A few reasons– Many SW images for each box

• IOS: 5+ trains, 8 packages for routers, 5 packages for switches– Primitive CLI interface– Commands may behave differently a design may work on paper, but not in practice– Local configuration at a switch/router could have global impact

• Sample manifestations– More outages are caused by operator errors rather than equipment failures 1,2

– 3 in 4 new BGP prefix advertisements are results of misconfiguration (200-1200prefixes/day) 3

– Average 7.17 to 9.63 errors per firewall config 4

1. “Configuration management delivers business resiliency,” The Yankee Group, Nov. 20022. D. Oppenheimer, A. Ganapathi, and D. Patterson, “Why internet services fail and what can be done

about these,” in Proc. USENIX USITS, Oct. 2003.3. R. Mahajan, D. Wetherall and T. Anderson, “Understanding BGP Misconfiguration”, SIGCOMM 20024. A. Wool, “A Quantitative Study of Firewall Configuration Errors”, Computer, 2004

2

Current solutions and their problems• Solution 1: Simulator (e.g., RouterSim, OPNET)

– Cannot capture all aspects of real hardware (e.g., many IOS images)– Support limited # of commands– Have simulation models for a limited number of routers

• Solution 2: Emulator (Dynamips)– Limited set of interfaces are simulated– Support a limited set of Cisco routers

• Solution 3: Build a smaller scale network in Lab to mimic the production network, and test in lab before roll out– Costly investment (routers are not cheap)– Poorly utilized (only during testing), yet necessary in case configuration changes again– May not be available when you need it (spare parts problem)– Time consuming to wire up– Hard to make sure the test network is the same as designed on paper.

3

What your lab looks like?

Our solution: Remote Network Labs• Setup a virtual lab, then test

– Equipment could be anywhere (no longer needs to be co-located)– Design from anywhere (no longer needs to be physically on site)– Quickly and easily reconfigurable

• Advantages– Enable efficient sharing of equipment between projects

• No more procurement delay• Essential zero cost to each project• No need for physical lab space

– Enable design from a Web browser• Everything digitized, including the wiring

– Save/restore legacy environment– Fully automate lab set up

• No physical work to mount equipment and wire• Reduce travel (go Green)

5

Distributed architecture

InternetInternet

San JoseChicago

Client site

http://netlabs.accenture.com

Web browsers

6

How it works

InternetInternet

Interface SW

Interface SWWhen press Deploy

button, send instruction to build tunnel

Front end Back end

7

Making a router/firewall/server part of the lab inventory

InternetInternet

Interface SW

8

Key differences from prior work


Real router Distributed Wire

Emulab N N L2 tunnel

PlanetLab N Y L3

ONL N N L2 tunnel

VINI N Y L3 tunnel

WAIL Y N Real wire

RNL Y Y L2 + L3 tunnel

Use case 1 (layer 2 config)

• Configuring failover using Cisco Catalyst 6500 switches with a Firewall Services Module (FWSM)

• Can also capture transient behavior


Use case 2 (test automation) • Full automation (from setup to teardown)

– Implementing web services API– Implementing library for traffic generation/capture, CLI parsing.

• Add delay/jitter• Inject/observe anywhere


x

x

x

x

Inject

Observe

Add delay/jitterWANWAN

Challenges/drawbacks

• Additional delay

• WAN bandwidth is not free

• L2 interface diversity

• Performance testing (see paper for more details)


Remote Network Labs: An On-Demand Network Cloud for Configuration Testing Huan Liu, Dan Orban...

Documents

Transcript of Remote Network Labs: An On-Demand Network Cloud for Configuration Testing Huan Liu, Dan Orban...