Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18,...
-
Upload
philip-lewis -
Category
Documents
-
view
216 -
download
1
Transcript of Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18,...
![Page 1: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/1.jpg)
Emulaband its lessons and value for
A Distributed Testbed
Jay Lepreau
University of Utah
March 18, 2002
![Page 2: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/2.jpg)
What?
A configurable Internet emulator in a room– Today: 168+160 nodes, 1646 cables, 4x BFS (switch)– virtualizable topology, links, software
Bare hardware with lots of tools: Management Software
An instrument for experimental CS research Universally available to any remote
experimenter Simple to use
![Page 3: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/3.jpg)
Points
Programmable, automated mgmt, complete virtualization:– Qualitatively new environment
Most of it will work in wide area
![Page 4: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/4.jpg)
New Stuff Integrated event system
– Underlying pub/sub system– Integrated into ‘ns’ (statically scheduled)– Start/stop programs– Replayable– Dynamic events– User-accessible
Traffic generation– Automatic, from ns script– New generators:
• TG (tcp, udp)• ‘nse’ with udp, tcp, ftp, telnet
![Page 5: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/5.jpg)
New Stuff (cont’d)
4 node types:– Real, running in the local rack, controlled env.– Real, running ‘nse’– [Simulated]– [Real, in wide-area]
Link configuration and monitoring– Latency, bw, plr, RED, queue size– Link monitoring and capture
GUI network config applet Full-day SIGCOMM tutorial Aug’02
![Page 6: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/6.jpg)
“Programmable Patch Panel”
PCPC
Web/DB/SNMPSwitch MgmtUsers
Internet
Control Switch/Router
Serial
Sharks Sharks
160168
PowerCntl
![Page 7: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/7.jpg)
Fundamental Leverage:
Extremely ConfigurableEasy to Use
– Power– Performance– Virtualization
![Page 8: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/8.jpg)
Key Design Aspects Allow experimenter complete control
– Configurable link bandwidth, latency, and loss rates, via transparently interposed “traffic shaping” nodes that provide WAN emulation
… but provide fast tools for common cases– OS’s, state mgmt tools, IP, batch, ...– Disk loading – 6GB disk image FreeBSD+Linux
• Unicast tool: 88 seconds to load• Multicast tool: 40 nodes simultaneously in < 5 minutes
Virtualization– of all experimenter-visible resources– node names, network interface names, network
addrs– Allows swapin/swapout, easily scriptable
![Page 9: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/9.jpg)
Key Design Aspects (cont’d)Flexible, extensible, powerful
allocation algorithm– Matches desired “virtual” topology to
currently available physical resourcesPersistent state maintenance:
– none on nodes, all in database– work from known state at boot time
Familiar, powerful, extensible configuration language: ns
Separate, isolated control network
![Page 10: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/10.jpg)
Lessons for wide area testbed
Central control: at this scale (1000s) it’s easy
Database!Control node for each site: great
benefits, cheap marginal cost– Trusted, firewall, local disk cache, power
control, console line
Ease of use is dominant driver
![Page 11: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/11.jpg)
Lessons… Generalized resource alloc/mapping
algorithm is great (eg, vs Grid) Get it going quickly, keep it going while
add new stuff– Like a startup– Use feedback and demand– 2.5 years in
Simple authorization model Most of our model and code will work in
wide-area
![Page 12: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/12.jpg)
Lessons…
Freedom for users is freedom for the management software and people
“You’ve got root, use it.”
Over-provision
FreeBSD Jail, or Eclipse/BSD, or VMWare, or ….
![Page 13: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/13.jpg)
Testing is tricky
Have real hardware that can’t virtualize
Test suite part of build Clone DB works some… 8-node minibed Nightly regression testing Schema evolution script/diff/check Developers use/test 3 diff. browsers
![Page 14: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/14.jpg)
Code Base Today 24,100 Web front end 23,900 Back end 2000 ns front end 4200 Resource mapping 4900 Diskimg compression/casting/load 8400 Scripts/daemons from nodes to DB 5000 Event system 6200 Remote console interaction/logging 3300 Regression testing harness and tests 700 Node health monitoring 3700 Documention of internals
![Page 15: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/15.jpg)
More stats
21 “programs”318 “scripts” (including 90 php
scripts, 71 small boot-time scripts)35% Perl32% C19% php12% html, Java, tcl, other
![Page 16: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/16.jpg)
The Database Today
Started with ~18 tables54 tables, 413 columnsGeneral categories
– Physical world: 11 tables, 65 cols– Virtual world: 7 tables, 83 cols– Operational state: 22 tables, 180 cols– Admin data: 14 tables, 85 cols
Note how much operational state shows how much work needs to be done
![Page 17: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/17.jpg)
Testbed Users
30 active projects– more registered– 25 External– About 40/30/30%
dist sys/activenets/traditional networking
~110 users 990 “experiments” in last 8 months
• 7.5/day recently• 40% testbed development
![Page 18: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/18.jpg)
More Sites
More emulab’s under construction:– Kentucky– Umass– Duke, CMU, Cornell, Stuttgart– Others stated intent:
MIT, WUSTL, Princeton, HPLabs, Intel/UCB, Mt. Holyoke, …
![Page 19: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/19.jpg)
Ongoing and Future Work
Federation– heteregeneous sites– resource allocation
Wireless nodes, mobile nodes IXP1200 nodes, tools, code fragments
– Routers, high-capacity shapers Simulation/emulation transparency Event system Scheduling system Topology generation tools and GUI Data capture, logging, visualization tools Microsoft OSs, high speed links, more nodes!
![Page 20: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/20.jpg)
A Global-scale Testbed Federation key Bottom-up “organic” growth
– Local autonomy and priority– Existing hardware resources– Provides diverse hardware
• PCs• Wireless, mobile• Real routers, switches (Wisconsin, …)• Network processors (IXP’s)• Research switches (WUSTL)
But, top-down is much easier: a good start
![Page 21: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/21.jpg)
NSF ITR Proposal (Nov 01)
Global-scale testbedUtah primary
– Research emphasis: software component for heterogeneity; resource allocation/mapping
Collaborators:– Brown, co-PI (resource allocation)– MIT (RON overlay, wireless)– Duke (ModelNet muxing, early adopter)– Mt. Holyoke (education)
![Page 22: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/22.jpg)
Types of Sites
High-end facilitiesGeneric clustersGeneric labs“Virtual machines”
Internet2 links between some sites
![Page 23: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/23.jpg)
Result…
Loosely coupled distributed system
Controlled isolation
“Internet Petri Dish”
![Page 24: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/24.jpg)
New Stuff: Extending to Wireless and
MobileProblems with existing approaches
Same problems as wired domain But worse (simulation scaling, ...) And more (no models for new technologies, ...)
![Page 25: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/25.jpg)
Wireless Virtual to Physical Mapping
![Page 26: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/26.jpg)
Available for universities, labs, and
companies, for research and teaching,
at:
www.emulab.net
![Page 27: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/27.jpg)
A Few Research Issues and Challenges
Network management of unknown and untrusted entities
Security (root!) Scheduling of experiments Calibration, validation, and scaling Artifact detection and control NP-hard virtual --> physical mapping
problem Providing a reasonable user interface ….
![Page 28: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/28.jpg)
How To Use It ... Submit ns script or GUI via web form Behind the scenes:
– Generates config from script & stores in DB– Maps specified virtual topology to physical nodes– Allocate resources– Provides user accounts for node access– Assigns IP addresses and host names– Configures VLANs– Loads disks, reboots nodes, configures Oss– Starts event system, traffic generators, link monitoring/control– Yet more odds and ends ...– User does his/her experiment– [Reports results if batch]
Takes ~3 min to set up 25 nodes, 5 secs/node
![Page 29: Emulab and its lessons and value for A Distributed Testbed Jay Lepreau University of Utah March 18, 2002.](https://reader035.fdocuments.in/reader035/viewer/2022062720/56649efa5503460f94c0d12d/html5/thumbnails/29.jpg)
An “Experiment”
emulab’s central operational entity Directly generated by an ns script, … then represented entirely by
database state
Steps: Web, compile ns script, map, allocate, provide access, assign IP addrs, host names, configure VLANs, load disks, reboot, configure OS’s, run, report