Report of NSF Workshop on Network Research Testbeds

NSF Workshop on Network Research Testbeds: Workshop Report, November 2002. 1

Title Page

Report to the National Science Foundation Directorate for Computer and Information Science and Engineering (CISE)

Advanced Networking Infrastructure & Research Division

Report of NSF Workshop on Network Research Testbeds

http://gaia.cs.umass.edu/testbed_workshop

Any opinions, findings, conclusions or recommendations expressed in this Report are those of the authors and do not necessarily reflect the views of the authors’ institutions or the NSF.

The NSF Workshop on Network Research testbeds was supported by NSF under grant ANI-0237569.


Authors and Contributors The following individuals attended the NSF Workshop on Network Research Testbeds and contributed to the writing of this report.

Name Affiliation Email Bob Aiken Cisco [email protected] Victor Bahl Microsoft [email protected] Bobby Bhattacharjee U. Maryland [email protected] Bob Braden USC/ISI [email protected] Mark Crovella Boston U. [email protected] Victor Frost U. Kansas [email protected] Mario Gerla UCLA [email protected] John Heidemann USC/ISI [email protected] Kevin Jeffay U. North Carolina [email protected] Jim Kurose U. Massachusetts [email protected] Jay Lepreau U. Utah [email protected] Russ Mundy TIS Labs [email protected] Craig Partridge BBN [email protected] Larry Peterson Princeton [email protected] Ramesh Rao UC San Diego [email protected] D. Raychaudhuri Rutgers/WINLAB [email protected] Henning Schulzrinne Columbia [email protected] Jonathan Shapiro U. Massachusetts [email protected] Karen Sollins MIT [email protected] Jon Turner Washington U. [email protected] Amin Vahdat Duke [email protected] Harrick Vin U. Texas [email protected] Ellen Zegura Georgia Tech [email protected]

Acknowledgments We wish to express our appreciation to the NSF Directorate for Computer and Information Science and Engineering (CISE) and its Divisions of Advanced Networking Infrastructure & Research (ANIR) and Advanced Computational Infrastructure & Research (ACIR) for their support and encouragement. In particular, we wish to acknowledge the assistance and participation of the following individuals.

Blatecky, Alan NSF [email protected] Bush, Aubrey NSF [email protected] Darleen Fisher NSF [email protected] Jukan, Admela NSF [email protected] Maeda, Mari nsf [email protected] Znati, Ty NSF [email protected]


The workshop organizers were Bob Braden (ISI), Mario Gerla (UCLA), Jim Kurose (U. Massachusetts, Chair), Jay Lepreau (U. Utah), Ramesh Rao (UC San Diego), and Jon Turner (Washington U.).


Table of Contents Title Page 1Authors and Contributors 2Acknowledgements 2Table of Contents 4Executive Summary 51. Introduction 102. Workshop Background 123. Surveying the Testbed Spectrum 144. The Past is Prelude: Past Experience with Testbeds 215. Important Testbed Characteristics 276. Research Opportunities and Benefits 357. Recommendations 46Appendix A: Workshop Agenda 50


Executive Summary Advances in science and engineering, new research paradigms, and new forms of commerce, information dissemination, and communication have all been made possible by the infrastructure provided by today’s computer networks. These networks have their foundations in four decades of networking research. Throughout this period, network research testbeds have been the crucial proving grounds in which new networking research ideas could be tested, stressed, observed, reformulated, and ultimately proven before making their way into operational systems.

A new generation of networked applications, as well as changing uses and requirements for existing networks, demand continuing advances in networking research. Spatially-dense and large-scale sensor networks, pervasive computing networks, wireless networks, application-level overlay networks, extensible networks, and emergency-response networks will all challenge and perhaps ultimately change the way networks operate. Some of these networks may be as different from today’s Internet, as the Internet was from the telephone networks before it. But where will these new network architectures and protocols find their proving grounds? Where will the necessary cycles of design, implementation, measurement, and experimentation take place? The NSF Workshop on Network Research Testbeds was held on October 17-18, 2002 , bringing together 23 key networking researchers to discuss the focus, needs, and hardware and software structures for a new generation of network research testbeds - testbeds that will allow for disruptive experimentation with network infrastructure, and that will enable and enhance cutting edge networking and network-application research over the next decade and beyond. This report documents the findings and recommendations of this workshop. Given the critical role of testbeds in networking research, and the many benefits that flow from a vibrant research activity in such testbeds, workshop participants recommend that NSF initiate a significant new program to provide core funding for network research testbeds. The findings and recommendations in this report can provide valuable guidance to the NSF in defining such a new program. The body of this report provides the detailed context and discussion for the workshop’s findings. We summarize these findings here first, followed by our overall recommendations. The statements of our findings and recommendations in this Executive Summary below summarize the more detailed statements found in the body of this report. Summary of Findings A broad spectrum of testbed activities can play a role in advancing experimental networking research. Our first set of findings place the various types of testbed activities in perspective, and highlight emerging new types of testbeds that hold special promise:

FINDING 1: Classes of research testbeds. Two broad classes of network testbeds can be distinguished. Multi-user experimental facilities and proof-of-concept testbeds both play an important role in an overall networking research program, but serve quite different purposes.


FINDING 2: New research testbed structures. The development of cluster testbeds, overlay testbeds, federated testbeds and network research kits by the networking research community constitutes an innovative response to long-felt needs within the community. These approaches show considerable promise. FINDING 3: Moving research results into practice. While the networking community needs its own research testbeds to support systems and experimental research, it also needs vehicles for moving research results into commercial practice.

Many lessons can be learned from our community’s past experience with network research testbeds:

FINDING 4: Successful past research testbed efforts. Network testbeds have a proven track record of providing the means by which important research advances in our community could be tested, stressed, observed, refined, and ultimately proven. Important representative efforts are discussed in Section 4. FINDING 5: Unsuccessful past research testbed efforts. Network research testbeds can also fail (in the sense of not producing research insights that inform the community) for many reasons, including being overwhelmed by the operational aspects of building and maintaining the testbed, failure to publish or distribute results and/or software, and lack of focus on the testbed’s research goals.

Our next set of findings identifies eleven important characteristics of testbeds that should be considered when thinking about future testbed efforts. These findings reflect the workshop participants’ collective experience with past and current testbeds, as well their hopes for how future research testbeds will be shaped:

FINDING 6: Network research testbeds - driven by a research agenda and meeting networking researcher needs. The most important characteristic of a network research testbed is perhaps also the most self-evident: that network research testbeds need to be driven by research coming from the networking community. FINDING 7: The tension between application users and networking research. Network testbeds that focus primarily on users and applications are typically not the best vehicles for supporting networking research, since users’ expectations for reliable, production network service are incompatible with networking researchers’ need to experiment with and modify network infrastructure components. FINDING 8. Geographic Distance. Testbed networks need not span large geographic distances to serve as valuable resources for networking researchers. Indeed geographic distribution can make certain kinds of experiments more difficult to execute and can act as a disincentive to performing such experiments. Local area and cluster testbeds can be highly effective research vehicles in spite of their limited geographic extent.


FINDING 9: The role of bandwidth. Dedicated, high-bandwidth wide-area links are not a necessary prerequisite for creating useful networking testbeds. Emulation of wide-area links in local labs represents a cost-effective alternative that should be considered when high-speed links are a necessary testbed component. FINDING 10: The importance (and cost) of operations and management. The operation and management of testbeds requires careful planning. The “people costs” for testbed software and hardware development, and for ongoing operational support can often significantly exceed the hardware costs. FINDING 11: Importance of testbed management software and tools. Investment in testbed management software and tools is crucial, allowing higher research productivity and significant cost efficiencies in the long run. FINDING 12: Timing is critical to success. Timing - both start date and duration - plays a critical role in the success of testbeds. Testbeds will require varying combinations of architectural design, development and use, which should be reflected in their duration. FINDING 13: The need for testbed instrumentation and measurement. A lack of good measurements and models of traffic, protocols, and applications hinders the network research community. Appropriately instrumented testbeds can help solve this problem. FINDING 14: Open hardware and software needed. The network research community needs open programmable hardware and software at all levels. FINDING 15: The important role of research community. Many successful testbeds claim a vibrant community, either pre-existing or catalyzed by the testbed itself. Testbeds can serve a special role in bringing together diverse communities (e.g., industry/government/academia, EE/CS/applications). FINDING 16: An important educational role for testbeds. Testbeds provide a unique educational opportunity for the students involved in their development, as well as a potential platform for class projects. Students, faculty, and others involved in testbeds benefit from developing systems-building skills and other experimental research capabilities.

Our final two findings address the ultimate goal of network research testbeds - the research itself:

FINDING 17: Research, reality, and technology transfer. Network research testbeds enable a critical stage in the process of understanding the relationship between the research and reality. Implementation and experimentation in a testbed can also facilitate the process of transferring research results into industry and the broader community FINDING 18: Many research areas enabled by testbeds. Testbeds can enable research in many different areas of networking. Areas that are particularly “ripe” for testbed activity include overlay networks, sensor networks, wireless networks, security, dynamically extensible networks, and measurement.


WORKSHOP RECOMMENDATIONS Based on these findings and other observations documented in this report, workshop participants make the following recommendations to the National Science Foundation: Recommendation 1: Initiate a new program in network research testbeds. Given the critical role of research testbeds in networking research, and the many benefits that flow from a vibrant research activity in such testbeds, NSF should initiate a significant new program to provide core funding for network research testbeds. These testbeds should be driven by the needs and research goals of networking researchers.

Recommendation 2: Recognize the value of a broad spectrum of different testbed activities. Testbeds can range widely in size, scope and geographic extent. They may take the form of traditional network testbeds, cluster testbeds, overlay testbeds, federated testbeds or network research kits. Testbeds can serve a variety of specific purposes and emphasize a range of different research concerns. Dedicated, high-bandwidth, wide-area links are not a necessary prerequisite for creating a useful networking testbed. We recommend that a new networking testbed program embrace the full spectrum of testbed activities. Recommendation 3: Appropriately fund the human resources needed to conceive, design, construct, operate, and maintain testbeds and their management software. Professional staff will be required to design, build, maintain, and operate testbeds, and provide support for the testbed’s user community. Truly innovative testbeds and management models are likely to require full-fledged research efforts. Such “people costs” can often significantly exceed hardware infrastructure costs. Recommendation 4: Adopt funding models that provide long-term support for testbeds. Significant investments in human, software, and hardware resources will often be required in a testbed over time. To realize the full potential of such investments, and to allow researchers maximum opportunity to work with well-developed smoothly-functioning testbeds, long term support for specific network research testbed efforts will be required, e.g., a funding cycle of 5 years, with renewal possible. Recommendation 5: Network research testbeds should be driven by the needs and research goals of networking researchers. Network research testbeds are distinct from generic networked testbeds that provide production-quality service in that research testbeds must support the needs of networking researchers These needs include the ability to experiment with, modify and replace network infrastructure components - an inherently destabilizing process. The presence of specific applications should be considered as long as their presence supports the needs of networking researchers - by driving the networking research agenda, by allowing destabilizing experimentation with infrastructure components, by providing realistic workloads, and by validating networking research through compelling demonstrations.


Recommendation 6: Emphasize the importance of instrumentation, measurement, archiving, and dissemination of measured data in research testbeds. In evaluating proposed testbeds, particularly those with an envisioned evolution from research testbed to engineering testbed to operational network, intelligent integration of measurement into the architectural design should be given due weight. Recommendation 7: Consider providing substantial, long-term support for a small number of large (center-like), multi-user, experimental facilities, providing open, well-managed experimental services to remote users. This support should include support for professional staff to manage the facilities and provide for their on-going development. Such facilities could also play an important role in building research community around the shared research infrastructure, and provide educational opportunities for faculty and students alike in the area of experimental systems research through internships, tutorials, and other educational activities. Recommendation 8: Provide network research testbed opportunities for a broad set of networking research topics, while providing specific attention to research in overlay-networks, fixed and mobile wireless networks, sensor networks, security, dynamically extensible networks, and measurement. These specific areas were identified by workshop participants as being particularly “ripe” for research advances and particularly well-suited to leverage testbed activity. Recommendation 9: Initiate a broad-based effort to find ways to facilitate technology transfer for networking research results. This should include exploration of ways to bridge the gap between network research testbeds and user/application oriented testbeds, to enable new networking technologies to be demonstrated within user/application oriented testbeds.


1. Introduction Computer networks have profoundly changed our personal and professional lives. Advances in science and engineering, new research paradigms, and new forms of commerce, information dissemination, and communication have been made possible by the infrastructure provided by today’s computer networks. The networks that have enabled these revolutions have their foundations in four decades of networking research. Throughout this period, network research testbeds have provided the crucial proving grounds for much of this research. Today’s Internet protocols trace their roots to the ARPA network in which they and their ancestors were first developed, experimented with, and later deployed. More recent advances such as IP telephony, high-performance protocols and architectures, multimedia communication, and new communication paradigms came of age in network research testbeds such as DARTnet and the Gigabit testbeds of the 1990’s. Throughout the past four decades, network research testbeds have provided the experimental infrastructure in which new ideas could be tested, stressed, observed, reformulated, and ultimately proven before making their way into operational systems. A new generation of networked applications, as well as changing uses and requirements for existing networks (e.g., a more secure cyber-infrastructure), require new advances in networking research. Spatially-dense and large-scale sensor networks, pervasive computing networks, wireless networks, application-level overlay networks, and emergency-response networks will all challenge and perhaps ultimately change the way we currently build networks. Some of these networks may be as different from today’s Internet, as the Internet was from the telephone networks before it. But where will these new network architectures and protocols find their proving grounds? Where will the necessary cycles of design, implementation, measurement, and experimentation take place? The purpose of the NSF Workshop on Network Research Testbeds was to bring together approximately 20 key networking researchers to discuss the focus, needs, and hardware and software structures for a new generation of network research testbeds - testbeds that will enable and enhance cutting edge networking and network-application research over the next decade and beyond. This report documents the findings and recommendations of this workshop. Workshop participants strongly recommend that the National Science Foundation initiate a bold new program that will create this crucially important next generation of network research testbeds, thereby enabling the networking research advances that will fuel continuing innovations in science, in engineering, and in myriad societal applications of networked communication and distributed computation. Workshop participants also stressed that network research testbeds need to be driven by research coming from the networking community, and that (unlike production networks) this often requires disruptive experimentation with the network infrastructure. The remainder of this report is structured as follows. Section 2 provides additional background about the workshop and provides context for the subsequent sections. Section 3 describes a broad spectrum of research testbed types, and a number of important attributes of such testbeds.


Section 4 describes a number of past research testbeds. Section 5 identifies ten important issues to be considered when thinking about future network research testbeds. These issues reflect the workshop participants’ collective experience (“lessons learned”) with past and current testbeds, as well their hopes for how future research testbeds will be shaped. Since the purpose of network research testbeds is to drive networking research, Section 6 begins with a broad discussion of the value of network research testbeds in the research enterprise, and then identifies a number of specific research opportunities that could be enabled through a research testbed program. Throughout sections 3 through 6, we call out our conclusions and findings in boxes accompanied by surrounding amplifying text. The material in sections 3 through 6 provide the foundation for the overall recommendations of this report, which are presented in section 7.


2. Workshop Background On October 17-18, 2002, a group of 23 networking researchers and six NSF staff met for a one-and-a-half-day workshop on network research testbeds. The workshop was organized with the encouragement and support of NSF CISE ANIR program managers. The goal of the workshop was to identify research opportunities, and the needs, characteristics, and hardware and software structures for a successful new generation of network research testbeds. The background, findings, and overall recommendations of the workshop are presented in this report. The workshop agenda is included as Appendix A; workshop participants are listed at the beginning of this report. In order to place the workshop’s discussion of network research testbeds in the proper context, the participants found it valuable to distinguish between testbeds that require production-quality network services and those that allow for disruptive experimentation with the network infrastructure.

• Testbeds requiring production-quality network services. These testbeds require high-performance networks that typically provide high-bandwidth, low latency connectivity between research sites. These networks are often dedicated to the research applications (e.g., Grid, e-Science) running on the network. Such networks are typically not meant to be used in ways that could disrupt the delivery of services to users. Rather, these networks exist to support “application-dictated development of application software toolkits, middleware, computing and networking. [These networks] must provide delivered experimental services on a persistent basis and yet encourage experimentation with innovative and novel concepts.” 1 Workshop participants noted that a key distinguishing requirement here is that these testbeds require production-quality network services, while network research testbeds allow for disruptive experimentation with the underlying network infrastructure. NSF CISE ANIR Division Director Aubrey Bush has coined the term “Experimental Networks” for testbeds that require production-quality network services but carry only scientific research-related traffic. Like commodity high-performance networks (such as Internet2’s Abilene network and WorldCom’s vBNS+ network) these networks are available on a persistent basis (barring unforeseen outages) and “dependable.”

• Network Research Testbeds are the topic of this workshop. These testbeds are driven by fundamental systems and experimental research in networking. Network testbeds can range widely in size, scope and geographic extent, and can employ a wide range of underlying technologies. They are often under complete control of the networking researcher(s), and are often used for experiments, which have the potential to be disruptive to normal network usage. They do not necessarily run “live” applications, although applications play an important role in defining the workload, constraints, and requirements for the networking concepts under research.

1 From “NSF CISE Grand Challenges in e-Science Workshop Report,” submitted to NSF, January 24, 2002.


Network research testbeds, and their relationship to experimental and production networks can also be viewed in light of the progression of a research innovation (e.g., as embodied in a new protocol) from first concept to deployment in a commercial product, as shown below in Figure 1.

theory simulation local prototyping

research testbed

early stage products, commercialization,

standardization

NSF-funded research

productvendors

research testbeds

Figure 1: From research to research-testbed to commercialization

While industry supports basic research, builds testbeds, and sells products, workshop participants note that the left end of this progression is the province of NSF-funded networking research, while the extreme right end is typically the province of product vendors. Industrial consortia and Interop have also played a significant role in organizing testbeds that fall in the middle-right.


3. Surveying the Testbed Spectrum The networking community recognizes that there is a broad spectrum of testbed activities that can play an important role in advancing systems and experimental research in networking. Testbeds can range widely in size, scope and geographic extent. They can serve a variety of specific purposes and emphasize a range of different research concerns. While we view this variety as a strength, reflecting the vitality of the networking research community, it can be bewildering to policy-makers seeking to determine the most promising opportunities for support. This section seeks to put the various types of testbed activities in perspective, and to highlight several emerging types of testbeds that hold special promise for contributing to on-going advances in the field.

3.1 Understanding the objective There are two broad classes of testbed activities that are useful to distinguish. While both types of activities have important roles to play, their objectives are quite distinct and the differences should be clearly understood, to avoid confusion that could compromise the effectiveness of both particular testbeds and a larger testbed program.

• Multi-user, experimental facilities (MXF). Testbeds of this type are designed and operated to provide a service to a user community of networking researchers. They provide well-supported infrastructure and tools to enable researchers to experiment with new network technologies, protocols, architectures and services. A successful MXF requires a strong professional staff, providing tools, documentation, training and other assistance to testbed users. It should cultivate interaction among its users and elicit feedback directed toward improving the services provided. MXFs may also include a significant development component, since many of the tools needed to effectively deliver services to research users require on-going development, in order to keep pace with the community’s advancing research agenda. Some MXFs may be so innovative that developing them involves substantial research and associated risks. Just as in the hard sciences, constructing a sophisticated facility or instrument can be a worthy object of research itself.

A large-scale MXF can have a transformative effect on a field. An example is the effect that the MOSIS (Metal Oxide Silicon Implementation Service) facility had on the VLSI community [Conway 1999]:

“Prior to MOSIS (and the original MPC service), academic researchers had few economical ways of implementing and testing new semiconductor designs, few universities could afford their own fabrication lines, and the proliferation of different commercial systems of rules for specifying semiconductor circuit designs--most of which were kept proprietary--made collaboration between universities and industry difficult. With MOSIS, researchers could submit designs for fabrication in a standardized format through the ARPANET or, subsequently, e-mail. … This system obviated the need for direct access to a fabrication line or for dealing with the complexity of arranging fabrication time at an industrial facility, by providing access to a qualified group of fabrication facilities through a single interface. … Prominent VLSI researcher Charles Seitz commented that MOSIS represented the first period since the pioneering work of


Eckert and Mauchley on the ENIAC in the late 1940s that universities and small companies had access to state-of-the-art digital technology.”

• Proof-of-concept testbed (PCT). The purpose of a proof-of-concept testbed is to advance a particular research objective, with an emphasis on demonstrating research ideas in a compelling way, in order to accelerate technology transfer and commercialization. The value of PCTs is embodied primarily in the knowledge derived during the research and development process that culminates in the construction of the testbed and the subsequent demonstrations of its use. PCTs can play a crucial role in transitioning research ideas from the laboratory to commercial practice, and are an important component of networking research, which places a high value on its continuing contributions to society at large. While there can be a strong temptation to maintain a PCT as an on-going resource following completion, many PCT efforts have served their purpose once the demonstration and evaluation period is complete. With rare exceptions, the research community is best served, not by maintaining such testbeds beyond their time, but by taking the lessons learned and applying them to advance the research agenda to the next level.

It should be understood that the missions of MXFs and PCTs are quite distinct. One is dedicated to providing a service to a larger group of research users with many different research objectives, while the other seeks to advance a specific research objective. Evaluation of either existing or prospective testbeds should be done with a conscious recognition of which type they are, so that appropriate criteria for evaluation can be applied.

3.2 Meeting the Needs of Networking Researchers: traditional and newer forms of research testbeds

The infrastructure of many traditional network research testbeds (including DARTnet, MAGIC, and the NGI testbeds, all of which are discussed in section 4.1) has consisted of wide-area links, routers, and end-systems that were all under control of the network researcher. In these cases, the form of the research testbed network mirrored the form of a traditional wide-area network. The crucial differences were that research protocols and cutting-edge technologies were deployed in the research testbeds, and the testbeds could be configured, measured and experimented with in ways that could be completely disruptive to network users. Recently, several new and particularly promising types of testbed activities have emerged to meet the needs of networking researchers and complement the more traditional forms of network research testbeds. These developments represent a very positive and creative, grass-roots

FINDING 1: Classes of research testbeds. Two broad classes of network testbeds can be distinguished. Multi-user experimental facilities (MXF) and Proof-of-concept testbeds (PCT) both play an important role in an overall networking research program, but serve quite different purposes. It is important to understand and recognize these differences to ensure an appropriate balance between these two classes in an overall testbed program.


response to long-felt needs within the networking research community. As such, these efforts deserve recognition and strong support from NSF, as it looks for new ways to provide the infrastructure needed to support a vibrant and innovative research program in networking and communications.

• Cluster testbeds are an emerging class of experimental research facilities that are based on the concept of network emulation. Such testbeds bring together a large number of network components (links, switches, PCs that are interchangeable as hosts, routers, or WAN emulators), in a common facility that can be remotely accessed by users through a web interface. They support flexible configuration, so that users may perform experiments on a variety of distinct network topologies with programmable delays that emulate the latencies of wide-area networks. Through virtualization and space-sharing that fully isolates simultaneous experiments, they efficiently use cluster resources. They use open, extensible platforms so that researchers can study the behavior of individual network components in detail, and modify those components to experiment with new algorithms and protocols (e.g., replace packet classifiers, scheduling mechanisms, routing algorithms, etc.). Cluster testbeds offer a much greater degree of realism than simulation. While existing clusters are designed to support research in “wired networking,” the concept can be usefully extended to support research in wireless, mobile, and sensor networks as well [White 2002b]. Because cluster testbeds centralize the infrastructure support required for a network laboratory, they relieve individual researchers of both the expense and labor of maintaining comparable facilities within their own labs. This allows researchers to concentrate on what they do best, enabling them to produce more and better research. An example of a cluster testbed is the Emulab component of Netbed [White 2002a].

• Overlay testbeds have a long history in networking research. Indeed, the early Internet was essentially an overlay on the telephone network. More recent overlay testbeds include the MBone [Macedonia 1994] (a multicast overlay network over the public Internet), the ABone [Berson 2002] (an active network overlay over the public Internet), and three generic application-level overlay infrastructures over the public Internet: MIT’s manually-managed “RON” testbed [Andersen 2001], the wide-area component of Netbed [White 2002a] (integrating RON nodes and others under Netbed’s management), and Planetlab [Peterson 2002], which has the dual long-term goal of becoming a deployment platform. The overlay strategy is widely recognized as the most effective general method for introducing new protocols and services, and has been adopted by companies seeking to deliver network services that fall outside the scope of the IP protocol suite. Typically, the nodes in an overlay testbed are commodity computers with networking code operating at the application layer. Recent research overlay testbed initiatives are seeking to deploy hundreds or thousands of nodes at many hundreds of different sites in the Internet. Such an experimental facility would allow experimentation with new types of content distribution networks as well as new core network services (e.g., reliable multicast). Unlike clusters, overlay testbeds allow experimental delivery of new services to end users. At the same time, they can be (and often are) constrained by the limitations (e.g., bandwidth, delay) of the underlying Internet service that connects the overlay nodes.


• Federated testbeds can be created by interconnecting several independent, locally managed research testbeds. “Federation” allows researchers to carry out larger scale experiments than they could do otherwise, using the Internet to provide connectivity among the participating sites. For example, a set of institutions engaged in wireless network research might link their local testbeds to explore broader research issues than would be possible if working with a single testbed. Users can test their network protocols and applications on remote testbeds and can use “virtualization” to effectively merge individual testbeds (often employing heterogeneous technologies) into an apparently seamless whole to permit larger scale experiments. The interconnection of testbeds can also provide a wide variety of hardware to choose from, allowing researchers to match their needs much more closely than would be feasible if limited to an individual facility. Federated testbeds also promote cost efficiency - some testbeds can sometimes be idle, while others will not have adequate resources to meet the needs of their users. When federated, usage can be evenly spread across facilities, and inefficiencies removed.

• Networking research kits are collections of software and hardware components that can be used by networking researchers to set up their own local network laboratories. Kits can be of two types: reusable testbed management software, or prototypes of research network components. The former can make testbed use far more efficient, especially when the two types are coupled. By providing a larger research community with copies of research prototypes, kits can often provide a time- and cost-efficient means for allowing a larger community to investigate issues that would otherwise be infeasible for them to explore. Kits also provide an important evaluation mechanism to those groups developing experimental prototypes, allowing them to learn from the experience of research users and apply those lessons to their on-going research efforts. Also, the use of a common, kits-based research platform by different research projects facilitates comparison and validation of results across projects and can help stimulate collaborations among groups with complementary strengths. Kits also enable experimentation with high performance, QOS-sensitive applications, something which is more difficult to do in clusters and overlays. The ATM switch kits distributed by Washington University [Turner 1999] and Intel’s IXP1200 Network Processor Kits [Intel 2002] are two such examples. One workshop participant suggested that a kit consisting of a number of radio-controlled model cars with limited-range wireless networks (e.g., Bluetooth) could provide a useful vehicle for exploring issues of wireless networking and coordinated control. In Recommendation 2 of this report, we note that it is primarily the development of the kits themselves (as opposed to the use of the kits by researchers) that workshop participants felt should be funded in a network research testbed program.

FINDING 2: New research testbed structures. The development of cluster testbeds, overlay testbeds, federated testbeds and network research kits by the networking research community constitutes an innovative response to long-felt needs within the community. These approaches show considerable promise and are worthy of vigorous support.


Research clusters, overlays, federates and kits represent innovative community-driven responses to the perceived needs of networking researchers. All but federates have working prototypes and are of proven value; federation efforts are at an earlier stage. While the current efforts in all four areas are promising, they operate on a fairly limited scale, with typically just enough funding to demonstrate their potential, but not enough to enable the long term commitments needed to fully realize that potential.

3.3 The importance of virtualization Virtualization has emerged as an important technique for networking research testbeds. In this context, virtualization refers to a variety of methods that have been developed to cope with different issues in the construction and use of networking testbeds. Several examples are provided here to illustrate how virtualization is emerging as an important tool for research testbeds. First, cluster testbeds have been designed with virtual links, which can be configured in arbitrary topologies and with arbitrary bandwidth, delay and error characteristics. This can be accomplished using configuration switches and programmable link emulators (such as Dummynet [Rizzo 1998] and Modelnet [Vahdat 2002]). Cluster testbeds also provide artificial traffic sources to create loads on network routers and links. Some sources are programmable traffic generators that create traffic loads based on stochastic traffic models or packet traces. Others may involve real application components (e.g., web servers), allowing researchers to study the interactions among applications, transport protocols and network components. Network simulation (passing real network packets through a simulator [Fall 1999]) can also be viewed as a form of virtualization, which can be used to complement experiments in a physical testbed. Advanced testbeds virtualize the entire network environment, essentially providing a “virtual machine for network experimentation,” and can transparently combine simulated, emulated, and “real Internet” resources [White 2002a]. Virtualization of a testbed infrastructure was the central architectural component of MORPHnet [Aiken 1997], which provided support for multiple, concurrent, multi-layer views of the network for the applications and the network researchers, while accommodating the sometimes conflicting requirements of both and ensuring one type of traffic does not adversely affect the other, i.e., production networks versus network research. Virtualization has also been used in overlay networks (ABone, Planetlab, and Netbed) to allow a single physical resource (a router, for example) to be shared by multiple researchers. This is similar to the classical virtual machine concept in operating systems. In this context, virtualization isolates each research user from the others sharing the physical resource, allowing users to perform their own experiments without interference. While there are limitations to this approach in the overlay context (it does not easily support applications that have high bandwidth or that require consistent quality-of-service), it does provide a way for a common testbed to support certain types of networking experiments, while also supporting real users and applications.

3.4 Spanning multiple layers While many of the testbeds and testbed types discussed above may place an emphasis on particular layers in the network stack (e.g., network layer in DARTnet, physical layer in MONET), an important characteristic of a testbed can also be its ability to expose and facilitate investigation of interactions that occur across layers. This is particularly true in the wireless and mobile networking environment, where there is a growing emphasis on the design of “adaptive”


application and transport protocols that can adjust to changes in connectivity and in delivered throughput. Work in this area requires careful, joint experimentation at several layers of the protocol stack, ranging from radio properties (e.g., robustness to multi-path fading) to selection of different video encoding schemes. While prominent in wireless systems, the spanning of multiple layers is present also in optical networks, where for example the routing of wavelengths at the physical layer in a WDM network is correlated and optimized in parallel with the routing at the IP layer in the Internet, and in fact it may be driven by high bandwidth applications (e.g., iGRID). We thus foresee that many future testbeds will incorporate cross-layer protocol development and testing, with different teams having complementary expertise contributing at different layers.

3.5 Enabling technology transfer As discussed above, networking researchers need research infrastructure that is open to experimentation at all levels. Since many of the experimental activities central to networking research are inconsistent with the delivery of production network services, it is necessary to limit the role of users and applications in networking research testbeds to only those that are willing to accepts such constraints and realities. However, the networking research community also needs ways to transition research ideas into commercial practice and user/application oriented testbeds (UAT) may provide a vehicle to facilitate this transition. In some sense, there is a natural progression from the idea stage, to experimental verification in a networking testbed, to validation in UATs to deployment in commercial networks. If the larger research community (including both networking and distributed applications researchers) can find ways to enable this progression to occur, it could greatly benefit all concerned and society at large. There are significant obstacles to realizing this objective, which should not be under-estimated. Most recent UATs have been built using commercial network platforms (routers and other equipment). The closed nature of these platforms can make them unsuitable as vehicles for networking research. For UATs to serve as technology transfer vehicles for networking research, there must be a way to implement new network capabilities within them. This currently requires the active participation of router vendors, who often have little motivation to implement new capabilities for which they cannot perceive an immediate market. The demonstration of new capabilities in UATs may be the essential step needed to convince router vendors and network service providers of the value of new capabilities, but it is this step which is inhibited by the current reliance on commercial products in UATs. The key to overcoming this problem is to include open network platforms in UATs. UAT operators would have to be convinced that such platforms could offer consistent, reliable production service before they would consent to include them in their networks. There are several projects currently ongoing in the networking research community that could be brought together to provide such a platform and demonstrate its ability to serve the needs of UATs. A collaborative effort to implement an open network platform for deployment within UATs could be a powerful stimulus for innovation in networking research and its interaction with higher level applications.


An alternative way to address this issue is to convert selected proof-of-concept testbeds (PCT) into UATs. A model for this approach is provided by the transition of the MONET testbed into ATDnet. For PCTs that provide an important new capability and for which the research products are suitable to support production services, this can be an effective pathway for transitioning a research technology into a larger demonstration environment that allows the technology to be validated by experience with real users and applications.

FINDING 3: Moving research results into practice. While the networking community needs its own research testbeds to support systems and experimental research, it also needs vehicles for moving research results into commercial practice. User/application oriented networks can play an important role here, if the various stakeholders can find ways to open up these networks to research technologies, without compromising the service expectations of their users.


4. The Past is Prelude: Past Experience with Testbeds Network research testbeds have played a pivotal role in the development of today's Internet hardware and software technology. To lay a foundation for recommendations to the NSF about network testbeds, the workshop began by examining the history and experience from past testbeds. The results are summarized in this section. The ARPANET was the ancestor of all network testbeds. The ARPAnet was originally an experimental platform to demonstrate the feasibility of packet-switching and to develop prototype software for packet switches and for end hosts. The evolution of the ARPAnet testbed, from a leading-edge research project that could have failed to a stable production facility that could permit only limited research, illustrates many of the issues faced by network testbeds. For this study, we considered a sample of more recent network research testbeds: DARTnet, the MAGIC testbed, the NGI testbeds, the ABone, and Emulab. The first three have accomplished their purposes and been dismantled; the last two are still in operation. These were chosen to cover a wide range of testbed types, in the hopes of obtaining a sufficiently representative sample. Each of these testbeds is summarized very briefly below, followed by a discussion of issues contributing to testbed failures. Building on this discussion, Section 5 identifies and discusses issues important to network research testbeds. 4.1. DARTnet DARTnet (DARPA Research Testbed network) was a DARPA-funded network research testbed for the DARPA Internet research community. DARTnet was a wide-area experimental environment for research on the network layer and transport layer protocols and on advanced applications that would drive this research. The primary advanced application turned out to be packet audio and video, especially video teleconferencing. The construction of DARTnet in 1991 was necessitated by the commercial success of the Internet, which no longer permitted disruptive experimentation. DARTnet was designed to allow experimentation that could be completely disruptive to the network infrastructure. DARTnet produced, directly or indirectly, many important research results. These included: IP multicasting (which became the MBone), packet scheduling algorithms (including CBQ), resource reservation protocols (ST-II and RSVP), video teleconferencing applications (the “MBone tools”, including vat, vic, and sdr), network time keeping, and experimental tools (e.g., mtrace and traffic generation tools). Productive use of DARTnet continued until its termination around 1996.

FINDING 4: Successful past research testbed efforts. Network testbeds have a proven track record of providing the means by which important research advances in our community could be tested, stressed, observed, refined, and ultimately proven.


DARTnet included 15 sites spanning the US continent, linked by T1 (1.5 Mbps) lines. It used an open router platform, Sun Sparcstations running SunOS, which was Sun's variant of the BSD Unix system. DARTnet also included experimental hosts that mostly used the same hardware and software platforms. DARTnet users formed a research community with a culture of collaboration. Although this culture pre-dated the formation of DARTnet, it was undoubtedly fostered and sharpened by the distributed DARTnet testbed. Installing a DARTnet router made each site a part of the research community. Periodic video teleconferences and frequent audio consultation among the participants furthered personal communication and collaboration. A key research result of DARTnet was an evolving set of kernel modifications that the research community developed collaboratively. 4.2. MAGIC The Gigabit testbed program conducted in the 1990's explored technologies and architectures for gigabit networking, while investigating the utility of gigabit networks to the end user. Thus each testbed effort had an applications focus. For example, the Multidimensional Applications and Gigabit Internetwork Consortium (MAGIC) gigabit project [Fuller 1996] developed and deployed a network and associated application that could demonstrate real-time, interactive exchange of data at gigabit-per-second rates among multiple distributed resources. The MAGIC testbed involved academia (the University of Kansas and the Minnesota Super Computer Center), National Laboratories (Lawrence Berkley Labs) and SRI, MITRE, and CNRI. The effort was supported by DARPA from 1992-1999. Industry (Sprint and Southwestern Bell) provided infrastructure and participated in the research. The MAGIC project included close working collaborations among its participants, led by a coordinating organization. MAGIC used a single compelling application, terrain visualization, that required massive amounts of remotely stored data. A distributed image server system was developed, and a high-speed internetwork was used to link the computing resources required for real-time rendering of the terrain. Initially the testbed included five sites directly connected to the OC-48 SONET backbone; later the distributed application was deployed across a variety of high performance networks. The effort successfully produced new tools for understanding high performance distributed applications, a distributed data server system, and a network-based terrain visualization application. MAGIC’s focus was thus primarily on applications. However, it also provided insights into the many aspects of cutting edge networking technology at the time, such as SONET and TCP-over-ATM networking. . Presentation of distributed terrain visualization to users and others provided persuasive evidence of the value of the overall the high-performance distributed system, including the network. 4.3 NGI Testbeds From 1994 -2002 the DARPA Broadband Information Technology (BIT) and Next Generation Internet (NGI) programs resulted in the creation of several networking testbeds. These testbeds had multiple purposes, ranging from experiments in optical technologies to testing protocols including control and management functions, to deployment and evaluation of prototype


hardware, and the demonstration of bandwidth intensive applications with potential national security impact. These programs included several interconnected and interoperating testbeds that enabled testing and application of end-to-end Gigabit/sec infrastructures. Each testbed effort had its own management structure, often in the form of a consortium that included regular coordination among the participants. However, experiment coordination and facilitation was provided by ISI. The testbeds included:

• Multiwavelength Optical NETworking Consortium (MONET) [MONET 2002] • Advanced Technology Demonstration Network (ATDNET) [ATDnet 2002] • Optical Network for Regional Access with Multi-wavelength Protocols (ONRAMP)

Consortium • BoSton-South NETwork (BOSSNET) • National Transparent Optical Network Consortium (NTON) • High Speed Connectivity Consortium (HSCC)

Many of these testbeds considered lower layer networking issues, e.g., control and management of optical networks, optical burst switching, and physical transmission. Since this effort involved the development and deployment of prototype network hardware, significant time was allocated to devising the architecture, designing and implementing new networking concepts. In this case the first 24 months were devoted to architecture, design, and building the prototype hardware. The time dedicated to develop a coherent network architecture was an important element to the success of these testbeds. In general, testbeds that require managing or deploying new physical layer network elements demand significant planning and additional time, often longer than expected. A number of technologies were successfully developed and deployed including, IP/WDM networking, re-configurable all-optical WDM network technology, dynamic optical WDM systems, optical burst switching, and interoperability. However the program was also interested in distributed applications requiring high bandwidth. The testbeds supported a number of demonstrations, e.g., wideband sensor network experiments involving the coherent processing of radar signals to emulate wideband radars, stereoscopic rendered images and video streaming with real-time compression methods and telepresence in the operating room utilizing IP video. Compelling applications on testbed networks do not come for free. In the case of the NGI testbeds, investments were needed to engage the appropriate set of applications researchers. Participants in the NGI testbeds included a number of companies and universities, as well as government agencies. Bringing together these communities, e.g., researchers with telecommunications and computer science backgrounds, was of value. The chemistry among the researchers was an important ingredient in the success of these testbeds. 4.4. The ABone The ABone (Active networks Backbone) [Braden 2002] testbed attempted to apply the DARTnet model of wide-area collaboration to DARPA's active networks research program. The ABone became operational in 2000, and it is still in operation although the DARPA active networks research program has nearly completed.


The ABone is composed of 50-80 active nodes supplied by research sites, connected using overlay networks (usually UDP tunnels) across the commodity Internet. Nodes may run Linux, FreeBSD, or Solaris. Active networking in the ABone is done in user space, mostly using portable Java code. The ABone software provides a remote daemon process on each node that allows researchers to fork and monitor Java execution processes remotely. The ABone was successfully constructed and demonstrated, and several quite complex active applications have been demonstrated using it. However, the ABone did not realize its potential for fostering research collaboration or for convincing deployment experience with real active applications. ABone development came relatively late in the 5-year program cycle for active networking, so researchers had made commitments to local testing. On the other hand, the ABone was in a real sense too early, since the active network research program had not yet progressed to the point of collaborative construction of a common technology, for which a wide-area testbed would have been invaluable. 4.5. Emulab Emulab [White 2002a] is a highly successful example of a cluster testbed, i.e., a set of network nodes and configurable links within one laboratory, which can be accessed remotely by experimenters over the Web. The current Emulab at the University of Utah has a cluster of 168 PC nodes, flexibly interconnected by Ethernet links using switch VLANs. A sophisticated, convenient, and familiar interface (an “ns” script or a GUI) allows a remote experimenter to allocate a subset of the nodes, interconnect them with a chosen link topology, load software into the nodes, and run an experiment. The user is able to replace all node software, but each experiment is completely isolated from other concurrent experiments on the cluster. The user is provided with a rich set of tools for control. In contrast, few special measurement tools are provided; since experimenters have “root” privileges, the familiar tools are available, and better tools have not yet been a high demand. Analogous to an OS process, entire experiments can be “swapped” in and out to arbitrary nodes, providing efficient use of cluster resources through time-sharing. It can be used as a network research testbed, but it is also useful for distributed system and OS research as well. It has already proven to be popular and very productive in terms of publishable research results. Early and incremental deployment was crucial to Emulab’s success. Low-budget development started in 1999, with a 10-node cluster and then a 40-node cluster being made publicly available by October 2000. The deployed management software took a quantum jump in sophistication in March 2001. Throughout this time, Emulab was kept near capacity by external research users. An important lesson was to quickly get something running to attract users and get feedback, and not hold up deployment for ideal solutions to administrative, policy, or technical issues. Problems were solved (e.g., problems arising from increased scale) as they arose. Users were considered central to the process: they generate feedback, resources, and more users. Emulab’s focus on interactive ease-of-use reflected the centrality of users. First-class operational support was always required, as users are always naive, hardware always goes bad, and the software is constantly evolving.


It turns out that the bulk of the software underlying Emulab (120,000 lines of code) is general enough that it can be extended to handle diverse kinds of testbeds. Emulab has recently been extended into “Netbed,” which provides integrated support for simulated and wide-area nodes and links. Several other universities and labs have built, are building, or are planning to build testbeds using the Utah software. Future challenges includes componentizing the software, pursuing federation, and decreasing operational load by improving reliability an order of magnitude. Despite the extremely positive community response, obtaining ongoing funding has been surprisingly difficult. Major funding only started in October 2000 with an NSF grant; Cisco hardware donations were also crucial. Since building Emulab has large elements of both research and infrastructure, reviewers of each type of program often found the presence of the other component to be problematic—even if funding for that component was not requested. 4.6 Testbed Failures Testbed participants also discussed how and why testbeds fail. Failure, in this case, is defined from the research perspective: namely, did the testbed produce insights or innovations (including those that come from outstanding technical work that produced disappointing results) and that informed the larger technical community. A few general themes appeared. First, the organization and operational aspects of running a testbed can overwhelm the research. There are cases where getting the testbed running absorbed the vast majority of the testbed funding, such that very little research was actually done. Testbeds which are overambitious in their goals are most vulnerable. Testbeds that seek to connect many sites over a wide geographic area often find the logistics of connecting sites to be difficult. Testbeds which seek to stretch the envelope of performance (especially testbeds that seek to go fast for the prestige of being fast, without a clear research need for high performance) are often absorbed by the tremendous challenges of getting cutting edge equipment simply to work, and are often undertaken without the appropriate number and type of personnel. A particularly pernicious form of this failure comes when a testbed accepts donated technology (on the grounds it is “free”) only to discover that the costs of actually operating and using the technology predominate the effort. Another way in which testbeds can fail is if they do not effectively disseminate results through publication, software distribution, or other means. Sometimes testbed teams simply fail to publish. In other cases, their most interesting results become encumbered with intellectual property problems. For instance, researchers may be unable to distribute innovative software because it includes licensed technology from a company that licensed the technology only to the testbed participants. There are also cases where testbeds exhibit

FINDING 5: Unsuccessful past research testbed efforts. Network research testbeds can fail (in the sense of not producing research insights or innovations that inform the community) for many reasons, including being overwhelmed by the operational aspects of building and maintaining the testbed, failure to publish or distribute results and/or software, and lack of focus on the research goals of the testbed.


confusion about their mission. One of the topics that generated the most discussion at the workshop (see section 5.2) is the problem of trying to combine infrastructure for users with infrastructure for networking research. Other pitfalls include building wide-area testbeds when local area testbeds would be more than sufficient, and continuing to operate a testbed network after its primary research objectives have been achieved. As a community, we need to ensure that testbeds are constructed and operated in a fashion consistent with their research objectives. Finally, we should mention a kind of “failure” that is acceptable. Because the essence of testbeds is research, it is always a possibility that the testbed will fail to achieve its research objectives for technical reasons. (Indeed, by some definitions of research if the testbed effort cannot fail, it is not research). We should never lose sight of the fact that the failure of technology is a learning experience.


5. Important Testbed Characteristics

Having described a number of past and on-going research testbed efforts, we now turn our attention to the future. What lessons have we learned in the past that can inform future testbed efforts and maximize their probability of success? What are the important characteristics that one is likely to find in successful testbeds? What issues are future testbed developers likely to face, and how can these challenges be met? This section discusses nine important issues (in sections 5.1 through 5.9) that will confront future testbed developers and their research users. 5.1 Network Research Testbeds: Driven by a Networking Research Agenda Workshop participants noted that the most important characteristic of a network research testbed is perhaps also the most obvious: that network research testbeds need to be driven by research coming from the networking community. Research is the raison d’etre of network research testbeds, and thus all aspects of testbed structure and operation should be developed with this primary goal in mind. While this characteristic may seem self-evident, workshop participants noted that some networks that were developed, ostensibly with the goal of supporting networking research, were not well-suited for networking research, because they were simultaneously attempting to satisfy other, conflicting objectives. For example, as discussed below in Section 5.2, a network that needs to provide production-quality service to some users, will often find it impossible to support the potentially disruptive experimentation that is a central part of networking research. 5.2 The Role of Users and Applications Since communication networks exist to enable users to communicate more effectively with each other and with remote computing and information resources, it is natural to expect that networking research testbeds would include real users. The presence of real users and the applications that serve them can be important to networking researchers. Applications help drive requirements for new network services, they can play an important role in their evaluation, and are a tangible way to demonstrate the capabilities of a testbed. However, the presence of real users (with their own research goals and expectations of reliable service) in a research testbed can also inhibit the networking research agenda. Application users require a stable platform and application writers require stable interfaces. This requirement is often at odds with a researcher’s need to experiment with, modify and replace core network components (possibly disrupting the network in the process of debugging and experimentation), and to define new interfaces. Such activities, which are central to networking research, are not consistent with users’ desire for reliable, production-quality network service. Experimental networks that have sought to provide production-quality services to end users (e.g., vBNS and Internet-2), have effectively closed the

FINDING 6: Network research testbeds - driven by a research agenda and meeting networking researcher needs. The most important characteristic of a network research testbed is perhaps also the most self-evident: that network research testbeds need to be driven by research coming from the networking community.


door on the vast majority of interesting networking research, and indeed do not fill the needed role of network research testbeds. There is a place for application-driven testbeds, but they do not serve the same purpose as network testbeds.

Historically, network research testbeds have tended to work best when the role of applications has been more limited, with an emphasis on demonstration rather than production use (as in DARTnet, for example). Demonstration applications can play a useful role in shaping and sharpening the requirements for network services, without inhibiting the kind of experimentation that is central to advancing the research agenda. A small user population can also be useful, as long as there are no expectations on the users’ part for consistent, production-quality services. Some networking testbeds (e.g., Emulab and the ABone) function as service facilities for a broad class of networking research, and thus (appropriately) have no specific application focus. By adopting an “application-neutral” stance, such testbeds can host an extremely wide variety of networking research as well as promote the development of networking technologies well suited to the unpredictable future evolution of applications. Testbed directions should be informed by the community's judgment on the most important research challenges and application classes, but it would be a mistake to be constrained by them. It should be noted that there are drawbacks in limiting the role of users and applications in networking research testbeds. First, it limits the usefulness of such testbeds as sources of data on the behavior of “real” users and the behavior of networks subjected to user-driven traffic, since only users who can tolerate disruption in the underlying network infrastructure can be accommodated. Second, application developers are unlikely to invest the considerable time and effort needed to produce high quality, user-oriented applications if there are no users to take advantage of them. This means that applications developed for such testbeds may have to be developed by networking researchers (or in collaboration with the users that will use them) and consequently may be relatively unpolished, suitable primarily for demonstrating new capabilities. While these are legitimate concerns, the networking research community has generally concluded that it is better to accept these limitations than to constrain the kinds of experimentation that networking researchers are able to purse. The tension between applications and the research use of testbeds has been noted by the networking community before, e.g., in the 1997 report of the NSF Scalable Information Infrastructure Workshop [Partridge 1997].

FINDING 7: The tension between application users and networking research. Network testbeds that focus on users and applications are typically not the best vehicles for supporting networking research, since users’ expectations for reliable, production network service are incompatible with networking researchers’ need to experiment with and modify network infrastructure components.


5.3 Geographic Extent and Bandwidth Considerations Given that the purpose of communication networks is to enable communication over distance, it may seem self-evident that network testbeds should span significant geographic distances. Paradoxically perhaps, geographically distributed testbeds do not always best serve the needs of networking researchers. There are several reasons for this. First, testbeds that span large geographic distances typically do so in order to link together remote users, or to link users to specific networked resources. For the reasons discussed above, networks that serve user populations with high expectations for production service, cannot also support the kind of disruptive experimentation that is central to so much of networking research. Second, geographically distributed testbeds tend to reach only a relatively small number of locations, due to the high cost of providing dedicated transmission bandwidth. Overlay networks, however, use commodity Internet services for connectivity and thus can be economically deployed to a much larger number of locations than can testbeds that require dedicated wide-area links. Third, geographic distribution often means distributed management, with different organizations being responsible for equipment at different locations. This often introduces artificial complexities that can make experimentation more time-consuming. This in turn, discourages researchers from using such testbeds, since the payoff in experimental results may not be worth the time and effort required to obtain them. Recent work on wireless and sensor networking (see Section 6: Research Opportunities) also does not require geographically distributed testbeds, since radios and physical sensing both have limited range. While these kinds of testbeds cannot be physically distributed, they can still share in the benefits of distributed testbeds by involving researchers from many institutions. Wireless testbeds can benefit from remote access in many of the same ways as cluster testbeds do today, although additional work is needed to understand how to allow remote users to incorporate motion and sensing [White 2002b]. In addition, in many ways, computers are inexpensive enough today that network testbed kits can enable many small, distributed wireless testbeds at different institutions. Even if not network-connected, such a distributed testbed would share the cost of designing testbed nodes and benefit from building a community around a common hardware and software platform. Historically, the issue of bandwidth has often gone hand-in-hand with the issue of geographic scope, since dedicated high-bandwidth links between geographically distributed sites have often been an important (and expensive) aspect

FINDING 8. Geographic Distance. Testbed networks need not span large geographic distances to serve as valuable resources for networking researchers. Indeed geographic distribution can make certain kinds of experiments more difficult to execute and can act as a disincentive to performing such experiments. Local area and cluster testbeds can be highly effective research vehicles in spite of their limited geographic extent.

FINDING 9: The role of bandwidth. Dedicated, high-bandwidth wide-area links are not a necessary prerequisite for creating useful networking testbeds. Emulation of wide-area links in local labs represents a cost-effective alternative that should be considered when high-speed links are a necessary testbed component.


of past geographically distributed testbeds (such as the Gigabit testbeds). Workshop participants note that much interesting and important networking research has been conducted on testbeds that do not include high bandwidth links. DARTnet, for example, used T1 lines to interconnect geographically dispersed sites, and focused on problems such as multicast routing and congestion control (where the ease of congesting the testbed was a feature). In cases where high bandwidth is important to the problems of interest, the ability to emulate wide-area links in local labs has reduced (though not eliminated) the need to invest in testbeds that contain high-speed wide-area links. 5.4 Testbed Management, Software and Tools The operation and management of testbeds, especially those that provide a service facility intended for use by a large and diverse research population, requires careful planning and considerable personnel resources.

Experience shows that people costs vastly exceed the hardware costs of most testbeds that have high research output. The people costs are for software and hardware development - and often research - of the testbed infrastructure itself, as well as for ongoing operational support. Using the testbed is also labor-intensive, since experimenters typically need special control and monitoring, and because the testbed is often unstable. In addition to supporting disruptive experimentation, perhaps the most important characteristic of a research testbed is the toolset needed to build, manage, monitor and control it. The existence of these tools and their ease-of-use is an extremely strong determinant of success for testbeds, with many associated implications. Perhaps most important, ease-of-use implies significant investment in professional staff to develop tools and provide continuity of support. However, real breakthroughs in ease-of-use and efficiency require imagination, insight, and detailed knowledge of the network research process. For these reasons, researchers themselves need to drive and guide this process.

Investment in both innovative and mundane testbed management software and tools pays rich dividends in numerous ways: 1) improving experimenter productivity, 2) improving the efficiency of testbed operations, 3) improving the utilization of often scarce testbed hardware resources, 4) enabling qualitatively new methods of experimentation, and 5) fostering a shallow learning curve, allowing new testbed users to “come up to speed” rapidly. These advantages yield tremendous cost efficiencies when appropriate investment is made in testbed management software. For example, Emulab, a cluster testbed that emphasizes rich management software, has demonstrated two orders of magnitude improvement in experimenter efficiency over manual experimentation, and further order of magnitude improvements in utilization. Emulab users perform kinds of experiments that were not previously possible, such as large-scale parameter-space explorations and “what-if” experiments.

FINDING 10: The importance (and cost) of operations and management. The operation and management of testbeds requires careful planning. The “people costs” for testbed software and hardware development, and for ongoing operational support can often significantly exceed the hardware costs.


The networking research kits (collections of software and hardware components that can be used by networking researchers to set up their own local network laboratories) discussed in section 3.2 are another instance where high-quality testbed management software (in this case, testbed management software accompanying the networking research kits) could significantly improve the experimental setting and dramatically enhance research productivity.

Testbed software for controlled experimentation can also greatly facilitate the researcher’s task. Such software can allow deterministic variation of experimental conditions, including traffic load, background traffic, and the relative times at which certain events such as link failures occur. Such controlled behavior is important for three reasons:

• It allows individual researchers to gain a better understanding of their own systems since it allows fine control over individual parameters.

• It allows research groups to validate results obtained by other groups and more easily experiment with alternative strategies.

• It allows the experimental prototypes to be subjected to extremes, testing their behavior under corner conditions, where experience shows many robustness problems arise.

We note that different experiments require, and different network technologies permit, vastly different degrees of determinism. Deterministic variation of all parameters is clearly not an option for some testbeds, such as overlay testbeds that utilize the public Internet for wide-area connectivity. Wireless testbeds also have imperfect repeatability due to difficult-to-control RF characteristics, or difficult-to-repeat mobility patterns. Nonetheless, the overall principle of controllable experimental conditions remains important for networking testbeds. Confidence in results can be gained through statistical techniques and multiple trials. Even though such experiments are not entirely repeatable, we recognize them to be vitally important for successful experimental networking research. 5.5 The Importance of Timing Timing plays a critical role in the success of testbeds. Testbeds that provide a service facility (i.e., multi-user experimental facilities) must exist long enough to allow for both the development of the testbed and the use of the testbed by researchers. The ABone testbed, for example, was not operational until the later stages of DARPA support for active network research, limiting its role as a service facility. Most active networking researchers invested time and effort in other experimentation platforms and were understandably reluctant to migrate to the ABone. One-time-prototype

FINDING 11: Importance of testbed management software and tools. Investment in testbed management software and tools is crucial, allowing higher research productivity and significant cost efficiencies in the long run.

FINDING 12: Timing is critical to success. Timing -- both start date and duration -- plays a critical role in the success of testbeds. Testbeds will require varying combinations of architectural design, development and use, which should be reflected in their duration.


testbeds may require significant time for architecture and design. For example, the MONET program and several of NGI projects spent the initial 24-36 months on architecture, design, and building of prototype hardware and software. 5.6 Measurement The impact on the networking community resulting from any testbed can be dramatically multiplied through the dissemination of measurements obtained from the testbed. Compared to the number of direct users of a testbed, the number of researchers who may make use of data collected in a testbed is many times greater. Thus, given a well-instrumented testbed, the measurement data collected during its operation may be a first-class research output in its own right. This is especially true for testbeds that are integrated into the real operational Internet (e.g., overlay testbeds) or those that support real applications. Furthermore, the adage that “Good data outlasts bad theory” [Patterson 2001] can perhaps be extended to “good data outlasts obsolete testbeds.” In a discussion of past testbed projects, several participants characterized a lack of measurement as a lost opportunity that they would not like to see repeated in future projects. In addition to the need to understand new networks and applications, a particularly urgent need is for better knowledge of the Internet itself (as discussed in Section 6.2.1). For this reason, several workshop participants felt that the lack of existing measurement data for the Internet was reason enough to consider building a testbed (or adapting a testbed, such as an overlay network testbed) for the purpose of gathering information about the live Internet. Such a testbed might be composed of a set of widely dispersed measurement points along with software for experiment setup and data analysis (factoring in lessons learned from projects like NIMI, Surveyor, and CAIDA). Given the wide range of potential testbed uses, the ability to measure a large variety of network properties is a desirable characteristic. Depending on the testbed, measurements of interest may include time series of offered load and queue length at routers, packet loss and signal-to-noise ratio on wireless links, user mobility data, routing updates and other control information, and even full packet traces taken at various points in the network. It is clearly desirable to instrument a testbed beyond the level required for its primary research mission, since some measures of interest may not be foreseen. This is particularly true for testbeds whose envisioned lifetime includes a transition to an operational engineering testbed, or even a production network. Indeed, even testbeds whose primary role is to support non-networking research (e.g., a high-bandwidth Grid interconnect) could provide significant value to network researchers if dissemination of measurement data were integrated into the testbed mission. 5.7 The Issue of Scale

FINDING 13: The need for testbed instrumentation and measurement. A lack of good measurements and models of traffic, protocols, and applications hinders the network research community. Appropriately instrumented testbeds can help solve this problem.


Testbeds face a key challenge in the area of scale. Many current challenges in network research arise from the scale of the number of network entities (end points, flows, routes, users). However, network testbeds, unless they somehow attract very large numbers of real users through fulfilling some real need, are inherently of modest scale, at least compared to the real Internet. The issue of scale also arises in the dimensions of speed, diversity of hardware/software/applications, and geographic scope, but the scaling challenges posed by large numbers of entities is the most important. There are four ways to confront and mitigate this issue.

• Work in domains that are often of more limited scale, such as wireless, mobile, and sensor nets. These domains are often inherently of sufficiently modest geographic scope that testbeds of realistic scale can be constructed. They also are relatively young as a research topic and so rife with research challenges that modest scale testbeds can make a tremendous impact. Finally, they are novel enough that real applications can be deployed and real users attracted. If large scale issues need to be explored in wireless domains (such as battlefield or disaster recovery scenarios including thousands of mobile nodes), they can (as discussed below) be efficiently attacked with hybrid simulation or more generally “virtualization”.

• Operate at the overlay level, making the overlays extensible and programmable by researchers and others, to provide services that would otherwise be unavailable. Overlays leverage the existing Internet's ubiquity to reach everywhere. Therefore, if an experimental service should become popular, overlays can often provide a transition to deployment with large numbers of real users.

• Pursue an aggressive program of federation of very large numbers of testbeds, test networks, and test machines. A single testbed is unlikely to scale to very large numbers for reasons not only economic and political, but also from limitations in flexibility and planning. However, through federation massive scaling may be possible, just as the Internet has scaled through linking multiple networks and autonomous systems. The challenges are many, but the potential rewards great.

• Make progress both in the measurement of real networks and in modeling to generate synthetic traffic that is realistic in quality and scale. Each is a big challenge, but the potential rewards merit serious effort and investment.

5.8 Open Platforms There is great value in having network testbeds whose components (hosts, routers/switches, software) are open, and whose detailed operation can be studied and modified. Open platforms foster deeper understanding of underlying mechanisms and principles. In experimental research settings, the level of access to software and hardware (in the form of source code and technical documents) often determines whether particular problems can be solved or even attempted. An important reason for the success of the DARTnet testbed was the level of access the researchers had to all layers of the SunOS Operating System software. Thus, we strongly encourage the use of open software and hardware platforms in research testbeds. The "XORP" extensible open router [Handley 2002] is a promising example of such a software platform. However, we acknowledge that certain cutting-edge components - especially hardware components - are often developed in proprietary settings, and may not be available in totally open form. Whenever possible, such components should be included in testbeds only if they are imperative for the


proposed research, or are themselves at layers that are entirely outside the scope of the testbed's research targets.

Development of a reasonably robust and flexible open router, with separate hardware, OS, and application (routing protocol) components could transform both network research and production networks. Breaking the vertically-integrated router market into three separate components would most likely spur waves of innovation, similar to what occurred in the PC industry in the 1980's. 5.9 The Role of Research Community Many successful testbeds can claim a vibrant community. In some cases, the community came together to create and use the testbed (e.g., DARTnet, NGI testbeds); in other cases, the testbed served to catalyze a collection of researchers who would otherwise work relatively independently; in other cases, collaborating researchers were initially drawn to a testbed because it provided a place to work together (e.g., Emulab). Past experience indicates that communities cannot be dictated; successful communities are comprised of people who want to work together, not those who are forced to do so. One-time-prototype testbeds may not serve a large networking research community. On the other hand, such efforts often involve a combination of industry and academic researchers (e.g., MAGIC, MONET) or are interdisciplinary, drawing participants from varied fields such as electrical engineering, computer science and application areas. 5.10 The Educational Role of Testbeds Testbeds have provided a unique educational opportunity for the students involved in their development. The chance to participate in the development of an operational network gives students a set of skills and experiences that cannot be duplicated by other activities. Some testbeds have also provided a platform for class assignments, reaching a larger group of students than those involved directly in the construction or use of the testbed. For example, the ABone was used in courses at USC. More generally, students whose research involves testbed use benefit from demonstrating their research in a “real” network and develop experimental design capabilities and hands-on skills in systems-building and experimental computer science.

FINDING 15: The important role of research community. Many successful testbeds claim a vibrant community, either pre-existing or catalyzed by the testbed itself. Testbeds can serve a special role in bringing together diverse communities (e.g., industry/government/academia, EE/CS/applications).

FINDING 16: An important educational role for testbeds. Testbeds provide a unique educational opportunity for the students involved in their development, as well as a potential platform for class projects. Students, faculty, and others involved in testbeds benefit from developing systems-building skills and other experimental research capabilities.

FINDING 14: Open hardware and software needed. The network research community needs open programmable hardware and software at all levels.


6. Research Opportunities and Benefits When considering a program on network research testbeds, the issue of the value of such testbeds will certainly arise. In this section, we first discuss the broad benefits to networking research of a vibrant testbed program. We then consider a number of specific research areas that are particularly well-positioned to benefit from a research testbed program. 6.1 A Broad Discussion of Benefits In the previous sections of this report, we have catalogued a number of testbed efforts and the fact that they play important roles at certain points in the production of valid and valuable research results. As noted in Figure 1, we typically see researchers moving from theory (the whiteboard) to algorithms and design to simulation to realization in a testbed to experimentation in a full-fledged network2. We have also seen earlier in this report that testbed environments may vary from a local testbed in a lab, to larger proof-of-concept testbeds, to more general multi-user experimental facilities. If we compare our field with other fields of science and engineering, testbeds can be compared with laboratory facilities ranging from the single lab to the large facility such as a shared wind-tunnel or an instrument at SLAC. As in these other fields of engineering and science there is significant value that accrues not only to the network research community, but also to the broader community that is either influenced by or builds on top of networking. In discussing the value of network research testbeds, one participant succinctly summarized the benefits of network research testbeds as enabling “quality research and its tech transfer into the real world of networking.” Delving more deeply into the notion of “quality,” research participants noted that testbeds provide the “real-world” environment in which simplifications needed in either theoretical analysis or simulations can be tested. A testbed environment may provide validation of the research, as well as a more realistic environment for studying the thesis of the research, than the synthetic universes of analysis or simulation. Because of the reality-testing nature of a testbed it may also lead to the unexpected or serendipitous discovery of interactions with other artifacts and activities that may have been abstracted away in a simulation environment. These are often the stimulus for previously unrecognized directions for further research. Contributions sent to the workshop committee in advance of the workshop amplified on the theme of a testbed providing the critical linkage between the idealized environment in which research ideas are first conceived and tested, and the “real-world,” where they will ultimately be proven and used.

• “I see testbeds as providing a sharper and more relevant focus for the work of the community. A lack of wide-area testbeds would contribute to a growing tendency towards paper solutions to thesis-factory problems, leaving the real networking world short of new ideas and technologies.”

2 We recognize that for a wide variety of reasons many research projects do not follow this pattern; for example,

because they do not have adequate funding or because what was learned from simulation suggested an alternative direction for research.


• “Testbeds might make clear which simplifications [in theoretical work] are valid and which fly directly in the face of reality.”

• “Without testbed research, one could expect to see more fragmented and/or theory contributions from smaller projects.”

Testbeds provide another important benefit - research results that are demonstrated in a testbed are more easily transferred into commercial settings, precisely because they have been implemented, tested, and experimented with in a non-simulated, non-paper-and-pencil environment. Testbeds thus serve to broaden the audience for research results, can provide alternative modes of explanation/demonstration for those with a “I won’t believe it until I see it implemented” worldview, and can have impact on standards communities. Participants noted:

• “Prototypes and testbeds are required to gain acceptance of new concepts with potential user communities.”

• An advantage of testbeds is … “promoting the acceptance of good research ideas into the IETF [ed.: or other standards bodies] and therefore into the vendor world.”

Workshop participants also noted that research testbeds play a crucial role in maintaining a vibrant experimental systems research community. The importance, and difficulties, of experimental computer science research have been noted earlier in [NRC Experimental 1994]. These benefits were also noted earlier in our Finding 11. Participants noted that testbeds:

• “…encourage experimental work” • “…encourage innovative experimental/systems thinking and training among university

graduates (and faculty)” • testbeds …“play an important role in stimulating and maintaining a vibrant on-going

effort in experimental/systems research in networking.” The final broad benefit of a vibrant testbed program - research community – was identified earlier as a lesson learned from successful past testbeds (Finding 7). In the words of workshop participants:

• “testbed programs are important for focusing the research community on a few broad challenges”

• a value of testbeds is “…building and maintaining research collaborations and communities”

• testbeds provide for “… collaborative interactions between communities of interest, e.g., crisis managers and the computing/networking research community.”

• “shared testbed and infrastructure, and shared research goals, help build community.” 6.2 Research Opportunities

FINDING 17: Research, reality, and technology transfer. Network research testbeds enable a critical stage in the process of understanding the relationship between the research and reality. Implementation and experimentation in a testbed also facilitate the process of transferring research results into industry and the broader community.


Network technology has reached an important milestone with the growth of the Internet during the 1990’s, made possible in large part by research testbed investments by the NSF, DARPA and other agencies. But many challenges still remain. Some of these challenges arise from the need for new network architectures (for scalable mechanisms for providing quality-of-service guarantees, for enhanced interaction between access networks and the network core), new network services (such as caching, content distribution, and streaming), and new service architectures (dynamically extensible networks, application overlay networks). Other challenges arise from the emergence of new technologies (e.g., sensor networks, pervasive computing) or the widespread deployment and new uses for current cutting edge technology (wireless and mobile computing). Other challenges, such as network measurement, are longstanding and reflect a lack of critical data with which to understand the artifacts (networks) we have built. Workshops participants also discussed research associated with access networks and the issues of peering between access and core networks. Given the limited amount of time at the workshop, participants “drilled” down deeply into only a selected number of research areas: overlay networks, sensor networks, wireless networks, security, dynamically extensible networks and measurement. These were all identified as crucial research areas in which testbed activity would be particularly valuable. The selection of these particular research areas for discussion undoubtedly reflects the research preferences of the workshop attendees; they are not meant to be exclusive, i.e., the participants noted that there is much interesting research to be accomplished in areas other than these. In the following sections, these research areas, and the impact of research testbeds in these areas are considered. 6.2.1 Network Measurement and Characterization

As the Internet has evolved, insights gained from network measurement and characterization have had considerable impact on its design --- including its protocols, components, and applications. Yet, given the natural role of workload characterization as an early part of any system design process, it’s perhaps surprising that many important network workload properties were not immediately understood or appreciated as the Internet was deployed. Examples of such after-the-fact discoveries include traffic self-similarity and the wide variety of long-tailed distributions observed in the Internet and the Web. Comparing our current knowledge of the Internet with that of other instances of large-scale social infrastructure (e.g., the telephone system, or regional power grids) highlights how little we know about the configuration, workload, and failure modes of the Internet as a whole. To avoid compounding our ignorance, it is important that we obtain a thorough understanding of the behavior of new network testbeds that we deploy. In fact, adequate understanding of the research results flowing from certain kinds of testbed efforts is impossible without accurate, pervasive instrumentation and measurement.

FINDING 18: Many research areas enabled by testbeds. Testbeds can enable research in many different areas of networking. Areas that are particularly “ripe” for testbed activity include overlay networks, sensor networks, wireless networks, security, dynamically extensible networks, and measurement.


Complete, detailed measurement of an operating network presents a number of research challenges. Issues of scalability, privacy, and methods for effective analysis of large-scale network measurements must be addressed. Making progress on these research problems should be an important collateral benefit of the deployment of new network testbeds. An important example of the kind of network measurement needed in future testbeds is traffic measurement and characterization. High quality measurement data generated by real users offers researchers the opportunity to ``close the loop'' with user models, with a potentially dramatic impact on the state-of-the-art in synthetic traffic generation. In the area of wireless networks, for example, most researchers believe that the user mobility models for wireless networks in current use do not adequately capture the behavior of real network users. However, there is little, if any, publicly available measurement data of real user behavior with which to validate existing models or develop new ones. Consequently, performance observed in simulation may not accurately reflect that of real networks. Thus, results from traffic measurements coming from new testbeds – whether exploring new networks or new applications -- can have immediate impact, by allowing a wide community of researchers to reproduce testbed properties in simulation. Furthermore, accurate traffic generation models can perform a critical function within testbeds to stress applications and network elements. Whether working in simulation or in a live testbed, experimentation often proceeds by simulating the use of the network by a given population of users using applications (such as web browsers or peer-to-peer file sharing applications). Synthetic traffic generators are used to inject traffic into the network according to a model of how the corresponding application or user behaves. This paradigm of simulation uses source-level descriptions of network traffic rather than packet-level descriptions of network traffic, as the latter reflects not only the source-level characteristics but also (in the case of closed loop control such as TCP) the congestion state of the network. A critical problem in simulation and network testbeds is thus the generation of application-dependent, network-independent synthetic traffic that corresponds to a valid, contemporary model of application or user behavior. Basic research is required at a number of levels to enable the construction of valid models. In order to construct models of application or user behaviors, measurements of application traffic in live networks must be performed, ideally at multiple locations over multiple measurement periods. The measurement process is complicated by the mundane problems of acquiring and archiving traces at gigabit and beyond line speeds. Research is required into methods for formalizing (and ideally automating) the process of network trace acquisition and the generation of application (or application class) traffic models. This includes issues of network measurement, trace analysis (informed by privacy considerations), model generation, model validation, and synthetic traffic generation. Any measurement effort will also have to confront the fundamental tension that exists between providing useful data about user behavior and ensuring the privacy of the users who generated it. For some testbeds, the impact of this tension may be minimal; just as users of the testbed have no assumption of production-quality availability, neither do they expect privacy. For other testbeds, however, privacy concerns may represent a significant impediment to data collection and


dissemination. Given this tension, research into transformations of network data in ways that preserve both privacy and usefulness could be of significant benefit to network research. 6.2.2. Dynamically Extensible Networks The last ten years has seen dramatic advances in the performance of computer networks, so much so, that we are approaching a point where performance is no longer such an overriding concern. We now have the opportunity to go beyond merely moving bits faster, to developing more sophisticated network architectures that take advantage of our greatly expanded ability to process information as it moves through a network, enabling innovative new services and applications that magnify the productivity of network users. Modern internet routers are engineered and optimized to move packets through a network efficiently and at high speed. While they are often constructed using programmable components, they are essentially single-purpose devices, with only limited programmability. Continuing progress in electronics will soon make it feasible for routers to evolve from simple packet forwarders to dynamically extensible network services platforms that will combine the data transport functions of a router with the ability to host application-specific plugins serving individual application sessions. Dynamically extensible networks can enable the rapid deployment of new network services and distributed applications, allowing a shift from a technology-centric focus to an application-centric focus, in which the interaction between users/hosts and the network is defined entirely in terms that are meaningful to the application and the user. Extensibility makes it possible to endow the network with application-specific knowledge and processing mechanisms, allowing network resources to be used more effectively, to provide the best possible service to users. By enabling these new capabilities through a general set of network extension mechanisms, we can maintain the network’s application-neutrality, ensuring its continuing ability adapt to meet new challenges as they arise. Such an infrastructure also naturally allows application/service providers to deploy a new service, without investing in a dedicated infrastructure to support it. Extensible networking raises a wide range of important research challenges. There are fundamental network architecture issues that must be resolved to define the appropriate set of abstractions for network extensibility and to determine how those abstractions can be mapped into protocols, algorithms, software and hardware. Given an appropriate model for network extension, we will need general methods for mapping the resource needs of individual application sessions onto available resources in a real network configuration. Extensible network services platforms will require the development of new system architectures to move and process large volumes of information as it moves through a network. The dramatic progress in the sophistication of configurable logic devices over the last several years, creates intriguing possibilities for combining the flexibility of software with the intrinsic parallelism and high performance of hardware. Extensible networking represents a radical departure from conventional data networks and presents a wide range of possible design choices at all levels, from network architecture on down. Determining the best choices in this design space will require the construction of prototype systems and experimental evaluation of these systems in a realistic testbed


environment. It will require development of sample applications so that application developers can evaluate network extension mechanisms and recommend ways that they can be improved. It will require measurement studies to determine how extensible networks can be expected to perform under a wide range of traffic conditions. It will require the development of compelling demonstrations to inspire router vendors and network operators to embrace the technology, so that its benefits can be brought to the public at large.

6.2.3. Overlay networks Given its popularity and wide-spread use, it is easy to forget that at one time the Internet was a laboratory for researchers to experiment with packet-switched networking. The more the Internet has become a commercial success, however, the less useful it is as a platform for exploring new ideas. Today, commercial interests shape the Internet's continued development.

In fact, a recent report from the National Research Council [Patterson 2001] points to the ossification of the Internet, both intellectually (pressure for compatibility with current standards stifles innovation) and in terms of the infrastructure itself (it is nearly impossible for researchers to affect the core infrastructure). The report goes on to observe that at the same time, a whole new set of challenges are emerging that may require a fresh approach. The dilemma, according to the report, is that

...successful and widely adopted technologies are subject to ossification, which makes it hard to introduce new capabilities or, if the current technology has run its course, to replace it with something better. Existing industry players are not generally motivated to develop or deploy disruptive technologies...

Finding the right way to introduce disruptive technologies is an interesting issue. Such innovations are likely to do some things very well, but overall they lag present technology in other important areas. For example, to introduce a new routing strategy into the Internet, one would have to build a router that not only supports this new strategy, but also competes with established vendors in terms of performance, reliability, management toolset, and so on. This is an extremely tall order. What the innovator needs is a way to allow users to take advantage of the new idea without having to write the hundreds of thousands of lines of code needed to support just the base system.

Overlay networks provide exactly this opportunity. Overlay nodes can be programmed to support the new capability or feature, while using existing infrastructure (i.e., conventional nodes and links in existing networks) to provide connectivity between overlay nodes. Over time, if the idea deployed in the overlay proves useful, there may be economic motivation to migrate the functionality into the feature set of commercial routers.

Overlay networks define an interesting model for introducing and experimenting with novel networking functionality. Design of overlay networks offers a number of basic research challenges, including: (1) specification of overlay network characteristics; (2) mapping of overlay networks onto physical networks; (3) resource reservation for each overlay; (4) protection between overlays; (5) tools for auto-configuration and management of overlays. Overlay network testbeds provide a platform to design, experiment and evaluate solutions to these problems. Further, overlay networks can foster development of novel network applications, although these overlay applications will be constrained by the limitations (bandwidth, delay, variability in service) of the underlying network(s) providing connectivity.


It is expected that first generation overlay networks will be constructed using commodity components (e.g., PCs interconnected using IP tunnels). With the rapid improvement in technology, there is a strong possibility that future networks (and hence overlays) will be constructed using programmable network systems. Hence, the research community will benefit significantly from the creation of testbeds constructed from programmable network systems (each network system may be built using some combination of network processors, general-purpose processors, function-specific co-processors, and programmable logic). Such testbeds will operate with high-bandwidth links; hence, they will expose to network application designers the challenges in deploying rich functionality into real routers. Further, they will foster research in (1) designing hardware kits for programmable network systems, and (2) designing programming methodologies and tools (e.g., compilers, run-time systems, etc.) that can hide the architectural details of the network systems from network application designers and thereby make network systems as easy to program as conventional PCs.

As an example of the role an overlay testbed can play, consider the possibility of developing new routing services. Unlike the Internet, which supports a default routing service based on BGP, an overlay would make it possible to design and deploy alternative routing services. For example, such new services might exploit distributed hash tables, advance a publish/subscribe model, provide support for multicast, or leverage new measurement services that do a particularly good job of mapping the network traffic conditions. Applications might select the routing service that best meets their needs, and over time, researchers will discover the set of primitive services (abstractions) upon which higher-level routing services can be built. In the end, it seems likely that these primitives will migrate from the overlay into the Internet itself.

In general, the potential of geographically distributed overlays lies in the fact that they provide multiple viewpoints of the network. New routing services might evolve to exploit distributed information about network conditions, and in the same way, new congestion managers might be able to take advantage of distributed information about bottlenecks. Similarly, one could imagine DDoS thwarting mechanisms and spam filters that take advantage of being able to observe network-wide activity rather than only being able to observe traffic as it crosses the last hop.

6.2.4 Wireless/Mobile Networks The wireless/mobile area is viewed as a particularly strong candidate for future network testbeds in view of the growing importance of portable computing and embedded sensor devices. Since sensor networks will be considered separately in section 6.2.5, we focus in this section on other forms of wireless and mobile networks. Over a five-to-ten year period, it may be expected that the majority of networked devices will have wireless access to the Internet, and that a growing proportion of these devices will be embedded into the physical environment to facilitate daily life. There are many technical challenges associated with this future wireless scenario, in view of the unique device heterogeneity, system scale and network robustness requirements. An important architectural issue is that of defining a next-generation wireless system which acts as a “network-of-wireless-networks” accommodating a variety of radio technologies and mobile service requirements in a seamless manner. Another important thread of research is aimed at self-organization in wireless


systems based on ad-hoc discovery and routing protocols between network nodes. There is also a need for a new generation of “smart” radio modems capable of extremely high spectral efficiencies and agility to support increasingly dense wireless deployments in the future. Lastly, there are various “cross-layer” protocol optimization issues that need to be studied across the range of radio, link, network, transport and application layer protocols under consideration in future systems. Research testbeds that focus on future wireless networking challenges are a necessary complement to theory and design activities because it is impossible to fully capture real-world radio channels with analytical or simulation models. There is no substitute for a real life environment to capture the complexity of multipath and fading that is inherent to wireless. Also, wireless systems tend to have complex traffic and mobility models with potentially significant cross-layer protocol interactions which are difficult to evaluate in the absence of a complete and realistic end-to-end system model. As a result, understanding of wireless networks requires extensive prototyping and experimentation, in order to uncover new insights that lead to improvements in system design. As wireless networks evolve from today’s centrally managed cellular and WLAN networks towards self-organizing ad-hoc systems, their performance and dynamics will be even more difficult to predict without system prototypes. In addition, the availability of such testbeds may be expected to stimulate application-level research on these emerging scenarios, leading to a better understanding of traffic requirements and cross-layer design issues. Two examples of wireless testbeds that we consider valuable for current and future network research are discussed below. These examples are meant to be illustrative only; many other valuable wireless testbed efforts are certainly possible.

• Network-of-Wireless-Networks Testbed: A testbed of this type would be aimed at

heterogeneous wireless systems that integrate short-range (sensors, pico-cell radios, UWB, etc.), medium-range (WLAN, UWB) and long-range (cellular 3G, 4G) radios. Research issues include global “network-of-wireless-networks” protocol architecture (“beyond mobile IP”), cellular/WLAN interworking, dynamic mobility, cross-layer adaptation, transport-layer and security. A research testbed for such a system would consist of an open/programmable mobile network infrastructure consisting of routers, switches, base stations and access points, interworking gateways, application servers, a number of highly adaptive, fully instrumented radio transceiver platforms, and various mobile computing platforms. Such a testbed will make it possible for researchers to (1) develop an understanding of wireless systems that have largely remained closed to the research community; (2) make controlled measurements across various systems to gain an understanding of the dynamics that result from their federation; and (3) test architectural innovations in a live environment. It is noted that no such facility is currently available to networking researchers, and it is unlikely that industry will provide such an experimental platform due to competitive issues. A testbed of this sort may need to secure experimental licenses from the FCC and maintain a limited wide-area infrastructure, and is thus a candidate for a single national facility involving multiple institutions.


• Unlicensed Spectrum Testbed: This type of testbed would be aimed at evaluating the efficiency of spectrum usage given specified radio technologies operating in unlicensed spectrum bands. Experiments of this sort are motivated by the immediate need for improved spectrum efficiency in the unlicensed wireless LAN (2.4 GHz 802.11b and 5 GHz 802.11a) bands which are currently experiencing rapid increases in usage. At the same time, there is a growing need to improve fundamental understanding of co-existence and collaboration techniques in such unlicensed bands, particularly in the presence of multiple radio technologies such as 802.11b and Bluetooth. Such an experimental network would consist of: (1) large numbers of end-user radio devices under controlled or realistic field conditions; (2) facilities for implementing different spectrum sharing methods, including policy servers, spectrum etiquette protocols, etc.; and (3) extensive measurement and data collection tools for evaluating system performance and spectral efficiency. A research project with this focus requires a large base of observed data, and is thus suitable for a single collaborative project aimed at integrating spectrum measurements from numerous locations and institutions.

6.2.5. Sensor networks A promising area that's closely related to wireless/mobile networks is sensor networks. Although research on sensors connected by packet radio stretches back to the 1980s, the recent development of low-cost wireless radios, sensors and embedded computers makes it possible to deploy thousands of small, sensing nodes dedicated to a single task. A recent NRC study explores both the promise and challenge of sensor networks in depth [NRC-EE 2001]. The long-term benefit of sensor networking is the ability to bring computing, communication, and sensing into the physical world. Many applications exist in the sciences, the military, government, and industry. The recently started Center for Embedded Networked Sensing (http://www.cens.ucla.edu/) has identified applications in four areas: environmental science (contaminant transport monitoring), marine biology (studying marine microorganisms), biology (micro-habitat monitoring), and geophysics (seismic monitoring). Challenges new to sensor networks include strong resource constraints, particularly in energy, but also in CPU and bandwidth; how to operate with and exploit very dense deployments (thousands of nodes in close proximity), and how to exploit the distributed processing capabilities of these many nodes. These challenges present research opportunities across all layers of the networking stack. At the lowest layer, improved physical radio technologies are needed to provide low-energy, short-range, moderate-bit-rate channels. New technologies such as ultra-wide-band radios offer promise. At the MAC layer additional work is needed to address fairness, energy efficiency and adaptive protocols that can scale to thousands of nearby nodes. In routing, sensor networks present new challenges to ad hoc Internet routing, and also non-Internet approaches such as directed diffusion. Distributed resources lead to a range of resource discovery and resource selection problems that may be integrated with routing or explored as separate services. New transport layers are needed to deal with streaming, real-time sensor data. Finally, applications


must understand how to adapt to this environment that is so different from today's well-connected, high-bandwidth Internet. Sensor networking provides strongly interdisciplinary research challenges as well. The work borders on a number of non-networking and non-computer science fields, including robotics, signal processing, sensor development, and even ethics and public policy, in addition to applications in most of the other sciences, the military, government, and industry. A sensor networking testbed differs in several ways from a wireless/mobile networking testbed. Most importantly, sensor network testbeds must consider not only the wireless and computing issues, but also the sensing component of the problem, with simulated or real sensors and targets. Because sensors are deployed in the environment rather than with people (as in current wireless work with PDAs, cell phones, and laptop computers), deployment densities of nodes can be much greater. Also unlike these technologies, many or all of the nodes in some sensor networks are stationary. Finally, because of the interdisciplinary nature, and because so little is known about the workloads of these applications, a sensor networking testbed will perhaps benefit from a stronger applications focus than other network testbeds. The traditional network testbed model of wide-area, high-bandwidth wired links is clearly not applicable to wireless sensor networks. One opportunity is a centralized sensor network testbed. Such an approach would be ideal if the testbed could be remotely accessed; additional research is needed to understand how to enable physical interaction with a remote testbed. (One approach suggested at the workshop was that model radio-controlled cars can provide a limited ability for remote-controlled node and target movement.) A complementary opportunity is a sensor networking kit (with the components and advantages discussed earlier in section 3.2) that would provide many institutions with an experimental, hands-on facility in which to perform sensor network research. 6.2.6. Testbeds and network-security related research Network security research can involve a wide range of functions and protocols that many times can only be effectively researched in a testbed environment. Network security-related research may require activities that overwhelm the capacity of network resources or otherwise disrupt the normal functions of networks. Similarly, research into defenses against attacks on network infrastructure components and resources will, by the very nature of the attack, disrupt the normal network functions and capabilities. Environments for research that require such disruptive activities are essential to the success of research into defenses against network-based attacks. The only feasible way to provide such environments is with testbeds whose network functions and capabilities can be disrupted. One specific example of the need for such a testbed is research related to defenses for distributed denial of service (DDOS) attacks. These attacks are distinguished by an overwhelming flood of network packets that are generated from many different sources with the intent of preventing legitimate use of services. Due to the nature of the attacks, they have proven to be extremely difficult to defend against and, as a result, are driving significant research activities. Since the attacks themselves result from use of malicious code, installation and use of such malicious code


on operational networks is infeasible (and illegal in some jurisdictions) so research related to defenses against DDOS attacks must be conducted in testbed environments. Since DDOS attacks are waged by hundreds to tens of thousands of machines from any portion of the Internet, DDOS defense testbeds must be able to provide a reasonable approximation of extremely large and diverse attacks. A reasonably large-scale isolated network testbed is also needed when there is only the risk of uncontrolled disruption. For example, safe but accurate evaluation, either for offensive or defensive purposes, of a potentially highly destructive worm, could only be done in a quarantined, yet realistic environment. Evaluation of intrusion-response systems that involve large-scale automated responses to cyber attacks poses similar risks.


7. Recommendations Throughout the workshop discussions (both at the workshop and in earlier and later interactions by e-mail), workshop participants worked hard to identify specific workshop findings, so that these conclusions could be passed on to others in the form of this report. Our findings have been noted throughout this report, accompanied by surrounding amplifying text. Based on these findings and other observations documented in previous sections, workshop participants make the following recommendations to the National Science Foundation: Recommendation 1: Initiate a new program in network research testbeds. Given the critical role of research testbeds in networking research, and the many benefits that flow from a vibrant research activity in such testbeds, NSF should initiate a significant new program to provide core funding for network research testbeds. These testbeds should be driven by the needs and research goals of networking researchers. The research opportunities identified in Section 6 of this report represent an initial step by the community in defining these goals.

Recommendation 2: Recognize the value of a broad spectrum of different testbed activities. Testbeds can range widely in size, scope and geographic extent. They may take the form of traditional network testbeds, as well as newer forms including cluster testbeds, overlay testbeds, federated testbeds and network research kits. Testbeds can serve a variety of specific purposes and emphasize a range of different research concerns. Dedicated, high-bandwidth, wide-area links are not a necessary prerequisite for creating a useful networking testbed. We view this diversity as a strength, reflecting the vitality of the networking research community and recommend that a new testbed program in networking embrace the full spectrum of testbed activities. Recommendation 3: Appropriately fund the human resources needed to conceive, design, construct, operate, and maintain testbeds and their management software. Professional staff will be required to design, build, maintain, and operate testbeds, and provide support for the testbed’s user community. Truly innovative testbeds and management models are likely to require full-fledged research efforts. Such “people costs” can often significantly exceed hardware infrastructure costs. Recommendation 4: Adopt funding models that provide long-term support for testbeds. Significant investments in human, software, and hardware resources will often be required in a testbed over time. To realize the full potential of such investments, and to allow researchers maximum opportunity to work with well-developed smoothly-functioning testbeds, long term support for specific network research testbed efforts will be required, e.g., a funding cycle of 5 years, with renewal possible. Explicit feedback loops from testbed users to testbed developers and operations are critical throughout the cycle.

Recommendation 5: Network research testbeds should be driven by the needs and research goals of networking researchers. Network research testbeds are distinct from generic networked testbeds that provide production-quality service in that research testbeds must support


the needs of networking researchers. These needs include the ability to experiment with, modify and replace network infrastructure components - an inherently destabilizing process. Network testbeds that focus on users and applications are typically not the best vehicles for supporting networking research, since users’ expectations for reliable, production network service are incompatible with networking researchers’ need to experiment with and modify network infrastructure components. Almost by definition, an application-only testbed is not a network research testbed. However, the presence of specific applications (or classes of applications) should be considered as long as their presence supports the needs of networking researchers - by driving the networking research agenda, by allowing destabilizing experimentation with infrastructure components, by providing realistic workloads, and by validating networking research through compelling demonstrations. Recommendation 6: Emphasize the importance of instrumentation, measurement, archiving, and dissemination of measured data in research testbeds. In evaluating proposed testbeds, particularly those with an envisioned evolution from research testbed to engineering testbed to operational network, intelligent integration of measurement into the architectural design should be given due weight. Furthermore, it should be recognized that dissemination of measured data to the larger research community - to the extent that such data would be useful - adds to the research value of a testbed project as the data itself becomes a shared community resource. Additionally, NSF should consider funding research into privacy-preserving and usefulness-preserving transformations of traffic data, as such research could increase the research value of many testbed projects by enabling the dissemination of data that otherwise would not be available. Recommendation 7: Consider providing substantial, long-term support for a small number of large (center-like) multi-user, experimental facilities, providing open, well-managed experimental services to remote users. This support should include support for professional staff to manage the facilities and provide for their on-going development. Such facilities could also play an important role in building research community around the shared research infrastructure, and provide educational opportunities for faculty and students alike in the area of experimental systems research through internships, tutorials, and other educational activities. Recommendation 8: Provide network research testbed opportunities for a broad set of networking research topics, while providing specific attention to research in overlay-networks, fixed and mobile wireless networks, sensor networks, security, dynamically extensible networks, and measurement. As discussed in Section 6.2, these specific areas were identified by workshop participants as being particularly “ripe” for research advances and particularly well-suited to leverage testbed activity. Recommendation 9: Initiate a broad-based effort to find ways to facilitate technology transfer for networking research results. This should include exploration of ways to bridge the gap between network research testbeds and user/application oriented testbeds, to enable new networking technologies to be demonstrated within user/application oriented testbeds.


References [Aiken 1997] R. J. Aiken, R.A. Carlson, I.T. Foster, T.C. Kuhfuss, R.L. Stevens, L.

Winkler,"Architecture of the Multi-Modal Organizational Research and Production Heterogeneous Network (MORPHnet)," Argonne National Laboratory, ECT and MCS Divisions. 1997 http://www.anl.gov/ECT/Public/research/morphnet.html

[Andersen 2001] D. Andersen, H. Balakrishnan, F. Kaashoek, R. Morris, “Resilient Overlay Networks,” Proceedings of the 18th ACM Symposium on Operating Systems Principles, Oct. 2001, pp. 131--145.

[ATDnet 2002] ATDnet homepage, http://www.atd.net/

[Berson 2002] S. Berson, S. Dawson, R. Braden, "Evolution of an Active Networks Testbed", Proceedings of the IEEE Computer Society DARPA Active Networks Conference and Exposition, May 29-30, 2002, San Francisco, California USA

[Conway 1999] Committee on Innovations in Computing and Communications: Lessons from History, Funding a Revolution: Government Support for Computing Research, National Research Council, National Academy Press, ISBN: 0-309-06278-0, 1999. http://www.nap.edu/readingroom/books/far/

[Fall 1999] K. Fall, “Network Emulation in the Vint/NS Simulator,” In Proceedings of the Fourth IEEE Symposium on Computers and Communications, pp. 244-250. Sharm El Sheikh, Red Sea, Egypt, IEEE. July, 1999. http://www.cs.berkeley.edu/~kfall/ns-em-paper.ps

[Fuller 1996] B. Fuller, I. Richer, “The MAGlC project: From vision to reality,” IEEE Network, vol. 10, no. 3, May/June 1996, pp. 15-25.

[Handley 2002] M. Handley, O. Hodson, E. Kohler, “Open Platforms for Network Research,” Proceedings of the 1st ACM Workshop on Hot Topics in Networks (HotNets-I), October 2002, http://www.icir.org/mjh/xorp-hotnets.pdf

[Intel 2002] Intel IXP1200 Network Processor, http://www.intel.com/design/edk/product/ixp1200_edk.htm

[Macedonia 1994] M. Macedonia, D. Brutzman, "MBone Provides Audio and Video Across the Internet," IEEE Computer, Vol.27 No.4, April 1994, pp. 30-36.

[MONET 2002] MONET homepage, http://www.bell-labs.com/project/MONET/

[NRC Experimental 1994] Committee on Academic Careers for Experimental Computer Scientists, Academic Careers for Experimental Computer Scientists and Engineers, National Research Council, National Academy Press, Washington, D.C. 1994. http://www.nap.edu/books/0309049318/html/index.html

[NRC-EE 2001] National Research Council Committee on Networked Systems of Embedded Computers, Deborah Estrin (chair). Embedded, Everywhere, National Academy Press, 2001.

[Patterson 2001] D. Patterson, D. Clark, A. Karlin, J. Kurose, E. Lazowska, D. Liddle, D. McAuley, V. Paxson, S. Savage, E. Zegura, Looking Over the Fence at Networks: A Neighbor's View of Networking Research, Committee on Research Horizons in Networking, Computer Science and Telecommunications Board, National Research Council , National Academy Press, ISBN: 0-309-07613-7, 2001. http://www.nap.edu/catalog/10183.html

[Partridge 1998] Craig Partridge (ed.), "Report of the NSF Scalable Information Infrastructure Workshop," Oct. 1998. http://www.ir.bbn.com/~craig/NSF-SII-workshop.html


[Peterson 2002] L. Peterson, T. Anderson , D. Culler, T. Roscoe, “A Blueprint for Introducing Disruptive Technology into the Internet” Proceedings of the 1st ACM Workshop on Hot Topics in Networks (HotNets-I), Princeton, NJ October 28-29, 2002. http://www.cs.princeton.edu/nsg/papers/hotnets.html

[Rizzo 1998] Luigi Rizzo. Dummynet and Forward Error Correction. In Proceedings of the USENIX Technical Conference, New Orleans, USA, USENIX. June, 1998. http://www.iet.unipi.it/~luigi/freenix98-dummynet.ps.gz.

[Turner 1999] J. Turner, “First Gigabits Kits Workshop,” 1999. http://www.arl.wustl.edu/gigabitkits/Workshop_99_07/slides/turnerintro.pdf

[Vahdat 2002] A. Vahdat, K. Yocum, K. Walsh, P. Mahadevan, D. Kostic, J. Chase, D. Becker,

“Scalability and Accuracy in a Large-Scale Network Emulator,” Proc. OSDI 2002 (5th Symposium on Operating Systems Design and Implementation), Dec. 2002. http://issg.cs.duke.edu/modelnet.pdf

[White 2002a] B. White, J. Lepreau, L. Stoller, R. Ricci, S. Guruprasad, M. Newbold, M. Hibler, C. Barb, A. Joglekar, “An Integrated Experimental Environment for Distributed Systems and Networks,” Proc. OSDI 2002 (5th Symposium on Operating Systems Design and Implementation), Dec. 2002. http://www.cs.utah.edu/flux/papers/netbed-osdi02-base.html

[White 2002b] B. White, J. Lepreau, S. Guruprasad, “Lowering the Barrier to Wireless and Mobile Experimentation,” Proceedings of the 1st ACM Workshop on Hot Topics in Networks (HotNets-I), October 2002. http://www.cs.utah.edu/flux/papers/barrier-hotnets1-base.html


Appendix A: Workshop Agenda

Agenda NSF Network Research Testbeds Workshop

October 17, 18 Chicago Hilton O'Hare

Meeting Location: Room 2033

Thursday

8:00 - 9:00 - Buffet Breakfast. Room 2033.

9:00 - 9:15 Welcome and introductions. [Jim Kurose and Mari Maeda]

9:15 - 10:45 Network Research testbeds: looking back and thinking ahead

Mari Maeda NGI Testbeds Victor Frost Magic Testbed Bob Braden DARTnet & ABone Jay Lepreau Emulab Craig Partridge Past Testbeds: An Overview

Speakers will summarize: Purpose, Community served, infrastructure and organization, successes, ingredients of success (or failure), and lessons learned. Panel chair: Bob Braden.

11:00 - 12:30 Research Opportunities This open discussion will identify research areas and directions that would be enabled with research testbed infrastructure(s). What networking research areas are ripe for testbed activity? Without a new generation of research testbeds, what good things will not happen? What role testbeds can play in stimulating and maintaining a vibrant on-going effort in experimental/systems research in networking? Thinking five or ten years out, how would the field be different with, or without, significant research tested activity? The discussion will begin with a summary of thoughts submitted in advance by workshop participants. Discussion moderator: Jim Kurose

12:30 - 1:30 Lunch

1:30 - 3:15 Components of a successful research testbed program.

This panel will identify and discuss (and, no doubt, expressing strong opinions on) the myriad issues that arise when considering a program of network research testbeds. What kinds of research testbeds can be imagined - small and/or localized, wide area, federated, testbed “kits,” a testbed center - and what are the pros and cons of each? What are the cost/benefit tradeoffs? What is needed to effectively manage a testbed - both the technology and infrastructure, as well


as the user population? What tools and technologies could be provided within a testbed for use by researchers? What are the roles of applications in a research testbed? Should a testbed effort be sustainable long term, or should testbeds be built, used, and then “thrown away”? What models are most likely to lead to a successful program? What's not needed in a testbed? What role can industry play? Are there new "out-of-the-box" ideas and approaches that we should consider? Moderators: Jay Lepreau and Jon Turner.

3:15 - 3:30 Break

3:30 - 5:00 Breakout sessions: synthesis.

We’ll break out into two smaller groups to summarize what has been discussed today. We'll map participants into groups via late-binding depending on participant interest, but we are thinking of doing this on the basis of wireless/mobile testbeds, and wide-area/wired/overlay. What “findings” would we like to make in our report? What recommendations do we want to make in terms of a “program” for network research testbeds? Breakout session leaders: Breakout session leaders: Mario Gerla, Jay Lepreau, Ramesh Rao.

Breakout Session 1 (Gerla: broad) Breakout Session 2 (Rao: wireless) Breakout Session 3 (Lepreau: infrastructure)

5:00 - 6:00 Report back, summarization, report planning.

7:00 - 9:00 Dinner. (Tokyo/Zurich Room)

Friday

8:00 - 8:30 Continental Breakfast. Room 2033.

8:30 - 10:00 Discussion of overnight thoughts, report planning

10:00 - 12:15 Writing breakout groups. TBD.

12:15 – 12:45 Wrap up, homework, and report completion schedule.

12:30 Lunch. Room 2033.

Report of NSF Workshop on Network Research Testbeds

Documents

Transcript of Report of NSF Workshop on Network Research Testbeds