Transcript of The Abilene Observatory
Internet2 Presentation TemplateUniversity of Minnesota 23 October
2003
11/10/2003 2
Internet2 Infrastructure • Abilene, HOPI, RONs • Observatory will
evolve to new infrastructures
Measurements • History • Operational / Research aspects • Abilene
Upgrade
The Observatory • Data Collections • Data Views • Collocation •
Input from research community
11/10/2003 3
Abilene – Design
11/10/2003 4
Abilene – Current
11/10/2003 5
Abilene Scale IP-over-DWDM (OC-192c) and IP-over-SONET
OC-48c Backbone, Juniper T-640 Routers 48 Connectors (OC-3c → 10
GigE) 222 participants – research universities & labs
• All 50 states, District of Columbia, & Puerto Rico •
Aggregation on the rise!
Expanded access • 92 sponsored participants, expected to increase •
30 State networks, expected to increase
Performance • Early IPv4/IPv6 tests – 8 gbps cross country (packet
blasts) • Recent tests – 5.6 gbps TCP flows from CalTech to
CERN!
11/10/2003 6
Abilene International Peering
Abilene Federal/Research Peering
Internet2/NYSERnet partnership Ethernet Switch
• Cisco 6513 (with new fabric/backplane soon) • Located in same
building as Abilene New York City Router Node • CA*net bringing in
OC-192 to same building
Abilene Tyco/SurfNet connection • OC-192 from Amsterdam
GLIF (Global Lambda Integration Facility) Installing Cisco
15454
• Located in MANLAN rack • Will bring CA*net and SurfNet OC-192s to
that switch • Potential for experimental TDM paths • Ethernets to
MANLAN • SONET to Abilene New York City Router Node
11/10/2003 9
• Abilene Packet Based Infrastructure • Regional Optical Networks –
RONs • National LambaRail Wave
Major Questions • Vision for the Network • Packet switching /
circuit switching futures • How do we go forward in the near
future
– Design team – Initial design February 2004
• How do we measure!! – Research community involvement
11/10/2003 10
California (CALREN) Colorado (FRGP/BRAN) Connecticut (Connecticut
Education Network) Florida (Florida LambdaRail) Indiana (I-LIGHT)
Illinois (I-WIRE) Maryland, D.C. & northern Virginia (MAX)
Michigan New York + New England region (NEREN) North Carolina (NC
LambdaRail) Ohio (Third Frontier Network) Oregon Rhode Island
(OSHEAN) SURA Crossroads (southeastern U.S.) Texas Utah
11/10/2003 11
11/10/2003 12
Measurement Infrastructure
Abilene Measurement Infrastructure History
• Original Abilene racks included measurement devices – Included a
single PC – Early OWAMP, surveyor measurements – Optical splitters
at some locations
Data collection motivation • Operational data – collected by the
NOC
– How is the network performing? – All users, other network
operators to understand network
• Some research data collected from the beginning
11/10/2003 13
Measurement Infrastructure
Upgrade to Juniper T-640 routers and OC-192c
• Important decision about rack space • Two racks with one
dedicated to measurement platform
• Potential for research community to collocate equipment
11/10/2003 14
Houston Router Node
nms1: throughput tester (iperf) • gigE direct to T-640, MTU
9000
nms2: ad-hoc on-demand (+ routing?) • gigE to switch to T-640, MTU
1500
nms3: statistics collection (flow, snmp) • 100bT to switch to T-640
• Local data collection to capture during network instability
nms4: latency tester (owamp, troute) • 100bT to switch to T-640 •
CDMA “GPS” timing source from endruntechnologies.com
11/10/2003 17
Hardware
Intel SR2200 from ioncomputer.com Intel SCB2 motherboard Dual 1.26
GHz Pentium III, 512K L2 1GB PC-133 DRAM in two banks ServerWorks
ServerSet HE-SL chipset 64 bit 64 MHz PCI Syskonnect GigE SK-9843
SX Redundant 48VDC
11/10/2003 18
OS
nms1’s ( in transition to: ) • Linux 2.4.20, SMP, no changes • All
traffic default routed through gigE • Buffers tuned a-la LBL
document • 990 Mb/s TCP between any two • (better TCP
platform)
nms2-4: • FreeBSD 4.6-STABLE • Buffers tuned • (better UDP
platform)
11/10/2003 19
Measurement Capabilities
One way latency, jitter, loss • IPv4 and IPv6
Regular TCP/UDP throughput tests – ~1 Gbps • IPv4 and IPv6;
On-demand available (see “pipes”)
SNMP (NOC) [octets, packets, errors; collected frequently]
• NOC working on SNMP proxy
“Netflow” (ITEC Ohio) [anon. by 0-ing last 11 bits] Multicast
beacon with historical data Routing data (BGP & IGP) [IGP under
development]
• Looking at Zebra + mods, Japanese routing research is
driver
(Optical splitter taps on backbone links at select
location(s))
11/10/2003 20
Throughput Tester Evolution
Original design had all four servers with the same operating
system
• Ease of administration • Lots of experience using FreeBSD for TCP
testing • FreeBSD features for capturing UDP packets (router flow
capture)
• FreeBSD had more respected NTP implementation
• Linux has Web100 development and large high- energy physics
installed base
FreeBSD 4.6-Release deployed
Throughput Tester Evolution
Over the last year, more TCP experience in wide area (9000 byte
MTU, gigE direct to T-640)
• DC to LA: 980 Mb/s UDP, but ~380 Mb/s TCP • NC-ITEC to LA: 990
Mb/s TCP, if sender has tuned, but otherwise stock 2.4.20 Linux
kernel
• Replicated in testing from Indianapolis (same location, same
connectivity, same hardware)
• The FreeBSD sender’s CPU is pegged, TCP stack apparently starved
(PUSH set on all frames)
11/10/2003 22
OWAMP
One Way Latency •Requires NTP on endpoints •Control connection used
to broker test request based upon policy restrictions and available
resources. (Bandwidth/disk limits)
•Enables the combination of regularly scheduled tests with
on-demand tests.
•http://owamp.internet2.edu/ •Reference implementation of
Draft:
http://www.ietf.org/internet-drafts/draft-ietf-ippm-owdp-06.txt
11/10/2003 23
Abilene Observatory
A program to provide enhanced support of computer science research
over Abilene
• Create network data archive – Consists of many separate databases
on a variety of servers – Forms a large correlated database –
Create tools to access the database – Support from and for graduate
programs
• Create Data views – A snapshot of Abilene • Collocation Component
- provision for direct network
measurement and experimentation –Resources reserved for additional
servers
• Power (DC), rack space (2RU), router uplink ports (GigE) –Initial
deployment is PlanetLab –Additional requests from AMP project and
research
team from Japan
• Usage Data • Netflow Data • Routing Data
–BGP now, ISIS in the future
• Latency Data • Throughput Data • Router Data • Syslog Data
11/10/2003 25
Databases
Variety of Interfaces to data • Simple web based for usages data •
Rsync for netflow • Simple web based for routing • Soap interface
for latency data • Throughput data undecided as yet • Router data –
Soap interface • Syslog data undecided as yet
Should this be standardized?
Large, correlated, distributed databases? Other types of data? More
importantly, other types of projects? Including networks closer to
the edge?
Gigapops and universities? Involving the international community?
Future infrastructures? Involving graduate programs –
graduate
students?
http://abilene.internet2.edu/observatory/data-views.html
Single page to see ‘10 worst’ measurements • Throughput data •
Losses • Variation in latency (95th – min) for links • Maybe top
utilization measurements too for comparison
Expect to iterate presentation to find “best” presentation for
us
11/10/2003 28
http://netflow.internet2.edu/weekly/ • Summarized flow data
11/10/2003 29
Outline
Measurement Infrastructure
Measurement Infrastructure
Hardware
OS