Post on 10-Jan-2016
description
Building the Measurement Infrastructure and Virtual Data Observatory
in the European Future Internet Experimental Facility
Péter Hága Eötvös Loránd University
why to build a new experimental facility?
• in the Internet it is often not possible to measure traffic flows and other aspects of usage
• models and analysis of measurement data should play crucial role in the research on a Future Internet
• the design of new network architectures should be amenable to modeling and measurement the network itself
(in contrast of the today’s Internet)
• the Experimental Facility should provide a platform with a rich set of tools for measurement and monitoring
• the EF should offer full observability of the experiment and the related data
• theoretical analysis and modeling can benefit from the rich capture and logging of data
FuturICT 2009, Budapest, Hungary, 29 June 2009.
FI research initiatives
• GENI/FIND/NetSE, US• AKARI Project, Japan• Future Internet Forum, Korea• ANR, France• G-Lab Initiative,
Germany• SHOK, Fin• Internet del Futuro,
Spain• EU Future Internet Research and Experimentation (FIRE)
FuturICT 2009, Budapest, Hungary, 29 June 2009.
GENI infrastructure
Best Testbed Award
• The European Traffic Observatory Measurement InfrastruCture (etomic) was created in 2004 within the Evergrow Integrated Project.
• Since 2005 also supported by the Hungarian Office for Research and Technology
• Its goals:– to provide open access, public testbed for researchers
experimenting the Internet – to serve as a Virtual Observatory active measurement data on
the European part of the Internet
• Since 2008 in Onelab2 Etomic is federated in the Advanced Network Monitoring Equipment
etomic history
FuturICT 2009, Budapest, Hungary, 29 June 2009.
The GENI connection
FuturICT 2009, Budapest, Hungary, 29 June 2009.
• Future Internet Research and Experimentation (FIRE) programme• OneLab2: An Open Federated Laboratory Supporting Network• OneLab2 EU Integrated Project 2008-2010 • 29 participants Europe wide + 1 from Japan• budget 7.5 M€• aims to federate several measurement infrastructures
(including Planetlab Europe)
• includes building (extending) and maintaining a monitoring and measurement infrastructure (based on the Etomic)
• partners involved: – Université Pierre et Marie Curie, France – Universidad Autonoma de Madrid, Spain– Tel Aviv University, Israel– Eötvös Loránd University, Hungary
FuturICT 2009, Budapest, Hungary, 29 June 2009.
Advanced Network Monitoring Equipment
• Web interface available via www.etomic.org• Account application -> own measurement design• Free access to periodic measurement end-to-end data
• Measurement time slot reservation for registered users, unique slot (experiment are not affected by other users)
• Programming DAG and ARGOS cards via a user friendly API• Controlling APE box• Fully configurable active measurement scenarios• Measurements are distributed automatically to the
measurement stations• Measurement data is stored in a VO fashion
FuturICT 2009, Budapest, Hungary, 29 June 2009.
Hardware setupAdvanced Network Monitoring Equipment (ANME)
FuturICT 2009, Budapest, Hungary, 29 June 2009.
FuturICT 2009, Budapest, Hungary, 29 June 2009.
Main components
ANME hardware components
• Etomic – Precise active measurements with DAG 3.6GE and ARGOS
FPGA
• CoMo– Monitoring the traffic of Planetlab nodes
• APE– lightweight measurement box– standalone
• GPS receiver to provide time synchronization
FuturICT 2009, Budapest, Hungary, 29 June 2009.
Etomic/CoMo architecture
• Server PC architecture• Linux OS• Endace DAG 3.6 GE card
or• ARGOS FPGA measurement card• with packet sending capability
(packet offset ~60ns)• GPS antenna for
time synchronization
FuturICT 2009, Budapest, Hungary, 29 June 2009.
FuturICT 2009, Budapest, Hungary, 29 June 2009.
APE lightweight measurement box
• standalone• based on Blackfin
programmable board• dedicated IP packet
timestamping module (< μsec)• low cost (300 €)
Infrastructure managementetomic Central Management System
FuturICT 2009, Budapest, Hungary, 29 June 2009.
Central Management System
FuturICT 2009, Budapest, Hungary, 29 June 2009.
• Etomic CMS• IBM blade server• User management• Node maintenance• Experiment scheduling• Storing experiment results
(temporally)• Web GUI
Slices vs. unique timeslots no virtualization
• balancing sliceability and fidelity is one of the most fundamental challenges facing the EF
• virtualization allows many researchers to share a common set of resources
• while virtualization introduces too much unpredictability in timing measurements
• dedicated measurement hardware elements can be allocated to some slices. Measurement hardware should operate under temporal partitioning principle.
Experimental use casesSome ongoing basic topics
FuturICT 2009, Budapest, Hungary, 29 June 2009.
Use cases
FuturICT 2009, Budapest, Hungary, 29 June 2009.
• one way delay (60nanosec resolution)
• tracking topology changes• available bandwidth meter• transport protocol testing• queuing delay
tomography• geolocation experiments• …
Data handlingStore & share
FuturICT 2009, Budapest, Hungary, 29 June 2009.
Experimental facilities
• most network measurement projects:▫ use a single dedicated infrastructure
▫ scan only narrow subsegments
▫ analyze a limited set of networkcharacteristics
▫ centralized and separated fromeach other
• key idea: try to interconnectseparate measurement data!▫ large-scale behaviour
▫ long-term evolution
Traditional approach
Traditionally measurements are designed to collect only specific data, important from the point of view of the researcher’s agenda.
FuturICT 2009, Budapest, Hungary, 29 June 2009.
Sharing science
Genome databases
Astronomy
Related work: CAIDA/DatCat
Related work: MoMe database
Related work: MAWI repository
Data publication efforts
1. DatCat (USA): searchable catalog of metadata about measurements passive traffic traces, traceroutes, BGP tables, virus propagation
studies
2. MOME (EU): database for meta-measurement data packet & flow traces, routing data, HTTP traces standardization efforts sharing of analysis tools is possible (e.g. jitter calculation)
3. MAWI (Japan): repository of passive traces from the WIDE backbone
(collected since 1999)
raw data isnot stored
raw datafrom single
infrastructure
• you can:GREP 1 MB in ~ secondsGREP 1 GB in ~ minutesGREP 1 TB in ~ daysGREP 1 PB in ~ years
• for 1 PB you need ~ 1000 disks
• at some point you need – to limit search– parallel data search and analysis– move the analysis to the data storage
• this is where databases can help
FTP and GREP are not adequate
−FTP ~ 1 MB/sec−FTP ~ 1 GB/min (= 1 $/GB)−FTP ~ 1 TB/ days and 1K$−FTP ~ 1 PB/ 3 years and
1M$
Key ideas in data handling
• store & share raw data– joint analysis of different types of measurement data– reanalysis (with new evaluation methods)– reference data (historical comparison)
• share analysis tools– server side processing simplifies client applications– no need to transfer bulk data packages: online processing
• standardization, network XML
Network Measurement Virtual Observatory (nmVO)
FuturICT 2009, Budapest, Hungary, 29 June 2009.
VO approach
• The modern approach is to collect and store all measurable data and make it available for „virtual observation”. Virtual measurements can have set of goals different from the original
FuturICT 2009, Budapest, Hungary, 29 June 2009.
Unified interface
• VO can be realized by collecting measurement data from different infrastructures. Data structures should be standardized → netXML
FuturICT 2009, Budapest, Hungary, 29 June 2009.
Casjobs - database interface
• the nmVO concept: efficient way to organize and share different types of measurement data and corresponding analysis tools
• prototype node implementation• client application example• a semantic extension to integrate
multiple datasources
Summary
• Experimental facility:– „old” etomic is running– „new” etomic will be launched this summer
• Main features:– Precise active measurements– Unique time slot reservation– Easy to use GUI– Data is collected in VO fashion
• European deployment (planned):
FuturICT 2009, Budapest, Hungary, 29 June 2009.
FuturICT 2009, Budapest, Hungary, 29 June 2009.
Thanks and register!
Visit: www.etomic.org
E-mail: haga@etomic.org