Péter Hága Eötvös Loránd University

Building the Measurement Infrastructure and Virtual Data Observatory

in the European Future Internet Experimental Facility

Péter Hága Eötvös Loránd University

why to build a new experimental facility?

• in the Internet it is often not possible to measure traffic flows and other aspects of usage

• models and analysis of measurement data should play crucial role in the research on a Future Internet

• the design of new network architectures should be amenable to modeling and measurement the network itself

(in contrast of the today’s Internet)

• the Experimental Facility should provide a platform with a rich set of tools for measurement and monitoring

• the EF should offer full observability of the experiment and the related data

• theoretical analysis and modeling can benefit from the rich capture and logging of data

FuturICT 2009, Budapest, Hungary, 29 June 2009.

FI research initiatives

• GENI/FIND/NetSE, US• AKARI Project, Japan• Future Internet Forum, Korea• ANR, France• G-Lab Initiative,

Germany• SHOK, Fin• Internet del Futuro,

Spain• EU Future Internet Research and Experimentation (FIRE)


GENI infrastructure

Best Testbed Award

• The European Traffic Observatory Measurement InfrastruCture (etomic) was created in 2004 within the Evergrow Integrated Project.

• Since 2005 also supported by the Hungarian Office for Research and Technology

• Its goals:– to provide open access, public testbed for researchers

experimenting the Internet – to serve as a Virtual Observatory active measurement data on

the European part of the Internet

• Since 2008 in Onelab2 Etomic is federated in the Advanced Network Monitoring Equipment

etomic history


http://www.tridentcom.org/tridentcom05/bta05.htm

The GENI connection


Infrastructure stations

Best Testbed Award

http://www.tridentcom.org/tridentcom05/bta05.htm

• Future Internet Research and Experimentation (FIRE) programme• OneLab2: An Open Federated Laboratory Supporting Network• OneLab2 EU Integrated Project 2008-2010 • 29 participants Europe wide + 1 from Japan• budget 7.5 M€• aims to federate several measurement infrastructures

(including Planetlab Europe)

• includes building (extending) and maintaining a monitoring and measurement infrastructure (based on the Etomic)

• partners involved: – Université Pierre et Marie Curie, France – Universidad Autonoma de Madrid, Spain– Tel Aviv University, Israel– Eötvös Loránd University, Hungary


Advanced Network Monitoring Equipment

• Web interface available via www.etomic.org• Account application -> own measurement design• Free access to periodic measurement end-to-end data

• Measurement time slot reservation for registered users, unique slot (experiment are not affected by other users)

• Programming DAG and ARGOS cards via a user friendly API• Controlling APE box• Fully configurable active measurement scenarios• Measurements are distributed automatically to the

measurement stations• Measurement data is stored in a VO fashion


http://www.etomic.org/

Hardware setupAdvanced Network Monitoring Equipment (ANME)



Main components

ANME hardware components

• Etomic – Precise active measurements with DAG 3.6GE and ARGOS

FPGA

• CoMo– Monitoring the traffic of Planetlab nodes

• APE– lightweight measurement box– standalone

• GPS receiver to provide time synchronization


Etomic/CoMo architecture

• Server PC architecture• Linux OS• Endace DAG 3.6 GE card

or• ARGOS FPGA measurement card• with packet sending capability

(packet offset ~60ns)• GPS antenna for

time synchronization



APE lightweight measurement box

• standalone• based on Blackfin

programmable board• dedicated IP packet

timestamping module (< μsec)• low cost (300 €)

Infrastructure managementetomic Central Management System


Central Management System


• Etomic CMS• IBM blade server• User management• Node maintenance• Experiment scheduling• Storing experiment results

(temporally)• Web GUI

Slices vs. unique timeslots no virtualization

• balancing sliceability and fidelity is one of the most fundamental challenges facing the EF

• virtualization allows many researchers to share a common set of resources

• while virtualization introduces too much unpredictability in timing measurements

• dedicated measurement hardware elements can be allocated to some slices. Measurement hardware should operate under temporal partitioning principle.

realization


http://measurement.etomic.org/

Experimental use casesSome ongoing basic topics


Use cases


• one way delay (60nanosec resolution)

• tracking topology changes• available bandwidth meter• transport protocol testing• queuing delay

tomography• geolocation experiments• …

Data handlingStore & share


Experimental facilities

• most network measurement projects:▫ use a single dedicated infrastructure

▫ scan only narrow subsegments

▫ analyze a limited set of networkcharacteristics

▫ centralized and separated fromeach other

• key idea: try to interconnectseparate measurement data!▫ large-scale behaviour

▫ long-term evolution

Traditional approach

Traditionally measurements are designed to collect only specific data, important from the point of view of the researcher’s agenda.


Sharing science

Genome databases

Astronomy

Related work: CAIDA/DatCat

Related work: MoMe database

Related work: MAWI repository

Data publication efforts

1. DatCat (USA): searchable catalog of metadata about measurements passive traffic traces, traceroutes, BGP tables, virus propagation

studies

2. MOME (EU): database for meta-measurement data packet & flow traces, routing data, HTTP traces standardization efforts sharing of analysis tools is possible (e.g. jitter calculation)

3. MAWI (Japan): repository of passive traces from the WIDE backbone

(collected since 1999)

raw data isnot stored

raw datafrom single

infrastructure

• you can:GREP 1 MB in ~ secondsGREP 1 GB in ~ minutesGREP 1 TB in ~ daysGREP 1 PB in ~ years

• for 1 PB you need ~ 1000 disks

• at some point you need – to limit search– parallel data search and analysis– move the analysis to the data storage

• this is where databases can help

FTP and GREP are not adequate

−FTP ~ 1 MB/sec−FTP ~ 1 GB/min (= 1 $/GB)−FTP ~ 1 TB/ days and 1K$−FTP ~ 1 PB/ 3 years and

1M$

Key ideas in data handling

• store & share raw data– joint analysis of different types of measurement data– reanalysis (with new evaluation methods)– reference data (historical comparison)

• share analysis tools– server side processing simplifies client applications– no need to transfer bulk data packages: online processing

• standardization, network XML

Network Measurement Virtual Observatory (nmVO)


VO approach

• The modern approach is to collect and store all measurable data and make it available for „virtual observation”. Virtual measurements can have set of goals different from the original


Unified interface

• VO can be realized by collecting measurement data from different infrastructures. Data structures should be standardized → netXML


Casjobs - database interface

• the nmVO concept: efficient way to organize and share different types of measurement data and corresponding analysis tools

• prototype node implementation• client application example• a semantic extension to integrate

multiple datasources

Summary

• Experimental facility:– „old” etomic is running– „new” etomic will be launched this summer

• Main features:– Precise active measurements– Unique time slot reservation– Easy to use GUI– Data is collected in VO fashion

• European deployment (planned):


Thanks and register!

Visit: www.etomic.org

E-mail: [email protected]

Péter Hága Eötvös Loránd University

Documents

Transcript of Péter Hága Eötvös Loránd University