Network Monitoring using MonALISA and NetFlowPractical Applications born out of Collaborative ResearchNSF Award Number SCI – 0231844

Ernesto Miguel RubiResearch Network Engineer

CIARA / CHEPREO / AMPATHFlorida International University

eMail: ernesto@cs.fiu.edu

PASI Conference - Mendoza, Argentina - May 16th to 21st, 2005

Some Background & Acknowledgments • Born out of Summer 2003 NSF REU Award

– Studied at-length the network utilization of an AmPATH link leading to U.S. FedNets ( non-educational US research networks )

– Used NetFlow v5, deployed throughout the AmPATH core to explore WHO, WHEN, HOW and HOW MUCH of a particular network segment.

• Why is this important?

– It’s like knowing which genes in your body serve what purpose...

– Can adjust various network policies according to findings ( QoS, routing )

– Can justify to funding agencies the need for increased funding for a particular link and present factual evidence with specific networks and agency usage numbers

– Well, it’s really cool! No really, if you’re a network engineer you want / can verify your routing policies are working and can have a DETAILED view of your network traffic.

• I haven’t done the Acknowledgments yet...

Would like to thank...• AmPATH / CIARA

– Julio Ibarra / Heidi Alvarez

• CalTech

– Xun Su - Network Engineer

• FIU

– Jose Fernandez

– Eric Johnson

• CERN / SLAC - SDSC

– MonALISA Team

– Joeerg Michel, entire Monitoring Team ( NLANR / DAST )

• Coffee break is soon, I promise it’ll be interesting...let’s check it out!

Where do you want to go?• Really far....

– Realtime In-Depth network monitoring, scalable with a built-in historical component; quick, cheap platform independent solution built on something reliable and already proven

• Issues which we knew we’d find

– Large data sets to analyze ( typical AmPATH circuits 10GigE, GigE, OC-12, OC-3 )

– What is a good enough time to keep historical data ( week, month, year...can we keep that much? )

– What defines real-time network monitoring? Up to the second, minute, hour?

– What network blindspots does NetFlow create with its sampling characteristics

– We don’t want to re-invent the wheel, who can accommodate the data we have into a convenient already proven framework.

– Can we collaborate with our REU colleagues across multiple timezones and across language barriers and busy schedules?

• Let’s start at the beginning, NetFlow is the starting point since we already have plenty of experience with it from everyday use and previous REU work.

NetFlow and Flow-Tools•NetFlow

– Was originally developed by Darren Kerr and Barry Bruins at Cisco Systems in 1996 as a switching path. Today NetFlow is primarily used for network accounting. AmPATH uses NetFlow v5 in its core routers; different versions available.

– Different vendors use NetFlow and call it different things but everyone supports it.

•NetFlow is a process that:– Runs in the background on the router,

– Samples data at a pre-configured rate and

– Exports it to a pre-determined host or hosts.

• So...what does a NetFlow configuration on a router looks like? Well..just a line or two..

Cisco GSR 12012 – NetFlow Configuration

ip flow-export source Loopback0ip flow-export version 5ip flow-export destination 131.94.191.101 2058ip flow-sampling-mode packet-interval 100

–

Let’s first define a flow

• A flow is IP data which has the following seven identical characteristics:

•Source IP address•Destination IP address•Source port•Destination port•Layer 3 protocol type•TOS byte ( RFC 791 - Type of Service )•Input logical interface on a router

So what does the output look like?• It’s trivial!!!

So Sometimes you Need an interpreter....

• Flow-Tools ( http://www.splintered.net/sw/flow-tools )

– A collection of programs and libraries used to collect and process NetFlow data. These tools allow users to process stored flow data from a series of command line interfaces. ( WARNING: Only works on Cisco and Juniper routers ). Other interpreters out there....

– For our analysis we used Flow-Tools version 0.66:

• The example below shows traffic destined outside of the router via interface 61 using reporting format 5 (destination port) and sorting by octets.

$flow-cat ft-v05.2005-01-19.135207-0500 | flow-filter -I 61 | flow-stat -f 5 -S 2# --- ---- ---- Report Information --- --- ---## Fields: Total# Symbols: Disabled# Sorting: Descending Field 2# Name: UDP/TCP destination port## Args: flow-stat -f 5 -S 2### port flows octets packets#80 7036 2693420 1415744191 233 1602211 245625 793 1041730 239446878 2 937500 625

The AmPATH network comes under a watchful eye...

MonALISA ( from Iosif Legrand )• Reliable Registration and Discovery for Services and Applications.

• Monitoring all aspects of complex systems:

- System information for computer nodes and clusters

- Network information : WAN and LAN

- Monitoring the performance of Applications or services

- The End User Systems

• Can interact with any other services to provide in near real-time customized / filtered information based on monitoring data

• Secure, remote administration for services and applications

• Agents to supervise applications, to restart or reconfigure them, and to notify other services when certain conditions are detected.

• The MonALISA framework can be used to develop higher level decision services, implemented as a distributed network of communicating agents, to perform global optimization tasks.

• Powerful Graphical User Interfaces

An Aside - MonALISA and Grids

MonALISA and ApMon• ApMon is an Application Programming Interface (API) which interacts with the

MonALISA service.

• ApMon allows any application to send parameterized monitoring information to MonALISA. The data can be sent as UDP datagrams to multiple hosts running the MonALISA service.

– ApMon has been implemented in C, C++, Java, Perl, Python.

• NetFlow allows for the monitoring of a large number of parameters. We decided to limit the parameters monitor to...

MonALISA and ApMon

- UDP/TCP port destination/source traffic

- IP destination/source traffic

- IP protocol traffic

- IP Next Hop traffic

- AS destination/source traffic

- Prefix destination/source traffic

• For most of these parameters we will be monitoring that total traffic in octets over a period of time. The initial period of time was 5 minutes.

Our Development...

The application written will parse the flow-stat report and generate a name/value pair.

The name of the parameter will always be the first column of the flow-stat report. This name will be one of the following:

• A Port number (or name when applicatble)• An IP address• A protocol name or number• An AS number• A Prefix (in the form of network/mask bit)

The value of the parameter will always be the total number of reported octets for that parameter. This will generate a large number of parameters which will be difficult to sort visually.

Two Sides to the Coin...

• A configuration file

– Drawback: Large and clumsy to edit. Not easy to manage but this can be easily solved with a couple of more weeks of development time.

• Code to parse configuration file and submit to MonALISA the results of the parsing algorithm.

cluster_name:GSR-StarLight VLANnode_name:Ingress Destination Portsflow_file_directory:/home/netflow/flows/gsr/flow_stat_options:-f 5 -S 2 -nflow_filter_options:-i 60value_param:2max_params:15next

Processing...Our application would parse this information and generate the following report:

$flow-cat ft-v05.2005-01-19.224516-0500 | flow-filter -i 60 | flow-stat -f 5 -S 2 -n # --- ---- ---- Report Information --- --- ---## Fields: Total# Symbols: Enabled# Sorting: Descending Field 2# Name: UDP/TCP destination port## Args: flow-stat -f 5 -S 2 -n### port flows octets packets#57893 1 913876 110632808 1 730500 4876881 52 210301 28332920 1 151132 10132918 1 85132 571144 3 84854 61http 39 68171 1061279 1 67500 45eDonkey-20 67 64742 1073113 2 63044 4933498 3 58552 40

The application will parse the generated report and retrieve the parameters name and values as specified in the configuration.

For this example the parameter names will be the port numbers and the values will be the octets as specified by the configuration file (value_param:2).

MonALISA Customizations

• I knew I was forgetting something...we wrote this in Perl.

• ml.properties file heavily modified due to large number of data.

– lia.Monitor.use_epgsqldb=true– lia.Monitor.jdbcDriverString=com.mckoi.JDBCDriver

• These parameters tell MonALISA to use the embedded PGSQL database as opposed to the MYSQL database. – After working with the MonALISA development team they suggested to use the

PGSQL database as it was faster and more stable for our configuration.

MonALISA customizationslia.Monitor.Store.TransparentStoreFast.web_writes = 3

Specifies that 3 tables will be created for historical data.

lia.Monitor.Store.TransparentStoreFast.writer_0.total_time=43200lia.Monitor.Store.TransparentStoreFast.writer_0.table_name=monitor_s_12hourlia.Monitor.Store.TransparentStoreFast.writer_0.writemode=1

Defines the first table to maintain data for 12 hours. The data stored in this table written in writemode 1 which will write every value received by MonALISA.

lia.Monitor.Store.TransparentStoreFast.writer_1.total_time=10800lia.Monitor.Store.TransparentStoreFast.writer_1.table_name=monitor_s_e_3hourlia.Monitor.Store.TransparentStoreFast.writer_1.writemode=2

Defines the second table which will maintain data for 3 hours. This table will use writemode 2 which will store all data included objects which can not be displayed with the MonALISA client.

lia.Monitor.Store.TransparentStoreFast.writer_2.total_time=16070400lia.Monitor.Store.TransparentStoreFast.writer_2.samples=4464lia.Monitor.Store.TransparentStoreFast.writer_2.table_name=monitor_s_6monthslia.Monitor.Store.TransparentStoreFast.writer_2.writemode=0

This last table will store sampled stat for 186 days approximately 6 months. Writemode 0 specifies that the data will be sampled.

Use Cases

• Let’s do a live demo...risks are fun...

In case that didn’t work...

21

Tool Detail

22

Tool Detail ( cont. )

23

Tool Detail ( cont. )

You never talked about realtime...• Right, so NetFlow is not realtime, data collected in 5 minute snaps, so when we

analyze the data it’s already 5 minutes old!

• Also, it’s not a 1:1 view of packets in the network; remember it’s sampled and 1:100 is the best ration we can get without crashing our router.

– Wouldn’t be nice to capture 1:1 samples on a router with 25 - 10GigE interfaces...besides routers are made to route, their primary function is not monitoring.

• Right around.....here...we ran out of time but...here’s a teaser..we’ve been awarded the Cisco University Research Program Grant to continue monitoring research and integrate the NLANR Passive Monitoring Agent into our work

• Can help answer the question: How good is NetFlow, how many flows are we missing when we use NetFlow...

• It helps to answer them by:

PMA Overview

• Visit: http://pma.nlanr.net/

• Tap into physical network media ( fiber ) in a non-obstrusive way and generate traces of packet headers.

• Limited to short time samples because of data generated

• IP address of hosts on flows masked due to some security concerns ( can cause issues for research )

• Not much of an overview but let’s talk afterwards if you want more info.

• Which leads to us to....

Next Steps

• Integrate PMA results into current tool

• Correlate data obtained in both environments to come up with comprehensive answers to problems posed.

• Begin dissimenating the REU results and collaborating others who are interested in helping / deploying.

• Look at emerging issues such as user privacy and address them to be able to release a public version of software.

• Sleep.

See, that wasn’t that bad....

• Questions...

• How about a more extensive demo later?

– Please say yes....!!