Experiences with tools for network anomaly detection in the GÉANT2 core

Connect. Communicate. Collaborate

Experiences with tools for network anomaly detection in the GÉANT2 core

Maurizio Molina, DANTE

COST TMA tech. Seminar

Samos, 23rd Sep 2008

Connect. Communicate. CollaborateThe GÉANT Network

• DANTE operates GÉANT2• Backbone network for National Research and Education

Networks in Europe• 30+ NRENs, 2 global connectivity providers (Telia and

GCrossing), peerings with other research networks (Abilene, Canarie, Clara, TEIN2, SINET…)


The GÉANT Network (IP layer)

• 20 Juniper routers

• tenths of GBit/s of aggregated traffic

• Main accesses and the backbone 10Gbit/s

Pls see www.dante.net

Connect. Communicate. CollaborateThe Services

• So…. Just a big pipe? No!• Services

– Dedicated L1-L2 circuits via multiple technologies– Performance Monitoring services (perfSONAR)– Support for federation of National AA Infrastructures

(eduGAIN) and wireless roaming (eduROAM)– Security Service Very NEW!

NEW!


The vision:enhance NRENs security

• NRENs have their (+ - evolved…) CERTs to deal with security

• and DANTE can filter traffic on GÉANT upon NRENs request….

! BUT !

• Can we be more proactive to NREN CERTs exploiting

the visibility of the GN2 core?


The vision (cont.):enhance NRENs security

• Approach: NetFlow (+ Routing data) & good processing tools

NetFlow v5 collector

• Netflow collected on all peering interfaces

• 1 / 1,000 Sampling

• ~3k flows/s


Proof of concept: Can we identify anomalies in the core?

• Anomalies are often “hidden”

Requirements:

High detection rate

Low false positives

Anomaly classification

Evidence collection

NfSen


From “volume” to “IP feature entropies”


•“IP features entropies”•Simple linear filter

Connect. Communicate. CollaborateDrilling down on peaks Connect. Communicate. Collaborate

-Concentration of DST IPs and DST ports receiving flows

-Dispersion of SRC IPs and SRC ports

• IRC server in Slovenia, receiving a lot of 60 bytes syn pkts on port 6667, mainly from a /16 Subnetwork of an University in the Netherlands.

• Likely a “BotNet war”?


Drilling down on peaks (cont.) Connect. Communicate. Collaborate

- Concentration of SRC and DST IPs and SRC ports

- Dispersion of DST ports

• Portscan of host in CARNET, from 4 hosts, 29 bytes packets

Connect. Communicate. CollaborateOpen source tools

• Results:– anomalies are observable in the GÉANT2 core– Novel methodologies (IP Features entropy) for their

classifications are applicable• Limits:

– NfSen does not fuse NetFlow and Routing data– Extensions would need to be run (and tuned) on all

ingress/egress points– No support, no guaranteed development

Connect. Communicate. CollaborateCommercial tools

• Test started Jun 08 (3 tools)– Tool 1

• PCA, entropy– Tool 2

• Large scale DDoS and Worm spread– Tool 3

• Per host behaviour

Connect. Communicate. CollaborateTool 1 (as a security tool…)

• Two main novel elements– Principal Component Analysis (PCA)– Both Volume and IP features Entropy anomaly

detection• Address what makes anomaly detection a complex task

– PCA: single parameter to control detection sensitivity, even if anomalies are attributed to specific OD pairs

– Entropy: Detection of both low volume (scans) and high volume (DoS) anomalies

Connect. Communicate. CollaborateDemo….

• …. Or Screenshots….

Connect. Communicate. CollaborateTool 2

• Well-established (and expensive!) solution for detecting “large” events

• Originally based on large volume shifts only• Now enhanced to give alerts on “fingerprints” (e.g.

communication with C&C servers)– Shared by (part) of the user community (50 out of 120)

• No usage of routing data– though “zones” can be manually created via BGP

prefixes lists• Traditional threshold based detection (although adaptive)

Connect. Communicate. CollaborateTool 3

• Per host behavioural analysis• rather complex “scoring” system to distinguish normal from

abnormal behaviour. Proprietary algorithms• Doesn’t use routing info

– though “zones” can be manually created via BGP prefixes lists

• Potentially attractive methodology• Concerns on scalability and accuracy with 1,000 sampling


lessons learnt and directions for research

• Manual validation is required to confirm/correct anomalies– More automatic intelligence to help this process– Fusion with other data sources (router logs?

Honeynets?)• Detection space of 3 tools often disjoint

– (Standard) anomaly injection• Operations need supported tools to support services• If choice is among published but “not a tool” or “secret but

supported and (claiming to) work” => risk to stick to those!– Fill the gap towards TOOLS!


Thank you!

[email protected]

Experiences with tools for network anomaly detection in the GÉANT2 core

Documents

Transcript of Experiences with tools for network anomaly detection in the GÉANT2 core