Collecting and processing of data from security tools in CESNET

CESNET, z. s. p. o.

Andrea Kropáčováandrea@cesnet.cz

CNMS2016, Prague, 25 Apr

● Operates Czech Educational and Scientific Network● Established in 1996● Has 27 members and ~300 participants ● Main goals:

– research and development of information and communication technologies

– construction and development of e­CESNET infrastructure for research and education

– promotion and dissemination of education, culture and knowledge

● 2011 – 2015: Project „Big infrastructure CESNET“● 2015 – 2020: Project „E­infrastructure CESNET“● Operates security teams CESNET­CERTS

– accredited team by Trusted Introducer in 2008 (established in 2003)

– responsible for solving security issues in CESNET2 (AS2852)

– http://csirt.cesnet.cz/, abuse@cesnet.cz

https://www.cesnet.cz/

CNMS2016, Prague, 25 Apr

Network management

Transparent

No restriction of legitimate traffic (until the problem came up)

Connected networks can use tools developed and operated by CESNET for their own self­protection and self­regulation

1)

2)

3)

CESNET2

- HW accelerated probes- large scale (backbone-wide) flow based monitoring (NetFlow data sources)- Honey Pots- IDS, IPS, tar pit based systems, etc.. - SNMP based monitoring

CNMS2016, Prague, 25 Apr

Begining ... ● Administrators often run their own IDS`, security probes, central 

syslog, honeypots, IPS ...– For networks and services monitoring– Finding compromised machines, botnet activity, malware, antispam– Detection of networks anomalies and attacks ...

● Problem – they pick just the data important for them, what to do with the rest?– Throw away?

➔ Noo, it is information wasting ...– Make report?

➔ Too much work ... recipients may need help, another information ...

➔ How? Data format? Protocol? Data clasification? Protection? Policy?

             

SHARE!!!

CNMS2016, Prague, 25 Apr

S● System for efficient information sharing● Client/server architecture (transport, not storage)● Community (aka „let's build security together")

– Reciprocity – all your data is available to the whole Warden community...

– … and all the community data is available to you

● Sending and receiving clients● Format: IDEA (https://idea.cesnet.cz)

● Protocol: JSON/HTTPS

● Sec/auth: TLS/X509

● Platform: Python/WSGI

● Bulk operations, incoming filtering

● Security (X509, encryption, “sanity” checks, peer review)                                                               

Dionea

Kippo

Dionea

IDSLaBrea NEMEA 3rd

Shadow, N6,X2, X4

NSHARP

FTASCESNET-CERTS

NOCPSSCSIRT.SK

VŠBVUTBR

Dionea

Kippo

Data flow (sending client)

Data flow (download client)

CNMS2016, Prague, 25 Apr

Lesson learned I

Connected organizations do not have sufficient human resources to use the open community approach to 

=

They can not download and processe data themselves.

But they want to obtain this data, the data is useful

=

it is necessary to deliver the processed data.

                                                                     

Dionea

Kippo

Dionea

IDSLaBrea NEMEA 3rd

Shadow, N6,X2, X4

NSHARP

FTASCESNET-CERTS

NOCPSS

Data flow (sending client)

Data flow (download client)

CSIRT.SKVŠB

VUTBR

Dionea

Kippo

CNMS2016, Prague, 25 Apr

S

                                                                     

CNMS2016, Prague, 25 Apr

● Mentat is downloading client in Warden architecture.

● SIEM

● Data storage

● Divides events according to end networks (creating reports)

● Send reports to the end networks (abuse @ ...)

– RIPE DB

CNMS2016, Prague, 25 Apr

Lesson learned II

● Too little info, we do not know what to do.

● I do not want a e­mail report, I wont structured data format.

● How is the severity? 

● We do not want this information, we get it from the source.

● Data from 3rd parties have different quality

● NAT, FW, DHCP …

● Big networks like universities … 

– we must divide the information in the report and create the new reports.

● Why do I receive the same report? I solved it yesterday.

                                                                     

                                     ... report recipients say ...

CNMS2016, Prague, 25 Apr

IDEA Format

● JSON (NoSQL friendly), but mostly flat and typed structure (SQL friendly)● Extensibility (producers can use their own keys and tags)● Marking of anonymised, imprecise, forged data● Able to distinct third party events, correlated events, updated/referenced events● Taxonomies (mkII categories, tag based Source/Target/Detector description)● https://idea.cesnet.cz

CNMS2016, Prague, 25 Apr

S

                                                                     

CNMS2016, Prague, 25 Apr

S

                                                                     

CNMS2016, Prague, 25 Apr

S

                                                                     

CNMS2016, Prague, 25 Apr

Filtering● End­networks admin may set up reporting

– Ignore one IP address– Ignore one source of data– Ignore some types of events

CNMS2016, Prague, 25 Apr

                                                                     inetnum 147.32.1.0 – 147.32.50.255remarks Report network abuse --> abuse@x.cvut.cz

inetnum 147.32.1.0 – 147.32.50.255netname CVUT-TCZdescr Praha 1remarks Report network abuse --> abuse@x.cvut.cz

inetnum 147.32.1.0 – 147.32.50.255netname CVUT-TCZdescr Praha 1remarks Report network abuse --> abuse@p1.cvut.cz

inetnum 147.32.60.0 – 147.32.100.255netname CVUT-TCZdescr Praha 6remarks Report network abuse --> abuse@p6.cvut.cz

inetnum 147.32.101.0 – 147.32.150.255netname CVUT-TCZdescr Praha 10remarks Report network abuse --> abuse@p10.cvut.cz

inetnum 147.32.160.0 – 147.32.180.255netname CVUT-TCZdescr Praha 8remarks Report network abuse --> abuse@p8.cvut.cz

inetnum 147.32.200.0 – 147.32.220.255netname CVUT-TCZdescr Praha 6remarks Report network abuse --> abuse@p66.cvut.cz

CNMS2016, Prague, 25 Apr

Lesson learned III

● We can gather data into one place and report them.

● BUT!

– Share primary data (via report) is not enough!

– Data obtained from security tools in one network is not enough! 

– Share data in one and from one network is not sufficient!

● Why?

– Primary data are many and have different information value.

– We do no see some problems. 

– Missing context, we do not see the big picture.

                                                                     

                                     ... present & future ...

CNMS2016, Prague, 25 Apr

What next?● New and more sources of primary data in CESNET.● New and more sources of primary data out of CESNET.● New source from 3rd parties.● Better validation and classification.● Data enrichment.● Inteligent analysis and data correlation.● Information and data sharing at national and international level.

„ ... more, better, faster...“

CNMS2016, Prague, 25 Apr

SABU

● Project “Sharing and Analysis of Security Events“

● 2016 – 2020, funded by Ministry of Interior of Czech Republic 

● CESNET, Masaryk University in Brno

● https://sabu.cesnet.cz – in development

● sabu­info@cesnet.cz

● Partners:

– CSIRT.SK

– ISP

– Bank sector

– Invea Technologies

(Sdílení a analýza bezpečnostních událostí)

CNMS2016, Prague, 25 Apr

CNMS2016, Prague, 25 Apr

Thank you for your attention!