EGI-InSPIRE

EGI-CSIRT presentation

Adam Smutnicki

Wroclaw Centre for Networking and SupercomputingPoland

10 May 2012 36th TF-CSIRT Meeting 1EGI-InSPIRE RI-261323 www.egi.eu

European Grid Infrastructure

• a federation of over 350+ resource centres in 50+ contries• approx 400k compute cores• continuation of EGEE I–III projects• computing and storage resources for researchers• cooperation of European and national projects• in practice not only European countries but also, Americas,

Asia and Pacific

10 May 2012 36th TF-CSIRT Meeting 2EGI-InSPIRE RI-261323 www.egi.eu

EGI in the world

10 May 2012 36th TF-CSIRT Meeting 3EGI-InSPIRE RI-261323 www.egi.eu

EGI-CSIRT

• top level CSIRT team for all the European GridInfrastructure• formally operational since 01.05.2010• created based on OSCT from EGEE• TI listed team• distributed team consisting of NGI’s representatives• not purely a virtual team, we meet each other few times a

year

10 May 2012 36th TF-CSIRT Meeting 4EGI-InSPIRE RI-261323 www.egi.eu

EGI Security Structure

10 May 2012 36th TF-CSIRT Meeting 5EGI-InSPIRE RI-261323 www.egi.eu

EGI-CSIRT Teams

• IRTF — Incident Response Task Force• SMG — Security Monitoring Group• SDG — Security Drill Group• TDG — Training and Disemination Group

10 May 2012 36th TF-CSIRT Meeting 6EGI-InSPIRE RI-261323 www.egi.eu

EGI-CSIRT Teams

• IRTF — Incident Response Task Force• SMG — Security Monitoring Group• SDG — Security Drill Group• TDG — Training and Disemination Group

10 May 2012 36th TF-CSIRT Meeting 6EGI-InSPIRE RI-261323 www.egi.eu

EGI-CSIRT Teams

• IRTF — Incident Response Task Force• SMG — Security Monitoring Group• SDG — Security Drill Group• TDG — Training and Disemination Group

10 May 2012 36th TF-CSIRT Meeting 6EGI-InSPIRE RI-261323 www.egi.eu

EGI-CSIRT Teams

• IRTF — Incident Response Task Force• SMG — Security Monitoring Group• SDG — Security Drill Group• TDG — Training and Disemination Group

10 May 2012 36th TF-CSIRT Meeting 6EGI-InSPIRE RI-261323 www.egi.eu

Incident Response Task Force

• 14 actively participating, among 34 NGI’s• Vulnerability Assesment Team• incident handling and coordination• forensics• strong and good cooperation within group→ forensics

done by members for other NGI’s• good cooperation with EGI Software Vulnerability Group• direct communication with IM

10 May 2012 36th TF-CSIRT Meeting 7EGI-InSPIRE RI-261323 www.egi.eu

Security Monitoring

• Security Dashboard:• Pakiti• Nagios• metrics• stats

• monitoring of a distributed infrastructure on a system levelrather than network level• active monitoring with notifications• in a short time, we can easily deploy, a dedicated security

checks, to be run on all sites; e.g. when there is a newvulnerability• Security Intelligence Group• we are very close/part of our constituencies (NGI’s), so we

know them very well, we are focused on proactive actions

10 May 2012 36th TF-CSIRT Meeting 8EGI-InSPIRE RI-261323 www.egi.eu

Security Monitoring

• Security Dashboard:• Pakiti• Nagios• metrics• stats

• monitoring of a distributed infrastructure on a system levelrather than network level• active monitoring with notifications• in a short time, we can easily deploy, a dedicated security

checks, to be run on all sites; e.g. when there is a newvulnerability• Security Intelligence Group• we are very close/part of our constituencies (NGI’s), so we

know them very well, we are focused on proactive actions

10 May 2012 36th TF-CSIRT Meeting 8EGI-InSPIRE RI-261323 www.egi.eu

Security Monitoring

• Security Dashboard:• Pakiti• Nagios• metrics• stats

• monitoring of a distributed infrastructure on a system levelrather than network level• active monitoring with notifications• in a short time, we can easily deploy, a dedicated security

checks, to be run on all sites; e.g. when there is a newvulnerability• Security Intelligence Group• we are very close/part of our constituencies (NGI’s), so we

know them very well, we are focused on proactive actions

10 May 2012 36th TF-CSIRT Meeting 8EGI-InSPIRE RI-261323 www.egi.eu

Security Monitoring

• Security Dashboard:• Pakiti• Nagios• metrics• stats

• monitoring of a distributed infrastructure on a system levelrather than network level• active monitoring with notifications• in a short time, we can easily deploy, a dedicated security

checks, to be run on all sites; e.g. when there is a newvulnerability• Security Intelligence Group• we are very close/part of our constituencies (NGI’s), so we

know them very well, we are focused on proactive actions

10 May 2012 36th TF-CSIRT Meeting 8EGI-InSPIRE RI-261323 www.egi.eu

Security Monitoring

• Security Dashboard:• Pakiti• Nagios• metrics• stats

• monitoring of a distributed infrastructure on a system levelrather than network level• active monitoring with notifications• in a short time, we can easily deploy, a dedicated security

checks, to be run on all sites; e.g. when there is a newvulnerability• Security Intelligence Group• we are very close/part of our constituencies (NGI’s), so we

know them very well, we are focused on proactive actions

10 May 2012 36th TF-CSIRT Meeting 8EGI-InSPIRE RI-261323 www.egi.eu

Security Monitoring

• Security Dashboard:• Pakiti• Nagios• metrics• stats

• monitoring of a distributed infrastructure on a system levelrather than network level• active monitoring with notifications• in a short time, we can easily deploy, a dedicated security

checks, to be run on all sites; e.g. when there is a newvulnerability• Security Intelligence Group• we are very close/part of our constituencies (NGI’s), so we

know them very well, we are focused on proactive actions

10 May 2012 36th TF-CSIRT Meeting 8EGI-InSPIRE RI-261323 www.egi.eu

Security Dashboard

10 May 2012 36th TF-CSIRT Meeting 9EGI-InSPIRE RI-261323 www.egi.eu

Security Dashboard

10 May 2012 36th TF-CSIRT Meeting 9EGI-InSPIRE RI-261323 www.egi.eu

Security Drills

Separate presentation“Security Drills in a Grid Environment”on Friday 11.05 at 11:00 by Oscar Koeroo from Nikhef.

10 May 2012 36th TF-CSIRT Meeting 10EGI-InSPIRE RI-261323 www.egi.eu

Training and Disemination

• wiki with a lot of operational information• Security Training sessions for staff during project meeting,

there was a big interest• involved in GridKa School trainings in Karlsruhe• real case incident scenarios in preparation with SDG

10 May 2012 36th TF-CSIRT Meeting 11EGI-InSPIRE RI-261323 www.egi.eu

IRTF Operational actions

• 1 week duties with backups• continuous monitoring• critical vulnerabilities handling• preparing and distributing advisories• incident response coordination• well known, systematized security staff structure• all security and administrative contacts in a single

dedicated database• NGI SO (from IRTF) are the first point of contact with

shortest reaction time• even though some sites has their own security staff and

has access to all security tools, in practice CSIRTmembers “take care” about them

10 May 2012 36th TF-CSIRT Meeting 12EGI-InSPIRE RI-261323 www.egi.eu

Procedures

• Critical Vulnerability Handling• Incident Response and information distribution to all sites• information sharing model is implemented in procedures. . .

and is working

10 May 2012 36th TF-CSIRT Meeting 13EGI-InSPIRE RI-261323 www.egi.eu

Procedures

• Critical Vulnerability Handling• Incident Response and information distribution to all sites• information sharing model is implemented in procedures. . .

and is working

10 May 2012 36th TF-CSIRT Meeting 13EGI-InSPIRE RI-261323 www.egi.eu

Procedures

• Critical Vulnerability Handling• Incident Response and information distribution to all sites• information sharing model is implemented in procedures. . .

and is working

10 May 2012 36th TF-CSIRT Meeting 13EGI-InSPIRE RI-261323 www.egi.eu

Procedures

• Critical Vulnerability Handling• Incident Response and information distribution to all sites• information sharing model is implemented in procedures. . .

and is working

10 May 2012 36th TF-CSIRT Meeting 13EGI-InSPIRE RI-261323 www.egi.eu

Critical Vulnerability Handling

• used for vulnerabilities assessed critical by RiskAssessment Team, eg. local root exploit• all sites are checked against such vulnerability (monitoring

tools)• while patch not released, mitigations are suggested and

checked (if possible)• while patch available, site has 7 days to apply• for old, reoccurring vulnerabilities: 2 days to apply• vulnerabilities tend to reappear even 1.5 year after patch

released (eg. CVE 2010-3081)• death penalty: site suspension — works pretty well

10 May 2012 36th TF-CSIRT Meeting 14EGI-InSPIRE RI-261323 www.egi.eu

Critical Vulnerability Handling

• used for vulnerabilities assessed critical by RiskAssessment Team, eg. local root exploit• all sites are checked against such vulnerability (monitoring

tools)• while patch not released, mitigations are suggested and

checked (if possible)• while patch available, site has 7 days to apply• for old, reoccurring vulnerabilities: 2 days to apply• vulnerabilities tend to reappear even 1.5 year after patch

released (eg. CVE 2010-3081)• death penalty: site suspension — works pretty well

10 May 2012 36th TF-CSIRT Meeting 14EGI-InSPIRE RI-261323 www.egi.eu

Critical Vulnerability Handling

• used for vulnerabilities assessed critical by RiskAssessment Team, eg. local root exploit• all sites are checked against such vulnerability (monitoring

tools)• while patch not released, mitigations are suggested and

checked (if possible)• while patch available, site has 7 days to apply• for old, reoccurring vulnerabilities: 2 days to apply• vulnerabilities tend to reappear even 1.5 year after patch

released (eg. CVE 2010-3081)• death penalty: site suspension — works pretty well

10 May 2012 36th TF-CSIRT Meeting 14EGI-InSPIRE RI-261323 www.egi.eu

Critical Vulnerability Handling

• used for vulnerabilities assessed critical by RiskAssessment Team, eg. local root exploit• all sites are checked against such vulnerability (monitoring

tools)• while patch not released, mitigations are suggested and

checked (if possible)• while patch available, site has 7 days to apply• for old, reoccurring vulnerabilities: 2 days to apply• vulnerabilities tend to reappear even 1.5 year after patch

released (eg. CVE 2010-3081)• death penalty: site suspension — works pretty well

10 May 2012 36th TF-CSIRT Meeting 14EGI-InSPIRE RI-261323 www.egi.eu

Critical Vulnerability Handling

• used for vulnerabilities assessed critical by RiskAssessment Team, eg. local root exploit• all sites are checked against such vulnerability (monitoring

tools)• while patch not released, mitigations are suggested and

checked (if possible)• while patch available, site has 7 days to apply• for old, reoccurring vulnerabilities: 2 days to apply• vulnerabilities tend to reappear even 1.5 year after patch

released (eg. CVE 2010-3081)• death penalty: site suspension — works pretty well

10 May 2012 36th TF-CSIRT Meeting 14EGI-InSPIRE RI-261323 www.egi.eu

Critical Vulnerability Handling

• used for vulnerabilities assessed critical by RiskAssessment Team, eg. local root exploit• all sites are checked against such vulnerability (monitoring

tools)• while patch not released, mitigations are suggested and

checked (if possible)• while patch available, site has 7 days to apply• for old, reoccurring vulnerabilities: 2 days to apply• vulnerabilities tend to reappear even 1.5 year after patch

released (eg. CVE 2010-3081)• death penalty: site suspension — works pretty well

10 May 2012 36th TF-CSIRT Meeting 14EGI-InSPIRE RI-261323 www.egi.eu

Critical Vulnerability Handling

• used for vulnerabilities assessed critical by RiskAssessment Team, eg. local root exploit• all sites are checked against such vulnerability (monitoring

tools)• while patch not released, mitigations are suggested and

checked (if possible)• while patch available, site has 7 days to apply• for old, reoccurring vulnerabilities: 2 days to apply• vulnerabilities tend to reappear even 1.5 year after patch

released (eg. CVE 2010-3081)• death penalty: site suspension — works pretty well

10 May 2012 36th TF-CSIRT Meeting 14EGI-InSPIRE RI-261323 www.egi.eu

Incident Response

• CSIRT SO on duty is an incident coordinator (sometimesNGI SO)• NGI SO involved as a NGI level coordinator if needed (eg.

multisite incident)• site response time requirements• first guidelines, what kind of information need to be

checked/provided• all sites are informed constantly — updates send by CSIRT

SO on duty• final report required and circulated among all sites (not

only involved ones)• templates for reporting, updates and final report

10 May 2012 36th TF-CSIRT Meeting 15EGI-InSPIRE RI-261323 www.egi.eu

Incident Response

• CSIRT SO on duty is an incident coordinator (sometimesNGI SO)• NGI SO involved as a NGI level coordinator if needed (eg.

multisite incident)• site response time requirements• first guidelines, what kind of information need to be

checked/provided• all sites are informed constantly — updates send by CSIRT

SO on duty• final report required and circulated among all sites (not

only involved ones)• templates for reporting, updates and final report

10 May 2012 36th TF-CSIRT Meeting 15EGI-InSPIRE RI-261323 www.egi.eu

Incident Response

• CSIRT SO on duty is an incident coordinator (sometimesNGI SO)• NGI SO involved as a NGI level coordinator if needed (eg.

multisite incident)• site response time requirements• first guidelines, what kind of information need to be

checked/provided• all sites are informed constantly — updates send by CSIRT

SO on duty• final report required and circulated among all sites (not

only involved ones)• templates for reporting, updates and final report

10 May 2012 36th TF-CSIRT Meeting 15EGI-InSPIRE RI-261323 www.egi.eu

Incident Response

• CSIRT SO on duty is an incident coordinator (sometimesNGI SO)• NGI SO involved as a NGI level coordinator if needed (eg.

multisite incident)• site response time requirements• first guidelines, what kind of information need to be

checked/provided• all sites are informed constantly — updates send by CSIRT

SO on duty• final report required and circulated among all sites (not

only involved ones)• templates for reporting, updates and final report

10 May 2012 36th TF-CSIRT Meeting 15EGI-InSPIRE RI-261323 www.egi.eu

Incident Response

• CSIRT SO on duty is an incident coordinator (sometimesNGI SO)• NGI SO involved as a NGI level coordinator if needed (eg.

multisite incident)• site response time requirements• first guidelines, what kind of information need to be

checked/provided• all sites are informed constantly — updates send by CSIRT

SO on duty• final report required and circulated among all sites (not

only involved ones)• templates for reporting, updates and final report

10 May 2012 36th TF-CSIRT Meeting 15EGI-InSPIRE RI-261323 www.egi.eu

Incident Response

• CSIRT SO on duty is an incident coordinator (sometimesNGI SO)• NGI SO involved as a NGI level coordinator if needed (eg.

multisite incident)• site response time requirements• first guidelines, what kind of information need to be

checked/provided• all sites are informed constantly — updates send by CSIRT

SO on duty• final report required and circulated among all sites (not

only involved ones)• templates for reporting, updates and final report

10 May 2012 36th TF-CSIRT Meeting 15EGI-InSPIRE RI-261323 www.egi.eu

Incident Response

• CSIRT SO on duty is an incident coordinator (sometimesNGI SO)• NGI SO involved as a NGI level coordinator if needed (eg.

multisite incident)• site response time requirements• first guidelines, what kind of information need to be

checked/provided• all sites are informed constantly — updates send by CSIRT

SO on duty• final report required and circulated among all sites (not

only involved ones)• templates for reporting, updates and final report

10 May 2012 36th TF-CSIRT Meeting 15EGI-InSPIRE RI-261323 www.egi.eu

Incidents info/stats (1)

• our incidents can spread very quickly through “leaf” sites indifferent countries, belonging to different NREN’s anddifferent jurisdictions• spread easily with compromised users’ credentials• since 05.2010 we had 18 incidents, most of them single

site• incidents due to: stolen/week passwords, unprotected ssh

keys, vulnerable services open to the world and unpatchedsoftware. . .

10 May 2012 36th TF-CSIRT Meeting 16EGI-InSPIRE RI-261323 www.egi.eu

Incidents info/stats (1)

• our incidents can spread very quickly through “leaf” sites indifferent countries, belonging to different NREN’s anddifferent jurisdictions• spread easily with compromised users’ credentials• since 05.2010 we had 18 incidents, most of them single

site• incidents due to: stolen/week passwords, unprotected ssh

keys, vulnerable services open to the world and unpatchedsoftware. . .

10 May 2012 36th TF-CSIRT Meeting 16EGI-InSPIRE RI-261323 www.egi.eu

Incidents info/stats (1)

• our incidents can spread very quickly through “leaf” sites indifferent countries, belonging to different NREN’s anddifferent jurisdictions• spread easily with compromised users’ credentials• since 05.2010 we had 18 incidents, most of them single

site• incidents due to: stolen/week passwords, unprotected ssh

keys, vulnerable services open to the world and unpatchedsoftware. . .

10 May 2012 36th TF-CSIRT Meeting 16EGI-InSPIRE RI-261323 www.egi.eu

Incidents info/stats (1)

• our incidents can spread very quickly through “leaf” sites indifferent countries, belonging to different NREN’s anddifferent jurisdictions• spread easily with compromised users’ credentials• since 05.2010 we had 18 incidents, most of them single

site• incidents due to: stolen/week passwords, unprotected ssh

keys, vulnerable services open to the world and unpatchedsoftware. . .

10 May 2012 36th TF-CSIRT Meeting 16EGI-InSPIRE RI-261323 www.egi.eu

Incidents info/stats (2)

• . . . so far none related to grid technologies/credentials• in most of the cases an attacker is not aware, what kind of

infrastructure he was able to penetrate• it is important to have good relationships with NREN

CSIRT’s• in one case, attackers were caught by LE: dwaan and xS

(KPN incident)

10 May 2012 36th TF-CSIRT Meeting 17EGI-InSPIRE RI-261323 www.egi.eu

Incidents info/stats (2)

• . . . so far none related to grid technologies/credentials• in most of the cases an attacker is not aware, what kind of

infrastructure he was able to penetrate• it is important to have good relationships with NREN

CSIRT’s• in one case, attackers were caught by LE: dwaan and xS

(KPN incident)

10 May 2012 36th TF-CSIRT Meeting 17EGI-InSPIRE RI-261323 www.egi.eu

Incidents info/stats (2)

• . . . so far none related to grid technologies/credentials• in most of the cases an attacker is not aware, what kind of

infrastructure he was able to penetrate• it is important to have good relationships with NREN

CSIRT’s• in one case, attackers were caught by LE: dwaan and xS

(KPN incident)

10 May 2012 36th TF-CSIRT Meeting 17EGI-InSPIRE RI-261323 www.egi.eu

Incidents info/stats (2)

• . . . so far none related to grid technologies/credentials• in most of the cases an attacker is not aware, what kind of

infrastructure he was able to penetrate• it is important to have good relationships with NREN

CSIRT’s• in one case, attackers were caught by LE: dwaan and xS

(KPN incident)

10 May 2012 36th TF-CSIRT Meeting 17EGI-InSPIRE RI-261323 www.egi.eu

Incident Response workflow

One may see our response scheme as:Site→ NGI CSIRT→ NREN CSIRT→ other NREN’s, NGI’sand Sites

In practice:Site→ NGI CSIRT→ EGI-CSIRT→ other NGI’s and Sitesor even:Site→ NGI CSIRT→ Other NGI CSIRT

10 May 2012 36th TF-CSIRT Meeting 18EGI-InSPIRE RI-261323 www.egi.eu

Incident Response workflow

One may see our response scheme as:Site→ NGI CSIRT→ NREN CSIRT→ other NREN’s, NGI’sand Sites

In practice:Site→ NGI CSIRT→ EGI-CSIRT→ other NGI’s and Sites

or even:Site→ NGI CSIRT→ Other NGI CSIRT

10 May 2012 36th TF-CSIRT Meeting 18EGI-InSPIRE RI-261323 www.egi.eu

Incident Response workflow

One may see our response scheme as:Site→ NGI CSIRT→ NREN CSIRT→ other NREN’s, NGI’sand Sites

In practice:Site→ NGI CSIRT→ EGI-CSIRT→ other NGI’s and Sitesor even:Site→ NGI CSIRT→ Other NGI CSIRT

10 May 2012 36th TF-CSIRT Meeting 18EGI-InSPIRE RI-261323 www.egi.eu

Links

EGI: http://www.egi.eu

EGI-CSIRT: https://wiki.egi.eu/csirt

10 May 2012 36th TF-CSIRT Meeting 19EGI-InSPIRE RI-261323 www.egi.eu

EGI CSIRT

Questions ?

10 May 2012 36th TF-CSIRT Meeting 20EGI-InSPIRE RI-261323 www.egi.eu