Keeping EGI secureEGI CSIRT: Prevention - Response - Training

EGI is a federated e-Infrastructure set up to provide advancedcomputing, storage and data services for research and innovation.

The EGI e-Infrastructure is publicly-funded and comprises almost 300data centres and cloud providers spread across Europe and worldwide.

www.egi.eu

Keeping EGI secureEGI CSIRT: Prevention - Response - Training

Meet the EGI CSIRTComputer Security and Incident Response Team

The EGI CSIRT builds on networks ofexperts.

The team is made up of securityofficers and experts distributed overseveral countries and based atnational e-Infrastructures (oftenabbreviated as NGIs) and oneEuropean research organisation(CERN). They work together to keepthe e-Infrastructure safe, makingsure that disreputable types don'tfind a red carpet on the way in.

The EGI CSIRT is led and coordinatedby the EGI Security Officer, whoserole and mission is defined by thesecurity policies approved by the EGIFederation.

EGI CSIRT | 01

EGI CSIRT is certified by TrustedIntroducer since October 2014.

Trusted Introducer is a communityof about 300 CSIRT teams from largescale organisations classifiedaccording to three levels: listed,accredited and certified. The EGICSIRT was the 5th team to achievethe top certified status.

The team provides operationalsecurity for the EGI infrastructure.

The EGI CSIRT looks after the EGIFederation of data centres as well asits infrastructure components andusers. Following key principles ofEGI, the data centres are bound bythe policies and procedures of theEGI security framework, whileretaining operational autonomy.

A large part of the EGI CSIRT work isto ensure that the procedures arefollowed and that every data centreknows what to do and when.

This means that theentire EGI CSIRT, itsprocedures, policies& operations werepositively evaluatedafter an external peerreview.

EGI CSIRT activities cover:> prevention of security incidents(security monitoring, vulnerabilityhandling, risk assessment andmitigation),> incident response (digital forensicsand mitigation) and> security training.

The team relies on the security staffat the EGI federated data centres,national e-Infrastructures and CERNto keep EGI secure for the benefit ofits users.

EGI CSIRT ensures effective securitycoordination within the EGI Fede-ration and liaises with other e-Infrastructures (e.g. EUDAT, PRACE,XSEDE) and the National Researchand Education Networks (NRENs).

EGI CSIRT | 02

ActivitiesWhat do we do?

Day-to-day security operations arehandled by the Incident ResponseTask Force - a small team of abouthalf a dozen security officersworking across countries andorganisations. These are the guys &the gals on watch who pick up thephone and act as first responders toreports of security incidents withinEGI. They are volunteers working inshifts and you will find one on duty24/7, holidays included.

The Security Monitoring Groupmaintains a monitoring frameworkthat collects and evaluates data fromall EGI federated data centres. Thisframework enables the EGI CSIRT tocheck basic security characteristicsof services, including software

patching levels and deployedmitigations.

The Security Drills Group tests theincident response capabilities of theEGI federated data centres andimproves collaboration among thedistributed teams. Because you cannever get too comfortable, the grouporganises drills where a red team"attacks" the infrastructure while theblue team "defends" it using andtesting incident response tools andexisting procedures.

The Training and DisseminationGroup collects best practices andorganises different types of eventsto train security staff across the EGIFederation.

Activity groups

EGI CSIRT | 03

The EGI Software VulnerabilityGroup works closely with the EGICSIRT to minimise risks to the e-infrastructure arising from softwarevulnerabilities.

This group investigates potential andexisting software vulnerabilities,their relevance to EGI and assessesthe risk to the EGI infrastructure.When appropriate, the group issuesadvisories to the EGI federated datacentres.

The larger part of the work of the EGICSIRT is to prevent security incidents.The EGI Software Vulnerability Groupassists in this aim, by handlingvulnerabilities reported in thesoftware used in the infrastructure.

Software vulnerabilities may bereported by anyone. Vulnerabilitiesin operating systems are oftenannounced by the technologydistributors and flagged by systemadministrators across the federation.Vulnerabilities in software producedby collaborating projects andinstitutes are likely to be reported bydevelopers, group members andother security experts in EGI.

After a vulnerability is found, thegroup investigates if it has potentialto become a problem to the EGI

EGI CSIRT | 04

PreventionKeeping the bad guys out

infrastructure and, if yes, what is therisk involved. During riskassessment, the vulnerability isplaced the in one of the categories:'Critical', 'High', 'Moderate' or 'Low'.

If deemed necessary, the SoftwareVulnerability Group will issue anadvisory with recommended actionsfor data centres to fix the problembefore someone decides to exploitthe vulnerability for selfish interest.

Vulnerabilities can also be detectedby monitoring. The EGI CSIRT deploysa framework to monitor softwarevulnerabilities with potential to affectthe infrastructure. The monitoringframework collects informationabout packages installed on computenodes available to users and detectsmissing software updates.

Once a Critical vulnerability isdetected by the monitoring, the EGICSIRT contacts the centre asking foran action, usually to apply therequested updates. Given the largenumber of sites and machine, theEGI CSIRT quite often fightregressions of vulnerabilities thatshow up repeatedly.

Thank you!

The EGI CSIRT thanks the goodsouls who found and reportedvulnerabilities to us.

It's appreciated!

EGI CSIRT | 05

If you find a vulnerability

Please report it to the SoftwareVulnerability Group:report-vulnerability@egi.eu.

Please do not discuss thevulnerability in open forums orblogs, as this may compromiseour mitigation strategies.

EGI CSIRT | 06

Vulnerabilities in numbers

More information:Some vulnerabilities are straightforward and can be handled very quickly,some need special treatment. For instance, CVE-2016-5195 is a severevulnerability in the Linux Kernel. Fixes were not easy to provide nor applyand many data centres opted to apply a mitigating configuration and,therefore, getting rid of the vulnerability completely took a longer time.TL;DR: even several hundreds of vulnerable data centres can be fixed withintwo weeks.

Security incidents often happenwhen someone exploits a vulnera-bility to gain unauthorised access tothe computing infrastructure.Intrusions are usually detected bythe EGI data centres and reported tothe EGI CSIRT who then coordinatesthe investigation and response.

The work starts with looking forclues and leads, just like what we seeat the beginning of crime-orientedtelevision episodes. Crime sceneinvestigators use forensic evidenceto understand what happened. EGICSIRT prefers logs and files tofingerprints and exotic fibres, but thegoals are similar.

Digital forensics is about investigatingevidence and leads to understand

EGI CSIRT | 07

ResponseWhen rainbows hit the fan

who the culprits are, what they areafter, how they got access to thesystem, and how their malwareworks.

Analysing malware helps to under-stand a particular attack but it canalso provide remote scanning tools,Indicators of Compromise and,potentially, internet addresses ofinfected systems that will help thecommunity to react to the threatand, if possible, track the maliciousintruder.

Security incidents often span multipleorganisations and country borders.EGI CSIRT relies on established linksand trusted relationships with thecommunity to collect and evaluateall the available information about

Number of incidentssince 2010

All incidents were successfully andquickly contained, thanks to the

security coordination that EGI CSIRTprovides across the federation.

an incident. Good communicationlines between EGI CSIRT and thedata centres is the secret weaponagainst large-scale incidents.

EGI CSIRT | 08

Case Study: VENOM rootkit

The VENOM rootkit was a malware designed to maintain unrestrictedand permanent access to compromised Linux systems.

The attackers, having obtained root access through other means, wereable to deploy a malicious kernel module and a userland binary, andconfigure them to be loaded upon the boot of the machine. Thismalware provided several features such as execution of an interactiveshell or manipulating files on the compromised system.

When the VENOM rootkit was detected in a server at an EGI data centrein December 2016, the EGI CSIRT moved into the digital crime scene. Byreverse engineering the malware, they were able to understand how themalware operated and, more importantly, what was its backdoormechanism. This allowed the team to test suspicious systems andprovide guidance how to check if the rootkit was installed.

This analysis lead to the discovery of an additional 25 servers outsidethe EGI Federation where VENOM was active but not yet detected.

Keeping the EGI infrastructuresecure requires an understanding ofattack and defence techniques thatgoes beyond the average skillset ofsystem administrators.

Security training is vital to guaranteethat local teams are able to useavailable information for a completeincident response.

Plugging the knowledge gaps is partof the EGI CSIRT mission and overthe years the team has worked toget the EGI community ready foraction. The EGI CSIRT has a diversecatalogue of training modules,developed by or the team or bypartner institutions.

EGI CSIRT | 09

TrainingPractice makes perfect

EGI CSIRT training sessions

Digital forensics training is all aboutfinding clues to understand whathappened. The teams look into thelogs and the files of a compromisedsystem and learn how to spot theorigin of the attack and what werethe weak points explored by theattackers.

Roleplay training brings it alltogether. The participants aredivided into teams and enact anincident inspired by real life. Theteams play all incident-responseroles, from site admins to managers,to learn that incident response is notonly about technical know-how butalso how it relies on effectivecommunications.

Defensive training unleashes anincident in a controlled environmentto test and improve defence skills.Based on actual attacks, the trainingis intended to look as realistic aspossible to prepare the teams forreal life attacks and familiarise themwith response procedures.

Offensive training turns the worldupside down and asks the securityteams to go on the attack. The teamslearn about attacking tools, how tospot weak points and how to disguiseone's tracks. By thinking as an attacker,they will know better what to expectduring incidents. The trainingmodule was provided by MasarykUniversity and its CSIRT team.

Keeping the EGI infrastructuresecure requires an understanding ofattack and defence techniques thatgoes beyond the average skillset ofsystem administrators.

Security training is vital to guaranteethat local teams are able to useavailable information for a completeincident response.

Plugging the knowledge gaps is partof the EGI CSIRT mission and overthe years the team has worked toget the EGI community ready foraction. The EGI CSIRT has a diversecatalogue of training modules,developed by or the team or bypartner institutions.

EGI CSIRT | 10

Training events attendants

Defensive Best practices for managing system security on Linux systems (ISGC 2015) * 18

Defensive Security training for Cloud Providers, VM admins, maintainers(EGI Community Forum 2015) *

15

Defensive Incident response and federated identity management (ISGC 2016) 15

Forensics Penetration testing hands-on training (UK HEP SYSMAN) 20

Forensics Security incident detection, analysis and handling in the world of Federated Cloudservices and distributed computing infrastructures (ISGC 2017)

18

Offensive Security session, Offensive Training (France Grilles Annual Meeting 2016) ** 12

Offensive Security Training (EGI Conference 2015) ** 30

Defensive Best practices for managing system security on Linux systems in Grids and Clouds(ISGC 2015)

18

Roleplay Federated AAI meets reality, Security Incident Handling Role Play(Trusted Introducer meeting 2016)

15

Roleplay Federated AAI meets reality, Security Incident Handling Role Play (DI4R 2016) 25

* with Nixon Security; ** with Masaryk University

Federated Cloud Security

EGI will continue to develop its cloudservices portfolio in the comingyears and Federated Cloud securitywill have to evolve accordingly.

The EGI Federated Cloud is achallenging environment because it'snot very forgiving for the users. Wecannot expect researchers to besystem experts, we cannot count onthem being aware of all the risks,vulnerabilities and procedures, andwe cannot blame them for this.

The challenge will be to make theFederated Cloud securely user-proof.It will be a major effort of operationsmanagement but there is precedentin commercial companies and thetechnology is available.

EGI CSIRT | 11

Future challengesNo rest for the wicked

Compromised federated identities

As Authentication and Authorisationmethods for large user communitiesmove towards federated accessmodels, how do we handle individualproblem users?

While this evolution is welcome, wehave to make sure that we are stillable to do the basic steps in incidentresponse, for example suspending acompromised account.

The challenge will be to retain thecapacity of responding to incidentspromptly in an increasingly complexlandscape where more and moreidentity providers are involved.

Widening collaborations

The next few years will see EGIwidening its collaboration andintegration with EUDAT and othere-Infrastructures. This will bringhuge benefits for the researcherswho currently rely on separateservice catalogues for their work.

The challenge will be to work closelywith other security teams toharmonise operational securitypolicies and procedures into acoherent set.

Contacts

Website:https://csirt.egi.eu

EGI Security Officer:Sven Gabriel (NIKHEF)

To report a vulnerability:report-vulnerability@egi.eu(please don't discuss it in openforums)

To report an incident:> EGI data centres : followhttps://wiki.egi.eu/wiki/SEC01> Everyone else : abuse@egi.eu

EGI CSIRT | 12

Acknowledgements

This publication was prepared by the EGI CSIRT and theCommunications Team of the EGI Foundation.

The EGI CSIRT is a coordination service of the EGIFederation funded by the EGI Council and contributions ofthe institutions represented in the team.

Copyright: EGI-Engage Consortium, Creative CommonsAttribution 4.0 International License.

The EGI-Engage project is co-funded by the EuropeanUnion (EU) Horizon 2020 program under grant 654142.

The content of this publication is correct to the best of ourknowledge as of July 2017.

www.egi.eu