THE ROLE OF CATALOGUES OF THREATS AND SECURITY … · 5.0 All Subjects Factors Mean quality of...
4
TUTORIAL ON GRC FOR SERVICES 2011 COPYRIGHT UNIVERSITY OF TRENTO 1 THE ROLE OF CATALOGUES OF THREATS AND SECURITY CONTROLS IN SECURITY RISK ASSESSMENT: AN EMPIRICAL STUDY WITH ATM PROFESSIONALS Joint work with M. De Gramatica, K. Labunets, F. Paci, A. Tedeschi Fabio Massacci University of Trento – Italy securitylab.disi.unitn.it THE PROBLEM ENAV – national large air traffic management authority • New cyber operational concepts (eg Remotely Operated Tower) • Must identify cyber security measures • Lots of domain specialists but few security experts Use a Risk Based Methodology, but which one? ISACA’s CoBIT, SABSA focus on Business Goals US NIST 800-53, UK’s IAS focus on Threats Eurocontrol or SESAR’s SecRAM focus on Assets Who should execute the methodology? Ask security experts they are expensive and in high demand. Use a threat/controls catalogue which one? is bigger = better? • ISO27002 general measures • German’s BSI general measures with specific details • Eurocontrol’s Risk Toolkit specific to Air Traffic Management 09/04/15 2 Fabio Massacci - REFSQ - ESSEN WHAT IS A SECURITY RISK ASSESSMENT METHOD? A SRA method examines system’s security threats proposes set of system changes (security measures, controls, requirements) to bring system within acceptable risk Example statements • X helps enterprises create optimal value from IT by maintaining a balance between realising benefits and optimising risk levels and resource use. • Y is a proven methodology for developing business-driven, risk and opportunity focused Security Architectures • W collect business requirements from risk owners and budget holders. Abstract them in business-language into business drivers for security then execute and measure value • The aim and purpose of Z is to analyse a proposed or existing system to identify risks and estimate the levels of those risks; Select appropriate controls to manage the treatable risks. 09/04/15 3 Fabio Massacci - REFSQ - ESSEN SIZE MATTERS… German’s IT-Grundschutz Catalogue (aka BSI) Intro 40 pages Assets 350 pages Threats 1.000 pages Controls 3.000 pages Eurocontrol’s ATM Security Risk Management Toolkit Guidance Material 100 pages ATM specific Threats 57 pages ATM specific controls 99 pages (pre 72 + 27 post) Remotely Operated Tower Scenario Operational Focus Area Description 100+ pages Short “essential” description 24 pages 09/04/15 4 Fabio Massacci - REFSQ - ESSEN
Transcript of THE ROLE OF CATALOGUES OF THREATS AND SECURITY … · 5.0 All Subjects Factors Mean quality of...
TUTORIAL ON GRC FOR SERVICES 2011
COPYRIGHT UNIVERSITY OF TRENTO 1
THE ROLE OF CATALOGUES OF THREATS AND SECURITY CONTROLS
IN SECURITY RISK ASSESSMENT: AN EMPIRICAL STUDY WITH ATM
PROFESSIONALS Joint work with M. De Gramatica,
K. Labunets, F. Paci,
A. Tedeschi
Fabio Massacci University of Trento – Italy
securitylab.disi.unitn.it
THE PROBLEM
q ENAV – national large air traffic management authority • New cyber operational concepts (eg Remotely Operated Tower) • Must identify cyber security measures • Lots of domain specialists but few security experts
q Use a Risk Based Methodology, but which one? ü ISACA’s CoBIT, SABSA à focus on Business Goals ü US NIST 800-53, UK’s IAS à focus on Threats ü Eurocontrol or SESAR’s SecRAM à focus on Assets q Who should execute the methodology? ü Ask security experts à they are expensive and in high demand. ü Use a threat/controls catalogue àwhich one? is bigger = better? • ISO27002à general measures • German’s BSI à general measures with specific details • Eurocontrol’s Risk Toolkit à specific to Air Traffic Management
09/04/15 2 Fabio Massacci - REFSQ - ESSEN
WHAT IS A SECURITY RISK ASSESSMENT METHOD? q A SRA method ü examines system’s security threats ü proposes set of system changes (security
measures, controls, requirements) ü to bring system within acceptable risk q Example statements
• X helps enterprises create optimal value from IT by maintaining a balance between realising benefits and optimising risk levels and resource use.
• Y is a proven methodology for developing business-driven, risk and opportunity focused Security Architectures
• W collect business requirements from risk owners and budget holders. Abstract them in business-language into business drivers for security then execute and measure value
• The aim and purpose of Z is to analyse a proposed or existing system to identify risks and estimate the levels of those risks; Select appropriate controls to manage the treatable risks.
09/04/15 3 Fabio Massacci - REFSQ - ESSEN
SIZE MATTERS…
q German’s IT-Grundschutz Catalogue (aka BSI) ü Intro à 40 pages ü Assets à 350 pages ü Threats à 1.000 pages ü Controls à 3.000 pages q Eurocontrol’s ATM Security Risk Management Toolkit ü Guidance Material à 100 pages ü ATM specific Threats à 57 pages ü ATM specific controls à 99 pages (pre 72 + 27 post) q Remotely Operated Tower Scenario ü Operational Focus Area Description à 100+ pages ü Short “essential” description à 24 pages
09/04/15 4 Fabio Massacci - REFSQ - ESSEN
TUTORIAL ON GRC FOR SERVICES 2011
COPYRIGHT UNIVERSITY OF TRENTO 2
RESEARCH QUESTIONS
q How do we evaluate a method with a catalogue? ü A clinical procedure with a physician’s desk reference (aka drug index)
à We know well how to evaluate it àclinical trial protocol ü A risk assessment procedure with security catalogue à Same idea q Research question: do catalogues make a difference? ü Evaluate 15 ATM Professionals applying an ATM Risk Method • Domain Security Experts without catalogue • Domain Experts with a generic catalogue (BSI) • Domain Experts with a specific catalogue (Eurocontrol)
ü We apply a trial protocol to estimate the efficacy of a methodology with and without catalogues
09/04/15 5 Fabio Massacci - REFSQ - ESSEN
OUR EXPERIMENTAL PROTOCOL
09/04/15 6 Fabio Massacci - REFSQ - ESSEN
PROTOCOL ACTORS
q Participant ü Important to have both students (novices to the treatment but unbiased
opinion) and practitioners (expert but may have prejudices on what it works). • Students à preliminary pilot • ATM Practitioners à THIS Paper at REFSQ 2015
q Designer à Expert in the method ü Provide the best possible training for the method. • Security Trainer at Eurocontrol
q Customer à Expert in the scenario ü Indipendent validation of quality of results (irrespective of treatment!) • Any method can produce “enough” security requirements if quality doesn’t matter.
ü Expert in method ≠ Expert in domain à former may give good score if method is followed à bias
09/04/15 7 Fabio Massacci - REFSQ - ESSEN
q Training Participants ü Designer(s) train on treatment ü Customer(s) describe scenario q Application ü Participants apply treatment for
a time span that is sufficiently long (>1day) to be challenging
q Evaluation ü Customers evaluate results of
participants ü Participants tell their opinion on
how the experiment went ü Designers evalute results
09/04/15 8 Fabio Massacci - REFSQ - ESSEN
q No initial learning bias ü Avoid my stuff vs competitor’s stuff ü Not wrong focus q Can test actual efficacy ü Experiments <1h too short to tell
results apart ü With large catalogues/scenarios 1h
not enough even to browse docs q Measures different things ü Customers à actual efficacy ü Participants à perceived efficacy ü Designers should only evaluate
compliance
PROTOCOL STEPS
TUTORIAL ON GRC FOR SERVICES 2011
COPYRIGHT UNIVERSITY OF TRENTO 3
PROTOCOL MEASUREMENTS
q Actual Efficacy ü Participants Reports • Quantitative (#threats/controls) à easy to generate huge numbers (of junk) • Qualitative Analysis àlikert scale
q Perceived Efficacy ü Questionnaires à likert scale q Qualitative analysis ü Post-it notes • Affinity Analysis
ü Focus Groups Interviews • Coding, qualitative analysis
09/04/15 9 Fabio Massacci - REFSQ - ESSEN
q Actual Efficacy - AE ü whether the treatment improves
performance of the task q Perceived Efficacy – PE ü Perceived Ease Of Use – PEOU • the degree to which a person
believes that using a treatment would be free of effort
ü Perceived Usefulness – PU • the degree to which a person
believes that a treatment will be effective in achieving its intended objectives
q Qualitative Feedback 09/04/15 10 Fabio Massacci - REFSQ - ESSEN
q AE Null Hypothesis ü No difference between the
treatements in identified risk/controls measured as • 5 point scale of expert evalutaion
q PE Null Hypothesis ü No difference between the
perceived efficacy (PEOU, PU) by the participants measured as • 5-point scale on questionnaire
about identifying threats • 5-point scale on questionnaire
about identifying security measures
PROTOCOL MEASUREMENTS - II
ACTUAL EFFECTIVENESS
q Quality of identified threats and controls
q Median Scores (Controls): Expert+No Cat NoExpert+ATM Cat ü Actual Efficacy à 3.0 vs 3.5 ü Perceived Ease of Use à 4.0 vs 3.0 q Summary ü No catalogues slightly worse results + Catalogues slightly worse to use ü No statistical diff q Domain Experts + Security Catalogues ≈ (Domain + Security) Experts
09/04/15 11 Fabio Massacci - REFSQ - ESSEN
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0All Subjects
Factors
Mea
n qu
ality
of i
dent
ified
Thr
eats
P01
P02
P03
P04
P05
P06
P07
P08
P09
P10P11
P12
P13
P14P15
P16DGCAT
DSCAT
NOCAT
ID Catalog1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0All Subjects
Factors
Mea
n qu
ality
of i
dent
ified
Sec
. Ctrl
s
P01
P02
P03
P04
P05P06
P07
P08P09
P10
P11
P12
P13P14
P15
P16
DGCAT
DSCAT
NOCAT
ID Catalog
q Key features emerging from qualitative analysis
ü Structure and Navigation ü Coverage and Size (*) ü Common Language (*) ü Checklist ü Quality of knowledge (*) q (*) present in qualitative study on
which features are important in a ATM risk assessment
ü 20 Experts working on Risk Assessement in SESAR • Labunets et al. SESAR’s Innovation
Conference 2014. q Big difference between expert
and non-experts
09/04/15 12 Fabio Massacci - REFSQ - ESSEN
q Experts ü Common Language ü Checklist • “The first step is to use your own
experience and then to use the catalogue to cover generic aspects that could be forgotten”
q Not Experts ü Navigation is judge, jury and
executioner • “I saw people near to me; they were not
able to find out stuff in the catalogue, they kept on getting lost in the pages and eventually they came up always with the same two or three items” • “Once identified the threat, finding out
controls was really a mechanical work”
WHY DO THINGS WORK?
TUTORIAL ON GRC FOR SERVICES 2011
COPYRIGHT UNIVERSITY OF TRENTO 4
SUMMARY OF EXPERIMENT
q Do catalogues work? ü MAYBE YES à not experts perfomed equally to experts without catalogues ü BUT à people work better with domain specific information ü AND à experts and non-experts use them in radically different ways q Open Issues ü What about comprehensibility of results? • Risk assessment must be piped down the line for implementation • This was a critical issue when we interviewed stakeholders in ATM
ü What about scaling to really large risk assessment? q What is next ü More Info? à http://securitylab.disi.unitn.it ü Want to join the effort? à we are looking for replications
09/04/15 13 Fabio Massacci - REFSQ - ESSEN
ADVERTISING
q We are hiring for a industry-academia lab ü European Electronic Crime Task force q Positions ü 2 Phd Students à deadline 20May ü 2 Post-doctoral positions à open q Further info on Trento ü [email protected] ü http://securitylab.disi.unitn.it ü http://en.wikipedia.org/wiki/European_Electronic_Crime_Task_Force
09/04/15 14 Fabio Massacci - REFSQ - ESSEN
DISCUSSION OF TONG’S PAPER
q What can we do now that we could not do before? ü Try to model different aspects of a design (from business to physical) q How sound is the solution? ü A formal model is behind the graphs à a minimal Tarski semantics exists ü A small scale scenario was modelled by the author(s) q Whose goals are served or helped by this? ü One can fend off attacks across different layers q What is the next step to take? ü Demonstrate that you really capture and fold cross layers attacks ü Address scale à goal models quickly evolve into “spaghetti” models q Controversial Question: How do you avoid the “beholder” effect? ü “Beauty is in the eye of the beholder” ü You are the only one who really used it. How do we know it really works?
09/04/15 15 Fabio Massacci - REFSQ - ESSEN
