@MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing...
Transcript of @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing...
![Page 1: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/1.jpg)
How to Get Promoted: Developing metrics to show how threat intel works
Toni GidwaniDirector of Research@t_gidwani
Marika ChauvinSenior Threat Intelligence Researcher@MarSChauvin
![Page 2: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/2.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 2
Marika Chauvin
Research nerd
King cake aficionado
Who are we?
Toni Gidwani
Side gig as a Georgetown professor
Maker of gelato
![Page 3: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/3.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 3
Table of Contents
The problem: showing value
Classes of metrics
Examples by maturity
![Page 4: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/4.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 4
The Problem
How do I show that threat intel provides value to my org?
![Page 5: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/5.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 5
“Building a Threat Intel Program” Survey Respondents
![Page 6: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/6.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 6
❏ Remove risks from cybercrime activities
❏ Protect personal client information
❏ Protect monetary assets of the organization
❏ Increase productivity for other parts of the organization
❏ Revenue generated for the organization
❏ Prevent service interruption for core business functions
❏ Avoid embarrassing public disclosures of information
What’s the most important success factor?
36%
![Page 7: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/7.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 7
Executives self-rate maturity much higherDisconnect: Just how mature are we?
![Page 8: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/8.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 8
When we’re not on the same page...The problem
![Page 9: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/9.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved.
“Metrics”
![Page 10: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/10.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 10
Metrics: Can’t live with them, can’t live without them
Good metrics
● Clear
● Measurable
● Correlate to business outcomes
Common pitfalls:
● What we can count
● Output, not impact
● Too tactical for your boss’ boss
![Page 11: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/11.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 11
Types of Metrics
Measures of Performance
Measure task completion and
efficiency
Am I doing things right?
Measures of Effectiveness
Measure what is accomplished and whether
goals are being met
Am I doing the right things?
![Page 12: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/12.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 12
Measures of PerformanceTypes of Metrics
Useful for:
● Impact of automation/efficiencies
● Process improvement
● Utilization of resources
● Incentivizing a baseline step
Examples:
● Total alerts issued
● Total Intel items reviewed/parsed
● % of malware samples detonated
● IOCs shared with Community
![Page 13: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/13.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 13
...ButTypes of Metrics
Limitations:
● Less useful for senior leaders
● Risk incentivizing poor behavior
● Less useful over long-term
![Page 14: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/14.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 14
Measures of EffectivenessTypes of Metrics
Useful for:
● Conveying program value to senior leaders
● Can be qualitative or quantitative
● Drive data collection
● Drive process development
Examples:
● Incidents discovered from TI
● Countermeasures enacted
● Total proactive blocks
● Mean time to detection
● Savings generated
![Page 15: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/15.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 15
...ButTypes of Metrics
Cons:
● More difficult to generate
● Not as easily countable
● Often require interaction and input from other teams
![Page 16: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/16.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 16
Key Takeaway
Measures of Effectiveness are more compelling to your boss’ boss
![Page 17: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/17.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 17
Showing Value at Different Maturity Levels
...because I can’t wait 5 years
![Page 18: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/18.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 18
• 77% saved a significant sum of money in the last year
○ Least mature: ~$2k
○ Mid-level programs: $1.7 million
○ Well-Defined programs: $10 million
Self-Reported Money Saved
![Page 19: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/19.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 19
Schrodinger’s Breach: When Getting Better Looks Worse
Gains for lower maturity programs come first from:
● Improving visibility
● Understanding the threat
● Enhanced detection
![Page 20: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/20.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 20
Metrics to Tell if Improving or Everything is on Fire
Getting started?
● IOCs observed
● Incidents discovered from TI
● Qualitative feedback loop
● Countermeasures enacted
![Page 21: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/21.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 21
Metrics to Tell if Improving or Everything is on Fire
More mature?
● False positive ratio
● Impact year over year
○ Mean time to detection
○ Mean time to respond
● New intelligence from cases
● Incident criticality impacted by TI
![Page 22: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/22.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved. 22
• Mean cost of breach
○ Downtime
○ Additional resources to address breach (consultants, identity theft protection, etc)
• Feedback loop can be used to justify salary, team budget, and direct analysis efforts
• IBM Cost of a Data Breach Calculator
Quantifying value
![Page 23: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/23.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved.
Key Takeaways: All Metrics aren’t Created Equal
Easy
Difficult
Least Valuable Most Valuable
● Mean time to discovery● Mean time to mitigation
● New intelligence from cases
● IOCs observed
● Feedback loop
● Number of IOCs● Number of ingested feeds
● Incidents worked
● AV detections
● Countermeasures enacted● False positive ratio
● Incident criticality impacted by TI
● Mean cost of breach
● Revenue saved
● New incidents from TI● Number of reports
![Page 24: @MarSChauvin metrics to show how threat intel works How to ... · How to Get Promoted: Developing metrics to show how threat intel works Toni Gidwani Director of Research @t_gidwani](https://reader030.fdocuments.in/reader030/viewer/2022040603/5e9edb8cebc1a7161e3692eb/html5/thumbnails/24.jpg)
© 2018 ThreatConnect, Inc. All Rights Reserved.
Thank Youwww.ThreatConnect.com