Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top
Transcript of Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top
![Page 1: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/1.jpg)
Security Metrics Rehab Breaking Free from Top ‘X’ Lists, Cultivating Organic Metrics,
& Realizing Operational Risk Management
April 11, 2014
![Page 2: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/2.jpg)
About Me
2
! Author of P.A.S.T.A threat modeling methodology (risk centric) (Wiley Life Sciences)
! Nearly 20 years of IT/Security experience (Developer, Security Arch, Dir of RM, CISO roles)
! Founder of VerSprite, global consulting security firm based in Atlanta
! OWASP ATL Chapter/ Project Leader for past 6 years
! VerSprite works across Retail, Hospitality, Financial, Gov’t, Higher Ed, Healthcare, Info Svc
! Co-organizer to BSides Atlanta grassroots security conference
! B.S from Cornell in International Finance
! Contact Info: ! [email protected] ! @t0nyuv / @VerSprite
![Page 3: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/3.jpg)
About This Talk
3
! Counterculture presentation on metrics, governance, and risk
! Depict pros/ cons around existing metrics/ frameworks in public domain
! Introduce seed of thought around building organic security metrics
![Page 4: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/4.jpg)
Consider the Following
4
If you had to only pick one… ! Option A: Fully developed security controls
framework w/ supporting metrics based upon leading industry lists and control frameworks
! Option B: Fully developed risk framework where
inherent and residual values are quantifiable, supported, and tied to business impact scenarios
![Page 5: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/5.jpg)
METRICS VS “BSETRICS”
![Page 6: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/6.jpg)
Separating Fact from Fiction
Metrics ! Objective focused ! Built from ‘What Do I Need’ (e.g. –
goal of providing evidence to effective technology/process management)
! Data source is dependable & vast ! Metrics should have a reliable data
source that augments over time ! Outliers are factored out ! Support clearly defined IT/ Biz goals
“BSetrics” ! Metrics that ‘feel/look
good’ (e.g. – closed risk issues)
! Built from ‘What Do I have’ (e.g. – tool begins to shape metrics discussion
! Based upon “industry standard” ! Keeping Up w/ the Jones’
Metrics ! Building metrics to manage
perception ! Data set is limited (e.g. – time,
breadth, pre-fixed) ! Outliers are not factored out
6
![Page 7: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/7.jpg)
Bad Metrics
7
! Issues remediated ! Unanswered: Tested/ not tested ! Unanswered: Was issue resolved, closed, transferred ! Unanswered: Is this ‘issue’ important
! # High Vulnerabilities Closed ! Deemed ‘High’ by whom? ! No context (High risk to a Low value asset)
! # of Code Imperfections to a Top X List (SAST Scan) ! Top X List begins to drive your risk perception devoid of anything
else ! Cultivating responses, remediation, and reports solely on top X items
! # of Pen Tests | # WebAppScans Conducted ! Doesn’t factor in automated or poorly conducted testing
![Page 8: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/8.jpg)
STATUS QUO SECURITY METRICS
![Page 9: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/9.jpg)
Process Metrics ! Is a SDL Process used? Are security
gates enforced? ! Secure application development
standards and testing criteria? ! Security status of a new application
at delivery (e.g., % compliance with organizational security standards and application system requirements).
! Existence of developer support website (FAQ's, Code Fixes, lessons learned, etc.)?
! % of developers trained, using organizational security best practice technology, architecture and processes
Management Metrics ! Management Metrics ! % of applications rated
“business-critical” that have been tested.
! % of applications which business partners, clients, regulators require be “certified”.
! Average time to correct vulnerabilities (trending).
! % of flaws by lifecycle phase. ! % of applications using
centralized security services. ! Business impact of critical
security incidents.
9
Examples of AppSec Metrics Today
![Page 10: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/10.jpg)
AppSec Metrics in Vuln Management
! Number and criticality of vulnerabilities found. ! Most commonly found vulnerabilities. ! Reported defect rates based on security testing (per
developer/team, per application) ! Root cause of “Vulnerability Recidivism”. ! % of code that is re-used from other products/projects*
! % of code that is third party (e.g., libraries)* ! Results of source code analysis**:
! Vulnerability severity by project, by organization ! Vulnerabilities by category by project, by organization ! Vulnerability +/- over time by project ! % of flaws by lifecycle phase (based on when testing occurs)
Source: * WebMethods, ** Fortify Software
![Page 11: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/11.jpg)
ROOM FOR IMPROVEMENT
![Page 12: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/12.jpg)
Forrester Survey: “What are your top three drivers for measuring information security?”
Source: “Measuring Information Security Through Metrics And Reporting”, Forrester Research, Inc., May 2006”
63%
11%
23%
26%
37%
51%
Manage risk
Report progress to business
Better stewardship
Loss of reputation
Regulations
Justification for security spending
Report progress to business
Better stewardship Base: 40 CISOs and senior security managers
![Page 13: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/13.jpg)
Good Metrics – Align w/ Maturity Model
Metrics ma+er most when they have direct or indirect relevance to opera5onal/ strategic goals
Align to Biz/ IT Goals
Directly or indirectly, categories to be measured need to map to key indicators that ma+er in IT Ops, Sales, Finance
Relate to Business Processes
Good start is to map metric areas to key processes sustained by a BIA
Map to a Business Impact
! Start simple ! Forget what everyone
else is doing – for now ! Perform an internal
PoC with LOBs/ BUs ! Grow base of coverage
over time ! Mature metrics by
benchmarking against industry reports/ analysis
13
![Page 14: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/14.jpg)
Opportunities for Metrics - Secure Development Life Cycle (SDL)
14
Secure questions during interviews
Concept Designs Complete
Test plans Complete
Code Complete
Deploy Post Deployment
Threat analysis
Security Review
Team member training
Data mutation & Least Priv Tests
Review old defects Check-ins checked Secure coding guidelines Use tools
Security push/audit
= on-going
Learn & Refine
External review
Source: Microsoft
Software assurance activities conducted at each lifecycle phase
![Page 15: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/15.jpg)
Organizing Metric Types Process Metrics
Information about the processes themselves. Evidence of maturity.
Vulnerability Metrics Metrics about application vulnerabilities themselves
Management Metrics specifically designed for senior management
Examples ! Secure coding standards in use ! Avg. time to correct critical vulnerabilities
Examples ! By vulnerability type ! By occurrence within a software development
life cycle phase
Examples ! % of applications that are currently security
“certified” and accepted by business partners
! Trending: critical unresolved, accepted risks
![Page 16: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/16.jpg)
Our Security Metric Challenge
“A major difference between a "well developed" science such as physics and some of the less "well-developed" sciences such as psychology or sociology is the degree to which things are measured.” Source: Fred S. Roberts, ROBE79
“Give information risk management the quantitative rigor of financial information management.” Source: CRA/NSF, 10 Year Agenda for Information Security Research, cited by Dr. Dan Geer
![Page 17: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/17.jpg)
BREAKING FREE FROM TOP ‘X’ LISTS
![Page 18: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/18.jpg)
Let’s Rethink Security Lists
Pros ! Great content from various
sources: OWASP Top Ten, SANS 20 Critical Security Controls, MITRE CWE Top 25, WASC TC v2, OWASP Top 10 - Mobile
! Provide a benchmark for testing | measurement
! Brings broader industry perspective
! Better suited for more mature programs where benchmarking is timely
Cons ! This defines an AppSec’s
program baseline ! Used as ground floor level
of metrics ! Tempts programs to look
outwardly vs. inwardly ! Doesn’t foster for Good
Metrics to take root ! Tools don’t make quitting
this trend easy (pre-defined profiles)
! Not a real basis for threat or risk analysis
![Page 19: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/19.jpg)
How Do Lists Break Us Free from This Cycle?
![Page 20: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/20.jpg)
METRICS & LISTS – TIMING IS EVERYTHING
![Page 21: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/21.jpg)
OWASP OpenSAMM Project ! Evaluate an organization’s
existing software security practices
! Build a balanced software security assurance program in well-defined iterations
! Demonstrate concrete improvements to a security assurance program
! Define and measure security-related activities throughout an organization
! http://www.opensamm.org ! Dedicated to defining,
improving, and testing the SAMM framework
! Always vendor-neutral, but lots of industry participation
! Open and community driven ! Targeting new releases every
6-12 months ! Change management process
![Page 22: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/22.jpg)
SAMM in a nutshell
! Evaluate an organization’s existing software security practices ! Build a balanced software security assurance program in well-
defined iterations ! Demonstrate concrete improvements to a security assurance
program ! Define and measure security-related activities throughout an
organization
![Page 23: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/23.jpg)
OWASP OpenSAMM (Software Assurance Maturity Model)
! Look inward ! Start with the core
activities tied to SDLC practices
! Named generically, but should resonate with any developer or manager
![Page 24: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/24.jpg)
Leveraging Lists at the Right Maturity Level ! Measure what you need across a framework’s
(OpenSAMM) area ! Identify ‘indicators’ that support business/ product
goals & objectives ! Apply use of lists for benchmarking as maturity
level rise
![Page 25: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/25.jpg)
Develop ‘Organic’ Security Metrics
Reasons ! Supports contextual
analysis based upon internal operations
! Top down approach to regressing to security metrics that matter
! Will substantiate security initiatives across non-InfoSec areas
Baking Organic Metrics Organiza5onal Objec5ves
Opera5onal Processes
Suppor5ng Technology & Infrastructure
BU/ LoB Objec5ves
Revenue Growth • Reputa5onal Loss • Non-‐Compliance
Cost Reduc5on • Fines & Penal5es
Product/ Service Objec5ves Product Innova5on • IP Security • Insider Threats • Incident Handling/ Response
Efficient Service Delivery • Con5nuity • Data Integrity
![Page 26: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/26.jpg)
Revisiting Lists
! Build your processes first
! Design metrics mapped to activities for those processes
! Develop scorecards that report on organic security metrics that relate to operational, financial areas
! Bake-in industry ‘lists’ in order to reflect more advanced quantitative analysis (Level 4)
![Page 27: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/27.jpg)
Creating Scorecards
! Gap analysis ! Capturing scores from detailed
assessments versus expected performance levels
! Demonstrating improvement ! Capturing scores from before
and after an iteration of assurance program build-out
! Ongoing measurement ! Capturing scores over
consistent time frames for an assurance program that is already in place
![Page 28: Tony UcedaVelez Security Metrics Rehab: Breaking Free from Top](https://reader036.fdocuments.in/reader036/viewer/2022081515/5868cb301a28ab677d8b7200/html5/thumbnails/28.jpg)
THANK YOU!