Preliminary DRAFT for community review Mike.Davis.SD@gmail.com 1

CISO Scorecard How do you answer the boss’ “how secure are we” question? Quantifying what really matters in cyber security is a first step in deciding what to measure, thus develop any sort of scorecard. The current cyber industry trend is towards a risk based security strategy, to best apply resources where the impact reduction is the greatest. Then what matters becomes what protection capabilities and vulnerability mitigations add the best value in reducing the organization’s overall risk. These risk reduction capabilities should be based on the organization’s key business success factors and support both threat vector minimization and vulnerability impact reduction as well as enhanced competitive advantage. Thus any “CISO scorecard” represents the overall “best risk value” for the organization from those key views. Whereas the ‘why’ of collecting security metrics also includes justification for security spending, compliance and regulations support, data breach risk minimization (loss of brand / reputation), etc. Many of downsides of ineffective risk management have hard to quantify intangible effects, specific damages potentially incurred, cost avoidance, etc – yet in the aggregate are very costly. Ideally the organization has documented business success objectives to help determine the related technical objectives by which the security controls are applied. If not, then one needs to use the vision and mission statements to distill a set of operational success objectives to use in translating those requirements to technical objectives. After which a risk assessment is applied and security controls are determined. In many cases this is the hardest step, as it involves two transformations and then mapping the major controls thereafter, with frequently fuzzy objectives to start with. Nevertheless, the security strategy must be risk value based, so this is a great exercise for the Security team to undertake to ensure their major cyber tasks and resources are focused on the most effective risk value elements. In fact, the ultimate security objective is to provide a competitive advantage, differentiating cyber strategy, which includes minimizing risks. The following sections will describe how we distill the core risk factors and major focus areas to develop the CISO scorecard, whereas as the initial notional views suggested above are suggested as below, to help frame the bigger picture as we start the ‘what matters’ journey.

Figure 1.0 – Notional risk value elements

Preliminary DRAFT for community review Mike.Davis.SD@gmail.com 2

The approach shown above is focused on the risk perspective, minimizing risks and barriers, yet all efforts must also contribute to a competitive advantage, as suggested below:

1. Known to be a trustworthy organization that ‘walks the walk’ - focusing on the P&L benefits of mitigating cyber risks, aligned with appropriate technical solutions and security architecture.

2. Embody safety, integrity & TRUST - demonstrate due care and due diligence therein. 3. Facilitate a class leader position, enhance BRAND, go beyond the norms - lead the industry. 4. Improve operational efficiency - high resiliency, lower security costs and risk impacts. 5. Improve compliance levels, reduce costs therein (Build in security, map to CIS 20 controls). 6. Show excellence in DATA protection – put security in the forefront, show privacy stewardship. 7. Improve processes and add new security services and products (directly support the new

technology initiatives and eternal services groups). 8. Support programs, processes, and use cases that foster innovation and growth (managed risk,

user transparency, “Special Projects” - etc). 9. Facilitate new business, enhance productivity and collaboration, commoditized services, etc.

Next we apply these major risk reduction and competitive advantage elements to the company’s strategic plan illustrated below and then distill the major business factors to base our CISO scorecard on.

Figure 2.0 – Company strategic business factors (example - insert YOURS here!)

Using the suggested risk reduction (and differentiating) measures quantified later in the document, along with these strategic plan objectives, we propose the key operational business success objectives that have a corresponding technical objectives relationship. The most likely risks and barriers to those objectives are then assessed by which the security measures will be selected to make up the CISO scorecard. These measures must be relatively easily captured and processed, to both minimize the resources overhead and support a clear picture of the ‘best risk value’ outcomes.

Preliminary DRAFT for community review Mike.Davis.SD@gmail.com 3

All measures need to be famed and managed within an overall enterprise ‘balanced scorecard (BSC)” perspective, to help show a clear, relative risk value to senior leadership. Traditionally, the BSC method describes the cause-and-effect linkages between four high-level perspectives of strategy and execution. Those four perspectives can be applied to an overall information security strategy: • Financials / Value is an expression of the information security organization’s strategic objectives and value that it provides to the organization (Safety, security, integrity, other safeguards for stakeholders). (e.g., Cost of security / workstation, assessment costs, compliance activities, security services status, etc) • Customers is an expression of how the security team believes they should be viewed in order to satisfy and deliver value to target customers or stakeholders in order to achieve its strategic objectives. (e.g., Reportable privacy breaches, customer & partner compliance, regulatory audits / period, etc.) • Business / Internal Operations is an identification of the handful of critical capabilities at which the security team needs to excel and demonstrate the overall company risk value to its target customers. (e.g., Average patch time, percentage (%) of endpoints with no vulnerabilities, time to de-provision users, time to fix critical vulnerabilities, cost of data breach, etc.) • Innovation, Learning and growth is an identification of the most critical people, processes and systems that enables security to be excellent at and add value to the important business capabilities. (e.g., Number of security consultations by business, % of users with weak passwords, security training statistics and effectiveness, etc.) Scorecards are used for strategic decision support – especially for financials and operations. For business decision support, scorecards are created to present targeted answers to common leadership questions:

Are we meeting our fiduciary and statutory requirements?

How do we compare to our peers? Are we advancing our objectives?

How are we identifying and managing risk? Are we improving our risk posture?

Are we investing in and advancing initiatives in the right order? A Cyber Security Scorecard will help steer an organization towards the desired cyber security strategy, while providing answers to both the leadership questions and others raised by executives: Board of Directors Questions

1. What is the status of our cyber resilience capabilities compared to the threat level? 2. What is the impact that cyber security risks have on our business strategy, risk posture? 3. How do our measures and investments compare to the rest of our sector? 4. Are we compliant with the relevant cyber security and related regulations?

CIO Questions 1. What are the key drivers in cyber security risk management and how are they developing? 2. What is the status of our cyber security detection and prevention capabilities? 3. What is the status of the risk framework and processes, including compliance / audit / V&V? 4. What were the root causes and actions taken for any high-impact incidents in last period?

Ultimately, the Security scorecard objective is to help CISOs be more successful at communicating the business value of information security and at linking the strategy with execution. A pseudo-formula for how to do it: Strategy Map + Measures and Targets + A Set of Funded Initiatives = A Complete Program of Action. Table 2.0 outlines the mapping between the company objectives and most likely risks (as listed in table 1.0 and further categorized later). Whereas the next section assesses the various security measures suggested by several authoritative sources (plus other operational metrics in the appendix) to best capture and align to those risks. We note upfront that that there is one over-riding risk that is pervasive in negatively affecting all objectives – a data breach. Which should not be a surprise to anyone, given the record numbers of events in 2016, in files, organizations and costs. Security developed a separate data breach risk posture approach that assessed the top ten vulnerabilities and risks therein,

Preliminary DRAFT for community review Mike.Davis.SD@gmail.com 4

where those results are used here. A summary risk table from that approach is excerpted below, then those risks are further categorized as ‘themes’ to better align with the business objectives in table 2.0. In table 1.0 the major data breach causal factors are listed as ‘methods’ since they contain the typical mix of policy, process, people and product (technology) elements. To start the risk estimation journey, especially for those who have only a general awareness of the security methods, we first start with a high level sense of the relative ‘goodness’ level, providing a heuristic view of the security ‘posture.’ The ‘key areas / exposure’ column (not shown below) provides a high level view of the major residual problem areas to help link the relationship and potential impacts in the other methods. The relative posture level and numeric risk value estimates for each method can provide a sense of the impact reduction potential for minimizing the vulnerabilities risk level, taking into account the defense in depth status of the system architecture as a whole (the dependencies, hierarchy, inheritance, etc. at play). (note - risk values are notional, examples used for illustrative purposes, NOT actuals, insert yours here!)

Table 1.0 – Main Data Breach methods and mitigations

Methods

Posture Risk (1-25)

Mitigations (common objectives)

1 – Adequate

encryption (and key

management).

Below

average

20 Implement targeted encryption at rest and in motion (HTTPS),

integrated with DLP solutions that control access, sharing and

protection means. Reduce shadow IT.

2 – Incident Response

Plan

Good 9 Processes exist, need to audit and ensure immediate actions are

clear. Run exercises with the major stakeholders, legal, etc

3 – Identity and Access

Management (IAM)

Average 15 Periodically audit all accounts. PAM policy / PI. Implement

enterprise MFA & DRM/IRM (align with DLP)

4 – Data Loss

Prevention (DLP)

Below

average

15

DLP strategy exists. Keep DLP simple and business focused.

Start with selecting DLP capabilities (with IRM). Get mission

critical apps buy-in, then other stakeholders, have POA&M.

5 – Vulnerability

management /

Adequate cyber suite

Average 15 Develop an effective CMDB. Automatic patch processes help.

Show CIS 20 security controls adequacy. More effective SIEM

/ SOC / tools integration

6 – Insider threat

detection

Marginal 15 Must monitor behaviors in both IT and HR. Control / limit file

and resource access. Leverage IdAM & DLP. Add controls -

limit users to own workstations or department, limit concurrent

logins and session time/hours,

7 –Cyber Education

and Awareness

Training

Average 9 User security actions need to be simple, direct. Defense in

depth technical controls are the essential core protection. More

periodic, targeted user engagement for cyber awareness,

phishing et al. Clearly identify consequences for poor behavior

8 – Cyber Insurance

and detailed

notification processes

Good 3 IR plan has effective steps, controls. Practice the steps and

communicate more often with all major stakeholders. Policies

can be inadequate (too many caveats). Usually based on client

data loss, versus also IP / PII (which is difficulty to put a value

on, to assess damages)

9 – Audit / monitor /

verification and

validation assessments

Below

average

9 Need a data security policy first, then an audit / V&V effort to

track and report; provide mitigations. Create a security

dashboard area / metric, with effective log reviews.

10 –Security and

Privacy Policy and

process instructions

Below

average

6 Need to enforce PII/Privacy and confidential data storage

policies. IT security policy needs implementing and linking to

company policies, supported by process instructions.

Preliminary DRAFT for community review Mike.Davis.SD@gmail.com 5

As discussed later, we define overarching themes and break those down into components with defined objectives, with each having defined major initiatives, which are themselves funded, tactical activities that support key risk mitigations activities. These themes are risk based and thus it’s structurally beneficial to categorize the key risks in table 1.0 into five key focus areas (as also done in the data breach risk approach). The risk heat chart therein categorizes the top risks in # 1 – 10 above as: (A) Data protection (encryption (#1) and DLP (#4)), (B) – IAM (#3), (C) - Vulnerability management / effective cyber suite (#5), and (D) – Intruder detection (Insider threat (#6), adequate rogue identification). To those four risk categories we add two management categories: (E) Policy (#10) and Monitoring (#9) and (F) – Risk Management (effectiveness, processes and metrics). Risks depend on the threats ability to take advantage of existing vulnerabilities. Whereas a risk based security strategy should focus on the highest threats to the company, specifically the “threat actors” that can take advantage of existing vulnerabilities; thus create the highest risks. These ‘intruders’ threat vectors also need to be minimized and accounted for in the metrics we capture, be they scorecard or operational. The typical threat sources that will be accounted for in the risk value process are: (1) - Insiders (85+% of all attacks); (2) - Privileged users (stolen credentials cause most data breaches); (3) - Hackers (nation states, criminals, all malevolent security types - typically they use malware), and (4) – Inattentive Users (phishing attacks, poor social media behavior, poor PWs, etc.). Table 2.0 aligns the operational objectives with the technical aspects therein and the corresponding key risks and barriers.

Table 2.0 – Business Objectives and Risks Mapping

Operational objectives Technical aspects Key Risks & overall barriers

A. Grow market share Demonstrate Safety, Service, Integrity & Trust to show clear value. Overall Industry ecosphere approach, versus just clients and customers, maintain visibility….

A, C and F. Unclear common companywide objectives No integrated capability roadmap

B. Sector Leader Enable Innovation, Establish Standards, support ISAC, support research and bold / disruptive changes for the industry – leapfrog legacy obsolesce….

B, D and E. Limited R&D mapping to future client’s needs Limited standards efforts to set the industry baseline

C. Drive efficiency Lean processes & remove overlap, key metrics & data driven decisions, smart monitoring and Audit / V&V….

B, C and E. Limited companywide process improvement initiatives No common “CM” or metrics processes or champion

D. Company R&D drives capability transformation

Special Project, Data / predictive analytics, Cyber Safety, effective standards implementation, whole company Risk Value view….

B, C and F. Limited top down ERM focus Unclear capability value chain

E. Company divisions synergies to improve product / services cohesion

Common processes, user productivity focus, business centric IT/security, new markets, technology agnostic solutions….

A, E and F. Unclear harmonization of overall company success factors No ‘commodity’ view of core capabilities key attributes

Preliminary DRAFT for community review Mike.Davis.SD@gmail.com 6

When it comes to the security metrics themselves, there is a whole litany of perspectives, views, references, et al to consider. The security team developed a companion security metrics strategy that covers most of the security metrics background in detail (where several key security measures are excerpted in the appendix for comparison). Whereas that metrics approach was from a security operations view versus the risk value determination herein. We will use the security metrics recommendations therein to help guide our choice of the CISO scorecard elements that follow, as they are for the most part, intertwined and inseparable – being different views of the same risk environment. Everyone has their own view on security metrics that matter (e.g., another metrics reference view is): http://www.csoonline.com/article/2118152/metrics-budgets/a-few-good-information-security-metrics.html

1: Baseline Defenses Coverage (Antivirus, Antispyware, Firewall, and so on 2: Patch Latency 3: Password Strength 4: Platform Compliance Scores 5: Legitimate E-Mail Traffic Analysis There are many other views on security metrics of course: http://www.csoonline.com/article/2123361/metrics-budgets/security-metrics--critical-issues.html

The NIST process to develop security metrics is shown below for another process frame of reference. Even with this high level illustration, it’s clear that it is an extensive process in itself, whereas our development and execution resources are fairly constrained in addition to not having any formal, overarching corporate metrics processes to align within. We will account for these steps as we begin to socialize and phase in the implementation of the recommended CISO scorecard metrics.

Figure 3.0 – NIST Metrics Life Cycle

Effective metrics should have five qualities. Most organizations know how to pick metrics that satisfy the first four qualities: namely, (1) that they are expressed as numbers, (2) have one or more units of measure, (3) are measured in a consistent and objective way, and (4) can be gathered efficiently (cheaply). Yet only a few entities choose metrics that satisfy the most important criterion: contextual relevance. That is, the metrics must help someone -- usually the boss -- make a decision about an important risk or business issue. Too many organizations use metrics to erect byzantine processes (excessively complicated, involving a great deal of administrative detail) to measure the minutiae of

Preliminary DRAFT for community review Mike.Davis.SD@gmail.com 7

what they understand rather than what the boss needs to know. Failure to pass the "so-what" test makes a metric potentially interesting but not insightful to support decisions. We use two sources of recommended security metrics for comparison and then distill those into a select set to use within the BSC framework suggested earlier (focusing on both risk reduction and competitive advantage benefits). (1) The “CISO Desk Reference Guide (DRG)” key suggested metrics and (2) the Center for Internet Security (CIS) top 20 controls recommendations – both discussed next. The CISO DRG key metrics (chapter 5 therein) are listed below (numbered for later reference, not priority) where those in bold text are the initial metrics proposed (phased in there as well): Administrative metrics: 1---Legal – (a) - Percentage of material contracts (defined as involving data sets that require privacy and security controls) that have been evaluated by security (typically as a high level risk assessment). (b) - Percentage of material contracts (defined as involving services with sensitive information that could impact the core company operations) that have been evaluated by security, including requirements for breach notification and confidentiality language. 2---Financial – (a) - Percentage of IT budget allocated to security (where this value varies widely depending on industry and the cyber baseline effectiveness - a minimum of 5% is normally expected). (b) - As applicable, status of major security projects for the year. (added) 3--- Human Resources (HR) – (a) - Percentage of employees who had a thorough background check (including any previous questionable activities, be that illegal or unethical). (b) - Percentage of job descriptions that highlight the employee’s responsibility to protect the organizations’ assets. (c) - Percentage of employees who have taken the annual security awareness training (and ideally passed an assessment of their retention of key cyber safe tenets) (d) - Percentage of employees who have read, acknowledged and been tested on the security policy. 4 ---Vendor Management – (a) Percentage of material vendors (defined as those that have the potential for critical or severe risk, should a breach or service interruption occur) who have been audited either directly or a third party (e.g., SSAE 16 SOC 1 & 2 audit, etc.). (b) - Percentage of material vendor relationships that are inventoried and documented. (c) - Percentage of material vendors that participate in quarterly reviews with the security function. Operational Metrics: 5---Software Inventory – (a) – Existence of an IT Asset Management database (“ITAM”) (added). (b) - Percentage of known assets & ‘systems’ accounted for in “ITAM/CMDB.” 6---Information (data) Inventory – (a) Percentage of information assets accurately inventoried (b) Percentage of information classified accurately (assumes a data classifications standard exists) (c) – Percentage of system documented (e.g., at least a data flow diagram exists, ideally a security plan) 7--- Systems Upgrades & Patching – (a) Percentage of major systems (both hardware and software) that are still supported by the manufactures’ or validated third party. (b) – Percentage of systems that are patched within 30 days of critical security patches. (c) – Percentage of system that are scanned for vulnerabilities on a monthly basis 8 – Multi-Factor Authentication (MFA) – (a) percentages of systems with IP, PII, or other sensitive data that use MFA (b) – Percentage of critical capabilities (Servers, firewalls, etc) & remote connections using MFA. (added) 9 – Mean time from incident response to remediation – Time delay from detection to vulnerability fix. 10 – Aggregate threat level – composite score of the leading cyber threat intel (CTI) indicators. (added) Governance Metrics (compliance): 11 – Incident Response (IR) Plan – (a) Existence of a management reviewed and approved IR Plan (b) – Time since IR plan was tested (at least in a table top, ideally a short breach exercise yearly).

Preliminary DRAFT for community review Mike.Davis.SD@gmail.com 8

12 - Business Impact Assessment (BIA) Analysis - (a) – Existence of a management approved BIA plan. (b) – Last update date. (c) – Number of high risk business processes with critical “RPOs” 13 – Business Continuity / Disaster Recovery (BP / DR) plans - (a) – Existence of a management approved BC / DR plan. (b) – Time since last test. c) – Percentage of systems, processes or applications that met RPO / RTO objectives. 14 – Project Security Plans / risk assessment – % of projects / applications that have one. (added) 15 – Cyber Maturity level – aggregated score of the major policies and processes required. (added) 16 – Cyber events / cost – number of yearly incidents and their cost to remediate. (added) Next we list the CIS top 20 security controls metrics recommendations (e.g. below table 3.0, 28 measures overall) as one key reference, followed by their recommendation for the top ten metrics to use in a BSC. These CIS metrics will be harmonized with the CISO DRG metrics recommendations above to provide our ‘best of breed” key risk reduction and competitive advantage BSC metrics, which maps back to the major business objectives; and can show a cost benefit where applicable as well.

Table 3.0 – CIS Security Controls Key Metrics Function Management Perspective Defined Metrics Incident Management How well do we detect, accurately

identify, handle, and recover from security incidents?

1. Cost of Incidents 2. Mean Cost of Incidents 3. Mean Incident Recovery Cost 4. Mean-Time to Incident Discovery 5. Number of Incidents 6. Mean-Time Between Security

Incidents 7. Mean-Time to Incident Recovery

Vulnerability Management How well do we manage the exposure of the organization to vulnerabilities by identifying and mitigating known vulnerabilities?

8. Vulnerability Scanning Coverage 9. Percent of Systems with No Known

Severe Vulnerabilities 10. Mean-Time to Mitigate Vulnerabilities 11. Number of Known Vulnerabilities 12. Mean Cost to Mitigate Vulnerabilities

Patch Management How well are we able to maintain the patch state of our systems?

13. Patch Policy Compliance 14. Patch Management Coverage 15. Mean-Time to Patch 16. Mean Cost to Patch

Configuration Management What is the configuration state of the systems in the organization?

17. Percentage of Configuration Compliance

18. Configuration Management Coverage 19. Current Anti-Malware Compliance

Change Management How do changes to system configurations affect the security of the organization?

20. Mean-Time to Complete Changes 21. % of Changes with Security Reviews 22. Percent of Changes with Security

Exceptions Application Security Can we rely on the security model of

business applications to operate as intended?

23. Number of Applications 24. Percent of Critical Applications 25. Risk Assessment Coverage 26. Security Testing Coverage

Financial Metrics What is the level and purpose of spending on information security?

27. IT Security Spending as Percent of IT Budget

28. IT Security Budget Allocation

Preliminary DRAFT for community review Mike.Davis.SD@gmail.com 9

We need to also integrate the security strategy key tasks ‘risk value contribution’ into the CISO scorecard, whereas any cyber strategy is essentially a hypothesis, since risk conditions change frequently. Strategy is the combination of leveraging resources and executing a plan of action to achieve a set of desired outcomes. A BSC/Metrics strategy is typically described from the top down (i.e., starting from the objectives to be achieved), but all strategies must be executed from the bottom up (i.e., starting with the allocation and alignment of the people, processes and technologies necessary to carry out a plan of action). In concert with the latter, the CIS core metrics BSC view is then:

Table 4.0 – CIS “BSC” Top Ten Metrics Scorecard Area Action Required Metrics Impact Report on security incidents and their

impact on the organization. 1. Number of Incidents 2. Cost of Incidents

Performance by Function: Outcomes

Report the outcome of business functions' Configuration Management, Patch Management and Vulnerability Management.

3. Configuration Policy Compliance (using CIS benchmarks) 4. Patch Policy Compliance (using current patch level) 5. Percent of Systems with No Known Severe Vulnerabilities (using CVSS base scores)

Performance by Function: Scope

Report the scope of business functions and the scope of outcome metrics for those functions.

6. Configuration Management Coverage 7. Patch Management Coverage 8. Vulnerability Scanning Coverage

Financial Metrics Report on the allocation and efficiency of security spending

9. IT Security Spending as Percent of IT Budget 10. IT Security Budget Allocation

Discussion - So now what? We’ve listed the CISO DRG overall measures and the CIS top 28 and BSC 10 measures and those risk based factors in the previous table that we proposed from the company’s business strategy. These measures all need to be harmonized into a ‘best of breed’ metric set and then associated to the four BSC categories described earlier. How do we reconcile them and provide performance indicators to senior leadership and then also decide which business outcome is weighted more - risk reduction (cost avoidance) or innovation (increase revenue). Including a competitive advantage aspect, in adding to the overall business success objectives value? In addition, the security team has to be able to effectively collect these metrics and transform them where required into the key leadership level interest items. Then we need to be able to benchmark the key measures, or use some other well-known company comparison methods, to show relative goodness. These risk focused areas then serve as the foundation for identifying the most relevant measures to be considered on a company’s dashboard. They cover the core areas of cyber security: risks, compliance, incidents, awareness & culture, threat level, maturity state, and key cyber security projects in development. (note - for more examples see:) https://assets.kpmg.com/content/dam/kpmg/pdf/2015/04/Cyber-Security-Dashboard1.pdf http://www.securitymetrics.org/content/Wiki.jsp?page=Chapter10ScorecardDesign So how do we define “cyber success” using the common BSC format that also demonstrates the security value sphere; which in turn support the company’s core business model, vision and mission? That is, aligning the CISO scorecard to show a triad of key contributions: (1) competitive advantage (facilitate new business, enhance reputation, establish collaborative partnerships), (2) operational efficiency (reducing security and compliance costs, increase productivity) and (3) risk reduction (cost avoidance, minimize the number and impact of security events).

Preliminary DRAFT for community review Mike.Davis.SD@gmail.com 10

We need to enhance the business strategy map using the security value triad elements to create a visual tool that illustrates the key company success factors from the four BSC perspectives providing a holistic view of the key risk mitigation factors. We start by defining overarching themes (the six risk categories defined earlier) and break those down into components with defined objectives for long term growth. Each objective then has quantified major initiatives, which are funded, tactical activities that support delivery and will be measured and tracked – these are the foundation of the triad’s model. This model then is self-sustaining as it shows the interplay and integrated nature between the overarching strategy, themes, objectives and initiatives. The four important considerations in the pursuit of a security value triad are: (a) Effective holistic solutions, (2) Key Stakeholder relationship management, (3) All levels of decision support and (4) Integrated performance management, which all then must be part of the planning and execution of each initiative. The earlier table mapped out our business objectives into technical objectives and main associated risks; thus we use those to frame our selection of CISO scorecard metrics. In addition the other day-to-day security metrics we recommend (listed in the appendix) that have an operational utility also need to show the value of supporting the scorecard view. Security adopted the RBSS approach and uses that to prioritize our tasks and resources; thus it follows that this approach can also be used to select the best value metrics for the targeted risks in the table. Given there are numerous ways and frameworks to map metrics to risks, we start by choosing the CISO DRG metrics to start, while also integrating the CIS metrics recommendations where needed into a core, best of breed, measures set, then map those metrics to the targeted risks. This approach provides scorecard traceability to the key business objectives and provides management with ‘data driven’ decision support to obtain the best risk value while also enhancing the company’s competitive advantage. Next we align the main risks with the corresponding metrics and the data they provide to assist in decision support. The table provides a link to the risks, suggested metrics and decision support data.

Table 5.0 – Risks / Metrics / Decisions supported

Risk themes Associated Metrics Data / Decision support

A – Data protection

1.A – Material contacts assessed 2.A - % IT budget for security 3..C - % employees annual training 4.A - % Vendors / contractors assessed 6.C - % systems documented (data flows) 11 – Time period since last IR Plan test 14 - % projects with risk assessment (SP) 16 - # cyber incidents / cost to remediate

- Potential third party privacy / data break risks - Resource the critical protections for this risk - Significant ‘user threat’ vector for this risk - Inherited security posture on their side - Lack of documentation means more data risks - Speed / accuracy of damage control is critical - Lack of security plan means unknown risks - Precursor to follow on data breach costs

B - IAM 2.A - % IT budget for security 3.A - % employees background check 8.A - % sensitive data systems using MFA 8.B - % critical IT systems using MFA

- Resource the critical protections for this risk - Critical ‘insider threat” propensity attribute - Stronger access controls for more data security - Improved protection of essential infrastructure

C – Vulnerability Mgmt

1.A – Material contacts assessed 2.A - % IT budget for security 4.A - % Vendors / contractors assessed 5..B - % key assets / systems in ITAM/CMDB 6.C - % systems documented (data flows) 7.B - # systems patched within 30 days 7.C - % key systems monthly scan 9 – Mean time from detect to remediation 10 – Aggregate threat level (and trends) 13.B – BCP/DR – time since last test 14 - % projects with risk assessment (SP) 16 - # cyber incidents / cost to remediate

- Potential third party privacy / data break risks - Resource the critical protections for this risk - Inherited security posture on their side - Ability to prioritize, quantify and track risks - Lack of documentation means more data risks - Risk impact grows with time and exposure - Verify & validate remaining vulnerabilities / risk - Risk impact grows with more time exposure - Focus / tune tools for increased threat actors - Resiliency risks increase with time exposure - Lack of security plan means unknown risks - Shows where vulnerability management is weak

Preliminary DRAFT for community review Mike.Davis.SD@gmail.com 11

D – Intruder detection

2.A - % IT budget for security 3.A - % employees background check 3..C - % employees annual training 4.A - % Vendors / contractors assessed 8.A - % sensitive data systems using MFA 8.B - % critical IT systems using MFA 10 – Aggregate threat level (and trends) 15 – Cyber maturity level (policy / processes)

- Resource the critical protections for this risk - Critical ‘insider threat” propensity attribute - Significant ‘user threat’ vector for this risk - Inherited security posture on their side - Stronger access controls for more data security - Improved protection of essential infrastructure - Focus / tune tools for increased threat actors - General readiness and effectiveness state

E – Policy & Monitoring

1.A – Material contacts assessed 2.A - % IT budget for security 11 – Time period since last IR Plan test 12.C – BIA - # high risk / critical “RPOs” 13.B – BCP/DR – time since last test 15 – Cyber maturity level (policy / processes)

- Potential third party privacy / data break risks - Resource the critical protections for this risk - Speed / accuracy of damage control is critical - Resiliency and risk time exposure increased - Resiliency risks increase with time exposure - General readiness and effectiveness state

F – Risk Management

2.A - % IT budget for security 4.A - % Vendors / contractors assessed 5..B - % key assets / systems in ITAM/CMDB 6.C - % systems documents (Sec Plan) 7.B - # systems patched within 30 days 7.C - % key systems monthly scan 9 – Mean time from detect to remediation 10 – Aggregate threat level (and trends) 11 – Time period since last IR Plan test 12.C – BIA - # high risk / critical “RPOs” 13.B – BCP/DR – time since last test 14 - % projects with risk assessment (SP) 15 – Cyber maturity level (policy / processes) 16 - # cyber incidents / cost to remediate

- Resource the critical protections for this risk - Inherited security posture on their side - Ability to prioritize, quantify and track risks - Lack of documentation means more data risks - Risk impact grows with time and exposure - Verify & validate remaining vulnerabilities / risk - Risk impact grows with more time exposure - Focus / tune tools for increased threat actors - Speed / accuracy of damage control is critical - Resiliency and risk time exposure increased - Resiliency risks increase with time exposure - Lack of security plan means unknown risks - General readiness and effectiveness state - Shows where risk management is weak

Any CISO Scorecard effort also needs to link those metrics into their security strategy of course and do that in one slide too! We offer proposed IT Security strategy statements below, which should capture the security value triad intent, and an example strategy slide. Once the BSC based measures are finalized that clarify the security value triad, then update this slide, whereas this proposed draft serves to help frame the final scorecard recommendations. (again, an example, notional strategy, put yours here!)

Preliminary DRAFT for community review Mike.Davis.SD@gmail.com 12

Next steps / Execution Plans. The key metrics proposed in table 5.0 can now be assessed for their utility to our end-state set of measures that matter. Whichever ones are chosen, we need to have an implementation plan, including how to best capture those not used currently (which is the majority of them). In addition, we need to decide on the best ‘starter set’ metrics to use in the CISO Scorecard – both visually and the key themes to highlight initially. We believe that leadership’s major expectation of security is to protect the company and minimize risk, whereas, operational efficiency and competitive advantage are then complementary to the former; thus we use a risk lens for our CISO scorecard.

Process note - Developing a CISO Scorecard (or any type for that matter) requires quite a lot of effort to get the final formalized and approved metrics and format (for example, see appendix item “D” / SANS Scorecard checklist, with over two dozen recommended steps). Our scorecard efforts start with this research / community paper to set the background, provide key references with potential recommended security metrics, etc., to provide an initial “strawman / notional” output to facilitate those steps and discussions required in the checklist. It’s rather like a ‘rapid programing’ effort where the general intent is known, along with best practices therein, so a baseline is developed from those key sources, wherein it is then generally to easier verify requirements, et al. Especially since many in management don’t have a full grasp of what security does or the high potential damage that a data breach risk can cause, for one example; thus showing a sample end-state scorecard can help facilitate their understanding and concurrence on the utility and value of the whole effort.

The operative question then becomes, after all the options, mapping and assessment so far, which metrics do we start with and what format should we use? Given we do not have a formal metrics program approved yet and we’ll have a long road to complete the checklist steps and educate management, a simple and short format is best. The risk vernacular is well known to all management, results based on impact to business success objectives; thus a natural format to start our CISO Scorecard is using the risk theme. In this case we’ll start with using the top four risks (less the policy and management ones) to tell the current cyber story, using an example ‘risk heat map’ illustration, followed by our proposed CISO Scorecard. This two slide communication method both sets the stage on what matters to the organization and provides a snapshot of the metrics that move the risks to lower levels.

Preliminary DRAFT for community review Mike.Davis.SD@gmail.com 13

Next we select the initial metrics that best captures those top four risks, starting with the suggested metrics in Table 3.0, which we can potentially capture in our current environment (Note – Once again, the values used below are notional examples for illustrative purpose only, insert YOURS therein).

Table 6.0 – Notional CISO Scorecard elements Perspective Objective Measures Value / trend (sample #’s) Risks / activities

Financial Manage cost 2A - % IT security 5% / steady 2017 budget approved

Data efficiency 9 – Time to fix critical risks 29 days - steady Assess more automation

Customers Breaches / Incidents 16 - # incidents / cost $ 1 (Ransomware) / 0$ CSC training on IR Plan

Partner compliance 4A - % vendors assessed 22% / decreasing Working on backlog

Threat awareness 10 – Aggregate threat level Moderate - Increasing Ransomware & Insiders

Business IdAM 3A - % background check 8B - % key IT using MFA

94% - steady 10% -increasing

user password changes tokens for critical devices

Find risks / issues 7C - % systems monthly scan 15% - Increasing Targeted scans in progress

Fix vulnerabilities 5B - % assets in CMDB 7B - % patched in 30 days

63% - Increasing 19% - increasing

Developing IT / OPS CMDB Legacy servers exposed

Innovation User Education 3C - % user train 88% / decreasing user awareness / phishing

Project support 14 - # of risk assessments 29% - steady Working on backlog

Cyber Maturity 15 – Maturity level 66% - increasing V&V and Conduct survey

We then translate the table into a C-Suite / BoD level set of graphics, as ‘presentation’ counts!

Preliminary DRAFT for community review Mike.Davis.SD@gmail.com 14

Now that we have our notional CISO Scorecard, we need to develop an implementation plan based on the key recommended steps in the SANS guide, appendix item D, providing a draft POA&M, for the below suggested follow on activities:

Develop a formal implementation plan (treat as a project - include objectives, stakeholders, charter as needed)(define team: IT / OPS, PMO and Sec (external POCs?)

Identify primary audience (CIO, BoD, others?). Determine major organizational and business unit objectives; review recent audit reports and recommendations; develop security survey questions and audience -- identify key risk concerns and priorities therein

Establish inventory of security controls associated with mitigation of aforementioned key risks; Identify roles for security controls using RACI and identify associated sponsors (with resources)

Research benchmarks and authoritative sources for security controls (e.g., best practices, quality standards, etc.); Research relevant regulation and contract obligations

Update the metrics that are to be presented in Security Scorecard; Establish data sources and data owners; Creation/Selection of tools to gather data and create Security Scorecard; Update draft Security Scorecard template.

Circulate template and request comment based on RACI / key stakeholders; Perform resource planning with functional managers and data owners; Establish communication protocol for Security Scorecard audience

Pilot first draft of Security Scorecard; Document Security Scorecard update procedures; Document lessons learned and make appropriate corrections to Security Scorecard; Finalize Security Scorecard layout and content; Establish Feedback mechanisms. Formally announce Security Scorecard, et al….

(Part of this metrics effort is benchmarking… http://www.bankinfosecurity.com/whitepapers/cybersecurity-benchmarking-cios-guide-for-reducing-security-anxiety-w-2888 )

Preliminary DRAFT for community review Mike.Davis.SD@gmail.com 15

Appendix: So the operative challenge then is to first decide WHAT information to collect to support security metrics that enable management decisions. We use the NIST, CISO DRG and CIS metrics recommendations to start and bound our selection set (which also aligns to ISO27001/2). Following items are taken from the Security Metrics Strategy to align that view with this CISO Scorecard view. A---NIST Special Publication (SP) 800-55, Performance Measurement Guide for Information Security http://csrc.nist.gov/publications/nistpubs/800-55-Rev1/SP800-55-rev1.pdf 1 – Security budget 2 – Vulnerability management 3 – Access control 4 – Awareness and training 5 – Audit and accountability 6 – Certification and accreditation 7 - Configuration management 8 – Contingency planning 9 – Identification and authentication 10 – Incident response 11 – Maintenance 12 – Media protection 13 – Physical environment 14 – Planning 15 – Personnel security 16 – Risk assessment 17 – Systems and services acquisition 18 – Systems and communications protection 19 – System and information integrity (see also NIST 800-100, Section 7.0 (summarizes 800-55)) B---CIS top 28 metrics suggestions (based / mapped on their CIS top 20 controls) (listed in main body): https://benchmarks.cisecurity.org/tools2/metrics/CIS_Security_Metrics-Quick_Start_Guide_v1.0.0.pdf The CIS Measurement Companion to the CIS Critical Security Controls provides specific information sets and suggested thresholds to measure over 90 of the CSCs; these will be used to execute and implement the final recommended metrics set that supports our BSC view. https://www.cisecurity.org/critical-controls/ In addition, SANS has a CIS 20 CSC based paper on metrics for each of the controls, to assist in execution: https://www.sans.edu/student-files/projects/jwp-caincouture-whitepaper.doc C---A SAMPLE Security Metrics list: Process Area Defined Metric ---Vulnerability Management VM-1: Vulnerability Scan Coverage VM-2: Percent of Systems without Known High Vulnerabilities VM-3: Number of Known Vulnerability Instances (High, Med, Low) ---Patch Management PM-1: Mean Time to Patch Covered Systems (Clients, Servers)

Preliminary DRAFT for community review Mike.Davis.SD@gmail.com 16

PM-2: Number of Patches Deployed ---Configuration Management CM-1: Number of Devices (Clients, Servers, Network, Other) CM-2: Number of Blocked Internet Emails CM-3: Number of Blocked Internet Access Attempts ---Incident Management IM-1: Number of Investigative Support Requests IM-2: Number of Incidents IM-3: Number of Discovered Malware Types IM-4: Number of Malware Agents Remediated IM-5: Number of Compromised Clients ---Risk Management RM-1: Number of Risk Assessments RM-2: List of Risk Assessments Completed RM-3: Number of US-CERT items that apply / remediated ---Awareness and Training AT-1: Number of Awareness Briefings/Communications ---Program Management PG-1: List of Audits (YTD with # of Findings…) PG-2: Project status D- Security Scorecard Checklist (SANS)

1. Create statement of objectives 2. Establish target audience and stakeholders 3. Create a formal project charter and plan 4. Present plan and obtain letter of authorization from CIO to stakeholders 5. Host project kick--‐off meeting (ideally with CIO present) 6. Identify major organizational and business unit objectives 7. Review recent audit reports and recommendations 8. Author questions for Executive Interview(s) 9. Perform executive interviews to identify key risk concerns and priorities 10. Establish inventory of security controls associated with mitigation of aforementioned key risks 11. Identify roles for security controls using RACI and identify associated sponsors ($) 12. Research benchmarks and authoritative sources for security controls (e.g., best practices, quality

standards, etc.) 13. Research relevant regulation and contract obligations 14. Identify and document metrics that are to be presented in Security Scorecard 15. Establish data sources and data owners 16. Creation/Selection of tools to gather data and create Security Scorecard 17. Draft Security Scorecard template 18. Circulate template and request comment based on RACI / $ 19. Perform resource planning with functional managers and data owners 20. Establish communication protocol for Security Scorecard audience 21. Pilot first draft of Security Scorecard 22. Document Security Scorecard update procedures 23. Document lessons learned and make appropriate corrections to Security Scorecard 24. Finalize Security Scorecard layout and content 25. Establish Feedback mechanisms 26. Formally announce Security Scorecard 27. Close project

Preliminary DRAFT for community review Mike.Davis.SD@gmail.com 17

E--- The final “IT” Security metrics approach should account for several perspectives: (1) Supporting the vision / mission / objectives as implemented in each major business unit, (2) Set of effective overall programmatic performance metrics that most organization should employ and (3) User / customer support metrics (e.g., satisfaction, productivity issues and value / RoI). Each “IT” Security effort has their own metrics to support their key efforts iso the overall “IT” Security objectives. These metrics must be well integrated and complementary between themselves and link to the overall organizational metrics where applicable. All security metrics must support key organizational goals and objectives, as listed earlier; thus working backwards to quantify the specific data that can be easily captured at some level and then used to support decision making, resource allocation, etc. For “COMPANY” we propose the following top down measurements, which also must support a balanced scorecard (BSC) view. We start with three major areas to report on: (1) key residual risks and impacts, (2) business value supported and (3) the overall security state. These can then be further defined with suggested supporting data elements. (1). Top technical security risks, which then must be put into a business impact view; (a) weak and inconsistently enforced IdAM (CIS #5, 9, 14, 15 & 16), (b) minimal ITAM / CMDB process and adhoc release management (CIS # 1,2, 3, & 11), (c) intruders & malware (CIS # 6, 12, 19), (d) unclear privacy and data protection methods (CIS # 10 & 13), and (e) weakly correlated security products (CIS #20)(minimizing the ability to quickly spot an intruder). These must all be quantified in terms of business impacts. (2). Business value as measured against the company’s key success factors. (… starting with the company objectives listed earlier, then list / map them…). Translate the key security priorities into supporting those factors, as well as enhanced productivity overall and added competitive advantage. (3) Overall state of security. This high level cyber metric can be overly complex if not confined to ‘security data that matters’ (e.g., the voluminous IT/security logs kept) and founded on ‘outcome’ based metrics. Areas to include at some level are: Cyber vulnerabilities, threat report, Security projects status, and process compliance (and maturity), again as mapped into a BSC view showing risk value and competitive advantage.

Proposed ‘operational’ security metrics (Still a master wish list, need to iterate and phase in key values measures)

Metric process Map to scorecard comments Trouble tickets

Security tool #1 statistics

Security tool #1 statistics

Security tool #1 statistics

Firewall statistics

Time to detect IoCs ???? Speed to halt attack / damages

Number of compromised systems

Botnet / malware infections per device

Help Desk collect? How prevalent malware is

Access controls / attempts

Snapshot of IdAM entry points

how often are employee accounts updated / re-verified

User password status Assess passwords monthly

Relative state of PWs, time from termination to account deletion

Admin account status Assess Admin PWs weekly (% / change)

Relative state of PWs

Third party / suppliers % reviewed, how often

Verifies controls

% of outbound DNS monitored

Preliminary DRAFT for community review Mike.Davis.SD@gmail.com 18

peer-to-peer sharing

number of open ports during some time frame

Incident volume Potential attacks / week?

Total “B” events per month?

Blocked emails Blocked internet access

Antivirus activity AV statistics? Devices reimaged that month

patch management efficiency

Vulnerabilities listed versus still open

# patched systems Mean time to patch Systems with NO highs ALL known vulnerabilities

2nd phase ---- ---- ----

baseline defense coverage metric

devices that have some sort of defense

Data in/data out how much data is moving in and out of the network and through which ports

Physical access Failures, IoT, CCTV, etc.

Industry benchmarks What others measure / report

Cyber maturity Deloitte follow on

Risk Assessments Programs statistic

Audit Status

Training / awareness Phishing, SME courses, new methods