Fix What MattersEd Bellis & Michael Roytman

Nice To Meet You

• CoFounder Risk I/O

About Us

Risk I/O

• Former CISO Orbitz• Contributing Author: Beautiful Security• CSO Magazine/Online Writer

• Data-Driven Vulnerability Intelligence Platform• DataWeek 2012 Top Security Innovator• 3 Startups to Watch - Information Week

• InfoSec Island Blogger

• 16 Hot Startups - eWeek

Ed Bellis

• Naive Grad Student• Still Plays With Legos• Barely Passed Regression Analysis

• Once Jailbroke His iPhone 3G• Has Coolest Job In InfoSec

Michael Roytman

Starting From Scratch

“It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories,

instead of theories to suit facts.”

-Sir Arthur Conan Doyle, 1887

Starting From Scratch

Starting From Scratch

Academia!• GScholar!•  JSTOR!•  IEEE!• ProQuest!

InfoSec Blogs!• CSIOs!• Pen Testers!• Threat Reports!• SOTI/DBIR!!

Twitter!• Thought Leaders (you

know who you are)!• BlackHats!• Vuln Researchers!

Primary Sources!• MITRE!• OSVDB!• NIST CVSS

Committee(s)!•  Internal Message

Boards for ^!Text

CISOs

Data Fundamentalism

Don’t Ignore What a Vulnerability Is: Creation Bias

(http://blog.risk.io/2013/04/data-fundamentalism/)

Jerico/Sushidude @ BlackHat

(https://www.blackhat.com/us-13/briefings.html#Martin)

Luca Allodi - CVSS DDOS

(http://disi.unitn.it/~allodi/allodi-12-badgers.pdf):

Data Fundamentalism - What’s The Big Deal?

”Since 2006 Vulnerabilities have declined by 26 percent.” (http://csrc.nist.gov/groups/SNS/rbac/documents/vulnerability-trends10.pdf)

“The total number of vulnerabilities in 2013 is up 16 percent so far when compared to what we saw in the same time period in 2012. ”

(http://www.symantec.com/content/en/us/enterprise/other_resources/b-intelligence_report_06-2013.en-us.pdf)

What’s Good?

Bad For Vulnerability Statistics:

NVD, OSVDB, ExploitDB, CVSS, Patches, Microsoft Reports, etc, et al, and so on.

Good For Vulnerability Statistics:

Vulnerabilities.

What’s Good?

What’s Good?

What’s Good?

What’s Good?

What’s Good?

What’s Good?

Counterterrorism

Known Groups

Surveillance

Threat Intel, Analysts

Targets, Layouts

Past Incidents, Close Calls

What’s Good?

Uh, Sports?

Opposing Teams, Specific Players

Gameplay

Scouting Reports, Gametape

Roster, Player Skills

Learning from Losing

InfoSec?

Defend Like You’ve Done It Before

Groups, Motivations

Exploits

Vulnerability Definitions

Asset Topology, Actual Vulns on System

Learning from Breaches

Work With What You’ve Got:

Akamai, Safenet

ExploitDB, Metasploit

NVD, MITRE

Add Some Spice

Show Me The Money

23,000,000 Vulnerabilities!

Across 1,000,000 Assets!

Representing 9,500 Companies!

Using 22 Unique Scanners!

Whatchu Know About Dat?(a)

Duplication

Vulnerability Density

Remediation

Duplication

0

225,000

450,000

675,000

900,000

1,125,000

1,350,000

1,575,000

1,800,000

2,025,000

2,250,000

2 or more scanners 3 or more 4 or more 5 or more 6 or more

Duplication - Lessons From a CISO

We Have: F(Number of Scanners) => Number of Duplicate Vulnerabilities

We Want: F(Number of Scanners) => Vulnerability Coverage

Make Decisions At The Margins!

<---------Good Luck!

0

25.0

50.0

75.0

100.0

0 1 2 3 4 5 6

Density

Type of Asset ~Count

Hostname 20,000

Netbios 1000

IP Address 200,000

File 10,000

Url 5,000

Hostname

Netbios

IP

File

Url

0 22.5 45.0 67.5 90.0

CVSS And Remediation Metrics

0

375.0

750.0

1125.0

1500.0

1 2 3 4 5 6 7 8 9 10

Average Time To Close By Severity Oldest Vulnerability By Severity

CVSS And Remediation - Lessons From A CISO

1 2 3 4 5 6 7 8 9 10

Remediation/Lack Thereof, by CVSS

1 2 3 4 5 6 7 8 9 10

NVD Distribution by CVSS

The Kicker - Live Breach Data

1,500,000 !Vulnerabilities Related to Live Breaches Recorded!

June, July 2013 !

CVSS And Remediation - Nope

0

1750.0

3500.0

5250.0

7000.0

1 2 3 4 5 6 7 8 9 10

Oldest Breached Vulnerability By Severity

CVSS - A VERY General Guide For Remediation - Yep

0

37500.0

75000.0

112500.0

150000.0

1 2 3 4 5 6 7 8 9 10

Open Vulns With Breaches Occuring By Severity

The One Billion Dollar Question

Probability(You Will Be Breached On A Particular Open Vulnerability)?

1.98%=(Open Vulnerabilities | Breaches Occurred On Their CVE)/(Total Open Vulnerabilities)

I Love It When You Call Me Big Data

RANDOM VULN

CVSS 10

CVSS 9

CVSS 8

CVSS 6

CVSS 7

CVSS 5

CVSS 4

Has Patch

0 0.01000 0.02000 0.03000 0.04000

Probability A Vulnerability Having Property X Has Observed Breaches

Enter The Security Mendoza Line

Wouldn’t it be nice if we had something that helped us divide who we considered “Amateur” and who we considered “Professional”?

http://riskmanagementinsight.com/riskanalysis/?p=294

Josh Corman expandsthe Security Mendoza Line

“Compute power grows at the rate of doubling about every 2

years”

“Casual attacker power grows at the rate of Metasploit”

http://blog.cognitivedissidents.com/2011/11/01/intro-to-hdmoores-law/

Alex Hutton comes up with Security Mendoza Line

I Love It When You Call Me Big Data

Random Vuln

CVSS 10

Exploit DB

Metasploit

MSP+EDB

0 0.1 0.2 0.2 0.3

Probability A Vulnerability Having Property X Has Observed Breaches

Be Better Than The Gap

I Love It When You Call Me Big Data

Spray and Pray => 2%

CVSS 10 => 4%

Metasploit + ExploitDB => 30%

Thank You

Follow UsBlog: http://blog.risk.ioTwitter: @mroytman

@ebellis@riskio

We’re Hiring! http://www.risk.io/jobs