Download - State of Bug Bounty Report

Transcript
Page 1: State of Bug Bounty Report

THE STATE OF BUG BOUNTY Bugcrowd’s second annual report on the current state of the bug bounty economy

JUNE 2016

Page 2: State of Bug Bounty Report

©BUGCROWD INC. STATE OF BUG BOUNTY REPORT 2015 2

TABLE OF CONTENTS

Introduction 3WHAT EXACTLY IS A BUG BOUNTY?

Executive Summary 5

About the Data Set 6BUGCROWD PLATFORM DATA PUBLIC DATA SOURCES SURVEY DATA IN THIS REPORT

Market Adoption 8ACCESSIBILITY OF BUG BOUNTIESPROGRAM GROWTH OVER TIMEINDUSTRY DIVERSIFICATIONENTERPRISE ENTERING THE MARKET

Submissions and Vulnerabilities 13VULNERABILITY RATING TAXONOMYVULNERABILITIES BY CRITICALITYVULNERABILITIES BY TYPE

Bounty Payouts 16DEFENSIVE VULNERABILITY PRICING MODEL

Researchers 17AGE AND EDUCATIONREGIONAL RESEARCHER ACTIVITYREGIONAL RESEARCHER QUALITYBUG TYPES AND SPECIALIZATIONSRESEARCHER ENGAGEMENT

Conclusion 22

Page 3: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 3©BUGCROWD INC.

INTRODUCTION

What we’re witnessing right now is the maturation of a model that will fundamentally change the way we approach the security, trust and safety of the Internet.

Bug bounty programs are moving from the realm of novelty towards becoming best practice. They provide an opportunity to level the cybersecurity playing field, strengthening the security of products as well as cultivating a mutually rewarding relationship with the security researcher community. While bug bounty programs have been used for over 20 years, widespread adoption by enterprise organizations has just begun to take off within the last few.

Developing, deploying, and managing secure products presents a massive challenge to all Internet-dependant organizations in 2016. The pressure on short time-to-market continues to increase, and attackers are upping their intensity and resourcefulness to capitalize on security vulnerabilities. Product owners must grow and evolve their vulnerability assessment and identification processes to match their adversaries and keep their users safe.

Our second annual State of Bug Bounty Report provides an inside look into the economics and emerging trends of bug bounties, with data collected from Bugcrowd’s platform and other sources throughout 2016. This report is published on a yearly basis for CISOs and other security decision makers to provide a transparent look at the evolving bug bounty market.

In this report, you’ll learn more about the bug bounty ecosystem, the researcher workforce, and how modern organizations are tackling their application security challenges with bug bounties.

THE FIRST BUG BOUNTYThe first bug bounty program was started at Netscape in late 1995 to find bugs in Netscape’s Navigator 2.0 Internet Browser. The idea of this program was to incentivize the security research community to provide feedback on the Netscape Navigator 2.0 by providing cash rewards to anyone who found bugs in their software. Although the program is noted as one of Netscape’s biggest successes, the bug bounty model did not spread quickly among other software companies.

Read the full history of bug bounties >

Page 4: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 4©BUGCROWD INC. RESEARCHERS

In the past year, the term “bug bounty” has become more well known and widely publicized through popular programs such as Tesla Motors’ car hacking program launched mid 2015, and “Hack the Pentagon.” This uptick in interest is portrayed below.

What Exactly is a Bug Bounty?

As bug bounties have gained traction and evolved to achieve organizations’ security assessment goals, additional variables have been introduced to the basic model. As a business, and for the purposes of the State of Bug Bounty Report, we use the term ‘bug bounty’ more holistically, encompassing programs that can be further classified into the below categories.

The majority of today’s bug bounty programs are scoped to web and mobile application targets, although there are several high profile examples of programs run on IoT devices and cars, such as Tesla Motor’s program and General Motor’s program. Other bounties focus on traditional, installable software, including Microsoft’s Bug Bounty program and Google’s Vulnerability Reward Program (VRP).

PROGRAM TYPE + GOAL VISIBILITY INCENTIVE SCOPE

Vulnerability Disclosure Programs: The primary objective of these programs is to ensure there is a single, public, well-defined channel for security issues.

PublicRecognition (i.e. public leaderboard)

Generally broad, accepting anything that could be considered a security risk

Public Bug Bounty Programs: The organization running the bounty typically interacts directly with researchers to incentivize them to submit vulnerabilities.

Public Cash, swag, misc. (i.e. airline miles)

Slightly less broad, anything that could be considered a security risk and requires a fix

Private Programs: A more exclusive and more highly incentivized program, often run via a crowdsourcing platform vendor that provides submission vetting and program management.

Private High cash incentive

Typically more specific scope or focus to encourage testing on a particular aspect of an attack surface - can be either time-boxed, or on an ongoing basis

Differences in the type of program, incentives, time frames, and exclusivity all affect the results of a program. In this report we will address these variables in terms of the various success metrics used by the market.

Figure 1: Google search keyword trends by interest from 2004 depicts an all time peak interest at the beginning of 2016.

DEFINING ‘BUG BOUNTY’A bug bounty is most simply defined as “an incentivized, results-focused program that encourages security researchers to report security issues to the sponsoring organization.”

Learn more about how it works >2005 2007 2009 2011 2013 2015

Page 5: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 5©BUGCROWD INC.

EXECUTIVE SUMMARY

Public bounties are just the beginning Organizations looking to reap the benefits of a traditional public bug bounty program are utilizing private, on-demand and ongoing, bounty programs more and more. 63% of all programs launched have been private. Jump to this finding.

Bug bounties move beyond just technology companies In nearly 300 programs run, our customer base has diversified from mostly tech companies, to now over 25% of programs launched by more traditional verticals such as Financial Services + Banking. Jump to this finding.

Average priority of submissions increases across all programs We saw an overall increase in average priority per vulnerability, up from what we reported in our last report, with regional differences in average priority. Jump to this finding.

XSS continues to dominateThe most commonly discovered vulnerability is still Cross-site Scripting (XSS), which represents over 66% of categorized vulnerabilities disclosed, followed by Cross-site Request Forgery (CSRF). Jump to this finding.

Payouts are on the riseRelated to the increased severity of vulnerability submissions, the all time average bug reward on Bugcrowd’s platform has risen from $200.81 in our first annual report, to $294.70, an increase of 47%. Jump to this finding.

“Super hunters” emergeEarning hundreds of thousands of dollars from bug bounties alone, a tier of ‘super hunters’ is emerging, often getting attention from organizations’ security team recruiting efforts. Jump to this finding.

1

2

3

4

5

6

Bugcrowd’s second annual State of Bug Bounty report provides comprehensive data from organizations running bug bounty programs, researchers participating in them, vulnerabilities discovered and rewards, with a specific focus on trends over the past year. Here are some of those top trends...

Page 6: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 6©BUGCROWD INC. RESEARCHERS

ABOUT THE DATA SET

Our inaugural State of Bug Bounty report, released mid-2015, included data from the programs run during an 18-month period between January 1, 2013 and June 30, 2015. This report adds to that data, including figures from programs run from January 1, 2013 to March 31, 2016. This data is analyzed with a specific focus on trends over the last year.

As one of the largest sources of vulnerability submission and bug bounty data, we aim to present a novel and impactful view of the current state of crowdsourced security testing, including data from external sources when relevant and necessary.

Public Data Sources

Since the origin of Bugcrowd, we’ve maintained a list of all public bug bounty programs as a service to researchers. We utilize this list and related data in this report to generate and estimate some figures related to overall market size.

We’ve also gathered and estimated other public statistics based on various sources of open source intelligence, as well as data from high profile public bounty programs:

• Google’s Vulnerability Reward Program (VRP) • Yahoo’s Bounty Program• Facebook’s Bounty Program• Microsoft’s Bounty Program• Mozilla / Firefox Bounty Program

2015 STATE OF BUG BOUNTY REPORTOur inaugural report launched in June 2015 was the first of its kind and gave a brief overview of the bug bounty economy, including the beginning stages of the bug bounty economy evolution.

View the full 2015 report >

THE LISTStarted in 2012 as an open source list of all companies with known public responsible disclosure programs and policies, ‘The List’ now has over 600 entries contributed by the community and maintained by Bugcrowd.

View the list >

Bugcrowd Platform Data

Bugcrowd platform data includes program data gathered from January 1, 2013 through March 31, 2016:

• 286 total programs, 64% private and 36% public• 54,114 total submissions• $2,054,721 paid out across 6,803 paid submissions and additional payments• 26,782 researchers as of March 31, 2016

Page 7: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 7©BUGCROWD INC. RESEARCHERS

Survey Data in this Report

This year we’ve included survey data from security researchers, as well as organizations engaging in bug bounty programs, through surveys carried out from the end of 2015 to March 2016.

RESEARCHER SURVEY DATA

At the beginning of 2016 we conducted a survey on a subset of our crowd, receiving responses from approximately 500 researchers with experience in all three types of the aforementioned bug bounty programs, and from 51 different countries. This survey data provides context for the growing security research community as well as insight into the potential growth and global sustainability of this economy. Readers should keep in mind that this data is primarily demographic, and is but a subset of Bugcrowd’s community.

COMPANY SURVEY DATA

We also surveyed organizations currently engaging in bug bounties, both with Bugcrowd and independent of Bugcrowd, as well as organizations who are not currently utilizing bug bounty programs. In total, we received survey responses from over 600 security professionals from every major industry and company size.

Figure 2: Bugcrowd researchers who responded to the 2016 State of Bug Bounty survey fit into one of two categories; researchers participating in solely public programs and researchers who are invited to private programs.

Figure 3: Countries of origin represented in the 2016 State of Bug Bounty researcher survey are representative of the crowd as a whole, with respondents hailing primarily from India and the United States.

Figure 4: Companies represented in the 2016 State of Bug Bounty survey by number of employees, representing a specific subset of the market.

Figure 5: Companies represented in the 2016 State of Bug Bounty survey by industry, representing a specific subset of the market.

39.23% India

11.79% United States

4.76% Philippines

4.08% Pakistan

2.72% United Kingdom

2.04% Netherlands

2.04% Italy

2.04% Germany

2.04% Egypt

2.04% Russia

27.44% Other

43.55%

8.20%

8.01%

5.27%

5.08%

4.88%

4.49%

3.71%

3.13%

2.54%

11.13%

Technology

Finance

Professional Services

Healthcare

Government

Education

Consumer

IT & Security

Non-profit

Manufacturing

Other

5,000 Employees+

501 to 5000 Employees

51 to 500 Employees

1 to 50 Employees

Did Not Disclose

27.55%

26.82%

18.98%

24.82%

1.82%

We use this survey data to understand the market in three main areas: the value of bug bounties, challenges in implementing bug bounty programs, and potential for future growth. We find this data is representative of the broader market, with a wide range of organizations and researchers involved.

62.98% Private Researchers

37.02% Public Researchers

Page 8: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 8©BUGCROWD INC. RESEARCHERS

MARKET ADOPTION

Bug bounties, in their traditional form, were originally uncapped “blank check” affairs, introduced by technology giants such as Facebook, Google, Yahoo and a few others. These giants activated marketplaces exchanging security vulnerabilities for cash. The organizations who started this market have spent over $13 million on bug bounty payouts through the beginning of 2016.

Today, organizations are able to access the benefits of bug bounties without a blank check, and thus they are being adopted by all types of organizations; from startups to enterprises, and from virtually every industry. Most notably in the past year, organizations such as United Airlines, the United States Department of Defense, Tesla Motors and General Motors started bug bounty programs, garnering attention from worldwide press outlets.

As they become more accessible, more organizations are starting bug bounty programs, and many more are reviewing the prospect of introducing them.

BUG BOUNTY GIANTS

The top paying bug bounty programs are Google VRP, with over $6M paid out over since 2010, Facebook, with $4.3M paid out since 2013, Yahoo, with $1.6M paid out since 2013, Mozilla and Firefox programs having paid out nearly $1M to date since 2010, and Microsoft with over $500K paid out over all their programs.

Read more about these programs >

Accessibility of Bug Bounties

From survey data, 37% of respondents work within companies who have either run, or are currently running a responsible disclosure or bug bounty program. Of those programs, over half offered rewards, including swag, while the rest offered recognition only.

From those respondents, the top three points of perceived value of bug bounties are the diversity in skill sets and methods used by hackers, the volume of bug hunters testing their applications and the pay for results model.

Figure 6: Top perceived value of bug bounties from survey respondents working in organizations running a bug bounty program, currently or previously.

Additional responses about the value of bug bounties included building a relationship with the security researcher community, positive external marketing, and internal education tools.

When asked which roadblocks or apprehensions their organization overcame in order to start a program, the top response was ‘skepticism around quality of results.’ However, when asked what about the quality of the program results compared to other, more traditional methods, 63% stated that the results from crowdsourced programs were better or the same, while just 4% stated that the results were worse. Additionally, 64% reported that they would spend additional or equal resources on their program, while 4% reported they would spend less.

0% 10% 20% 30% 40% 50% 60% 70%

Positive Marketing Impact

Results Based Rewards Model

Volume of Testers

Creative Testing Methods

Page 9: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 9©BUGCROWD INC. RESEARCHERS

Figure 7: Top application security efforts being carried out by companies represented in survey data.

Accessibility of Bug Bounties (cont.)

The bug bounty economy is growing rapidly, and yet it still has a long way to go as proven by recent research stating that 94% of companies on the Forbes 2000 list do not currently have a vulnerability disclosure or bug bounty program.

Our findings show that most organizations have a fairly comprehensive suite of application testing methods, as expected in a modern security organization. The below graph shows what application security efforts are utilized by companies who are not running a bounty program, 63% of total respondents.

Of that same data set, 18% stated that their organization is currently reviewing launching a bug bounty program, and of the remaining that are not currently considering it at present, 28% were optimistic about their organization considering it in the future.

Additionally, of those reviewing starting a bug bounty program within their organization, the top roadblock was budgeting (71.93%). Of those not currently reviewing a bug bounty program, the top reasons were uncertainty of where to begin (36.78%), internal bureaucracy (34.48%) and insufficient budget or resources (28.35%).

ADDITIONAL LEARNINGS:

COMPARED TO PENETRATION TESTING

Bug bounties are often compared to traditional application security assessment methods such as a penetration testing. The biggest differences between the two are volume of testers involved and the differing reward models. Bug bounties involve a large volume of researchers, as opposed to a select few penetration testers, and utilize a pay for results reward model rather than for effort.

The volume and diversity of security researchers participating in bug bounty programs results in a diverse range of bug types, classes and criticality of vulnerabilities, and testing is usually performed without prior knowledge of the target.

0% 10% 20% 30% 40% 50% 60% 70% 80%

Other

Threat modeling

Code review

Compliance reviews/audits

Application security training

Static analysis

Vulnerability scanning

Penetration testing

Page 10: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 10©BUGCROWD INC. RESEARCHERS

Program Growth Over Time

In our first annual State of Bug Bounty Report, we outlined the growth of programs on Bugcrowd’s platform. Program growth continues to increase over 210% on average year over year. Additionally, in the 2015 report we highlighted the emerging trend of private programs, surpassing the launch of public programs. This trend continues in its trajectory; as of March 31 2016, 63% of all Bugcrowd program launches have been private programs.

Figure 8: Private and public program launches on the Bugcrowd platform from January 2013 to March 2016 show that private launches surpassed public launches and continue to rise in popularity with more velocity than that of public programs.

PRIVATE PROGRAMSPrivate programs include both on-demand and ongoing programs in which researchers must be invited to participate. In general, we conclude that demand for private programs has continued for three main reasons:

• Organizations looking to start a public bug bounty program begin privately, incentivizing a smaller number of researchers while they build their response capabilities. Over time, the programs become public, allowing everyone to participate.

• Organizations looking to access the benefits of crowdsourcing with specific business goals, complex technologies or environments benefit from a smaller testing pool. These organizations pay higher bounties to attract and maintain interest from the top researcher talent.

PRIVATE RESEARCHERSIn order to receive invitations to Bugcrowd’s private programs, researchers must score high in all of the following measures: trust, acceptance rate and overall submission quality, finding severity and activity.

Private researchers not only have a good track record of adhering to community guidelines and program briefs, but also have a priority rate of better than 4.0, a minimum acceptance rate of 50% and have been active in the past 90 days.

More on private program invitations >

50

100

150

200

2016-Q12015-Q12014-Q12013-Q1

Private

Public

Private Programs

Page 11: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 11©BUGCROWD INC. RESEARCHERS

Industry Diversification

The expansion of bug bounties is proven by the growth in overall programs launched over time, as well as the diversity in industries. Out of the nearly 300 programs in the past three years, Bugcrowd has launched bug bounty programs for organizations from nearly every industry.

Of all programs launched on the Bugcrowd platform (left), the top two industries represented are Computer Software, including companies like Heroku and Adobe, and Internet, including companies like Pinterest and Indeed.com. Following those industries, most notably, are Financial Services + Banking, Information Technology + Services, Computer + Network Security, and Retail + E-Commerce.

These results are in line with public data of all known vulnerability disclosure programs (right). The most notable difference is a smaller portion of traditional industries such as Financial Services + Banking represented. This is likely because more of those organizations often choose to run private programs.

Figure 11: Programs launched by quarter, broken down by industry shows increasing ‘traditional’ sectors starting more programs, quarter over quarter.

Over time, we’ve seen continued industry diversification on the Bugcrowd platform, with prominent traction from Retail + E-Commerce and Financial Services + Banking over the past twelve months. Overall, organizations from more “traditional” industries have seen year over year growth of over 217% on average, including Financial Services + Banking, Automotive, Healthcare, Education, Telecommunications, Hospitality, Real Estate, Utilities and Consumer Goods.

Figure 9: Breakdown of all programs launched on Bugcrowd’s platform by industry. Figure 10: Company industries represented in public data of all known public bug bounty programs.

22.30%

21.60%

11.50%

10.45%

6.27%

4.88%

3.83%

3.14%

2.44%

2.44%

11.15%

Computer Software

Internet

IT + Services

Financial Services + Banking

Computer + Network Security

Non-profit

Retail + E-Commerce

Consumer Electronics

Computer Networking

Marketing + Advertising

Other

21.13%

15.33%

13.24%

6.85%

4.76%

4.76%

3.27%

3.27%

3.13%

2.53%

21.73%

Computer Software

Internet

IT + Services

Financial Services + Banking

Business Services

Computer + Network Security

Computer Networking

Entertainment

Marketing and Advertising

Retail + E-Commerce

Other

0

10

20

30

40

50

2016-Q12015-Q42015-Q32015-Q22015-Q12014-Q42014-Q32014-Q22014-Q12013-Q42013-Q32013-Q22013-Q1

22.30%

21.60%

11.50%

10.45%

6.27%

4.88%

3.83%

3.14%

2.44%

2.44%

11.15%

Computer Software

Internet

IT + Services

Financial Services + Banking

Computer + Network Security

Non-profit

Retail + E-Commerce

Consumer Electronics

Computer Networking

Marketing + Advertising

Other

Industry Diversification

Page 12: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 12©BUGCROWD INC. RESEARCHERS

5,000 Employees+

500 to 4,999 Employees

200 to 499 Employees

50 to 199 Employees

1 to 49 Employees

Enterprise Entering the Market

In addition to more varied industries adopting the bug bounty model, we’ve seen diversification in the sizes of companies adopting the bug bounty model as well.

Most companies launched on the Bugcrowd platform have 500 to 5,000 employees (28.67%), followed by 51 to 200 employees (24.48%).

As shown in the figure below, smaller companies were the first to adopt the model, but we are seeing more mid-market and enterprise interest.

The enterprise, defined as organizations with 5,000+ employees, account for the fastest growth of program launches on the Bugcrowd platform over the past twelve months.

Figure 13: Programs launched by quarter, by number of employees, showing that bigger sized companies are adopting bug bounties.

Figure 12: Company size by number of employees in all time programs launched.

We attribute larger companies’ bug bounty adoption to the evolution of the traditional bug bounty model, and to the popularization of private programs. Private programs are more conducive to organzations with more compliance requirements such as the Payment Card Industry Data Security Standard (PCI DSS) and Sarbanes Oxley (SOX), while retaining the integrity of the bug bounty model and delivering the value of the crowd.

0

10

20

30

40

50

2016-Q12015-Q12014-Q12013-Q1

5,000 Employees+

500 to 4,999 Employees

200 to 499 Employees

50 to 199 Employees

1 to 49 Employees

9.79%

24.48%

13.99%

23.08%

28.67%

Page 13: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 13©BUGCROWD INC. RESEARCHERS

SUBMISSIONS AND VULNERABILITIES

We have received 54,114 submissions on the Bugcrowd platform between January 1, 2013 and March 31, 2016.

Of those total submissions, 24,516 (45.38%) were marked invalid and 19,574 (36.23%) were marked duplicate. Valid, non-duplicate submissions account for the remaining 9,963 submissions, resulting in a signal-to-noise ratio of 18%.

The signal-to-noise ratio significantly affects the total ownership cost of a program. The more time an organization spends on processing submissions that don’t produce a signal, the more overhead they experience in the program. Private programs overall have a much better signal to noise ratio of 29% compared to 13% in public programs.

Market education, guidance and standardization will be key for both researchers and organizations as the bug bounty model matures, encouraging more high value and valid submissions, better communication with one another and aligned expectations.

Figure 14: Cumulative submission data broken out by all submissions, all valid submissions, and all valid non-duplicate submissions.

Vulnerability Rating Taxonomy

In 2016, Bugcrowd released its Vulnerability Rating Taxonomy (VRT) to help align expectations about the criticality of a bug bounty submission. Bugcrowd’s Technical Operations team follows the VRT, using it to rate the technical priority of submissions.

Once Bugcrowd receives a submission, a VRT submission type and technical priority level are assigned, allowing the program owner to quickly understand the urgency of a submission.

The criticality scale for a submission ranges from Priority 1 (P1) to Priority 5 (P5), 1 being the most critical, 5 being the least critical. This scale provides researchers and organizations a baseline for prioritization of a fix and potential reward amount.

VRT EVOLUTION The VRT is a living document that changes regularly, so specific submission ratings and notes are frequently updated. To understand all considerations, implications and use cases for this document, download a copy of the VRT and read the accompanying report.

Get the full guide >

0

10K

20K

30K

40K

50K

60K

2016-Q12015-Q12014-Q12013-Q1

Total Submissions (cumulative)

Total Valid Submissions (cumulative)

Non Duplicate Valid Submissions

(cumulative)

Page 14: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 14©BUGCROWD INC. RESEARCHERS

Vulnerabilities by Criticality

To measure the health of individual bounty programs and the bug bounty economy as a whole, we have tracked the average priority of a bug across submissions. The current average priority of all time submitted bugs is 3.75, better than it was twelve months ago: 3.88. Lower impact issues (i.e. P5s) are easier to find and can often be discovered by automated vulnerability scanners, thus researchers are often discouraged from submitting them. On the flip side, more critical vulnerabilities are much harder to find. For that reason, P1 and P5 vulnerabilities account for a smaller portion of valid submissions.

Since we started to disincentivize P5 submissions, they have accounted for less of all submissions over time. Higher impact submissions are highly incentivized on Bugcrowd’s platform and reports are rising in frequency. As more P1s and P2s are submitted over time, coupled with less P5s over time, we’ve seen the average priority of a bug get significantly better.

Figure 16: Submission volume by priority over time shows a steady decrease in lower priority submissions and an increase in higher priority submissions.

Figure 15: All valid submissions, including duplicates, by priority.

VULNERABILITIES BY PRIORITY:

• P1 - Critical: Vulnerabilities that cause a privilege escalation from unprivileged to admin or allow for remote execution, financial theft, etc.

• P2 - High: Vulnerabilities that affect the security of the platform including the processes it supports

• P3 - Medium: Vulnerabilities that affect multiple users and require little or no user interaction to trigger

• P4 - Low: Vulnerabilities that affect singular users and require interaction or significant prerequisites to trigger (MitM) to trigger

• P5 - Acceptable Risk: Non-exploitable vulnerabilities in functionality. Vulnerabilities that are by design or are deemed acceptable business risk to the customer

2.81%

14.62%

24.95%

45.23%

12.39%

P1

P2

P3

P4

P5

1K

2K

3K

4K

5K

2016-Q12015-Q12014-Q12013-Q1

P1

P2

P3

P4

P5

Priority up

Page 15: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 15©BUGCROWD INC. RESEARCHERS

Vulnerabilities by Type

Cross site scripting (XSS) and Cross Site Request Forgery (CSRF) are still the top vulnerability submissions to all Bugcrowd programs, which is consistent with other publicly available bug bounty data. 38% of all valid and duplicate submissions fall into the category of XSS, CSRF, mobile, SQLi and clickjack. Uncategorized bugs omitted, XSS vulnerabilities account for 66% of valid submissions, followed by 20% categorized as CSRF. Read more about the prevalence of XSS, as well as other bugs not classified in this data in our October 2016 Addendum (pages 23-25 of this report).

Consistent with the VRT, our ranking system discourages issues with low technical impact such as clickjacking, thus we’ve seen a decline in clickjacking and other low-impact reports over time.

Figure 17: Bug types across all valid submissions including uncategorized submissions. Figure 18: Bug types across valid submissions excluding uncategorized submissions.

Figure 19: Bug types across valid submissions shows a decline in low value bug types such as clickjacking, and steady submissions in XSS and mobile bugs.

0

1K

2K

3K

2016-Q12015-Q12014-Q1

XSS

SQLi

Mobile

CSRF

Clickjack

25.35%

7.54%

3.37%

1.39%

0.62%

61.72%

XSS

CSRF

Mobile

SQLi

Clickjack

Other

66.24%

19.71%

8.79%

3.64%

1.62%

XSS

CSRF

Mobile

SQLi

Clickjack

XSS

Page 16: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 16©BUGCROWD INC. RESEARCHERS

$20K

$40K

$60K

$80K

$100K

$120KTotal sum of Amount Cents

Unique Paid Submitters

BOUNTY PAYOUTS

Bugcrowd has paid out $2,054,721 across 6,803 paid vulnerability submissions and additional payments as of March 31, 2016. Total payout distribution is skewed heavily to a small portion of the crowd, signifying the strength and success of ‘super hunters’ which we will address in the following section.

TOP BUG PAYOUTSOn the Bugcrowd platform, multiple top payouts of $15K have been made in the past year, up from last year’s top payout of $10K.

DVPM EXAMPLE:According to the DVPM, a P4 submitted to an organization on the ‘basic’ end of the security maturity scale should be rewarded $100, while a P1 submitted to an organization on the ‘advanced’ end of the security maturity scale should be rewarded $15,000.

Read the full guide >

Beyond the distribution of payouts to individuals, we look at the average payout per bug as a metric of health for the entire bug bounty economy. The all time average payout of a bug is currently $294.70, up 47% from the all-time average reported in the 2015 State of Bug Bounty Report, $200.81.

Defensive Vulnerability Pricing Model

The Defensive Vulnerability Pricing Model (DVPM) first focuses on security maturity based on four variables: security philosophy, people, processes, and technologies. Based on the security maturity , an organization may be classified as ‘basic,’ ‘progressing’ or advanced.’ Using technical priority and an organization or application’s security maturity level, the DVPM helps organizations determine what a bug is worth. The DVPM set the first ever market rate for bug bounties, guiding and setting expectations for both researchers and organizations.

Figure 20: Distribution of cumulative payments cross individual paid submitters.

Figure 21: Average payout per bug shows steady increase quarter over quarter.

Notice that as the average payout begins to stabilize at the beginning of 2015, we notice a significant upward trend quarter over quarter.

In early 2016, we published our first ever Defensive Vulnerability Pricing Model to provide guidance to organizations considering a bug bounty on how to budget and reward submissions.

This guidance is beginning to pay off; the average bug payout in just the first quarter of 2016 was at an all time high of $505.79.

$100

$200

$300

$400

$500

$600

2016-Q12015-Q12014-Q12013-Q1

Payouts

Page 17: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 17©BUGCROWD INC. RESEARCHERS

RESEARCHERS

As of March 31, 2016, 26,782 researchers had signed up on the Bugcrowd platform. Notably, 41% of those researchers signed up in the past twelve months alone, signifying steady community growth year over year.

Age and Education

Of those who responded to our survey, 75% of researchers were between the ages of 18 and 29 followed by the second largest age group, aged 30 to 44, representing 19% of respondents. Additionally, 88% of them have at least one year of college under their belts. 55% of them have graduated with a bachelor’s or postgraduate degree. All respondents had at least a high school degree.

Figure 21: Cumulative researcher sign-ups quarterly since January 2013.

Figure 22: Researcher survey data showing the distribution of ages represented amongst researchers.

Figure 23: Researcher survey data showing level of education completed.

0

5K

10K

15K

20K

25K

30K

2016-Q12015-Q12014-Q12013-Q1

42.14%

28.70%

12.76%

12.30%

4.10%

College Degree

Some College

Graduate Degree

High School Degree

Some Graduate School

0%

20%

40%

60%

80%

45 - 5930 - 4418 - 29< 18

Age in Years

Page 18: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 18©BUGCROWD INC. RESEARCHERS

Regional Researcher Activity

Bugcrowd researchers hail from 112 different countries; activity and quality varies by region. The vast majority of researcher sign ups are from India (28.2%) and the United States (24.4%), followed by the United Kingdom (3.9%), Pakistan (3.5%) and Australia (2.4%).

In terms of submission volume, however, the top ten submitting countries are, in order of highest submission volume to lowest, India, the United States, Pakistan, the United Kingdom, the Philippines, Germany, Malaysia, the Netherlands, Australia and Tunisia. Over time, we have seen steady submissions from India and the United States, with a steady decrease in submissions from Pakistan.

We analyze activity and quality trends primarily in terms of total submissions, valid submissions and total amount paid. Total submission volume (left) versus total paid out (right) show differences in quality by region.

Figure 25: Breakdown of total submission volume by geography. Figure 26: Breakdown of total payment volume by geography.

Figure 24: Submission volume by geographical location quarter over quarter.

India ranked first for total submission volume as well as total amount of money paid out. Notably, Portugal is the second ranked country for payout volume without making the top ten submitting regions by total volume. Additionally, while countries like Australia and Tunisia are in the top ten for submission volume, they don’t make it in the top ten for total money paid out. This signifies that researchers in those regions contributed less valid submissions that resulted in payment.

India

United States

Pakistan

United Kingdom

Philippines

Other

0

2K

4K

6K

8K

10K

2016-Q12015-Q12014-Q12013-Q1

43.04%

12.79%

11.50%

3.87%

3.28%

1.63%

1.52%

1.31%

1.30%

1.23%

18.53%

India

United States

Pakistan

United Kingdom

Philippines

Germany

Malaysia

Netherlands

Australia

Tunisia

Other

35.51%

12.94%

12.15%

12.13%

6.52%

2.67%

2.47%

2.18%

1.68%

1.40%

10.35%

India

Portugal

United States

United Kingdom

Malaysia

Ukraine

Philippines

Pakistan

Netherlands

Germany

Other

Page 19: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 19©BUGCROWD INC. RESEARCHERS

37.44%

19.60%

12.04%

2.38%

2.17%

2.14%

1.96%

1.25%

1.22%

1.16%

18.65%

India

United States

Pakistan

United Kingdom

Tunisia

Hong Kong

Philippines

Germany

Australia

Netherlands

Other

Regional Researcher Quality

The above trends are also reflected in regional submission priority, as depicted by analyzing the breakdown of vulnerabilities by priority. The top five P1 submitters by region are the United States (33.81%), India (13.12%), Portugal (7.42%), the United Kingdom (6.42%) and Germany (3.99%).

Figure 27: P5 submission volume broken down by geography.

Figure 29: Average priority by top submitting regions depicts above average priority in European nations such as Portugal, France, United Kingdom and Netherlands, and below average priority from India, Pakistan, Tunisia and Egypt.

Looking at top submitting regions, one measure of quality is average vulnerability priority. Remember, the lower the average numerical priority, the better, meaning more critical vulnerabilities out of total submissions. Of the top fifteen submitting countries, Portugal has the top average priority of 2.91, compared to the overall average submission priority of 3.75 across all submissions.

Figure 28: P1 submission volume broken down by geography.

33.81%

13.12%

7.42%

6.42%

3.99%

3.42%

3.28%

3.00%

3.00%

2.28%

20.26%

United States

India

Portugal

United Kingdom

Germany

Russia

Netherlands

Canada

France

Australia

Other

P3P1 P4 P5

Italy

Portugal

Egypt

Russia

France

Tunisia

Australia

Netherlands

Malaysia

Germany

Philippines

United Kingdom

Pakistan

United States

India 3.97

3.65

4.58

3.28

3.67

3.58

3.42

3.34

3.60

4.57

3.14

3.40

4.07

2.91

3.57

‘SUPER HUNTERS’As reported previously, a vast majority of payouts go to a select group of individuals. The top ten paid out researchers have made, collectively, 23% of total payouts.

These individuals from around the world have made names for themselves, garnering attention from the security researcher community.

Some, from less expected regions, have been so consistent and successful, they have put their entire countries on our radar. For example, Portugal’s success is from just a few researchers.

Super hunters, although not a entirely new phenomenon, are making more money than ever, as more complex and high profile bounty programs launch with higher stakes.

Meet Bugcrowd’s top bug hunters >

Page 20: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 20©BUGCROWD INC. RESEARCHERS

Bug Types and Specializations

Another way we measure and segment researchers is by skillsets or specializations. Although there are less variances in bug type by geography, we notice a few key trends.

While volume isn’t represented in the graph below, the breakdown of bug types of the top ten submitting region reveals key trends. SQLi, which is often categorized as a high value vulnerability, accounts for the greatest percentage of all vulnerabilities submitted by researchers in Tunisia, Portugal, and the United States. Similarly, Tunisia, Pakistan, India and the Philippines account for the most clickjack vulnerabilities, often categorized as a low value vulnerability. Additionally, mobile submissions account for the biggest portion of submissions from Australia and the United States.

When asked which technologies they had intermediate to advanced skill in, 95% of respondents of our aforementioned survey felt they had intermediate or advanced knowledge of web application testing, 48% in Android, 28% in iOS and 15% in IoT. While the Bugcrowd community is made up of security researchers with expertise across numerous technologies, accessibility, complexity and opportunity contribute largely to these responses.

Figure 31: Survey data showing researchers’ advanced and intermediate level skill sets.

0% 25% 50% 75% 100%

SCADA

Mobile OS/Baseband

Mobile App - other

IoT/hardware

Malware Analysis

Network Appliance

Reverse Engineering

Mobile App - iOS

Desktop/Server software

Linux

Network Infrastructure

Mobile App - Android

Code Review

APIs/Web Services

Web App

Figure 30: Top submitting countries’ total valid submission volume by percentage of each bug type, excluding unclassified bugs.

0% 100%

Australia

Portugal

Russia

Tunisia

Netherlands

France

Malaysia

Germany

Philippines

United Kingdom

Pakistan

United States

India XSS

SQLi

Mobile

CSRF

Clickjack

As more organizations utilize bug bounties in different technologies and applications and education around complex technologies becomes more accessible and available, we foresee diversification of depth and breadth of skills to be manifested in the crowd.

Page 21: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 21©BUGCROWD INC. RESEARCHERS

Researcher Engagement

Based on survey responses, we discovered that just 15% of researchers participate in bug bounty programs full-time, with an additional 31% hoping to participate full-time in the future. Thus, 85% of Bugcrowd researchers participate in bug bounty programs as a hobby or view it as a part-time job currently, from which we infer their primary source of income is independent from bug hunting. Additionally, 70% spend fewer than 10 hours a week working on bounties.

From our experience, while we’ve seen multiple super payouts of $10K and $15K, and ‘super hunters’ making anywhere from $9K to $20K monthly, most of the crowd is bug hunting as a secondary source of income. In other words, the majority of bug hunters are employed, making their living independent of bug hunting, oftentimes as developers, security engineers or penetration testers.

Furthermore, when asked how much money they would need to make in order to do security research full time, respondents’ answers varied greatly, representative of the geographical diversity and differences in regional average incomes. 43% of respondents would need to make $0 to $50,000 while nearly 30% would need to make more than $100,000.

Figure 34: Survey responses of how much they would need to make, annually to bug hunt full time.

Figure 32: Survey responses to the question “Do you bug hunt full time?” Figure 33: Average hours a week spent participating in bug bounty programs.

Researchers are motivated by a range of incentives, extrinsic and intrinsic, from prestige or profit, to philanthropy or professional development. As the community grows and we learn more about it, we leverage these motivations to better assist this flourishing marketplace. While there is certainly more money becoming available in this marketplace, as proof of this report, Bugcrowd also has the unique opportunity to continue supporting the crowd.

This community will be forever evolving and growing, and we will continue to analyze and report on the state of bug hunters and the security research economy.

15.00% Yes

54.09% No, part time only

30.91% No, but hopefully someday

0

10%

20%

30%

40%

50%

40+31 - 4021 - 3011 - 206 - 100 - 5

Time Spent Per Week in Hours

0% 5% 10% 15% 20% 25%

$125,000+

$100,000 - $124,999

$75,000 - $99,999

$50,000 - $74,999

$25,000 - $49,999

$0 - $24,999

Page 22: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 22©BUGCROWD INC.

CONCLUSION

In 2016, bug bounty programs are emerging as a key component of organizations’ security programs. The bug bounty path, paved by tech giants, is widening, enabling security teams of all sizes to create and manage robust application security assessment programs, get ahead of adversaries, and level the cybersecurity playing field. While we are clearly still in the early- to mid-adopter phase of this new market, this report proves that bug bounties are gaining momentum and evolving to meet those needs.

Case-in-point, with over $2M paid to researchers and velocity increasing with a current average reward of over $500, bug bounty programs are becoming very lucrative for researchers around the world. Researchers are building their reputation and obtaining access to private programs, allowing them to earn even more.

With more Financial Services + Banking organizations adopting the model and starting programs, we predict that rewards will continue to rise both in frequency and size.

Bugcrowd is here to help you get started. If you’re interested in starting a program or learning more, we encourage you to reach out to us at https://bugcrowd.com.

ABOUT BUGCROWDThe pioneer and innovator in crowdsourced security testing for the enterprise, Bugcrowd harnesses the power of more than 30,000 security researchers to surface critical software vulnerabilities and level the playing field in cybersecurity. Bugcrowd also provides a range of responsible disclosure and managed service options that allow companies to commission a customized security testing program that fits their specific requirements. Bugcrowd’s proprietary vulnerability disclosure platform is deployed by Drupal, Pinterest, Western Union and many others. Based in San Francisco, Bugcrowd is backed by Blackbird Ventures, Costanoa Venture Capital, Industry Ventures, Paladin Capital Group, Rally Ventures and Salesforce Ventures.

Page 23: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 23©Bugcrowd inc. RESEARCHERS

VOLUME AND IMPACT OF CROSS-SITE

SCRIPTING IN BUG BOUNTIES

SEPTEMBER 26, 2016

As of March 31, 2015, we reported that over 25% of all valid submissions were categorized as Cross-Site Scripting vulnerabilities, following the largest category of vulnerabilities, ‘Other’ (62%). Since releasing our second annual ‘State of Bug Bounty’ report in June, we’ve received inquiries about this statistic. This addendum will address those questions, as well as...

• Provide context around the volume and velocity of XSS over the past 10+ years• Explore the potential impact XSS can have • Analyze our general state of our vulnerability data and other high impact vulnerabilities

Current State of Cross-Site Scripting

Before we address the potential impact of XSS, it’s important to provide context around the current perception of XSS in the vulnerability disclosure space. The frequency and persistence of XSS in headlines, POCs and vulnerability databases over the past 10+ years have created ‘XSS-fatigue.’

The security community–consultants and researchers–have been submitting XSS in reports for a long time. But instead of proving the impact of an XSS vulnerability and exploiting a full attack by taking over accounts etc., the industry has standardized on a JavaScript pop-up or prompt box. In short, the industry has written off XSS as low-hanging fruit, partially because it has been around for so long, and also because bug reports are downplayed.

In reality, however, this isn’t how it happens in the real world. There are many exploit frameworks that rely on XSS, including ransomware attacks, nation-state attacks, and more. While it is true that some XSS vulnerabilities have little notable impact, that is certainly not always the case. Bug bounties are here for when that isn’t the case.

Cross-Site Scripting Impact

Our vulnerability data shows that 27% of all valid XSS were classified as P1 or P2, followed by 29% P3, which is to say not all XSS are ‘low hanging fruit.’ Our Vulnerability Rating Taxonomy (VRT) has multiple classifications for XSS, capturing priority variations for XSS within applications with multiple user privilege levels. Stored Cross-Site Scripting is listed as a Priority 2 (P2) vulnerability when privilege escalation can occur from non-admin to anyone, and a Priority 3 (P3) vulnerability for XSS when privilege escalation can occur from admin to anyone.

When coupled with the appropriate business impact, XSS can, and has been, classified and rewarded as a Priority 1 (P1) vulnerability as explored in the following section. Generally, XSS can have critical business impact in two instances; when the attack vector is unusual or obscure, or when the users involved or business impact are notable.

ADDENDUM

Page 24: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 24©BUGCROWD INC. RESEARCHERS

XSS in Bug Bounties

This longevity and persistence is unique to XSS due to the technical cause of the vulnerability, as well as how difficult it is to avoid it. What is more–it shows no signs of being resolved anytime soon.

It’s also important to note that most organizations don’t start a bug bounty program, public or private, without already having a mature application security program in place. The vast majority of our customers have already run various vulnerability scanners against their attack surface, as well as penetration tests. Most scanners historically fail to pick up many of these XSS findings–the power of human creativity cannot be replicated by scanners. Bug bounties are often useful to find ‘the unknown unknowns.’ Below are two specific examples XSS bugs submitted through a Bugcrowd bug bounty program with particularly high impact.

EXAMPLE 1: BLIND XSS

The Company: Communications company

The Bug: Blind XSS is a variant of XSS in which the attacker injects the script payload via a form that doesn’t immediately indicate to the attacker that the script was executed. In this specific example, the researcher injected the payload via the company’s interactive support chat. Eventually, this payload was executed by an employee of the company who viewed the other end of the chat conversation, allowing the researcher to thereby control a browser within the company’s internal network.

Impact: In this scenario, the researcher was able to bypass several security mechanisms protecting the company’s internal network by executing a payload within the help desk technician’s internal browser. This kind of access to an internal network – especially with a privileged user account – is often the entry point for a much larger-scale network compromise. While some think that Blind XSS is just a form of advanced phishing, this is actually not the case, as the root cause is that the website itself allowed an attacker to send and ultimately rendered the attacker’s script payload. All websites – whether externally or internally facing – should always perform proper input validation and output encoding.

EXAMPLE 2: PRIVILEGE ESCALATION AND REMOTE CODE EXECUTION VIA XSS

The Company: E-commerce company using multi-tenant content management systems (CMS)

The Bug: In this example, a researcher found an XSS that would allow an attacker to force administrative users to promote non-admin users to an admin role. Privilege escalation is common in CMSs with multiple roles within an application, and in this instance, the admin had virtually unlimited access within in the CMS. After using XSS to allow for silent and automatic account upgrading to admin, the attacker was then able to to use his new account to inject more code, including adding a web shell to the server to take over the server.

Impact: Some will not accept this bug because they believe it requires a phish on the part of the admin, which is unlikely to happen, but in most cases, admins are just as susceptible to phishing and XSS as anybody else. In this case, the attack in the JavaScript was so cleverly crafted with CSS modification and left all the hooks and codes in the background, so the admin never even knew what was going on. This wouldn’t have been possible without XSS.

Page 25: State of Bug Bounty Report

STATE OF BUG BOUNTY 2016 25©BUGCROWD INC. RESEARCHERS

For additional examples of high-impact Cross-Site Scripting vulnerabilities, listen to our recent ‘Big Bugs’ Episode, XSS Fatigue, with Jason Haddix, Bugcrowd’s Head of Trust and Privacy.

‘Other’ Bugs

As stated in this report, 62% of all bugs fall into the but type category of ‘Other.’ In writing this report, we included our vulnerability data as-is to provide an accurate, albeit simplified, snapshot of our vulnerability data while we work to classify all submissions into more granular categories. This will minimize vulnerabilities listed in ‘Other’ which actually holds a multitude of vulnerability types, providing richer, more meaningful data to the market as a whole.

The goal in providing categories or classes of vulnerability types is to better understand the current state of web applications, mobile applications, and connected devices. Again, it’s important to keep in mind that the vulnerabilities we are finding are those that have been missed by scanners, or overlooked by penetration testers, which is why it is notable to call out that XSS and CSRF are such a big percentage of valid vulnerabilities, behind our unclassified portion.

So what kinds of bugs are in this category? As one can imagine, it varies greatly, but notable bug types include IDOR, XXE, RCE and more. Below are just a few examples of ‘Other’ bug write-ups with significant impact.

A FEW EXAMPLES

• Blackphone remote memory corruption vulnerability• Netgear remote code execution (RCE)• Aruba Reversing Firmware

Addendum Sources and Additional Resources

• http://www.thesecuritypractice.com/the_security_practice/2010/11/how-cross-site-scripting-was-discovered.html • https://www.owasp.org/index.php/Cross-site_Scripting_(XSS) • http://techbeacon.com/sites/default/files/gated_asset/hpe-cyber-risk-report-2016.pdf • https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Taking_still_photos• https://www.owasp.org/index.php/XSS_Filter_Evasion_Cheat_Sheet• https://www.owasp.org/index.php/XSS_%28Cross_Site_Scripting%29_Prevention_Cheat_Sheet