An Empirical Study of Vulnerability Rewards Programs

Matthew Finifter, Devdatta Akhawe, and David WagnerUniversity of California, Berkeley

\{ finifter, devdatta, daw\} @cs.berkeley.edu

AbstractWe perform an empirical study to better understand twowell-known vulnerability rewards programs, or VRPs,which software vendors use to encourage communityparticipation in finding and responsibly disclosing soft-ware vulnerabilities. The Chrome VRP has cost approx-imately $580,000 over 3 years and has resulted in 501bounties paid for the identification of security vulnerabili-ties. The Firefox VRP has cost approximately $570,000over the last 3 years and has yielded 190 bounties. 28%of Chrome’s patched vulnerabilities appearing in secu-rity advisories over this period, and 24% of Firefox’s,are the result of VRP contributions. Both programs ap-pear economically efficient, comparing favorably to thecost of hiring full-time security researchers. The ChromeVRP features low expected payouts accompanied by highpotential payouts, while the Firefox VRP features fixedpayouts. Finding vulnerabilities for VRPs typically doesnot yield a salary comparable to a full-time job; the com-mon case for recipients of rewards in either program isthat they have received only one reward. Firefox has farmore critical-severity vulnerabilities than Chrome, whichwe believe is attributable to an architectural differencebetween the two browsers.

1 IntroductionSome software vendors pay security researchers for the re-sponsible disclosure of a security vulnerability. Programsimplementing the rules for this exchange are known asvulnerability rewards programs (VRPs) or bug bountyprograms. The last couple of years have seen an upsurgeof interest in VRPs, with some vendors expanding theirexisting programs [1, 19], others introducing new pro-grams [3, 34, 38], and some companies offering to act asan intermediary between security researchers and vendorsoffering VRPs [53].

VRPs offer a number of potential attractions to softwarevendors. Offering adequate incentives entices security re-searchers to look for vulnerabilities, and this increasedattention improves the likelihood of finding latent vulner-abilities.1 Second, coordinating with security researchersallows vendors to more effectively manage vulnerabilitydisclosures, reducing the likelihood of unexpected and

1For instance, Linus’s Law suggests “Given enough eyeballs, allbugs are shallow.” [48]

costly zero-day disclosures. Monetary rewards provide anincentive for security researchers not to sell their researchresults to malicious actors in the underground economyor the gray world of vulnerability markets. Third, VRPsmay make it more difficult for black hats to find vulnera-bilities to exploit. Patching vulnerabilities found througha VRP increases the difficulty and therefore cost for mali-cious actors to find zero-days because the pool of latentvulnerabilities has been diminished. Additionally, expe-rience gained from VRPs (and exploit bounties [23, 28])can yield improvements to mitigation techniques and helpidentify other related vulnerabilities and sources of bugs.Finally, VRPs often engender goodwill amongst the com-munity of security researchers. Taken together, VRPsprovide an attractive tool for increasing product securityand protecting customers.

Despite their potential benefits, there is an active de-bate over the value and effectiveness of VRPs. A numberof vendors, notably Microsoft,2 Adobe, and Oracle, donot maintain a VRP, with Microsoft arguing that VRPsdo not represent the best return on investment on a per-bug basis [26]. Further, it is also not clear if the boun-ties awarded are a sufficient attraction for security re-searchers motivated by money—underground economyprices for vulnerabilities are far higher than those offeredby VRPs [20, 37].

Given the emergence of VRPs as a component of thesecure development lifecycle and the debate over the effi-cacy of such programs, we use available data to better un-derstand existing VRPs. We focus on the Google Chromeand Mozilla Firefox web browsers, both of which arewidely considered to have mature VRPs, as case studies.We analyze these VRPs along several dimensions withthe intention of better understanding the characteristics,metrics, and trajectory of a VRP.

We make the following contributions:

• We collect and analyze data on vulnerability rewardsover the last 3 years for the Google Chrome VRPand the Mozilla Firefox VRP (Section 3).

• We assess the state of these VRPs along severaldimensions, including costs, benefits, popularity, and

2On June 19, 2013, during final preparation of this manuscript,Microsoft announced a month-long VRP for the IE11 developer pre-view [54].

1

efficacy (Section 4), finding that these VRPs appearboth effective and cost-effective.

• We make concrete recommendations for softwarevendors aiming to start or evolve their own VRP(Section 5.2).

• We generate hypotheses, which identify opportuni-ties for future research on VRPs and secure softwaredevelopment.

2 BackgroundA secure software development lifecycle (SDLC) aims toaddress software security throughout the entire softwaredevelopment process, from before specifications are de-veloped to long after software has been released [15]. Avulnerability remediation strategy is any systematic ap-proach whose goal is to reduce the number of softwarevulnerabilities [57]. Vulnerability remediation strategiesare one important part of an SDLC, complemented bythings like incident response [32], operational securityconsiderations [46], and defense in depth [16].

Potential vulnerability remediation strategies include:

• Code reviews. These can range from informal, as-needed requests for code review to systematized, for-mal processes for mandatory code inspection. Typi-cally, SDLCs also include an extra security reviewfor security critical features.

• Penetration testing. Software vendors may performin-house penetration testing or may hire externalcompanies who specialize in this service. Penetra-tion testing ranges from manual to automated.

• Use of dynamic and static analysis tools. Special-ized tools exist for catching a wide range of flaws,e.g., memory safety vulnerabilities, configurationerrors, and concurrency bugs.

• Vulnerability rewards programs. The focus ofour study, VRPs have recently received increasedattention from the security community.

How such strategies are systematized and realizedvaries widely between software vendors. One companymight require mandatory code reviews before code check-in, while another might hire outside penetration testingexperts a month before product release. Vendors oftencombine or innovate on existing strategies.

Vulnerability rewards programs (VRPs) appear to beemerging as a viable vulnerability remediation strategy.Many companies have them, and their popularity contin-ues to grow [6,9]. But VRPs have gone largely unstudied.For a company considering the use of a VRP in theirSDLC, guidance is limited.

By studying mature, high-profile VRPs, we aim toprovide guidance on the development of new VRPs andthe evolution and maturation of existing VRPs. Vendorslooking to grow their VRPs can benefit from an improvedunderstanding of those VRPs we study.

Toward this end, we measure, characterize, and dis-cuss the Google Chrome and Mozilla Firefox VRPs. Wechoose these VRPs in particular because browsers are apopular target for malicious actors today. Their ubiqui-tous nature and their massive, complex codebase withsignificant legacy components make them especially vul-nerable. Complex, high-performance components with alarge attack surface such as JavaScript JITs also providean alluring target for malicious actors. For the same rea-sons, they are also widely studied by security researchers;they therefore provide a large sample size for our study.In addition, browser vendors were among the first to of-fer rewards for vulnerabilities: Mozilla’s VRP started in2004 and Google introduced the Chrome VRP in 2010,before the security community at large adopted VRPs asa vulnerability remediation strategy.

2.1 Goals

We intend to improve our understanding of the followingcharacteristics of a mature VRP: (1) Expected cost, (2)expected benefits, (3) incentive levels effective for encour-aging and sustaining community participation, and (4)volume of VRP activity (e.g., number of patches comingout of VRP reports).

We do so by studying available data coming out of twoexemplars of well-known, mature VRPs, that of GoogleChrome and Mozilla Firefox. Understanding these VRPswill allow these vendors to evaluate and improve theirprograms, and it will suggest targets for other vendorsto strive toward with their VRPs. At minimum, we hopeto arrive at a better understanding of the current stateof VRPs and how they have evolved. At best, we aimto make concrete suggestions for the development andimprovement of VRPs.

2.2 Google Chrome VRP

The Google Chrome VRP3 is widely considered an ex-emplar of a mature, successful VRP. When first intro-duced in January 2010, the Google Chrome VRP offeredresearchers rewards ranging from $500 for high- andcritical-severity bugs, with a special $1337 reward forparticularly critical or clever bugs. Over time, the typicalpayout increased to a $1000 minimum with a maximumpayout of $3133.7 for high-impact vulnerabilities. Addi-tionally, the Chrome team, has provided rewards of up to$31,336 for exceptional vulnerability reports [21].

3The program is officially the Chromium VRP with prizes sponsoredby Google. We refer to it as the Google Chrome VRP for ease ofexposition.

2

Google also sponsors a separate, semi-regular exploitbounty called the “pwnium” competition [23]. This pro-gram focuses on full exploits; a mere vulnerability is notenough. In return, it awards higher bounties (as high as$150,000) for these exploits [8]. Reliable exploits formodern browsers typically involve multiple vulnerabili-ties and significant engineering effort. For example, thetwo winning entries in a recent “pwnium” contest requiredsix and ten vulnerabilities in addition to “impressive” en-gineering in order to achieve a successful exploit [7, 45].Our focus is on programs that provide bounties for vul-nerabilities; we do not consider exploit bounties in thiswork.

The severity of a vulnerability plays a key role in decid-ing reward amounts. Google Chrome uses a clear guide-line for deciding severity [12]. In short, a critical vulnera-bility allows an attacker to run arbitrary native code on theuser’s machine; for instance, web-accessible memory cor-ruption vulnerabilities that appear in the Chrome kernel4

are typically critical severity. A high-severity vulnerabil-ity is one that allows an attacker to bypass the same-originpolicy, e.g., via a Universal XSS vulnerability (which en-ables an attacker to mount an XSS attack on any web site)or a memory corruption error in the sandbox. A vulner-ability is of medium severity if achieving a high/criticalstatus requires user interaction, or if the vulnerability onlyallows limited disclosure of information. Finally, a low-severity vulnerability refers to all the remaining securityvulnerabilities that do not give the attacker control overcritical browser features. Medium-severity vulnerabili-ties typically receive rewards of $500, and low-severityvulnerabilities typically do not receive rewards.

2.3 Mozilla Firefox VRP

Mozilla’s VRP is, to the best of our knowledge, one of theoldest VRPs in the industry. It was first started in 2004 andbased on a similar project at Netscape in 1995 [41]. TheMozilla VRP initially awarded researchers $500 for high-or critical-severity security bugs. Starting July 1, 2010Mozilla expanded its program to award all high/criticalvulnerabilities $3000 [1].

Mozilla’s security ratings are similar to that of Chrome.Critical vulnerabilities allow arbitrary code execution onthe user’s computer. Vulnerabilities that allow an attackerto bypass the same-origin policy or access confidentialinformation on the user’s computer are high severity.Due to the absence of privilege separation in the Fire-fox browser, all memory corruption vulnerabilities arecritical, regardless of the component affected. Mozillais currently investigating a privilege-separated design forFirefox [17, 36, 39].

Mozilla’s VRP also qualitatively differs from the

4Chrome follows a privilege-separated design [4]. The Chromekernel refers to the privileged component.

Google program. First, Mozilla awards bounties evenif the researcher publicly discusses the vulnerabilityinstead of reporting it to Mozilla.5 Second, Mozilla alsoexplicitly awards vulnerabilities discovered in “nightly”(or “trunk”) versions of Firefox. In contrast, Googlediscourages researchers from using “canary” builds andonly awards bounties in canary builds if internal testingwould miss those bugs [55].

3 MethodologyFor both browsers, we collect all bugs for which rewardswere issued through the browser vendor’s VRP. To evalu-ate the impact of the VRP as a component of the SDLC,we also collected all security bugs affecting stable releases.We chose to look only at bugs affecting stable releases toignore the impact of transient bugs and regressions caughtby internal testing.

For each bug in the above two datasets, we gatheredthe following details: (1) severity of the bug, (2) rewardamount, (3) reporter name, (4) report date. For bugsaffecting stable releases, we also aimed to gather the datea release patching the bug became available for download.As we discuss below, we were able to gather this data foronly a subset of all bugs.

For all bugs, we mark a bug as internally or externallyreported via a simple heuristic: if a reward was issued,the reporter was external, and otherwise the reporter wasinternal. Because low and medium severity vulnerabil-ities usually do not receive bounties, we only look atcritical/high vulnerabilities when comparing internal andexternal bug reports. While all high/critical vulnerabilitiesare eligible for an award, a researcher can still refuse anaward, in which case, our heuristic falsely marks the bug“internal.” We are aware of a handful of such instances,but there are not enough of these in our dataset to affectour analysis.

We are also aware of some researchers who transitionedfrom external to internal over the course of our study pe-riod. Because our heuristic operates on a per-bug basis(as opposed to marking each person as internal or exter-nal), the same person may be (intentionally) consideredinternal for one bug and external for another.

In this section, we present how we gathered this datasetfor Chrome and Firefox. We first discuss how we identifythe list of bugs affecting stable releases and bugs awardedbounties, followed by a discussion on how we identified,for each bug, other details such as severity. Finally, wediscuss threats to the validity of our measurement study.

3.1 Gathering the Google Chrome dataset

We gathered data from the official Chromium bugtracker [13] after confirming with Google employees that

5But Mozilla reports that this was a rare occurrence over the period oftime we consider, possibly because the VRP became widely known [56].

3

the bug tracker contained up-to-date, authoritative dataon rewards issued through their VRP. We search the bugtracker for bugs marked with the special “Reward” labelto collect bugs identified through the VRP. Next, wesearched the bug tracker for bugs marked with the special“Security-Impact: Stable” to collect bugs affecting stablereleases. Next, we remove the special Pwnium [23]rewards from all datasets because Pwnium rewardsexploits instead of vulnerabilities as in the regular VRP.This provides us with 501 bugs identified through theVRP and 1347 bugs affecting stable releases.

The Chromium Bug tracker provides a convenient in-terface to export detailed bug metadata, including severity,reporter, and report date, into a CSV file, which we useto appropriately populate our dataset. We identify thereward amounts using the “Reward” label.

Unfortunately, the Chromium bug tracker does not in-clude the release date of bug fixes. Instead, we gatherthis data from the Google Chromium release blog [27].For each stable release of the Chromium browser, Googlereleases a blog post listing security bugs fixed in a release.For the subset of bugs mentioned in these release notes,we extract the release date of the stable version of Chromethat patches the bug.

3.2 Gathering the Mozilla Firefox dataset

Similar to Google Chrome, we searched Bugzilla, theFirefox bug tracker, for an attachment used to tag a bugbounty.6 We identified 190 bugs via this search.

Unlike the Chrome bug tracker, Bugzilla does not pro-vide a convenient label to identify bugs affecting stablereleases. Instead, Mozilla releases Mozilla FoundationSecurity Advisories (MFSA) with every stable release ofMozilla Firefox [40]. We scraped these advisories fora list of bugs affecting stable releases. We also use theMFSAs to identify the release date of a patched, stableversion of Firefox. We gathered 613 unique bugs from theMFSA advisories dating back to March 22, 2010 (Firefox3.6).

Similar to the Chromium Bug tracker, the Bugzillawebsite provides a convenient interface to export detailedbug data into a CSV file for further analysis. We usedBugzilla to collect, for each bug above, the bug reporter,the severity rating, and the date reported. The securityseverity rating for a bug is part of the Bugzilla keywordsfield and not Bugzilla’s severity field. We do not sep-arately collect the amount paid because, as previouslydiscussed, Mozilla’s expanded bounty program awards$3,000 for all critical/high vulnerabilities.

6The existence of this attachment is not always visible to the public.We acknowledge the support of Mozilla contributor Dan Veditz for hishelp gathering this data.

Severity Chrome FirefoxStable Bounty Stable Bounty

Low 226 1 16 1Medium 288 72 66 9High 793 395 79 38Critical 32 20 393 142Unknown 8 13 59 0Total 1347 501 613 190

Table 1: Number of observations in our dataset.

3.3 Dataset

Table 1 presents information about the final dataset weused for our analysis. We have made our dataset availableonline for independent analysis [33].

3.4 Threats to validity

In this section, we document potential threats to validityso readers can better understand and take into account thesources of error and bias in our study.

It is possible that our datasets are incomplete, i.e., thereexist patched vulnerabilities that do not appear in ourdata. For example, for both Chrome and Firefox, we relyheavily on the keyword/label metadata to identify bugs;since this labeling is a manual process, it is possible thatwe are missing bugs. To gather the list of bugs affectingstable releases, we use the bug tracker for Chrome butuse security advisories for Mozilla, which could be in-complete. Given the large number of vulnerabilities wedo have in our datasets, we expect that a small number ofmissing observations would not materially influence theconclusions we draw.

We treat all rewards in the Firefox VRP as $3,000despite knowing that 8% of the rewards were for lessthan this amount [56]. Data on which rewards were forless money and what those amounts were is not publiclyavailable. Any results we present regarding amounts paidout for Firefox vulnerabilities may therefore have as muchas 8% error, though we expect a far lower error, if any. Wedo not believe this affects the conclusions of our analysis.

Parts of our analysis also compare Firefox and ChromeVRPs in terms of number of bugs found, which assumesthat finding security vulnerabilities in these browsers re-quires comparable skills and resources. It could be thecase that it is just easier to find bugs in one over theother, or one browser has a lower barrier to entry forvulnerability researchers. For example, the popular Ad-dress Sanitizer tool worked only on Google Chrome untilMozilla developers tweaked their build infrastructure toenable support for the same [31]. Another confound isthe possibility that researchers target a browser based onpersonal factors beyond VRPs. For example, researcherscould look for vulnerabilities only in the browser theypersonally use.

Assigning bug severity is a manual process. While

4

the severity assignment guidelines for both browsers aresimilar, it is possible that vendors diverge in their actualseverity assignment practices. As a result, the bug severi-ties could be incomparable across the two browsers.

We study only two VRPs; our results do not necessarilygeneralize to any other VRPs. We caution the reader toavoid generalizing to other VRPs, but instead take ourresults as case studies of two mature, well-known VRPs.

4 ResultsWe study VRPs from the perspectives of three interestedparties: the software vendor, the independent securityresearcher, and the security researcher employed full-timeby the software vendor.

4.1 Software vendor

We model the software vendor’s goal as follows: to in-crease product security as much as possible while spend-ing as little money as possible. There are many potentialstrategies for working toward this goal, but in this paperwe consider the strategy of launching a VRP. We presentdata on costs and benefits for two VRPs, and generatehypotheses from this data. The software vendor’s motiva-tion can also include publicity and engaging the securityresearch community. We do not measure the impact ofVRPs on these.

4.1.1 Number of vulnerabilities

The intended benefit of a VRP is to improve product se-curity. A reduction in the number of latent vulnerabilitiesis one way of improving product security. We find thatthe Chrome VRP uncovers about 2.6 times as many vul-nerabilities as that of Firefox (501 vs. 190), despite thefact that Chrome’s total number of security vulnerabilitiesin our dataset is only 2.2 times that of Firefox (Table 1).27.5% of bugs affecting Chrome releases originate fromVRP contributions (371 of 1347), and 24.1% of bugs af-fecting Firefox releases (148 of 613) result from VRPcontributions.

Discussion Both VRPs yield a significant fraction ofthe total number of security advisories, which is a clearbenefit. Chrome is seeing approximately 1.14 times thebenefit of Firefox by our metric of fraction of advisoriesresulting from VRP contributions. We only study bugs af-fecting stable releases in this metric and caution the readerfrom assuming that VRPs are competitive with internalresearchers. For both browsers, internal researchers findfar more bugs during the months of testing that precedea typical browser release. For example, from January toMay 2013, across all release channels, Google researchersfound 140 high or critical vulnerabilities in Chrome, whilethe Chrome VRP only found 40 vulnerabilities in the sametime period.

0.2

.4.6

.8Fr

actio

n vu

lns

in c

urre

nt p

lus

last

5 m

onth

s

7/09 1/10 7/10 1/11 7/11 1/12 7/12 1/13

Low MediumHigh Critical

(a) Chrome

0.2

.4.6

.8Fr

actio

n vu

lns

in c

urre

nt p

lus

last

5 m

onth

s

7/09 1/10 7/10 1/11 7/11 1/12 7/12 1/13

Low MediumHigh Critical

(b) Firefox

Figure 1: Moving average over the current plus 5 previousmonths of the percentage of vulnerabilities at each severity level(low is blue solid line, medium is red long-dashed line, high isgreen short-dashed line, and critical is yellow dash-dotted line).In this and subsequent line graphs, the data are aggregated bymonth to improve graph readability, and the x-axis representsthe open date of the bug.

4.1.2 Vulnerability severity

Another measure of improvement to product security ischange in vulnerability severity over time. It is a goodsign, for example, if the percentage of critical-severityvulnerabilities has decreased over time.

Table 1 lists the total number of vulnerabilities by sever-ity for Firefox and Chrome. Figure 1 plots the fraction ofvulnerabilities at each severity level over the current plus5 previous months.

Discussion Firefox has a much higher ratio of criti-cal vulnerabilities to high vulnerabilities than Chrome.We expect that many of Firefox’s critical vulnerabilitieswould instead be high severity if, like Chrome, it also hada privilege-separated architecture. The lack of such anarchitecture means that any memory corruption vulnera-

5

bility in Firefox is a critical vulnerability. We thereforehypothesize that:

Hypothesis 1 This architectural difference betweenChrome and Firefox—that the former is privilege-separated and the latter is not—is the most influentialfactor in causing such a large difference in vulnerabilityseverity classification.

The fraction of critical severity bugs has remained rela-tively constant for Chrome. We also notice the start of atrend in Chrome—the fraction of high severity vulnera-bilities is declining and the fraction of medium severityvulnerabilities is increasing.

Chrome’s privilege-separated architecture means that acritical vulnerability indicates malicious code executionin the privileged process. We see that there continue tobe new bugs resulting in code execution in the privilegedprocess. Further investigation into these bugs can helpunderstand how and why they continue to surface.

Low-severity vulnerabilities in Google Chrome makeup a significant fraction of all vulnerabilities reported.In contrast, the fraction of low- and medium-severityvulnerabilities in Firefox remains negligible.

Note that our dataset does not allow us to attributeany change in vulnerability severity over time to the useor success of a VRP. However, severity over time is ametric worth tracking for a software vendor because it canindicate trends in the overall efforts to improve productsecurity, of which a VRP may be one component.

4.1.3 Community engagement

One advantage of VRPs is engagement with the broadersecurity community. We studied this engagement alongtwo axes: (1) the contribution of internal and externalresearchers towards identifying security vulnerabilities,and (2) the number of external participants in each VRP.

Figure 2 depicts the cumulative number of high- andcritical-severity vulnerabilities patched and Figure 3 de-picts the same, but for only critical vulnerabilities. Table 2shows the distribution of the total number of vulnerabil-ities reported by each external participant in each of thetwo VRPs. Although a few external participants submitmany bugs, there is a clear long tail of participants inboth VRPs. Table 3 shows the same distribution, but forinternal (i.e., employee) reporters of vulnerabilities.

Discussion For both browsers, internal contributionsfor high- and critical-severity vulnerabilities have consis-tently yielded the majority of patches. The number ofexternal contributions to Chrome has nearly caught upwith the number of internal contributions (i.e., around4/11 and 3/12, in Figure 2a) at various times, and as ofthe end of our study, these two quantities are comparable.Considering only critical-severity vulnerabilities, externalcontributions have exceeded internal contributions as of

0200

400

600

800

1000

Num

ber

hig

h+

critical vuln

s o

pened (

cum

ula

tive)

1/09 7/09 1/10 7/10 1/11 7/11 1/12 7/12 1/13

(a) Chrome

0100

200

300

400

500

Num

ber

hig

h+

critical vuln

s o

pened (

cum

ula

tive)

1/07 1/08 1/09 1/10 1/11 1/12 1/13

(b) Firefox

Figure 2: Number of high- plus critical-severity vulnerabilitiesreported over time, discovered internally (blue long-dashed line),externally (red short-dashed line), and total (green solid line).

the end of our study. For Firefox, on the other hand, thenumber of external contributions has consistently beenfar lower than the number of internal contributions.

We observe an increase in the rate of external contri-butions to Chrome starting around July 2010, six monthsafter the inception of the VRP. As seen in Figure 3a,this is more pronounced when considering only critical-severity vulnerabilities. We conjecture that this changecorresponds to increased publicity for the Chrome VRPafter Google increased reward amounts [19].

Linus’s Law, defined by Eric Raymond as “Givenenough eyes, all bugs are shallow,” suggests that it isin the interests of the software vendor to encourage morepeople to participate in the search for bugs. The distribu-tions of bugs found by external participants indicate thatboth VRPs have been successful in encouraging broadcommunity participation. The existence of a long tailof contributors holds for internal contributors as well asexternal contributors.

6

010

20

30

40

Num

ber

critical vuln

s o

pened (

cum

ula

tive)

7/09 1/10 7/10 1/11 7/11 1/12 7/12 1/13

(a) Chrome

0100

200

300

400

Num

ber

critical vuln

s o

pened (

cum

ula

tive)

1/07 1/08 1/09 1/10 1/11 1/12 1/13

(b) Firefox

Figure 3: Number of critical-severity vulnerabilities reportedover time, discovered internally (blue long-dashed line), exter-nally (red short-dashed line), and total (green solid line).

4.1.4 Diversity

There is the potential benefit that the wide variety ofexternal participants may find different types of vulner-abilities than internal members of the security team. Afew pieces of anecdotal evidence support this. Chromehas awarded bounty amounts that include $1,337, $2,337,$3,133.7, and $7,331 for bugs that they considered cleveror novel [21], and our dataset contains 31 such awards.Additionally, one of PinkiePie’s Pwnium exploits led toa full review of the Chrome kernel file API, which re-sulted in the discovery of several additional vulnerabili-ties [21, 51]. The Chrome security team missed all theseissues until PinkiePie discovered and exploited one suchissue [14]. We therefore hypothesize that:

Hypothesis 2 An increase in the number of researcherslooking for vulnerabilities yields an increase in the diver-sity of vulnerabilities discovered.

# Bugs Freq.

1 453 24 16 1

10 112 213 316 117 122 124 127 135 148 192 1

Total 63

(a) Chrome

# Bugs Freq.

1 462 93 45 16 19 1

10 112 114 147 1

Total 66

(b) Firefox

Table 2: Frequency distribution of number of high- or critical-severity vulnerabilities found by external contributors.

# Bugs Freq.

1 672 103 104 25 2

14 120 267 1

263 1Total 96

(a) Chrome

# Bugs Freq.

1 432 103 74 35 26 27 1

12 113 115 117 218 121 123 144 1

Total 77

(b) Firefox

Table 3: Frequency distribution of number of high- or critical-severity bugs found by internal contributors.

4.1.5 Cost of rewards

Though the number of bounties suggests that VRPs pro-vide a number of benefits, a thorough analysis necessarilyincludes an analysis of the costs of these programs. Inthis section, we examine whether VRPs provide a cost-effective mechanism for software vendors. We analyzeone ongoing cost of the VRP: the amount of money paidto researchers as rewards for responsible disclosure. Run-ning a VRP has additional overhead costs that our datasetdoes not provide any insight into.

Figure 4 displays the total cost of paying out rewardsfor vulnerabilities affecting stable releases. We find thatover the course of three years, the costs for Chrome andFirefox are similar: approximately $400,000.

7

01

23

45

Cum

ula

tive r

ew

ard

s p

aid

($100k)

7/09 1/10 7/10 1/11 7/11 1/12 7/12 1/13

Figure 4: Cumulative rewards paid out for Chrome (blue solidline) and Firefox (red dashed line), excluding rewards for vul-nerabilities not affecting stable versions.

Rewards for Development Releases Both Firefox andChrome issue rewards for vulnerabilities that do not af-fect stable release versions, increasing the total cost ofthe VRP beyond the cost of rewarding vulnerabilities af-fecting stable releases. One potential drawback of suchrewards is that the VRPs awards transient bugs that maynever make their way into a user-facing build in the firstplace. On the other hand, such rewards could catch bugsearlier in the development cycle, reducing the likelihoodof expensive out-of-cycle releases.

Figure 5 shows the cumulative rewards issued by eachof the two VRPs for vulnerabilities affecting stable re-leases, vulnerabilities not affecting stable releases, andthe sum of the two. We observe that the Chrome VRP haspaid out $186,839, 32% of its total cost of $579,605 overthe study period for vulnerabilities not affecting a stablerelease. The Firefox VRP has paid out $126,000, 22% ofits total cost of $570,000, over the study period for suchvulnerabilities.

Discussion The total cost of each of the two VRPs isremarkably similar. Both spend a significant fractionof the total cost on vulnerabilities not present in stablerelease versions.

4.1.6 Average daily cost

Figure 6 plots the average daily cost to date of each VRPover time. We see that Chrome’s VRP has cost $485 perday on average, and that of Firefox has cost $658 per day.

Discussion If we consider that an average North Amer-ican developer on a browser security team (i.e., that ofChrome or Firefox) would cost the vendor around $500per day (assuming a $100,000 salary with a 50% over-head), we see that the cost of either of these VRPs iscomparable to the cost of just one member of the browsersecurity team. On the other hand, the benefit of a VRP

01

23

45

6C

um

ula

tive r

ew

ard

s p

aid

($100k)

7/09 1/10 7/10 1/11 7/11 1/12 7/12 1/13

(a) Chrome

01

23

45

6C

um

ula

tive r

ew

ard

s p

aid

($100k)

7/10 1/11 7/11 1/12 7/12 1/13

(b) Firefox

Figure 5: Cumulative rewards paid out for vulnerabilities affect-ing a stable release (blue long-dashed line), vulnerabilities notaffecting any stable release (red short-dashed line), and the sumof the two (green solid line).

far outweighs that of a single security researcher becauseeach of these VRPs finds many more vulnerabilities thanany one researcher is likely to be able to find. For bugsaffecting stable releases, the Chrome VRP has paid 371bounties, and the most prolific internal security researcherhas found 263 vulnerabilities. For Firefox, these num-bers are 148 and 48, respectively. Based on this simplecost/benefit analysis, we hypothesize that:

Hypothesis 3 A VRP can be a cost-effective mechanismfor finding security vulnerabilities.

4.2 External security researcher

We model the goal of an external security researcher asfollows: to make as much money as possible in as lit-tle time as possible.7 The researcher can contribute to

7Naturally, this does not reflect the reality of every security re-searcher’s goal.

8

0100

200

300

400

500

Avera

ge d

aily

cost to

date

($/d

ay)

7/09 1/10 7/10 1/11 7/11 1/12 7/12 1/13

(a) Chrome

300

400

500

600

700

Avera

ge d

aily

cost to

date

($/d

ay)

7/10 1/11 7/11 1/12 7/12 1/13

(b) Firefox

Figure 6: Average daily cost to date since first reward.

any VRP he chooses, each of which pays out accordingto some rewards distribution. The researcher has someperception of security of each product, which reflects theexpected amount of time the researcher will have to spendto find a vulnerability.

A rational strategy for the security researcher is to lookfor products perceived to be insecure that also happen topay large bounties. This implies that a product with ahigher perceived security must pay relatively higher boun-ties to encourage researchers to look for vulnerabilities init as opposed to in a different product that is perceived tobe less secure. We therefore hypothesize that:

Hypothesis 4 In an efficient market with many VRPsand fluid reward structures, larger rewards reflect ahigher level of perceived security by the population ofresearchers who contribute to VRPs.

4.2.1 Reward amounts

Our dataset provides insight into the distributions of re-wards for two products. Firefox offers a standard rewardof $3,000 for all vulnerabilities. In contrast, the Chrome

Amount ($) Frequency (%)

500 24.751,000 60.081,337 3.591,500 2.992,000 2.992,337 0.602,500 0.603,000 0.203,133 1.803,500 0.204,000 0.204,500 0.205,000 0.207,331 0.20

10,000 1.40

Table 4: Percentage of rewards given for each dollar amount inChrome VRP.

VRP’s incentive structure provides different reward levelsbased on a number of subjective factors like difficulty ofexploit, presence of a test case, novelty, and impact, all ofwhich is at the discretion of Google developers.

Table 4 depicts the reward amounts paid to externalresearchers by the Chrome VRP. The majority of therewards are for only $500 or $1,000. Large rewards, suchas $10,000 rewards, are infrequent.

Discussion We hypothesize that high maximum rewardsentice researchers to participate, but low ($500 or $1,000)rewards are typical, and the total cost remains low. Themedian (mean) payout for Chrome bug bounty is $1,000($1,156.9), suggesting that a successful VRP can be in-expensive with a low expected individual payout. Muchlike the lottery, a large maximum payout ($30,000 forChrome), despite a small expected return (or even nega-tive, as is the case of anyone who searches for bugs butnever successfully finds any) appears to suffice in attract-ing enough participants. Bhattacharyya and Garrett [5]explain this phenomenon as follows:

Lotteries are instruments with negative ex-pected returns. So when people buy lotterytickets, they are trading off negative expectedreturns for skewness. Thus, if a lottery gamehas a larger prize amount, then a buyer will bewilling to accept a lower chance of winningthat prize.

4.2.2 VRPs as employment

Our dataset also allows limited insight into how muchmoney independent security researchers make. Table 5adisplays the total amounts of money earned by each exter-nal contributor to the Chrome VRP. Only three externalcontributors (out of eighty two) have earned over $80,000over the lifetime of the VRP, and an additional five haveearned over $20,000.

9

$ earned Freq.

500 261,000 251,337 61,500 22,000 13,000 23,133 13,500 24,000 15,000 17,500 1

11,000 111,500 111,837 115,000 117,133 118,337 120,633 124,133 128,500 128,633 137,470 180,679 185,992 1

105,103 1Total 82

(a) Chrome

$ earned Freq.

3,000 466,000 129,000 4

12,000 115,000 121,000 127,000 130,000 136,000 142,000 1

141,000 1Total 70

(b) Firefox

Table 5: Frequency distributions of total amounts earned byexternal VRP contributors.

In contrast to Google Chrome, we see in Table 5b that asingle Firefox contributor has earned $141,000 ($47,000per year) since the beginning of our study period. Tenof this individual’s rewards, representing $30,000, werefor vulnerabilities that did not impact a stable release.Six contributors have earned more than $20,000 via theMozilla VRP.

Discussion Based on the data from 2 VRPs, we hypoth-esize that:

Hypothesis 5 Contributing to a single VRP is, in gen-eral, not a viable full-time job, though contributing tomultiple VRPs may be, especially for unusually successfulvulnerability researchers.

4.2.3 Repeat contributors

Figure 7 shows the cumulative number of vulnerabilitiespatched due to reports from first-time VRP participantsand repeat participants. For both programs, first-timeparticipant rewards are steadily increasing, and repeatparticipant rewards are increasing even more quickly.

Discussion Both VRP incentive structures are evidentlysufficient for both attracting new participants and contin-uing to entice existing participants, though we do notedifferences between Chrome and Firefox. Until recently,repeat participants in Firefox’s VRP represented a rel-atively small fraction of the number of awards issued.

0100

200

300

400

500

Num

ber

vuln

era

bili

ties (

cum

ula

tive)

7/09 1/10 7/10 1/11 7/11 1/12 7/12 1/13

(a) Chrome

050

100

150

200

Num

ber

vuln

era

bili

ties (

cum

ula

tive)

7/10 1/11 7/11 1/12 7/12 1/13

(b) Firefox

Figure 7: Cumulative number of vulnerabilities rewarded, asreported by (1) first-time VRP contributors (blue short-dashedline), (2) repeat contributors (red long-dashed line), and (3) allcontributors (green solid line).

Chrome, on the other hand, has seen the majority of itsreports come from repeat participants for nearly the wholelifetime of its VRP.

4.3 Internal security researcher

An internal security researcher is a full-time employee ofa software vendor who is paid a salary to find as manyvulnerabilities as possible. Google hired at least three re-searchers who first came to light via the Chrome VRP [21]and Mozilla hired at least three researchers as well [56].

Discussion A software vendor may choose to hire anunusually successful independent security researcher. Theresearcher’s past performance indicates how many vul-nerabilities the vendor can expect them to find, and thevendor may prefer to pay a fixed salary instead of a per-vulnerability reward. The researcher may also prefer this;the researcher trades a potentially higher amount of cashfor less compensation, but more benefits and job security.

10

Median, Std. dev. Median, Std. dev.,Severity Chrome Chrome Firefox Firefox

Low 58.5 110.6 114 256.1Medium 45.5 78.9 106 157.6

High 28.0 35.3 62.5 85.7Critical 20.0 26.6 76 116.5

Table 6: Median and standard deviation of number of daysbetween vulnerability report and release that patches the vulner-ability, for each severity level.

Accordingly, we hypothesize that:

Hypothesis 6 Successful independent security re-searchers bubble to the top, where a full-time job awaitsthem.

4.4 Other factors

Our dataset provides an additional opportunity to betterunderstand the state of the SDLC (software developmentlifecycle) at Mozilla and Google. In particular, we ana-lyze (1) the elapsed time between a vulnerability reportand the release of a patched browser version that fixes thevulnerability, and (2) how often vulnerabilities are inde-pendently discovered, and what the potential implicationsare of this rediscovery rate.

4.4.1 Time to patch

As previously discussed, we choose to study time to re-lease a patched version, not time to commit a patch. Al-though relying on time to release a patch means we an-alyze only a subset of the data (Section 3), we believethe time to release a patched version of the browser is themore useful metric for end users. Mozilla Firefox andGoogle Chrome both follow a rapid-release cycle, with anew release every 6 or 7 weeks [11, 25]. In some cases,browser vendors release an out-of-band (or “chemspill”)release for vulnerabilities with active exploits in the wild.Such out-of-band releases are one of the most expensiveincidents for software companies, with costs running intomillions of dollars [30]. Our metric awards the engineer-ing and management commitment required in choosingto release such versions.

Figure 8 depicts the time between initial report of avulnerability and the stable release that patches it. Table 6gives summary statistics for these distributions.

Figure 9 is a scatter plot of the same data, which al-lows us to see changes in time to patch over time. Fig-ure 10 shows the change in standard deviation of time topatch over time. More specifically, for a given releasedate, the y-value is the standard deviation for all bugspatched in that release or up to five prior releases. Thesegraphs indicate that the standard deviation in time to patchcritical vulnerabilities has slowly dropped for Firefox,while Chrome’s time to patch critical vulnerabilities hasremained relatively constant over time.

0100

200

300

400

500

Da

ys b

etw

ee

n in

itia

l re

po

rt a

nd

sta

ble

re

lea

se

with

pa

tch

Low Medium High Critical

(a) Chrome

0500

1,0

00

Days b

etw

een initia

l re

port

and s

table

rele

ase w

ith p

atc

h

Low Medium High Critical

(b) Firefox

Figure 8: Box and whisker plots depicting the distributions oftime between vulnerability report and release that patches thevulnerability, for each severity level.

Discussion For Chrome, both the median time to patchand the variance are lower for higher severity vulnerabili-ties. This is an important parameter for a VRP becauseresponsible disclosure depends critically on vendor re-sponse time [22, 50]. If a vendor does not patch in a rea-sonable time frame, security researchers are less likely toexercise responsible disclosure. Accordingly, this may bea contributing factor in Firefox’s lower degree of commu-nity participation (as compared to Chrome), given that thetime to patch critical vulnerabilities in Firefox is longerand has very high variance.

In Chrome, the time to patch is faster for critical vulner-abilities than it is for high severity vulnerabilities. Thistrend continues for medium- and low-severity vulnera-bilities as well. This indicates correct prioritization ofhigher-severity vulnerabilities by Chrome security en-gineers. The same cannot be said for Firefox; high andcritical severity vulnerabilities tend to take about the sameamount of time to fix.

11

010

020

030

040

050

00

100

200

300

400

500

7-09 7-10 7-11 7-12 7-09 7-10 7-11 7-12

Low Medium

High Critical

Tim

e to

pat

ch (d

ays)

Stable release date

(a) Chrome

0500

1000

0500

1000

1−10 1−11 1−12 1−131−10 1−11 1−12 1−13

Low Medium

High Critical

Tim

e to p

atc

h (

days)

Stable release date

(b) Firefox

Figure 9: Scatter plots depicting the time between vulnerabilityreport and release that patches the vulnerability vs. time, foreach severity level.

The high variance in Firefox’s time to patch criticalvulnerabilities may be partly attributable to the lack ofprivilege separation in Firefox, since a larger TCB forcritical vulnerabilities means that there is a larger pool ofengineers owning code that might hold a critical vulnera-bility. However, it is an encouraging sign that Firefox hasgradually reduced this variance. Nonetheless, the variancein patch times and typical time to patch for Firefox bothremain far higher than we see for Chrome, suggesting theneed for a concerted effort at reducing this.

4.4.2 Independent discovery

Using the Chromium release blog, we manually codedan additional variable independent. This variablerepresents the number of times a vulnerability was inde-pendently discovered. We coded it using the text of thecredit variable, which mentions “independent discov-ery” of a vulnerability in the case of multiple independentdiscoveries.

050

100

150

200

Std

. dev. of tim

e to p

atc

h

7−10 1−11 7−11 1−12 7−12 1−13

(a) Critical-severity vulnerabilities.

050

100

150

Std

. dev. of tim

e to p

atc

h

7−09 1−10 7−10 1−11 7−11 1−12 7−12 1−13

(b) Critical and high severity vulnerabilities.

Figure 10: Standard deviation of time to patch over time. Fora given release date, the y-value is the standard deviation of allbugs patched in that release or up to five prior releases. The redsolid line is for Firefox, and the blue dashed line is for Chrome.

Our Chrome dataset indicates when a vulnerability wasindependently discovered by multiple parties, identifiesthe parties, and in some cases, gives an upper boundon the time between discovery and rediscovery. Of the668 vulnerabilities in our Chrome VRP dataset, fifteen(2.25\%) of them had at least two independent discoveries,and two of these had three independent discoveries. Thisis a lower bound on the number of independent discover-ies of these vulnerabilities, since it represents only thoseknown to the vendor.

Figure 11 displays the independent rediscovery ratesfor individuals. Each dot represents an individual contrib-utor in our dataset. Its x-value gives the number of vul-nerabilities discovered by this individual, and its y-valuegives the number of these vulnerabilities independentlyrediscovered by another contributor in our dataset. Ofthose individuals who reported five or more vulnerabili-ties, the highest rediscovery rate is 25\% and the mean is

12

01

23

45

Nu

mb

er

of vu

lne

rab

ilitie

s in

de

pe

nd

en

tly r

ep

ort

ed

by s

om

eo

ne

els

e

0 20 40 60 80 100Number of vulnerabilities reported

Figure 11: Independent vulnerability discoveries within theChrome VRP dataset. Each dot represents an individual con-tributor in our dataset. Its x-value gives the number of vulnera-bilities contributed by this individual, and its y-value gives thenumber of these contributions that were independently discov-ered by another contributor in our dataset.

4.6\%.Our Firefox dataset does not indicate independent re-

discovery, but we have limited data from personal com-munication with a Firefox security engineer [56]. Heindicated that there had been at least 4–7 vulnerabilitiesreported through the VRP for which there had been twoindependent discoveries, a rate of 2.7% to 4.7%, which isconsistent with what we see in our Chrome dataset.

Discussion Independent rediscovery rates can have im-plications for estimating the number of latent bugs insoftware [29] as well as understanding the expected decayrate of a stash of zero-day vulnerabilities.

A zero-day loses its value when the vendor becomesaware of it, which happens via independent discovery ofthe vulnerability. Thus, a stash of zero-days will decay atsome rate. From the limited data available to us via ourstudy, we hypothesize that:

Hypothesis 7 The decay rate of a stash of zero-day vul-nerabilities is low enough to be inconsequential as a resultof relatively low empirical independent rediscovery rates.

We encourage future studies that aim to confirm or refutethis hypothesis using larger, more appropriate datasets.

5 Discussion and recommendationsIn this section, we synthesize what we have learned andpresent concrete recommendations for software vendorsbased on our data analysis.

5.1 Mozilla Firefox vs. Google Chrome

Despite costing approximately the same as the Mozillaprogram, the Chrome VRP has identified more than three

times as many bugs, is more popular and shows simi-lar participation from repeat and first-time participants.There is a stark difference between the levels of externalparticipation in the two VRPs (Figure 2).

Despite having the oldest bounty program, externalcontributions lag far behind internal contributions to Fire-fox’s security advisories. In contrast, external contribu-tions to Chrome’s security advisories closely rival internalcontributions. Investigating further, we find three key dif-ferences between the two programs:

Tiered structure with large special rewards Mozilla’sprogram has a fixed payout of $3,000, which is approxi-mately equal to the normal maximum payout for Chrome($3,1337). Nonetheless, Chrome’s tiered structure, witheven higher payouts (e.g., $10,000) possible for cleverbugs and special cases appears to be far more effectivein encouraging participation. This makes sense with anunderstanding of incentives in lotteries: the larger the po-tential prize amount, the more willing participants are toaccept a lower expected return, which, for VRPs, meansthe program can expect more participants [5].

Time to patch We see a far higher variance in thetime-to-release-patch metric for critical vulnerabilities inMozilla Firefox. It is generally accepted that the viabilityof responsible disclosure depends on a reasonable vendorresponse time [50]. Thus, the high variance in Mozilla’sresponse time could affect responsible disclosure throughthe VRP.

Higher profile Chrome’s VRP has a higher profile, withannual competitions like Pwnium providing particularlyhigh rewards (up to $150,000). Chrome authors alsoprovide extra reward top-ups for “interesting” bugs. Webelieve this sort of “gamification” leads to a higher profilefor the Chrome VRP, which may help encourage partici-pation, particularly from researchers interested in widerrecognition.

Our methodology does not provide insight into the mo-tivations of security researchers and the impact of VRPdesigns on the same—a topic we leave for future work.Nevertheless, we hypothesize that these three factors com-bined explain the disparity in participation between theFirefox and Chrome VRPs. Accordingly, we recommendMozilla change their reward structure to a tiered systemlike that of Chrome. We urge Mozilla to do whatever ittakes to continue to reduce the variance in time to releasea patch for critical vulnerabilities, though we also realizethe difficulty involved in doing so. Ongoing attempts atprivilege separation might enable reducing the variance intime to patch critical vulnerabilities [17, 36, 39]. Mozillacan also consider holding its own annual competitions orotherwise increasing the PR surrounding its VRP.

13

5.2 Recommendations for vendors

Our study of the Chrome and Firefox VRPs yield a num-ber of observations that we believe can guide vendorsinterested in launching or evolving their own VRPs.

We find that VRPs appear to provide an economicallyefficient mechanism for finding vulnerabilities, with a rea-sonable cost/benefit trade-off (Sections 4.1.1 and 4.1.6).In particular, they appear to be 2-100 times more cost-effective than hiring expert security researchers to findvulnerabilities. We therefore recommend that more ven-dors consider using them to their (and their users’) advan-tage. The cost/benefit trade-off may vary for other typesof (i.e., non-browser) software vendors; in particular, theless costly a security incident is for a vendor, the lessuseful we can expect a VRP to be. Additionally, we ex-pect that the higher-profile the software project is (amongdevelopers and security researchers), the more effective aVRP will be.

Response time, especially for critical vulnerabilities,is important (Section 4.4.1). High variance in time-to-patch is not appreciated by the security community. It canreasonably be expected to reduce participation because itmakes responsible disclosure through the VRP a less at-tractive option than the other options available to securityresearchers.

VRP incentive design is important and should be care-fully considered. Chrome’s tiered incentive structure ap-pears more effective at encouraging community participa-tion than Firefox’s fixed-amount incentive structure (Sec-tion 4.2.1). Additionally, both Chrome and Firefox haveincreased their rewards over time. Doing so increasespublicity, entices participants, and signals that a vendoris betting that their product has become more secure overtime.

Our analysis demonstrates the impact of privilege sep-aration on the Chrome VRP (Section 4.1.2). Privilegeseparation also provides flexibility to the Chrome team.For example, a simple way for Chrome to cut costs whilestill increasing participation could be to reduce rewardamounts for high-severity vulnerabilities and increase re-ward amounts for critical-severity vulnerabilities. Mozilladoes not have this flexibility. Vendors should considerusing their security architecture to their advantage.

6 Related WorkMein and Evans share our motivation and presentGoogle’s experience with its vulnerability rewardsprograms [35]. In contrast, our focus is on understandingand comparing two popular VRPs run by competingbrowser vendors. We also perform a number of analysesnot performed by the previous work as well as makeour data available for other researchers. We alsoindependently confirm that, for both Google and Mozilla,VRPs are cost-effective mechanisms for finding security

vulnerabilities.

Development lifecycle datasets Many authors havelooked to large datasets, including code repositories, bugtrackers, and vulnerability databases, to gather and ana-lyze data in an effort to better understand some aspect ofthe development lifecycle. Rescorla gathered data fromNIST’s ICAT database (which has since been updated andrenamed to NVD [44]) to analyze whether vulnerabilityrates tend to decrease over time [49]. He found no evi-dence that it is in fact worthwhile for software vendorsto attempt to find vulnerabilities in their own softwarebecause there is no evidence that such efforts are reducingvulnerability rates.

Ozment and Schechter used the OpenBSD CVS reposi-tory to ask and answer similar questions as Rescorla [47].They find that the rate of discovery of what they callfoundational vulnerabilities—those present since the be-ginning of the study period—had decreased over the studyperiod.

Neuhaus and Plattner use vulnerability reports forMozilla, Apache httpd, and Apache Tomcat to evalu-ate whether vulnerability fix rates have changed overtime [42]. They conclude that the supply of vulnerabili-ties is not declining, and therefore that attackers and/orvulnerability researchers have not hit diminishing returnsin looking for vulnerabilities.

Neuhaus et al. use a dataset of Firefox security advi-sories in combination with the Firefox codebase to mapvulnerabilities to software components and predict whichcomponents are likely to contain vulnerabilities [43].

Scholte et al. use the NVD to evaluate how cross-sitescripting and SQL injection vulnerabilities have evolvedover time [52]. They find that the complexity of such vul-nerabilities does not appear to have changed over time andthat many foundational cross-site scripting vulnerabilitiesare still being discovered.

Evaluating vulnerability-finding techniques Otherwork has focused specifically on evaluating the manyavailable techniques for finding vulnerabilities, thoughwe are unaware of any previous work that has consideredpublic-facing VRPs as one such technique.

Austin and Williams evaluated four different tech-niques for vulnerability discovery on two health recordsystems: “systematic and exploratory manual penetrationtesting, static analysis, and automated penetration test-ing” [2], finding that very few vulnerabilities are in factfound by multiple techniques and that automated penetra-tion testing is the most effective in terms of vulnerabilitiesfound per hour.

Finifter and Wagner compared manual source codeanalysis to automated penetration testing on a web appli-cation, with similar findings: the techniques are comple-mentary, and manual analysis found more vulnerabilities,

14

but took much more time than automated penetration test-ing [24].

Edmundson et al. found that different reviewers tendto find different vulnerabilities and, even in a small code-base, it takes many reviewers to spot all or even a sig-nificant fraction of the vulnerabilities present [18]. Thisis consistent with our findings about the effectiveness ofcrowdsourced VRPs.

A large body of work investigates defect predictionusing empirical techniques; we refer the reader to a surveyby Catal et al. [10].

7 Conclusion and future workWe examined the characteristics of well-known vulner-ability rewards programs (VRPs) by studying two suchVRPs. Both programs appear economically efficient, com-paring favorably to the cost of hiring full-time securityresearchers. The Chrome VRP features low expected pay-outs accompanied by high potential payouts, a strategythat appears to be effective in engaging a broad commu-nity of vulnerability researchers.

We hope that our study of these two VRPs serves as avaluable reference for software vendors aiming to evolvean existing VRP or start a new one. Potential future workon understanding VRPs includes economic modeling ofVRPs; identifying typical patterns, trajectories, or phasesin a VRP; and studying failed or unsuccessful VRPs toget a better sense of possible pitfalls in VRP development.Gathering and analyzing data from more VRPs will surelypaint a more complete picture of their potential costs andbenefits.

AcknowledgmentsWe are particularly grateful to Chris Evans and Dan Veditzfor their help, encouragement, and feedback throughoutthe research. We also thank Chris Hofmann, Parisa Tabriz,Vern Paxson, Adrienne Felt, the anonymous reviewers,and our shepherd, Sam King, for their feedback on draftsof the paper.

This work was supported by Intel through the ISTC forSecure Computing; by the National Science Foundationunder a Graduate Research Fellowship and grant numbersCCF-0424422, 0842695, and 0831501-CT-L; by the AirForce Office of Scientific Research under MURI awardsFA9550-08-1-0352, FA9550-09-1-0539, and FA9550-12-1-0040; and by the Office of Naval Research under MURIgrant no. N000140911081. Any opinions, findings, andconclusions or recommendations expressed in this ma-terial are those of the author(s) and do not necessarilyreflect the views of the NSF, the AFOSR, the ONR, orIntel.

References[1] ADAMSKI, L. Refresh of the Mozilla Security Bug Bounty

Program, July 2010. https://blog.mozilla.org/

security/2010/07/15/refresh/.

[2] AUSTIN, A., AND WILLIAMS, L. One technique is not enough:A comparison of vulnerability discovery techniques. In EmpiricalSoftware Engineering and Measurement (ESEM), 2011 Interna-tional Symposium on (2011), IEEE, pp. 97–106.

[3] BARRETT, M. PayPal “Bug Bounty” Program for SecurityResearchers, June 2012. https://www.thepaypalblog.com/2012/06/paypal-bug-bounty-program/.

[4] BARTH, A., JACKSON, C., REIS, C., AND TEAM, T. G. C.The Security Architecture of the Chromium Browser. Tech. rep.,Stanford University, 2008.

[5] BHATTACHARYYA, N., AND GARRETT, T. A. Why PeopleChoose Negative Expected Return Assets - An Empirical Examina-tion of a Utility Theoretic Explanation. Federal Reserve Bank of St.Louis Working Paper Series (March 2006). http://research.stlouisfed.org/wp/2006/2006-014.pdf.

[6] BLINDU, E. Vulnerabilities reward programs, July 2012.http://www.testalways.com/2012/07/13/vulnerabilities-reward-programs/.

[7] BUCHANAN, K., EVANS, C., REIS, C., AND SEPEZ, T. A Taleof Two Pwnies (Part 2), June 2012. http://blog.chromium.org/2012/06/tale-of-two-pwnies-part-2.html.

[8] BUCHANAN, K., EVANS, C., REIS, C., AND SEPEZ, T. Showoff Your Security Skills: Pwn2Own and Pwnium 3, January 2013.http://blog.chromium.org/2013/01/show-off-your-security-skills-pwn2own.html.

[9] CARETTONI, L. “No More Free Bugs” Initiative, Octo-ber 2011. http://blog.nibblesec.org/2011/10/no-more-free-bugs-initiatives.html.

[10] CATAL, C., AND DIRI, B. A systematic review of software faultprediction studies. Expert Systems with Applications 36, 4 (2009),7346–7354.

[11] Chromium Development Calendar and Release Info. http://www.chromium.org/developers/calendar.

[12] Severity Guidelines for Security Issues. https://sites.google.com/a/chromium.org/dev/developers/severity-guidelines.

[13] Chromium Bug Tracker, 2013. http://crbug.com.

[14] Security: Pwnium 2 tcmalloc profile bug, 2012. http://crbug.com/154983.

[15] DAVIS, N. Secure Software Development Life Cycle Processes,July 2012. https://buildsecurityin.us-cert.gov/bsi/articles/knowledge/sdlc/326-BSI.html.

[16] Defense in Depth. http://www.nsa.gov/ia/_files/support/defenseindepth.pdf.

[17] MozillaWiki: Electrolysis, April 2011. https://wiki.mozilla.org/Electrolysis.

[18] EDMUNDSON, A., HOLTKAMP, B., RIVERA, E., FINIFTER, M.,METTLER, A., AND WAGNER, D. An Empirical Study on theEffectiveness of Security Code Review. In Proceedings of theInternational Symposium on Engineering Secure Software andSystems (March 2013).

[19] EVANS, C. Celebrating Six Months of Chromium Security Re-wards, July 2010. http://blog.chromium.org/2010/07/celebrating-six-months-of-chromium.html.

[20] EVANS, C. Bug bounties vs. black (& grey) markets, May2011. http://scarybeastsecurity.blogspot.com/2011/05/bug-bounties-vs-black-grey-markets.html.

15

[21] EVANS, C. Personal Communication, March 2013.

[22] EVANS, C., GROSSE, E., MEHTA, N., MOORE, M., ORMANDY,T., TINNES, J., ZALEWSKI, M., AND TEAM, G. S. RebootingResponsible Disclosure: a focus on protecting end users, July2010. http://googleonlinesecurity.blogspot.com/2010/07/rebooting-responsible-disclosure-focus.html.

[23] EVANS, C., AND SCHUH, J. Pwnium: rewards for exploits,February 2012. http://blog.chromium.org/2012/02/pwnium-rewards-for-exploits.html.

[24] FINIFTER, M., AND WAGNER, D. Exploring the relationship be-tween web application development tools and security. In USENIXconference on Web application development (2011).

[25] MozillaWiki: RapidRelease/Calendar, January 2013. https://wiki.mozilla.org/RapidRelease/Calendar.

[26] FISHER, D. Microsoft Says No to Paying Bug Bounties, July2010. http://threatpost.com/microsoft-says-no-paying-bug-bounties-072210/.

[27] Chrome Releases: Stable Updates. http://googlechromereleases.blogspot.com/search/label/Stable%20updates.

[28] GORENC, B. Pwn2Own 2013, January 2013. http://dvlabs.tippingpoint.com/blog/2013/01/17/pwn2own-2013.

[29] HATTON, L. Predicting the Total Num-ber of Faults Using Parallel Code Inspections.http://www.leshatton.org/2005/05/total-number-of-faults-using-parallel-code-inspections/, May 2005.

[30] HOFMANN, C. Personal Communication, March 2013.

[31] HOLLER, C. Trying new code analysis techniques, January 2012.https://blog.mozilla.org/decoder/2012/01/27/trying-new-code-analysis-techniques/.

[32] Creating a Computer Security Incident Response Team: A Pro-cess for Getting Started, February 2006. https://www.cert.org/csirts/Creating-A-CSIRT.html.

[33] MATTHEW FINIFTER AND DEVDATTA AKHAWE ANDDAVID WAGNER. Chrome and Firefox VRP datasets,June 2013. https://gist.github.com/devd/a62f2afae9f1c93397f5.

[34] The MEGA Vulnerability Reward Program, February 2013.https://mega.co.nz/#blog_6.

[35] MEIN, A., AND EVANS, C. Dosh4Vulns: Google’s VulnerabilityReward Programs”, March 2011.

[36] MELVEN, I. MozillaWiki: Features/Security/Low rights Firefox,August 2012. https://wiki.mozilla.org/Features/Security/Low_rights_Firefox.

[37] MILLER, C. The legitimate vulnerability market: the secretiveworld of 0-day exploit sales. In WEIS (2007).

[38] MILLS, E. Facebook launches bug bounty program, July 2011.http://news.cnet.com/8301-27080 3-20085163-245/facebook-launches-bug-bounty-program/.

[39] MOZILLA BUGZILLA. Bug 790923: Content process sandboxingvia seccomp filter. https://bugzil.la/790923.

[40] MOZILLA FOUNDATION. Mozilla Foundation Security Ad-visories, January 2013. https://www.mozilla.org/security/announce/.

[41] Netscape announces ”netscape bugs bounty” with release ofnetscape navigator 2.0 beta. The Internet Archive. http://web.archive.org/web/19970501041756/www101.netscape.com/newsref/pr/newsrelease48.html.

[42] NEUHAUS, S., AND PLATTNER, B. Software security economics:Theory, in practice. In WEIS (2012).

[43] NEUHAUS, S., ZIMMERMANN, T., HOLLER, C., AND ZELLER,A. Predicting vulnerable software components. In Proceedingsof the 14th ACM conference on Computer and communicationssecurity (2007), ACM, pp. 529–540.

[44] National Vulnerability Database. http://nvd.nist.gov/.

[45] OBES, J. L., AND SCHUH, J. A Tale of Two Pwnies (Part1), May 2012. http://blog.chromium.org/2012/05/tale-of-two-pwnies-part-1.html.

[46] Understanding Operational Security. http://www.cisco.com/web/about/security/intelligence/opsecurity.html.

[47] OZMENT, A., AND SCHECHTER, S. E. Milk or wine: does soft-ware security improve with age. In In USENIX-SS06: Proceedingsof the 15th conference on USENIX Security Symposium (2006),USENIX Association.

[48] RAYMOND, E. S. The Cathedral and the Bazaar, 1st ed. O’Reilly& Associates, Inc., Sebastopol, CA, USA, 1999.

[49] RESCORLA, E. Is finding security holes a good idea? IEEESecurity & Privacy 3, 1 (2005), 14–19.

[50] CERT/CC Vulnerability Disclosure Policy, November 2012.https://www.cert.org/kb/vul_disclosure.html.

[51] Chromium bug tracker: Sandbox bypasses found in review, 2013.http://goo.gl/13ZlR.

[52] SCHOLTE, T., BALZAROTTI, D., AND KIRDA, E. Quo vadis? astudy of the evolution of input validation vulnerabilities in webapplications. Financial Cryptography and Data Security (2012),284–298.

[53] Secunia Vulnerability Coordination Reward Program (SVCRP).https://secunia.com/community/research/svcrp/.

[54] THE BLUEHAT TEAM. Microsoft Security Bounty Programs.http://www.microsoft.com/security/msrc/report/bountyprograms.aspx, June 2013.

[55] THE CHROMIUM AUTHORS. Vulnerability Rewards Program:Rewards FAQ, 2010. http://goo.gl/m1MdV.

[56] VEDITZ, D. Personal Communication, February 2013.

[57] Vulnerability Remediation, September 2010. https://www.cert.org/vuls/remediation.html.

16