Scalable and Effective Fuzzing of Google Chrome Browser

Positive Hack Days VIMoscow, 17-18 May 2016Max Moroz, Google

● Google Chrome Security team, Bugs--

● BalalaikaCr3w, LC↯BC

● CTF, BugBounty, etc

Bio

2

1. ClusterFuzz overview

2. Memory Debugging Tools

3. LibFuzzer

4. Bugs statistics

5. Vulnerability Reward Program

6. Get some $$$ with ClusterFuzz

Agenda

3

A software testing technique, often automated or semi-automated, that involves passing invalid, unexpected or random input to a program and monitor result for crashes, failed assertions, races, leaks, etc.

Definition: Fuzzing

4

The Big 3s in Fuzzing

5

Fuzzers Memory Tools ScaleKnow

WHEREto

reach

KnowWHEN

youreached

KnowHOW

toreach

FASTER

6

ClusterFuzz overview

Suppliers of Chrome vulns

7

ClusterFuzz

8

BackendFrontend

Bots

Builder Bots1. Trunk (Release (-

O2+-O1), Debug) 2. Stable, Beta

Sync Bots using GlusterFS

1. Fuzzer Data bundles

2. Build Cache3. Code Coverage

Local Storage1. Tests

2. Fuzzers 3. Some Fuzzer Data Bundles

4. Builds

Google Cloud Storage

Blobstore1. Custom Binaries2. Crash testcases

3. User uploaded testcases4. Fuzzers

5. Small data bundles

Task Pull Queue

High replication datastoreAll the metadata - Jobs, Crash List, Crash Stats, Fuzzers, Bot

tasks, Bot heartbeats, etc

Tasks1. Fuzz

2. Minimize 3. Impact

4. Regression5. Fixed

6. Sheriff7. Coverage

ClusterFuzz UI

Appengine Google Compute Engine and Chrome Lab

1. Automated crash detection, analysis and management

2. Fully reproducible and minimized testcases

3. Real-time regression and fixed testing

ClusterFuzz: Goals

9

ClusterFuzz: Goals

10

ClusterFuzz: Life of a Crash

11

Find Actionable, Reproducible and Minimized Tests

Verify

Test Deduplication, Regresse, Production Impact AnalyzeFixed, Merged, Released

Prevent Regressions

12

Goal 1:Crash automation & management

Fuzzer Types

13

Generation BasedGenerate from scratch with no prior state

Examplehttps://bugs.webkit.org/show_bug.cgi?id=60831

<script>document.body = document.createElement(‘iframe’);</script>

Fuzzer Types

14

Mutation BasedMutate existing state based on some rules

Examplecrbug.com/552046

--- orig.pdf+++ crash.pdf@@ -57,7 +57,7 @@ /DecodeParms [null 8 0 R] /Type /XObject /Width 1760-/Filter [/FlateDecode /DCTDecode]+/Filter [/JBIG2Decode /DCTDecode] /Height 1248 /Length 2277

Fuzzer Types

15

EvolutionaryGeneration or mutation based or both, in-process with code coverage feedback

Examplecrbug.com/575205

SELECT'\xef(\xfb;DS\x1aLEETABL\xfeES'REGEXP';0\t\tC LE|A*(\xc8*.+!*)*h*00\x0b$T''&'

● Setup build {trunk}, run application with test

● Choose params

● Gestures, tool settings, timeout multiplier, window

location+size, etc.

● Figure out resource dependencies

● Test for reproducibility

● Test for duplicates

● Store crash, coverage, stats, etc

Infrastructure

16

● Disable inline frames (-inlining in llvm-symbolizer)

● Crash_type - e.g. “Heap-buffer-overflow READ 2”

● Crash_state - top 3 frames

● +namespaces, -line_numbers

● E.g.

● WebCore::SVGDocumentExtensions::removeAnimationElement

● WebCore::SVGSMILElement::removedFrom

● WebCore::ContainerNode::removeChild

● security_flag

Testcase Duplication Check

17

Code Coverage: Aggregate View Sample

18

Scale, Scale and Scale (in cores)

19

12k

5kLinux

4kAndroid x86200+Android devices

3kWindows

20Mac

{wip}iOS

20

Goal 2:Fully reproducible and minimized testcases

● Same bot configuration as crash

● Multi-threaded minimization based on Delta Debugging

● Custom minimizers for some file types

● + Gesture minimization

● + Resource minimization

● + Tool settings correction (redzone {asan}, history size

{tsan}, origins {msan}, etc)

Fully reproducible & minimized tests

21

● Tokenize the input

● Generate hypotheses that certain groups of tokens are not

required for the crash

● Test hypotheses by running the test with the tokens from the

hypothesis removed

● If it crashes, removing them was fine

● If not, try breaking it into smaller groups

How does the minimizer work?

22

● Stack trace with default redzone (128)==9485== ERROR: AddressSanitizer heap-use-after-free on address 0x7f8f653ff11e atpc 0x7f8f849fbb10 bp 0x7f8f5514a0a0 sp 0x7f8f5514a098READ of size 2 at 0x7f8f653ff11e thread T14 #0 0x7f8f849fbb10 in WTF::charactersToIntStrict(unsigned short const*,unsigned long, bool*, int) #1 0x7f8f8589d863 in WebCore::InlineTextBox::isLineBreak() const #2 0x7f8f858a771d in WebCore::InlineTextBox::containsCaretOffset(int)

● Actual stack trace with bigger redzone (1024 / 2048)==14334== ERROR: AddressSanitizer heap-buffer-overflow on address 0x7f7e42b9b81cat pc 0x7f7e8f79a6ca bp 0x7f7e3cc30040 sp 0x7f7e3cc30038READ of size 2 at 0x7f7e42b9b81c thread T15#0 0x7f7e8f79a6ca in WebCore::InlineTextBox::isLineBreak() const#1 0x7f7e8f7abddd in WebCore::InlineTextBox::containsCaretOffset(int) const#2 0x7f7e8e06b19d in WebCore::Position::inRenderedText() const

Tools Settings Correction: ASAN example crbug.com/118662

23

24

Goal 3:Real-time regression and fixed testing

● Use LKGR builds archived on Google Cloud

● Account for bad builds / startup crashes

● Use a LOOK_BEHIND_WINDOW

● If previous step failed, then use binary bisect

● Use FindIt to find culprit changelist

Real-time regression and fixed testing

25

● Manual owner triage is usually …

● Slow

● Inaccurate

● Complex

● From factors such as refactorings, size of regression range, etc

FindIt: Culprit CL Finder

26

1. Parse the stacktrace

FindIt: How it works

27

2. Parse the changelog(s) in the regression range

FindIt: How it works

28

3. Generate a list of suspected CLs, and sort / filter the results

FindIt: How it works

29

4. Show blame information if no results are available

FindIt: How it works

30

FindIt: How it works

31

32

ClusterFuzz: Sample

Testcase Report

33

FindIt: Culprit CL Finder

34

Filed bug: crbug.com/430925

35

Patched + Verified: < 1 day

36

Or not patched...

37

38

Memory Debugging Tools

Memory Debugging Tools

39

Tool \ OS Android Chrome OS Linux OS X Windows

ASan + + + + +

MSan +

UBSan +

TSan +

LSan +

CFI +

SyzyASAN +

DrFuzz +

DrMemory +

● AddressSanitizer (aka ASan)

● Detects use-after-free, buffer overflows (heap, stack, globals),

stack-use-after-return, container-overflow

● Cpu: 2x, memory 1.5x-3x

● ThreadSanitizer (aka TSan)

● Detects data races, esp on use-after-frees. Object vtpr

● Cpu: 4x-10x, memory: 5x-8x

Memory Debugging Tools

40

● MemorySanitizer (aka MSan)

● Detects uninitialized memory reads

● Cpu: 3x, memory: 2x

● Special mode: origins

● UndefinedBehaviorSanitizer (aka UBSan)

● Detects several classes of bugs (19?), esp on type confusion,

signed-integer-overflow, undefined shift, etc.

● Cpu: 10-50%

● Memory: ~1x (no allocator, no shadow)

Memory Debugging Tools

41

● Others

● SyzyASAN

● DrFuzz

● DrMemory

Memory Debugging Tools

42

● Container-overflow (ASan):#include <vector>#include <assert.h>typedef long T;int main() { std::vector<T> v; v.push_back(0); v.push_back(1); v.push_back(2); assert(v.capacity() >= 4); assert(v.size() == 3); T *p = &v[0];

// Here the memory is accessed inside a heap-allocated buffer // but outside of the region `[v.begin(), v.end())`. return p[3]; // OOPS.}

Memory Debugging Tools: Examples

43

44

LibFuzzer

● Fuzzing very wide scope

● Cannot fuzz specific function

● Hard to fuzz network protocols

● Speed of regular fuzzers (html, css, dom, etc mutators):

● From ~30K to ~2.5M testcases per week

● Regular fuzzers are great, but not enough...

Problems of regular fuzzing of browsers

45

● More granulated fuzzing

● Faster fuzzing

● Smarter fuzzing

● Easier fuzzer writing

Goals

46

● In-process guided fuzz testing

● Very effective at function / protocol level

● It’s easy not hard to write a “target function”

● Can be checked along with unit-tests

● Maintained by Googlers :)

LibFuzzer

47

libFuzzer example [code]

48

#include <stdint.h>

#include "third_party/icu/source/i18n/unicode/regex.h"

// Entry point for LibFuzzer.extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) { UParseError pe = { 0 }; UErrorCode status = U_ZERO_ERROR; URegularExpression* re = uregex_open(reinterpret_cast<const UChar*>(data), static_cast<int>(size) / sizeof(UChar), 0, &pe, &status); if (re) uregex_close(re);

return 0;}

libFuzzer example [crash]

49

==19494==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x4d306d in icu_54::RegexCompile::doParseActions(int) third_party/icu/source/i18n/regexcmp.cpp:1702:13 #1 0x4a1f3c in icu_54::RegexCompile::compile(UText*, UParseError&, UErrorCode&) third_party/icu/source/i18n/regexcmp.cpp:228:13 #2 0x5fc98e in icu_54::RegexPattern::compile(UText*, unsigned int, UParseError&, UErrorCode&) third_party/icu/source/i18n/repattrn.cpp:357:5 #3 0x601be7 in uregex_open_54 third_party/icu/source/i18n/uregex.cpp:155:20 #4 0x49dc3f in LLVMFuzzerTestOneInput testing/libfuzzer/fuzzers/icu_uregex_open_fuzzer.cc:13:28

Uninitialized value was created by an allocation of compiler in the stack frame of function _ZN6icu_5412RegexPattern7compileEP5UTextjR11UParseErrorR10UErrorCode #0 0x5fc220 in icu_54::RegexPattern::compile(UText*, unsigned int, UParseError&, UErrorCode&) third_party/icu/source/i18n/repattrn.cpp:325

● Dictionaries

● Seed corpus

● Custom mutators

● Coverage visualisation

Ways to improve a libFuzzer

50

● dictionary and custom options for ICU uregex_open() fuzzer:○ coverage ↑ 25+%

● dictionary for libxml fuzzer:○ 5 new security bugs

● dictionary and seed corpus for libpng fuzzer:○ coverage ↑↑↑ 70+%

Dictionaries are awesome!

51

52

53

54

55

● Timeout (DoS)

● Heap-buffer-overflow

● Use-of-uninitialized-value

● Heap-buffer-overflow

● Heap-buffer-overflow

sqlite3 fuzzer with dictionary at ClusterFuzz

56

● ASan, MSan, UBSan builds

● Corpus Synchronization with Google Cloud Storage

● Support for dictionaries and custom runtime options

● Automated exploitability scoring for crashes

● Statistics dashboard

LibFuzzer & ClusterFuzz

57

● ~85 target functions for Chromium

● 88 bugs filed

○ 46 security bugs

● 700 GCE bots running target functions at ClusterFuzz 24/7

● ~310 billion testcases daily

● ~8,000 corpus synchronization jobs daily

● From ~200k to ~800k corpus files

● Best coverage: 100.00% of funcs, 89.32% of edges

Numbers

58

● 200+ bugs in third party, Google, other software

● http://llvm.org/docs/LibFuzzer.html#trophies:

Trophies of LibFuzzer outside of Chrome

59

● Slow inputs and timeouts

● Out-of-memory with valid inputs

● You have to improve / refresh some fuzzers

● Stuck at easy crashes

Problems

60

Example: 10 lines fuzzer

61

extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {

auto var = xxxxxxxxxxxxxx(data, size, NULL, 0); if (var) { xxxxxxxxxx_Free(var); }

return 0;}

Example: 10 lines fuzzer ⇨ 10+ bugs

62

● Memory leak

● 7 × Heap-buffer-overflow

● Timeout (DoS)

● Heap-based buffer-underreads

● Heap-based buffer overread

● Undesired side effects

Self-fuzzing!

63

WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x56b2594 in fuzzer::Fuzzer::AlarmCallback() third_party/llvm/lib/Fuzzer/FuzzerLoop.cpp:138:7 <...> #9 0x56abbb7 in fuzzer::FileToVector() third_party/llvm/lib/Fuzzer/FuzzerIO.cpp:65:3 <...> #14 0x56a08a8 in main third_party/llvm/lib/Fuzzer/FuzzerMain.cpp:25:10

Uninitialized value was created by an allocation of 'F.i' in the stack frame of function '_ZN6fuzzer12FuzzerDriverEiPPcPFiPKhmE' #0 0x56a1040 in fuzzer::FuzzerDriver() third_party/llvm/lib/Fuzzer/FuzzerDriver.cpp:395

64

● The same target function

● Shared corpus:

● Need to be pruned or minimized on regular basis

● Different mutations:

● When one fuzzer got stuck, another one may help

● PoC is ready:● https://github.com/llvm-mirror/llvm/blob/master/lib/Fuzzer/afl/afl_driver.cpp

● Work in progress

Multiple fuzzing engines approach

65

66

return 0;

67

Bugs statistics

Bugs statistics

68

69

Bugs statistics

70

Bugs statistics

71

Vulnerability Reward Program

● Reporter submits a bug on crbug.com

● Bug gets fixed

● Reward panel determines reward based on table and precedent

● Reporter gets cash money

Chromium VRP: high level process

72

Reward amounts

73 https://www.google.com/about/appsecurity/chrome-rewards/index.html

Recent updates

74 https://www.google.com/about/appsecurity/chrome-rewards/index.html

● Max reward: $50,000 ⇨ $100,000:● compromise a Chromebook or Chromebox with device persistence in guest

mode (i.e. guest to guest persistence with interim reboot, delivered via a web page)

● Download Protection bypass (i.e. SafeBrowsing) *:

● Baseline: $0 - $500

● High-quality report: $1,000

* Landing a blacklisted test binary on disk where a typical user could execute it, on Mac or Windows. The file type on disk must lead to non-sandboxed code execution after minimal user interaction with the file.

Chromium VRP stats

75

● Over $500,000 paid out last year

● More than $2,500,000 in rewards since 2010

● Median payment went from $2,000 to $3,000

● Top 0x05 reporters take ~70% of the earnings

76

Trusted Researcher Program

ClusterFuzz’s first days...

77

ClusterFuzz now

78

● Invitation-only

● Skilled fuzzer developers

● Fuzz at Google scale

● Get 100% of the reward for bugs + $500 bonus:

● Bugs shouldn’t be found by our* fuzzers within 48 hours

● *our - written by Googlers

● Target functions for LibFuzzer are also in scope!

How to get an invite?

● Submit at least one bug found by one of your fuzzers

● mailto:mmoroz@chromium.org

Trusted Researcher program

79https://www.google.com/about/appsecurity/chrome-rewards/index.htmlhttp://www.chromium.org/Home/chromium-security/bugs/developing-fuzzers-for-clusterfuzz

● More than $120,000 paid out for 2014-2015

● Rewards in range: $500 - $4,500

Stats

80

Thank you

Contacts● email: mmoroz@chromium.org● twitter: @dor3s● telegram: @dor1s

Q & A

81