The Bloom Paradox Ori Rottenstreich Joint work with Isaac Keslassy Technion, Israel.

The Bloom Paradox

Ori Rottenstreich

Joint work with Isaac Keslassy

Technion, Israel

• Requirement: A data structure in user with fast answer to• Solutions:

o O(n) – Searching in a listo O(log(n)) – Searching in a sorted listo O(1) – But with false positives / negatives

Slocal cache

Problem Definition

2

Mcentral memory with

all elements

vuzyxzx

x

usercost = 10

cost = 1x

y

cost = 10

y

user

y

• False Positive: but the data structure answers

• Results in a redundant access to the local cache.

Additional cost of 1.

• False Negative: but the data structure answers

• Results in an expensive access to the central memory instead of the local cache.

Additional cost of 10-1=9.

Two Possible Errors

3

x

y

1

• Initialization: Array of zero bits.

• Insertion: Each of the elements is hashed times, the corresponding bits are set.

• Query: Hashing the element, checking that all bits are set.

• False positive rate (probability) of .

• No false negatives.

Bloom Filters (Bloom, 1970)

4

0000000000 00

1

y1 1

0000000000 00

1 1

z

x11

1 1

1 11 1 1

x11 1 w

1 11

• Cache/Memory Framework• Packet Classification• Intrusion Detection• Routing• Accounting• Beyond networking: Spell Checking, DNA Classification

• Can be found in o Google's web browser Chromeo Google's database system BigTableo Facebook's distributed storage system Cassandrao Mellanox's IB Switch System

Bloom Filters are Widely Used

5

The Bloom Paradox

6

Sometimes, it is better to disregard the Bloom filter results, and in fact not to even query it,

thus making the Bloom filter useless.

Outline

Introduction to Bloom Filters The Bloom Paradox

o The Bloom Paradox in Bloom Filterso Analysis of the Bloom Paradox o The Bloom Paradox in the Counting Bloom Filter

Summary

7

• Parameters:

• Extreme case without locality: All elements with equal probability of

belonging to the cache.o Toy example

Bloom Paradox Example

8

Bloom filter

• Parameters:• Let be the set of elements that the Bloom filter indicates are in

o In particular, no false negatives in Bloom filter

• Intuition:

Slocal cache


all elements

vuzyxzx

cost = 10cost = 1

cost = 10


. .

userBBloom filterBloom filter

9



• Surprise:

cost = 1

Slocal cache


all elements

vuzyxzx

cost = 10

cost = 10


. . 9

BBloom filter



• Surprise:

The Bloom filter indicates the membership of

elements. Only of them are indeed in .


. .

BBloom filter

• When the Bloom filter states that , it is wrong with probability

• Average cost if we listen to the Bloom filter:

• Average cost if we don’t:

The Bloom filter is useless!


11

Don’t listen to the Bloom filter

= =

Outline



Summary

12

• The cost of a false positive : 1• The cost of a false negative :

• In the cache example:

Costs of the Two Possible Errors

13

• Let be the a priori membership probability of o i.e. before getting the answer of the Bloom filter

• Intuition: The Bloom paradox occurs more often when:o is small

Conditions for the Bloom Paradox

14

localcache

Bloom filter

central memory


• Intuition: The Bloom paradox occurs more often when:o is smallo is large (i.e. is small)


14central memory

localcache

Bloom filter


• Intuition: The Bloom paradox occurs more often when:o is small o is large (i.e. is small)o is small (because the Bloom filter implicitly assumes )


14

Bloom filtercentral memory

localcache


• Intuition: The Bloom paradox occurs more often when:o is small o is large (i.e. is small)o is small (because the Bloom filter implicitly assumes )

• Theorem 1:The Bloom paradox occurs if and only if

• Boundaries of the Bloom Paradox: (for )


14

If and the Bloom paradox occurs if

• Theorem 1:The Bloom paradox occurs if and only if

Bloom Filter Improvements

15

• Use the formula to improve the Bloom filter o Only insert / query Bloom filter if the formula expects it to be

useful

Bloom filtercentral memory

localcache

Outline



Summary

16

1

• Bloom filters do not support deletions of elements. Simply resetting bits might cause false negatives.

• The solution: Counting Bloom filters - Storing array of counters instead of bits.o Insertion: Incrementing counters by one.o Deletion: Decrementing counters by one. o Query: Checking that counters are positive.

• The same false positive probability.• Require too much memory, e.g. 57 bits per element for .

Counting Bloom Filters (CBFs)

y+1 +1

0102001010 01

+1 +1x

+1+1

0000001010 00

x11 111

• Queryo Checking that counters are positive.

o Question: Which is more likely to be correct? y or z?

Counting Bloom Filter Query

18

0381052010 12

zy

y

• Theorem 2:Let denote the values of the counters pointed by the

set of hash functions. Then,

19

The Bloom Paradox in the Counting Bloom Filter

Only counters product matters!

• Parameters: n=3328, m = 28485, k=6 20

CBF Based Membership Probability

-Before checking CBF, a priori membership probability = ≈ 0.03-CBF indicates counters product=8 a posteriori membership probability ≈ 0.69

• Internet trace (equinix-chicago) with real hash functions.

Counting Bloom filter parameters: n=210, m / n = 30, k=5, 220

queries

21

Experimental Results

• Discovery of the Bloom paradox

• Importance of the a priori membership probability

• Using the counters product to estimate the correctness of a positive indication of the CBF

Concluding Remarks

22

Thank You

The Bloom Paradox Ori Rottenstreich Joint work with Isaac Keslassy Technion, Israel.

Documents

Transcript of The Bloom Paradox Ori Rottenstreich Joint work with Isaac Keslassy Technion, Israel.