Efficient Approximate Search on String CollectionsPart II

Marios Hadjieleftheriou Chen Li

Outline Motivation and preliminaries Inverted list based algorithms Gram Signature algorithms Length normalized algorithms Selectivity estimation Conclusion and future directions

2/68

N-Gram Signatures Use string signatures that upper bound

similarity Use signatures as filtering step Properties:

Signature has to have small size Signature verification must be fast False positives/False negatives

Signatures have to be “indexable”

3/68

Known signatures Minhash

Jaccard, Edit distance Prefix filter (CGK06)

Jaccard, Edit distance PartEnum (AGK06)

Hamming, Jaccard, Edit distance LSH (GIM99)

Jaccard, Edit distance Mismatch filter (XWL08)

Edit distance

4/68

Prefix Filter Bit vectors:

Mismatch vector:

s: matches 6, missing 2, extra 2 If |sq|6 then s’s s.t. |s’|3, |s’q| For at least k matches, |s’| = l - k + 1

1 2 6 1411

q

s

3 4 5 7 8 9 10 12 13

5/68

Using Prefixes Take a random permutation of n-gram

universe:

Take prefixes from both sets: |s’|=|q’|=3, if |sq|6 then s’q’

6 9 1 137

q

s

11 14 8 2 3 4 5 10 12

6/68

t1 t2 t6 t14t11

sq

t4 t8

Prefix Filter for Weighted Sets For example:

Order n-grams by weight (new coordinate space)

Query: w(qs)=Σiqs wi τ Keep prefix s’ s.t. w(s’) w(s) - α

Best case: w(q/q’ s/s’) = α Hence, we need w(q’s’) τ - α

w1 w2 … w14

w1 w2 w40 0w1 w2 w40 0

w(s)-α αs/s’s’

7/68

Prefix Filter Properties The larger we make α, the smaller the prefix The larger we make α, the smaller the range

of thresholds we can support: Because τα, otherwise τ-α is negative.

We need to pre-specify minimum τ Can apply to Jaccard, Edit Distance, IDF

8/68

Other Signatures Minhash (still to come) PartEnum:

Upper bounds Hamming Select multiple subsets instead of one prefix Larger signature, but stronger guarantee

LSH: Probabilistic with guarantees Based on hashing

Mismatch filter: Use positional mismatching n-grams within the prefix to

attain lower bound of Edit Distance

9/68

Signature Indexing Straightforward solution:

Create an inverted index on signature n-grams Merge inverted lists to compute signature

intersections For a given string q:

- Access only lists in q’- Find strings s with w(q’ ∩ s’) ≥ τ - α

10/68

The Inverted Signature Hashtable (CCVX08) Maintain a signature vector for every n-gram Consider prefix signatures for simplicity:

s’1={ ‘tt ’, ‘t L’}, s’2={‘t&t’, ‘t L’}, s’3=… co-occurence lists: ‘t L’: ‘tt ’ ‘t&t’ …

‘&tt’: ‘t L’ … Hash all n-grams (h: n-gram [0, m]) Convert co-occurrence lists to bit-vectors of size m

11/68

Example

at&t&tlabt L la…

Hashtable100011010101 …

labat&t&tt L la…

Hash54510

s’1s’2s’3s’4s’5…

Signaturesat&, lat&t, at&t L, at&abo, t&tt&t, la

12/68

Using the Hashtable? Let list ‘at&’ correspond to bit-vector 100011

There exists string s s.t. ‘at&’ s’ and s’ also contains some n-grams that hash to 0, 1, or 5

Given query q: Construct query signature matrix:

Consider only solid sub-matrices P: rq’, pq We need to look only at rq’ such that w(r)τ-α and

w(p)τ

p

rat&

lab

q’q at& lab t&t res …

1 1 1 0

1 1 0 1

…

…

13/68

Verification How do we find which strings correspond to a

given sub-matrix? Create an inverted index on string n-grams Examine only lists in r and strings with w(s)τ

- Remember that rq’ Can be used with other signatures as well

14/68

Outline Motivation and preliminaries Inverted list based algorithms Gram Signature algorithms Length normalized algorithms Selectivity estimation Conclusion and future directions

15/68

Length Normalized Measures What is normalization?

Normalize similarity scores by the length of the strings.- Can result in more meaningful matches.

Can use L0 (i.e., the length of the string), L1, L2, etc. For example L2:

- Let w2(s) Σtsw(t)2

- Weight can be IDF, unary, language model, etc.- ||s||2 = w2(s)-1/2

16/68

The L2-Length Filter (HCKS08)

Why L2? For almost exact matches. Two strings match only if:

- They have very similar n-gram sets, and hence L2 lengths

- The “extra” n-grams have truly insignificant weights in aggregate (hence, resulting in similar L2 lengths).

17/68

Example “AT&T Labs – Research” L2=100 “ATT Labs – Research” L2=95 “AT&T Labs” L2=70

If “Research” happened to be very popular and had small weight?

“The Dark Knight” L2=75 “Dark Night” L2=72

18/68

Why L2 (continued) Tight L2-based length filtering will result in very

efficient pruning. L2 yields scores bounded within [0, 1]:

1 means a truly perfect match. Easier to interpret scores. L0 and L1 do not have the same properties

- Scores are bounded only by the largest string length in the database.

- For L0 an exact match can have score smaller than a non-exact match!

19/68

Example q={‘ATT’, ‘TT ’, ‘T L’, ‘LAB’, ‘ABS’} L0=5 s1={‘ATT’} L0=1 s2=q L0=5

S(q, s1)=Σw(qs1)/(||q||0 ||s1||0)=10/5 = 2 S(q, s2)=Σw(qs2)/(||q||0 ||s2||0)=40/25<2

20/68

Problems L2 normalization poses challenges.

For example:- S(q, s) = w2(qs)/(||q||2 ||s||2) - Prefix filter cannot be applied.- Minimum prefix weight α?

Value depends both on ||s||2 and ||q||2. But ||q||2 is unknown at index construction time

21/68

Important L2 Properties Length filtering:

For S(q, s) ≥ τ τ ||q||2 ||s||2 ||q||2 / τ We are only looking for strings within these

lengths. Proof in paper

Monotonicity …

22/68

Monotonicity Let s={t1, t2, …, tm}. Let pw(s, t)=w(t) / ||s||2 (partial weight of s)

Then: S(q, s) = Σ tqs w(t)2 / (||q||2 ||s||2)=

Σtqs pw(s, t) pw(q, t) If pw(s, t) > pw(r, t):

w(t)/||s||2 > w(t)/||r||2 ||s||2 < ||r||2

Hence, for any t’ t: w(t’)/||s||2 > w(t’)/||r||2 pw(s, t’) > pw(r, t’)

23/68

Indexing Use inverted lists sorted by pw():

id strings01234

richstickstichstuckstatic

4

3 104 2

2-grams

atchckicristtatituuc

203 10 4 1 2

44 1 233

• pw(0, ic) > pw(4, ic) > pw(1, ic) > pw(2, ic)

||0||2 < ||4||2 < ||1||2 < ||2||2 24/68

L2 Length Filter Given q and τ, and using length filtering:

02

2

2

4

4

4

00

44

• We examine only a small fraction of the lists

1

atchckicristtatituuc

21

1

1

3

3

33

4

04

00

44

4

2

2

2

25/68

Monotonicity

4

3 104 2

atchckicristtatituuc

203 10 4 1 2

44 1 233

If I have seen 1 already, then 4 is not in the list:

26/68

Other Improvements Use properties of weighting scheme

Scan high weight lists first Prune according to string length and maximum

potential score Ignore low weight lists altogether

27/68

Conclusion Concepts can be extended easily for:

BM25 Weighted Jaccard DICE IDF

Take away message: Properties of similarity/distance function can play

big role in designing very fast indexes. L2 super fast for almost exact matches

28/68

Outline Motivation and preliminaries Inverted list based algorithms Gram signature algorithms Length-normalized measures Selectivity estimation Conclusion and future directions

29/68

The Problem Estimate the number of strings with:

Edit distance smaller than k from query q Cosine similarity higher than τ to query q Jaccard, Hamming, etc…

Issues: Estimation accuracy Size of estimator Cost of estimation

30/68

Motivation Query optimization:

Selectivity of query predicates Need to support selectivity of approximate string

predicates Visualization/Querying:

Expected result set size helps with visualization Result set size important for remote query

processing

31/68

Flavors Edit distance:

Based on clustering (JL05)

Based on min-hash (MBKS07)

Based on wild-card n-grams (LNS07)

Cosine similarity: Based on sampling (HYKS08)

32/68

Selectivity Estimation for Edit Distance Problem:

Given query string q Estimate number of strings s D Such that ed(q, s) δ

33/68

Sepia - Clustering (JL05, JLV08)

Partition strings using clustering: Enables pruning of whole clusters

Store per cluster histograms: Number of strings within edit distance 0,1,…,δ from the cluster

center Compute global dataset statistics:

Use a training query set to compute frequency of strings within edit distance 0,1,…,δ from each query

34/68

Edit Vectors Edit distance is not discriminative:

Use Edit Vectors

3D space vs 1D space

pi

LuciaLuciano

Lucas2

2Lukas

q3

Ci<1,1,1> <2,0,0>

<1,1,0>

35/68

Visually...p1C1 p2

C2 pnCn

#Edit Vector

<0, 0, 0>

<0, 0, 1>

<1, 0, 2>

4

12

7

…

F1

#Edit Vector

<0, 0, 0>

<0, 1, 0>

<1, 0, 1>

3

40

6

…

F2

#Edit Vector

<0, 0, 0>

<1, 0, 2>

<1, 1, 1>

2

84

1

…

Fn

Global Table

v(q,pi) v(pi,s) ed(q,s) # %

<1, 0, 1>

<1, 0, 1>

<1, 0, 1>

…

<1, 1, 0>

<1, 1, 0>

<1, 1, 0> <1, 0, 2>

<1, 0, 2>

<1, 0, 2>

…

<0, 0, 1>

<0, 0, 1>

<0, 0, 1>

… …

1

2

3

3

4

5

1

4

7

21

63

84

14

57

100

25

75

100

36/68

Selectivity Estimation Use triangle inequality:

Compute edit vector v(q,pi) for all clusters i If |v(q,pi)| ri+δ disregard cluster Ci

ri

δ

piq

37/68

Selectivity Estimation Use triangle inequality:

Compute edit vector v(q,pi) for all clusters i If |v(q,pi)| ri+δ disregard cluster Ci

For all entries in frequency table:- If |v(q,pi)| + |v(pi,s)| δ then ed(q,s) δ for all s- If ||v(q,pi)| - |v(pi,s)|| δ ignore these strings- Else use global table:

Lookup entry <v(q,pi), v(pi,s), δ> in global table Use the estimated fraction of strings

38/68

Example δ =3 v(q,p1) = <1,1,0>

v(p1,s) = <1,0,2>

Global lookup:[<1,1,0>,<1,0,2>, 3] Fraction is 25% x 7 =

1.75 Iterate through F1, and

add up contributions

#Edit Vector

<0, 0, 0>

<0, 0, 1>

<1, 0, 2>

4

12

7

…

F1

v(q,pi) v(pi,s) ed(q,s) # %

<1, 0, 1>

<1, 0, 1>

<1, 0, 1>

…

<1, 1, 0>

<1, 1, 0>

<1, 1, 0> <1, 0, 2>

<1, 0, 2>

<1, 0, 2>

…

<0, 0, 1>

<0, 0, 1>

<0, 0, 1>

… …

1

2

3

3

4

5

1

4

7

21

63

84

14

57

100

25

75

100

Global Table

39/68

Cons Hard to maintain if clusters start drifting Hard to find good number of clusters

Space/Time tradeoffs Needs training to construct good dataset

statistics table

40/68

VSol – minhash (MBKS07)

Solution based on minhash minhash is used for:

Estimate the size of a set |s| Estimate resemblance of two sets

- I.e., estimating the size of J=|s1s2| / |s1s2|

Estimate the size of the union |s1s2| Hence, estimating the size of the intersection

- |s1s2| J~(s1, s2) ~(s1, s2)

41/68

Minhash Given a set s = {t1, …, tm} Use independent hash functions h1, …, hk:

hi: n-gram [0, 1] Hash elements of s, k times Keep the k elements that hashed to the

smallest value each time We reduced set s, from m to k elements Denote minhash signature with s’

42/68

How to use minhash Given two signatures q’, s’:

J(q, s) Σ1ik I{q’[i]=s’[i]} / k |s| ( k / Σ1ik s’[i] ) - 1 (qs)’ = q’ s’ = min1ik(q’[i], s’[i]) Hence:

- |qs| (k / Σ1ik (qs)’[i]) - 1

43/68

VSol Estimator Construct one inverted list per n-gram in D

The lists are our sets Compute a minhash signature for each list

t1 t2 t10…

15

25

…

35

14

…

18

43

…Inverted list

Minhash

44/68

Selectivity Estimation Use edit distance length filter:

If ed(q, s) δ, then q and s share at least L = |s| - 1 - n (δ-1)

n-grams Given query q = {t1, …, tm}:

Answer is the size of the union of all non-empty L-intersections (binomial coefficient: m choose L)

We can estimate sizes of L-intersections using minhash signatures

45/68

Example δ = 2, n = 3 L = 6

Look at all 6-intersections of inverted lists Α = |ι1, ..., ι6 [1,10] (ti1 ti2 … ti6)| There are (10 choose 6) such terms

q = t1 t2 t10…

15

25

…

35

14

…

18

43

…Inverted list

46/68

The m-L Similarity Can be done efficiently using

minhashesAnswer:

ρ = Σ1jk I{ i1, …, iL: ti1’[j] = … = tiL’[j] } A ρ |t1… tm|

Proof very similar to the proof for minhashes

47/68

Cons Will overestimate results

Many L-intersections will share strings Edit distance length filter is loose

48/68

OptEQ – wild-card n-grams (LNS07)

Use extended n-grams: Introduce wild-card symbol ‘?’ E.g., “ab?” can be:

- “aba”, “abb”, “abc”, … Build an extended n-gram table:

Extract all 1-grams, 2-grams, …, n-grams Generalize to extended 2-grams, …, n-grams Maintain an extended n-grams/frequency

hashtable49/68

Example

stringDataset

abcdefghi…

n-gram Frequencyn-gram table

abbcdeefghhi…?ba??c…

101541212…131723…

abcdef…

52…

50/68

Query Expansion (Replacements only) Given query q=“abcd” δ=2 And replacements only:

Base strings:- “??cd”, “?b?d”, “?bc?”, “a??d”, “a?c?”, “ab??”

Query answer:- S1={sD: s ”??cd”}, S2=…- A = |S1 S2 S3 S4 S5 S6| =

Σ1n6 (-1)n-1 |S1 … Sn|51/68

Replacement Intersection Lattice

A = Σ1n6 (-1)n-1 |S1 … Sn|

Need to evaluate size of all 2-intersections, 3-intersections, …, 6-intersections

Then, use n-gram table to compute sum A Exponential number of intersections But ... there is well-defined structure

52/68

Replacement Lattice Build replacement lattice:

Many intersections are empty Others produce the same results

we need to count everything only once

??cd ?b?d ?bc? a??d a?c? ab??

?bcd a?cd ab?d abc?

abcd

2 ‘?’

1 ‘?’

0 ‘?’

53/68

General Formulas Similar reasoning for:

r replacements d deletions

Other combinations difficult: Multiple insertions Combinations of insertions/replacements

But … we can generate the corresponding lattice algorithmically! Expensive but possible

54/68

BasicEQ Partition strings by length:

Query q with length l Possible matching strings with lengths:

- [l-δ, l+δ] For k = l-δ to l+δ

- Find all combinations of i+d+r = δ and l+i-d=k- If (i,d,r) is a special case use formula- Else generate lattice incrementally:

Start from query base strings (easy to generate) Begin with 2-intersections and build from there

55/68

OptEq Details are cumbersome

Left for homework Various optimizations possible to reduce

complexity

56/68

Cons Fairly complicated implementation Expensive Works for small edit distance only

57/68

Hashed Sampling (HYKS08)

Used to estimate selectivity of TF/IDF, BM25, DICE (vector space model)

Main idea: Take a sample of the inverted index But do it intelligently to improve variance

58/68

Example

1

04 2

1 2

4

304 2

atchckicristtatituuc

23 10 1

4433

24

1

21

0

Take a sample of the inverted index

59/68

Example (Cont.)

1

0

00 1

1

133

4

4 2

atchckicristtatituuc

23

44

4 2

2

3

But do it intelligently to improve variance

1

1

1

00

01

60/68

Construction Draw samples deterministically:

Use a hash function h: N [0, 100] Keep ids that hash to values smaller than σ

Invariant: If a given id is sampled in one list, it will always be

sampled in all other lists that contain it:- S(q, s) can be computed directly from the sample- No need to store complete sets in the sample- No need for extra I/O to compute scores

61/68

Selectivity Estimation The union of arbitrary list samples is an σ% sample Given query q = {t1, …, tm}:

A = |Aσ| |t1 … tm| / |tσ1 … tσm|:- Aσ is the query answer size from the sample- The fraction is the actual scale-up factor- But there are duplicates in these unions!

We need to know:- The distinct number of ids in t1 … tm

- The distinct number of ids in tσ1 … tσm

62/68

Count Distinct Distinct |tσ1 … tσm| is easy:

Scan the sampled lists Distinct |t1 … tm| is hard:

Scanning the lists is the same as computing the exact answer to the query … naively

We are lucky:- Each list sample doubles up as a k-minimum value

estimator by construction!- We can use the list samples to estimate the distinct |t1

… tm|

63/68

The k-Minimum Value Synopsis It is used to estimated the distinct size of

arbitrary set unions (the same as FM sketch): Take hash function h: N [0, 100] Hash each element of the set The r-th smallest hash value is an unbiased

estimator of count distinct:

0 100hr

r

hr r100 ?

64/68

Outline Motivation and preliminaries Inverted list based algorithms Gram signature algorithms Length normalized algorithms Selectivity estimation Conclusion and future directions

65/68

Future Directions Result ranking

In practice need to run multiple types of searches Need to identify the “best” results

Diversity of query results Some queries have multiple meanings E.g., “Jaguar”

Updates Incremental maintenance

66/68

References [AGK06] Arvind Arasu, Venkatesh Ganti, Raghav Kaushik: Efficient Exact Set-Similarity Joins. VLDB

2006 [BJL+09] Space-Constrained Gram-Based Indexing for Efficient Approximate String Search,

Alexander Behm, Shengyue Ji, Chen Li, and Jiaheng Lu, ICDE 2009 [HCK+08] Marios Hadjieleftheriou, Amit Chandel, Nick Koudas, Divesh Srivastava: Fast Indexes and

Algorithms for Set Similarity Selection Queries. ICDE 2008 [HYK+08] Marios Hadjieleftheriou, Xiaohui Yu, Nick Koudas, Divesh Srivastava: Hashed samples:

selectivity estimators for set similarity selection queries. PVLDB 2008. [JL05] Selectivity Estimation for Fuzzy String Predicates in Large Data Sets, Liang Jin, and Chen Li.

VLDB 2005. [KSS06] Record linkage: Similarity measures and algorithms. Nick Koudas, Sunita Sarawagi, and

Divesh Srivastava. SIGMOD 2006. [LLL08] Efficient Merging and Filtering Algorithms for Approximate String Searches, Chen Li, Jiaheng

Lu, and Yiming Lu. ICDE 2008. [LNS07] Hongrae Lee, Raymond T. Ng, Kyuseok Shim: Extending Q-Grams to Estimate Selectivity of

String Matching with Low Edit Distance. VLDB 2007 [LWY07] VGRAM: Improving Performance of Approximate Queries on String Collections Using

Variable-Length Grams, Chen Li, Bin Wang, and Xiaochun Yang. VLDB 2007 [MBK+07] Arturas Mazeika, Michael H. Böhlen, Nick Koudas, Divesh Srivastava: Estimating the

selectivity of approximate string queries. ACM TODS 2007 [XWL08] Chuan Xiao, Wei Wang, Xuemin Lin: Ed-Join: an efficient algorithm for similarity joins with

edit distance constraints. PVLDB 2008

67/68

References [XWL+08] Chuan Xiao, Wei Wang, Xuemin Lin, Jeffrey Xu Yu: Efficient similarity joins for near

duplicate detection. WWW 2008. [YWL08] Cost-Based Variable-Length-Gram Selection for String Collections to Support

Approximate Queries Efficiently, Xiaochun Yang, Bin Wang, and Chen Li, SIGMOD 2008 [JLV08]L. Jin, C. Li, R. Vernica: SEPIA: Estimating Selectivities of Approximate String Predicates

in Large Databases, VLDBJ08 [CGK06] S. Chaudhuri, V. Ganti, R. Kaushik : A Primitive Operator for Similarity Joins in Data

Cleaning, ICDE06 [CCGX08]K. Chakrabarti, S. Chaudhuri, V. Ganti, D. Xin: An Efficient Filter for Approximate

Membership Checking, SIGMOD08 [SK04] Sunita Sarawagi, Alok Kirpal: Efficient set joins on similarity predicates. SIGMOD

Conference 2004: 743-754 [BK02] Jérémy Barbay, Claire Kenyon: Adaptive intersection and t-threshold problems. SODA

2002: 390-399 [CGG+05] Surajit Chaudhuri, Kris Ganjam, Venkatesh Ganti, Rahul Kapoor, Vivek R. Narasayya,

Theo Vassilakis: Data cleaning in microsoft SQL server 2005. SIGMOD Conference 2005: 918-920

68/68