Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa...
-
Upload
francis-davis -
Category
Documents
-
view
218 -
download
1
Transcript of Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa...
![Page 1: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/1.jpg)
Query processing:optimizations
Paolo FerraginaDipartimento di Informatica
Università di Pisa
Reading 2.3
![Page 2: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/2.jpg)
Augment postings with skip pointers (at indexing time)
How do we deploy them ? Where do we place them ?
1282 4 8 41 48 64
311 2 3 8 11 17 21
3111
41 128
Sec. 2.3
![Page 3: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/3.jpg)
Using skips
1282 4 8 41 48 64
311 2 3 8 11 17 21
3111
41 128
Suppose we’ve stepped through the lists until we process 8 on each list. We match it and advance.
We then have 41 and 11 on the lower. 11 is smaller.
But the skip successor of 11 on the lower list is 31, sowe can skip ahead past the intervening postings.
Sec. 2.3
![Page 4: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/4.jpg)
Placing skips
Tradeoff: More skips shorter skip spans more
likely to skip. But lots of comparisons to skip pointers.
Fewer skips few pointer comparison, but then long skip spans few successful skips.
Sec. 2.3
![Page 5: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/5.jpg)
Placing skips
Simple heuristic: for postings of length L, use L evenly-spaced skip pointers.
This ignores the distribution of query terms. Easy if the index is relatively static; harder if
L keeps changing because of updates.
This definitely used to help; with modern hardware it may not unless you’re memory-based The I/O cost of loading a bigger postings list
can outweigh the gains from quicker in memory merging!
Sec. 2.3
![Page 6: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/6.jpg)
Faster query = caching?Two opposite approaches:
I. Cache the query results (exploits query locality)
II. Cache pages of posting lists (exploits term
locality)
![Page 7: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/7.jpg)
Query processing:phrase queries and positional
indexes
Paolo FerraginaDipartimento di Informatica
Università di Pisa
Reading 2.4
![Page 8: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/8.jpg)
Phrase queries
Want to be able to answer queries such as “stanford university” – as a phrase
Thus the sentence “I went at Stanford my university” is not a match.
Sec. 2.4
![Page 9: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/9.jpg)
Solution #1: Biword indexes
For example the text “Friends, Romans, Countrymen” would generate the biwords friends romans romans countrymen
Each of these biwords is now an entry in the dictionary
Two-word phrase query-processing is immediate.
Sec. 2.4.1
![Page 10: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/10.jpg)
Longer phrase queries
Longer phrases are processed by reducing them to bi-word queries in AND
stanford university palo alto can be broken into the Boolean query on biwords, such as
stanford university AND university palo AND palo alto
Need the docs to verify+They are combined with other solutions
Can have false positives!Index blows up
Sec. 2.4.1
![Page 11: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/11.jpg)
Solution #2: Positional indexes
In the postings, store for each term and document the position(s) in which that term occurs:
<term, number of docs containing term;doc1: position1, position2 … ;doc2: position1, position2 … ;etc.>
Sec. 2.4.2
![Page 12: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/12.jpg)
Processing a phrase query
“to be or not to be”. to:
2:1,17,74,222,551; 4:8,16,190,429,433; 7:13,23,191; ...
be: 1:17,19; 4:17,191,291,430,434;
5:14,19,101; ...
Same general method for proximity searches
Sec. 2.4.2
![Page 13: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/13.jpg)
Query term proximity
Free text queries: just a set of terms typed into the query box – common on the web
Users prefer docs in which query terms occur within close proximity of each other
Would like scoring function to take this into account – how?
Sec. 7.2.2
![Page 14: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/14.jpg)
Positional index size
You can compress position values/offsets Nevertheless, a positional index expands
postings storage by a factor 2-4 in English
Nevertheless, a positional index is now commonly used because of the power and usefulness of phrase and proximity queries … whether used explicitly or implicitly in a ranking retrieval system.
Sec. 2.4.2
![Page 15: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/15.jpg)
Combination schemes
BiWord + Positional index is a profitable combination
Biword is particularly useful for particular phrases (“Michael Jackson”, “Britney Spears”)
More complicated mixing strategies do exist!
Sec. 2.4.3
![Page 16: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/16.jpg)
Soft-AND
E.g. query rising interest rates
Run the query as a phrase query
If <K docs contain the phrase rising interest
rates, run the two phrase queries rising
interest and interest rates
If we still have <K docs, run the “vector space
query” rising interest rates (…see next…)
“Rank” the matching docs (…see next…)
Sec. 7.2.3
![Page 17: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/17.jpg)
Query processing:other sophisticated queries
Paolo FerraginaDipartimento di Informatica
Università di Pisa
Reading 3.2 and 3.3
![Page 18: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/18.jpg)
Wild-card queries: *
mon*: find all docs containing words beginning with “mon”. Use a Prefix-search data structure
*mon: find words ending in “mon” Maintain a prefix-search data structure for
reverse terms.
How can we solve the wild-card query pro*cent ?
Sec. 3.2
![Page 19: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/19.jpg)
What about * in the middle?
co*tion We could look up co* AND *tion and
intersect the two lists Expensive
se*ate AND fil*erThis may result in many Boolean AND queries.
The solution: transform wild-card queries so that the *’s occur at the end
This gives rise to the Permuterm Index.
Sec. 3.2
![Page 20: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/20.jpg)
Permuterm index
For term hello, index under: hello$, ello$h, llo$he, lo$hel, o$hell,
$hellowhere $ is a special symbol.
Queries: X lookup on X$ X* lookup on $X* *X lookup on X$* *X* lookup on X* X*Y lookup on Y$X* X*Y*Z ??? Exercise!
Sec. 3.2.1
![Page 21: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/21.jpg)
Permuterm query processing
Rotate query wild-card to the right P*Q Q$P*
Now use prefix-search data structure Permuterm problem: ≈ 4x lexicon size
Empirical observation for English.
Sec. 3.2.1
![Page 22: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/22.jpg)
K-gram indexes
The k-gram index finds terms based on a query consisting of k-grams (here k=2).
mo
on
among
$m mace
among
amortize
madden
arond
Sec. 3.2.2
![Page 23: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/23.jpg)
K-gram for wild-cards queries
Query mon* can now be run as
$m AND mo AND on
Gets terms that match AND version of our
wildcard query.
Must post-filter these terms against query.
Sec. 3.2.2
![Page 24: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/24.jpg)
Isolated word correction
Given a lexicon and a character sequence Q, return the words in the lexicon closest to Q
What’s “closest”? Edit distance (Levenshtein distance) Weighted edit distance n-gram overlap
Useful in query-mispellings
Sec. 3.3.2
![Page 25: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/25.jpg)
Edit distance
Given two strings S1 and S2, the minimum number of operations to convert one to the other
Operations are typically character-level Insert, Delete, Replace, (Transposition)
E.g., the edit distance from dof to dog is 1 From cat to act is 2 (Just 1 with transpose.) from cat to dog is 3.
Generally found by dynamic programming.
Sec. 3.3.3
![Page 26: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/26.jpg)
Let E(i,j) = edit distance between T1,j and P1,i.
DynProg for Edit Distance
E(i,0)=E(0,i)=i
E(i, j) = E(i–1, j–1) if Pi=Tj
E(i, j) = min{E(i, j–1),
E(i–1, j),
E(i–1, j–1)}+1 if Pi Tj
![Page 27: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/27.jpg)
Example
T0 1 2 3 4 5 6
p t t a p a
P
0 0 1 2 3 4 5 61 p 1 0 1 2 3 4 52 a 2 1 1 2 2 3 43 t 3 2 1 1 2 3 44 t 4 3 2 1 2 3 4
![Page 28: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/28.jpg)
Weighted edit distance
As above, but the weight of an operation depends on the character(s) involved Meant to capture OCR or keyboard errors,
e.g. m more likely to be mis-typed as n than as q
Therefore, replacing m by n is a smaller edit distance than by q
Requires weight matrix as input Modify dynamic programming to handle
weights
Sec. 3.3.3
![Page 29: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/29.jpg)
k-gram overlap for Edit Distance
Enumerate all the k-grams in the query string as well as in the lexicon
Use the k-gram index (recall wild-card search) to retrieve all lexicon terms matching any of the query k-grams
Threshold by number of matching k-grams If the term is L chars long If E is the number of allowed errors (E*k, k-grams are killed) At least (L-k+1) – E*k of the query k-grams must match a
dictionary term to be a candidate answer.
Sec. 3.3.4
![Page 30: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/30.jpg)
Context-sensitive spell correction
Text: I flew from Heathrow to Narita. Consider the phrase query “flew form
Heathrow” We’d like to respond
Did you mean “flew from Heathrow ”?
because no docs matched the query phrase.
Sec. 3.3.5
![Page 31: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/31.jpg)
Zone indexes
Paolo FerraginaDipartimento di Informatica
Università di Pisa
Reading 6.1
![Page 32: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/32.jpg)
Parametric and zone indexes
Thus far, a doc has been a term sequence
But documents have multiple parts: Author Title Date of publication Language Format etc.
These are the metadata about a document
Sec. 6.1
![Page 33: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/33.jpg)
Zone
A zone is a region of the doc that can contain an arbitrary amount of text e.g., Title Abstract References …
Build inverted indexes on fields AND zones to permit querying
E.g., “find docs with merchant in the title zone and matching the query gentle rain”
Sec. 6.1
![Page 34: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/34.jpg)
Example zone indexes
Encode zones in dictionary vs. postings.
Sec. 6.1
![Page 35: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/35.jpg)
Tiered indexes
Break postings up into a hierarchy of lists Most important … Least important
Inverted index thus broken up into tiers of decreasing importance
At query time use top tier unless it fails to yield K docs If so drop to lower tiers
Sec. 7.2.1
![Page 36: Query processing: optimizations Paolo Ferragina Dipartimento di Informatica Università di Pisa Reading 2.3.](https://reader036.fdocuments.in/reader036/viewer/2022062423/56649e9d5503460f94b9d988/html5/thumbnails/36.jpg)
Example tiered index
Sec. 7.2.1