Web Information Retrieval
description
Transcript of Web Information Retrieval
![Page 1: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/1.jpg)
1
Web Information Retrieval
Web Science Course
![Page 2: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/2.jpg)
2
![Page 3: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/3.jpg)
What to Expect
• Information Retrieval Basics– IR Systems– History of IR
• Retrieval Models– Vector Space Model
• Information Retrieval on the Web– Differences to traditional IR
• Selected Papers
3
![Page 4: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/4.jpg)
4
Information Retrieval Basics
![Page 5: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/5.jpg)
5
Information Retrieval (IR)
• The indexing and retrieval of textual documents.
• Concerned firstly with retrieving relevant documents to a query.
• Concerned secondly with retrieving from large sets of documents efficiently.
![Page 6: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/6.jpg)
6
Typical IR Task
• Given:– A corpus of textual natural-language
documents.– A user query in the form of a textual string.
• Find:– A ranked set of documents that are relevant to
the query.
![Page 7: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/7.jpg)
7
IR System
IRSystem
Query String
Documentcorpus
RankedDocuments
1. Doc12. Doc23. Doc3 . .
![Page 8: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/8.jpg)
8
Relevance
• Relevance is a subjective judgment and may include:– Being on the proper subject.– Being timely (recent information).– Being authoritative (from a trusted source).– Satisfying the goals of the user and his/her
intended use of the information (information need).
![Page 9: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/9.jpg)
9
Keyword Search
• Simplest notion of relevance is that the query string appears verbatim in the document.
• Slightly less strict notion is that the words in the query appear frequently in the document, in any order (bag of words).
![Page 10: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/10.jpg)
10
Problems with Keywords
• May not retrieve relevant documents that include synonymous terms.– “restaurant” vs. “café”– “PRC” vs. “China”
• May retrieve irrelevant documents that include ambiguous terms.– “bat” (baseball vs. mammal)– “Apple” (company vs. fruit)– “bit” (unit of data vs. act of eating)
![Page 11: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/11.jpg)
11
Intelligent IR
• Taking into account the meaning of the words used.
• Taking into account the order of words in the query.
• Adapting to the user based on direct or indirect feedback.
• Taking into account the authority of the source.
![Page 12: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/12.jpg)
12
IR System Architecture
TextDatabase
DatabaseManagerIndexing
Index
QueryOperations
Searching
RankingRankedDocs
UserFeedback
Text Operations
User Interface
RetrievedDocs
UserNeed
Text
Query
Logical View
Inverted file
![Page 13: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/13.jpg)
13
IR System Components
• Text Operations forms index words (tokens).– Stopword removal– Stemming
• Indexing constructs an inverted index of word to document pointers.
• Searching retrieves documents that contain a given query token from the inverted index.
• Ranking scores all retrieved documents according to a relevance metric.
![Page 14: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/14.jpg)
14
IR System Components (continued)
• User Interface manages interaction with the user:– Query input and document output.– Relevance feedback.– Visualization of results.
• Query Operations transform the query to improve retrieval:– Query expansion using a thesaurus.– Query transformation using relevance feedback.
![Page 15: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/15.jpg)
15
History of IR
• 1960-70’s:– Initial exploration of text retrieval systems for
“small” corpora of scientific abstracts, and law and business documents.
• 1980’s:– Large document database systems, many run by
companies• 1990’s:
– Searching FTPable documents on the Internet– Searching the World Wide Web
![Page 16: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/16.jpg)
16
Recent IR History
• 2000’s– Link analysis for Web Search– Automated Information Extraction– Question Answering– Multimedia IR– Cross-Language IR– Document Summarization
![Page 17: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/17.jpg)
17
Vector Space Retrieval Model
![Page 18: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/18.jpg)
18
Retrieval Models
• A retrieval model specifies the details of:–Document representation–Query representation–Retrieval function
• Determines a notion of relevance.• Notion of relevance can be binary or
continuous (i.e. ranked retrieval).
![Page 19: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/19.jpg)
19
Preprocessing Steps• Strip unwanted characters/markup (e.g.
HTML tags, punctuation, numbers, etc.).• Break into tokens (keywords) on
whitespace.• Stem tokens to “root” words
– computational comput• Remove common stopwords (e.g. a, the, it,
etc.).• Build inverted index (keyword list of
docs containing it).
![Page 20: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/20.jpg)
20
The Vector-Space Model
• Assume t distinct terms after preprocessing; call them index terms or the vocabulary.
• These “orthogonal” terms form a vector space. Dimension = t = |vocabulary|
• Each term, i, in a document or query, j, is given a real-valued weight, wij.
• Both documents and queries are expressed as t-dimensional vectors: dj = (w1j, w2j, …, wtj)
![Page 21: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/21.jpg)
21
Graphic Representation
Example:D1 = 2T1 + 3T2 + 5T3
D2 = 3T1 + 7T2 + T3
Q = 0T1 + 0T2 + 2T3
T3
T1
T2
D1 = 2T1+ 3T2 + 5T3
D2 = 3T1 + 7T2 + T3
Q = 0T1 + 0T2 + 2T3
7
32
5
• Is D1 or D2 more similar to Q?• How to measure the degree of
similarity? Distance? Angle? Projection?
![Page 22: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/22.jpg)
22
Term Weights: Term Frequency
• More frequent terms in a document are more important, i.e. more indicative of the topic. fij = frequency of term i in document j
• May want to normalize term frequency (tf) by dividing by the frequency of the most common term in the document: tfij = fij / maxi{fij}
![Page 23: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/23.jpg)
23
Term Weights: Inverse Document Frequency
• Terms that appear in many different documents are less indicative of overall topic.
df i = document frequency of term i = number of documents containing term i idfi = inverse document frequency of term i, = log2 (N/ df i) (N: total number of documents)• An indication of a term’s discrimination power.• Log used to dampen the effect relative to tf.
![Page 24: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/24.jpg)
24
TF-IDF Weighting• A typical combined term importance
indicator is tf-idf weighting:wij = tfij idfi = tfij log2 (N/ dfi)
• A term occurring frequently in the document but rarely in the rest of the collection is given high weight.
• Many other ways of determining term weights have been proposed.
• Experimentally, tf-idf has been found to work well.
![Page 25: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/25.jpg)
25
Computing TF-IDF -- An Example
Given a document containing terms with given frequencies: A(3), B(2), C(1)Assume collection contains 10,000 documents and document frequencies of these terms are: A(50), B(1300), C(250)Then:A: tf = 3/3; idf = log2(10000/50) = 7.6; tf-idf = 7.6B: tf = 2/3; idf = log2 (10000/1300) = 2.9; tf-idf = 2.0C: tf = 1/3; idf = log2 (10000/250) = 5.3; tf-idf = 1.8
![Page 26: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/26.jpg)
26
Query Vector
• Query vector is typically treated as a document and also tf-idf weighted.
• Alternative is for the user to supply weights for the given query terms.
![Page 27: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/27.jpg)
27
Similarity Measure
• A similarity measure is a function that computes the degree of similarity between two vectors.
• Using a similarity measure between the query and each document:– It is possible to rank the retrieved documents in
the order of presumed relevance.– It is possible to enforce a certain threshold so
that the size of the retrieved set can be controlled.
![Page 28: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/28.jpg)
28
Cosine Similarity Measure• Cosine similarity measures the cosine of
the angle between two vectors.• Inner product normalized by the vector
lengths.
D1 = 2T1 + 3T2 + 5T3 CosSim(D1 , Q) = 10 / (4+9+25)(0+0+4) = 0.81D2 = 3T1 + 7T2 + 1T3 CosSim(D2 , Q) = 2 / (9+49+1)(0+0+4) = 0.13
Q = 0T1 + 0T2 + 2T3
2
t3
t1
t2
D1
D2
Q
1
t
i
t
i
t
i
ww
ww
qdqd
iqij
iqij
j
j
1 1
22
1
)(
CosSim(dj, q) =
![Page 29: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/29.jpg)
29
Naïve ImplementationConvert all documents in collection D to tf-idf
weighted vectors, dj, for keyword vocabulary V.Convert query to a tf-idf-weighted vector q.For each dj in D do Compute score sj = cosSim(dj, q)Sort documents by decreasing score.Present top ranked documents to the user.
Time complexity: O(|V|·|D|) Bad for large V & D !|V| = 10,000; |D| = 100,000; |V|·|D| = 1,000,000,000
![Page 30: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/30.jpg)
Inverted Index
30
![Page 31: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/31.jpg)
31
Comments on Vector Space Models
• Simple, mathematically based approach. • Considers both local (tf) and global (idf)
word occurrence frequencies.• Provides partial matching and ranked
results.• Tends to work quite well in practice despite
obvious weaknesses.• Allows efficient implementation for large
document collections.• Does not require all terms in the query
![Page 32: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/32.jpg)
32
Web Search
![Page 33: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/33.jpg)
33
Web Search
• Application of IR to HTML documents on the World Wide Web.
• Differences:– Must assemble document corpus by spidering
the web.– Can exploit the structural layout information
in HTML (XML).– Documents change uncontrollably.– Can exploit the link structure of the web.
![Page 34: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/34.jpg)
34
Web Search Using IR
Query String
IRSystem
RankedDocuments
1. Page12. Page23. Page3
. .
Documentcorpus
Web Spider
![Page 35: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/35.jpg)
35
The World Wide Web
• Developed by Tim Berners-Lee in 1990 at CERN to organize research documents available on the Internet.
• Combined idea of documents available by FTP with the idea of hypertext to link documents.
• Developed initial HTTP network protocol, URLs, HTML, and first “web server.”
![Page 36: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/36.jpg)
36
Web Search Recent History
• In 1998, Larry Page and Sergey Brin, Ph.D. students at Stanford, started Google. Main advance is use of link analysis to rank results partially based on authority.
![Page 37: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/37.jpg)
37
Web Challenges for IR• Distributed Data: Documents spread over millions of
different web servers.• Volatile Data: Many documents change or disappear
rapidly (e.g. dead links).• Large Volume: Billions of separate documents.• Unstructured and Redundant Data: No uniform
structure, HTML errors, up to 30% (near) duplicate documents.
• Quality of Data: No editorial control, false information, poor quality writing, typos, etc.
• Heterogeneous Data: Multiple media types (images, video, VRML), languages, character sets, etc.
![Page 38: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/38.jpg)
38
Growth of Web Pages Indexed
SearchEngineWatch Link to Note from Jan 2004
Assuming 20KB per page, 1 billion pages is about 20 terabytes of data.
Billi
ons
of P
ages
GoogleInktomi
AllTheWeb
TeomaAltavista
![Page 39: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/39.jpg)
39
Graph Structure in the Web
http://www9.org/w9cdrom/160/160.html
![Page 40: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/40.jpg)
Selected Papers
40
![Page 41: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/41.jpg)
1. A Taxonomy of Web Search
• Andrei Broder, 2002• Query log analysis & user survey• Classify web queries according to their
intent into 3 classes– Navigational– Informational– Transactional
• How global search engines evolved to deal with web-specific needs
41
![Page 42: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/42.jpg)
2. Personalizing Search via Automated Analysis of Interests and Activities
• Jaime Teevan, Susan Dumais, Eric Horvitz, 2005
• Formulate and study search personalization algorithms
• Relevance feedback framework• Rich models of user interests built from
– Previously issued queries – Previously visited Web pages – Documents and emails the user has read and
created42
![Page 43: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/43.jpg)
3. Personalized Query Expansion for the Web
• Paul Chirita, Claudiu Firan, Wolfgang Nejdl, 2007
• Improve Web queries by expanding them • Five broad techniques for generating the
additional query keywords – Term and compound level analysis – Global co-occurrence statistics– Use external thesauri
43
![Page 44: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/44.jpg)
4. Boilerplate Detection using Shallow Text Features
• Christian Kohlschütter, Peter Fankhauser, Wolfgang Nejdl, 2010
• Boilerplate text typically is not related to the main content
• Analyze a small set of shallow text features for classifying the individual text elements in a Web page
• Test impact of boilerplate removal to retrieval performance
44
![Page 45: Web Information Retrieval](https://reader035.fdocuments.in/reader035/viewer/2022062814/5681685c550346895dde94e1/html5/thumbnails/45.jpg)
For You to Choose:
1. A Taxonomy of Web Search2. Personalizing Search via Automated
Analysis of Interests and Activities3. Personalized Query Expansion
for the Web4. Boilerplate Detection using Shallow Text
Features
45