Automatic Search Engine Evaluation with Click-through Data
Transcript of Automatic Search Engine Evaluation with Click-through Data
![Page 1: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/1.jpg)
Automatic Search Engine Evaluation Automatic Search Engine Evaluation with Clickwith Click--through Data Analysisthrough Data Analysis
YiqunYiqun LiuLiuState Key Lab of Intelligent Tech. & SysState Key Lab of Intelligent Tech. & Sys
Jun. 3th, 2007Jun. 3th, 2007
![Page 2: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/2.jpg)
Recent work:• Using query log and click-through data analysis to:
• identify search engine users’ information need types• evaluate search engine performance automatically• separate key resource pages from others • estimate Web page quality
Our Lab:• A joint lab• R&D Support to a widely-used Chinese Search Engine Sogou.com, platform to get research results realized.
Yiqun LiuPh.D student from Tsinghua University, Beijing, [email protected]
![Page 3: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/3.jpg)
Yiqun LiuPh.D student from Tsinghua University, Beijing, [email protected]
• Web Data Cleansing• Using query-Independent features and ML algorithms• 5% web pages can meet >90% user’s search needs
• Query type identification• Identify the type of user’s information need• Over 80% queries are correctly classified
• Search engine performance evaluation• Construct query topic set and answer set Automatically .• Obtain similar evaluation results with manual based methods, and cost far less time and labor.
![Page 4: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/4.jpg)
IntroductionIntroduction
• Lots of search engines offer services on the Web
• Search Engine Performance Evaluation– Web Users
• over 120 million users in mainland
– Search Advertisers• spending 5.6 billion RMBs in 2007
– Search engineers and researchers
![Page 5: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/5.jpg)
IntroductionIntroduction
• Evaluation is a key issue in IR research– Evaluation became central to R&D in IR to such an
extent that new designs and proposals and their evaluation became one. (Saracevic, 1995)
• Cranfield-like evaluation methodology– Proposed by Cleverdon et al in 1966.– A set of query topics, their corresponding answers
(usually called qrels) and evaluation metrics.– Adopted by IR workshops such
as TREC and NTCIR.
![Page 6: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/6.jpg)
IntroductionIntroduction• Problems with Web IR evaluation
– 9 people months are required to judge one topic for a collection of 8 million documents. (Voorhees, 2001)
– Search engines (Yahoo!, Google) index over 10 billion Web documents.
– Almost Impossible to use human-assessed query and qrel sets in Web IR system evaluation.
![Page 7: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/7.jpg)
Related worksRelated works• Efforts in automatic search engine
performance evaluation (Cranfield-like)– Considering pseudo feedback documents as
correct answers(Soboroff, 2001; Nuray, 2003)
– Adopting query topics and qrels extracted from Web page directories such as open directory project (ODP)(Chowdhury, 2002; Beitzel, 2003)
![Page 8: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/8.jpg)
Related worksRelated works
• Efforts in automatic search engine performance evaluation (other evaluation approaches)– Term Relevance Sets (Trels) method.
Define a pre-specified list of terms relevant and irrelevant to these queries. (Amitay, 2004)
– The use of click-through data.Construct a unified meta search interface to collect users’ behaviour information. (Joachims, 2002)
![Page 9: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/9.jpg)
Our methodOur method
• A cranfield-like approach– Accepted by major IR research efforts– Difficulty: annotating all correct answers
automatically
• Click-through behavior analysis– Single user may be cheated by search spams
or SEOs.– User group’s behavior
information is more reliable.
![Page 10: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/10.jpg)
Automatic Evaluation Process
• Information need behind user queries– Proposed by Broder (2003) – Navigational type:
One query have only one correct answer. – Informational type:
One query may have several correct answers.
• Different behavior over different types of information needs
![Page 11: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/11.jpg)
Information needs and EvaluationInformation needs and Evaluation
• Informational queries cannot be annotated– People click different answers while using
different search engines.
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
baidu google yahoo sogou
![Page 12: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/12.jpg)
Automatic Evaluation Process
![Page 13: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/13.jpg)
Query Set Classification
• Less Effort Assumption & N Clicks Satisfied (nCS) Evidence– While performing a navigational type search request,
user tend to click a small number of URLs in the result list.
0%
10%
20%
30%
40%
50%
60%
0.9 0.8 0.7 0.6 0.5 0.4
Navigational Informational & Transactional
![Page 14: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/14.jpg)
Query Set Classification
• Cover Page Assumption and Top N Results Satisfied (nRS) Evidence– While performing a navigational type search request,
user tend to click only the first few URLs in the result list.
0%
10%
20%
30%
40%
50%
60%
0.9 0.8 0.7 0.6 0.5
Navigational Informational & Transactional
![Page 15: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/15.jpg)
Query Set Classification
• Click Distribution Evidence– Proposed by Lee (Lee, 2005). Also based on click-
through information.– Users tend to click the same result while proposing a
same navigational type query
– Less than 5% informational / Transactional queries’CD value is over ½, while 51% navigational queries’corresponding value is more than ½.
![Page 16: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/16.jpg)
Query Set Classification
• A decision tree algorithm
![Page 17: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/17.jpg)
Answer AnnotationAnswer Annotation
• Navigational type query annotation– Define: Click focus
– Annotate q with the result r whose ClickFocusvalue is the largest.
)(#)(#
),(qofSession
rclicksthatqofSessionrResultqQueryClickFocus =
![Page 18: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/18.jpg)
Answer AnnotationAnswer Annotation
• Annotation AlgorithmFor a given Query Q in the Query Set and its clicked result list r1, r2, …, rM : IF Q is navigational
Find R in r1, r2, …, rM, ClickFocus(Q,R) = ClickDistribution(Q); IF CD(Q) > T1
Annotate Q with R;EXIT;
ELSEQ cannot be annotated;
END IFELSE //Q is informational
Q cannot be annotated;END IF
![Page 19: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/19.jpg)
Experiment Results
• Experiment data– Collected by Sogou.com from Jun 2006 to Jan 2007.– Over 700 million querying or clicking events totally.
• Annotation experiment results– 5% of all results are checked mannually.
#(Annotated queries)
#(Checked sample set) Accuracy
Jun. 06 - Aug. 06 13,902 695 98.13%
Sept.06 - Nov. 06 13,884 694 97.41%
Dec. 06 - Jan. 07 11,296 565 96.64%
![Page 20: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/20.jpg)
Experiment Results
• Performance evaluation experiment– 320 manual-developed queries and corresponding
answers are used in the evaluation experiment.– Correlation value between MRRs of the manual and
the automatically methods is 0.965.
![Page 21: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/21.jpg)
Applications and Future worksApplications and Future works
• Choosing the correct search portal– Overall performance – Performance for queries in a certain field
• Search engine monitoring– Complicated computer cluster systems are
used in modern search engines– To notify the engineers when
the search engine fails. (performance going down)
![Page 22: Automatic Search Engine Evaluation with Click-through Data](https://reader031.fdocuments.in/reader031/viewer/2022021307/620748ba49d709492c2fde3d/html5/thumbnails/22.jpg)
Thank you!
Questions or comments?