Slow Search With People
description
Transcript of Slow Search With People
![Page 1: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/1.jpg)
SLOW SEARCHWITH PEOPLEJaime Teevan, Microsoft Research, @jteevanMicrosoft: Kevyn Collins-Thompson, Susan Dumais, Eric Horvitz, Adam Kalai, Ece Kamar, Dan Liebling, Merrie Morris, Ryen WhiteCollaborators: Michael Bernstein, Jin-Woo Jeong, Yubin Kim, Walter Lasecki, Rob Miller, Peter Organisciak, Katrina Panovich
![Page 2: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/2.jpg)
Slow Movements
![Page 3: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/3.jpg)
Speed Focus in Search Reasonable
![Page 4: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/4.jpg)
Not All Searches Need to Be Fast• Long-term tasks
• Long search sessions• Multi-session searches
• Social search• Question asking
• Technologically limited• Mobile devices• Limited connectivity• Search from space
![Page 5: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/5.jpg)
Making Use of Additional Time
![Page 6: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/6.jpg)
CROWDSOURCINGUsing human computation to improve search
![Page 7: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/7.jpg)
Replace Components with People• Search process
• Understand query• Retrieve • Understand results
• Machines are good at operating at scale
• People are good at understanding
with Kim, Collins-Thompson
![Page 8: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/8.jpg)
Understand Query: Query Expansion• Original query: hubble telescope achievements• Automatically identify expansion terms:
• space, star, astronomy, galaxy, solar, astro, earth, astronomer• Best expansion terms cover multiple aspects of the query
• Ask crowd to relate expansion terms to a query term
• Identify best expansion terms:• astronomer, astronomy, star
space star astronomy galaxy solar astro earth astronomer
hubble 1 1 2 1 0 0 0 1
telescope 1 2 2 0 0 0 0 1
achievements 0 0 0 0 0 0 0 1
𝑝 (𝑡𝑒𝑟𝑚 𝑗|𝑞𝑢𝑒𝑟𝑦 )= ∏𝑖∈𝑞𝑢𝑒𝑟𝑦
𝑣𝑜𝑡𝑒 𝑗 ,𝑖
∑𝑗𝑣𝑜𝑡𝑒 𝑗 , 𝑖
![Page 9: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/9.jpg)
Understand Results: Filtering• Remove irrelevant results from list
• Ask crowd workers to vote on relevance
• Example: • hubble telescope
achievements
![Page 10: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/10.jpg)
People Are Not Good Components• Test corpora
• Difficult Web queries• TREC Web Track queries
• Query expansion generally ineffective• Query filtering
• Improves quality slightly• Improves robustness
• Not worth the time and cost• Need to use people in new ways
![Page 11: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/11.jpg)
Understand Query: Identify Entities• Search engines do poorly with long, complex queries• Query: Italian restaurant in Squirrel Hill or Greenfield with
a gluten-free menu and a fairly sophisticated atmosphere• Crowd workers identify important attributes
• Given list of potential attributes• Option add new attributes• Example: cuisine, location, special diet, atmosphere
• Crowd workers match attributes to query• Attributes used to issue a structured search
with Kim, Collins-Thompson
![Page 12: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/12.jpg)
Understand Results: Tabulate• Crowd workers used to tabulate search results
• Given a query, result, attribute and value• Does the result meet the attribute?
![Page 13: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/13.jpg)
People Can Provide Rich Input• Test corpus: Complex restaurant queries to Yelp• Query understanding improves results
• Particularly for ambiguous or unconventional attributes• Strong preference for the tabulated results
• People who liked traditional results valued familiarity• People asked for additional columns (e.g., star rating)
![Page 14: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/14.jpg)
Create Answers from Search Results
• Understand query• Use log analysis to expand query to related queries• Ask crowd if the query has an answer
• Retrieve: Identify a page with the answer via log analysis• Understand results: Extract, format, and edit an answer
with Bernstein, Dumais, Liebling, Horvitz
![Page 15: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/15.jpg)
Create Answers to Social Queries
• Understand query: Use crowd to identify questions• Retrieve: Crowd generates a response• Understand results: Vote on answers from crowd, friends
with Jeong, Morris, Liebling
![Page 16: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/16.jpg)
PROS & CONS OF THE CROWDOpportunities and challenges of crowdsourcing search
![Page 17: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/17.jpg)
Personalization with the Crowd
?
with Organisciak, Kalai, Dumais, Miller
![Page 18: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/18.jpg)
Matching Workers versus Guessing• Matching workers
• Requires many workers to find a good match
• Easy for workers• Data reusable
• Guessing• Requires fewer workers• Fun for workers• Hard to capture complex
preferences
Rand. Match Guess
Salt shakers 1.64 1.43 1.07
Food (Boston) 1.51 1.19 1.38
Food (Seattle) 1.68 1.26 1.28
(RMSE for 5 workers)
![Page 19: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/19.jpg)
![Page 20: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/20.jpg)
Extraction and Manipulation Threats
with Lasecki, Kamar
![Page 21: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/21.jpg)
Information Extraction• Target task: Text recognition
• Attack task• Complete target task• Return answer from target:
1234 5678 9123 4567
1234 5678 9123 4567
62.1% 32.8%
![Page 22: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/22.jpg)
gun (36%), fun (26%), sun (12%)
Task Manipulation• Target task: Text recognition
• Attack task• Enter “sun” as the answer for the attack task
sun (75%) sun (28%)
![Page 23: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/23.jpg)
FRIENDSOURCINGUsing friends as a resource during the search process
![Page 24: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/24.jpg)
Searching versus Asking
![Page 25: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/25.jpg)
Searching versus Asking• Friends respond quickly
• 58% of questions answered by the end of search• Almost all answered by the end of the day
• Some answers confirmed search findings• But many provided new information
• Information not available online• Information not actively sought• Social content
with Morris, Panovich
![Page 26: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/26.jpg)
Shaping the Replies from Friends
Should I watch E.T.?
![Page 27: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/27.jpg)
Shaping the Replies from Friends• Larger networks provide better replies• Faster replies in the morning, more in the evening• Question phrasing important
• Include question mark• Target the question at a group (even at anyone)• Be brief (although context changes nature of replies)
• Early replies shape future replies• Opportunity for friends and algorithms to collaborate to find the best content
with Morris, Panovich
![Page 28: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/28.jpg)
Summary
![Page 29: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/29.jpg)
Further Reading in Slow Search• Slow search
• Teevan, J., Collins-Thompson, K., White, R., Dumais, S.T. & Kim, Y. Slow Search: Information Retrieval without Time Constraints. HCIR 2013.
• Teevan, J., Collins-Thompson, K., White, R. & Dumais, S.T. Slow Search. CACM 2014 (to appear).
• Crowdsourcing• Jeong, J.W., Morris, M.R., Teevan, J. & Liebling, D. A Crowd-Powered Socially Embedded Search
Engine. ICWSM 2013.• Bernstein, M., Teevan, J., Dumais, S.T., Libeling, D. & Horvitz, E. Direct Answers for Search Queries in
the Long Tail. CHI 2012.
• Pros and cons of the crowd• Lasecki, W., Teevan, J., & Kamar, E. Information Extraction and Manipulation Threats in Crowd-
Powered Systems. CSCW 2014.• Organisciak, P., Teevan, J., Dumais, S.T., Miller, R.C. & Kalai, A.T. Personalized Human Computation.
HCOMP 2013.
• Friendsourcing• M.R. Morris, J. Teevan & K. Panovich. A Comparison of Information Seeking Using Search Engines
and Social Networks. ICWSM 2010.• J. Teevan, M.R. Morris & K. Panovich. Factors Affecting Response Quantity, Quality and Speed in
Questions Asked via Online Social Networks. ICWSM 2011.
![Page 30: Slow Search With People](https://reader035.fdocuments.in/reader035/viewer/2022062323/568165fe550346895dd92e0e/html5/thumbnails/30.jpg)
QUESTIONS?Slow Search with PeopleJaime Teevan, Microsoft Research, @jteevan