Torsten Suel Associate Professor CSE Department Polytechnic Institute of NYU suel@poly
description
Transcript of Torsten Suel Associate Professor CSE Department Polytechnic Institute of NYU suel@poly
Databases
Computer SecuritySoftware Engineering
Computer Graphics
NetworkingDistributed Systems
Web Search Engines:Technologies, Applications, and Opportunities
Torsten SuelAssociate ProfessorCSE DepartmentPolytechnic Institute of [email protected]
What this talk is about: - web search engines - how they work - underlying technologies - applications and impact - what is next? - opportunities and education
The Basics: • type in some words• get back results (usually 10 at a time)• hopefully good results among these• if not, change your query, try again
The Basics: • type in some words• get back results (usually 10 at a time)• hopefully good results among these• if not, change your query, try again
• as the web grew, we needed a way to find sites
• has grown into many-billion-$ industry at heart of the web
• search engines: largest supercomputers of the world
• 100’s of millions of queries per day, 0.1 sec latency/query
• evaluated over tens of billions of documents
• the majors: Google, MSFT Bing, Yahoo!(?), Baidu, Yandex
• but many other businesses use similar technologies, or rely on search engines for their business
Web Search Technology
• 1. Search engines in a nutshell: - How does the web work? - How do search engines work? - Basic search architecture - Historical background
• 2. Technical challenges and opportunities: - link analysis - computational advertising
•3. Education and opportunities - the search landscape - search engines and the curriculum
Overview of this Lecture:
1. Search Engines in a Nutshell
The Web:
text …
A lot of text …
>100 billionpages of text
and other stuff …
• pages containing (fairly unstructured) text
• images, audio, etc. embedded in (hanging off) pages
• structure defined using HTML (Hypertext Markup Language)
• hyperlinks between pages!
• over 100 billion pages
• over 3 trillion hyperlinks
a giant graph!
What is the web? (another view)
• pages reside in servers• sites often contain related pages• site/host structure• local versus global links
How the web is organized: site structure
Web Server (Host)
Web Server (Host)
Web Server (Host)
www.poly.edu
www.cnn.com
www.irs.gov
How Browsing Works
Desktop(with browser)
give me the file “/world/index.html”
here is the file: “...”
Web Server
www.cnn.com
Fetching “www.cnn.com/world/index.html”
HTTP:
desktop or crawler
web server
GET /world/index.html HTTP/1.0User-Agent: Mozilla/3.0 (Windows 95/NT)Host: www.cnn.comFrom: …Referer: …If-Modified-Since: ...
HTTP/1.0 200 OKServer: Netscape-Communications/1.1Date: Tuesday, 8-Feb-99 01:22:04 GMTLast-modified: Thursday, 3-Feb-99 10:44:11 GMTContent-length: 5462Content-type: text/html
<the html file>
Basic structure of a search engine:
Crawler
disks
Index
mining &indexing
Search.comQuery: “computer”
look up
Crawling
Crawler
disks
• crawler: also called spider, web robot• fetches pages from the web• starts at set of “seed pages”• parses fetched pages for hyperlinks• then follows those links (e.g., BFS)• until all pages fetches (i.e., never)
Data Mining:
• data fetched by the crawler is analyzed• many different tasks: - link analysis (later) - detection of spam pages and dangerous pages - analyzing data about past queries (clicks etc.) - data extraction (products, people, locations, …)
• mining becoming increasingly important• needs experts in statistics & machine learning• large-scale mining platforms: mapReduce (Google), Hadoop, Pig (Yahoo!), Dryad (MSFT), …
Indexing
disks
• parse & build lexicon & build index• index very large
I/O-efficient techniques needed
“inverted index”
indexing
aardvark 3452, 11437, ….......arm 4, 19, 29, 98, 143, ...armada 145, 457, 789, ...armadillo 678, 2134, 3970, ...armani 90, 256, 372, 511, ........zebra 602, 1189, 3209, ...
Querying
Boolean queries: (zebra AND armadillo) OR armani compute unions/intersections of lists
Ranked queries: zebra, armadillo give scores to all docs in union or intersection of lists
look up
aardvark 3452, 11437, ….......arm 4, 19, 29, 98, 143, ...armada 145, 457, 789, ...armadillo 678, 2134, 3970, ...armani 90, 256, 372, 511, ........zebra 602, 1189, 3209, ...
Ranked Querying: • return best pages first• term- vs. link vs. log-based approaches• multiple phases, many features• also add meaningful “snippets”
• coverage (need to cover large part of the web)
• good ranking (in the case of broad and narrow queries)
• freshness (need to update content)
• user load (> 50000 queries/sec - Google)
• manipulation (sites want to be listed first)
Challenges for search engines:
need to crawl and store massive data sets
smart information retrieval techniques
frequent recrawling of content
many queries on massive data
naïve techniques will be exploited quickly
Summary:
• Four main tasks: - crawling (data acquisition) - data/web mining - index building and maintenance - query execution
• highly optimized ranking functions and systems• tens to hundreds of thousands of machines• major engines need hundreds of engineers• and PhD-level experts in stats, ML, systems, etc. • many more jobs in related companies
“IR is concerned with the representation, storage, organization of, and access to information items”
• focus on automatic processing (indexing, clustering, search) of unstructured data (text, images, audio, ...)
• subfield of Computer Science, but with roots in Library Science, Information Science, and Linguistics
• main focus on text data, but also images, audio, video• applications: - searching in a library catalog - categorizing a collection of medical articles by area - web search engines
Historical Roots: Information Retrieval (IR)
• WW2 era computers built for number crunching: ballistics computations, code breaking
• since earliest days, also used to “organize information” - Memex (Vannevar Bush, 1945)
• today, this is the main application! - store and organize data and documents - model organizations and work processes
• Computer Organizer (also: communications/media)
• … however, no clean separation
Historical Perspective
• IR: lesser known cousin of field of Databases• Databases: focus on structured data
• IR: unstructured data: “documents”
• IR focused on providing info directly to human users• IR has false positives and negatives (is fuzzy)
Structured vs. Unstructured Data
- scientific articles, novels, poems, jokes, web pages
• Babylonians, Greeks, Romans, etc.• Indexing & creation of concordances (by hand) - “algorithms for full-text indexing” !• Lib. of Cong. and Dewey Library Classifications• Documentalism• Bibliometric and Informetric distributions: - Bradford, Lotka, Zipf, Pareto, Yule (1920s-40s)
• Citation Analysis and Social Network Analysis• Microfilm rapid selectors: (e.g., E. Goldberg 1931)• Memex (Vannevar Bush, 1939/45)
IR Before 1945:
• “As We May Think”, Atlantic Monthly, 1945 (mostly written 1939)
Memex: Vannevar Bush (1890-1974)
• Querying documents by keywords• Classifying documents by topic• Hypertext and analyzing links between documents
• Library catalogs and digital libraries• Searching a collection of news or medical articles• National security: - analyzing communication streams, financial networks
• Most widely used application: web search!
IR Techniques and Applications:
2. Technical Challenges and Opportunities:
• What is a graph? - nodes and edges - edges directed or undirected
• Graphs are used to model many scenarios - social relationships: who knows whom, who is friends with whom? - citations in literature: which physicist cites which other physicist? - email, telephone: who communicates with whom? - follow the money: who gives money to whom? - the web: who links to whom?
Graphs and Social Networks
• nodes are researchers• connected by edge if one cites the other• From: University of Cottbus
Example: Scientific Literature
• nodes are employees• connected by edge if they exchanged more than 5 emails• From: Shetty/Adibi (USC)
Example: Enron Email Network
• search engines: use hyperlink graph to improve ranking
• national security: who calls whom and what does it mean?
• social sciences: understanding societies
Social Networks: Why do we care?
• Basic idea: exploits judgments by millions of web pages
• A page that is highly referenced should be better or more important
• Pagerank (Brin&Page at Google) “significance of a page depends on significance of those referencing it”
• s(a) = s(b)/2 + s(c)/3 + s(d)/1
• System of equations• Unique solution under some assumptions
Link-Based Ranking Techniques
• initialize the rank value of each node to 1/n (0.2 for 5 nodes)
• a node with k outgoing links transmits a 1/k fraction of its current rank value over that edge to its neighbor
• iterate this process many times until it converges• NOTE: this is a random walk on the link graph• Pagerank: stationary distribution of this random walk
1/21/2
1/21/2
1/2
1/2 1
1
Pagerank
0.2
0.2
0.2
0.2
0.2
1/21/2
1/21/2
1/2
1/2 1
1 0.2
0.2
0.2
0.2
0.2
1/21/2
1/21/2
1/2
1/2 1
1 0.1
0.2
0.2
0.3
0.2
1/21/2
1/21/2
1/2
1/2 1
1 0.1
0.3
0.15
0.25
0.2
1/21/2
1/21/2
1/2
1/2 1
1 0.143
0.286
0.143
0.286
0.143
..
(1) (2)
(3) (n)
• stationary distribution: vector x with xA = x• A is primitive, and x Eigenvector of A• computed using Jacobi or Gauss Seidel iteration
1/21/2
1/21/2
1/2
1/2 1
1
Matrix notation
0.2
0.2
0.2
0.2
0.2
0 1/2 0 0 1/2
0 0 1/2 0 1/2
0 0 0 1 0
0 0 1/2 0 1/2
1 0 0 0 0
A
• Now every engine is using link-based ranking
• But the idea is much older!
• Who is the most important physicist? - citation analysis, 1960s
• Who is the most important person in town? - social network analysis, 1950s
• Who is the most important person on Facebook? - the most friends? - or the most important friends? (recursive) - or the best friends? …. and on Twitter ?
Computational Advertising:
• New field dealing with mathematical techniques for targeting ads in electronic media• a multi-billion $ business• economic foundation of search engines and web• many startups in New York City• ITV: TV is moving to the internet• how to match ads with the right consumers • scary privacy implications
Comp. Adv. Basics• Search ads: matching ads to queries - ads on top and right hand side of search results - based on query and past behavior (cookies etc) - pay-per-click model with bids by advertisers - or pay-per-action: goal is immediate action by user • Display ads: large, shiny ads for brands - banner ads on major sites for cars, movies, etc. - per-par-display: e.g., $100 per million impressions - sold by contract: e.g., ads for upcoming movies• AdSense (Google): text ads on 3rd-party pages - placing ads on pages based on content and user - Google sharing money with site owner - danger of manipulation by site owner
Comp. Adv. Marketplace• Very complicated zoo of companies and roles - ad networks - ad campaign coordination/optimization - companies providing user data - arbitrage & manipulation - hundreds/thousands of companies - real-time auctions in tens of millisecs • Emerging ads scenarios - monetizing social networks (facebook, linkedIn etc.) - mobile ads and ads in app-space (e.g., flurry) - ITV and internet radio: ads not a broadcast (hulu) - ads in games and virtual worlds• Privacy: looking bad at the moment - (almost) everything is for sale …
• search engine positions is $$$• web pages are cheap• make lots of automatic junk …• use ads to make $ (or $$$)
Adversarial Information Retrieval
3. Education and Opportunities:
• Web search at center of the online world• interesting technical challenges• many professional opportunities• NYC is a center of this industry• Google, plus media, ads, e-commerce, mobile
• what qualifications are needed?
Perspective:
algorithms
systemsinformation retrieval
databases
machine learning
natural languageprocessin
g
AI
library &information
science
websearch
• Search is somewhat Computer Science oriented
• … it’s all software (mostly)
• also, media/design exp. becoming important
• courses/topics to learn: - algorithms - distributed systems - databases - web search & information retrieval - machine learning, data mining, and statistics - natural language processing
In the curriculum
Course Offering & Student Activities
• CS6913: Web Search Engines (Spring Sem.)
- web and search engine architecture how does it all work?- working with massive data sets storing and analyzing terabytes- introduction to Information Retrieval unstructured data, text- system building skills building distributed systems- the Web as a social network adversarial behavior, spam, communities
• Course Objectives
• Student Activities- course projects build your own (small-scale) search engine- independent research projects, theses, assistantships