Unit 3 Web Search Engines

Unit 3 Web Search Engines

Can You Find the Answers?Can You Find the Answers?

Connect to Connect to GoogleGoogle Search for items on Iran Search for items on Iran Records ________Records ________ Combine Iran with nuclear weaponsCombine Iran with nuclear weapons Rec _______Rec _______ Combine Iran with the phrase Combine Iran with the phrase

nuclear weaponsnuclear weapons Rec _______Rec _______ Use Advanced Search:Use Advanced Search: Combine Iran with the phrase Combine Iran with the phrase nuclear weaponsnuclear weapons

so that all the words appear in title of so that all the words appear in title of

documentsdocuments Records ___Records ___

Unit 3Web Search Engines

Unit 3Web Search Engines

How People Search on the WebHow People Search on the Web What Are Search Engines?What Are Search Engines? How Search Engines Work How Search Engines Work What’s in Search Engines?What’s in Search Engines? How to Find Search EnginesHow to Find Search Engines Search BasicsSearch Basics

Three Ways People SearchThree Ways People Search

SurfSurf No direction, clear idea, issueNo direction, clear idea, issue Consult people, news, magazines, Web for ideasConsult people, news, magazines, Web for ideas

BrowseBrowse Have some idea, but vague, flexible, unclearHave some idea, but vague, flexible, unclear Consult reference sources, Web directories for Consult reference sources, Web directories for

direction, topic, themedirection, topic, theme Search in-depthSearch in-depth

Have defined topic, narrow focusHave defined topic, narrow focus Consult databases, search and metasearch enginesConsult databases, search and metasearch engines

What Are Search EnginesWhat Are Search Engines

SoftwareSoftware Captures web sites, pagesCaptures web sites, pages Indexes full-text of web pageIndexes full-text of web page Provides interface to search web pages Provides interface to search web pages

DatabaseDatabase Large, billions of pages (unlike directories)Large, billions of pages (unlike directories) Computer built (robots, spiders)Computer built (robots, spiders) No selectivity, no evaluationNo selectivity, no evaluation

How Search EnginesWorkHow Search EnginesWork

Spider comb, “capture” web pagesSpider comb, “capture” web pages Software builds databaseSoftware builds database Words from web pages “indexed”Words from web pages “indexed” Search interface finds words on pagesSearch interface finds words on pages Engine ranks, describes resultsEngine ranks, describes results How engines and directories differHow engines and directories differ

Spiders Comb, Capture Web PagesSpiders Comb, Capture Web Pages

Software decides which web pages to Software decides which web pages to collectcollect

Spiders check for updated pages Spiders check for updated pages Spiders remove dead sitesSpiders remove dead sites

Spider Software Builds DatabaseSpider Software Builds Database

Current web size: over 15 billion pagesCurrent web size: over 15 billion pages No engine’s database covers it allNo engine’s database covers it all

Google covers 22% (3.3 billion+)Google covers 22% (3.3 billion+) AlltheWeb covers 21% (3.2 million+)AlltheWeb covers 21% (3.2 million+) HotBot (Inktomi) covers 20% (3 billion+)HotBot (Inktomi) covers 20% (3 billion+) Teoma covers 10% (1.5 billion) Teoma covers 10% (1.5 billion)

Words from Web Pages “Indexed”

Words from Web Pages “Indexed”

““Index” means creating lists of words for Index” means creating lists of words for database and linking words to web pagesdatabase and linking words to web pages

Some index full text in documentSome index full text in document Some index part of textSome index part of text

First 100 words in documentFirst 100 words in document Words in abstract, or title of documentWords in abstract, or title of document

How indexing works affects search How indexing works affects search resultsresults

Search Interface Finds Web PagesSearch Interface Finds Web Pages

Provides keyword search boxProvides keyword search box

Offers simple or advanced searchingOffers simple or advanced searching Offers search options to affect results:Offers search options to affect results:

Most assume AND between words: Russian mafiaMost assume AND between words: Russian mafia Most accept “quotes” to search a PHRASE: Most accept “quotes” to search a PHRASE:

“Russian mafia”“Russian mafia” Most allow FIELD searches : ti:Russian mafiaMost allow FIELD searches : ti:Russian mafia

AlltheWebAlltheWeb

Engine Ranks, Describes Results Engine Ranks, Describes Results

Software lists most “relevant” items firstSoftware lists most “relevant” items first Word popularity: word repetitions, locationWord popularity: word repetitions, location Site popularity – visitations of web siteSite popularity – visitations of web site Link popularity – how often link citedLink popularity – how often link cited

Results describedResults described Few words to a paragraphFew words to a paragraph Sometimes stars, other indicators of Sometimes stars, other indicators of

relevancyrelevancy

How Engines and Directories Differ

How Engines and Directories Differ

Computers vs peopleComputers vs people Engine spiders not editors select documentsEngine spiders not editors select documents

Quantity vs qualityQuantity vs quality Engines big: want all, accept anythingEngines big: want all, accept anything Directories small: want “best” “important”Directories small: want “best” “important”

Technology vs human judgmentTechnology vs human judgment Engine software ranks, no human evaluationEngine software ranks, no human evaluation

Top Search Engines Top Search Engines

GoogleGoogle 3.3 billion+3.3 billion+ AlltheWebAlltheWeb 3.2 billion+3.2 billion+ HotBotHotBot (Inktomi) (Inktomi) 3 billion+3 billion+ TeomaTeoma 1.5 billion+1.5 billion+

Directories, Search Engines and Defaults

Directories, Search Engines and Defaults

If directories find little, they default to enginesIf directories find little, they default to engines Yahoo defaults to GoogleYahoo defaults to Google Open Directory defaults to GoogleOpen Directory defaults to Google Looksmart defaults to Inktomi (Hotbot)Looksmart defaults to Inktomi (Hotbot)

Some search engines borrow directoriesSome search engines borrow directories GoogleGoogle uses “Open Directory” uses “Open Directory”

Learn the source of information when using a Learn the source of information when using a directory search box or search engine’s directory search box or search engine’s directorydirectory

Metasearch EnginesMetasearch Engines

Technologies that search several search Technologies that search several search engines at the same timeengines at the same time

ProsPros

Increase results when search engine Increase results when search engine produce littleproduce little

Save time by searching several engines at Save time by searching several engines at onceonce

Show results of several engines on one Show results of several engines on one pagepage

ConsCons Retrieve too many hitsRetrieve too many hits Retrieve less relevant resultsRetrieve less relevant results

Do not individualize search syntax for each Do not individualize search syntax for each engineengine Do not know whether to use and, AND, +, OR, or, Do not know whether to use and, AND, +, OR, or,

cannot interpret phrases, etc.cannot interpret phrases, etc.

Exclude certain large engines like GoogleExclude certain large engines like Google

Top Metasearch EnginesTop Metasearch Engines

VivisimoVivisimo Categorizes results, narrows topicsCategorizes results, narrows topics

Ez2findEz2find Includes most major enginesIncludes most major engines

DogpileDogpile Refines results, covers major enginesRefines results, covers major engines

What’s In Search Engines?What’s In Search Engines?

Business, commercial informationBusiness, commercial information Organizational publicationsOrganizational publications Government resourcesGovernment resources Magazine, newspaper excerptsMagazine, newspaper excerpts Some scholarly informationSome scholarly information

Teaching materials, unpublished articlesTeaching materials, unpublished articles Books, articles whose copyright expiredBooks, articles whose copyright expired

What’s Not in Search EnginesWhat’s Not in Search Engines

Books under copyrightBooks under copyright Most Fiction, non-fiction in existence Most Fiction, non-fiction in existence

Journal, magazine, newspaper articlesJournal, magazine, newspaper articles Most current and past researchMost current and past research

Reference booksReference books Most recent, quality publicationsMost recent, quality publications

In shortIn short Bulk of human knowledge and researchBulk of human knowledge and research

How to Find Search EnginesHow to Find Search Engines

Word of mouth, hearsayWord of mouth, hearsay Newspaper, magazine articlesNewspaper, magazine articles Library web pagesLibrary web pages

Guides to search enginesGuides to search engines HyperResearchHyperResearch

Search BasicsSearch Basics

Identify, select keywordsIdentify, select keywords Effects of internet use on children Effects of internet use on children

Internet, children, effectsInternet, children, effects

Combine keywords to focus resultsCombine keywords to focus results Use OR, ANDUse OR, AND Use phrase searchingUse phrase searching Limit search to field like title or URLLimit search to field like title or URL

OrBroadens

OrBroadens

Retrieves an article if it contains Retrieves an article if it contains eithereither keyword keyword

Use to connect Use to connect similarsimilar words words Use to increase resultsUse to increase results

OR Expands Results

OR Expands Results

InternetInternet 1515 Internet OR WebInternet OR Web 50 50 Internet or Web or digital Internet or Web or digital 9090

ANDNarrows

ANDNarrows

Use to connect two Use to connect two differentdifferent ideas ideas AND between keywords means AND between keywords means bothboth

terms must be in recordterms must be in record Use to decrease resultsUse to decrease results

AND Reduces Results

AND Reduces Results

ChildrenChildren 2,956, 0002,956, 000 Children AND InternetChildren AND Internet

1,7561,756 Children AND Internet AND HomeworkChildren AND Internet AND Homework

Children internet homeworkChildren internet homework 26 26

AND, OR andSearch Engine Syntax

AND, OR andSearch Engine Syntax

Use Use helphelp or or tipstips AND, and, OR, or, “+” “-” ?AND, and, OR, or, “+” “-” ? Does engine default to AND or OR? Does engine default to AND or OR? Do AND or OR have to be upper case? Do AND or OR have to be upper case? Use ADVANCED SEARCH to learn optionsUse ADVANCED SEARCH to learn options

Is there a pull-down menu box?Is there a pull-down menu box? AND can mean “All the words” AND can mean “All the words” OR can mean “Any of the words”OR can mean “Any of the words”

Phrase SearchingPhrase Searching

Two words in consecutive orderTwo words in consecutive order Juvenile delinquency Russian mafiaJuvenile delinquency Russian mafia

How does computer recognize “phrase”?How does computer recognize “phrase”? Pull-down menu: EXACT PHRASEPull-down menu: EXACT PHRASE Quotation marks: Quotation marks: “drug abuse”“drug abuse”

Phrase searches reduce hits, improve Phrase searches reduce hits, improve relevancyrelevancy Russian mafia Russian mafia 23,234 23,234 ““Russian mafia” Russian mafia” 789 789

Field SearchingField Searching Common document “fields” Common document “fields”

Author, Title, Subject, Abstract, Text, URLAuthor, Title, Subject, Abstract, Text, URL Limits search to words in particular fieldsLimits search to words in particular fields

Learn syntax:Learn syntax:title: ti: url:title: ti: url: Ti: Russian mafiaTi: Russian mafia url: russianmafiaurl: russianmafia

Use ADVANCED SEARCHUse ADVANCED SEARCH Use pull-down menu (in the title, in the URL)Use pull-down menu (in the title, in the URL)

Reduces hits, improves relevancyReduces hits, improves relevancy Russian mafia (all the words) = 23,234Russian mafia (all the words) = 23,234 Russian mafia (in title) = 254Russian mafia (in title) = 254

Can You Find the Answers?Can You Find the Answers?

Connect to Connect to GoogleGoogle Search for items on Iran Search for items on Iran Records Records 11,00,00011,00,000 Combine Iran with nuclear weaponsCombine Iran with nuclear weapons Rec 7Rec 790,00090,000 Combine Iran with the phraseCombine Iran with the phrase

nuclear weaponsnuclear weapons Rec 428Rec 428,000,000 Use Advanced Search:Use Advanced Search: Combine Iran with the phrase Combine Iran with the phrase nuclear weaponsnuclear weapons

so that all the words appear in title of so that all the words appear in title of

documentsdocuments Records Records 274274

HomeworkHomework

Use major search enginesUse major search engines Alltheweb, Google, TeomaAlltheweb, Google, Teoma

Use a metasearch engine- VivisimoUse a metasearch engine- Vivisimo Practice using AND, OR, phrase, field Practice using AND, OR, phrase, field

searchingsearching