Role of Text Mining in Search Engine

Role of Text Mining in Search Engines

IST 345 Term Project

Team 3

Fall 2008

Role of Text Mining in Search Engine

Agenda

1111 IntroductionIntroduction

2222 Current TrendsCurrent Trends

3333

4444 Future TrendFuture Trend

5555

6666 ConclusionsConclusions


Text Mining in Search EngineText Mining in Search Engine

Case StudyCase Study

Really

RESULTS

How do Search Engine Work?


1. New Website is posted, Linked to, or has its content altered


2. Search Engine’s Spider Crawl the page


Goglebot: Google

Slurp: Yahoo

MSNbot: MSN

Skims text, image descriptions, meta data, page titles and URL


Follow links, count links In and Out


SearchWikiEngineSEOSEM

Web


Index key terms, count word frequency

Why Mining in Search Engine?

Overwhelming information in typical user query results

Results are only partly related to each other

Many users investigate only the two or three top ranked documents

Traditional lists of ranked documents do not seem to be sufficient for the exploratory search tasks


Additional techniques are needed to help the users analyze the search results efficiently and drill down to the information they are looking for.

Users need to explore the information• Discovering new patterns, • New entities, and • Knowledge they do not even realize they needed


Why Mining in Search Engine?

Text Mining

Text mining applications today draw on a wide range of techniques and serve many purposes in information management and business intelligence.

TM techniques can be organized into four categories:

1) Classification techniques

2) Association analysis

3) Information extraction techniques

4) Clustering techniques


Use of Text Mining in Search Engine

Text categorization (faceted search systems)

Using multi-dimensional categories to describe (groups of) documents Richer descriptions Expensive to develop the categories

Semantic Web Search

Linguistic analysis of text Addition to purely statistical techniques



Contextualized clustering

Group the search results by topic Clustering of documents according to terms found in the documents

Use of Text Mining in Search Engine

Clustering Engines

Scatter/Gather

Grouper

The Lingo system

The Clusty/Vivisimo engine (www.clusty.com and www.vivisimo.com)

SnakeT

HOBSearch


Clusty / Vivisimo Engine



Future Trends

Anticipated to expand 1000 times

With the introduction of full-text search engines such as AltaVista, Excite, HotBot, Infoseek, Lycos, and Northern Light, the Web can be viewed as a searchable 15-billion-word encyclopedia.

Nutch engine is the future search engine

Fetch several billion pages per month Maintain an index of these pages Search the index up to 1000 times per second Provide very high quality search results Operate at minimal cost

Future Trend Introduction of full-text search engines such as AltaVista, Excite, HotBot, Infoseek, Lycos, and Northern Light

Summarization of online documents

More efficient Categorization/Clustering for the search results

Entity Extraction by using linguistics and pattern detection

Answering intelligent questions



Case Study: Data Crow


Customized Search

Music

Data Crow contains two separate modules The Music Album and Audio CD module

Use one of the online services (MusicBrainz, Amazon, Discogs and others) to find information on your CD and or music files

Parse information from your mp3, flac, ape and or ogg file and fill missing information using an online service


Conclusion

Learnt the evolution of Search Engine

Efficiency of Search Engine can be increased by using mining techniques

Increased demand from customized search


Which of the following is not a search engine?

• Google• Open Directory• Yahoo search• Lycos

Open Directory


What search engine has the largest index of listings on the web?

• Yahoo• Google• MSN• Microsoft

Google


Search Engines and directory are both the same thing because:

• They both index information• They’re both on the internet• They’re not the same thing• They both search for information

They’re not the same thing


What was the first search engine ever created?

• WWW Wanderer• Google; created by Larry and Sergey in 1995• Yahoo; created in 1994• MSN Search; created in 1982

WWW Wanderer: technically not really a search engine, but a pioneer of the crawling process

Q: Google is limited to how many search terms in one query:

• 16• 18• 13• 15

15


Q: How do search engines find out about sites on the Web?

• Osmosis• Search engines automatically know everything on the Web.• Two different ways: search engine spiders index the information, or site owners submit it manually• Search engines have special features that enable them to know when your site is uploaded. It's called "crystal ball technology."

Two different ways: search engine spiders index the information, or site owners submit it manually.


Role of Text Mining in Search Engine

Technology

Transcript of Role of Text Mining in Search Engine