Presented By:Akshat Saxena Anjul Sahu

Definition

Application of data mining techniques on the web to discover interesting patterns.

Introduction

Size of web is extremely large Data present on web is unstructured Good scope of data mining Types of data on web

Content of actual webpage Intrapage structure Interpage structure Usage data User profiles and cookies

Web Mining Taxonomy

Web Content Mining

Extends work of search engineImproves on traditional crawler

techniqueUse data mining for efficiency,

effectiveness and scalabilityFurther divided into

◦ Agent based approach◦ Database based approach

Text mining is/isn’t content miningCrawlersPersonalization

Web Content Mining Subtasks

Resource finding Retrieving intended documents

Information selection/pre-processing Select and pre-process specific information

from selected documents Generalization

Discover general patterns within and across web sites

Analysis Validation and/or interpretation of mined

patterns

Text Mining

Web Crawler

Program which browses WWW in a methodical, automated manner

Copy in cache and do Indexing Starts from a seed url Searches and finds links, keywords Types of Crawler

Context focused Focused Incremental Periodic

Focused Crawler

Focused Crawler

Visits only pages of interest Architecture consists of:

Hyperlink Classifier Distiller Crawler

Hub pages - links to relevant pages Hard focus - parent node relevant Soft focus - probability of relevance Harvest rate – precision rate

Context Focused Crawler

Focused crawler was static Drawbacks:

Non-relevant pages having links to relevant ones. These to be followed

Relevant ones not having links to other relevant ones. Backward crawling

CFC in two steps Construct context graphs and classifiers Crawl using these classifiers

Harvest System

Uses caching, indexing and crawling Act as a tool in gathering information

from other sources Components:

Gatherer - obtains information Broker - provides index and query

interface Essence systems Semantic indexing

Virtual Web View

Web as multiple layer database A view of MLDB is virtual web view No spiders used Websites send their indices to others WebML – DMQL for web mining KEYWORDS – covers, covered by,

like, close to Difficult to implement

Personalization

Contents of web are modified as per user’s desires

Personalized not targeted Use cookies, userID, profile

information Legal issues to be considered Includes clustering, classification or

even prediction

Personalization

Types: User preference Collaborative filtering Content based filtering

Example : My Yahoo! was first. Now almost every service offers personalization.

Personalization

Yahoo was the first to introduce the concept of a ’personalized portal’, i.e. a Web site designed to have the look-and-feel as well as content personalized to the needs of an individual end-user.

Mining MyYahoo usage logs provides Yahoo valuable insight into an individual’s Web usage habits, enabling Yahoo to provide compelling personalized content, which in turn has led to the tremendous popularity of the Yahoo Web site.

Web Structure Mining

Creating a model of web organization

Classify web pages Create similarity measures between

web pages Page Rank The Clever system Hyperlink induced topic search(HITS)

PageRankTM

Link analysis algorithm which assigns numerical weight to a webpage.

The numerical weight that it assigns to any given element E is also called the PageRank of E and denoted by PR(E).

the PageRank value for a page u is dependent on the PageRank values for each page v out of the set Bu (this set contains all pages linking to page u), divided by the number L(v) of links from page v.

Page Rank

Increase effectiveness of search engines

Based on number of back links Rank sink problem exists

Clever System

Finds both authoritative pages and hubs

Authoritative - best source Hub - link to authoritative pages Most value page returned Hyperlink Induced Topic Search

Keywords Authority and hub measure

Alternatives to PageRank

HITS Algorithm IBM Clever Project TrustRank But PageRank is the most popular

and widely used algorithm by search engines

Web Usage Mining

Applies mining on web usage data or weblogs or clickstream data

Client perspective Server perspective Aid in personalization Helps in evaluating quality and

effectiveness Preprocessing, pattern discovery and

data structures

Trackers for site usage and analysis

Issues in Web Log

Identify exact user

Exact sequence of pages visited

Security, privacy and legal issues

Preprocessing

Information not in presentable format

Data cleaning required Log: (<src

id>,<literal>,<timestamp>) Data might be grouped Sessions Path completion

Data Structure

DS needed to keep track of patterns identified

DS used is trie A rooted tree where each path from

root to node represents a sequence

Pattern Discovery

Traversal pattern - pages visited in a session

Properties: Duplicate reference may / may not be

allowed Consist of only contiguous page reference Pattern may / may not be maximal

Association rules - pages accessed together

Pattern Discovery

Sequential Pattern - ordered set satisfying a support and maximal

Similar to apriori algorithm Web access pattern - efficient

counting Episodes – partially ordered by

access time; users not identified Pattern analysis

Queries ‘N Suggestions

References: http://maya.cs.depaul.edu/~mobasher/w

ebminer/survey/ Google.com/Technology http://www.almaden.ibm.com/projects/

clever.shtml

Thanks !! {akshatsaxena11, anjulsahu}@gmail.com