Web Mining
-
Upload
guestb73ec6 -
Category
Technology
-
view
11.608 -
download
1
description
Transcript of Web Mining
Presented By:Akshat Saxena Anjul Sahu
Definition
Application of data mining techniques on the web to discover interesting patterns.
Introduction
Size of web is extremely large Data present on web is unstructured Good scope of data mining Types of data on web
Content of actual webpage Intrapage structure Interpage structure Usage data User profiles and cookies
Web Mining Taxonomy
Web Content Mining
Extends work of search engineImproves on traditional crawler
techniqueUse data mining for efficiency,
effectiveness and scalabilityFurther divided into
◦ Agent based approach◦ Database based approach
Text mining is/isn’t content miningCrawlersPersonalization
Web Content Mining Subtasks
Resource finding Retrieving intended documents
Information selection/pre-processing Select and pre-process specific information
from selected documents Generalization
Discover general patterns within and across web sites
Analysis Validation and/or interpretation of mined
patterns
Text Mining
Web Crawler
Program which browses WWW in a methodical, automated manner
Copy in cache and do Indexing Starts from a seed url Searches and finds links, keywords Types of Crawler
Context focused Focused Incremental Periodic
Focused Crawler
Focused Crawler
Visits only pages of interest Architecture consists of:
Hyperlink Classifier Distiller Crawler
Hub pages - links to relevant pages Hard focus - parent node relevant Soft focus - probability of relevance Harvest rate – precision rate
Context Focused Crawler
Focused crawler was static Drawbacks:
Non-relevant pages having links to relevant ones. These to be followed
Relevant ones not having links to other relevant ones. Backward crawling
CFC in two steps Construct context graphs and classifiers Crawl using these classifiers
Harvest System
Uses caching, indexing and crawling Act as a tool in gathering information
from other sources Components:
Gatherer - obtains information Broker - provides index and query
interface Essence systems Semantic indexing
Virtual Web View
Web as multiple layer database A view of MLDB is virtual web view No spiders used Websites send their indices to others WebML – DMQL for web mining KEYWORDS – covers, covered by,
like, close to Difficult to implement
Personalization
Contents of web are modified as per user’s desires
Personalized not targeted Use cookies, userID, profile
information Legal issues to be considered Includes clustering, classification or
even prediction
Personalization
Types: User preference Collaborative filtering Content based filtering
Example : My Yahoo! was first. Now almost every service offers personalization.
Personalization
Yahoo was the first to introduce the concept of a ’personalized portal’, i.e. a Web site designed to have the look-and-feel as well as content personalized to the needs of an individual end-user.
Mining MyYahoo usage logs provides Yahoo valuable insight into an individual’s Web usage habits, enabling Yahoo to provide compelling personalized content, which in turn has led to the tremendous popularity of the Yahoo Web site.
Web Structure Mining
Creating a model of web organization
Classify web pages Create similarity measures between
web pages Page Rank The Clever system Hyperlink induced topic search(HITS)
PageRankTM
Link analysis algorithm which assigns numerical weight to a webpage.
The numerical weight that it assigns to any given element E is also called the PageRank of E and denoted by PR(E).
the PageRank value for a page u is dependent on the PageRank values for each page v out of the set Bu (this set contains all pages linking to page u), divided by the number L(v) of links from page v.
Page Rank
Increase effectiveness of search engines
Based on number of back links Rank sink problem exists
Clever System
Finds both authoritative pages and hubs
Authoritative - best source Hub - link to authoritative pages Most value page returned Hyperlink Induced Topic Search
Keywords Authority and hub measure
Alternatives to PageRank
HITS Algorithm IBM Clever Project TrustRank But PageRank is the most popular
and widely used algorithm by search engines
Web Usage Mining
Applies mining on web usage data or weblogs or clickstream data
Client perspective Server perspective Aid in personalization Helps in evaluating quality and
effectiveness Preprocessing, pattern discovery and
data structures
Trackers for site usage and analysis
Issues in Web Log
Identify exact user
Exact sequence of pages visited
Security, privacy and legal issues
Preprocessing
Information not in presentable format
Data cleaning required Log: (<src
id>,<literal>,<timestamp>) Data might be grouped Sessions Path completion
Data Structure
DS needed to keep track of patterns identified
DS used is trie A rooted tree where each path from
root to node represents a sequence
Pattern Discovery
Traversal pattern - pages visited in a session
Properties: Duplicate reference may / may not be
allowed Consist of only contiguous page reference Pattern may / may not be maximal
Association rules - pages accessed together
Pattern Discovery
Sequential Pattern - ordered set satisfying a support and maximal
Similar to apriori algorithm Web access pattern - efficient
counting Episodes – partially ordered by
access time; users not identified Pattern analysis
Queries ‘N Suggestions
References: http://maya.cs.depaul.edu/~mobasher/w
ebminer/survey/ Google.com/Technology http://www.almaden.ibm.com/projects/
clever.shtml
Thanks !! {akshatsaxena11, anjulsahu}@gmail.com