Crawling and Social Ranking CSCI 572 Class Project Huy Pham PhD @ USC April 28 th, 2011.
Web Crawling, Analysis and Archiving. PhD Presentation
-
Upload
vangelis-banos -
Category
Internet
-
view
1.068 -
download
0
Transcript of Web Crawling, Analysis and Archiving. PhD Presentation
Web Crawling, Analysis and ArchivingPHD DEFENSE VANGELIS BANOS
DEPARTMENT OF INFORMATICS, ARISTOTLE UNIVERSITY OF THESSALONIKIOCTOBER 2015
COMMITTEE MEMBERSYannis Manolopoulos, Apostolos Papadopoulos, Dimitrios Katsaros,
Athena Vakali, Anastasios Gounaris, Georgios Evangelidis, Sarantos Kapidakis.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 2 of 67
Problem definition: The web is disappearing
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 3 of 67
Web Archiving• Web archiving is the process of collecting portions of
the Web to ensure the information is preserved in an archive for researchers, historians, and the public.
• Many important organisations work on web archiving since 1996.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 4 of 67
Our ContributionsWe focus on Web Crawling, Analysis and Archiving.1. New metrics and systems to appreciate the possibilities of
archiving websites,2. New algorithms and systems to improve web crawling efficiency
and performance,3. New approaches and systems to archive weblogs,4. New algorithms focused on weblog data extraction.
◦ Publications:• 4 scientific journals (1 still under review),• 7 international conference proceedings,• 1 book chapter.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 5 of 67
Presentation Structure1. An Innovative Method to Evaluate Website
Archivability,2. Near-duplicate and Cycle Detection in Webgraphs
towards Optimised Web Crawling,3. The BlogForever Platform: An Integrated Approach
to Preserve Weblogs,4. A Scalable Approach to Harvest Modern Weblogs,5. Conclusions and Future Work.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE
1. An Innovative Method to Evaluate Website ArchivabilityProblem description• Not all websites can be archived correctly. • Web bots face difficulties in harvesting websites (Technical problems, low performance,
invalid code, blocking web crawlers).• After web harvesting, archive administrators review manually the content.• Web crawing is automated while Quality Assurance (QA) is manual.
Our contributions1. The Credible Live Evaluation of Archive Readiness Plus (CLEAR+) Method to evaluate
Website Archivability.2. The ArchiveReady.com system which is the reference implementation of the method.3. Evaluation and observation regarding 12 prominent Web Content Management
Systems’ (CMS) Archivability.
6
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 7 of 67
CLEAR+: A Credible Live Method to Evaluate Website Archivability• Website Archivability (WA) captures the core aspects of a website
crucial in diagnosing whether it has the potentiality to be archived with completeness and accuracy.o Not to be confused with website reliability, availability, security, etc.
• CLEAR+: A method to produce a credible on-the-fly measurement of Website Archivability by:o Imitating web bots to crawl a website.o Evaluating captured information such as file encoding and errors.o Evaluating compliance with standards, formats and metadata.o Calculating a WA Score (0 – 100%).
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 8 of 67
CLEAR+ Archivability Facets and Website Attributes
FAAccessibility
FcCohesion
FMMetada
ta
FSTStandard
sCompliance
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 9 of 67
CLEAR+ Method Summary1. Perform specific evaluations on Website Attributes2. Each evaluation has the following attributes:
1. Belongs to one or more WA Facets.2. Has low, medium, or high Significance (different weight).3. Has a score range from 0 – 100%.
3. The score of each Facet is the weighted average of all evaluations’ scores.
4. The final Website Archivability is the average of all Facets’ scores.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 10 of 67
Accessibility FacetFacet Evaluation Rating Significance Total
FA
Accessibility
No sitemap.xml 0% High
63%
21 valid and 1 invalid link 95% High
2 inline JavaScript files 0% High
HTTP Caching Headers 100% Medium
Average response time 30ms, very fast
100% High
Not using proprietary formats (e.g. Flash or QuickTime)
100% High
ADBIS 2015 Website Accessibility Evaluation 1st Sept 2015
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 11 of 67
Cohesion Facet
• If files constituting a single website are dispersed across different web locations, the acquisition and ingest is likely to suffer if one or more web locations fail.
• 3rd party resources increase website volatility.
Facet Evaluation Rating Significance Total
FC
Cohesion
6 local and no external scripts 100% Medium 100%9 local and no external images 100% Medium
2 local and no external CSS 100% Medium
ADBIS 2015 Website Accessibility Evaluation 1st Sept 2015
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 12 of 67
Metadata Facet
• Adequate metadata are a big concern for digital curation.• The lack of metadata impairs the archive’s ability to manage,
organise, retrieve and interact with content effectively.
Facet Evaluation Rating Significance Total
FM
Metadata
HTTP Content type 100% Medium 100%
HTTP Caching headers 100% Medium
ADBIS 2015 Website Accessibility Evaluation 1st Sept 2015
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 13 of 67
Standards Compliance FacetFacet Evaluation Rating Significance Total
FST
Standards Compliance
2 Invalid CSS files 0% Medium
74%
Invalid HTML file 0% Medium
No HTTP Content transfer encoding 50% Medium
HTTP Content type found 100% Medium
HTTP Caching headers found 100% Medium
9 images found and validated with JHOVE 100% Medium
Not using proprietary formats (e.g. Flash or QuickTime)
100% High
ADBIS 2015 Website Accessibility Evaluation 1st Sept 2015
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 14 of 67
ADBIS’2015 Website Archivability Evaluation
• Web application implementing CLEAR+• Web interface and REST API• Developed using Python, MySQL, Redis,
PhantomJS, Nginx, Linux.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 15 of 67
Experimentation with Assorted Datasets• D1: National libraries, D2: Top 200 universities,• D3: Government organizations, D4: Random spam websites from Alexa.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 16 of 67
Evaluation by experts• Experts evaluate how well a website is archived in the Internet
Archive and assign a score.• We evaluate the WA Score using ArchiveReady.com.• Pearson’s Correlation Coefficient for WA, WA Facets and experts’
score.• Correlation: 0.516
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 17 of 67
WA Variance in the Same Website
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 18 of 67
Web Content Management Systems Archivability• Aim: Identify strengths and weaknesses in different web
CMS regarding their WA.• Corpus: 5.821 random WCMS Samples from the Alexa
top 1m websites. Systems:o Blogger, DataLife Engine, DotNetNuke, Drupal,
Joomla, Mediawiki, MovableType, Plone, PrestaShop, Typo3, vBulletin, Wordpress.
• Evaluation using the ArchiveReady.com API• Results saved in MySQL and analysed.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 19 of 67
WCMS Accessibility Variations
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 20 of 67
WCMS Standards Compliance Variations
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 21 of 67
WCMS Metadata Results
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 22 of 67
WCMS Archivability Results Summary
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE
Website Archivability Impact• Deutches Literatur Archiv, Marbach, is using the ArchiveReady API in its
web archiving workflow since early 2014.• Stanford University Libraries Web Archiving Resources recommends using
the CLEAR method and ArchiveReady. • The University of South Australia is using ArchiveReady in their Digital
Preservation Course (INFS 5082).• Invited to present at the Library of Congress, National Digital Information
Infrastructure & Preservation, Web Archiving, 2015, and the Internet Archive Web Archiving meeting (University of Innsbruck, 2013).
• Many contacts and users from: University of Newcastle, University of Manchester, Columbia University, Stanford University, University of Michigan Bentley Historical Library, Old Dominion University.
• 120 unique daily visitors, 80.000+ evaluations at http://archiveready.com/.
23
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 24 of 67
Presentation Structure1. An Innovative Method to Evaluate Website Archivability,2. Near-duplicate and Cycle Detection in Webgraphs
towards Optimised Web Crawling,3. The BlogForever Platform: An Integrated Approach to
Preserve Weblogs,4. A Scalable Approach to Harvest Modern Weblogs,5. Conclusions and Future Work.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 25 of 67
2. Near-duplicate and Cycle Detection in Webgraphs towards Optimised Web Crawling
Problem description• Web bots capture a lot of duplicate and near-duplicate data.
o There are methods to detect and remove duplicate data after crawling.o There are few methods to remove near-duplicate data in web archives.
• Web bots fall into web spider traps, webpages that cause infinite loops. No automated solution to detect them.
Our Contributions
1. a set of methods to detect duplicate and near-duplicate webpages in real time during web crawling.
2. a set of methods to detect web spider traps using webgraphs in real time during web crawling.
3. The WebGraph-It.com system, a web platform which implements the proposed methods.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 26 of 67
Key Concepts
• Unique Webpage Identifier?• Webpage similarity metric?• Web crawling modeled as a graph?
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 27 of 67
Key Concepts: Unique Webpage Identifier• URI is not always optimal as a unique webpage identifier.
o http://edition.cnn.com/videos - http://edition.cnn.com/videos#some-point
o http://edition.cnn.com/videos?v1=1&v2=2 o http://edition.cnn.com/videos?v2=2&v1=1
• Sort-friendly URI Reordering Transform (SURT) URI Conversion.o URI: scheme://[email protected]:port/path?query#fragment o SURT: scheme://(tld,domain,:port@user)/path?query
o URI: http://edition.cnn.com/tech -> SURT: com,cnn,edition/tech• SURT encoding is lossy. SURT is not always reversible to URI.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 28 of 67
Key Concepts: Unique Webpage Identifier Similarity• Dear duplicate URIs/SURTs may have duplicate content.
o http://vbanos.gr/page?show-greater=10 - http://vbanos.gr/page?show-greater=11
o http://vbanos.gr/blog/tag/cakephp/ - http://vbanos.gr/blog/tag/php/
• We use the Sorensen-Dice coefficient similarity to search for near-duplicate webpage identifiers with a 95% similarity threshold.o Low sensitivity to word ordering,o Low sensitivity to length variations,o Runs in linear time.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 29 of 67
Key Concepts: Unique Webpage Identifier Similarity
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 30 of 67
Key Concepts: Webpage content similarity• Content similarity:• Exact duplicate webpages• Near-duplicate webpages (ads, dates, counters may change)• We use the simhash algorithm (Charikar) to calculate bit
signatures from each webpage.• 96 bit webpage signature.• Near duplicate webpages have very few different bits.• Fast to compare the similarity of two webpages.• Efficient storage (save only the signature, keep it in memory).
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 31 of 67
Key Concepts: Webpage content similarity
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE
Key concepts: Webgraph cycle detection Step 1 Step 2 Step 3
New Node F Get Nearby Nodes (dist=3) and Cycle Detection using DFS (dist=3)
check for duplicate / near duplicate
32 of 67
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 33 of 67
Web Crawling Algorithms
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 34 of 67
WebGraph-It.com System• Web application implementing all presented algorithms. API Available.• Built using Python, PhantomJS, Redis, MariaDB, Linux.• Easy to expand and create new web crawling algorithms as plugins.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 35 of 67
Evaluation1. Dataset: 100 random websites from Alexa top 1M.2. Crawl with all 8 algorithms (C1-C8) using the WebGraph-it system.3. Record metrics for each web crawl.4. Analyse the results and compare with the base web crawl.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 36 of 67
Indicative results for a single website
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE
Results
37
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE
Evaluation conclusions• Best method is D8: Cycle detection with content similarity• 17.1% faster than the base crawl.• 60% of base crawl webpages captured.• 98.3% results completeness.• Always use SURT instead of URL as a unique webpage
identifier.• Use URL/SURT similarity AND content similarity together.• Using URL/SURL similarity alone results in incomplete results.
38
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 39 of 67
Presentation Structure1. An Innovative Method to Evaluate Website Archivability,2. Near-duplicate and Cycle Detection in Webgraphs
towards Optimised Web Crawling,3. The BlogForever Platform: An Integrated Approach to
Preserve Weblogs,4. A Scalable Approach to Harvest Modern Weblogs,5. Conclusions and Future Work.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 40 of 67
3. The BlogForever Platform: An Integrated Approach to Preserve WeblogsProblem descriptionCurrent web archiving tools have issues with weblog archiving.• Scheduling (timely intervals vs archive when new content is available.• Content selection (archive everything instead of archiving the updated content only),• Ignoring weblog features (rich set of information entities, structured content, RSS, tags,
etc.)Our contributions
1. A survey of the technical characteristics of weblogs.2. Methods to improve weblog harvesting, archiving and management.3. Methods to integrate weblog archives with existing archive technologies.4. The BlogForever platform: A system to support harvesting, ingestion, management and
reuse of weblogs.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 41 of 67
Technical survey of the blogosphere• Dataset: 259.930 blogs• Evaluate the use of:
o Blog platforms,o Web standards (HTTP Headers, HTML markup etc),o XML feeds,o Image formats,o JavaScript frameworks,o Semantic markup (Microformats, XFN, OpenGraph, etc)
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 42 of 67
Indicative survey results: Blog platforms
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 43 of 67
Indicative survey results: Image and feed types
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 44 of 67
standard_descrcontent
date
Blog has Entry
is a
PostPage
has
Comment
Content
has
Authorhas
has
Categorised ContentCategorised Content
CommunityCommunity
Web FeedWeb Feed
External WidgetsExternal Widgets
Network and Linked DataNetwork and Linked DataBlog ContextBlog Context
SemanticsSemantics
BlogForever: Conceptual Data Model
Version 0.6
Spam DetectionSpam Detection
embeds
WidgetType
crawlerAouth
Widget
Feed
idformat
last_updated
generatorlast_build_date
related_feedLayout
themecss
images
SnapshotView
dateformat
src
hashas
Expression_ Meta
descriptiondef_keywords
Spam
dateflag
contains
SpamCategory
Keyword SentimentContent_Similarity
scoreflag
scoresrc
contains
contains
usernameURIUserProfile
ExternalProfile ProfileType
URI
Association Triple
subjectpredicate
object
Association Type
Multimedia
Text
Link
Tag
srcalt
caption/descrGEO
srcdescription
type
valueformat
tags
copyrightembedding
thumbnail
language
Ranking, Category and SimilarityRanking, Category and Similarity
valuedate
Ranking given
Similarity
Crawling InfoCrawling Info
Crawl captured
Category
similarity_scorealgorithm
AffiliationTypeAffiliation
Eventdate locationname URL
Topic
avatar
creator
service_uri
hasFeed_Type
value
Structured_ Meta
nameproperty
has
Standard and Ontology MappingStandard and Ontology Mapping
OntologyMapping
OntClass
OntProperty
SpamAlgorithm
ImageAudio
VideoDocument
LinkType
is a
BlogEntity
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 45 of 67
The BlogForever platform
45
Blog crawlers
Real-time monitoring Html data extraction engine Spam filtering Web services extraction engine
Unstructured information
Web servicesBlog APIs
Original data andXML metadata
Blog digital repository
Digital preservation and QA Collections curation Public access APIs Web interface to browse, search, export Personalised services
Harvesting
PreservingManaging and reusing
Web servicesWeb interface
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 46 of 67
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 47 of 67
The BlogForever platform
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 48 of 67
Evaluation using external testers
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 49 of 67
Presentation Structure1. An Innovative Method to Evaluate Website Archivability,2. Near-duplicate and Cycle Detection in Webgraphs
towards Optimised Web Crawling,3. The BlogForever Platform: An Integrated Approach to
Preserve Weblogs,4. A Scalable Approach to Harvest Modern Weblogs,5. Conclusions and Future Work.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 50 of 67
4. A scalable approach to harvest modern weblogsProblem description• Inefficient weblog harvesting with generic solutions.• Unpredictable publishing rate of weblogs.Our contributions
1. A new algorithm to build extraction rules from blog web feeds with linear time complexity,
2. Applications of the algorithm to extract authors, publication dates and comments,
3. A new web crawler architecture and system capable of extracting blog articles, authors, publication dates and comments.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 51 of 67
Motivation & Method Overview• Extracting metadata and content from HTML is hard because web
stardards usage is low. 95% of websites do not pass HTML validation. • Focusing on blogs, we observed that:
1. Blogs provide XML feeds: standardized views of their latest ~10 posts.2. We have to access more posts than the ones referenced in web feeds.3. Posts of the same blog share a similar HTML structure.
• Content Extraction Method Overview1. Use blog XML feeds and referenced HTML pages as training data to build
extraction rules.2. For each XML element (Title, Author, Description, Publication date, etc)
create the relevant HTML extraction rule.3. Use the defined extraction rules to process all blog pages.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 52 of 67
Locate in HTML page all RSS referenced elements
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 53 of 67
Generic procedure to build extraction rules
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 54 of 67
• Rules are XPath queries.• For each rule, we compute the score based on string similarity.• The choice of ScoreFunction greatly influences the running time and
precision of the extraction process.
• Why we chose Sorensen–Dice coefficient similarity:1. Low sensitivity to word orderingand length variations2. Runs in linear time
Extraction rules and string similarity
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 55 of 67
Example: blog post title best extraction rule• Find RSS blog post title: “volumelaser.eim.gr” in html page:
http://vbanos.gr/blog/2014/03/09/volumelaser-eim-gr-2/• The Best Extraction Rule for the blog post title is: /body/div[@id=“page”]/header/h1
XPath HTML Element Value Similarity Score
/body/div[@id=“page”]/header/h1 volumelaser.eim.gr 100%
/body/div[@id=“page”]/div[@class=“entry-code”]/p/a
http://volumelaser.eim.gr/ 80%
/head/title volumelaser.eim.gr | Βαγγέλης Μπάνος
66%
... ... ...
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 56 of 67
Variations for authors, dates, comments• Authors, dates and comments are special cases as they appear
many times throughout a post.• To resolve this issue, we implement an extra component in the
Score function:o For authors: an HTML tree distance between the evaluated node and
the post content node.o For dates: we check the alternative formats of each date in addition
to the HTML tree distance between the evaluated node and the post content node.
o Example: “1970-01-01” == “January 1 1970”o For comments: we use the special comment RSS feed.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 57 of 67
System Pipeline of operations:
1. Render HTML and JavaScript,2. Extract content,3. Extract comments,4. Download multimedia files,5. Propagate resulting records to
the back-end. Interesting areas:
◦ Blog post page identification,◦ Handle blogs with a large number of pages,◦ JavaScript rendering,◦ Scalability.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 58 of 67
Evaluation• Extract articles and titles from web pages and compare
extraction success rate and running time• Comparison against three open-source projects:
o Readability (Javascript), Boilerpipe (Java), Goose (Scala).
• Dataset: 2300 blog posts from 230 blogs from Spinn3r.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 59 of 67
5. Conclusions• We proposed tangible ways to improve web crawling, web
archiving and blog archiving with new algorithms and systems.
• The Credible Live Evaluation of Archive Readiness Plus (CLEAR+) method to evaluate Website Archivability.
• Methods to improve web crawling via detecting duplicates, near-duplicates and web spider traps on the fly.
• A new approach to harvest, manage, preserve and reuse weblogs.
• A new scalable algorithm to harvest modern weblogs.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 60 of 67
PublicationsPublications in scientific journals:
1. Banos V., Manolopoulos Y.: “Near-duplicate and Cycle Detection in Webgraphs towards Optimised Web Crawling”, ACM Transactions on the Web Journal, submitted, 2015.
2. Banos V., Manolopoulos Y.: “A Quantitative Approach to Evaluate Website Archivability Using the CLEAR+ Method”, International Journal on Digital Libraries, 2015.
3. Banos V., Blanvillain O., Kasioumis N., Manolopoulos Y.: “A Scalable Approach to Harvest Modern Weblogs”, International Journal of AI Tools, Vol.24, No.2, 2015.
4. Kasioumis N., Banos V., Kalb H.: “Towards Building a Blog Preservation Platform”, World Wide Web Journal, Special Issue on Social Media Preservation and Applications, Springer, 2013.
Publications in international conference proceedings:
5. Banos V., Manolopoulos Y.: “Web Content Management Systems Archivability”, Proceedings 19th East-European Conference on Advances in Databases & Information Systems (ADBIS), Springer Verlag, LNCS Vol.9282, Poitiers, France, 2015.
6. Blanvillain O., Banos V., Kasioumis N.: BlogForever Crawler: “Techniques and Algorithms to Harvest Modern Weblogs”, Proceedings 4th International Conference on Web Intelligence, Mining & Semantics (WIMS), ACM Press, Thessaloniki, Greece, 2014.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 61 of 67
Publications3. Banos V., Kim Y., Ross S., Manolopoulos Y.: “CLEAR: a Credible Method to Evaluate Website
Archivability”, Proceedings 10th International Conference on Preservation of Digital Objects (iPRES), Lisbon, Portugal, 2013.
4. Kalb H., Lazaridou P., Banos V., Kasioumis N., Trier M.: “BlogForever: From Web Archiving to Blog Archiving”, Proceedings ‘Informatik Angepast an Mensch, Organisation und Umwelt‘ (INFORMATIK), Koblenz, Germany, 2013.
5. Stepanyan K., Gkotsis G., Banos V., Cristea A., Joy M.: “A Hybrid Approach for Spotting, Disambiguating and Annotating Places in User-Generated Text”, Proceedings 22nd International Conference on World Wide Web (WWW), Rio de Janeiro, Brazil, 2013.
6. Banos V., Baltas N., Manolopoulos Y.: “Trends in Blog Preservation”, Proceedings 14th International Conference on Enterprise Information Systems (ICEIS), Vol.1, pp.13-22, Wroclaw, Poland, 2012.
7. Banos V., Stepanyan K., Manolopoulos Y., Joy M., Cristea A.: “Technological Foundations of the Current Blogosphere”, Proceedings 2nd International Conference on Web Intelligence, Mining & Semantics (WIMS), ACM Press, Craiova, Romania, 2012.
Book chapters:
8. Banos V., Baltas N., Manolopoulos Y.: “Blog Preservation: Current Challenges and a New Paradigm”, chapter 3 in book Enterprise Information Systems XIII, by Cordeiro J., Maciaszek L. and Filipe J. (eds.), Springer LNBIP Vol.141, pp.29–51, 2013.
of 63WEB CRAWLING, ANALYSIS AND ARCHIVING - PHD DEFENSE 62 of 67
Future Work1. Website Archivability
1. Augment the CLEAR+ method with new metrics.2. Disseminate to wider audiences (e.g. web developers)3. Integrate with web archiving systems.4. Improve http://archiveready.com/
2. Web crawling duplicate and near-duplicate detection1. Develop new algorithm variants.2. Integrate into open source web crawlers.3. Provide support services to web crawling operations.4. Improve http://webgraph-it.com/
3. BlogForever platform1. Automate content curation processes.2. Improve entity detection in archived content.3. Support more types of weblogs.4. http://webternity.eu/
Web Crawling, Analysis and ArchivingPHD DEFENSE VANGELIS BANOS
DEPARTMENT OF INFORMATICS, ARISTOTLE UNIVERSITY OF THESSALONIKIOCTOBER 2015
THANK YOU!