Ecdl2008 Jeromedl Evaluation Long
-
Upload
sebastian-kruk -
Category
Technology
-
view
1.580 -
download
2
description
Transcript of Ecdl2008 Jeromedl Evaluation Long
![Page 1: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/1.jpg)
Chapter Copyright 2007 Digital Enterprise Research Institute. All rights reserved. www.deri.org
Digital Enterprise Research Institute www.deri.ie
Evaluation of Semantic and Social Technologies for Digital Libraries
Sebastian R. Kruk, Ewelina Kruk, Katarzyna Stankiewicz
![Page 2: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/2.jpg)
Digital Enterprise Research Institute www.deri.ie
Outline of presentation
What are Semantic Digital Libraries Overview of JeromeDL Semantic and social information discovery solutions Evaluation goals and setup Evaluation results Conclusions
2
![Page 3: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/3.jpg)
Digital Enterprise Research Institute www.deri.ie
Semantic Digital Libraries - Motivation
• How to integrate and search information from different sources? • How to share and interconnect knowledge among people?
3
![Page 4: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/4.jpg)
Digital Enterprise Research Institute www.deri.ie
JeromeDL - Semantic Digital Library
Open source prototype of Semantic DL research; jointly developed by Gdansk University of Technology, Poland and DERI
Social Semantic Information Spaces Semantic description (interconnected metadata)
Annotations provided by users (social metadata)
Collaborative search and browsing (interface)
Features Search and browsing based on semantics empowers users
Users contribute to the classification process
Users can understand community driven annotations
Users enhance digital content using blogs, wikis on the side
Library can interact with other Internet services
4
![Page 5: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/5.jpg)
Digital Enterprise Research Institute www.deri.ie
9
Metadata and Services in JeromeDL
5
![Page 6: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/6.jpg)
Digital Enterprise Research Institute www.deri.ie
Search and browse on semantics
Natural language templates allows to perform complex queries using natural language
Dynamic Collections easily extensible
Resource-based Recommendations customizable view of recommendations
TagsTreeMaps clustered tags rendered with treemaps algorithm
zoomable interface paradigm
MultiBeeBrowse collaborative and adaptive approach to perform complex browsing
Exhibit (SIMILE, MIT) powerful faceted filtering
6
![Page 7: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/7.jpg)
Digital Enterprise Research Institute www.deri.ie
Natural language templates
Find articles related to mission in the context of aerospace
...QueryTemplates(Regular Expressions)
English Portuguese
Aerospacemission
skos:relatedskos:narrower
resultsmarcont:hasKeyword marcont:hasDomain
SELECT * FROM ....
![Page 8: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/8.jpg)
Digital Enterprise Research Institute www.deri.ie
Resource-based Recommendations
![Page 9: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/9.jpg)
Digital Enterprise Research Institute www.deri.ie
Resource-based Recommendations
Library resource
![Page 10: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/10.jpg)
Digital Enterprise Research Institute www.deri.ie
Resource-based Recommendations
Library resource
hasKeyword
hasDomain
hasCreator
...
![Page 11: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/11.jpg)
Digital Enterprise Research Institute www.deri.ie
Resource-based Recommendations
Library resource
hasKeyword
hasDomain
hasCreator
A
C
D
E
F
Step 1: Find similar resources
G
...
E
C
B
A
![Page 12: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/12.jpg)
Digital Enterprise Research Institute www.deri.ie
Resource-based Recommendations
Library resource
hasKeyword
hasDomain
hasCreator
AE
Step 1: Find similar resources
Step 2: Rank and filter according to user’s settings
...
EBA
![Page 13: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/13.jpg)
Digital Enterprise Research Institute www.deri.ie
Resource-based Recommendations
Library resource
hasKeyword
hasDomain
hasCreator
AE
Step 1: Find similar resources
Step 2: Rank and filter according to user’s settings
...
EBA
summary (max. 3)
![Page 14: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/14.jpg)
Digital Enterprise Research Institute www.deri.ie
Browsing on semantics
TagsTreeMaps filtering based on clustered tags
using treemaps to present the tag space
zoomable interface paradigm
MultiBeeBrowse collaborative browsing
allows to perform complex browsing operations
user can overview browsing context and look up browsing history
Exhibit (SIMILE, MIT) powerful faceted filtering
9
![Page 15: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/15.jpg)
Digital Enterprise Research Institute www.deri.ie
TTM - Filtering with hierarchical tags
Problems with Tag Clouds: information overload (for large tag clouds)
cannot carry structure and/or semantics
querying model: only conjunctive queries
Solution: limits the information overload
– clustering tagging space
– limiting popularity range
zoomable browser on the tagging space
selecting multiple tags– fulltext filtering - easy highlight matching tags
– optional conjunctive (AND) and union (OR) mode
defined interfaces for delivering processors in the pipeline (e.g., clustering, filtering, coloring)
10
![Page 16: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/16.jpg)
Digital Enterprise Research Institute www.deri.ie
TTM - Filtering with hierarchical tags
11
![Page 17: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/17.jpg)
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - Adaptive
Presenting results human-readable names of concepts
type-specific rendering
limiting information overload with stretch-text
Refining queries in-situ each concept is seed to new query
different actions based on concept type
Suggesting properties and concepts most frequently used
recently used
Accessible predicated names human-readable names of properties
support for inverted properties
12
![Page 18: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/18.jpg)
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - Collaborative
Nowadays people share: photos, music, links, etc. - why not queries ?
Collaborative filtering solution adapted for sharing browsing experience based on Social Semantic Collaborative Filtering service
users can tag/annotate their queries
users can share queries with their friends
13
![Page 19: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/19.jpg)
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - Zoomable
Helping users with different problems Finding results
Going back and forth in the refinement process
Overview of current browsing context
Replaying previous queries
4 views: Basic browsing view
Structured history view
HoneyComb view
Life-long history view
14
![Page 20: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/20.jpg)
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - Structured
15
![Page 21: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/21.jpg)
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - Structured
15
![Page 22: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/22.jpg)
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - Structured
15
![Page 23: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/23.jpg)
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - Structured
15
![Page 24: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/24.jpg)
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - Structured
15
![Page 25: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/25.jpg)
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - HoneyComb View
16
![Page 26: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/26.jpg)
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - HoneyComb View
16
![Page 27: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/27.jpg)
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - HoneyComb View
16
![Page 28: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/28.jpg)
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - HoneyComb View
16
![Page 29: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/29.jpg)
Digital Enterprise Research Institute www.deri.ie
MultiBeeBrowse - HoneyComb View
16
![Page 30: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/30.jpg)
Digital Enterprise Research Institute www.deri.ie
Social Services in JeromeDL
• Involve users into sharing knowledge
– Blogs – comments and discussions about documents and
resources
– Tagging – collaborative classification
– Wikis – collaboratively edited additional descriptions, such as
summaries and interesting facts
• Preserve knowledge for future use
– users can learn from experience of others instantly
– recommend new, interesting resources based on users’ profiles
17
![Page 31: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/31.jpg)
Digital Enterprise Research Institute www.deri.ie
What is Social Semantic Collaborative Filtering?
• Goal: to enhance individual bookmarks with shared knowledge within a community
• Users annotate catalogues of bookmarks with semantic information taken from DMoz or WordNet vocabularies
• Catalogs can include (transclusion) friend's catalogues
• Access to catalogues can be restricted with social networking-based polices
• SSCF delivers:– Community-oriented, semantically-rich taxonomies
– Information about a user's interest
– Flows of expertise from the domain expert
– Recommendations based on users previous actions
– Support for SIOC metadata
18
![Page 32: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/32.jpg)
Digital Enterprise Research Institute www.deri.ie
knows
include
bookmark
Social Semantic Collaborative Filtering
19
recommend
![Page 33: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/33.jpg)
Digital Enterprise Research Institute www.deri.ie
Scope of Evaluation
20
![Page 34: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/34.jpg)
Digital Enterprise Research Institute www.deri.ie
Questions for Evaluation
Question 1: Do the social and semantic services increase the quality of the answers provided by the users in response to given problems?
Question 2: Do the social and semantic services increase the accuracy of the references provided by the users to answer given questions?
Question 3: Do the social and semantic services increase overall satisfaction of using the digital library?
Question 4: Which services, i.e., semantic, social, or recommendations, are found to be most useful by the end users?
Question 5: Do social and semantic services improve the information retention?
![Page 35: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/35.jpg)
Digital Enterprise Research Institute www.deri.ie
Evaluation Dimensions
Impact of social and semantic features: One group of users should use a library with and the other group
without semantic and social services.
Half of the evaluation participants use a semantic digital library, the other (control) group used a popular, classic digital library.
Learning curve: Does the particular digital library facilitate users in learning it, and
hence improving the quality of their work, over the time?
To answer this question we engaged participants in a number of tasks spread over longer period of time.
Types of enhancing services: three types: search and browsing on semantics, collaborative
services, recommendation services (reasoning on semantics).
Perform three QA tasks; during each task a new type of enhanced information discovery features was introduced.
22
![Page 36: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/36.jpg)
Digital Enterprise Research Institute www.deri.ie
Evaluation Setup
Two vanilla versions of digital libraries: DSpace - open source “classic” digital library
JeromeDL - open source semantic digital library
Database: noise: 529 articles from http://library.deri.ie/ and http://
books.deri.ie/
references database: 35 articles on Internet psychology
Evaluation site: opened Dec 18th, 2007 and Feb 7th, 2008
advertised on national (Poland) and international social networks
Participants: 59 initiated, 26 completed
23
![Page 37: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/37.jpg)
Digital Enterprise Research Institute www.deri.ie
Evaluation Scenario
Pre-evaluation questionnaire - demographics Initial task: getting to know your library Core tasks: question-answering:
3 rounds, max 45 minutes each
pool of 7 questions from the Internet psychology
up to 300 words answer
unlimited number of supporting references
at least 6 hour breaks
Memory task: question-answering after a month Questionnaire after each task: measuring user satisfaction
24
![Page 38: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/38.jpg)
Digital Enterprise Research Institute www.deri.ie
Evaluation Scenario
25
![Page 39: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/39.jpg)
Digital Enterprise Research Institute www.deri.ie
Satisfaction metrics
Solution-related satisfaction metrics easy to use vs hard to use
complex, mind boggling vs simple,clearly organized
hard to master, unintuitive vs intuitive, straight forward
boring vs interesting
ugly, unattractive vs attractive
useless vs useful, handy.
Task-related satisfaction metrics hard to understand vs easy to understand
hard to execute vs easy to execute
hard to master, unintuitive vs intuitive, straight forward.
Overall satisfaction metrics
26
![Page 40: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/40.jpg)
Digital Enterprise Research Institute www.deri.ie
Q1: Quality of answers
27
![Page 41: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/41.jpg)
Digital Enterprise Research Institute www.deri.ie
Q2: Accuracy of provided references
28
Prec
isio
n
R
ecal
l
![Page 42: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/42.jpg)
Digital Enterprise Research Institute www.deri.ie
Q2: Accuracy of provided references
29
Recall
Precision
![Page 43: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/43.jpg)
Digital Enterprise Research Institute www.deri.ie
Q3: Overall satisfaction (all tasks)
30
![Page 44: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/44.jpg)
Digital Enterprise Research Institute www.deri.ie
Q3: Overall satisfaction (search tasks)
31
![Page 45: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/45.jpg)
Digital Enterprise Research Institute www.deri.ie
Q4: Most useful type of services
32
NLQTTM
ExhibitMBB
col. browsingbookmarks/SSCF
blogranking
bookmarks rec.resource rec. 19.32
15.8312.47
13.4417.76
10.282.68
9.723.62
6.42
![Page 46: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/46.jpg)
Digital Enterprise Research Institute www.deri.ie
Q5: Knowledge Retention
33
Quality of answers: JeromeDL - 2.78, DSpace - 2.44 Correct References: JeromeDL - 6, DSpace - 1 Satisfaction:
![Page 47: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/47.jpg)
Digital Enterprise Research Institute www.deri.ie
Would like to continue using this library ?
JeromeDL DSpace
![Page 48: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/48.jpg)
Digital Enterprise Research Institute www.deri.ie
Would like to continue using this library ?
84.62%
46.15%
JeromeDL DSpace
![Page 49: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/49.jpg)
Digital Enterprise Research Institute www.deri.ie
Conclusions
Quality and accuracy of answers is slightly higher when using semantic and social features
User satisfaction is much higher when using JeromeDL
Further research should focus more on: collaborative and recommendation services - they were
perceived to be the most welcome hiding semantic features behind the scene, e.g., making
simple search much “smarter” Lowering number and diversity of solutions, or
introducing gradual engagemement
35
![Page 50: Ecdl2008 Jeromedl Evaluation Long](https://reader034.fdocuments.in/reader034/viewer/2022042813/54bfd41e4a795933068b4576/html5/thumbnails/50.jpg)
Digital Enterprise Research Institute www.deri.ie
Related Information
Complete report on JeromeDL evaluation:
http://library.deri.ie/resource/ARfuVUi8
More information about Semantic Digital Libraries:
http://semdl.corrib.org/Book/
http://semdl.corrib.org/Tutorial/
JeromeDL home page:
http://www.jeromedl.org/
JeromeDL users mailing list:
Sebastian Ryszard KrukDERI NUI Galway
36