Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference...
-
Upload
roy-warner -
Category
Documents
-
view
219 -
download
0
Transcript of Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference...
![Page 1: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/1.jpg)
Discovery Tool Evaluation: A Three-Pronged Approach
9th Annual AMICAL Meeting and ConferenceSharjah, UAE
![Page 2: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/2.jpg)
Background
• AUS implemented Encore Discovery in 2009• Other products like Serials Solutions Summon and
EBSCO’s EDS are actively marketing in the Middle East• Decision in 2010 to evaluate all products• Having Encore installed gave us opportunity to evaluate
products against the same databases, and think about how an ILS-integrated discovery solution compares to a less integrated solution
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 3: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/3.jpg)
Methodology
• Setting up trials of Summon and EDS• Scheduling vendor presentations of Summon and EDS• Testing all 3 products for ability to produce useful,
relevant and accessible results• Reviewing the features of each of the products• Performing usability testing with all products• Assessing the costs of implementing each product• Assessing the impact of implementing each product on
other library systems and operations
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 4: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/4.jpg)
Usability Testing
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 5: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/5.jpg)
Informal Usability Testing
• Think Aloud Protocol
• 5 participants per product
• Participants only told they would be testing a new discovery tool to find library resources
• Participants used their own research topic to complete realistic tasks
• Observers did not interact with participant except to give tasks and encourage thinking aloud
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 6: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/6.jpg)
Testing Environment
• 1 test administrator • 1 observer• Laptop with built in microphone for recording voice• Camtasia Studio with Powerpoint plug-in to capture
screen and voice• Browser of choice• Other software, websites as requested
– Google and Microsoft Word for spelling
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 7: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/7.jpg)
Tasks
Based on common assignments requirements seen at the Research Help Desk, participants were asked to find:
• an article• an article from a peer reviewed/scholarly/academic
journal• a newspaper article• a reference book• a general collection book• a book published in 2008 or 2009
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 8: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/8.jpg)
Observations
• Limit / facet confusion– “Format” or “Content Type” vs “Location” – Books, e-books, e-journal
• E-journal format was used to find articles
• Limits reset for each new search
• Click through to article– Expect to get article, but given a WebBridge or 360 Link page
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 9: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/9.jpg)
System Usability Scale
• Participants filled out System Usability Scale questionnaire for perceived ease-of-use score
• 10 item Likert scale questionnaire used to determine user’s perception of the system for– Effectiveness– Efficiency– Satisfaction
• Resulting scores range from 0 to 100– Average SUS score across 500+ studies is 68
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 10: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/10.jpg)
System Usability ScoresSystem Usability Scale Encore EDS Summon
1. I think that I would like to use this system frequently 4 3.2 4.4
2. I found the system unnecessarily complex 2.4 3 2.4
3. I thought the system was easy to use 4 3.4 4.2
4. I think that I would need the support of a technical person to be able to use this system
2.4 2.2 1.8
5. I found the various functions in this system were well integrated 4.4 3.4 3.8
6. I thought there was too much inconsistency in this system 2 2.6 1.4
7. I would imagine that most people would learn to use this system very quickly
4 3.2 4
8. I found the system very cumbersome to use 2.2 2.4 1.8
9. I felt very confident using the system 4 3.6 4
10. I needed to learn a lot of things before I could get going with this system
2.2 3.2 2
Average System Usability Score 73.5 58 77.5
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 11: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/11.jpg)
Overall Usability
• Summon– Pop-up abstract and number of results for limit in brackets
were very popular
• Encore – Worked very well for books– Article search interface (Synergy) not well understood or used
• EDS – Limits not intuitively named, extra “Update” click to refine– Can set mutually exclusive limits giving 0 results
• Preference to use web for finding newspaper articles
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 12: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/12.jpg)
Relevancy and Functionality
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 13: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/13.jpg)
Relevancy Test
• Testers - AUS public service librarians• Searches - Actual searches done this year in Encore• Scoring - Scale 1 - 5 + comments• Scorecards - One scorecard for each search in each
discovery tool. Extensive comments• Discovery tools tested
– Encore – Summon– EDS
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 14: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/14.jpg)
Actual SearchesSelected from Google Analytics
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 15: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/15.jpg)
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 16: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/16.jpg)
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 17: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/17.jpg)
Encore URLs
• /iii/encore/search/C|Scarbon+fiber+reinforced+polymer|Orightresult|U1?lang=eng&suite=pearl
• /iii/encore/search/C|Sglobalization|Orightresult|U1?lang=eng&suite=pearl
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 18: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/18.jpg)
Searches
• arranged+marriage• carbon+fiber+reinforced+polymer• Electronics+All-In-One+Desk+Reference+For+Dummies• electronics+for+dummies• globalization • jean+luc+godard• sex+education• uae+federal+laws• walt+disney• "Artificial+intelligence"+And+"Manufacturing"
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 19: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/19.jpg)
Types of Results
• Top 5 results– Number of:
• Books • Articles• Other (e-books, e-journals, DVDs, e-resources,
print journals, videos, etc)
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 20: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/20.jpg)
Types of ResultsBooks Articles Other
EDS 1 2 3 0EDS 2 0 0 5 reportsEDS 3 1 2 2 e-booksEDS 4 1 4 0EDS 5 0 5 0
Summon 1 0 5 0Summon 2 0 3 2 newspapersSummon 3 3 2 0Summon 4 1 3 1 e-journalSummon 5 0 5 0
Encore 1 3 2 0Encore 2 3 2 0Encore 3 1 2 2 e-booksEncore 4 2 2 1 e-journalEncore 5 3 2 0
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 21: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/21.jpg)
Relevancy Assessment
Overall reaction to top 5 results:
5 Excellent (just what I was looking for)4 Very good (on target and useful)3 Fair (acceptable)2 Poor (not useful)1 Bad (unrelated to topic)
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 22: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/22.jpg)
Test Searches Scorecard
Discovery Tool
Test Search
Top 5 results
Number of:
Books
Articles
Other (e-books, e-journals, DVDs, resources, print journals, videos, etc)
Relevancy assessment – overall reaction to top 5 results:
5 Excellent (just what I was looking for) 4 Very good (on target and useful) x 3 Fair (acceptable) 2 Poor (not useful) 1 Bad (unrelated to topic)
Comments:
Books were excellent, articles were poor.
Encore Synergy
3
2
arranged marriage
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 23: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/23.jpg)
ResultsTester 1 Tester 2 Tester 3 Tester 4 Tester 5 AVG Total Score
Summon 1 1 1 4 1 1 1.6 8Summon 2 3 3 3 4 3 3.2 16Summon 3 5 5 5 5 4 4.8 24summon 4 4 5 5 5 3 4.4 22Summon 5 4 5 4 5 3 4.2 21SUMMON TOTAL 17 19 21 20 14 18.2 91
EDS 1 4 4 4 4 3 3.8 19EDS 2 2 3 2 3 2 2.4 12EDS 3 4 4 4 4 4 4 20EDS 4 4 4 3 3 3 3.4 17EDS 5 4 4 4 4 3 3.8 19EDS TOTAL 18 19 17 18 15 17.4 87
Synergy 1 2 3 1 3 3 2.4 12Synergy 2 3 3.5 2 3 5 3.3 16.5Synergy 3 2 3 3 5 2 3 15Synergy 4 2 3 3 2 3 2.6 13Synergy 5 4 4 2 4 4 3.6 18SYNERGY TOTAL 13 16.5 11 17 17 14.9 74.5
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 24: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/24.jpg)
Functionality
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 25: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/25.jpg)
Functionality Conclusions
Basically, all 3 systems offer similar user functionality – sorting, limiting, emailing and “did you mean” are available. There is no superior feature that would make one system preferable to another, just as there is no missing feature that would eliminate a system.
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 26: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/26.jpg)
Impact Assessment
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 27: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/27.jpg)
Pieces & Configurations
A-Z List Link Resolver Discovery Federated Search
1 Existing 360 Core WebBridge Encore Synergy Research Pro
2 Summon 360 Core 360 Link Summon na
3 Summon 360 Core WebBridge Summon na
4 EDS EH A-Z LinkSource EDS
5 EDS 360 Core WebBridge EDS
New
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 28: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/28.jpg)
Functional Equalization
• EDS Federated Search• EDS & Summon – Millennium Integration Pieces• III Mobile Interface (free if Airpac)• III Encore Harvesting• III Query API – in development
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 29: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/29.jpg)
Costs
• Sunk Costs to write off– Encore Server– WebBridge License
• Ongoing Costs– Quotes for each component
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 30: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/30.jpg)
Work Needed for any Change• Implementation
– Profiling of resources and catalog records– Interface customization where possible– Changes to website– Changes to electronic resources– Staff training– Modify instruction materials and curriculum
• Ongoing Work– Bibliographic Data Loads– Changes to Holdings Loads for EDS
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 31: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/31.jpg)
Potential Lost Functionality
• Tag Clouds & Authorities for related searching• Availability Facet• WebBridge/Pathfinder for Collection Maps• User tags• Relevancy modification• Millennium integration
– Automatic update/syncing– Hold placement– Collection map links– My Lists– Saving RP Groups in My Millennium
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 32: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/32.jpg)
Potential Added Functionality• Administrative interface for searching & user interface• Mobile Interface bundled in• Repository Integration bundled in• API’s bundled in• 1-Click in 360 Link
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012
![Page 33: Discovery Tool Evaluation: A Three-Pronged Approach 9 th Annual AMICAL Meeting and Conference Sharjah, UAE.](https://reader034.fdocuments.in/reader034/viewer/2022051315/56649e425503460f94b344e1/html5/thumbnails/33.jpg)
Final Scoring
Factor Summon EDS Encore Weight Summon EDS Encore
Relevance 3 2 1 0.3 0.9 0.6 0.3
Usability 3 1 2 0.3 0.9 0.3 0.6
Financial Impact 3 1 2 0.2 0.6 0.2 0.4Implementation Work Impact 1 2 3 0.1 0.1 0.2 0.3
Ongoing Work Impact 2 1 3 0.1 0.2 0.1 0.3
12 7 11 2.7 1.4 1.9
Discovery Tool Evaluation | American University of Sharjah | 6.04.2012