Assessing an adaptive, profile-based PubMed search tool

16
Assessing an adaptive, profile-based PubMed search tool Marisa Conte [email protected] NLM Associate Fellow University of Michigan Health Sciences Libraries

description

Presentation at MLA annual conference on MiSearch, MLA ID 08-pap-479-mla

Transcript of Assessing an adaptive, profile-based PubMed search tool

Page 1: Assessing an adaptive, profile-based PubMed search tool

Assessing an adaptive,profile-based PubMed

search toolMarisa Conte

[email protected] Associate Fellow

University of Michigan Health Sciences Libraries

Page 2: Assessing an adaptive, profile-based PubMed search tool

MiSearch Adaptive biomedical literature search tool Utilizes implicit relevance feedback Builds statistical profile to predict which results will be most

relevant to user

Page 3: Assessing an adaptive, profile-based PubMed search tool

Objective To compare the performance of a locally-

developed third-party PubMed search toolto PubMed in terms of user satisfaction,search success and efficiency

Page 4: Assessing an adaptive, profile-based PubMed search tool

Collaboration

Page 5: Assessing an adaptive, profile-based PubMed search tool

Methodology N = 11 librarians (12 recruited) Tasks derived from mediated search files Each subject completed 2 tasks with each

search tool Each task completed by 3 subjects with

each search tool Pre-training conducted by tool developer

Page 6: Assessing an adaptive, profile-based PubMed search tool

Data collection Web-based surveys

Demographic survey Task-specific survey

PubMed search histories MiSearch data logfiles

Page 7: Assessing an adaptive, profile-based PubMed search tool

Sample questions In patients with diabetes mellitus, does

increasing hemoglobin A1c reduce the riskof amputation and kidney failure?

What are the attitudes, beliefs and socio-cultural factors that influence cervicalcancer screening in the US? What aboutspecifically among immigrant groups? AndI’m only interested in research from 1990 –present.

Page 8: Assessing an adaptive, profile-based PubMed search tool

Limitations Convenience sample Unclear survey questions Study tasks too complicated Problems with data collection

Page 9: Assessing an adaptive, profile-based PubMed search tool

Sample results – Query 2

1 = low21.7Completeness21.3Accuracy

1.71.3Speed1.71.3Tool1.71Search activity

PubMedMiSearchAveragesatisfaction

Page 10: Assessing an adaptive, profile-based PubMed search tool

Overall satisfaction with searches

0

10

20

30

40

50

60

% r

esp

on

ses

High Fair Low

User satisfaction

PubMed

Misearch

Page 11: Assessing an adaptive, profile-based PubMed search tool

Overall satisfaction with tools

0

10

20

30

40

50

60

70

% r

esp

on

ses

High Fair Low

User satisfaction

PubMed

MiSearch

Page 12: Assessing an adaptive, profile-based PubMed search tool

Suggestions Improve database response time Add more filters Add a “reset” function Make search details more transparent

Page 13: Assessing an adaptive, profile-based PubMed search tool

Next steps

Librarians suggested changes which: were easy to implement dramatically improved the tool’s functionality

First usability test – helped establish formalusability protocols and heuristics

New roles for librarians

Page 14: Assessing an adaptive, profile-based PubMed search tool

New roles for librarians Usability experts

Heuristic evaluation of tools Portal development Formal usability studies Weblog analyses

Education, promotion, dissemination Research collaborations

Page 15: Assessing an adaptive, profile-based PubMed search tool

Resources MiSearch: http://misearch.ncibi.org/ NCIBI: https://portal.ncibi.org/gateway/

States, D.J., Ade, A.S., Wright, Z.C., Bookvich,A.V. & Athey, B.D. (2008) Misearch AdaptivePubMed Search Tool. Bioinformatics.PMID 18326507

Page 16: Assessing an adaptive, profile-based PubMed search tool

Acknowledgments Jean Song, Nirit Glazer, Barbara Mirel, David States Librarians from UM’s Health Sciences and Shapiro

Science Libraries National Center for Integrative Biomedical Informatics

(NIH grant #U54DA021519) National Library of Medicine

This research was supported in part by an appointment to the NLMAssociate Fellowship Program sponsored by the National Library ofMedicine and administered by the Oak Ridge Institute for Science andEducation.