Use, Usability, Usefulness and Impact of an Infobutton Manager
description
Transcript of Use, Usability, Usefulness and Impact of an Infobutton Manager
![Page 1: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/1.jpg)
Use, Usability, Usefulness and Impact of an Infobutton Manager
James J. Cimino, M.D.Department of Biomedical Informatics
Columbia University College of Physicians and Surgeons2006 AMIA Fall Symposium
![Page 2: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/2.jpg)
Evaluating the Infobutton Manager
• What are Infobuttons?
• What is the Infobutton Manager?
• How do we study it (modalities)?
• What did we learn?– How is it used?– How usable is it?– How useful is it?– What is its impact?
![Page 3: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/3.jpg)
![Page 4: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/4.jpg)
![Page 5: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/5.jpg)
![Page 6: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/6.jpg)
![Page 7: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/7.jpg)
Infobuttons vs. Infobutton Manager
Pageof
Hyperlinks
InfobuttonClinical System Resourc
e
InfobuttonManager
Context
s
![Page 8: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/8.jpg)
![Page 9: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/9.jpg)
![Page 10: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/10.jpg)
![Page 11: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/11.jpg)
![Page 12: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/12.jpg)
![Page 13: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/13.jpg)
![Page 14: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/14.jpg)
![Page 15: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/15.jpg)
![Page 16: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/16.jpg)
CPMC Infobutton Managers Deployed:
• New York Presbyterian Hospital’s WebCIS
• New York Presbyterian Hospital’s Eclipsys
• New York State Psychiatric Institute’s PSYCKES
• Regenstrief Medical Record System
• Crystal Run Healthcare’s NextGen
![Page 17: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/17.jpg)
Evaluation of InfobuttonsRosenbloom ST, Geissbuhler AJ, Dupont WD, et al. Effect of
CPOE user interface design on user-initiated access to educational and patient information during clinical care. JAMIA. 2005;12(4):458-73.
• 105 house officers studied over 11 months• 278 uses• Findings: 15-fold increased use of resourcesMaviglia SM, Yoon CS, Bates DW, Kuperman G. KnowledgeLink:
impact of context-sensitive information retrieval on clinicians' information needs. JAMIA. 2006;13(1):67-73.
• 359 users over 12 months• 7,972 uses• Findings: 84% found answers to their question,
positive impact on patient care 15% of the time
![Page 18: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/18.jpg)
Evaluation Modalities
• System log files
![Page 19: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/19.jpg)
Mar 5 06:46:03|xxx9017|156.145.18.47|lab^524732|view
Mar 5 06:46:33|xxx9017|156.145.18.47|healthresource|view
Mar 5 06:46:35|xxx9017|156.145.18.47|url^http://www.crlonline.com
Mar 5 06:47:21|xxx9017|156.111.245.19|LabDetail|ibutt^35776
Mar 5 06:47:24|xxx9017|156.111.245.19|LabDetail|Infomgr^70^35776
What does the CPMC Lab Manual say about the Serum Ferritin test?
![Page 20: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/20.jpg)
Evaluation Modalities
• System log files
• Pop-up surveys
![Page 21: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/21.jpg)
![Page 22: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/22.jpg)
Evaluation Modalities
• System log files
• Pop-up surveys
• On-line feedback
![Page 23: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/23.jpg)
![Page 24: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/24.jpg)
Evaluation Modalities
• System log files
• Pop-up surveys
• On-line feedback
• E-mail surveys
![Page 25: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/25.jpg)
1) How often do you use Infobuttons __ Once a month or less __ A few times a month __ A few times a week __ A few times a day-----------------------------------------------------------2) Infobuttons are easy to use: __ Strongly Agree __ Agree Somewhat __ Neither Agree nor Disagree __ Disagree Somewhat __ Strongly Disagree------------------------------------------------------------3) When I click on an Infobutton in WebCIS, the
question I am interested in is: __ Almost never on the list __ Usually not on the list __ Is only on the list about half the time __ Usually on the list __ Almost always on the list------------------------------------------------------------4) Compared to the way I usually get answers to
my questions, the Infobuttons are: __ Much slower __ Somewhat slower __ About the same speed __ Somewhat faster __ Much faster------------------------------------------------------------
5) When I click on a question on the question list __ I almost always get a helpful answer __ I usually get a helpful answer __ I get a helpful answer about half the time __ I usually do not get a helpful answer __ I almost never get a helpful answer------------------------------------------------------------6) Which statement most accurately describes your
experience with Infobuttons? __ Infobuttons almost always have a negative effect on
my patient care decisions __ Infobuttons sometimes have a negative effect on
patient care __ Infobuttons have no effect (positive or negative) on
my patient care decisions __ Infobuttons sometimes have a positive effect on my
patient care decisions __ Infobuttons almost always have a positive effect on
my patient care decisions------------------------------------------------------------7) Please tell us about a time that Infobuttons
improved a patient care decision (please do not include any information that would identify a patient)
------------------------------------------------------------8) Please tell us about a time that Infobuttons hindered
a patient care decision (please do not include any information that would identify a patient)
![Page 26: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/26.jpg)
Evaluation
• Log file analysis
• On-line survey
• Popup questionnaires
• E-mail survey
• Log file analysis: 32 months of data
![Page 27: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/27.jpg)
Results: System Log Files
• March 2004-October 2006:– IM: 3,009 users, 29,541 times
![Page 28: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/28.jpg)
Infobutton Manager (IM) Usage
0
100
200
300
400
500
600
700
800
900
1000A
ug-0
2
Nov
-02
Feb-
03
May
-03
Aug
-03
Nov
-03
Feb-
04
May
-04
Aug
-04
Nov
-04
Feb-
05
May
-05
Aug
-05
Nov
-05
Feb-
06
HousestaffAttendingNurseStudentOther
![Page 29: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/29.jpg)
Resource Use by Context
0
1000
2000
3000 Laboratory Results Review
0
500
1000 Inpatient Drug Order Review
0
50
100Microbiology Sensitivity Results
0
100
200
300Diagnosis List
0
20
40
60
80
100
120Inpatient Lab Order Entry
0
50
100
150
200 Inpatient Drug Order Entry
Health Resources Infobutton Manager
![Page 30: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/30.jpg)
Results: System Log Files
• March 2004 - October 2006:– IM: 3,009 users, 29,541 times
• October 2006:– IM: 281 users, 1,022 times (33/day)– HR: 708 users, 5,744 times (185/day)
– HR: 3,609 users, 155,718 times
![Page 31: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/31.jpg)
Resource Use by Context - October
0
500
1000
1500
2000
2500
LabResults
InpatientDrugs
OutpatientDrugs
OrderEntry
Infobuttons
HealthResources
![Page 32: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/32.jpg)
Finding a Resource or Question
• Users chose questions 48.7% of the timeDiagnosis list: 10.5-24.3%Outpatient drug order review: 23.0-63.3%Inpatient drug order review: 63.3-78.7%
• Users chose at least one resource from HR page 92.4% of the time (82.6-93.7%, depending on user type and context)
![Page 33: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/33.jpg)
Evaluation
• Log file analysis
• On-line survey
• Popup questionnaires
• E-mail survey
• Log file analysis: 32 months of data
• On-line survey: 108 comments
![Page 34: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/34.jpg)
Results: On-line Feedback
• 108 questions or topics submitted by users– 54 (50%) in Laboratory Results Review context
• 30 were about test interpretation– 22 (20%) in Inpatient Drug Order Review
• 15 were about side effects, dosage, cost, trade names, and interactions
![Page 35: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/35.jpg)
Evaluation
• Log file analysis: 32 months of data
• On-line survey: 108 comments
• Popup questionnaires:
• E-mail survey• Popup questionnaires: 195/~3,642 (5.4%)
![Page 36: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/36.jpg)
Pop-up Survey Responses
Q1: Easy to useQ2: Got answerQ3: Helpful
![Page 37: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/37.jpg)
Evaluation
• Log file analysis: 32 months of data
• On-line survey: 108 comments
• Popup questionnaires:
• E-mail survey• Popup questionnaires: 195/~3,642 (5.4%)
• E-mail survey: 73/1,228 (5.8%)
![Page 38: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/38.jpg)
E-mail Survey Responses
0%
20%
40%
60%
80%
100%A
ttend
ing
Hou
sest
aff
Nur
seS
tude
ntO
ther
Atte
ndin
gH
ouse
staf
fN
urse
Stu
dent
Oth
er
Atte
ndin
gH
ouse
staf
fN
urse
Stu
dent
Oth
er
Atte
ndin
gH
ouse
staf
fN
urse
Stu
dent
Oth
er
Atte
ndin
gH
ouse
staf
fN
urse
Stu
dent
Oth
er
Q2 Q3 Q4 Q5 Q6
5
4
3
2
1
Q2: Infobuttons are easy to useQ3: Presence of question on the listQ4: Speed of answerQ5: Helpful answerQ6: Positive effect on patient care decisions1 (strongly positive) to 5 (strongly negative)
![Page 39: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/39.jpg)
Results: Pop-Up and E-Mail Surveys
Pop-Up E-MailEasy to use 83% 92%Question on list >50% of time 89%Answered question 69%Useful 77%Helpful >50% of time 90%Positive effect on care 74%Specific cases of improved care 14(40% of negative responses due to Lab Manual)(Students were the least positive)
![Page 40: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/40.jpg)
Summary of Impact
• Over 29,000 additional accesses to resources (19%)
• Positive impact at least half the time: 74%
• 14 respondents identified one or more specific situations in which patient care improved
![Page 41: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/41.jpg)
Lessons Learned• Use of the IM has been more gradual than desired
• User fails to select a question half the time– Is the IM failing to anticipate the information need?– Example - questions sugested during lab review:
Housestaff did not select a question 57.8% of timeOnline survey: 4/11 dealt with reference rangesIM almost always provides a link to lab manual
– Is the user unable to find questions on the IM page?
• Solutions:– Education– Add questions– User interface
![Page 42: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/42.jpg)
![Page 43: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/43.jpg)
Conclusions
1. Context-specific access to health knowledge resources has been successfully accomplished
2. Impact on patient care decisions has been positive
3. Need to improve user interface for navigation
4. Increased resource use should result in clinicians making better informed patient care decisions
![Page 44: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/44.jpg)
Acknowledgments
• This work is supported by NLM grant R01LM07593• Evlauation plan:
– Vimla Patel– Sue Bakken– Leanne Currie– Beth Friedman
• Programming: Jianhua Li• Log files: Rick Gallagher
![Page 45: Use, Usability, Usefulness and Impact of an Infobutton Manager](https://reader031.fdocuments.in/reader031/viewer/2022013012/56815d37550346895dcb38f2/html5/thumbnails/45.jpg)
Questions?
www.dbmi.columbia.edu/cimino/amia06-app.pdf