Evidence-based usability design principles for medication alerting … · 2018. 7. 24. · (MCBZ...
Transcript of Evidence-based usability design principles for medication alerting … · 2018. 7. 24. · (MCBZ...
![Page 1: Evidence-based usability design principles for medication alerting … · 2018. 7. 24. · (MCBZ and RM) decided on the final list of publications by consensus. Once relevant publications](https://reader035.fdocuments.in/reader035/viewer/2022071415/610f9fa4887af236364559bc/html5/thumbnails/1.jpg)
RESEARCH ARTICLE Open Access
Evidence-based usability design principlesfor medication alerting systemsRomaric Marcilly1* , Elske Ammenwerth2, Erin Roehrer3, Julie Niès4 and Marie-Catherine Beuscart-Zéphir1
Abstract
Background: Usability flaws in medication alerting systems may have a negative impact on clinical use and patientsafety. In order to prevent the release of alerting systems that contain such flaws, it is necessary to provide designersand evaluators with evidence-based usability design principles. The objective of the present study was to develop acomprehensive, structured list of evidence-based usability design principles for medication alerting systems.
Methods: Nine sets of design principles for medication alerting systems were analyzed, summarized, andstructured. We then matched the summarized principles with a list of usability flaws in order to determinethe level of underlying evidence.
Results: Fifty-eight principles were summarized from the literature and two additional principles were defined,so that each flaw was matched with a principle. We organized the 60 summarized usability design principlesinto 6 meta-principles, 38 principles, and 16 sub-principles. Only 15 principles were not matched with a usability flaw.The 6 meta-principles respectively covered the improvement of the signal-to-noise ratio, the support for collaborativeworking, the fit with a clinician’s workflow, the data display, the transparency of the alerting system, and the actionabletools to be provided within an alert.
Conclusions: It is possible to develop an evidence-based, structured, comprehensive list of usability design principlesthat are specific to medication alerting systems and are illustrated by the corresponding usability flaws. This list representsan improvement over the current literature. Each principle is now associated with the best available evidence of itsviolation. This knowledge may help to improve the usability of medication alerting systems and, ultimately, decrease theharmful consequences of the systems’ usability flaws.
Keywords: Human engineering, Usability, Alerting system, Decision support, Design
BackgroundMedication alerting systems “provide real-time notifica-tion of errors, potential hazards or omissions” related tothe prescription of medications, and thus help cliniciansto make informed decisions (nota bene: in the presentreport, a “clinician” is defined as any healthcare profes-sional who interacts with the patient; the term thereforeencompasses physicians, nurses and pharmacists) [1].These promising technologies can change prescribers’behavior by helping them avoid errors [2] and, ulti-mately, can improve the quality of the medication
management process [3]. Nonetheless, the design andthe implementation of these tools may introduce nega-tive, unforeseen side effects: poor integration into theclinical workflow [4], acceptance issues, and decreasedsafety and quality of care, for example [5]. Some of theseissues are related to the usability of the alerting systems[6]; they are caused by defects in the design of the sys-tem, i.e. usability flaws. For instance, alerts may bepoorly integrated into the workflow and may appear toolate in the decision-making process – rendering thealerting system useless [7, 8]. In other cases, the contentof the alert is either incomplete or not visible enough toadequately support a clinician’s decision making – lead-ing to incorrect clinical decisions [9]. This lack of infor-mation also increases the clinician’s cognitive load [10].Alerts may be poorly written or explained - causing
* Correspondence: [email protected]. Lille, INSERM, CHU Lille, CIC-IT / Evalab 1403 - Centre d’Investigationclinique, EA 2694, F-59000 Lille, France, Maison Régionale de la RechercheClinique, 6 rue du professeur Laguesse, 59000 Lille FranceFull list of author information is available at the end of the article
© The Author(s). 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Marcilly et al. BMC Medical Informatics and Decision Making (2018) 18:69 https://doi.org/10.1186/s12911-018-0615-9
![Page 2: Evidence-based usability design principles for medication alerting … · 2018. 7. 24. · (MCBZ and RM) decided on the final list of publications by consensus. Once relevant publications](https://reader035.fdocuments.in/reader035/viewer/2022071415/610f9fa4887af236364559bc/html5/thumbnails/2.jpg)
misunderstandings or at least creating difficulties in un-derstanding them. These cognitive issues may also leadto incorrect clinical decisions [11–13]. In summary,these and other usability flaws in the alerting systemmay have severe consequences, such as rejection of thealerting system, and incorrect clinical decisions. There-fore, the usability of an alerting system warrants specialscrutiny, with a view to avoiding usability-induced useerrors at least.To prevent the usability of alerting systems from intro-
ducing errors, usability activities (e.g. design specificationsand prototype evaluation) must be undertaken during thetechnology development process [14]. The implementa-tion of those activities requires a sound knowledge ofgood usability design principles (also known as usabilityheuristics and usability criteria). Violation of thoseprinciples may generate usability flaws in the technology.With a view to helping companies to avoid the release ofmedication alerting systems that contain unintentionalviolations of these principles, it is necessary to providedesigners and evaluators with easy access to relevant, illus-trated usability design principles and to convince them ofthe value of applying these principles to design decisions.In summary, designers and evaluators of medication alert-ing systems need to access evidence-based usability designprinciples, i.e. usability design principles that have proventheir value in practice [15]. As far as we know, the presentstudy is the first to have provided evidence-based usabilitydesign principles for medication alerting systems.Putting together a body of evidence relies on the accu-
mulation of results that demonstrate the positive valueof applying design principles. Unfortunately, publicationsin the field of usability evaluation tend to report onlynegative results, i.e. instances of usability flaws. Thisreporting bias prevents the collection of evidence toshow that applying principles is beneficial. Hence,although it is not yet possible to demonstrate the posi-tive value of applying usability design principles, it is still
possible to demonstrate the negative consequences ofviolating them.In previous research, we started to develop a usability
knowledge framework (Fig. 1; [16]). We have used thisframework to gather evidence-based usability designprinciples for medication alerting systems. In a first step,we performed a systematic review of the literature toidentify the usability flaws in medication alerting systemsused in hospital and/or primary care (active or passivealerts, and use as a standalone system or integrated intoa larger information system) [17]. In a second step, wesearched for the consequences of these flaws on users(usage problems; e.g. alert fatigue and missed informa-tion) and on the work system (negative outcomes; e.g. adecrease in effectiveness, and patient safety issues), andlinked them to their cause [6].The third step involves identifying, summarizing, and
organizing published design principles so as to avoid“reinventing” principles as far as possible. The fourthstep (in line with previous work by Nielsen [18]) seeksto match usability flaws to the usability design principlesthat could fix them and thus obtain empirical illustra-tions of the principles’ violation. The present study tack-led the third and fourth steps. The results will help toestablishing evidence in support of these principles.The present study had two objectives. Firstly, it sought
to identify and organize literature reports of usability de-sign principles for medication alerting systems in hos-pital or primary care settings into a comprehensive,structured list of design principles. Secondly, the studysought to match this list with the set of usability flawsidentified in the systematic review [17], in order to as-sess the fit between known usability flaws and knownexisting design principles and thus illustrate violations ofthese principles.
MethodsA two-step methodology was applied.
Fig. 1 Top: a graphical representation of the evidence-based usability knowledge framework. The numbering refers to the four steps, asdescribed in the text. The question marks refer to the steps tackled in the present study. Bottom: an instance of the cause-consequence chainlinking a usability flaw, a usage problem and a negative outcome (adapted from [27])
Marcilly et al. BMC Medical Informatics and Decision Making (2018) 18:69 Page 2 of 17
![Page 3: Evidence-based usability design principles for medication alerting … · 2018. 7. 24. · (MCBZ and RM) decided on the final list of publications by consensus. Once relevant publications](https://reader035.fdocuments.in/reader035/viewer/2022071415/610f9fa4887af236364559bc/html5/thumbnails/3.jpg)
Gathering and structuring usability design principlesWe searched peer-reviewed journals and conference pa-pers for published consensus sets of usability designprinciples for medication alerting systems (i.e. principlesthat experts in the field had agreed on). The “grey” litera-ture was excluded because the quality of the informationmay vary. Hence, we searched PubMed, Scopus, andErgonomics Abstracts databases for articles addressingboth “medication alerting systems” and “usability” topics.With this goal in mind, we used the screening and eligibil-ity assessment steps from our previous systematic review[17] to identify papers purposefully providing at least oneusability design principle dedicated to medication alertingsystems. We excluded system-specific papers providingrecommendations on improving usability because theseprinciples are not applicable to a broad range of systems.This task was updated on March 30th 2016. The literaturesearch was intended to provide an overview of publishedsets of usability design principles for medication alertingsystems, rather than being systematic and reproducible.The database search was completed by examining the in-vestigators’ personal libraries and by screening the refer-ences of the selected publications. Two investigators(MCBZ and RM) decided on the final list of publicationsby consensus.Once relevant publications had been identified, one in-
vestigator (RM) extracted all items referring to usabilitydesign principles from each publication. Next, the inves-tigator grouped together principles with similar purposesand organized them hierarchically. A second investigator(MCBZ) independently crosschecked the hierarchicalorganization of the principles. Disagreements weresolved by discussion until a consensus was reached.Lastly, the two investigators summarized the principlesthat had been grouped together.
Matching usability design principles to known usabilityflawsOne investigator (RM) checked the list of usability flawspublished in the on-line appendices of Marcilly et al. [17]against the structured list of usability design principles. Asecond investigator (MCBZ) crosschecked the results. Dis-agreements were discussed until a consensus was reached.The items referring to usability flaws were either descrip-tions of the technology’s defects observed during fieldstudies or usability tests, answers to interviews/question-naires, or users’ positive or negative comments about thecharacteristics of the technology collected during theirinteraction with the system. A usability flaw was matchedto a given usability design principle if it was an instance ofa violation of the said principle. Reciprocally, a usabilitydesign principle matched a usability flaw if the applicationof the principle stopped the flaw from occurring. If a flawdid not match any of the usability design principles, then
we considered the possible extension (broadening) of anexisting principle to other contexts so that it covered awider range of flaws. If no principles could be extended tocover the flaw, we defined a new principle.The matching process was intended to be as unequivocal
as possible, i.e. one flaw matched one principle. However, ifa given usability flaw violated several principles (e.g. atdifferent levels of granularity), we matched that flaw to themost significantly violated principles (based on our experi-ence). It should be noted that a given principle could bematched with several instances of the same flaw.Figure 2 illustrates the matching process. Both investi-
gators performed the descriptive analysis of the matches.
ResultsGathering and structuring usability design principlesWe identified 9 publications on design principles dedi-cated to medication alerting systems (Table 1).Figure 3 describes the sets of publications analyzed.
One publication (Zachariah et al. [22]) was included inboth sets; although this publication was an extension ofanother set of design principles described by Phansalkaret al. [20], it contained a few usability design principlesnot found in the original publication [20]. The publica-tion also gave a list of usability flaws detected usingheuristics. Despite the potential for self-matching bias,this publication was included because our objective wasto obtain the most comprehensive possible list of designprinciples. Moreover, it was found that virtually all firstauthors of the set of usability design principles wereco-authors of one or more studies included in the reviewof usability flaws (e.g.[1, 4, 20, 23]).A total of 345 items referring to usability design princi-
ples were extracted from the 9 publications (see Add-itional file 1: Appendix 1) and then organized. The level ofagreement between the two investigators regarding theorganization of the items was very high, with full agree-ment for 92.6% of the combinations, discussion neededfor 6%, and disagreement for 1.3%. After a consensusmeeting, the items were summarized into 58 principles.No significant inconsistencies between principles from dif-ferent publications were noticed. The summarized princi-ples displayed different granularity levels, and some weremore tangible and precise than others; they could there-fore be organized hierarchically into 6 meta-principles, 36principles, and 16 sub-principles (Fig. 4, Table 2).Overall, the 9 publications contributed to different ex-
tents to the set of summarized principles: the contribu-tions ranged from 12% [22] to 69% [24] (see Additionalfile 2: Appendix 2).The level of support for each of the summarized prin-
ciples (in terms of number of publications in which theywere found) varied: one summarized principle was sup-ported by all 9 papers (#49 “Include actionable tools
Marcilly et al. BMC Medical Informatics and Decision Making (2018) 18:69 Page 3 of 17
![Page 4: Evidence-based usability design principles for medication alerting … · 2018. 7. 24. · (MCBZ and RM) decided on the final list of publications by consensus. Once relevant publications](https://reader035.fdocuments.in/reader035/viewer/2022071415/610f9fa4887af236364559bc/html5/thumbnails/4.jpg)
within the alert”), “Suggest, do not impose” (#42) wassupported by 8 publications, and 6 principles were foundin 1 publication. When considering the overallmeta-principles and their components (i.e. related prin-ciples and sub-principles), the level of support rangedfrom 5 publications for meta-principle E (“Make the sys-tem transparent for the user”) to 9 publications formeta-principle D (“Display relevant data within thealert”) and meta-principle F (“Include actionable toolswithin the alert”).
Matching usability design principles with known usabilityflawsThe two investigators agreed well on the matchingbetween usability flaws and usability design principles,with full agreement for 54.6% of matches, partial
agreement for 31% (agreement on the main corre-sponding principle but a need to match the flaw witha second principle), discussion needed for 13.4%, anddisagreement for 1%. Of the 58 principles, 34 directlymatched at least one instance of a usability flaw, ninewere broadened to cover a flaw, and 15 were notmatched at all (see Additional file 3: Appendix 3).Two new principles were defined so that all flawsmatched a principle (#46, provide “a description ofthe characteristics of the tools included in the alert”to users, and #57, include a “send the alert into theclinical note template” function in the alert).After the addition of the 2 new principles, the final set
comprised 60 summarized usability design principles: 6meta-principles, 38 principles, and 16 sub-principles.The 6 meta-principles were as follows:
Fig. 2 Illustration of the matching process, using meta-principle E (#44) and one of its sub-principles (#48). The usability design principles foundin the literature were summarized and organized hierarchically (left). The usability flaws identified in the systematic review were collated by topic(right). Next, the correspondence between a given type of flaw and a given summarized principle was established based on the principle’s ability(if applied) to fix the usability flaw. This correspondence is represented by a double arrow. When a usability flaw could not be fixed by any of thedesign principles in the literature, we either extended an existing principle or created a new one (single arrow). The illustration presents anextension of principle #48 (in italics)
Table 1 Main characteristics of the publications on usability design principles
First author Year Focus Method used to provide the principles
[2] Bates DW 2003 Design, implementation, monitoring Lessons learned
[19] Kuperman GJ 2007 Design, implementation Expert consensus
[4] Sittig DF 2008 Design, implementation, research Lessons learned / expert consensus
[20] Phansalkar S 2010 Usability Targeted review
[21] Pelayo S 2011 Usability Targeted review & analysis of cognitive andcollaborative tasks
[22] Zachariah M 2011 Development of a usability evaluationinstrument
Phansalkar et al.s’ review and feedback froma preliminary evaluation
[1] Horsky J 2012 Usability Targeted review
[23] Horsky J 2013 Usability Targeted review
[24] Payne T 2015 Usability Expert consensus
Marcilly et al. BMC Medical Informatics and Decision Making (2018) 18:69 Page 4 of 17
![Page 5: Evidence-based usability design principles for medication alerting … · 2018. 7. 24. · (MCBZ and RM) decided on the final list of publications by consensus. Once relevant publications](https://reader035.fdocuments.in/reader035/viewer/2022071415/610f9fa4887af236364559bc/html5/thumbnails/5.jpg)
A. Improve the system’s signal-to-noise ratio, inorder to decrease the frequency of over-alerting. Inaddition to the drugs ordered, the alert strategyshould take into account parameters such as thepatient’s clinical context or the clinician’s specialty.Moreover, the system must provide tools tocustomize the knowledge implemented within itand to monitor alert overrides.
B. Support collaborative work, advocate a teamapproach, and make the system a team player.The alerting system must encourage collaborationbetween the healthcare professional managingmedications (e.g. physicians, pharmacists andnurses). Overall, alerts must deliver the sameinformation to all clinicians, even if additionalsupplementary data can be presented as a functionof the healthcare professional’s role. The alertingsystem must help clinicians to understand howother healthcare professionals have alreadymanaged the alert.
C. Fit with clinicians’ workflow and their mentalmodel. The alerting system must comply withclinicians’ needs and tasks. Alerts must bepresented at the right moment in the decision-making process. Only the most severe alerts mustinterrupt the users; other alerts must be displayedmore discreetly. Alerts must be concise, under-standable and consistently structured so that userscan easily find the relevant data. Once the alert hasbeen satisfied, the clinicians must be able to resumetheir tasks easily.
D. Display relevant data within the alert. Thesystem must provide clinicians with the
information needed to make informed decisions.This includes the cause of the unsafe event (themedications involved), the description of theunsafe event, the severity/priority of the event,the mechanism of the interaction, the patient’sclinical context, and evidence supporting thealert. Lastly, the system must suggest – but notimpose – a means of remedying or monitoringthe unsafe event.
E. Make the system transparent for the user. Thealerting system must help clinicians to understandwhat the system can and cannot do and how itworks, in order to prevent erroneous interpretationof its behavior. The user must have access to (i) thetypes of data that are checked, (ii) the formulas andrules applied, (iii) the list of the unsafe events thatare targeted, and (iv) a description of the alerts’levels of severity.
F. Include actionable tools within the alert. Thealert must provide several tools that help cliniciansto easily and quickly translate their alert-informedclinical decision into actions: for example, buttonsto modify/cancel/discontinue an order or overridethe alert, to order actions for monitoring an event,and to provide patient education. Other tools arerecommended for managing the alert: pulling upthe alert at a later time, sending the alert into aclinical note, removing the alert for a patient, andgaining access to the patient’s medical records.
The final list of summarized usability design principlesis given in Fig. 4. Table 2 provides a detailed version ofthe principles and corresponding flaws.
DiscussionAnswers to study questionsThe present study sought primarily to provide a specific,comprehensive, structured list of usability design princi-ples for the medication alerting systems implemented inhospital or primary care settings. The secondary object-ive was to pair this list with the set of documented us-ability flaws, assess the match between the usabilityflaws that are known and the existing design principles,obtain illustrations of the existing violations of the prin-ciples, and present evidence that not applying usabilitydesign principles may be detrimental.A total of 60 specific usability design principles for
medication alerting systems were identified andorganized hierarchically around 6 meta-principles: (A)improve the signal-to-noise ratio, (B) support collabora-tive work, (C) fit the clinicians’ workflow and theirmental model, (D) display relevant data within thealert, (E) make the system transparent for the user,and (F) include actionable tools within the alert. The
Fig. 3 Sets of papers analyzed. Left: the set of papers analyzed toestablish a structured list of usability design principles formedication alerting systems. Right: the set of papers analyzed toestablish the list of usability flaws in medication alerting systems [17]
Marcilly et al. BMC Medical Informatics and Decision Making (2018) 18:69 Page 5 of 17
![Page 6: Evidence-based usability design principles for medication alerting … · 2018. 7. 24. · (MCBZ and RM) decided on the final list of publications by consensus. Once relevant publications](https://reader035.fdocuments.in/reader035/viewer/2022071415/610f9fa4887af236364559bc/html5/thumbnails/6.jpg)
Fig.4
Hierarchicalo
rganizationof
themeta-principles,p
rinciples,and
sub-principles
specifically
relatedto
med
icationalertin
gsystem
s.Meta-principles
aredisplayedat
thetopin
coloredbo
xes.
Principles
arepresen
tedin
thelinkedcoloredbo
xes.Sub-principles
arepresen
tedin
thebo
rder-free
areasbe
low
theprinciples
Marcilly et al. BMC Medical Informatics and Decision Making (2018) 18:69 Page 6 of 17
![Page 7: Evidence-based usability design principles for medication alerting … · 2018. 7. 24. · (MCBZ and RM) decided on the final list of publications by consensus. Once relevant publications](https://reader035.fdocuments.in/reader035/viewer/2022071415/610f9fa4887af236364559bc/html5/thumbnails/7.jpg)
Table
2Summarized
usability
design
principles,and
descrip
tions
ofthemaincorrespo
ndingflaws.Principles
andsub-principles
arepresen
tedrespectivelyas
firstandsecond
inde
nts.Principles
that
have
been
adde
dor
extend
edto
completethematchingprocessaregivenin
italics.Theob
lique
barsindicate
theabsenceof
correspo
ndingflaws
Usabilityde
sign
principles
Summaryof
correspo
ndingflaws
Meta-princ
iple
A:
#1Im
prove
thesigna
l-to-no
iseratioby
improvingthesensitivity
andthespecificity
ofthealertin
gsystem
inorde
rto
decrease
thenu
mbe
rof
irrelevantalerts[1,4,19,20,23]
(e.g.,system
(non
-med
ical)alerts,alertswith
little
eviden
ce,low
clinicalrelevanceor
redu
ndantalerts,alertsthat
require
noaction).
Therearetoomanyalerts[8,25–29]someareredu
ndant[10,25,27,30,31]
orirrelevant[11,31],
othe
rdo
notne
edanyactio
n[29].Poten
tialeventsareover-or
unde
r-de
tected
[10]
dueto
sensitivity/spe
cificity
issues
[32]
orinapprop
riate
trigge
ringthresholds
[11].
#2Che
cktheaccu
racy
oftheinform
ationretrieve
dfro
mtheCPO
E(Com
puterized
PhysicianOrder
Entry)/EHR(ElectronicHealth
Record)[19],che
ckwhe
ther
they
are
outdated
and/or
reconciliated
[1,23].
#3Use
adeq
uate
eviden
ce-based
alertkn
owledg
ebase[19].Itshou
ldbe
regu
larly
up-dated
/maintaine
d[1,2](see
#13).
Alertsareinconsistent
with
EHRdata
[10]
espe
ciallywith
listsof
patient’sactualmed
ications
[10,
12,32,33]or
with
patient’sdiagno
ses[34].
Med
ications
interactions
high
lighted
bythealertsareun
know
nin
pharmaceuticalreferencebo
oks
[27].Kno
wledg
esupp
ortin
gthealertsisno
tup
dated[34].
#4Increa
sethevarietyof
thesourcesof
inform
ationused
inthetrigge
ringmod
el(e.g.severalallergybases[19])a
ndreconcile
multip
lesen
tries[1];whe
ndata
are
missing
(deg
rade
dcond
ition
s),the
system
mustcontinue
tofunctio
n[1].
/
#5Con
sider
temporal
dim
ension
s:intervalbe
tweendrug
s’administration[23]:
distingu
ish“now
”,“stand
ing”,and
“future”orde
rs,evolutio
nof
theun
safe
even
t:increase
theseverityof
theun
safe
even
tifitge
tsworse
[21],tim
elabtestsare
overdu
e[1,19],intervalbetweentheappearance
oftheunsafeeventandthe
administrationofdrugs.
Thealertisirrelevantbe
causetheadverseeffect
itpresen
tshapp
enstoofastto
bemanageable
[29].The
system
does
notdistingu
ishorde
rsspecified
as“now
”andthosespecified
as“fu
ture”
or“stand
ing”
[31].
#6Con
sider
patient
clinical
context[1,19,23,24]:b
esides
thespecificdrug
regimen
(s)(e.g.
dose,rou
te,d
urationof
therapy,sequ
ence
ofinitiatingco-the
rapy,
timingof
co-adm
inistration),add
patient
andlabo
ratory
data
into
theexpe
cted
interaction(e.g.age
,gen
der,bo
dyweigh
t,mitigatin
gcircum
stances,pred
ispo
sing
risks
factors,drug
serum
level,renalfun
ction,co-m
orbidity,and
previous
expe
riences).Co
nsider
thepointduringpatient’sstay
atwhich
thealertispresented.
Med
icationorde
rchecking
isno
tpatient
tailored[26,35]:alertsmay
bevalid
butno
tapplicable
topatient
clinicalcontext[10,31,36]:e.g.pregn
ancy
alertsformalepatientsandwom
enof
non-child-bearin
gage[32],nodistinctionbe
tweentrue
allergiesandside
effects[10,27].An
alertthat
issupp
osed
toappe
arthelastdayof
thestay
(which
isun
foreseeable)
appe
arsevery
day[34].
#7Con
sider
action
salread
ytake
nbytheprovide
r(e.g.d
oseadjustmen
t)[21]
andprovider-specificdata[1,23]
(e.g.clinicalspecialty:som
edrugsmay
beused
off-label,o
thersmay
bevoluntarily
duplicated
trigge
ring“dup
licateorde
rs”[19]).
Alertsappe
arwhilethecorrectiveor
mon
itorin
gactio
nshave
alreadybe
entakenby
the
physicians
[29].Som
ecorrectiveactio
nsthat
areclinicallyrelevant
areno
taccepted
bythe
system
[36].Insomespecialties,adverse
even
tsareintentional[29,32]
(e.g.p
sychiatry[32]);in
othe
r,drug
sareused
off-label(e.g.p
ediatrics[32]).Clinicians
alreadyknow
thealerts[25,29].
#8Includ
efuzzylogic-based
algorithm
s,multi-attributeutility
mod
elandfilters
into
thetrigge
ringmod
elto
change
alerts’activationwhe
ncertaincond
ition
sapply
[1,19,20].Defineapprop
riatelythresholds
totrigge
rthealertsandto
prioritizethe
alertsaccordingto
thepatient’sclinicalcontextandtheseverityof
theun
safe
even
t[1,4,20,23](see
#28).
Alertstrigge
ringthresholds
aretoolow
[26].A
lertsareno
torde
redby
severitylevel[22].
Non
-significant,or
low
incide
nce,alertsarepresen
ted[11,12,29].
#9Com
binealerts
[1]:
/
#10Re
commen
dations
mustbeco
mbined
inaco
nsistent
way
forpatients
withco
morbidities[4].
/
#11Aggreg
atelow
seve
rity
alerts
inasing
ledisplayto
bereview
edallat
once
ataconven
ient
pointin
theworkflow
[1,20,23].
Alertsareno
tgrou
pedaccordingto
theirseverity[22].
#12Su
pportmon
itoring
oftheusag
eof
alerts
andtheircustom
ization
[1,2,19,23,24]
/
Marcilly et al. BMC Medical Informatics and Decision Making (2018) 18:69 Page 7 of 17
![Page 8: Evidence-based usability design principles for medication alerting … · 2018. 7. 24. · (MCBZ and RM) decided on the final list of publications by consensus. Once relevant publications](https://reader035.fdocuments.in/reader035/viewer/2022071415/610f9fa4887af236364559bc/html5/thumbnails/8.jpg)
Table
2Summarized
usability
design
principles,and
descrip
tions
ofthemaincorrespo
ndingflaws.Principles
andsub-principles
arepresen
tedrespectivelyas
firstandsecond
inde
nts.Principles
that
have
been
adde
dor
extend
edto
completethematchingprocessaregivenin
italics.Theob
lique
barsindicate
theabsenceof
correspo
ndingflaws
(Con
tinued)
Usabilityde
sign
principles
Summaryof
correspo
ndingflaws
#13Allowexpe
rtcommitteesin
each
orga
nizatio
nto
adap
tthekn
owledg
e-ba
sean
dsugg
estio
nsforactio
nto
localp
ractices
andgu
idelines
andto
removepo
tential
errorsfro
mthebase.Customizations
shou
ldpersist
acrossversionup
grades
[19,23,24].
Alertsarein
conflictswith
localand
common
practices
[10,27,36].
#14Mon
itor
alerts’ov
erridereason
s:alertsfre
quen
tlyoverrid
denandof
little
valueshou
ldbe
considered
forremovalor
forachange
oftheirpresen
tatio
nform
at(e.g.intrusivene
ss)[1,2,19,23,24].H
owever,d
ono
telim
inateor
turn
off
relevant
alertseven
forspecialists[24].
/
#15Allo
winstitutiona
lflexibility
indetermininginterrup
tive
vs.
non-interrup
tive
alerts
[23,24].
/
Meta-princ
iple
B:
#16Su
pportthecollaborativework.
Advocateateam
approach
[24],m
akethealertin
gsystem
ateam
player
[21].
Thesystem
does
notinform
physicians
whe
ther
pharmacistsreview
theirjustificatio
nof
overrid
eand/or
findthem
useful
[10].
#17Prov
idefunc
tion
sto
supporttheteam
awaren
essof
thealertman
agem
ent
[21,24]:(see
#56).
/
#18Sign
altheavailabilityof
inform
ationto
allu
sers[21]
(evento
non-prescribing
clinicians
[24]).
/
#19Displayov
erride
reason
sen
teredby
aph
ysicianto
nurses
andph
armacistsin
orde
rto
allow
them
toun
derstand
theratio
naleforoverrid
ing[1,19,21,23,24].
Justificatio
nsandcommen
tsaredisplayedto
noon
e[37]
#20Display
alerts
toconcerne
dclinicians
andthen
tono
n-prescribingclinicians
asa
second
check[24].Redirectalertsthatdo
notconcernph
ysicians
tosupp
ortstaff[1].
Pharmacistsreceivealertsthat
concernph
ysicians
(e.g.d
ruginteraction[38]).Ph
ysicians
receive
alertsrelatedto
drug
sadministrationthat
concernnu
rses
[11,29].Ph
ysiotherapistsandnu
rses
receivealertsrelatedto
theorde
ringof
drug
swhilethey
dono
tprescribe[26].
#21Displayco
nsistently
thebasicalertco
nten
t,i.e.the
mainelem
ents
ofthe
alert,am
ongst
allclin
icians
[21,24].Non
ethe
less,the
detailedpresen
tatio
nmay
differb
ased
onclinicians’expertise(e.g.pharm
acistsmay
need
moreph
armacolog
ical
data)[1,23],on
theirrole(privileges,respon
sibilities)andon
thecontextof
use[24].
Detailsmay
bepresentedup
onrequ
est[21].
Theway
data
aredisplayedisno
tadeq
uate
forallclinicians’types
[10].
#22Sh
arepatient
clinical
inform
ationin
thealertsummaryscreen
withall
clinicians
(e.g.w
ithph
armacists)[1,4].
/
Meta-princ
iple
C:
#23Fittheclinicians’w
orkflowan
dtheirmen
talm
odel
[1,2,4,20].
Men
talm
odelim
plem
entedin
thesystem
does
notfit
users'on
e[12].
#24Displaythealertat
theap
propriatetimedu
ringthede
cision
making[1,2,24]
orlaterdu
ringthemed
icationmanagem
entprocess[19].
Alertsappearou
tof
thelogicalw
orkflow[30],eith
ertoolate
intheorderingprocess[7,8,25,31,
38,39]or
tooearly
[7,31].
#25Alertsmustbedisplaye
dan
dfully
accessible
quickly:screentransitio
ntim
emustbe
wellu
nder
asecond
[1,2],avoidscrolling
andtabs
[24].
Alertsappearance
isdelayed[12]:lags/do
wn-tim
esof
8sec.[28]up
to15
sec.[10].Clinicians
must
exploreseveralpartsof
thealertsto
getallrelevantd
ata[9]:they
mustscroll[10,11,22],explore
severaltabs[9,40],orfindinform
ationintooltips[26]becauseshortversio
nsof
alertsareno
tsufficiently
inform
ative[31].
#26Displayalerts
over
theCPO
E/EH
Rscreen
incloseproxim
ityto
therelevant
controlsanddisplays
[20,23].
Alertsareou
tsidetheregion
ofthescreen
whe
reclinicians
arelooking[26,34].
Marcilly et al. BMC Medical Informatics and Decision Making (2018) 18:69 Page 8 of 17
![Page 9: Evidence-based usability design principles for medication alerting … · 2018. 7. 24. · (MCBZ and RM) decided on the final list of publications by consensus. Once relevant publications](https://reader035.fdocuments.in/reader035/viewer/2022071415/610f9fa4887af236364559bc/html5/thumbnails/9.jpg)
Table
2Summarized
usability
design
principles,and
descrip
tions
ofthemaincorrespo
ndingflaws.Principles
andsub-principles
arepresen
tedrespectivelyas
firstandsecond
inde
nts.Principles
that
have
been
adde
dor
extend
edto
completethematchingprocessaregivenin
italics.Theob
lique
barsindicate
theabsenceof
correspo
ndingflaws
(Con
tinued)
Usabilityde
sign
principles
Summaryof
correspo
ndingflaws
#27Stream
lineusers’workflow
inrespon
seto
alerts
[22].M
aketheresolutio
nof
alertsqu
ickandeasy
(fewsteps)throug
hscreen
operations
[1,22,24]:cancelor
reset
alertsinrespon
seto
theapprop
riate
correctiveactio
n,do
notrequire
acknow
ledg
ment
beforeacorrectiveaction[20].Afterthe
alertisresolved,resum
etheworkflow[19](see
#49).
/
#28Adap
tthedisplayof
thealertto
itstype
(med
icationalerts,system
alerts)and
itsseverity[19,20,23,24].
Alerts
ofdifferent
severitylevelsandof
different
typesareno
tdistingu
ished
[22].
#29Adap
talert’sform
at(e.g.color,sym
bol)andlocatio
non
thescreen
[20,24].
Moresevere
alertsmustbe
placed
with
inthefocalreg
ionof
theuser’svisualfield
inorde
rof
impo
rtance
whileno
nsevere
alertsmustbe
placed
inside
region
s[1,20,23].Distin
guishsystem
vs.m
edicationalertmessage
s[20].
Alertsareno
tdistingu
ishe
dby
severityno
rby
type
[10,22,32].A
llalertslook
thesame[22]
#30Adap
talert’sintrusiven
ess[1,4,19,21,2324].Interrup
tivealertsshou
ldbe
reserved
forh
ighseveritywarning
andused
judiciou
sly:theyshou
ldrequ
irean
explicit
respon
se.Lessimpo
rtantalertsmustbe
displayedlessintrusively(e.g.on-demand)
asmessagesno
trequ
iring
anyactio
ns.D
onotu
sepop-up
alertsforsystem
messages.
Highriskalertsareno
tseen
whenno
tintrusive[31].O
nthecontrary,low
riskalertsthatpo
p-up
anno
yusers.
Moreover,no
n-medicationalertsaretoointrusive[30]andcontrib
uteto
desensitize
theusers[10,27].
#31Use
aco
nsistent
structured
alerttemplate
across
thevariou
ssystem
sused
bytheclinicians
[23,24].
Alertscombine
dbu
tno
tstructured
causevisualizationdifficulties
[10,12,27,34]:usersface
difficulties
tofindspecificdata
[11].The
lack
ofgu
idance
bother
usersandtheirun
derstand
ing
[30,35,41].U
sefulo
ndem
andinform
ationavailablein
EHRisno
tavailablein
thealerts[22].
#32Makethealertconcisean
dactio
nable;thede
scrip
tionof
theprob
lem
shou
ldbe
shorterthan10
words
[1,19,24,24];labelsof
button
mustb
econcise
too[1](see#49).Alertscontaintoomuchtext
orextraneo
usinform
ation[10,25,30,41].
#33Presen
tthemostcriticalinform
ationon
thetop-levelo
fthealert:theun
safe
even
t,its
causes
andits
severity[1,23,24].Th
endisplayon
-dem
and(link
ed)
inform
ationon
backgrou
ndandsecond
aryconsiderations
(con
textualinformation,
mechanism
ofinteractionandeviden
ce[19,24].Thesugg
estio
nof
actio
ncouldbe
presen
tedeither
atthetoplevelo
ron
-dem
and[20,24](see
#36).
Nodataarehigh
lighted
with
inthealert[9,10],the
alertison
aparagraphform
[41].
#34Use
consistent
term
s,ph
rases,classifications,colorsandde
finition
s(e.g.for
theseverity)[1,21,22].
/
#35Te
rminolog
yan
dmessages
shou
ldsuitlocalcon
ventions
[23]
andbe
unde
rstand
ableandno
n-am
bigu
ous[1,23].
Themessage
conveyed
bythealertisno
tun
derstand
able[10,12,40]:the
text
isam
bigu
ous
[41].Icons
used
aremisinterpreted[28,41]as
wellasbu
tton
slabe
ls[35].
Meta-princ
iple
D:
#36Displayrelevant
datawithinthealert[1,20,23,24]
(see
#33).For
therelevant
toolsto
prop
ose,seemeta-principles
F./
#37Th
ecauseof
theun
safe
even
tan
dits
characteristics(e.g.dose)[1,19,20,23].U
sethemed
icationnameas
orde
redas
wellasge
neric
drug
names
whe
niden
tifying
the
interaction[24],d
ono
tfocuson
pharmacolog
ical/therapeutic
classes[24].
Alertdo
esno
tprovideinform
ationon
why
itistrigge
red[10,12,27].
#38Th
eun
safe
even
t(potentialorcurrentlyhapp
ening)
[19,20,23,24].Dono
tuse
generic
term
(e.g.risk),preferconcrete
description[24].Present
thefrequ
ency
orincidence
oftheun
safeevent[24].
Alertdo
esno
tinform
ontheun
safe
even
t[9,10,12,27].
#39Th
eseve
rity
/priorityof
theun
safe
even
t[1,20,23,24]:use
colorcode
anda
sign
alwordto
inform
ontheseverity[20].
Alertdo
esno
tinform
ontheseverityof
theun
safe
even
t[9,10,27].
#40Th
emecha
nism
oftheinteraction(possiblyby
embe
dded
links)[23,24].
Alertdo
esno
tprovideinform
ationon
how
thecauses
oftheun
safe
even
tcond
uctto
the
unsafe
even
t[9].
Marcilly et al. BMC Medical Informatics and Decision Making (2018) 18:69 Page 9 of 17
![Page 10: Evidence-based usability design principles for medication alerting … · 2018. 7. 24. · (MCBZ and RM) decided on the final list of publications by consensus. Once relevant publications](https://reader035.fdocuments.in/reader035/viewer/2022071415/610f9fa4887af236364559bc/html5/thumbnails/10.jpg)
Table
2Summarized
usability
design
principles,and
descrip
tions
ofthemaincorrespo
ndingflaws.Principles
andsub-principles
arepresen
tedrespectivelyas
firstandsecond
inde
nts.Principles
that
have
been
adde
dor
extend
edto
completethematchingprocessaregivenin
italics.Theob
lique
barsindicate
theabsenceof
correspo
ndingflaws
(Con
tinued)
Usabilityde
sign
principles
Summaryof
correspo
ndingflaws
#41Re
levant
datasupporting
thedecision-makingprocess
andthesugg
estio
nsof
actio
n[1,4,23,24]:e.g.
contextualinform
ation,mod
ifyingandpred
ispo
sing
factors
(e.g.co-morbidityor
lab-values).Providealinkto
asummaryof
patient
clinicaldata[23].
Disp
laydata
necessaryto
interpretvalues
provided
(e.g.thresholds).
Thealertdo
esno
tprovideessentialp
atient
inform
ationfortheprescriber
[10].U
sercanlinkto
outsidesourcesof
inform
ationfro
melsewhe
rein
thesystem
,but
thereisno
linkwith
inthealert
[22].Evenwhe
nalertsprovidepatient
biolog
icalresults,the
thresholds
tointerpretthem
are
notpresen
ted[9].
#42Su
ggest,dono
tim
pose.
Makethesystem
aclinician’spartne
r[21]:p
rovide
clinicallyapprop
riate
sugg
estio
nsof
actio
nformitigatin
gthepo
tentialh
arm,d
ono
tim
pose
[1,19,20,23,24].Presen
tpo
ssibleancillary
orde
rssuch
asmon
itorin
g/surveillanceactio
ns,d
rugalternative(incl.D
oseandfre
quen
cy)and/or
orde
rmod
ificatio
nor
cancellatio
n[1,2,23,24].Makethesugg
estio
nsactio
nable[24]
(see
meta-principleF).Incase
ofmultip
lesugg
estio
ns,p
rioritizethem
andpresen
ttheircon
ditio
nsof
application[24].Justifysugg
estio
ns[21,24]:checklocally
sugg
estio
ns[24]andinclud
elinkto
institutio
n-specificgu
idelines
[19],m
akeconsensualsugg
estio
ns[1,19].M
onito
rwhether
orno
tusersfollowed
throug
hwith
thesugg
estedactio
nthey
started;ifusersfailno
tifythem
they
didno
tfinish
theactio
nthey
started[22].
Alertsdo
notprovidesugg
estio
nsof
actio
n[10,22,29,41]no
ralternativetreatm
entop
tions
[9].
Alertsprovideerrone
oussugg
estio
nsof
actio
n[35].
#43Ev
iden
cesupporting
thealert(incl.Stren
gthandsource)usingsymbo
ls/le
tter/
numbers[24].Include
alinkto
amorecompletedo
cumentatio
n(m
onog
raph
,evidence,
extend
edinform
ation,context)[1,2,19,23,24].
Alertsdo
notprovideexistin
geviden
ceor
theeviden
cethat
supp
ortsthealertispo
orand
contradict
clinicians’kno
wledg
e[10,27,29].The
alertdo
esno
tpresen
teviden
cereferences
[9].
Meta-princ
iple
E:#4
4Makethesystem
tran
sparen
tfortheuser.The
system
mustno
tbe
ablackbo
xandits
coverage
mustbe
accessibleto
itsuser
[19].Inform
userswhe
nne
cessarydata
aremissing
(e.g.severity
oftheun
safe
even
t)[24].M
akeaccessible:
Thede
cision
supp
ortisinvisibleto
users[42]:capabilitiesandlim
itatio
nsof
thesystem
alon
gwith
type
sof
data
that
arecheckedareno
tshow
nto
theusers[10,27,42].
#45Th
ealerting
algorithm
/logic/form
ulas
implemen
tedwithinthesystem
[1,20,23].
Noalertsareappe
aringafterorde
ringmed
ications
althou
ghclinicians
expe
cton
eto
comeup
forapatient
[12].The
calculationform
ulas
that
thesystem
appliesareno
tun
derstood
byclinicians
[7].
#46Ade
scriptionof
thecharacteristicsof
thetoolsinclud
edin
thealert
(e.g.d
urationof
activ
ation).
Thesystem
isno
texplicitabou
tho
wto
useandmanagealertseffectively[31]:itdo
esno
tmakeitclearthat
onecanturn
offsomealerts[10]
andforho
wmuchtim
e[36].
#47Explana
tion
son
thegradingsystem
s:levelsof
severityused
bythealertin
gsystem
(and
theirnu
mbe
rby
unsafe
even
t)[20,24],andexplanations
oftheir
classificationas
unsafe
even
ts[20]).
Thesystem
does
notexplainthelevelsof
severitythat
areused
[22].
#48Th
eev
ents
that
arech
ecke
dbythealerting
system
[20]
andthetype
and
form
atof
data
(e.g.,free-text,origin
ofthedata
(otherho
spitaldata),nam
eof
drugsvs.
Anatom
icalTherapeutic
Chem
ical-ATC
codes)
Thesystem
does
notexplainwhich
interactions
areactuallychecked[10],w
hich
patients’data
areanalyzed
[31],w
hether
orde
rsin
freetext
arechecked[10]
andwhe
ther
checking
isbased
ondrug
s’names
oron
ATC
code
s[13].
Meta-princ
iple
F:#4
9Includ
eaction
able
toolswithinthealertto
allow
clinicians
totake
actio
nsintuitively,easily
andqu
ickly[1,2,4,19–24];displaythosetoolscloseto
thesugg
estio
nsthey
arerelatedto
[23]
(see
#27and#32).The
listof
actio
nabletoolsshou
ldinclud
e:
Therearede
aden
dsin
which
clinicians
face
noreason
ableop
tions
toproceed[28];the
reare
nouseful
actio
nableop
tions
[22].
#50Mod
ifytheorde
rbe
ingen
tered(orits
dose)[1,22–24].Forinstance,p
ropo
seform
ularydrug
slists[19]forformularydrug
alerts.Alloworderingadrug
sugg
ested[23]
oranewdrug
:inthiscase,clearlystatethattheexistingdrug
willbe
discon
tinuedifthe
newon
eisfinalized
[23],openapre-po
pulatedorderin
gscreen
forthe
newdrug
[23].
Thesystem
does
notgu
ideusersforsw
itching
amed
ication[41].
#51Disco
ntinue
thepreexistingactivedrug
[22–24].
Thesystem
does
notgu
ideuser
fordiscon
tinuing
amed
ication[41].
#52Can
celthe
existin
g/ne
worde
r[1,22–24].
/
Marcilly et al. BMC Medical Informatics and Decision Making (2018) 18:69 Page 10 of 17
![Page 11: Evidence-based usability design principles for medication alerting … · 2018. 7. 24. · (MCBZ and RM) decided on the final list of publications by consensus. Once relevant publications](https://reader035.fdocuments.in/reader035/viewer/2022071415/610f9fa4887af236364559bc/html5/thumbnails/11.jpg)
Table
2Summarized
usability
design
principles,and
descrip
tions
ofthemaincorrespo
ndingflaws.Principles
andsub-principles
arepresen
tedrespectivelyas
firstandsecond
inde
nts.Principles
that
have
been
adde
dor
extend
edto
completethematchingprocessaregivenin
italics.Theob
lique
barsindicate
theabsenceof
correspo
ndingflaws
(Con
tinued)
Usabilityde
sign
principles
Summaryof
correspo
ndingflaws
#53Order
labtestsor
action
sformon
itorin
gas
justified
bythealert[1,24].
/
#54Prov
idepatient
educ
ation[24].
/
#55Delay
thealertforapred
etermined
amou
ntof
time(“sno
oze”
functio
n)[24],
allowusersto
getthealertagain.
Alertscann
otbe
pulledup
later,hind
eringalertresolutio
n[10,36].Moreo
ver,itisno
tpo
ssible
toge
tthealertagain[11].
#56Se
ndthealertto
anothe
rclinician[24].
Thesystem
does
notsupp
ortthetransm
issionof
alertsto
othersclinicians
[28].
#57Send
thealertinto
theclinical
note
template
Thesystem
does
notallow
clinicians
tosend
thealertsinto
atemplateforpatient’srecord
[28,41].
#58Ove
rridethealert(m
eaning
continue
orde
ring,
igno
ringthealert)[22–24].Most
severe
unsafe
even
tsmustbe
moredifficultto
overrid
e(e.g.req
uire
asecond
confirm
ation,or
even
nopo
ssibilityof
overrid
ing[23])thanlesssevere
ones.Alertsmust
requ
irethereason
foro
verride[24](especially
themostcriticalones,op
tionalotherwise
[23].Avoidtextentries,prop
osealistof
3–4(m
ax5items)selectablecodedreason
s;reason
smustbe
1–2wordlong
[19,23,24].
Thesystem
does
notprovideapprop
riate
optio
nsforjustifyingoverrid
es[28].The
system
isno
texplicitabou
tthene
cessity
toen
terajustificatio
n[37];m
oreo
ver,fre
e-text
entriesareno
teffectivein
theoverrid
ejustificatio
nlogic[10]
anden
terin
gdata
tojustify
overrid
esisseen
astim
ebu
rden
[10,36].
#59Allo
wprovidersto
remov
eredun
dant
alerts
forapatient
who
haspreviously
toleratedadrug
scombinatio
n(after
morethan
oneoverrid
e)or
whe
nprovidersfeel
they
have
sufficien
tpracticeandknow
ledg
eabou
tthisalertor
whe
nthealertis
outdated
foraspecificpatient
[1,23].
Thesystem
does
notprovideusersthepo
ssibility
toremoveirrelevantalertsthat
therefore
continue
toappe
ar[28].
#60Allo
wusersaccess
easily
patient’sreco
rdfrom
thealertscreen
tochange
errone
ousdata
(e.g.allergy)or
toaddne
wdata.D
ono
trequ
ireen
terin
gadditio
nal
data
inthealert[1,19].
Thesystem
asks
theusersto
enterdata
inthealertandthen
inthepatient
record,leading
towastin
gtim
eanddo
ubledo
cumen
tatio
n[28].
Marcilly et al. BMC Medical Informatics and Decision Making (2018) 18:69 Page 11 of 17
![Page 12: Evidence-based usability design principles for medication alerting … · 2018. 7. 24. · (MCBZ and RM) decided on the final list of publications by consensus. Once relevant publications](https://reader035.fdocuments.in/reader035/viewer/2022071415/610f9fa4887af236364559bc/html5/thumbnails/12.jpg)
9 analyzed publications contributed to this list to dif-ferent extents; we consider that the collation of sev-eral sets of usability design principles found in theliterature expands the variety of topics represented ineach individual set.The match between the summarized usability design
principles and the list of documented usability flaws wasquite good: 34 principles were directly matched, and thecontext of application was extended for 9 principles.Nonetheless, 15 principles did not match any of the doc-umented usability flaws. In view of the hierarchicalorganization of the principles, some principles are alsonot matched because their meta-principle or one ormore of their sub-principles are matched - thus artifi-cially reducing the quality of the match. We also identi-fied limited gaps in the principles found in the literature;two new principles had to be created.From a qualitative point of view, a few instances of
usability flaws appear to contradict the correspondingusability design principles. For instance, some princi-ples recommend including non-prescribers (e.g. phar-macists and nurses) in the alert management process,in order to promote collaboration between healthcareprofessionals (e.g. #20). However, it has been reportedthat nurses are annoyed by medication alerts thatinterrupt their work [26]. The balance between pro-moting collaboration between healthcare professionalsand not disrupting non-prescribers’ tasks is delicate.Overall, instances of usability flaws must be used sothat the corresponding design principles are not takentoo literally.
Study strengthsThe results of the present study represent an improve-ment with respect to the current literature. We did notchange the principles extracted from the literature. Bycombining and summarizing the extracted principles,they are now clearly identified, listed, and organizedhierarchically into a comprehensive, consistent, andstructured hierarchy. Furthermore, the process ofmatching the principles to the usability flaws allows oneto identifying evidence to show that not applying theseprinciples has negative consequences. Each principle isnow associated with the best available evidence of itsviolation. As far as we know, the present study is thefirst to have drawn up this type of list.In addition to providing evidence, the matching
process also provided concrete illustrations of violationsof usability design principles. The illustrations may helppeople designing and evaluating alerting systems toidentify the “usability mistakes” that should not be madeor to catch these mistakes during the evaluation phases.In fact, the illustrations provide a clearer understandingof the design principles to be applied.
Study limitationsThe retrieval of the usability design principles mighthave biased the representativeness of the principlesand the flaws. We considered only publicationsreporting general sets of design principles, rather thanevaluations giving system-specific usability recommen-dations. Grey literature was excluded. Moreover, mostof the analysis was performed by one investigator,with a second investigator independently crosscheck-ing the results. Together, these biases might havecaused us to miss a few relevant principles. Conse-quently, the principles that we extended or created inthe present study may have already been described inother publications (e.g. as system-specific recommen-dations on usability). Likewise, some usability flawsmight have been missed during the systematic review[17] due to publication and reporting biases: it mighthave been possible to match principles not matchedin the present study with usability flaws documentedoutside our review [17]. Despite these limitations, thematch between the usability design principles and theusability flaws was quite good and ensured that theprinciples and flaws retrieved were representative.This good level of matching might be due (at least inpart) to the inclusion of Zachariah’s publication [22]and reports written by closely linked authors in bothsets of publications (i.e. the set used to establish thelist of principles and the set used to establish the listof flaws, e.g. [4, 20, 23]). In the present study, therisk of self-matching bias was considered to be ac-ceptable because our objective was to obtain the mostcomprehensive possible list of design principles andcorresponding flaws. On the contrary, not including apublication in one set because its authors had alsoworked on a publication included in the other setcould have led us to ignore relevant usability designprinciples and/or usability flaws.The frequency of appearance of the design principles
was analyzed in order to establish the level of supportfor the design principles (i.e. the number of publicationsthey were found in). However, we do not interpret thisnumber as an indicator of which principles should beprioritized. Firstly, reporting and publishing biases anddifferences in the focus of the publications analyzed mayhave biased the frequency of appearance. Even withoutthese biases, prioritizing the principles would imply thatwe are able to predict the severity of the consequencesof the related usability flaws. However, the severity de-pends on many other factors, such as the system’s otherfeatures and other flaws, and the context of use. This isone reason why most sets of design principles - whetherdeveloped for interactive systems (e.g. Nielsen’s [43] andScapin’s [44] sets) or for a specific type of technology(e.g. the ones included in our analysis) - do not prioritize
Marcilly et al. BMC Medical Informatics and Decision Making (2018) 18:69 Page 12 of 17
![Page 13: Evidence-based usability design principles for medication alerting … · 2018. 7. 24. · (MCBZ and RM) decided on the final list of publications by consensus. Once relevant publications](https://reader035.fdocuments.in/reader035/viewer/2022071415/610f9fa4887af236364559bc/html5/thumbnails/13.jpg)
Fig.5
Norman’s“seven
stages
ofactio
n”mod
el,asappliedto
theinteractionwith
amed
icationalertin
gsystem
.Top
:Loo
pa(ontheleft;“display/read”thealert)represen
tsthecore
interaction.
Loop
b(“ackno
wledg
emen
t”)represen
tsasecond
orde
rinteraction:no
tallalertingsystem
srequ
irealertsto
beacknow
ledg
ed.U
sabilityde
sign
principles
that
could(if
applied)
improvethequ
ality
oftheinteractionarelinkedto
thecorrespo
ndingstageof
themod
el,w
henpo
ssible
Marcilly et al. BMC Medical Informatics and Decision Making (2018) 18:69 Page 13 of 17
![Page 14: Evidence-based usability design principles for medication alerting … · 2018. 7. 24. · (MCBZ and RM) decided on the final list of publications by consensus. Once relevant publications](https://reader035.fdocuments.in/reader035/viewer/2022071415/610f9fa4887af236364559bc/html5/thumbnails/14.jpg)
principles. Design principles can be prioritized by a per-son who is aware of the alerting system’s characteristicsand context of use.In the present study, we addressed the evidence in
favor of usability design principles by examining theviolation of these principles. Evidence to suggest thatapplying design principles is beneficial has not yetbeen considered, due to reporting bias in the litera-ture. Even though our present evidence is not basedon instances of successful design, it may be convin-cing enough to persuade designers to apply usabilitydesign principles. Once researchers have begun toreport on the positive usability characteristics ofmedication alerting system, the present analysis willhave to be updated.
The significance of the present results for a userinteracting with a medication alerting systemUsability design principles are related to various compo-nents of the alerting system: the triggering model, theknowledge implemented, the cognitive model implementedin the system, the information displayed, and the tools pro-posed within the alert. Applying these usability design prin-ciples might improve the clinician-alerting systeminteraction and the collaboration between clinicians. Ac-cording to Norman’s “seven stages of action” model [45],the user’s interaction with a system encompasses twostages: the action stage translates a goal into an actionsequence, and the evaluation stage compares the changesperceived in the world with the initial goal of the action(see Fig. 5). A clinician interacts with an alerting system inorder to check the appropriateness of the prescriptions(step 1). Two “action and evaluation loops” may then bedescribed. The main loop is “display/read” the alert. Thesecond “acknowledgement” loop depends on the alertingsystem model; in some models, acknowledgment is notrequired.For the “display/read” loop (loop a, Fig. 5, left), improving
the overall usability of the alerting system by applying thewhole set of design principles may facilitate the interactionand increase the clinician’s intention to use the alerting sys-tem (step 2). More specifically, the whole “improving thesignal-to-noise ratio” meta-principle may help to improvethe relevance of the alerts, decrease alert fatigue, and thusincrease the clinician’s will to use the alerting system. Instep 3, principles such as “signal the availability of informa-tion to all users” (#18) and “display the alert at the appro-priate time” (#24) could make it easier to notice andretrieve alerts. In step 4, applying the “fit the clinician’sworkflow” meta-principle may help the clinician to displaythe alerts. Once alerts are displayed, clinicians have to readand interpret them (in steps 5 and 6). Applying the “use aconsistent structured alert template”, “display relevantdata”, and “make the system transparent” principles (#31,
#36, and #44, respectively) may make the alerts morereadable and help the clinicians to interpret them. Lastly,“extend[ing] the sources of information used in the trigger-ing model” (#4) may make it easier for clinicians to assessthe alerts’ relevance (step 7).Once alerts are interpreted, physicians may have to
acknowledge them (loop b, Fig. 5, right). Applying the“suggest - do not impose” principle (#42) may increasethe probability with which a clinician acknowledges thealert and perform corrective actions (step 2). Next,“includ[ing] actionable tools within the alerts” (#49) maymake it easier and quicker to specify and executecorrective actions (e.g. modify the order; step 3). If aphysician overrides an alert and enters the reason why,“display[ing] override reason” (#19) might help otherclinicians to interpret the alert’s acknowledgement status(step 6) and decide whether or not the alert has beenproperly assessed.In summary, applying this set of usability design princi-
ples might improve both the action and evaluation stagesof a user’s interaction with the alerting system - mainly inthe “display/read” loop but also in the “acknowledgement”loop. Some principles go beyond Norman’s model, whichrelates to an individual’s interaction with the alertingsystem and not interactions between clinicians or theclinicians’ workflow. Adhering to the “fit the clinicians’workflow” meta-principle might decrease the risk of rejec-tion. Moreover, if the “support collaborative work”meta-principle were to be applied, the alerting systemcould truly help clinicians to gain the same mental repre-sentation of the prescription being checked; this wouldhelp them to coordinate their actions and improve patientsafety.
Generalizability of the studyThe list of usability flaws used in the matching processmight increase over time, depending on whether newpublications report usability flaws. Moreover, technologyevolves rapidly, and the related principles might changeaccordingly. For instance, the principles presented hereare formulated for medication alerting systems imple-mented on laptop and/or desktop computers. However,as mobile health technologies are refined and expanded,alerting systems will be progressively installed on mobiledevices. This might modify the applicability of the us-ability design principles listed here. It will therefore beessential to update this work regularly and take accountof the latest trends and developments. However, themaintenance of this knowledge may be time-consuming,and represents a challenge for human factors specialistsin the field of medical informatics. Manufacturers shouldbe associated with this process.Some design principles insist on the need for promot-
ing collaboration between clinicians (#16) but ignore the
Marcilly et al. BMC Medical Informatics and Decision Making (2018) 18:69 Page 14 of 17
![Page 15: Evidence-based usability design principles for medication alerting … · 2018. 7. 24. · (MCBZ and RM) decided on the final list of publications by consensus. Once relevant publications](https://reader035.fdocuments.in/reader035/viewer/2022071415/610f9fa4887af236364559bc/html5/thumbnails/15.jpg)
key person in the medication management process - thepatient. Only principle #54 mentions the patient (beingable to “provide patient education”). However, as forother information technologies, the implementation ofan alerting system changes the nature of thepatient-clinician interaction [46]. It is important to en-sure that poor usability has not damaged thepatient-clinician interaction. On the contrary, increasedusability should underpin patient-clinician discussion,empower the patient [47], and ensure that care remainspatient-centered. The current literature on the usabilityof medication alerting systems does not consider the pa-tient as a stakeholder in medication management. Futureresearch on the usability of medication alerting systemshould integrate patients as stakeholders in medicationmanagement, so as to adapt or extend usability designprinciples to their specific features.Although the structured design principles target
only medication alerting systems implemented inhospital or primary care settings, some principles maybe applied to other kinds of alerting systems. For in-stance, part of the “fit the clinicians’ workflow”meta-principle could also be applied to laboratory re-sult alerting systems. Nonetheless, the evidence thatunderpins the principles presented here is valid formedication alerting systems only.In addition to the results, the method used to build
this set of evidence-based usability design principlescould also be applied to evidence-based usability designprinciples for other kinds of technology. However, thismethod is very time-consuming, and requires in-depthknowledge of the usability of the technology in questionif the data are to be analyzed correctly.
Turning the results into a usable, practical tool fordesigners and evaluatorsThe present set of evidence-based usability design princi-ples for medication alerting systems must be made access-ible to and usable by designers and evaluators. At present,the principles are presented as a printable table (Table 2)that might not be ideal for optimal use. We intend to usethe table to develop tools that present the evidence-basedknowledge in a way that suits the needs of the various sys-tem designers and evaluators (usability experts, computerscientists, etc.) in various contexts of use (design,evaluation, procurement processes, etc.). With that aim inmind, we have started to identify the needs of medicationalerting system designers and evaluators [48]. Accordingly,we developed (i) a checklist that measure the appropriateuse of evidence-based principles in the design of medica-tion alerting systems, and (ii) a set of interactive designinstructions illustrated by visual representations of goodand bad usability practices, in order to help designers makeinformed design decisions.
This list of usability design principles should help de-signers to make evidence-based usability design decisions.Nonetheless, and even though we believe that the list ishelpful, it is not intended to be used as a stand-alone sys-tem or to replace the requirement for expertise in usabilityand design. Firstly, the present list does not include gen-eral design principles for unspecified interactive systems;it must therefore be used in combination with sets ofgeneral usability design principles for interactive systems(e.g. [43, 44]). Secondly, several principles require insightsinto the users’ cognitive tasks and their decision-makingprocesses in order to adjust (for instance) an alert’s formatand the moment at which it appears (e.g. #24). Hence,work system and cognitive work analyses [49] must beperformed so the principles are applied in an optimal way.Thirdly, principles moderate each other; they must not beapplied alone or in an unquestioning manner. Humanfactors specialists and designers must use their expertiseto determine which principles must be applied and howthey must be applied, given the characteristics of the aler-ting system and the setting in which it is implemented in(hospital vs. primary care, for example). In summary, thisstructured list of usability design principles must be usedas a support for expertise and not as a substitute for it.Applying some of the principles listed here may
present specific technical and organizational challengeswhen seeking to tailor alerts. For instance, the “prioritizethe alerts according to patient’s clinical context and theseverity of the unsafe event” (#8) principle requiresaccess to valid data on the patient’s clinical context, stay,and treatment. However, these data are often not stan-dardized or structured enough to be used in the alertingsystem’s set of rules [50]. Further research is needed toovercome these challenges.Ultimately, presenting designers and evaluators with
evidence-based knowledge may help to decrease theoccurrence of unforeseen and potentially harmfulusability-induced use errors. Nonetheless, one must beaware that improving the usability of an existing system orensuring that the usability of a system under developmentis optimal is no guarantee of success. Other issues arisingduring the development of a medication alerting system(e.g. an error-ridden knowledge base, a poor implementa-tion process, unsuitable settings, etc.) can ruin even opti-mal levels of usability. Even though it is necessary toconsider usability during the design, development, andevaluation of medication alerting systems, one must neverneglect the relevant technical, social, and managerial fac-tors that also contribute to the system’s success or failure.
ConclusionsIn the present study, we developed an evidence-based,structured, specific, comprehensive list of usability designprinciples for medication alerting systems, and then
Marcilly et al. BMC Medical Informatics and Decision Making (2018) 18:69 Page 15 of 17
![Page 16: Evidence-based usability design principles for medication alerting … · 2018. 7. 24. · (MCBZ and RM) decided on the final list of publications by consensus. Once relevant publications](https://reader035.fdocuments.in/reader035/viewer/2022071415/610f9fa4887af236364559bc/html5/thumbnails/16.jpg)
illustrated them with the corresponding usability flaws. Thislist should help designers and usability experts to gain abetter understanding of usability design principles. We ex-pect that the list can be used during the design and evalu-ation processes of medication alerting systems, in order toprevent usability issues that could have a counterproductiveimpact on clinicians (e.g. alert fatigue) and potentiallyharmful outcomes for patients (e.g. errors in medicationdosing). Although operational barriers may complicate thedeployment and maintenance of the evidence-based usabil-ity design principles presented in the present study, our re-sults show that the approach is feasible. Indeed, ourapproach could be transferred to other health informationtechnologies for the generation of specific lists ofevidence-based usability design principles. In this way, de-signers and evaluators could be provided with tools to helpthem avoid usability design issues in health informationtechnology and thus decrease the likelihood of unforeseenand potentially harmful usability-induced use errors.
Additional files
Additional file 1: Appendix 1. List of usability design principlesidentified in the 9 papers and the corresponding usability design principlessummarized (for definitions, please refer to Table 2). (DOCX 248 kb)
Additional file 2: Appendix 2. The 9 papers’ contributions to thesummarized principles. Crosses show that a given principle is mentioned in apaper. The right-hand-most column gives the number of papers mentioning agiven principle. The bottom two rows present the number of principles men-tioned by each paper and the proportion of the full list of principles mentionedby each paper. It should be noted that the percentages are based on the 58principles summarized in step 1. Principles #46 and #57 were created after thematching process and therefore were not included here. (DOCX 56 kb)
Additional file 3 Appendix 3. Results of the matching betweeninstances of usability flaws (from Marcilly et al. [17]) and the usabilitydesign principles summarized in the present study. (DOCX 82 kb)
AbbreviationsATC: Anatomical Therapeutic Chemical; CPOE: Computerized Physician OrderEntry; EHR: Electronic Health Record
AcknowledgementsThe authors would like to thank the staff at the University of Lille 2 library fortheir very efficient work in retrieving the required publication. The authorwould like to thank Melissa Baysari for her feedback on the wording of Table2, Emmanuel Castets and Pierre-François Gautier for the figures’ design. Fi-nally, the authors would like to thank the reviewers and editors for their con-structive comments.
Availability of data and materialsThe dataset(s) supporting the conclusions of this article is(are) includedwithin the article (and its additional file(s)).
Authors’ contributionsRM designed the study, retrieved the data, performed the analysis and wrotethe paper. EA helped to design the study, provided methodological supportand supported the writing of the paper by reading it several times andproviding advice to improve the report of the study. ER supported thewriting of the paper by reading it several times and validating it.Additionally, ER checked English spelling and grammar. JN provided amethodological support and supported the writing of the paper by readingit several times and providing advice to improve the report of the study.MCBZ retrieved the data, performed the analysis and supported the writing
of the paper by reading it several times and providing advice to improve thereport of the study.All authors approved the present version of the paper.
Ethics approval and consent to participateNot applicable.
Competing interestsThe authors declare that they have no competing interests.
Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.
Author details1Univ. Lille, INSERM, CHU Lille, CIC-IT / Evalab 1403 - Centre d’Investigationclinique, EA 2694, F-59000 Lille, France, Maison Régionale de la RechercheClinique, 6 rue du professeur Laguesse, 59000 Lille France. 2Institute ofMedical Informatics, UMIT – University for Health Sciences, MedicalInformatics and Technology, 6060 Hall in Tirol, Austria. 3eHealth ServicesResearch Group, School of Engineering and ICT, University of Tasmania,Private Bag 87, Hobart, Tasmania 7001, Australia. 4General Electric HealthcarePartners, 92772, Boulogne Billancourt cedex, France.
Received: 22 June 2017 Accepted: 23 May 2018
References1. Horsky J, Schiff GD, Johnston D, Mercincavage L, Bell D, Middleton B.
Interface design principles for usable decision support: a targeted review ofbest practices for clinical prescribing interventions. J Biomed Inform. 2012;45:1202–16.
2. Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, Spurr C,Khorasani R, Tanasijevic M, Middleton B. Ten commendments for effectiveclinical decision support: making the practice of evidence-based medicine areality. J Am Med Inform Assoc. 2003;10:523–30.
3. Jaspers MW, Smeulers M, Vermeulen H, Peute LW. Effects of clinicaldecision-support systems on practitioner performance and patientoutcomes: a synthesis of high-quality systematic review findings. J Am MedInform Assoc. 2011;18:327–34.
4. Sittig DF, Wright A, Osheroff JA, Middleton B, Teich JM, Ash JS, Campbell E,Bates DW. Grand challenges in clinical decision support. J Biomed Inform.2008;41:387–92.
5. Campbell EM, Guappone KP, Sittig DF, Dykstra RH, Ash JS. Computerizedprovider order entry adoption: implications for clinical workflow. J GenIntern Med. 2009;24:21–6.
6. Marcilly R, Ammenwerth E, Roehrer E, Pelayo S, Vasseur F, Beuscart-ZephirMC. Usability flaws in medication alerting systems: impact on usage andwork system. Yearb Med Inform. 2015;10:55–67.
7. Horsky J, Kaufman DR, Patel VL. Computer-based drug ordering: evaluationof interaction with a decision-support system. Stud Health Technol Inform.2004;107:1063–7.
8. Ash JS, Sittig DF, Campbell EM, Guappone KP, Dykstra RH. Someunintended consequences of clinical decision support systems. AMIA AnnuSymp Proc. 2007:26–30.
9. Duke JD, Bolchini DA. Successful model and visual design for creatingcontext-aware drug-drug interaction alerts. AMIA Annu Symp Proc. 2011:339–48.
10. Russ AL, Zillich AJ, McManus MS, Doebbeling BN, Saleem JJ. Prescribers'interactions with medication alerts at the point of prescribing: a multi-method, in situ investigation of the human-computer interaction. Int J MedInform. 2012;81:232–43.
11. van der Sijs H, van Gelter T, Vulto A, Berg M, Aarts J. Understanding handlingof drug safety alerts: a simulation study. Int J Med Inform. 2010;79:361–9.
12. Russ AL, Saleem JJ, McManus MS, Zillich AJ, Doebbling BN. Computerizedmedication alerts and prescriber mental models: observing routine patientcare. Proc Human Factors Ergon Soc Annu Meet. 2009;53:655–9.
13. Hartmann Hamilton AR, Anhoj J, Hellebek A, Egebart J, Bjorn B, Lilja B.Computerised physician order entry (CPOE). Stud Health Technol Inform.2009;148:159–62.
Marcilly et al. BMC Medical Informatics and Decision Making (2018) 18:69 Page 16 of 17
![Page 17: Evidence-based usability design principles for medication alerting … · 2018. 7. 24. · (MCBZ and RM) decided on the final list of publications by consensus. Once relevant publications](https://reader035.fdocuments.in/reader035/viewer/2022071415/610f9fa4887af236364559bc/html5/thumbnails/17.jpg)
14. International Standardization Organization Ergonomics of human systeminteraction - part 210: human centered design for interactive systems (rep N°9241-210). Geneva: international standardization Organization; 2010.
15. Marcilly R, Peute L, Beuscart-Zephir MC. From usability engineering toevidence-based usability in health IT. Hat evidence supports the use ofcomputerized alerts and prompts to improve clinicians' prescribingbehavior? Stud Health Technol Inform. 2016;222:126–38.
16. Marcilly R, Beuscart-Zephir MC, Ammenwerth E, Pelayo S. Seeking evidenceto support usability principles for medication-related clinical decisionsupport (CDS) functions. Stud Health Technol Inform. 2013;192:427–31.
17. Marcilly R, Ammenwerth E, Vasseur F, Roehrer E, Beuscart-Zephir MC.Usability flaws of medication-related alerting functions: a systematicqualitative review. J Biomed Inform. 2015;55:260–71.
18. Nielsen J. Enhancing the explanatory power of usability heuristics. CHI '94proceedings of the SIGCHI conference on human factors in. Comput Syst.1994:152–8.
19. Kuperman GJ, Bobb A, Payne TH, Avery AJ, Gandhi TK, Burns G, Classen DC,Bates DW. Medication-related clinical decision support in computerizedprovider order entry systems: a review. J Am Med Inform Assoc. 2007;14:29–40.
20. Phansalkar S, Edworthy J, Hellier E, Seger DL, Schedlbauer A, Avery AJ, BatesDW. A review of human factors principles for the design andimplementation of medication safety alerts in clinical information systems.J Am Med Inform Assoc. 2010;17:493–501.
21. Pelayo S, Marcilly R, Bernonville S, Leroy N, Beuscart-Zephir MC. Humanfactors based recommendations for the design of medication relatedclinical decision support systems (CDSS). Stud Health Technol Inform. 2011;169:412–6.
22. Zachariah M, Phansalkar S, Seidling HM, Neri PM, Cresswell KM, Duke J,Bloomrosen M, Volk LA, Bates DW. Development and preliminary evidencefor the validity of an instrument assessing implementation of human-factorsprinciples in medication-related decision-support systems–I-MeDeSA. J AmMed Inform Assoc. 2011;18(Suppl 1):i62–72.
23. Horsky J, Phansalkar S, Desai A, Bell D, Middleton B. Design of decisionsupport interventions for medication prescribing. Int J Med Inform. 2013;82:492–503.
24. Payne TH, Hines LE, Chan RC, Hartman S, Kapusnik-Uner J, Russ AL, et al.Recommendations to improve the usability of drug-drug interaction clinicaldecision support alerts. J Am Med Inform Assoc. 2015;22:243–1250.
25. Baysari MT, Westbrook JI, Richardson KL, Day RO. The influence ofcomputerized decision support on prescribing during ward-rounds: are thedecision-makers targeted? J Am Med Inform Assoc. 2015;18:754–9.
26. Kortteisto T, Komulainen J, Makela M, Kunnamo I, Kaila M. Clinical decisionsupport must be useful, functional is not enough: a qualitative study ofcomputer-based clinical decision support in primary care. BMC Health ServRes. 2012;12:349–57.
27. Russ AL, Zillich AJ, McManus MS, Doebbeling BN, Saleem JJA. Humanfactors investigation of medication alerts: barriers to prescriber decision-making and clinical workflow. AMIA Annu Symp Proc. 2009:548–52.
28. Saleem JJ, Patterson ES, Militello L, Render ML, Orshansky G, Asch SM.Exploring barriers and facilitators to the use of computerized clinicalreminders. J Am Med Inform Assoc. 2005;12:438–47.
29. van der Sijs H, Aarts J, van Gelter T, Berg M, Vulto A. Turning off frequentlyoverridden drug alerts: limited opportunities for doing it safely. J Am MedInform Assoc. 2008;15:439–48.
30. Feldstein A, Simon SR, Schneider J, Krall M, Laferriere D, Smith DH, Sittig DF,Soumerai SB. How to design computerized alerts to safe prescribingpractices. Jt Comm J Qual Saf. 2004;30:602–13.
31. Krall MA, Sittig DF. Clinician's assessments of outpatient electronic medicalrecord alert and reminder usability and usefulness requirements. Proc AMIASymp. 2002:400–4.
32. Weingart SN, Massagli M, Cyrulik A, Isaac T, Morway L, Sands DZ, WeissmanJS. Assessing the value of electronic prescribing in ambulatory care: a focusgroup study. Int J Med Inform. 2009;78:571–8.
33. Russ AL, Saleem JJ, McManus MS, Frankel RM, Zillich AJ. The workflow ofcomputerized medication ordering in primary care is not prescriptive. ProcHuman Factors Ergon Soc Annu Meet. 2010;54:840–4.
34. Wipfli R, Betrancourt M, Guardia A, Lovis CA. Qualitative analysis ofprescription activity and alert usage in a computerized physician orderentry system. Stud Health Technol Inform. 2011;169:940–4.
35. Khajouei R, Peek N, Wierenga PC, Kersten MJ, Jaspers MW. Effect ofpredefined order sets and usability problems on efficiency of computerizedmedication ordering. Int J Med Inform. 2010;79:690–8.
36. Patterson ES, Nguyen AD, Halloran JP, Asch SM. Human factors barriers tothe effective use of ten HIV clinical reminders. J Am Med Inform Assoc.2004;11:50–9.
37. Chused AE, Kuperman GJ, Stetson PD. Alert override reasons: a failure tocommunicate. AMIA Annu Symp Proc. 2008:111–5.
38. Koppel R, Metlay JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, Strom BL.Role of computerized physician order entry systems in facilitatingmedication errors. JAMA. 2005;293:1197–203.
39. Khajouei R, de Jongh D, Jaspers MW. Usability evaluation of a computerizedphysician order entry for medication ordering. Stud Health Technol Inform.2009;150:532–6.
40. Saleem JJ, Patterson ES, Militello L, Anders S, Falciglia M, Wissman JA, RothEM, Asch SM. Impact of clinical reminder redesign on learnability, efficiency,usability, and workload for ambulatory clinic nurses. J Am Med InformAssoc. 2007;14:632–40.
41. Trafton J, Martins S, Michel M, Lewis E, Wang D, Combs A, Scates N, Tu S,Goldstein MK. Evaluation of the acceptability and usability of a decisionsupport system to encourage safe and effective use of opioid therapy forchronic, noncancer pain by primary care providers. Pain Med. 2010;11:575–85.
42. Chan J, Shojania KG, Easty AC, Etchells EE. Usability evaluation of order sets in acomputerised provider order entry system. BMJ Qual Saf. 2011;20:932–40.
43. Nielsen J. Usability Engineering Boston: Academic Press; 1993.44. Scapin DL, Bastien JMC. Ergonomic criteria for evaluating the ergonomic
quality of interactive systems. Behav Inf Technol. 1997;6:220–31.45. Norman DA. The Design of Everyday Things. New-York: Basic Book; 1988.46. Pearce C. Computers, patients, and doctors—theoretical and practical
perspectives. In: Shachak A, Borycki EM, Reis SP, editors. Health Professionals'Education in the Age of Clinical Information Systems, Mobile Computingand Social Networks: Academic Press, Elsevier; 2017.
47. Náfrádi L, Nakamoto K, Schulz PJ. Is patient empowerment the key to promoteadherence? A systematic review of the relationship between self-efficacy,health locus of control and medication adherence. PLoS One. 2017;12
48. Marcilly R, Monkman H, Villumsen S, Kaufman D, Beuscart-Zéphir M-C. Howto present evidence-based usability design principles dedicated tomedication-related alerting systems to designers and evaluators? Resultsfrom a workshop. Stud Health Technol Inform. 2016;228:609–13.
49. Vicente KJ. Cognitive work analysis. Mahwah, NJ: Lawrence ErlbaumAssociates; 1999.
50. Riedmann D, Jung M, Hackl WO, Stuhlinger W, van der SH, Ammenwerth E.Development of a context model to prioritize drug safety alerts in CPOEsystems. BMC Med Inform Decis Mak. 2011;11:35.
Marcilly et al. BMC Medical Informatics and Decision Making (2018) 18:69 Page 17 of 17