Research and Applications

Design and evaluation of a pharmacogenomics

information resource for pharmacists

Katrina M Romagnoli,1 Richard D Boyce,1 Philip E Empey,2 Yifan Ning,1 Solomon

Adams,2 and Harry Hochheiser1,3

1Department of Biomedical Informatics, School of Medicine, University of Pittsburgh, Pittsburgh, PA, USA, 2School of Pharmacy,

University of Pittsburgh and 3Intelligent Systems Program, University of Pittsburgh

Corresponding Author: Katrina Romagnoli, 5607 Baum Blvd, 417D, Pittsburgh, PA 15206, USA. E-mail: kak59@pitt.edu.

Received 4 August 2016; Revised 20 September 2016; Accepted 11 January 2017

ABSTRACT

Objective: To develop and evaluate a pharmacogenomics information resource for pharmacists.

Materials and Methods: We built a pharmacogenomics information resource presenting Food and Drug Admin-

istration (FDA) drug product labelling information, refined it based on feedback from pharmacists, and con-

ducted a comparative usability evaluation, measuring task completion time, task correctness and perceived

usability. Tasks involved hypothetical clinical situations requiring interpretation of pharmacogenomics informa-

tion to determine optimal prescribing for specific patients.

Results: Pharmacists were better able to perform certain tasks using the redesigned resource relative to the Phar-

macogenomic Knowledgebase (PharmGKB) and the FDA Table of Pharmacogenomic Biomarkers in Drug Labeling.

On average, participants completed tasks in 107.5 s using our resource, compared to 188.9 s using PharmGKB and

240.2 s using the FDA table. Using the System Usability Scale, participants rated our resource 79.62 on average,

compared to 53.27 for PharmGKB and 50.77 for the FDA table. Participants found the correct answers for 100% of

tasks using our resource, compared to 76.9% using PharmGKB and 69.2% using the FDA table.

Discussion: We present structured, clinically relevant pharmacogenomic FDA drug product label information

with visualizations to help explain the relationships between gene variants, drugs, and phenotypes. The results

from our evaluation suggest that user-centered interfaces for pharmacogenomics information can increase

ease of access and comprehension.

Conclusion: A clinician-focused pharmacogenomics information resource can answer pharmacogenomics-

related medication questions faster, more correctly, and more easily than widely used alternatives, as perceived

by pharmacists.

Key words: pharmacogenomics, structured product labeling, pharmacist, evaluation, usability, clinical decision making

BACKGROUND AND SIGNIFICANCE

As pharmacogenomics-guided drug decision-making becomes more

common, pharmacists seek trustworthy, clinically actionable pharma-

cogenomics information to guide their decisions.1–4 Using information

about the interaction between genotypes and medications to inform

medication and dosage choices can optimize treatment and prevent ad-

verse medication events,5 yet that knowledge is not regularly applied.6,7

Insufficient pharmacogenomics training, complicated terminology, and

a rapidly moving field complicate the task, making clinicians of all

types uncertain of their ability to make informed decisions about medi-

cations with pharmacogenomics implications.7–9

Consequences of not knowing a patient’s genotype and resulting

phenotype can be severe, even fatal, as in the death of a breastfed

newborn whose mother was an ultra-rapid metabolizer of codeine,

which caused her breast milk to contain toxic levels of codeine’s

VC The Author 2017. Published by Oxford University Press on behalf of the American Medical Informatics Association. All rights reserved.

For Permissions, please email: journals.permissions@oup.com

1

Journal of the American Medical Informatics Association, 0(0), 2017, 1–10

doi: 10.1093/jamia/ocx007

Research and Applications

active metabolite, morphine.10 The Food and Drug Administration

(FDA) now requires the boxed warning section of clopidogrel’s

product label to warn that poor metabolizers of clopidogrel may not

be effectively treated by the drug and may have a higher risk of heart

attack, stroke, and death than normal metabolizers,11 implying that

genetic testing may be appropriate prior to prescription. With the

help of genetic tests, clinicians can ideally prescribe the right dose of

the right drug at the right time to the right patient. In some cases,

such as with warfarin1 and clopidogrel,3 genotyping already informs

drug and dose selection in select hospitals.

However, medication-related genetic testing is still in its nascent

phase,9,12 and there is no consensus yet on the usefulness of pharmaco-

genomics vs other methods to determine drug choice or dosage,13–16

further complicating how to use genetic information. In 2011, the

Clinical Pharmacogenetic Implementation Consortium (CPIC) began

developing clinical guidelines to support biomarker-based clinical

decision-making in conjunction with the Pharmacogenomics Knowl-

edgebase and the National Institute of Health’s Pharmacogenomics

Research Network (www.pgrn.org).17 As pharmacogenomics be-

comes more integrated into standard practice, clinicians have an ever

greater need for clinically relevant, effectively presented, searchable

information that supports accurate interpretation and adoption of

treatment recommendations.18

Multiple efforts exist to support the integration of patient-

specific, clinically actionable pharmacogenomics information into

practice. Eleven sites in the Electronic Medical Records and

Genomics (eMERGE) Network are implementing pharmacogenom-

ics clinical decision support. A recent article found that while these

sites have experienced delays and barriers to implementation of

pharmacogenomics clinical decision support, these are generally sur-

mountable.12 The eMERGE group is developing an infobutton that

will write genomic medicine information to share genetic testing re-

sults with clinicians within the eMERGE network.19 Trustworthy,

clear pharmacogenomic information presented in a clinically useful

way and combined with patient data and local or CPIC practice

guidelines has the potential to move pharmacogenomics into clinical

practice.

As drug information experts poised to support the pharmacoge-

nomics information needs of other clinicians, pharmacists play a key

role in pharmacogenomics-informed drug decisions.20–24 Meeting

pharmacists’ information needs is critical to effectively incorporate

pharmacogenomics information into decision-making.22 Multiple re-

sources are available, including the FDA Table of Pharmacogenomic

Biomarkers in Drug Labeling25 and FDA labeling information in the

Pharmacogenomics Knowledge Base (PharmGKB),26 but none of these

resources are currently able to provide clinically relevant information

at the point of care. The FDA table lists the drugs that are mandated by

the FDA to contain relevant pharmacogenomics information, but it re-

quires a minimum of 4 nonobvious clicks to access a product label.

The user still has to scan through the entire product label to locate the

pharmacogenomic information, which is dispersed throughout the la-

bel in multiple sections. PharmGKB aims to provide comprehensive

pharmacogenomics information, with an emphasis on the CPIC guide-

lines,27 but the utility of practicing pharmacists using it to make pre-

scribing decisions has not been evaluated.

In previous work, we developed a semantic model of pharmaco-

genomics information in product labels and designed a preliminary

prototype interface to explore that information.28 We extracted

pharmacogenomics statements from drug product labels and pre-

sented that information in a prototype user interface, as a first step

toward the goal of providing clinically actionable pharmacogenom-

ics information at the point of care. We also conducted a series of in-

terviews with pharmacists to assess their pharmacogenomics

information needs and resource requirements.29

In this study, we redesigned our prototype pharmacogenomics

information resource based on the results of that assessment and

conducted a user study comparing it to the original prototype, the

FDA Table of Pharmacogenomic Biomarkers in Drug Labeling,25

and the FDA labeling information in the Pharmacogenomics Knowl-

edge Base.30 Here, we present the redesigned prototype interface

and the results of our evaluation.

MATERIALS AND METHODS

Version 1.0 designIn the initial version of the prototype, the top table provides infor-

mation about the gene variants, the phenotype, and whether the im-

pact is on the drug’s efficacy or toxicity. A pathway diagram

illustrates how the drug interacts with the gene product to influence

the drug-related phenotype (Figure 1). Below are tabs containing

pharmacogenomics information from the annotated structured

product labels (SPLs), and recommendations regarding how to

change the patient’s treatment (Figure 2).

Initial usability studyAs part of a previously published study,29 we asked 14 pharmacists

working in a variety of settings to use our initial prototype to answer

pharmacogenomics-related medication questions while we observed

their behavior. We recruited pharmacists through professional con-

nections of the authors, using a convenience sampling strategy de-

signed to represent the diversity of needs and perspectives associated

with different care contexts, including tertiary-care centers, commu-

nity practices, and private nursing home consulting pharmacies. We

used the results of this preliminary usability study to identify prob-

lems with the pharmacogenomics information resource interface.

These problems, along with the results of our inquiry into pharma-

cists’ perceived information needs, informed a redesign of the proto-

type. The study was approved by the University of Pittsburgh

Institutional Review Board.

Pharmacogenomics resource usability evaluationWe conducted laboratory-based user studies with 16 pharmacists,

comparing both versions of our prototype to potential alternative

sources of pharmacogenomics information contained in FDA drug

product labelling. The FDA table was selected as the authoritative

source issued by a key regulatory agency. PharmGKB was selected

based on its history as the most comprehensive and well-established

pharmacogenomics information resource providing pharmacoge-

nomics information from product labels.

Study participants completed 8 pharmacogenomics information

tasks that could be answered in all 4 information resources. These

tasks asked participants to consider drug and dosage choices appro-

priate for individuals with various genotypes, along with questions

regarding the applicability of testing. The tasks were designed to

cover a range of pharmacogenomics concepts that have been identi-

fied as important competencies for pharmacists by the American So-

ciety of Health System Pharmacists and the Pharmacogenomics

Education Program (PharmGenEd) at the University of California,

San Diego.31 The tasks and correct answers were written based on

information provided solely within the FDA product label for

each relevant drug, as that is the source of pharmacogenomics

2 Journal of the American Medical Informatics Association, 2017, Vol. 0, No. 0

information provided by all assessed resources. Four of the tasks in-

volved only 1 drug question, and 4 were multidrug questions. The

tasks are listed in Table 1.

The order in which tasks and resources were presented was var-

ied among participants using a pseudo-randomization technique, to

avoid order effects. The order of tasks was randomly assigned to the

first participant and was assigned to each subsequent participant

based on that participant, to ensure that each task was performed

on each resource an equal number of times. The order of resources

was assigned randomly to each participant.

The FDA table and PharmGKB resources were downloaded to

the study laptop to guarantee that task completion times would not

be dependent on Internet connectivity and network latency. We

used HTTrack Website Copier32 to download the FDA table (down-

loaded June 2, 2015) and the FDA Drug Labeling portion of

PharmGKB.org (downloaded June 5, 2015).

The pharmacists were instructed to use only the assigned re-

source for each task and not seek the information on any other

sites. We recorded their activity on the screen and their voices

simultaneously using SnagIt, a Chrome web browser extension

for screen capture.33 Participants were instructed to highlight the

text of the answer to each task on the screen when they found it.

Each task had a maximum completion time of 10 min. Tasks that

were not completed in this time were marked as incorrect. The

task completion time for incorrect and incomplete tasks was mea-

sured as the time it took to find the incorrect answer, give up, or

reach 10 min. After all 8 tasks were finished, participants com-

pleted the System Usability Score (SUS) questionnaire34 for each

resource, to assess perceived usability. Measurements included

the time to complete each task from start to finish, the correctness

of each answer as determined by the investigators, and the SUS

score for each resource.

Evaluations took place during July and August 2015 and were

conducted at the participants’ workplaces. The study was approved

by the University of Pittsburgh Institutional Review Board.

Data analysisData were analyzed using MiniTab 1735 and R version 3.2.3.36

Descriptive statistics were used to summarize participant demographics.

We performed a regression analysis to examine task completion

time vs task number to assess order effect. For task completion time,

we assessed the distribution of the data, which was lognormal. Due

to the lognormal distribution, we used the Kruskal-Wallis test, a

nonparametric test for assessing the variance of data, and the Dunn

test for nonparametric pairwise correction to analyze the signifi-

cance of task completion time, comparing the 4 resources across all

8 tasks. For the SUS scores, we assessed the distribution of the data.

Due to the normal distribution, we performed a one-way analysis of

variance (ANOVA) and Games-Howell test for post hoc pairwise

comparison to analyze the significance of SUS scores of all 4 re-

sources. Games-Howell test was used due to unequal variances

among the 4 resources. Finally, for the correctness of answers for

each resource, we performed a Chi-square test with a post hoc

Figure 1. Drug page of version 1.0

Journal of the American Medical Informatics Association, 2017, Vol. 0, No. 0 3

pairwise comparison Bonferroni correction. MiniTab was used for

all analyses except the Kruskal-Wallis and Dunn tests, for which R

was used. P values were considered significant at .05. We considered

3 types of incorrect answers: the user gave an answer that was

wrong, the user failed to find an answer before 10 min, and the

user gave up before finding an answer but before the 10-min mark.

All of these are considered incorrect, as they are a failure to find

the correct answer.

RESULTS

Usability issuesWe organized usability issues identified in our initial usability study

into 3 categories: layout, information display, and information. The

layout was problematic, because the accordion design of the section

containing the information from product labels seemed almost invis-

ible. Participants also often failed to notice the Drug Info Links ta-

ble. As a result, many of the pharmacists were unable to access

product label information, and they requested links to other sources

despite those links being available. While they liked the illustrations

of interactions between drugs and genes, they were often confused

by the table containing information about efficacy and toxicity, or

how that influenced the phenotype. Pharmacists were also confused

by the notation of the gene variants (“star” notation) and they

wanted clear recommendations. They were concerned about the

source of the information, as it was not clear to them that all the

statements originated from FDA-approved product labels. Finally,

pharmacists also wanted more general background information

about pharmacogenomics, specifically about the drugs and genes, to

understand the information presented to them.

Figure 2. Expanded FDA label annotations section of PGx@Pitt version 1.0

4 Journal of the American Medical Informatics Association, 2017, Vol. 0, No. 0

Version 2.0 redesignThe problems with version 1.0 of the prototype identified in the ini-

tial usability study and the changes made in response are summa-

rized in Table 2.

The most prominent change to the drug pages (Figure 3) was

combining the pathway portion of the page with the table to form a

visual map, leading users from interactions between drugs and vari-

ants to information about phenotypes, and finally to FDA recom-

mendations. This change was made in response to pharmacists’

difficulty in interpreting the table with information about genotype

and phenotype. Specifically, our participants requested assistance

with interpreting the relationship between genotype, phenotype, and

medication recommendations. The visual map was designed to pro-

vide a clear and predictable structure of this information for all

drugs, including drugs like warfarin that have multiple relevant bio-

markers (Figure 4). The tabs were reordered, placing boxed warn-

ings and recommendations first. The remaining information was

provided in the second tab, organized by product label section. This

layout was intended to draw attention to the most relevant informa-

tion first, while also providing access to the label information in the

second tab. The box including links to other resources, including to

the original product label used for the annotation, was modified to

be visible at all times.

Usability evaluationSixteen pharmacists representing a variety of clinical practice envi-

ronments, with experience ranging from 2 months to >20 years,

participated in the usability evaluation. Most of them were clinical

pharmacists (9). Of those, a majority (5) were nursing home consul-

tant pharmacists (Table 3). All pharmacists who agreed to partici-

pate completed the evaluation.

Task completion time

To verify that the order of presentation was not a factor in task

completion, we confirmed that task completion time and order

were not correlated (r2¼0.02, P¼ .08). Task completion times

followed a lognormal distribution, requiring the use of Kruskal-

Wallis instead of ANOVA. Kruskal-Wallis results indicated signif-

icant differences between the resources (Figure 5) (Kruskal-Wallis,

P¼ .001). Search tasks were completed more quickly with the ver-

sion 2.0 resource (mean 107.5 s) as compared to both the FDA ta-

ble (mean 240.2 s; Dunn, P¼ .003) and PharmGKB (mean 188.9 s;

Dunn, P¼ .0087). Search tasks using the original PGx 1.0 proto-

type (mean 143.3 s) were completed significantly faster than the

FDA table (Dunn, P¼ .047). All other pairs were not statistically

significantly different.

Task correctness

A Chi-square test with a Bonferroni correction showed a difference

among the resources in rate of task correctness (N¼128,

v2¼13.393 [df¼3], P¼ .004). Version 2.0 performed statistically

significantly better than all other resources (P¼ .006) and was the

only resource in which 100% of answers were found correctly

(Table 4). Of the incorrect answers (22), 2 involved timeouts (maxi-

mum 10 min), 6 involved participants giving up, and in the remain-

ing 14, they found answers they incorrectly believed were correct.

System usability scale scores

The SUS score data were normally distributed, allowing for use of

ANOVA. Significant differences in SUS scores exist between the var-

ious resources (Figure 6) (ANOVA, P< .0005). As variances were

not equal, Games-Howell was used for post hoc analysis.37 SUS

scores were higher for version 2.0 (mean SUS score 79.62) than ei-

ther the FDA table (mean SUS score 50.77; Games-Howell,

Table 1. Pharmacogenomics Evaluation Tasks

Task Answer

1. A new patient with a prescription for warfarin is genotyped for all major pharmaco-

genes. Results indicate that he is CYP2C9 *2/*3, VKORC1 G-1639A AA. Using this re-

source, what starting dose is indicated for this patient, based solely on the genotype?

0.5–2 mg, depending on other factors like weight, age,

etc.

2. A physician has a patient with rheumatoid arthritis. She has had genotyping performed

for all major pharmacogenes, and results indicate she is heterozygous *3A (1 copy of

*3A, 1 copy of *1) for TPMT. The doctor wants to know if she can prescribe azathio-

prine for this patient.

Consider lower dose

3. Why should a patient with a CYP2C19 genotype other than *1 limit his exposure to cita-

lopram to 20 mg/day?

Poor metabolizers are at risk for QT prolongation

4. A postpartum breastfeeding mother tried to pick up her medications after being released

from the hospital. The community pharmacist refused to fill the prescription for codeine

because of a possible genetic issue. Why is the community pharmacist refusing to fill a co-

deine prescription for this patient?

Codeine should not be given to breastfeeding mothers

due to the risk of neonate morphine overdose if the

mother is an unknown ultra-rapid metabolizer

5. A new patient is genotyped for all major pharmacogenes. Results indicate he is a

CYP2D6 poor metabolizer. The patient is taking lisinopril, citalopram, metformin, war-

farin, and zolpidem. Which of those drugs, if any, are affected by CYP2D6 genes?

None

6. If a patient is positive for HLA-B*5701, which of these drugs can she take: carbamaze-

pine or abacavir? Both? Neither?

She can take carbamazepine, but not abacavir

7. A cardiologist wants to know if genetic testing is available by the FDA prior to initiating

cardiology drugs he frequently prescribes, such as clopidogrel, warfarin, atorvastatin,

and prasugrel

Clopidogrel and warfarin

8. A patient who is predicted to be a CYP2D6 extensive (*1/*1) metabolizer based on geno-

type alone is taking fluoxetine. If her physician is considering adding aripiprazole, what

is the recommendation regarding dosing?

50% dose reduction

Journal of the American Medical Informatics Association, 2017, Vol. 0, No. 0 5

P¼ .001) or PharmGKB (mean SUS score 53.27; Games-Howell,

P¼ .013), and higher for version 1.0 (mean SUS score 75) than the

FDA table (Games-Howell, P¼ .01). All other pairs were not statis-

tically significantly different. SUS scores above 68 (on a scale of 0–

100) are considered above average, while scores below 68 are con-

sidered below average.

Pharmacists’ reactions to pharmacogenomics information resources

Participants in both the qualitative inquiries and the evaluations in-

dicated that while the information provided by our pharmacoge-

nomics resource was important, a stand-alone web interface would

not be adequate for effective use of pharmacogenomics information

in the clinic. Ideally, they would prefer a resource that is integrated

into the electronic health record (EHR) and connects actual patient

genotype data with FDA-approved drug label information. They

also felt that the information provided by the FDA did not always

give them satisfactory recommendations on how to proceed with

specific patients. Recommending that a dose be lowered is not as

helpful as recommending a specific dose; recommending that an al-

ternative drug be given is not as helpful as recommending a list of al-

ternatives. Furthermore, for many drugs, other nongenomic factors

have a significant impact on the optimal dose for a given patient,

such as age, weight, renal function, and other clinical factors. As a

result, pharmacists felt that the information and recommendations

provided by the FDA product labels were insufficient without

patient-specific clinical data and recommendations based on that

data in addition to genotype data.

DISCUSSION

Clinicians of all types struggle with pharmacogenomics informa-

tion.7,9,21,38,39 To make a decision regarding the applicability of

drugs for patients, clinicians must incorporate information about

genes, how they interact with drugs, the predicted phenotypes of

given gene variants, and the phenotypes associated with those vari-

ants with respect to affected drugs.39 Interpreting this information

presents multiple challenges, including unfamiliar and potentially

confusing notation for describing gene variants and phenotypes.

Studies have shown that clinicians feel poorly informed about phar-

macogenomics testing, and that they have a difficult time interpret-

ing test results7,9 and the complex interactions between drugs and

genes.39 Efforts at meeting clinicians’ pharmacogenomics informa-

tion needs are under way, such as the work of Devine et al.40 to de-

velop pharmacogenomics alerts within a computerized provider

order entry system. Devine, et al. found in their recent usability

study with clinicians found that clinicians seek trustworthy, rele-

vant, dosing-related recommendations.40 Another recent study

found that clinicians welcome pharmacogenomics information for

the clopidogrel–CYP2C19 drug-gene interaction, and that they

would like supplementary information about the interaction due to

insufficient knowledge about pharmacogenomics.41 Pharmacists, as

trained drug information experts, are naturally poised to support

the incorporation of pharmacogenomics into clinical practice, but

they also struggle with pharmacogenomics information.22,29,39,42

Many groups, including the Pharmacogenomics Research Network,

CPIC, and eMERGE, as discussed previously, are implementing clin-

ical pharmacogenomics testing, producing evidence-based guide-

lines, and incorporating genetic testing results into clinical decision-

making. All of these efforts have hurdles to overcome related to howTab

le2.U

sab

ilit

yO

bse

rva

tio

ns

inV

ers

ion

1.0

an

dD

esi

gn

Ch

an

ge

sfo

rV

ers

ion

2.0

Layout

Info

rmati

on

Dis

pla

yIn

form

ati

on

Obse

rvati

on

Use

rsdid

not

noti

ce

acc

ord

ion

sect

ions

Use

rsappea

red

not

to

noti

ceD

rug

Info

Lin

ks

table

and

links

Effi

cacy

and

toxic

ity

not

wel

lunder

-

stood,phen

oty

pe

expla

nati

ons

wer

e

confu

sing

“Sta

r”nota

tion

uncl

ear

Part

icip

ants

wante

d

clea

rre

com

men

da-

tions

Part

icip

ants

wante

d

toknow

info

rma-

tion

sourc

e

Part

icip

ants

reques

ted

back

gro

und

info

to

be

able

tounder

-

stand

pharm

aco

-

gen

om

ics

Change

Change

tab

des

ign

of

SPL

sect

ions

Change

“For

more

info

”box

Com

bin

eto

xic

ity

and

effica

cyin

form

ati

on

wit

hpath

way

dia

gra

min

new

ta-

ble

dia

gra

m

Pro

vid

eso

urc

eof

in-

form

ati

on

more

clea

rly

Pro

vid

eback

gro

und

info

rmati

on

Spec

ific

Changes

Reo

rganiz

eta

bs:

boxed

warn

ing,

Pharm

aco

dynam

ic/

pharm

aco

kin

etic

(PD

/PK

),and

rec-

om

men

dati

ons

in

Sum

mary

,re

stin

seco

nd

tab

Move

tori

ght

of

scre

en,per

ma-

nen

tly

open

Rep

lace

effica

cyand

toxic

ity

info

rma-

tion

wit

hphen

o-

type

info

rmati

on

Have

“vari

ant”

col-

um

nin

new

dia

-

gra

m

Pro

vid

ere

com

men

da-

tion,if

appro

pri

ate

,

wit

ha

vis

ualaid

Make

links

box

easi

er

tofind,w

ith

clea

r

link

toso

urc

epro

-

duce

label

at

Dai-

lyM

ed

Add

links

tooth

erre

-

sourc

es,“pri

mer

”

page

6 Journal of the American Medical Informatics Association, 2017, Vol. 0, No. 0

to structure and present genetic data and evidence in a way that is

clinically actionable.12

One solution to managing this overwhelming amount of informa-

tion is to present structured, concise, relevant information, particu-

larly with visual depictions of the relationships between gene

variants, drugs, and phenotypes. Our evaluation suggests that these

techniques can help pharmacists to access, understand, and incorpo-

rate pharmacogenomics information into their decision-making pro-

cess, as compared to other pharmacogenomic information resources.

Version 2.0 supported pharmacists in answering pharmacogenomics-

related drug questions more successfully than the FDA table on all 3

measures (task completion time, correctness, and perceived usability)

and more successfully than PharmGKB on 2 out of 3. Participants

saved 81.4 s, on average, when completing tasks using version 2.0

compared to PharmGKB, a 75.7% decrease, and 132.7 s compared to

the FDA table, a 123.4% decrease. Version 2.0 was the only resource

on which users answered every question correctly.

Some of the differences in performance may be due to a differ-

ence in focus among the resources. For instance, the FDA infor-

mation provided by these resources is not the only clinically

important pharmacogenomics information available: PharmGKB is

built around the clinical guidelines developed by the CPIC.17

Incorporating CPIC guidelines into our resource and integrating

them alongside the drug label information is a potential future im-

provement. Also, the FDA table is not intended to be a clinical in-

formation resource providing practice recommendations or

supporting clinical decision-making. In contrast, our resource is a

novel implementation of pharmacogenomics information, struc-

tured by a clinically informed, detailed semantic model that allows

us to present the information in a highly usable, searchable way,

while also providing the opportunity to connect the information

to other sources of information, including guidelines, evidence,

and patient-specific data.

The goal of this project was to develop an effective method of im-

parting pharmacogenomics information to clinicians, specifically

pharmacists, by providing clinically actionable pharmacogenomics

drug information connected to relevant FDA-mandated product label

sources. We aimed to access and present that information faster,

Figure 3. Version 2.0, drug page

Figure 4. Generic model presentation of pharmacogenomics information

Journal of the American Medical Informatics Association, 2017, Vol. 0, No. 0 7

more completely, with a more satisfactory user experience than avail-

able alternatives. However, a key piece of information missing from

our resource, as well as all other currently available resources, is accu-

rate patient data, both clinical and genotype. The semantic model on

which this resource is built is capable of connecting patient data from

EHRs to evidence from sources such as product labels, guidelines,

and institutional policies. We have achieved the first necessary steps

in building the resource, designing a highly usable way of presenting

pharmacogenomics information, and evaluating it against alternative

resources, with positive results in performance and usability. Our

next step is to connect it to patient data so pharmacists have access to

patient-specific, clinically relevant, actionable information at the

point of care. Furthermore, pharmacists indicated throughout this

project that a stand-alone system would not fully support their needs;

they want the information integrated into their existing systems, in-

cluding computerized provider order entry systems and EHRs.

Understanding how best to achieve this requires careful study of cur-

rent workflows of pharmacists and other clinicians as they engage

with medication-related pharmacogenomics information.

LIMITATIONS

Limitations include a small sample size and number of tasks, which

limit the generalizability of the conclusions. We cannot claim that

any one of the interface changes definitely improved usability or per-

formance. Clinical implementation of genetic testing and genotype-

guided clinical decision support is necessary to determine improved

clinical outcomes. Some pharmacogenomics information from prod-

uct labels might be of limited value due to the lack of actionable rec-

ommendations and connection to patient-specific genetic data.

Similarly, other consensus PGx guidelines (CPIC) exist and are com-

monly used to drive decision support. However, we chose to focus

on FDA product label information as a starting point as it is the US

regulatory standard for drug information. The information was pro-

vided by all assessed resources, and the tasks and correct answers

were written based solely on that information. Participants used

only the FDA labeling portion of the PharmGKB.org website rather

than the complete site to ensure that they were using the same type

of information to complete tasks across each resource. We also did

Table 3. Demographics

Characteristics N¼ 16

Sex (N)

Female 12

Male 4

Ethnicity (N)

Caucasian 13

Asian 1

Hispanic 1

Other 1

Job Title (N)

Pharmacy student 1

Pharmacy resident 2

Pharmacy faculty 2

Clinical pharmacist, nursing home consultant 5

Clinical pharmacist, unit based 3

Clinical pharmacist, director of patient care 1

Pharmacy director 1

Pharmacist, PhD candidate 1

Time as Pharmacist (years)

Mean 7.5

Median 3

SD 8.9

Figure 5. Box plot of time (s) by resource: boxes and whiskers represent median interquartile ranges, asterisks represent outliers, solid circles represent mean

task completion times

Table 4. Task correctness (%) data

Resource Correct Incorrect Total Percent Correct

PGx 1.0 27 5 32 84.6

PGx 2.0 32 0 32 100

PharmGKB 26 6 32 76.9

FDA 21 11 32 69.2

8 Journal of the American Medical Informatics Association, 2017, Vol. 0, No. 0

not compare tertiary resources that provide pharmacogenomics in-

formation because we were focusing on sources that provide prod-

uct label statements directly. Finally, further development and

implementation of the system as an integrated resource will require

long-term effort and institutional support. Much of the expense lies

in the translation of FDA pharmacogenomics information from free

text into a structured semantic model.28 Community adoption of a

common model for representing key pharmacogenomics details

would reduce this expense.

Future work will include exploring how this resource can fit

within pharmacists’ workflow; expanding the information and sour-

ces of information to include CPIC guidelines, local practice guide-

lines, and tertiary information sources; and, most importantly,

connecting it to patient genetic and clinical data to provide patient-

specific recommendations.

CONCLUSIONS

Our pharmacist-focused resource supports pharmacogenomics in-

formation tasks more successfully than alternative resources. In the

process of designing this resource, we identified a model for display-

ing pharmacogenomics information that pharmacists find under-

standable and useful while seeking this information to make drug

decisions. The model for presenting pharmacogenomics information

and the semantic model on which the resource is built are necessary

for effective implementation of patient-specific pharmacogenomics

decision support connected to EHRs. Researchers and designers of

alerts and practice guidelines related to pharmacogenomics may find

this model to be a strong guideline for presenting pharmacogenom-

ics information to clinicians in a concise and accessible way.

CONTRIBUTORS

KMR designed and implemented the work; acquired, analyzed, and

interpreted the data; and wrote the manuscript. RDB provided sub-

stantial contributions to the design of the work, interpretation of the

data, and drafting of the manuscript, and assessed the manuscript

critically for intellectual content. PEE provided substantial contribu-

tions to the design of the work and interpretation of the data, and

assessed the manuscript critically for intellectual content. YN pro-

vided substantial contributions to the design and implementation of

the work and assessed the manuscript. SA provided substantial con-

tributions to the design of the work and assessed the manuscript crit-

ically for intellectual content. HH provided substantial

contributions to the design of the work, interpretation of the data,

and drafting of the manuscript, and assessed the manuscript criti-

cally for intellectual content.

FUNDING

This research was supported by the National Library of Medicine

(R01LM011838 and T15 LM007059), the National Institute of Aging (NIA

K01 AG044433), and the Agency for Healthcare Research and Quality

(K12HS019461).

COMPETING INTEREST

The authors have no competing interests to declare.

ACKNOWLEDGMENTS

The authors would like to thank the pharmacists who participated in this

work for their valuable contributions to the annotation and evaluation of

pharmacogenomics information.

REFERENCES

1. Kangelaris KN, Bent S, Nussbaum RL, et al. Genetic testing before antico-

agulation? A systematic review of pharmacogenetic dosing of warfarin.

J Gen Intern Med. 2009;24:656–64.

Figure 6. Box plot of System Usability Scales by resource: boxes and whiskers represent median and interquartile ranges, asterisks represent outliers, solid cir-

cles represent mean SUS scores

Journal of the American Medical Informatics Association, 2017, Vol. 0, No. 0 9

2. Caudle KE, Klein TE, Hoffman JM, et al. Incorporation of pharmacoge-

nomics into routine clinical practice: the Clinical Pharmacogenetics Imple-

mentation Consortium (CPIC) guideline development process. Curr Drug

Meta.b 2014;15:209–17.

3. Pulley JM, Denny JC, Peterson JF, et al. Operational implementation of

prospective genotyping for personalized medicine: the design of the Van-

derbilt PREDICT project. Clin Pharmacol Ther. 2012;92:87–95.

4. Johnson JA, Elsey AR, Clare-Salzler MJ, et al. Institutional profile: University

of Florida and Shands Hospital Personalized Medicine Program: clinical im-

plementation of pharmacogenetics. Pharmacogenomics. 2013;14:723–26.

5. Empey PE. Genetic predisposition to adverse drug reactions in the inten-

sive care unit. Crit Care Med. 2010;38:S106–16.

6. Sheffield LJ, Phillimore HE. Clinical use of pharmacogenomic tests in

2009. Clin Biochem Rev. 2009;30:55–65. http://www.pubmedcentral.nih.

gov/articlerender.fcgi?artid¼2702214&tool¼pmcentrez&rendertype¼ab

stract. Accessed April 5, 2013.

7. Crews KR, Hicks JK, Pui C-H, et al. Pharmacogenomics and individual-

ized medicine: translating science into practice. Clin Pharmacol Ther.

2012;92:467–75.

8. Johansen Taber KA, Dickinson BD. Pharmacogenomic knowledge gaps

and educational resource needs among physicians in selected specialties.

Pharmgenomics Pers Med. 2014;7:145–62.

9. Stanek EJ, Sanders CL, Taber KAJ, et al. Adoption of pharmacogenomic

testing by US physicians: results of a nationwide survey. Clin Pharmacol

Ther. 2012;91:450–58.

10. Madadi P, Ross CJD, Hayden MR, et al. Pharmacogenetics of neonatal

opioid toxicity following maternal use of codeine during breastfeeding: a

case-control study. Clin Pharmacol Ther. 2009;85:31–35.

11. George J, Doney A, Palmer CN, et al. Pharmacogenetics testing: implica-

tions for cardiovascular therapeutics with clopidogrel and warfarin. Car-

diovasc Ther. 2010;28:135–38.

12. Herr T, Denny J, Hakonarson H, et al. Practical considerations in genomic

decision support: The eMERGE experience. J Pathol Inform. 2015;6:50.

13. Nutescu EA, Drozda K, Bress AP, et al. Feasibility of implementing a com-

prehensive warfarin pharmacogenetics service. Pharmacotherapy.

2013;33:1156–64.

14. Klein TE, Altman RB, Eriksson N, et al. Estimation of the warfarin dose

with clinical and pharmacogenetic data. N Engl J Med. 2009;360:753–64.

15. Kimmel SE, French B, Kasner SE, et al. A pharmacogenetic versus a clini-

cal algorithm for warfarin dosing. N Engl J Med. 2013;369:2283–93.

16. Drozda K, Wong S, Patel SR, et al. Poor warfarin dose prediction with

pharmacogenetic algorithms that exclude genotypes important for African

Americans. Pharmacogenet Genomic.s 2015;25:73–81.

17. Relling MV, Klein TE. CPIC: Clinical Pharmacogenetics Implementation

Consortium of the Pharmacogenomics Research Network. Clin Pharma-

col Ther. 2011;89:464–67.

18. de Graaff LCG, van Schaik RHN, van Gelder T. A clinical approach to

pharmacogenetics. Neth J Med. 2013;71:145–52. http://www.ncbi.nlm.

nih.gov/pubmed/23712814. Accessed November 1, 2013.

19. Overby CL, Rasmussen LV, Hartzler A, et al. A Template for Authoring

and Adapting Genomic Medicine Content in the eMERGE Infobutton

Project. AMIA Annu Symp Proc. 2014;2014:944–53. http://www.pub

medcentral.nih.gov/articlerender.fcgi?

artid¼4419923&tool¼pmcentrez&rendertype¼abstract. Accessed Octo-

ber 15, 2015.

20. El-Ibiary SY, Cheng C, Alldredge B. Potential roles for pharmacists in

pharmacogenetics. J Am Pharm Assoc. 2003;48:e21–29, e30–32.

21. Tuteja S, Haynes K, Zayac C, et al. Community pharmacists’ attitudes to-

wards clinical utility and ethical implications of pharmacogenetic testing.

Per Med. 2013;10:793–800.

22. Dias MM, Ward HM, Sorich MJ, et al. Exploration of the perceptions,

barriers and drivers of pharmacogenomics practice among hospital phar-

macists in Adelaide, South Australia. Pharmacogenomics J. Published

Online First: September 10, 2013, doi:10.1038/tpj.2013.31

23. Reiss SM. Integrating pharmacogenomics into pharmacy practice via med-

ication therapy management. J Am Pharm Assoc. 2003;51:e64–74.

24. American Society of Health System Pharmacists. ASHP Guidelines on the

Pharmacist’s Role in Providing Drug Information. In: ASHP Best Practices

2014–2015. 2014. 355–59. http://www.ashp.org/DocLibrary/BestPracti

ces/SpecificGdlMedInfo.aspx

25. FDA. Genomics: Table of Pharmacogenomic Biomarkers in Drug Labels.

http://www.fda.gov/drugs/scienceresearch/researchareas/pharmacoge

netics/ucm083378.htm. Accessed March 31, 2015.

26. Hewett M, Oliver DE, Rubin DL, et al. PharmGKB: the Pharmacogenetics

Knowledge Base. Nucleic Acids Res. 2002;30:163–65.

27. Altman RB. PharmGKB: a logical home for knowledge relating genotype

to drug response phenotype. Nat Genet. 2007;39:426.

28. Boyce RD, Freimuth RR, Romagnoli KM, et al. Toward semantic modeling

of pharmacogenomic knowledge for clinical and translational decision sup-

port. AMIA Summits Transl Sci Proc. 2013;28–32. http://www.pubmedcen

tral.nih.gov/articlerender.fcgi?artid¼3814496&tool¼pmcentrez&renderty

pe¼abstract. Accessed September 3, 2014.

29. Romagnoli KM, Boyce RD, Empey PE, et al. Bringing clinical pharmaco-

genomics information to pharmacists: a qualitative study of information

needs and resource requirements. Int J Med Inform. 2016;86:54–61.

30. Drug Labels. https://www.pharmgkb.org/view/drug-labels.do. Accessed

July 6, 2016.

31. Kuo GM, Ma JD, Lee KC, et al. Institutional Profile: University of Califor-

nia San Diego Pharmacogenomics Education Program (PharmGenEdTM):

bridging the gap between science and practice. Pharmacogenomics.

2011;12:149–53.

32. Roche X. HTTrack Website Copier. 2015. https://www.httrack.com/.

Accessed June 2, 2015.

33. TechSmith. Snagit. 2015. https://www.techsmith.com/snagit.html.

Accessed June 2, 2015.

34. Brooke J. SUS: A ‘quick and dirty’ usability scale. In: Jordan PW, Thomas

B, Weerdmeester BA et al., eds. Usability Evaluation in Industry. Bristol,

PA: Taylor & Francis; 1996:189–94.

35. MiniTab. MiniTab 17. State College, PA: Minitab, Inc.;2015.

36. Team RC. R: A Language and Environment for Statistical Computing.

2015. http://www.r-project.org/. Accessed August 1, 2015.

37. Games PA, Howell JF. Pairwise multiple comparison procedures with unequal

N’s and/or variances: a Monte Carlo study. J Educ Behav Stat. 1976;1:113–25.

38. Overby CL, Erwin AL, Abul-Husn NS, et al. Physician attitudes toward

adopting genome-guided prescribing through clinical decision support.

J Pers Med. 2014;4:35–49.

39. McCullough KB, Formea CM, Berg KD, et al. Assessment of the pharmacoge-

nomics educational needs of pharmacists. Am J Pharm Educ. 2011;75:51.

http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid¼3109805&tool¼pmcentrez&rendertype¼abstract. Accessed November 19, 2012.

40. Devine EB, Lee C-J, Overby CL, et al. Usability evaluation of pharmacoge-

nomics clinical decision support aids and clinical knowledge resources in a

computerized provider order entry system: a mixed methods approach.

Int J Med Inform. 2014;83:473–83.

41. Nishimura AA, Shirts BH, Salama J, et al. Physician perspectives of

CYP2C19 and clopidogrel drug-gene interaction active clinical decision

support alerts. Int J Med Inform. 2016;86:117–25.

42. de Denus S, Letarte N, Hurlimann T, et al. An evaluation of pharmacists’ ex-

pectations towards pharmacogenomics. Pharmacogenomics. 2013;14:165–75.

10 Journal of the American Medical Informatics Association, 2017, Vol. 0, No. 0