Table of Contents - DPCPSI · 2018. 5. 9. · NExT Program: Report of Overall Project Page 6 . III....
Transcript of Table of Contents - DPCPSI · 2018. 5. 9. · NExT Program: Report of Overall Project Page 6 . III....
NExT Program: Report of Overall Project Page 1
Table of Contents
I. EXECUTIVE SUMMARY ................................................................................................................................................ 3
II. INTRODUCTION ........................................................................................................................................................... 5 RESEARCH GOALS .................................................................................................................................................................................... 5
III. METHODOLOGY.......................................................................................................................................................... 6 EXPERT REVIEW ...................................................................................................................................................................................... 6 ROUND 1 USABILITY TEST ..................................................................................................................................................................... 6 ROUND 2 USABILITY TEST ..................................................................................................................................................................... 6
IV. EXPERT REVIEW FINDINGS .................................................................................................................................... 8 NAVIGATION ............................................................................................................................................................................................. 8 CONTENT ................................................................................................................................................................................................... 8 LABELING AND TERMINOLOGY .............................................................................................................................................................. 8 STYLISTIC ISSUES ..................................................................................................................................................................................... 8
V. ROUND ONE USABILITY TEST FINDINGS ........................................................................................................... 10 FINDING THE NEXT WEBSITE ............................................................................................................................................................. 10 DISCOVERING THE PURPOSE OF NEXT .............................................................................................................................................. 10 UNDERSTANDING FUNDING VS. RESOURCES ..................................................................................................................................... 11 IDENTIFYING AVAILABLE RESOURCES ............................................................................................................................................... 12 APPLYING TO THE PROGRAM ............................................................................................................................................................... 12 LOCATING INFORMATION WITHIN PAGES ......................................................................................................................................... 13 CLEAR INSTRUCTIONS FOR APPLYING ................................................................................................................................................ 14 UNDERSTANDING ALLOCATION OF RESOURCES ............................................................................................................................... 14 WEBSITE USAGE AND ADDITIONAL COMMENTS FROM EXPERT USERS ....................................................................................... 16
VI. ROUND TWO USABILITY TEST FINDINGS ........................................................................................................ 16 PARTICIPANTS UNDERSTOOD NEXT’S HIGH LEVEL GOAL ............................................................................................................. 16 NOT A GRANT MECHANISM ................................................................................................................................................................. 17 COMPLICATED TEXT .............................................................................................................................................................................. 18 WHAT APPLICANTS NEED TO KNOW ................................................................................................................................................. 19 HOMEPAGE GRAPHIC – SELL THE PROGRAM .................................................................................................................................... 19 ADDITIONAL PERSUASIVE CONTENT .................................................................................................................................................. 21 ANNOUNCING TOPICS OF INTEREST ................................................................................................................................................... 21 NEXT GOVERNANCE STRUCTURE ....................................................................................................................................................... 21 INTELLECTUAL PROPERTY ................................................................................................................................................................... 23 CBC PARTICIPANTS AGREEMENT ....................................................................................................................................................... 24 NEXT STAGE GAGES .............................................................................................................................................................................. 25 SUBMIT AN APPLICATION LINK ........................................................................................................................................................... 28 APPLICATION INSTRUCTIONS............................................................................................................................................................... 28
VII. CONCLUSION ............................................................................................................................................................ 30
VIII. APPENDICES ........................................................................................................................................................... 31
NExT Program: Report of Overall Project Page 2
VERBAL INFORMED CONSENT FORM .................................................................................................................................................. 32 ROUND 1 FACILITATOR’S GUIDE ......................................................................................................................................................... 33 ROUND 2 FACILITATOR’S GUIDE ......................................................................................................................................................... 36
NExT Program: Report of Overall Project Page 3
I. Executive Summary The National Cancer Institute’s Experimental Therapeutics (NExT) Program is collaborative effort between the
NCI Division of Cancer Treatment and Diagnosis (DCTD) and the Center for Cancer Research. NExT provides
support in the form of research resources to researchers in an effort to accelerate the transition of cancer
treatments from the lab to the clinic. Researchers are accepted into the NExT program by having their proposals
assessed for scientific merit by a review committee. User-Centered Design (UCD) was contracted to assess the
NExT website to improve its usability and usefulness, especially as it pertained to potential applicants. An expert
review and two usability tests were conducted on the website.
Expert Review
An initial expert review in October 2010 provided comments about the website regarding the overall organization
of content, navigation, and stylistic elements. Overall, it was found that:
• The treatment of the navigation could be improved, both visually and functionally.
• The website contained useful information, but was often too text heavy.
• There were labeling and terminology issues that might confuse end users.
• Various visual design, page layout, and stylistic issues could be addressed to make aspects of the
website more clear.
Usability Test Round 1
Results from the first usability test in May 2011 showed that most participants had difficulty with many tasks on
the NExT website. Both novices and individuals who were familiar with NExT had trouble discerning the purpose
of the NExT program from the website. Although the Home Page contained relevant information, it was not
noticed and did not resonate with most participants. Participants struggled to understand how NExT could provide
resources to researchers without providing funding. In fact, many were unclear about the relationship between
researchers and the NExT program.
Additionally, most participants could not find information about applying to the program. The “Entry to Pipeline”
tab did not resonate as a place to find instructions for applying.
Participants were also unable to determine how program resources would be allocated. No one, without previous
experience with the program, understood the relevance of the Governance Structure to resource allocation.
Expert users who were familiar with the program did use the website to find various materials and to introduce
other investigators to the program. However, most other participants had difficulty locating information within the
website. Many pages were text-heavy contributing to the participants’ difficulty in extracting information.
Usability Test Round 2
The second usability test in September 2012 was a comparative test between two mock-ups. Based on the
difficulty participants had understanding the various material on the Home Page and elsewhere in the previous
test, both mock-ups had been designed to provide better information about the purpose of the NExT program.
Results showed that participants preferred elements of both home page options: they preferred the visual aspect
of one design, but the more explanatory content of the other design.
In regards to other design options, internal pages that provided headings formatted to describe paragraph
contents were preferred over those without headings. Stage Gates were preferred when they were presented
NExT Program: Report of Overall Project Page 4
graphically rather than simply as an expanding list, although the small size of the font in the graphic design limited
participants’ ability to read and comprehend the material.
The change of wording to “How to Apply” (from the previous “Entry to the Pipeline”) greatly facilitated participants
finding the application instructions.
Most of the content on the website was understood, although participants commented that much of it was
considered to be too dense; this was especially true for those who were not native English speakers. Participants’
comments reflected that they wanted to know what was important to them or ”What’s in it for me?”
• What do I need to do to apply?
• What will I get from the program if I am accepted?
The website divides the answer to the first question between a link to the application and instructions of what is
needed. Many participants tended to go immediately to the application and skip the instructions. Further, the
instructions didn’t start until below the fold (when a user has scrolled past the first screen) and the format of the
instructions made it difficult to determine exactly what needed to be done. Additionally, guidance about the types
of projects that NExT preferred was not obvious.
Most participants felt that the second question, “What will I get from the program if I am accepted”, was not
answered on the website. They suggested that scrubbed examples of projects that are already part of NeXT
would help with their understanding. Participants could not determine how resources would be allocated to
projects and the Governance Structure material was not helpful in this regard. The governance structure did not
seem pertinent to potential applicants; participants instead wanted to know with whom they would be interacting
and how the interaction would take place. Participants believed that material about intellectual property, a critical
issue to most researchers, was not described on the website in a way that they could understand how NExT
handled the issue. The CBC Participant Agreement did not improve their understanding. In fact, the CBC
agreement made most participants realize that they would need a lawyer to protect their side in any legal
agreement.
Despite the significant improvements seen on the website mockup as compared to the website in the first test,
improvements were still thought necessary. UCD suggests that:
• A new homepage graphic be created that better sells the program to potential applicants.
• The content of the site be rewritten based on principles for writing for the web and Plain Language.
• Additional content be added as necessary to answer potential applicants questions about what to expect
from the program.
• The overall format of the Application and its Instructions be redone so that each item needed for the
application is obvious—perhaps as a checklist,
• Links going directly to the Application should instead have an intermediate page advising users that
reading the instructions is needed before filling out the application.
• NExT investigates whether or not a wider page layout is possible, since a limited page width in the
current design creates problems for the NExT Stage Gates page.
NExT Program: Report of Overall Project Page 5
II. Introduction The National Cancer Institute’s Experimental Therapeutics (NExT) Program is a collaborative effort between the
NCI Division of Cancer Treatment and Diagnosis (DCTD) and the Center for Cancer Research. It is intended to
accelerate the transition of cancer treatments from the lab to the clinic. NExT consolidates NCI’s anticancer drug
discovery and development resources into a therapeutics pipeline of high-priority projects and is capable of
supporting research from initial discovery through Phase II clinical trial evaluation. The NExT website
(http://next.cancer.gov/) serves to inform researchers in government, academia, and industry about the program
and how they can participate in it. It provides background information on the program and instructs users on how
to apply.
User-Centered Design, Inc. (UCD) was contracted to conduct three rounds of usability testing, with changes being
made to the website after each round based on the results. However, after the first round, it was determined that
significant changes to the homepage of the website would be needed prior to another round of testing. In lieu of a
third round of usability testing, UCD created two alternative homepages, and then conducted the second, and
final, round of usability testing. In addition, UCD conducted an expert review, prior to the first round of testing in
order to familiarize ourselves with the website and to present initial feedback based on industry standards and
best practices.
Research Goals
The purpose of the usability evaluations was to determine the effectiveness and usability of the NExT website to
ensure that potential users (both novice and expert) can find the information they need to understand and apply
for the NExT program. The goal was to refine the website in a step-wise manner by iteratively testing with small
pools of users, responding to issues, and then testing again. This allowed the second usability test to gather
feedback on the effectiveness of the modifications made after the first round of testing.
The key questions that the testing was designed to answer were:
• How easily can users who are unfamiliar with the NExT program learn about, and understand, what it is
and what it can offer them?
• How well can users locate materials and the information they need to submit a NExT application?
• How well does the website communicate information regarding goals, policies, guidelines, etc.?
NExT Program: Report of Overall Project Page 6
III. Methodology
Expert Review
In October 2010, UCD conducted an expert review of the NExT website in an effort to familiarize ourselves with
the website and provide feedback on the overall organization of content, navigation, and stylistic elements. One
Human Factors Engineers and one Interaction designer conducted independent reviews based on industry
standards and best practices. A compendium of both reviews was presented that addressed potential usability,
style guide, and information architecture issues.
Round 1 Usability Test
Remote usability testing was conducted on the existing NExT website with seven participants between May 24th -
27th, 2011. Each session lasted approximately 45 minutes. Participants were recruited from a list of potential
participants that NCI/NExT provided. Participants represented a range of familiarity with NExT, work settings, and
job positions.
Familiarity Number
Expert (associated with NExT program) 2
Novice (some knowledge of NExT) 2
Unfamiliar (no prior knowledge of NExT) 3
Work Setting Number
Academic 5
Government 2
Job Roles Number
Scientist 3
Investigator 1
Professor 2
Director of Drug Discovery and Development program or related program
2
Protocol: The live NExT website was used for the first round of testing. Both an exploratory and task-based
protocol was used. Participants were first allowed to freely explore the website to see what they were able to learn
about the program on their own. They were then given tasks to find specific information on the website, and
exploratory follow-up questions were asked in order to explore participants’ expectations and confusions.
Round 2 Usability Test
Based on the results of the first round, two alternative homepage designs were created in an effort to improve the
ability for users to understand the NExT program. In addition, alternative displays were created for several of the
sub-pages, and a “Home” tab was added to the main navigation. Two mockups were created that incorporated
these different options. Remote usability testing was conducted with nine participants between August 24th – 29th,
2012. Each session lasted approximately 45 minutes. Participants were recruited from a list of potential
participants that NCI/NExT provided.
As in Round 1, participants represented a range of familiarity with NExT and work settings. Seven participants
were directly involved with drug discovery and development. The other two were indirectly involved (protocol
writing, and imaging). In addition, two participants had English as a second language.
NExT Program: Report of Overall Project Page 7
Familiarity Number
Novice (some knowledge of NExT) 3
Unfamiliar (no prior knowledge of NExT) 6
Work Setting Number
Academic 3
Government 3
Industry 3
Protocol: A task-based protocol was used; however, due to time limitations and the complexity of the tasks, most
participants did not complete all of the tasks. Instead, they were allowed to focus on the tasks that were important
to them. Since there were two mockups, half the participants used one design, and half used the other design.
Toward the end of their session, participants were shown the alternative design and asked for their feedback.
NExT Program: Report of Overall Project Page 8
IV. Expert Review Findings
Navigation
A variety of potential issues related to navigating were found, including:
• There was no Home Page link in the main navigation.
• Links within content were not obvious due to their color and lack of an underline.
• The use of both drop-down menus and left navigation could be confusing.
• The left navigation did not stand out visually and could be missed by users.
• “Resubmission Instructions” was listed in the drop-down menu in the top navigation, but not in the left
navigation.
• The “previous” and “next” navigation links within sections of Discovery and Development sections were
not intuitive.
• It is not clear which links are internal to NExT and which take the user to a different site.
• The “Cancer Imaging Program” link on the “Other NCI Programs” page under “Discovery” is the only link
in the list that goes to a different part of the website, which could be confusing.
Content
The website contains useful information, but much of it was too text heavy and difficult to follow. In particular:
• The About NExT page describes different organizations that are involved at different stages, but it is hard
to see how they all fit together.
• The Instructions page is very long with several layers of indentation which could
In addition, there are places where more (or alternative) information may be necessary. For example, the website
says approved proposals gain access to drug discovery and development resources, but it is not obvious what
those resources are.
Labeling and Terminology
There were several places throughout the website where the labeling and terminology may confuse users:
• The homepage “Do you need” section may be better phrased as “Benefits of NExT.”
• The label “Pipeline” may not be familiar to new users; “How to Apply” might be clearer.
• The label “Pipeline Management” may be confusing; “How it Works” might be more clear.
• The use of “Main” as the label for the landing page of each section could be confused for the “Home” link.
Stylistic Issues
There were various visual design, page layout, and stylistic issues, including:
• The font styles and colors on the home page didn’t support the page hierarchy.
• The content box on the homepage may suffer from banner blindness (i.e., users might not notice it).
• The page title is duplicated on the page: in the banner and in the content area.
• The NExT Governance Structure diagram is at the bottom of the page, but would be more useful at the
top, and it doesn’t indicate any directional flow between groups (i.e., who advises who)
• The Stage Gate arrow graphic is difficult to read and the text is part of the image (508 concern).
• It is a little hard to tell the steps apart in the Stage Gate Guidelines table (bullets may help).
• The visual styling of the left navigation doesn’t differentiate between second and third level pages.
NExT Program: Report of Overall Project Page 9
• The “Toxicology and Pharmacology Branch” page within Discovery and within Development look nearly
identical.
• The “Pharmacodynamics Assay Development Pathway” image on the “Biomarkers” page uses a really
small text size that is difficult to read.
NExT Program: Report of Overall Project Page 10
V. Round One Usability Test Findings
Finding the NExT Website
When time permitted, participants were asked to try to get to the NExT website from cancer.gov and the DCTD
website. Participants used a variety of approaches:
From cancer.gov
• 1 successfully found the site using search (“NCI NExT”)
• 1 looked in “NCI Drug Dictionary” but was unsuccessful
• 1 did not know where to look
From DCTD site
• 1 noticed a link to the NExT website on the homepage
• 1 did not know where to look.
Though few participants performed a search, it is generally an effective method of finding information and worked
for the one participant that tried it.
Discovering the Purpose of NExT
Both novice participants and those completely unfamiliar with NExT had trouble discerning the purpose of the
program. Participants expected to find a description of the purpose on the homepage, but were not sure where to
look (Figure 1).
The “Do you need” section was helpful for some participants, but not all. Participants commented that it “looks like
PR,” is text-heavy, and is “stuff I already know.”
Several participants looked in the “About NExT” tab and still had difficulty understanding the purpose of NExT.
One person found the content in the Discovery and Development tabs to be helpful.
Most participants tried to discern the purpose of NExT from the text in the gray box on the right side of the
homepage. It includes, “Who,” “What,” “When,” “Where,” “How,” “Sponsors,” and “What NExT is not,” and was
read by all participants. However, despite reading the “What” section, several participants still did not understand
the purpose. Some described the purpose as: “To develop candidate drugs, provide screening, optimization, etc.,”
and “to provide compounds to clinicians for clinical trials.” In addition, several participants commented that the text
size was too small.
Recommended changes included:
• Make the content in the gray box more prominent, as it contains important information.
• Add an explanation of how the program benefits researchers.
• Move “What NExT is not” higher on the page since it helps describe the program.
• Link to examples of how the program works.
NExT Program: Report of Overall Project Page 11
Figure 1: Where participants looked for information on the purpose of NExT.
Understanding Funding vs. Resources
Participants struggled with the concept of NExT being a source for resources but not of funding. The explanation
for this is buried at the bottom of the gray box on the homepage (Figure 2). Some participants explicitly said that
the program does provide funding. Others expressed confusion over what the program provides, if not funding.
Most were unclear about the connection between researchers and the NExT program.
NExT Program: Report of Overall Project Page 12
Figure 2: Explanation that NExT is not a grant or contract mechanism.
Identifying Available Resources
Participants were asked to locate what resources were available to researchers through the NExT program. Some
participants looked in the “Discovery” or “Development” tabs, but all had difficulty finding a list of resources.
Some participants felt like the website required too much clicking in order to find information. For example,
“Discovery Activities in DTP” links to several subpages, each of which provides a short explanation of what the
branch supports, and then from there it links to the website of that branch. Since the content on each of the
branch pages is fairly small, it may be possible to combine them all onto one page (same with Development
Activities in DTP), thus requiring fewer clicks.
In addition, describing the “activities” in terms of “resources” offered to applicants may make it easier for users to understand what the NExT program can offer them.
Applying to the Program
Most participants had trouble locating information on how to apply to the program. Some noticed the link in the
gray box on the homepage to “NExT application,” but this took them to ProposalCentral, not the instructions
(Figure 3). The tab label, “Entry to Pipeline” (Figure 4), did not resonate with participants (including Experts) as a
path to the application process. They understood the term “pipeline,” but the label did not imply an application to
the program. Several participants suggested a new label be used, such as “Submit Application,” “Apply to
Program,” etc.
When participants clicked on the link to ProposalCentral, the website opened in a new tab; however most
participants thought they were still on the NExT website. When asked to return to the NExT website, they were
confused because they thought they could just hit the Back button. They did not realize they were in a new tab. [It
should be noted that in the second round of testing, this was not an issue even though the website still opened in
a new tab.] The debate about whether to open external websites in the same window vs. a new window is
ongoing and the solution often varies, depending on the website. In this particular case, since users may want to
refer back to the NExT website when they fill out an application, it is probably best to keep opening the site in a
new window. [As discussed later in this report, the creation of a landing page for “Submit an Application” may help
with this issue since users expect main navigation links to go to a page on the current website.]
NExT Program: Report of Overall Project Page 13
Figure 3: Link to ProposalCentral on homepage.
Figure 4: "Entry to Pipeline" label did not imply "application."
Locating Information Within Pages
Several key pages within the website are text-heavy, with no bold text or headers to break it up. This formatting
contributed to participants’ difficulty in extracting information. For example, one participant wanted to know how
long the application could be and had significant difficulty locating the answer, “should not exceed 5 pages,” on
the Instructions page (Figure 5). Rewriting the content in plain language [as discussed later in this report] and
NExT Program: Report of Overall Project Page 14
formatting it to aid scanning (i.e., bolding key words and phrases, adding headers, and using bulleted lists) would
help in the readability of long passages.
Figure 5: "Should not exceed 5 pages" instruction is buried in the text.
Clear Instructions for Applying
One “expert” participant said she had followed the instructions for applying and successfully submitted her
application. She felt the instructions were clear and thorough. However, she also thought the “Entry to Pipeline”
label should be changed.
Understanding Allocation of Resources
When asked how the NExT program resources would be allocated, most participants were unsure what this
meant. They thought it would be up to them to request certain resources. None of the participants spontaneously
looked in the “Pipeline Management” section (Figure 6) for information on resource allocation.
NExT Program: Report of Overall Project Page 15
Figure 6: "Pipeline Management" section.
Participants who looked at the NExT Governance Structure page upon prompting were confused by the text and
accompanying graphic (Figure 7), unless they were already familiar with the program. They did not see the
relevance of the content and did not understand how the governance structure would affect their connection with
NExT.
Figure 7: NExT Governance Structure graphic.
NExT Program: Report of Overall Project Page 16
Website Usage and Additional Comments from Expert Users
Participants who were expert users reported using the site to:
• Introduce other investigators to the program.
• Provide information about his/her center.
• Look for key documents (e.g., CBC Participants Agreement, MTA).
• Review “Stage Gate” information.
After showing the website to other investigators who might be interested, those potential applicants asked for
clarification on:
• How is this program different from other NCI programs?
• What is this program if it isn’t a funding mechanism?
• If I don’t get funding, what do I gain from the program?
• How does resource allocation work?
• How is intellectual property handled?
• What is the process for entering into the program?
• What is the process once you are in the program?
In addition, expert users suggested:
• Showing an example of how the overall process works.
• Showing the review process
• Clarifying “pipeline” as this may mean different things to different investigators. In particular, academic
researchers may not understand the applied-research perspective.
• Clarifying the relationship between NExT, DCTD, and DTP.
VI. Round Two Usability Test Findings
Participants Understood NExT’s High Level Goal
In this round of testing, participants easily determined the high level goal of the NExT program, which is to
facilitate collaboration in an effort to move cancer related drugs through the pipeline more efficiently. Some of the
items on the homepage that seemed to convey this included: the title in the header (“NCI Experimental
Therapeutics Program”), the tagline (“Partnering with the NCI for Oncology Drug Discovery and Development”),
the paragraph titled “What is NExT,” and the “Who” section which specifies that NExT is for researchers in
academia, government, and industry (Figure 8). Though participants understood that NExT would provide them
with resources, they had some difficulty explaining what those resources would be.
NExT Program: Report of Overall Project Page 17
Figure 8: The top section of the homepage.
Not A Grant Mechanism
In the first round of testing, it was not clear to participants that NExT is not a grant mechanism. One of the
changes made prior to the second round of testing was to clarify this on the website. It was mentioned on the
homepage (in bold), the “How to Apply” page, and the “Instructions” page (Figure 9). Participants all noticed the
message this time. Nearly all participants understood that this meant they would not be given money (though one
participant was still unsure).
One participant said that the message was too repetitive. He commented, “I don’t know how many times you’re
going to beat us over the head with, ‘This is not a grant application.’” However, given the misunderstanding in the
last round of testing, it is better for the website to state it too many times than not enough.
Another participant commented that on the homepage, the statement should not be the second sentence. Rather,
it should tell users what NExT does offer (successful applicants receive access to NCI resources), and then what
it does not offer (funding).
NExT Program: Report of Overall Project Page 18
Figure 9: Three places the website tells users NExT is not a grant mechanism.
Complicated Text
Participants had trouble getting a clear picture of the NExT program because the text was complicated and hard
to digest. Several participants commented that it should be written in simpler language, and that it should be more
concise and geared towards potential applicants. The complexity of the text was especially an issue for two
participants with English as a second language. Both of these participants had trouble scanning the text and
indicated that they would need a lot more time to really understand it all.
Since users of a website are unlikely to spend an hour on the website trying to figure it out, if they cannot easily
determine whether the NExT program would be appropriate for them, they will probably move on. Though the
subject matter is complex by nature, it is always possible to improve readability for the web. Some basic
formatting changes (such as the use of headers to break up content on the page and the use of concise bulleted
lists) can improve users’ ability to scan and understand the content. The website http://centerforplainlanguage.org
is a good resource for learning about how to write in plain language.
NExT Program: Report of Overall Project Page 19
What Applicants Need to Know
A reoccurring theme across participants was that the website needed to focus more on what was important to
potential applicants. Several participants commented that the website appeared well organized at first (the main
navigation labels made sense and they liked the drop down menus), but that once they started looking around, it
was difficult to find what they wanted to know. Basically, participants wanted to know what to expect. Some of the
questions they had included:
• What’s in it for the researcher?
• What is the process? Not only for applying, but what happens after a person is accepted?
• What is the researcher’s role vs. the role of the committees in the governance structure?
• How much collaboration is there when structuring the project?
• Will the researcher be interacting with all of the committees, or a single individual within NExT?
• What type of interactions will the researcher have with NExT?
• How much control does the researcher have at each of the decision points in the stage gates?
• How can the relationship with NExT be severed?
These questions, and any others that potential applicants might have, need to be addressed on the website (in
plain language). Though the answers to these questions will probably be addressed on various pages throughout
the site, it may also be helpful to create an FAQ. Having this information consolidated in one place (if it answers
users’ real questions and if it’s concise) may also help give users a better overview of the program and what to
expect.
Homepage Graphic – Sell the Program
Participants wanted the homepage to “sell” the program. They wanted it to quickly tell them what it would do for
them, how it works, and why they should even be interested. Neither of the graphics seemed to do this well.
Participants liked the visual styling of the pipeline graphic (Figure 10), but preferred the flowchart graphic (Figure
11) since it seemed to walk through the process (though it didn’t quite have all the right information). Participants
also liked the idea of the links in the flowchart, which led to more information, indicating that an interactive graphic
of some kind might be well received.
Industry participants were already very familiar with the stage gates, so seeing them on the homepage in the
pipeline provided no value to them. Even participants who were not as familiar with the pipeline also wanted to
see something more informative. Some of the suggestions for concepts that should be highlighted on the
homepage included: details about collaboration, availability of useful resources to participants, and the overall
process. Since there is a lot of information behind each of those concepts, an interactive graphic that provides
teaser information on each concept and allows the user to click a link to learn more may be useful. One
participant commented that the homepage overall was too bland, so a more colorful, interactive graphic would be
a good way to make the page more interesting and draw users in.
NExT Program: Report of Overall Project Page 20
Figure 10: One of two homepage graphic options – pipeline.
Figure 11: One of two homepage graphic options - flowchart.
NExT Program: Report of Overall Project Page 21
Additional Persuasive Content
Several participants said that providing scrubbed examples (i.e., success stories) of projects that have been a
part of NExT would help provide a better understanding of the program. It would help show applicants what they
could expect and would be a persuasive way to sell the program. Since a project can enter at any part of the
pipeline, the examples would not need to be exhaustive, but it would be ideal if they could showcase several
different types of projects. Including information about at what point the project entered the pipeline, what
resources NExT provided, how those resources helped, and the outcome of the project may be useful. Along the
same lines, one participant suggested that “press releases” about successful projects (if available) would be nice.
Announcing Topics of Interest
One participant suggested including announcements about what type of projects NExT is particularly interested in
at the time (as this is likely to change). Currently, the website does mention this on the application instructions
page (Figure 12), but the information is not obvious. It is surrounded by other text that participants tended to skim
over. Also, it is fairly general guidance. It may be helpful to create an “Areas of Focus” section on the homepage
to highlight this information.
Figure 12: Information on the topic areas NExT is particularly interested in.
NExT Governance Structure
Participants had difficulty determining how resources were allocated to projects. The “NExT Governance
Structure” page answers this question, and it is linked to from the “How NExT Works” page, which participants did
NExT Program: Report of Overall Project Page 22
visit. In one mockup, the paragraph containing the link had a “Resource Allocation” header (Figure 13). In the
other mockup, there was no header (Figure 14), just the surrounding text (which was the same in both versions).
Participants who saw the version with the header went straight to the “NExT Governance Structure” page.
Participants who saw the version without the header took a lot longer to find the correct page (if they found it at
all). Some participants understood that committees would make decisions about resource allocation based on
various factors, but it was not obvious to everyone. One participant said the page “started to explain it” but it
wasn’t entirely clear.
Figure 13: Paragraph surrounding "NExT Governance Structure" link with a header.
Figure 14: Paragraph surrounding "NExT Governance Structure" link, without a header.
Like much of the site, the “Next Governance Structure” page is very dense and is not written in way that aids
scanning, or that is geared towards potential applicants. One participant commented that the content is the type of
information you would present to committees, rather than researchers. Potential applicants do not need to know
all the details about the various committees, except for as it applies to their experience – who will they be
interacting with and how?
If this information does need to be presented on the website (perhaps for other user groups besides applicants), it
would benefit from breaking the content up into one paragraph per committee, with the use of headers to identify
each committee.
The label “NExT Governance Structure” was misleading for one participant. When she saw the word “structure”
she thought it would only show the image of the structure (Figure 15), like an org chart. She did not realize there
would be explanatory text with it. For her, the label “NExT Governance” would have been better. Since only one
participant had this problem, it is difficult to tell whether or not this would be a common misconception, but it is
worth considering.
NExT Program: Report of Overall Project Page 23
Figure 15: Graphic of NExT Governance Structure.
Intellectual Property
Intellectual property (IP) issues were a concern to several participants, especially those from industry. Due to their
concerns over IP, industry participants felt the site was geared more towards academics, though they realized it
could be useful for people in industry as well (particularly if they were from a small start-up company). The
specifics of how intellectual property would be handled were not clear to participants. Though the “IP and Data
Access” page explains some of this (Figure 16), participants did not find it obvious.
First, the label “IP” is used instead of spelling out “Intellectual Property.” It is best practice to spell out acronyms,
both in the navigation and the page title. This helps emphasize the focus of the page and ensures users know
what it means.
Second, though headers are used, they do not strongly imply that the content below them is about IP (e.g., CBC
Participants Agreement, NExT MTA, NCI Employee Inventions, Contract Research Organization, CTEP-
Sponsored Clinical Trials). For example, the “NExT MTA” is for applicants “entering the pipeline post lead
compound development but prior to clinical trials” (i.e., mid-phase projects), but the user doesn’t know this until
they read through the text. Rather than labeling the header “NExT MTA” (which means little to new users), it could
say, “IP Agreement for Mid-phase Projects, “ since users can likely identify with their project’s location in the
pipeline.
Third, the page references the “Bayh-Dole Act” but does not explain what it is. It assumes that users will know
what this means. Providing an explanation of what this means for applicants to NExT would be helpful.
Fourth, both the CBC Participants Agreement and NExT MTA are provided through links, but these documents
are written in legal language. Users are not likely to read through these documents to get information. Instead, a
summary about what the documents say regarding IP should be provided in plain language on the page itself.
NExT Program: Report of Overall Project Page 24
Finally, at the bottom of the page there is an image that shows which agreements are required, depending on a
project’s location in the pipeline. None of the participants noticed this image, but it seems like it would be very
helpful in guiding users to the relevant section of the page (depending on their project). Placing it at the top of the
page, instead of the bottom, may help.
Figure 16: "IP and Data Access" page.
CBC Participants Agreement
The CBC Participants Agreement is linked to in three different places on the website: “IP and Data Access,” “How
to Apply,” and “Instructions” (Figure 17 and Figure 18), however no participants viewed it on their own. They had
to be directed to click on it. There are several possible reasons for this.
First, the label itself does not imply what it is. Most participants did not seem to have a clear understanding of
CBC’s role and therefore “CBC Participant” did not mean much to them.
Second, the surrounding text is not particularly helpful. On the “IP and Data Access” page (Figure 17), it does not
clarify what the CBC is, who “members” are, or why the agreement is necessary. The text on the “How to Apply”
NExT Program: Report of Overall Project Page 25
and “Instructions” pages (Figure 18) is more clear about who will need to sign the agreement and why, but the link
is embedded within the page, surrounded by text which causes participants to skip over it.
Figure 17: Link to "CBC Participants Agreement" on "IP and Data Access" page.
Figure 18: Link to "CBC Participants Agreement" on "How to Apply" and "Instructions" pages.
Once participants were directed to review the agreement, they said it looked like a standard legal document and
therefore they would not read through it as a means of getting information. Several mentioned that this is
something they would give to their lawyers. If there is important information in this document that potential
applicants should know prior to applying, it needs to presented on the website in plain language.
One participant commented that this agreement would pose a possible roadblock since industry lawyers will often
want to use their own language.
Another participant commented that the use of “Participant” instead of “Researcher” was a little confusing. Though
we recognize that this language will not change, it is important to keep this in mind, as the current link to the
document is “CBC Participants Agreement.” It is not likely that users will identify with, and therefore an alternative
link label would be beneficial.
NExT Stage Gages
The term “Stage Gate” was familiar to some participants. Some of them thought the term was fine and that users
would understand it, but others, although familiar with the label, did not particularly like its use. One of these
participants said they are really “decision points,” and would have preferred that as the label. Though not
everyone was familiar with or liked the label, all participants were drawn to click on it on their own and seemed to
understand what it was when they saw it.
Two versions of the NExT Stage Gates page were tested: the existing design (Figure 19, left), and an alternate
design that listed the guidelines in bulleted lists rather than a table, with the Discovery and Development phases
in collapsible/expandable panels (Figure 19, right). Participants preferred the graphical presentation of the stage
gate guidelines as displayed in the table in the current design. On the alternate design, they thought that clicking
NExT Program: Report of Overall Project Page 26
the panels to expand them was not good, and that the vertical presentation of specific guidelines caused the page
to be too long.
In the current design, however, several participants commented that the text was hard to read. The blue arrow
image was familiar to people from industry, but participants did not like that acronyms in the image were labeled
below it. This required looking back and forth between the image and the key below it. In addition, the font size in
the table of specific guidelines was too small to read comfortably.
The legibility of the stage gates page in the current design (both the blue arrow graphic and the table) is hindered
by the width of the page – 752 pixels wide. This page width is designed to support an 800x600 screen resolution,
which is no longer a standard that needs to be supported. Widening the page design to support a 1024x768
screen resolution may allow the labels for each stage gate to be incorporated into the blue arrow itself, rather than
using acronyms and a key. In addition, it would be possible to use a larger font size in the table.
NExT Program: Report of Overall Project Page 27
Figure 19: Two options for the NExT Stage Gates. Current design (left), and alternate version (right).
NExT Program: Report of Overall Project Page 28
Submit an Application Link
When looking for information on submitting an application, participants often went straight to the “Submit an
Application” link in the “How To Apply” drop down menu (Figure 20), prior to reading the instructions. This link
goes to a different website, ProposalCentral. It is necessary to review the instructions before a user can submit an
application. In addition, it is generally bad practice to link to a different website directly from the main navigation,
as this can be disorienting to users.
Consider creating a landing page for the “Submit an Application” link on the NExT website that tells users they
should read the instructions before submitting an application, and then provides a link to the website where they
can submit the application. This will let users know they should read the instructions first, and it will let them know
they will be leaving the website if they continue to the application.
Figure 20: "Submit an Application" link in the "How To Apply" menu.
Application Instructions
The formatting of the “Instructions” page was problematic for some participants. One participant commented that
the instructions do not actually start with instructions. Instead, it repeats some of what has already been said
elsewhere on the site and is too verbose. Another said the instructions would be more helpful if they were
formatted as a step-by-step checklist. Doing so would ensure that users do not miss important information (if each
item had its own checkbox), and it would better emphasize the steps that need to be taken. For example, having
applicants review the CBC Participants Agreement could be incorporated as one of the steps for applying.
Currently bullets are used to identify each piece of the application, which in general is good practice, except that
on this page it makes it difficult to tell the difference between bullets indicating an item that needs to be included
(e.g., background, hypothesis, research strategy and specific request) vs. bullets providing more detail about an
item (e.g., “For early-stage drug discovery projects,” “For late-stage drug discovery projects,” and “For clinical
drug development projects”) (Figure 21). The use of checkboxes would help distinguish between the two, as
would making the sub sections under the “Research Strategy and Specific Request” collapsible/expandable
(collapsed by default). This would let users see at a glance that specific information is required depending on
NExT Program: Report of Overall Project Page 29
where a project is in the pipeline (i.e. early-stage discovery, late-stage discovery, or “clinical drug development”).
Users could then view just the section that is relevant to them.
Figure 21: Application instructions with several levels of indentation.
NExT Program: Report of Overall Project Page 30
VII. Conclusion The initial expert review revealed issues concerning navigation, content that is not written to aid readability,
labeling and terminology, and the visual design/page layout. Many of these issues were confirmed in the first
round of usability testing. Some changes were made prior to the second round of testing which did improve users’
ability to understand and use the website. However, additional changes are recommened to further improve the
website and ensure that it is appropriate for potential applicants. UCD suggests that:
• A new homepage graphic be created that better sells the program to potential applicants.
• The content of the site be rewritten based on principles for writing for the web and Plain Language.
• Additional content be added as necessary to answer potential applicants questions about what to expect
from the program.
• The overall format of the Application and its Instructions be redone so that each item needed for the
application is obvious—perhaps as a checklist,
• Links going directly to the Application should instead have an intermediate page advising users that
reading the instructions is needed before filling out the application.
• NExT investigates whether or not a wider page layout is possible, since a limited page width in the
current design creates problems for the NExT Stage Gates page.
NExT Program: Report of Overall Project Page 31
VIII. Appendices
NExT Program: Report of Overall Project Page 32
Verbal Informed Consent Form
Verbal Informed Consent Form for the
Evaluation of the NExT Website
As part of this research, I need to ask you to formally agree to this usability test and interview. As part
of a research project for the National Cancer Institute, we are seeking to evaluate the appropriateness
and usability of the NExT Website for researchers—both those who are already familiar with NExT
(“experts”) and people unfamiliar or relatively new to NExT (“novices”). Individuals who we think
would have an appropriate background are being asked to participate in a usability test and then to
reflect on the website. So we are asking you for about 45 minutes of your time.
We won’t be asking anything personal and no identifying information will be shared. Any findings will
be reported in aggregated form.
If it’s alright with you, we’d like to record the monitor used in the usability test and your voice in order
to help with note taking, but your image would not be videotaped. Is that OK with you or would you
prefer not to be recorded?
Your participation is voluntary, and if you choose not to participate it will not affect your relationship
with the National Cancer Institute. You may ask questions at any time. You are also free to stop the
interview at any time without penalty and without any questions being asked of you. Do you have any
questions about the process?
If you agree to participate, you are saying that you understand what I’ve told you and that any questions
you have were satisfactorily answered. You are also saying that you are at least 18 years old, and that
you voluntarily agree to participate. Do you agree to participate?
NExT Program: Report of Overall Project Page 33
Round 1 Facilitator’s Guide
[Note: The purpose of this document is to guide the moderator. The questions and tasks contained herein may not be asked as
written. The facilitator often draws on participant comments and the natural flow of the testing process to determine the flow of the
session. While the facilitator will try to follow the order of the guide, many times tasks will come up ahead of time or in different order. The
facilitator may allow the order of the tasks to change in order to let the process flow naturally. ]
Pre-Test
[Administer the informed consent and video release form.]
Introduction
Thank you for agreeing to participant in this study. Do you have any questions for me before we get started?
You signed a video release so you know I’ll be recording everything that shows up on the computer monitor. We’re going to be looking at a web site today that was created for the National Cancer Institute. I’m going to ask you to try to explore the web site and then to try to do some tasks with the site. Then I’d like to get your opinion about it. You and I will work on this together. Additional observers in another room may watch your monitor screen to assist me in taking notes.
We have invited individuals who have and have not experienced the NExT website. So we can understand the point of view that you are bringing to the testing, I’d like for you to tell me a little bit about your previous experience with or knowledge about the NExT website.
• Where do you work? (What have you worked previously?)
• What type of work do you do? (What have you done previously?)
• Have you ever visited and used the NExT website? How much have you used it in the last
month? In the last year?
[Participant talks….]
Today we are going to be talking about
There are two important things you should keep in mind while you work with this web site:
• First, I did not design it so you can’t hurt my feelings. If there are problems with the design I would like to discuss them with you to see if we can find a way to make the web site better.
• Second, we are evaluating the web site and not you, so you cannot make any mistakes. The web site is supposed to be intuitive and easy to use. If it isn’t, that’s a problem with the web site – not with you.
Any comments you have, either positive or negative, will help make the product better so feel free to tell me whatever is on your mind. After you’re done working, I’ll ask you a few questions and then give you some time to ask me anything you’d like. Do you have any questions for me before we get started?
Initial Impression
[Open the browser to http://next.cancer.gov/ ]
Before we get started, I’d like to let you know that as you’re using the site, if you have questions as we go along, feel free to ask. I may not answer your questions right away in order to see how well you can figure out things for yourself, but it is valuable to know if you do have questions about the site.
NExT Program: Report of Overall Project Page 34
[Facilitator should ask about specific pages and sections as they are encountered. If users explore areas listed as a task below, the facilitator should get the feedback during their exploration. Areas of the site that seem to be of interest and then are dropped should be asked about.]
Tasks
All Participants
1. How would you get to the NExT program site from the NCI website? 2. How would you get to it from the DCTD website?
[Ask participant if they have used the site before]
IF YES: Previous users of the site
1. Could you explain to me your understanding of what NExT is.
a. Did you learn this from using the website or from some other source?
b. Which parts of the site helped contribute to your understanding of what NExT is?
2. When was the last time you visited the site, and what did you use it for? [Ask user to take a look at it now]
a. Which sections of the site did you use?
b. Were you able to accomplish what you wanted to successfully?
c. Did you run into any problems? If so, what?
IF NO: New users of the site
1. First, I’m going to let you explore the web site by yourself. I want to see what you can find that might be helpful or interesting to you. I’d like you to tell me about what you see and whether it does or does not relate to finding information useful to you.
a. [Observe the participants and note what they find on their own. Keep track of any relevant comments.]
b. [If not offered, prompt for participant’s impression of the main page.]
c. [If not already covered, ask what prompted them to explore the sections they visited.]
2. [If not already discussed] Find out what NExT is, what they offer, and whether or not the program would be useful for you specifically.
All participants
1. You’ve had a chance to look at the site now and have a feel for what NExT is. How would you find out what specific services they have to offer?
2. Let’s say you have decided to apply to the program. What information do you need to submit?
a. Follow-up question: How would you submit that information?
3. If you have applied to the program and your application was rejected, what further options do you have?
4. If your project is accepted, who will be involved in deciding what resources are available to you?
5. If your project is accepted, what milestones are you expected to meet as your project progresses?
6. [For sections the participant has not explored]. Let’s take a look at the [section] section and tell me what you think about it.
Post Test Interview Questions
Now, I’d like to collect a bit of information from you using some standard tools before we discuss your experience.
[Administer the System Usability Scale (SUS) and Modified Cooper Harper questionnaires.]
NExT Program: Report of Overall Project Page 35
Next, there are a few specific questions about your experience with the website.
1. What is your overall impression of the site?
2. What are the three things you liked most about the way the site looks and works?
3. What three things did you like least about the way the site looks and works?
4. Was the site comprehensive enough to allow you to determine if the program fits your needs?
5. Were the application process and the details easy to find?
6. Do you feel that the site is logically designed so as to allow you to get details without overwhelming you?
7. What about the site surprised you the most?
8. Is there any information or other content you would have expected to find, but didn’t?
Wrap up
OK, we’re done. Are there any questions you would like to ask me about the site or about today’s experience that I did not ask you about? Thanks again for your participation.
[If this is an in-person test, pay stipend and obtain signature. For remote testing, arrangements to send checks will be arranged before this.]
NExT Program: Report of Overall Project Page 36
Round 2 Facilitator’s Guide
[Note: The purpose of this document is to guide the moderator. The questions and tasks contained herein may not be asked as written. The facilitator often draws on participant comments and the natural flow of the testing process to determine the flow of the session. While the facilitator will try to follow the order of the guide, many times tasks will come up ahead of time or in different order. The facilitator may allow the order of the tasks to change in order to let the process flow naturally.]
Pre-Test
[Administer the informed consent form.]
Introduction
Thank you for agreeing to participate in this study. Do you have any questions for me before we get started?
We’re going to be looking at a website today that was created for the National Cancer Institute. I’m going to ask you to explore the website and do some tasks with the site. Then I’d like to get your opinion about it. You and I will work on this together for about 45 minutes. Additional observers may watch your monitor screen to assist me in taking notes.
There are two important things you should keep in mind while you work with this website:
• First, I did not design it so you can’t hurt my feelings. If there are problems with the design I would like to discuss them with you to see if we can find a way to make the website better.
• Second, we are evaluating the website and not you, so you cannot make any mistakes. The website is supposed to be intuitive and easy to use. If it isn’t, that’s a problem with the website – not with you.
Any comments you have, either positive or negative, will help make the product better so feel free to tell me whatever is on your mind. As you’re using the site, if you have questions as we go along, feel free to ask. I may not answer your questions right away in order to see how well you can figure out things for yourself, but it is valuable to know if you do have questions about the site.
After you’re done working, I’ll ask you a few questions and then give you some time to ask me anything you’d like. Do you have any questions for me before we get started?
I’d like to make sure I understand your background before we get started.
OMB No.: 0925-0589-08
Expiry Date: 5/31/2011
Public reporting burden for this collection of information is estimated to average 30 minutes per response,
including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data
needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor,
and a person is not required to respond to, a collection of information unless it displays a currently valid
OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of
information, including suggestions for reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge
Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-0589-08). Do not return the completed form to
this address.
NExT Program: Report of Overall Project Page 37
[Confirm previously submitted responses on following questions:]
• Where do you work? (Industry, Government, Academia, Other?)
• Are you involved in creating drugs for potential use in cancer?
o If, no. What is your connection to NCI and potential anti-cancer agents?
o If yes, What type of work do you do? (Discovery, Development)
• Have you ever visited the NExT website (NCI Experimental Therapeutics Program)?
o [If yes] When was the last time you were there?
o About how frequently would you say that you have used it?
Tasks
[Explain that we will use a mockup of a new design for the site. Open one of the mockups 2 min (order alternated to assure that both are used and assigned, if needed, to assure each type of user does not see the same design (e.g., that not all Industry representatives see Design A ).]
1. Take a few minutes to look around this site, and tell me what your first impressions are.
a. Who do you think the site is intended for?
b. Briefly describe what the NExT program is.
2. From what you see here, can you tell me the purpose or goal of the NExT program.
a. [Answer should include something about moving drug discovery and development projects through the pipeline toward creation of new anti-cancer agents.]
3. From what is on the website, can you tell me what sorts of projects NExT support?
a. [Answer should include projects at both discovery and development stages.]
b. How could you determine whether one of your projects is a candidate for NExT?
4. From the website, what does NExT offer to researchers?
a. How does NExT differ from other programs offered by NCI, e.g., grants?
b. Does NExT provide funding for researchers? [No]
c. How do researchers benefit from becoming part of NExT? [List of resources]
5. How are NExT resources allocated to projects?
a. [Answer should include something about the NExT governance structure.]
b. How do projects proceed through the program, i.e., how do projects pass from one stage to another?
6. If you wanted to apply to the NExT program, where would you start?
a. What are the submission deadlines?
b. How long should the submission be?
c. What information do you need to submit?
d. If you applied to the program but your application was rejected, what further options do you have?
e. If your project is accepted, who will be involved in deciding what resources are available to you?
f. If your project is accepted, what milestones are you expected to meet as your project progresses?
7. Reflection on finding information on the website.
a. Which parts of the site contributed most to your understanding of what NExT is?
b. Do you think further elaboration is needed regarding how this program works?
NExT Program: Report of Overall Project Page 38
c. What is the role of the Principal Investigator in approved NExT projects?
i. If it’s not clear, what is missing and where should the information go?
Post Test Interview Questions
Next, I have a few specific questions about your experience with the website.
9. What was your overall impression of the site?
10. Do you think that the homepage makes an appropriate “first impression” on visitors?
a. If not, what would you change on the home page to make it more accurate in representing the program?
11. What are your thoughts on how well the site allowed you to determine if the NExT program fits your needs?
a. Probe: --Was there enough detail? --Was there too much detail? --Was the language easy to understand?
12. Is there any information or other content you would have expected to find, but didn’t?
13. What are the three things you liked most about the way the site looks and works?
14. What three things did you like least about the way the site looks and works?
15. What about the site surprised you the most?
16. What other suggestions do you have for improving the site?
Comparison
1. Take a look at the graphic on the homepage. (Earlier you said…)
a. Tell me in words what this is depicting.
b. Does this help you understand the NExT process?
i. Is there anything missing?
ii. How could it be improved?
c. Here is another graphic that we are considering. [Show alternative mockup.] Which of these do you think is more effective at conveying the NExT program?
2. [Show alternatives for stage gates] We have two ways of displaying the same information for NExT stage gates. Which do you prefer, and why? [In How Next Works]
3. [Show alternatives for landing pages] We have two ways of displaying information on these pages. Which do you prefer, and why?
[IF THERE’S TIME]
d. How would you get to the NExT program website from http://www.cancer.gov ?
e. How would you get to it from http://dctd.cancer.gov ?
Wrap up
OK, we’re done. Do you have any questions about today’s session? Thanks again for your participation. I’ll be sending you an email asking for your mailing address so that we can send you your $60 stipend.