Usability Report

Usability Study Report, Part 1Usability Testing

August 2011

Collaborative Institutional Training Initiative (CITI) www.citiprogram.org

Prepared by: Susan Mallgrave, Project Manager and Usability Specialist, UTEC, IIT Halcyon Lawrence, Usability Specialist, UTEC, IIT Prepared under the direction of: Dr. Susan Feinberg, Director, UTEC, IIT

1

Table of ContentsIntroduction........................................................................................................................3

Heuristic Evaluation..........................................................................................................3

Methodology....................................................................................................................3

Heuristics Used................................................................................................................4

Severity Rankings............................................................................................................5

Summary of Findings and Recommendations.................................................................5

Methodology of Testing...................................................................................................11

Recruitment and User Consent......................................................................................11

Gathering User Profiles.................................................................................................11

User Profiles..................................................................................................................11

Investigation of User’s Mental Model............................................................................12

Methodology..................................................................................................................12

User’s Mental Model Findings......................................................................................13

Usability Testing..............................................................................................................18

Methodology..................................................................................................................18

Usability Test Findings and Recommendations............................................................18

Usability Test Positive Findings....................................................................................18

Usability Test Findings For Improvements...................................................................18

Appendices........................................................................................................................28

Appendix 1: Facilitator Script......................................................................................28

Appendix 2: Consent Form...........................................................................................29

Appendix 3: User Profile Questions.............................................................................30

Appendix 4: Task Scenarios.........................................................................................31

Appendix 5: Time Taken on Tasks...............................................................................33

2

Introduction Under the supervision of Dr. Susan Feinberg, Director of the Usability Testing and Evaluation Center (UTEC) at Illinois Institute of Technology (IIT), a team of usability specialists conducted

1) a heuristic evaluation, used to compare a pre-defined set of usability principles to a product or website, in order to identify usability flaws;

2) an investigation of the user’s mental model, used to reach findings that lead to reorganization of the information on the site (information architecture);

3) a usability test employing task scenarios, used to observe users’ behavior while performing tasks within the website.

The purpose of using these three elements in this first phase of a planned three-phase study of the CITI website was to familiarize the team with the CITI website and with the features that need redesign.

Heuristic EvaluationMethodology

The first element of Phase 1 in the study of the CITI website is a Heuristic Evaluation conducted to familiarize the consulting team with the website. Typically, a heuristic evaluation is conducted by multiple evaluators as “different people find different usability problems.”1 However, in the case of this project, the evaluation was conducted by one team member as part of the website familiarization process. The outcomes of this process subsequently informed the consulting team as to the types of tasks they would ask participants to perform during the usability test conducted in Phase 1.

The evaluation therefore centered on nine areas student users typically interface with on the site. These areas were outlined by the client2 and are:

1. Public Pages2. Registration for private environment3. Registration of a new learner4. Available course menu (across courses)5. Available module menu (within course)6. Transcript (across courses)7. Module content delivery8. Quiz content delivery9. Satisfaction survey/feedback.

Heuristics Used1 http://www.useit.com/papers/heuristic/heuristic_evaluation.html2

3

There are 7 heuristics used in this evaluation. It is important to note that the number and type of heuristics vary from project to project. The consulting team took great care in selecting heuristics that would uncover the challenges experienced on CITI’s site. The following is a detailed description of the heuristics used in the evaluation:

1. Visibility of system status & provide user feedback Inform the user of what the system is doing. Inform the user when a task is completed. Show progress achieved toward a task, especially if the task has phases or

takes the user time to complete.

2. Explorable interface Allow easy access to frequently used commands. Provide navigation structure that mirrors the site’s architecture. Allow for shortcuts for expert users. Create logical groupings for tasks or for content.

3. Consistency of interface design & presentation of information Be consistent in the presentation of information (format and layout) across the

site. Be consistent in the labels used within tasks. Ensure that the same labels perform the same tasks.

4. Flexibility and efficiency of use Ensure that users can work quickly and efficiently to complete tasks. Provide multiple ways of completing tasks. Make commonly accessed commands easily available to users. Minimize user’s memory load by providing adequate user cues and tools.

5. Aesthetic and minimalist design Provide a simple structure which makes content easy to understand. Avoid irrelevant and extraneous information that may confuse the user.

6. Readability and user cues Present content in plain language. Use words based on the task to be completed. Present content in manageable portions for the reader to access. Provide adequate cues (e.g., consistent use of color) to assist the user to

navigate the content. Use appropriate font features (size, spacing etc.) to enhance readability.

7. Help, documentation, and error prevention: Allow the user to cancel a task without interruption of the system. Provide access to help on specific tasks through search functions. Allow error messages to indicate a specific problem in plain language. Suggest solutions to the user to fix the problem.

4

Severity Rankings

The following provides a guide to the ranking system used to prioritize problems encountered in the site and these rankings are based on those defined by Jakob Nielsen3. In the summary of findings which follows, problems which are rated four (4) are shaded grey to pinpoint the areas where the client might begin to focus their efforts in the site’s development.

Rating Definition0 Not a Problem: violates a heuristic but doesn’t seem to be a usability problem.1 Cosmetic Problem Only: need not be fixed unless extra time is available on

project.2 Minor Usability Problem: fixing this should be given low priority. 3 Major Usability Problem: important to fix, so should be given high priority. 4 Usability Catastrophe: imperative to fix this feature so as to improve the user

experience. This problem is in urgent need of attention.

Summary of Findings and Recommendations

1. Visibility of System Status

Site Area Specific problem Recommendation RatingNew Learner Registration

Applying for CME/CEU credit takes you to an information page, not an application page. It is unclear how this information relates to the application. There is a “Submit” button at the end of the page which can be missed.

Redesign this page to ensure that users understand the process involved in applying for CME/CEU credit.

.

4

New Learner Registration

There is no indication that students have successfully applied for CME/CEU or guidance as to what they do next in the process.

Include a status indicator when the application is successful

3

Module Menu

There is no indication where students are within a module;

Include a status indicator which can let students know

3

3 http://www.useit.com/papers/heuristic/severityrating.html

5

Site Area Specific problem Recommendation Rating(Within Course)

therefore, it may be difficult for students to leave and return to a module to complete it.

where they are in the module, what % work completed, etc. and allow them to pick up where they left off.

Quiz Content Delivery

There is no indication as to how many questions comprise a quiz.

Indicate how many questions are in a quiz.

2

Satisfaction Survey

Too much scrolling is required to complete the survey; a more dynamic approach to the survey can be taken.

Have each question appear on a new screen (not window), with a status indicator to tell users where they are in the survey. For example “Question 1 of 5.”

2


There is no indication of how many steps are required to complete registration. Steps 1-5 are numbered for the the login information, but this is just phase one of the registration process.

Include a visual status indicator for the registration process beyond the initial 5 steps needed to create an account.

2

2. Explorable Interface (Navigation, Labeling, Search Functions etc.)

Site Area Specific problem Recommendation RatingPublic Pages

There is no way of indicating to users where they are in the site’s structure. As a result, there is no way of developing a sense of the site’s organization.

Use breadcrumbs to tell users where they are in the hierarchy.

4

Public Pages

“HIPS” course, “International Course Site”, “Biosafety demo” and “RCR course” links take the user to the same location.

Include a “Courses” link in the main navigation, with a list of course names as sub-navigation items.

4

Public Pages

Information in hover boxes over the navigation links may suggest that a more intuitive labeling system is needed.

Use more specific labels for the navigation tabs. Please refer to page 13—User Mental Model Findings and Recommendations for a detailed discussion.

4

Public Pages

All second level navigation is in the form of text hyperlinks creating problems of structure.

Replace hyperlinks with a second, and if needed, a third tier of navigation.

Public Because students, institutions, and Place login information in an 4

6

Site Area Specific problem Recommendation RatingPages administrators all have to login to

access course or administrative information, users to the site should be able to log on from any of the public pages without having to return to the homepage.

area easily visible and recognizable to the user and in a consistent location from page to page.

Public Pages

The CME/CEU link at the bottom of the page is vague and links to an even more ambiguous page. Only those who are aware of applying for CME credits may be familiar with the term “CME/CEU.”

In the process of conducting this evaluation, the researcher did a google search to find a definition of CME/CEU, as there was none available on CITI’s site.

Make navigation labels more intuitive and recognizable. Include a clear definition of CME and CEU on the related pages.

3

Public Pages

There is no contact information provided on the “Contact Us” page.

Include full contact information for CITI agents.

3

Transcript (across courses)

The “Return” link on the transcript page (Completion Report) takes the user to a detailed transcript. The label “return” implies a very different action.

Replace the word “Return” with “Transcript details” or “Detailed Completion Report”

3

Quiz Content Delivery

The invitation to take the quiz appears at the bottom of the ‘answer sheet’ of the last quiz taken. There is no guarantee that the participant will scroll to the bottom to see this.

Move the “Take a Quiz” link to the top of the page

2

7

3. Consistency of Interface Design & Presentation of Information

Site Area Specific problem Recommendation RatingModule Content Delivery

The information presented in each module is formatted and presented differently across modules. This may affect the efficiency with which students can access and read the module.

Reorganize and reformat course modules so that they are consistently presented to the user. An instructional designer could be a useful resource in this task.

4

Public Pages Navigation items at the top and bottom of the page don’t consistently appear across all pages, e.g., HIPS course link.

Ensure that all navigation links appear consistently on every page.

3

Public Pages The purpose of the navigation links at the bottom is unclear. Are they to mirror main navigation links? For example it is unclear why “Human Subjects Research” (main navigation) and “About Us” (bottom navigation) take users to the same page.

Ensure that each label must have unique and clear navigation function.

2

Public Pages The appearance of CITI Knowledge Base is inconsistent, compared to the rest of the site.

Knowledge Base is part of the software used to manage help desk inquiries; Reid Cushman will inquire about the customizability of the interface.

1

Register for Private Environment

The website’s home page presents only a Login prompt. No other information is presented on the homepage.

Make better use of home page ‘real estate.’ Might include information on what CITI is and include topics such as Latest News; also make login area less prominent but still visible and easily accessible.

1

4. Flexibility and efficiency of use


In most cases, course material is accessed through a series of external links. As a result, students may need guidance as to

Use sub-menu navigation instead of links.

Include a required and

4

8

Site Area Specific problem Recommendation Ratingwhat is required reading and recommended reading, as the ubiquitous use of links does not allow the user to prioritize their study plan

recommended reading section. An instructional designer could be useful in this process.

Transcript (across courses)

The dialog box to print completion reports appears immediately, not giving users the opportunity to review the transcript before printing it. Users must cancel the print command if they wish to review their completion report.

Remove the print dialog box command. Present the completion report first and then allow users to select the option to print the report.

3

5. Aesthetic and Minimalist Design


There is a need to regulate how much information the student sees at any one time while taking a course.

Introduce a hide/show feature which gives users flexibility and functionality as they go through course material. An instructional designer could be useful in this process.

2

Public Pages

Forms for institutions to register for CITI services are embedded as hyperlinks to Excel documents. These forms are integral to the registration process and could be made more accessible to the institutions.

Include a main navigation link which accesses all registration forms.

2

6. Readability & User Cues

Site Area Specific problem Recommendation RatingPublic Pages

Text on the public pages is difficult to read. All text appears in Arial – a sans-serif font typically associated with headings. Text presentation is monotonous with little variation between paragraphs.

Convert text to a serif font, such as TNR. Present information in smaller units.

4

Course The “Instructions for CITI Include a short training clip 3

9

Site Area Specific problem Recommendation RatingMenu (Across Courses)

Learners” are very difficult to follow. No visual support is provided.

or screen shots to accompany instructions.

Public Pages

Registration forms seem complicated to fill out.

These forms can be integrated into the website as a database function, where institutions can register online and the data is sent directly to a database.

CITI’s programmers should be able to suggest some alternatives.

2

Module Menu (within courses)

Use of color can be problematic. For example, hyperlink text “Enter or Re-enter” to begin and continue modules and module status “incomplete” use the same colors.

Implement a consistent color scheme to enhance readability. If hyperlinks are red, then avoid using red for other text on the site.

1

7. Help, Documentation and Error Prevention

Site Area Specific problem Recommendation RatingPublic Pages Help and documentation appear

in the form of an FAQs link; however, the link takes users to a resource page where the FAQs link is the 5th link down the page.

Ensure that FAQs link takes users to the FAQs only. There can be a separate Resources link if deemed necessary.

4


It is unclear why a username may be initially rejected beyond the explanation that it already exists.

Allow the the system to suggest possible user names when one is rejected. It gives users a better idea of what’s available.

2


It may be unclear that students need to choose only ONE affiliate institution.

Repeat instructions that only one affiliated university can be added at a time. The word “OR” after each drop down box may be one way to address this, or a greyed-out area for non-selected items.

1

10

After the team reviewed and analyzed the heuristic evaluation findings, we identified a number of tasks that we would present to users in order to observe and record the user's experience on CITI's site. The following section explains how we conducted the testing of the website.

Methodology of Testing The usability test involved five users from the target audience (students who represented typical website users) and took place on August 10-11, 2010. All five users were tested at the UTEC lab on the campus of IIT.

Recruitment and User Consent

After inviting users to participate in the testing, we scheduled a time slot with each of them.

On the day of testing, we read a script to those participants who came to the UTEC for testing of the website and asked them to sign a consent form to confirm their agreement for participating in the usability study. (For the forms used, please refer to the Facilitator Script/Appendix 1 and Consent Form /Appendix 2).

Gathering User Profiles

Testing for Phase 1 required learning about the users through questions related to their work experience, their background in ethics, and their level of experience with the internet and with online training platforms. (For a list of complete questions, please refer to User Profile Questions /Appendix 3.)

User Profiles

The profiles of the five users who participated in this study are summarized below:

User #1 User #2 User #3 User #4 User #5Title Grad

StudentGrad Student

Research Assistant

EducationalConsultant

Grad student

Teaching experience

No Yes No Yes Yes

Has included ethics considerations in research proposals?

No No No Yes No

Has included ethics in the classroom?

No No No Yes No

Has used site before?

No No No No No

11

User #1 User #2 User #3 User #4 User #5Level of experience with the internet—low, medium, high

High Medium Medium High Medium

Level of experience with online training platforms—low, medium, high

Medium Medium Low Low Medium

Field of study Computer Network Security

TechnicalCommunication

Electrical/Biomedical Engineering

History/Education

Theater/Drama/Performing Arts

Voluntary information—age bracket

15-28 29-44 15-28 29-44 29-44

Voluntary information—gender

Male Female Male Male Female

Investigation of User’s Mental Model

Methodology

After learning more about the participants, we moved to the second element of Phase I in our study of the CITI website--an exercise to investigate the user’s mental model, based on the navigation labels provided on the home page. To prepare for the exercise, participants were asked to read the “About CITI” page and were then asked to provide feedback on the content they expected to find under each link. Participants were not allowed to hover over the links. Below is a summary of the findings; in Phases 2 and 3 of this project, these findings (based on the user’s mental model) will be used to make recommendations for the re-organization of information on the site.

12

User’s Mental Model Findings

The following findings and discussion are based on a user’s mental model exercise conducted during the usability test. As a guide to the findings, the following definitions are provided:

“Navigation Label” refers to the existing labels used for the main navigation links, for example, “Home” or “Human Subjects Research.”

“Existing Headings” refers to the headings used on the pages linked to the main navigation, for example, “About the Collaborative Institutional Training Initiative.”

“Existing Content” refers to the content or hyperlinks found under the existing headings, for example, the list of participating institutions and organizations.

“Participants' Expectations” are areas of content that users expected to find under each navigation label. “Findings” provides a discussion of the participants' expectations vis-à-vis the first navigation label, existing headings, and

existing content.

Navigation Label

Existing Headings Existing Content Participants’ Expectations

Findings

Home Welcome

CITI Login and Registration Page

Register

Login

Forgot Login information

General information

Link to home page

Contact information

FAQs

Login information

For the most part, the home page matches users’ expectations. The inclusion of general information on the home page, for example, “About CITI” may be good utilization of the home page real estate.

Human Subject Research

About the Collaborative Institutional Training Initiative (CITI)

List of participating institutions and organizations

Information about conducting Human Subjects Research

Information about

There is a mismatch between the navigation label, the existing heading and the content.

13

Navigation Label


Findings

CITI Developer’s group

CITI Advisory Committee

CITI Slideshow Presentation

biomedical research

Images of human subjects (people)

Education about how people learn

Research tools based on different subjects

This information needs to be streamlined. Users’ expectations varied widely even after having read the information presented on this page.

Become a Member

CITI Human Subjects Research Educational Program Information (CITI)

CITI order form

CITI RCR order form

Order form in Spanish

Services for members

Pay site

Registration

Sign up page

Join/become a member

It is unclear to users if this page is relevant to student users or institutional users. A label like “Become an Institutional Member” or “Register your Institution” may be more helpful navigation labels.

International Course Site

The CITI International Research Course Site

Register for the IRB training course

Registration for IRB course

Don’t know what to expect

For people who speak languages other than English

How you would register to be a member depending on what country you are from

It is not immediately clear how the International Course Site links with registration for the Institutional Review Board (IRB).

Participants varied widely on their expectations for this link. “International

14

Navigation Label


Findings

Links to courses from different countries

Research Courses” may be a more appropriate navigation label.

HIPS Course None Registration Link to HIPS course

Something related to privacy

Noted HIPS was referred to in the About page

Some users were able to recognize the initials RCR, HIPS from the “About CITI page” they were given to read before the exercise. Two users suggested having one tab for courses, with sub- navigation listing all the courses which CITI offers. This is an excellent recommendation which will allow more space to spell out RCR and HIPS instead of using initials.

Additionally, the links to these pages should consistently give specific information about these courses and not just a registration page, as is the case of HIPS course page.

RCR Course CITI Course in the Responsible Conduct of Research

Registration for RCR CourseDirections to instructors and administrators

CITI program home page

Don’t know what RCR stands for

Links to Responsible Conduct of Research

Lab Animal Course

The CITI Laboratory Animal Welfare Course

CITI lab animal welfare working group

CITI lab animal demo

Permissions for subscribers

Other advantages for institutions

Value Added

CITI lab animal working group

Lab animal order form

Knowledge base website

Link to Lab Animal Course

Information about meeting regulations in lab animal research

15

Navigation Label


Findings

Subscription information

Biosafety The CITI Biosafety Course

Registration for the Biosafety course “demo”

CITI Biosafety course order form

Information about biosafety course offering

Does not understand what the word means.

See comments for HIPS, RCR and Lab Animal Course

FAQ Resources Knowledge base

Contact CITI support

Questions and answers about how the courses conducted

There is a mismatch between the label “FAQ” and the heading “Resources.” This needs to be addressed. Content management for this page is needed and information needs to be organized accordingly, since participants only expect to find a list of FAQs under this link.

Language Select Language Arabic, Chinese, English, French, Japanese, Korean, Portuguese, Russian, Spanish, Thai

Other languages options

Site translated into 10 different languages

No change is recommended.

16

Usability TestingMethodology

After completion of the user’s mental model exercise, the usability team moved to the third element of Phase I in our study of the CITI website—a formal usability test, employing guided tasks. For this part of the testing, specific task scenarios had been prepared by the usability team, in accordance with the client’s requests, in order to test for ease of usability of the website. (For a list of task scenarios for testing, please refer to the Task Scenarios/Appendix 4.) Below are the usability test findings and recommendations.

Usability Test Findings and Recommendations

This section covers the main findings that resulted from the first usability study of the CITI website. An actionable recommendation follows each finding. Additionally, a comparison of estimated time and actual time taken on each task is provided in Appendix 5. It is useful to note that completion of tasks is not the only marker of success, the time taken allows us to judge if users are able to work efficiently as well.

Usability Test Positive Findings

One user (a student of “computer network security”) noted that the “site works perfectly fine, functionality-wise, with no errors” (none that this user detected).

A majority of the users found the content contained in the class projects basic course “very interesting” and several expressed a desire to read this content at their leisure.

Usability Test Findings for Improvements

Some basic areas to be tested had been agreed upon in consultation with the client. Once the heuristic evaluation was completed and the team was familiar with the site, the task scenarios were created. For example, the client had expressed interest in having elements of the public pages tested and the team designed task scenario 1 (see below) to test the use of public pages. The following are the tasks presented to each participant, the findings for each task and our team’s recommendations.

Task Scenario 1:

“You are a research assistant who must earn CME/CEU credits. Find the CITI Knowledge Base and do a search to find out how to apply for CME/CEU credits.”

The objective of the first task in the usability test was to involve users in browsing the public pages of CITI, without the need for login.

Finding 1 (Task 1):

17

A majority of users had significant trouble locating the CITI Knowledge Base. Several users devised various “work-around” strategies—one user ultimately used a Google search to locate the database and 2 users attempted to use local “find” function. One user commented, “I don’t feel like there is an area to find this [Knowledge Base] unless there was an area to search.” A user who was not successful with this first task commented “when I got to the [U Health page], I didn’t really find how I could apply for the credits. This [how to apply for CME/CEU credits] is really hard to find. So hard.”

Recommendations based on finding 1 (Task 1):

Integrate a search engine, like Google, on the site. Use breadcrumbs to tell users where they are in the hierarchy. Provide a site map on the website.

Finding 2 (Task 1):

Only 2 users found the database using the FAQs navigation tab. Two of 5 users did not find the database at all. Additionally, a majority of users thought it was mandatory, or might be necessary, to register on site before accessing any information, including information on public pages. Some user comments: “None of the links on the top gave any clue where to go for guests and the home page says to register if I’m a new user—there doesn’t seem like any other way to go for somebody who is just a guest;” and another user: “It’s very obvious that I need to login first.”


Locate a link to CITI Knowledge Base prominently on every page. The label “CITI Knowledge Base” may not be universally understood

as a place to find help. A main navigation label “Help” which takes the user to the Knowledge Base is an alternative.

Leave the login prompt on the home page, but demote it, so that it is visible and accessible but not the focus of the home page. As one user stated “don’t give all that space on the home page to registration – that can be a small box on the upper right or somewhere – it’s too prominent.” See Figure 1 for an example of login prompt placement which is accessible but not overly prominent.

18

Figure 1: Placement of login prompt

Task Scenario 2:

“You are a grad student enrolling in a course that will include a research project requiring that you complete a course on the CITI website. Create a new learner account and register for the “class projects basic course.” (Users were given some additional information they needed in order to complete the registration process, such as to use Halcyon University for their institution, etc.)

The objective of the second task in the usability test was to involve users in login and registration to gain access to the private environment of the website and was tested by creating a new learner account.

Finding 1 (Task 2):

A majority of users were confused by “choose only one” instruction for institution drop-down boxes in the registration process. One user commented, “it could be clearer that you are only supposed to check one,” and another said, “not clear whether I have to select any others . . . because they’re all available and parallel.”

Recommendation based on finding 1 (Task 2):

Block access to all other options when one is selected or use a greyed-out option.

Finding 2 (Task 2):

Login area is accessible but

not the main focus of the

page.

19

Several users were confused by only one box being offered for selection or non-selection of “good clinical practice” in the registration process. One user commented, “there is no choice given for “good clinical practice,” although it says “choose all that apply,” and “please make your selection—unclear exactly what the checkmark means . . . typically this would offer yes or no.”


Offer two check box options that clearly indicate acceptance and rejection of the “good clinical practice” option. Additionally, the statement “choose all that apply” may lead to confusion since there is only one option available. Delete this directive.

Task Scenario 3:

You are a student who needs CME/CEU credits for a medical affiliation. Go to the page where you can apply for these credits and also indicate that you are willing to take a survey at the end of the process.

The objective of the third task in the usability test was to involve users in login and registration to gain access to the private environment of the website and was tested by applying for CME/CEU credits and indicating willingness to take a survey at the end of the process.

Finding 1 (Task 3):

A majority of users were curious about what the letters CME/CEU mean.


Adhere to the rule of thumb for clear communication: spell out all words before you use acronyms and abbreviations.

Because of the technical nature of this site, it may be useful to have a glossary of terms available to users.

Finding 2 (Task 3):

Four out of 5 users gave no indication they saw the CME/CEU navigation tab on the bottom of pages; only one user clicked on this link.


Promote CME/CEU link to the main navigation area at the top of the page.

20

Finding 3 (Task 3):

Four out of 5 users were uncertain as to whether or not they had successfully applied for CME/CEU credits. One user commented, “how do I know if I’ve successfully applied? Something should have appeared to tell me.” When asked by the test facilitator if another user thought she had successfully applied for the CME/CEU credits, she responded, “I have no idea. I would say yes, because I went through the steps and then clicked submit . . . but it doesn’t . . . I don’t feel like there’s an area showing me that I’ve done that.”


Use a status indicator to let users know when a task or function is completed (see figure 2.) As one user commented when asked about the level of difficulty experienced in applying for CME/CEU credit, he responded: “I’m not sure I ever did that, so I’ll go with ‘somewhat difficult.’ ”

Figure 2: Example of a status indicator

Provide clearer instructions on the process of applying for CME/CEU credit. Currently, it seems that the institution requires that you read an information page, before you can click “submit”. The purpose of the information page may not be clear to the user. You can separate this function into two tasks. First, a link to an “About CME/CEU” and second, a command option that only allows users to apply for credit. An option to indicate that a user has read the “About” page before he/she applies for CME/CEU credit can be included.

Change the label “submit” to “apply”. This word choice may lead to less ambiguity for users as to the task they are completing. (See figure 3)

21

Figure 3: CME/CEU Credit PageTask Scenario 4:

You are a student who must complete a class projects basic course and take a quiz to demonstrate that you have passed the course. View the ‘Halcyon University Instructions’ page as a guide to how you can access and complete the “class projects basic course” and the required quiz. The answers to the quiz are attached on the following page.

The objective of the fourth task in the usability test was to involve users in the process of working within the available course menu (across courses).

Finding 1 (Task 4):

A majority of users thought the words ‘Enter/Re-enter’ in red (see figure 4), underlined text on the basic course instructions page was a clickable link as this is a common convention used in website content. The convention is used inconsistently on the CITI site, a practice that helps create unreliable expectations for users. After several attempts to click on the red, underlined ‘enter/re-enter’ one user commented, “I can’t go anywhere.” Because the words are also part of a phrase that says, ‘The red link Enter or Re-Enter will permit you to Begin/Continue the Course’, this reinforced the illusion that the red text would take users to the course directly from this page, as several users reported in the debriefing after being tested on tasks. As one user said after fruitlessly clicking on the words ‘Enter-Re-enter’, “the red link, this is supposed to take me there and it’s not.”

Submit button appears at the end of a long page

22

Figure 4: Problem of misleading “link look-alike” on the “Halcyon University Instructions” page.

Recommendations based on findings 1 (task 4)

Provide a more interactive page for users. As one user stated, “it should be both instructions and actionable; killing two birds with one stone...makes it more efficient.”

Use screen shots in the instructions rather than actual text; this way users will know you are referring to another page on the website rather than trying to click what they perceive as links to complete the tasks.

Consider adding a short 3-minute video clip as an alternative to written instructions.

Finding 2 (Task 4):

A majority of users hesitated over “Belmont Report” & CITI Course Introduction” on the “Required Modules” main page and were not sure this was the proper selection. The module is called “Class Projects Basic Course” until the user arrives at this page, at which point the user is offered the unfamiliar title “Belmont Report.”

Recommendation based on finding 2 (task 4)

Use a status indicator to tell students where they are at in the module and which modules they have completed. A status indicator has been used (see figure 5), but not consistently within course modules.

23

Figure 5: Status Indicator used in course modules menu

Finding 3 (Task 4):

A majority of users commented on how much text was contained on pages within this section of the site. One participant used the word “overwhelming” as she scrolled through “Students in Research” module. The same user went on to say, “you get in the middle of it and don’t know how you can get out of it . . . information is in one layer—should be ‘like a book,’ with many pages.” Another user calls a lengthy page “verbose.”

Recommendations based on finding 4 (task 4):

These course modules were interesting to users but difficult to navigate. These module pages require organization and proper instructional design to be effective. Modules need to be broken down into pages and students should be given both recommended and required readings so that they can prioritize their study programs. (Figure 6a and 6b provide a visual comparison of existing layout and proposed layout )

If a student returns to the course, there should be some indicator of what was last covered and where they resume their study rather than having to scroll through the material.

Status indicator used in modules

24

Figure 6a: Existing Content Layout

Figure 6b: sample of a re-designed module to avoid information overload

25

Task Scenario 5:

You have completed the class projects basic course and are required by your own university to submit proof of this achievement. Print a copy of your completion report and when you are finished, log off from the site.

The objective of the fifth and last task in the usability test was to involve users in accessing and printing a copy of completion report.

Finding 1 (Task 5):

A majority of users were confused by some element of printing the completion report. Users’ comments ranged from “confirmation should have shown up automatically,” to “the quiz itself should be printed as completion report,” to “I have no idea where to go, to print a copy.”


There are too many final reports and that may be confusing to students. To name a few, there are the “completion report” page, “modules completed for stage” page, the “course completion history” and the “archived completion reports.” Provide students with a reports tab that lists all the reports they can print. Provide a short description of each report and what it offers and indicate which reports are required for their institutions.

26

AppendicesAppendix 1: Facilitator Script

Hi,

I’m ____________ and I’m your facilitator for today’s usability testing. Let me tell you a little bit about usability study so that you fully understand what’s going to take place today and how valuable your input is to us.

In a nutshell, usability testing is the evaluation of a product or website based on how real users of the product or website are able to easily and successfully perform a task. This usability evaluation will be for the website of the Collaborative Institutional Training Initiative (CITI). The session will last about an hour and we have 5 tasks to go through.

Please be assured that YOU are NOT being evaluated—we’re evaluating the website. There are no wrong answers. We are not the designers or developers of the website; rather, we provide a third party non-biased website evaluation service.

Your identity will stay anonymous. We are only reporting results, without identifying users. As you fit the profile of the target audience for this website, you are going to help us learn how users find information on the CITI website. We encourage you to talk as you are browsing and tell us what you are thinking. It will really help us find ways to recommend improvements to the current website. Once again, we are NOT testing YOU, we’re testing the website.

Do you give us your consent to participate in this study?

(Have participant sign consent form.)

Again, we encourage you to think ALOUD and tell us your reasoning behind clicking on certain links!

Questions? Ready? Okay, let’s begin.

27

Appendix 2: Consent Form

Usability Study of Collaborative Institutional Training Initiative (CITI)

I agree to participate in the usability test of the University of Miami CITI site, being conducted by Susan Mallgrave and Halcyon Lawrence, under the direction of Dr. Susan Feinberg, Director of the Usability Testing and Evaluation Center (UTEC) at Illinois Institute of Technology (IIT). I understand that this participation is entirely voluntary; I can withdraw my consent at any time without penalty and have the results of the participation, to the extent that it can be identified as mine, returned to me, removed from the research records, or destroyed.

The research has been explained to me as follows:

● The purpose of the study is to evaluate the usability of certain portions of the website. The researchers wish to determine the ease of use of the site and how users perceive it.

● Participants will work at a computer that has access to the specified website and be given tasks related to accessing the site. While the participant attempts these tasks, he or she will be directly observed and videotaped so that the research team can review the participant’s interactions with the instruction and the participant’s comments during this interaction. The entire activity will take approximately 1 hour.

● The participant’s identity will be kept confidential in any transcriptions and reports generated from this research. Any comments or expressions made during the usability test may be used for the purpose of evaluating the website and showing the results of this research. All videotapes related to this research project will be retained by UTEC.

The researcher will answer any further questions about the research, now or during the course of the project.

Please sign both copies of this form. Keep one and return the other to the researcher.

_______________________ _________Signature of Participant Date

_______________________ __________ Signature of Researcher Date

28

Appendix 3: User Profile Questions

● What is your current title?

___________________________

● Do you have teaching experience?

Yes _________ No _________

● Have you included ethics considerations in research proposals?

Yes _________ No _________

● Have you studied ethics in the classroom?

Yes _________ No _________

● Have you used the CITI site before?

Yes _________ No _________

● How would you characterize your level of experience with the internet—low, medium, high?

Low _________ Medium _________ High _________

● How would you characterize your level of experience with online training platforms—low, medium, high?

Low _________ Medium _________ High _________

● What is your field of study?

____________________________

● Voluntary information—please provide your age bracket:

15-28 _________ 29-44 _________ 45-59 _________

● Voluntary information—please provide your gender:

Female _________ Male _________

29

Appendix 4: Tasks Scenarios

Task 1:

You are a research assistant who must earn CME/CEU credits. Find the CITI Knowledge Base and do a search to find out how to apply for CME/CEU credits.

Objective: This scenario covers browsing public pages, without the need for login.

Successful Path: http://citiprogram.supportcenterpro.com/knowledgebase/cme-credits/

how-do-i-apply-for-cmeceu-credit.html

Task 2:

You are a grad student enrolling in a course that will include a research project requiring that you complete a course on the CITI website. Create a new learner account and register for the “class projects basic course.”

Objective: This scenario covers login and registration for private environment

Successful Path: https://www.citiprogram.org/members/mainmenu.asp?

strKeyID=8D48CCC1-CB8D-40C1-B9BA-243FA4F1E6AD-6450701

Task 3:

You are a student who needs CME/CEU credits for a medical affiliation. Go to the page where you can apply for these credits and also indicate that you are willing to take a survey at the end of the process.

Objective: This scenario covers login and registration for private environment.

Successful Path: https://www.citiprogram.org/members/learnersii/cmecredits.asp?

strKeyID=9ECC459F-E4F0-46A3-9EFB-3BF4645D7C6D-6450701

30

Task 4:

You are a student who must complete a class projects basic course and take a quiz to demonstrate that you have passed the course. View the ‘Halcyon University Instructions’ page as a guide to how you can access and complete the “class projects basic course” and the required quiz. The answers to the quiz are attached on the following page.

Objective: This scenario covers working w/n the available course menu (across courses)

Successful Path:

https://www.citiprogram.org/members/learnersII/moduletext.asp?strKeyID=84F334F2-

89CF-4371-A9CC-A0A5DBAFCBBA-6450701&module=1321 and

https://www.citiprogram.org/members/learnersII/exam.asp?strKeyID=ECB00EB8-

D2B3-4F33-83E7-2971D14C0127-6450701

Task 5:

You have completed the class projects basic course and are required by your own university to submit proof of this achievement. Print a copy of your completion report and when you are finished, log off from the site.

Objective: This scenario covers accessing a transcript (across courses).

Successful Path:

https://www.citiprogram.org/members/learnersII/crbystage.asp?strKeyID=DFF8960C-

44A6-48EE-A1A2-A5D84CA936F1-6450701&gradebook=34861 and

https://www.citiprogram.org/login.asp?strKeyID=87BF5C23-91E8-42B0-87E3-

CDC4977299AF-6450701&language=english

31

Appendix 5: Time Taken on Tasks

Scenario 1 - Use CITI Knowledge Base Successful? Predicted Actual DifferenceParticipant 1 No 2:00 5:00 (3:00)Participant 2 No 2:00 3:20 (1:20)Participant 3 Yes 2:00 3:32 (1:32)Participant 4 Yes 2:00 4:30 (2:30)Participant 5 No 2:00 1:09 0:51

Scenario 2 - Create New Learner Account Successful? Predicted Actual DifferenceParticipant 1 Yes 5:30 7:02 (2:02)Participant 2 Yes 5:30 8:42 (3:12)Participant 3 Yes 5:30 4:29 1:01Participant 4 Yes 5:30 4:41 0:49Participant 5 Yes 5:30 7:16 (1:46)

Scenario 3 – Apply for CME/CEU Credits Successful? Predicted Actual DifferenceParticipant 1 Yes 1:30 1:56 (0:26)Participant 2 Yes 1:30 1:38 (0:08)Participant 3 No 1:30 2:37 (1:07)Participant 4 Yes 1:30 0:27 1:03Participant 5 Yes 1:30 3:31 (2:01)

Scenario 4 - Take a Course/Quiz Successful? Predicted Actual DifferenceParticipant 1 Yes 10:00 12:31 (2:31)Participant 2 Yes 10:00 15:55 (5:55)Participant 3 No 10:00 13:53 (3:53)Participant 4 Yes 10:00 8:30 1:30Participant 5 Yes 10:00 17:03 (7:03)

Scenario 5 - Print Completion Report/Log Off Successful? Predicted Actual DifferenceParticipant 1 Yes 1:00 1:48 (0:48)Participant 2 Yes 1:00 0:42 0:18 Participant 3 Yes 1:00 1:11 (0:11)Participant 4 Yes 1:00 1:08 (0:08)Participant 5 Yes 1:00 1:05 (0:05)

32

Usability Report

Documents

Transcript of Usability Report