CFF Final Report

7/28/2019 CFF Final Report

1/23

Usability Test Report:

The Cystic Fibrosis Foundation - CFF.org

Emily Lazo

Caron Garstka

December 3, 2012


2/23


3/23

Introduction

The Cystic Fibrosis Foundations website aims to provide accurate and comprehensive health

information for those battling with cystic fibrosis, or to family members and friends of those with

cystic fibrosis. The goal of the foundation is to facilitate the dissemination of information to those

who need it: patients, family members and friends, and donors to the foundation.

This usability test was conducted in order to determine the extent that the interface facilitates a

users ability to complete routine tasks based on various information needs.

Executive Summary

We conducted this usability test using a live version ofhttp://www.cff.org using Windows Internet

Explorer 6.0 located on a Windows 7 PC in the IT lab of the University of Texas School of

Information. The test administrator and data logger were present in the testing room. Each

session was audio recorded. We captured each participants navigational choices, task

completion rates, comments, overall satisfaction ratings, questions and feedback.

The users were recruited largely from a convenience sample, as six out of seven of the users

were fellow students at the School of Information. Our participants had a variety of experience

looking up health information on the web. 5 out of 6 participants (83%) used the web to search

for health information at least once a month.

Seven tests were conducted, including one pilot test and six regular tests. Each individual

session lasted approximately 20 minutes and included an introduction and informed consent,

background questionnaire, five tasks to be attempted, and an exit survey.

According to user comments, the majority of participants found the web site to be clear andstraightforward. 83% thought the website was easy to use, and the same majority found that

they were able to do everything on this health site that they had been able to do on other health

sites.

The test identified only a few minor problems including:

Text-heavy interface

Confusion surrounding the difference between a cystic fibrosis care center and a chapter

location of the Cystic Fibrosis Foundation

Inconsistent interface design for Great Strides information

Confusion surrounding sorting of chapters in the drop-down menu for finding a GreatStrides fundraising walk: listed alphabetically by chapter name instead of by geographic

location

This document contains the participant feedback, satisfactions ratings, task completion rates,

time on task, errors, and recommendations for improvements. A copy of the scenarios and

questionnaires are included in the Attachments section.
http://www.google.com/url?q=http%3A%2F%2Fwww.cff.org&sa=D&sntz=1&usg=AFQjCNEvFC8ht-mXNWceFr27vE3iGEnVVg


4/23

Methodology

This usability study gathered assessment data about the effectiveness of www.cff.com. In this

study, six participant worked through five tasks. Each session was approximately 30 minutes in

length. Five minutes of each session was used to explain the session agenda to the participant,

outline the responsibilities of being a participant, and review consent form information with the

participant. During the remaining 15 minutes of the session, participants completed abackground survey, the five tasks, and an exit survey covering their overall feelings and thoughts

toward the www.cff.org.

Test Environment

This usability study was conducted at the University of Texas in the School of Information

computer lab. More specifically, the classroom at the back of the computer lab was used

because it was a quiet space free of outside distractions. This controlled environment was used

in an attempt to control and eliminate environmental factors that might influence or impact user

behavior. The test portion of the usability study was conducted on a Windows 7 PC using

Internet Explorer 6.0 connected to the university's high speed internet. The browser was resetafter each session in order to eliminate the previous users browser data. Additionally, the

browser was set up with the necessary test materials before the next session began. All test

participants used the same desktop setup in order to maintain control over the digital test

environment.

During the tests, the moderator was positioned to the left of the participant while the notetaker

was stationed to the participants right. The distance between moderator and participant was

close enough so the moderator to see the screen but far enough away to give the participant

their own space. The same is true for the note-takers position.

While the sessions were not digitally video-recorded, they were audio-recorded. A digital voice

recorder in the form of an Apple iPhone was used to tape the responses and reactions the

participants had while using the www.cff.org.

Sessions

The participant sample used in this study was a convenience sample. The test moderator

recruited participants from the INF 385P Usability class. Each participant was recruited in

person to take part in the usability study. In total, there were seven participants (five females and

two males) who ranged in age (27-42). One participant was used for a pilot test, and therefore

that participants test results were not included in the final test samples results.Each individual session lasted approximately twenty minutes. During the session, the test

moderator explained the test session and asked the participant to fill out a brief background

questionnaire. Participants read the task scenarios aloud and tried to find the information on the

website.

After the last task was completed, the test administrator asked the participant to complete and
http://www.google.com/url?q=http%3A%2F%2Fwww.cff.org&sa=D&sntz=1&usg=AFQjCNEvFC8ht-mXNWceFr27vE3iGEnVVghttp://www.google.com/url?q=http%3A%2F%2Fwww.cff.org%2F&sa=D&sntz=1&usg=AFQjCNEO0ruQwbsnqXnliniut7qN7MOAWQhttp://www.google.com/url?q=http%3A%2F%2Fwww.cff.org&sa=D&sntz=1&usg=AFQjCNEvFC8ht-mXNWceFr27vE3iGEnVVg


5/23

exit survey which consisted of rating the website overall by using a 4-point Likert scale for eleven

subjective measures. The participant then had to textually input their personal thoughts and

feelings about the website in a personal response section of the survey. These questions were

structured as overall website questions:

What the participant liked most What the participant liked least

Recommendations for improvement

Finally, the participant had to select the best answer on a 5-point Likert scale for how they

generally felt about using cff.org and searching for the specified health information.

Participants

All participants, with the exception of one, were students in the INF 385P Usability class during

the fall semester. Seven participants were scheduled over two testing dates. Seven out of seven

participants completed the test. One participant was involved in testing on November 9th and sixon November 12th.

Friday, November 9 Monday, November 12

Meet, setup Session 2

Session 1 (Pilot session) Session 3

Debriefing Session 4

Revise session script if needed Session 5

Revise tasks Session 6

Session 7

Debriefing

Of the seven participants, two were male and five were female. The ages of the participants

ranged from 27-42 years of age.

Technology Use

Participants outlined their technology use on the background questionnaire. All participants

included in the study own a mobile phone as well as a laptop computer. In addition to that, all

participants selected Several times a day for participating in the following behavior:

Sending and receiving emails

Reading news items on web sites


6/23

Accessing social networking sites

Four participants (67%) say they send or receive text messages on their cell phone Several

times a day and 2 participants (33%) say they send or receive text messages on their cell

phone Once daily.

Overall, our sample participants use digital web technologies daily and can therefore be

categorized as technologically proficient in web browsing and web information seeking.

Evaluation Tasks/Scenarios

Test participants attempted completion of the following five tasks:

1. Find out more about the Foundation and its general purpose.

2. Find information on cystic fibrosis therapies.

3. Find out if there is a cystic fibrosis care center in Texas.

4. Find a Cystic Fibrosis Foundation chapter in Austin, Texas.5. Find out when the next Great Strides fundraiser will be in Austin, Texas.

Results

Task Completion Success Rate

All participants successfully completed Task 2 (find information on cystic fibrosis therapies) and

Task 4 (find a Cystic Fibrosis Foundation chapter in Austin, Texas). Five of the six (83%)

participants completed Task 1 (find out more about the Foundation and its general purpose) and

Task 3 (find out if there is a cystic fibrosis care center in Texas). Four of the six (67%) of

participants were able to complete Task 5 (find out when the next Great Strides fundraiser will be

in Austin, Texas).


7/23

Time on Task

We collected the time on task for each participant from the sessions digital recordings. Some

tasks were inherently more difficult to complete than others and is reflected by the median time

on task.

While the median completion times for Tasks 1, 2, 3, and 4 were forty-five seconds or below, themedian completion time for Task 5, which required participants to find out when the next Great

Strides fundraiser will be in Austin, Texas, was 101 seconds. Completion times ranged from 5

seconds to 103 seconds (approximately two and a half minutes) with most times fewer than 60

seconds.


8/23

Errors

The note taker captured the number of errors participants made while trying to complete the task

scenarios.

Errors were categorized as either non-criticalor critical. Noncritical errors involve confusion,

following incorrect links, and or/user frustration. Critical errors prevent the completion of a task or

result in incorrect data.

Non-critical Errors

1. Backtracking

2. Initially selected incorrect route

3. Found correct info but did not accept it

4. Found info through an alternative path

5. Found info but continued to search


9/23

Critical Errors

1. Could not complete task

2. Had to ask for help

3. Believed task was complete when it was not

Exit Survey

All of the participants found that the website was very professional (on a scale of 1 -

Professional to 4 - Unprofessional) for both its user interface and content. Six participants

(100%) of users rated the website as safe to use. Safe in this instance was an extension of

trustworthy and could be applied to trustworthiness of information provided and transparency of

the site creators. Most of the participants (67%) rated the website as being very high quality while

the remaining participants (33%) rated the website as being high quality. Although participantsencountered some difficulty and confusion with certain tasks, the participants average rating for

how clear the website was 1.5. This is particularly interesting since 1 stood for Clear and 4

stood for Confusing. Half of the participants (50%) rated the site as a 1 and half (50%) rated the

site as a 2. No participants rated the site a 3 or a 4, which indicates that the site was not

confusing.

Half of the participants (50%) agreed that the website was Easy to use. Two participants (33%)


10/23

rated the site a 2 for ease of use which indicates that while the site was for the most part easy to

use, there was some confusion on certain tasks. One participant (17%) rated the site a 3 for

ease of use and no participant rated the site a 4. 83% of participants found the site Easy to

Moderately easy to use, while 17% found the site to be Slightly complex to use.

Three participants (50%) rated the website a 1 for likability. Three participants (50%) also rated

the website a 2 for likability. Likability in this instance refers to overall feelings toward thewebsite. Therefore the participants expressed strong to moderate feelings of liking the website

while no participants expressed strong to moderate feelings of disliking the website as a whole.

Overall Ratings

After a task session completion, participants rated the site for three overall measures. These

measures include:

1. Relevancy and usefulness of site content

2. The site facilitates health information gathering

3. Ease of use

The following bar graphs show the three overall rating questions and how the participants

responded.

I thought the information I got from the web site was useful.

I can do everything I would expect to be able to do on a health information web site.


11/23

Overall, the site was easy to use.

As is evident from the graphs, almost all of the participants either Agreed or Strongly agreed

with the metric statements. Therefore, it is reasonable to say that the participants were

consistently satisfied with the functionality of the site. The one participant who rated the second

statement as Neither disagree or agree related to the moderator that he did not know what a

health information site should include and could not decide whether to agree or disagree.

Likes, Dislikes, Participant Recommendations

Upon completion of the tasks, participants provided survey feedback for what they liked most

and least about the website in the exit survey as well as recommendations for improving the

website.

Liked Most

The following comments capture what the participants liked most:

What are two things you liked most about the site?


12/23

Participant 1"Self explanatory terms for navigation tabs - clear

Clear and simple content"

Participant 2

"I liked the way the menus popped up when you roll over the top navigation. I

feel like that would save me a lot of time!

I thought the navigation labels were pretty clear."

Participant 3"The site did not have an overwhelming amount of menu options. Items wereeasy to find even when I got off track a bit. The site did not take long tonavigate. The use of colors and images was not distracting. "

Participant 4"The labels were very clear and easy to understand

Navigation was simple and brought me to where I expected to go"

Participant 5 "decent layout, pretty simple navigation"

Participant 6"It featured information for the first-time visitor prominently and presented it

clearly.

Accomplishing the chosen tasks seemed fairly straightforward."

Liked Least

The following comments capture what the participants liked the least:

What are two things you liked least about the site?

Participant 1"The Great Stride Section (locating a city)Locating a chapter vs. locating a branch office"

Participant 2

"There's a lot of information, and therefore a lot of choices. It seemed like a loof navigation options were duplicated in the left hand menu.The walk website looked substantially different from the rest of the CFwebsite. That was a little jarring."

Participant 3"I don't think I looked at the site long enough to find something that I didn't cafor. "

Participant 4"It was more difficult to find specific information about the walk without actuallbeing a participantSome of the link locations within text fields were not as easy to access"

Participant 5"Fnding local resources was not intuitive as they were listed by titlealphabetically rather than location"

Participant 6

"The links to finding chapters and care centers are relegated to the bottom ofthe left sidebar in gray text. These could be not visible without scrolling on asmaller screen.Maybe something more dynamic -- larger featured images of kids-- that reallyencourages me to want to help out.


13/23

On the "Find a fundraiser by chapter" drop down, the chapters are ordered balpha rather than alpha by state then alpha by chapter."

Recommendations for Improvement

The following comments capture what the participants liked the least:

What would you like to add to the site?

Participant 1 "A well organized locate a walk menu "

Participant 2

"It would be cool to have a feature that detects your location and automaticallprovided you with a list of local chapters, care centers and events. Or even asingle zip code question that could bring up a list of all of your localresources."

Participant 3

"From the local chapter site, I went to chapter events to find out about GreatStrides even though there was a huge link to it in the left nav bar. The resultsfrom my search for Great Strides did not show anything. Perhaps if the GreatStrides were also listed in the search results, that might be helpful to someusers. "

Participant 4 "Easier back navigation"

Participant 5 "Nothing"

Participant 6

"I didn't get a chance to check during the test, but featured stories about kidswith Cystic Fibrosis? (Okay, there's one. Maybe more/larger? Maybe a slideshow of clickable videos?) Featured images of successful, enjoyable

fundraising events.More prominent display of Find a . . . links."

Recommendations

This section provides recommended changes and justifications driven by the participant

success rate, behaviors, and comments. Each recommendation includes a severity rating. The

following recommendations will improve the overall ease of use and address the areas where

participants experienced problems or found the interface/information architecture unclear.

To reiterate:

All participants successfully completed Task 2 (find information on cystic fibrosis

therapies) and Task 4 (find a Cystic Fibrosis Foundation chapter in Austin, Texas). Five

of the six (83%) participants completed Task 1 (find out more about the Foundation and

its general purpose) and Task 3 (find out if there is a cystic fibrosis care center in Texas).

Four of the six (67%) of participants were able to complete Task 5 (find out when the next

Great Strides fundraiser will be in Austin, Texas).


14/23

Even though some errors occurred when participants tried completing Task 1 and no errors

occurred for Task 4, the recommendations we focus on come from Task 3, 4, and 5.

Task 3 Recommendations, Justification, and Severity Rating

Task 3 required participants to find out if there is a cystic fibrosis care center in Texas.

Change Justification Severity

Add a working definition for

care center.

Include a prominent search

feature to search locally for a

care center.

Three participants were

confused between the

function of a care

center and the function

of a chapter. The site

would benefit from

defining the function of

both and/or usingdifferent terminology.

Additionally, being able

to search for a care

center locally is

essential. Currently,

care center information

is categorized

regionally and not

categorized locally.

Participant commentsincluded categorizing

care center information

in a more concise

manner so it is easier

to find.

High


Task 4 required participants to find a Cystic Fibrosis Foundation chapter in Austin, Texas.


Add a working definition for

chapter.

More prominently display the

Find a chapter search tool.

Three participants were

confused between the

function of a care

center and the function

of a chapter. The site

High


15/23

would benefit from

defining the function of

both and/or using

different terminology.

Additionally, being able

to search for a local

chapter is essential.

Currently, chapter

information

is categorized

regionally and not

categorized locally.

Participant comments

included categorizing

chapter information in a

more concise manner

so it is easier to find.

Participants also

suggested placing a

Find a chapter search

option in a more

prominent spot on the

homepage.


Task 5 required participants to find out when the next Great Strides fundraiser will be in Austin,

Texas.


Add Great Strides Locate a

Walk in Quick Links or

main navigation menu.

Start page at the top.

Add search feature that

retrieves a list of local

chapters, care centers, and

events.

Participants suggested

placing a Great Strides

- Locate a walk link in

the Quick Links

menu. This would

facilitate faster

information retrieval

without having to go

through additional steps

to satisfy the

information need.

Additionally, when the

High


16/23

Great Strides page

opens, it needs to load

from the top of the

page, not the middle.

The middle page load is

confusing to users.

One participant

suggested that there

should be a general

local search that would

retrieve local chapters,

care centers, and

events all at once. This

has the potential to

satisfy multiple

information needs

simultaneously.

Conclusion

Most of the participants found CFF.org to be well-organized, comprehensive, clean and

uncluttered, very useful, and easy to use. Having a centralized site to find information is key to

many if not all of the participants. Implementing the recommendations and continuing to work

with users (i.e., real lay persons) will ensure a continued user-centered website.


17/23

Document A: Background Questionnaire


18/23


19/23

Document B: Script

Session Introduction

Thank you for agreeing to take part in our research study. My name is [Caron/Emily] and this is

my partner [Caron/Emily]. Today we are going to be running through a couple tasks using the

Cystic Fibrosis Foundations website.

Therefore, were here to learn about how users navigate the Cystic Fibrosis Foundation's

website.

During the session, I will ask you to use the web site to do a variety of things and will observe

you while you do them. We want our test to be as real as possible so as you do these things,

please try to do whatever you would normally do when seeking information on a website.

Please try to think out loud while youre working. Just tell me whatever is going through your

mind. Please know that were not testing you, and there is no such thing as a wrong answer. Youwill help us understand what works or doesnt work on the site by doing this.

Also, please be completely honest when expressing your thoughts. Im just here to moderate the

study, so nothing you say about this website will upset me or hurt my feelings. If something

about using the site is hard or doesnt make sense, be sure to let me know. Also, if something

works great or surprises you in a good way, be sure to tell me about that too. We want all of your

feedback.

Now, before we get started, I have a standard disclaimer that I say to everyone:

This study is completely voluntary. By listening to your thoughts and observing your actions, wehope to make using the Cystic Fibrosis Foundations website easier and more enjoyable.

Nothing you say or do will be used to evaluate you. We are only evaluating the Cystic Fibrosis

Foundations website. Also, no personally identifiable information gathered during this study will

be shared with anyone outside the project team. Once again, this study is completely voluntary.

You can stop the test at any time for any reason.

Does that sound ok with you?

[If yes] Great. Please read through and sign the consent form. The second consent form is for

recording purposes. Please understand that the recording will be confidential and will bedestroyed after the usability testing is complete.

The whole session will take about 20 minutes. Do you have any questions before we begin?

[Answer any questions.]


20/23

Task Matrix

Great. Lets get started.

First, please fill out our preliminary survey before starting the task portion of our test.

(pull up opening questionnaire)

Task

Now lets begin the task portion of the test. For each task, read it aloud and then complete the

task to the best of your ability. Do not move on to another task until you have completed the

present task. If, for whatever reason, you are unable to complete a task, just let us know and

you can move on to the next task.

Any questions?

(hand them task sheets)

Lets begin! Please read the first task.

(for each scenario, periodically ask"Please tell me what you see")

Close the Session

Having used the site now, we would like you to complete a closing questionnaire.

(pull up closing questionnaire)

(once participant is done with questionnaire)

Thank you for participating in our user study! Your input is valuable to us.


21/23

Document C: Tasks

Tasks

You are unsure what the Cystic Fibrosis Foundation's purpose is. Find out more about the

Foundation and its general purpose.

Now that you know a little bit about the foundation, you wonder what information the foundation

has available on cystic fibrosis therapies. Find information on cystic fibrosis therapies.

The Cystic Fibrosis Foundation provides funding for and accredits more than 110 cystic fibrosiscare centers. You are curious if there is a center located in Texas. Find out if there is a cystic

fibrosis care center in Texas.

The Cystic Fibrosis Foundation has more than 75 chapters and branch offices across the

country. You now want to locate a Cystic Fibrosis Foundation chapter locally. Find a Cystic

Fibrosis Foundation chapter in Austin, Texas.

You want to support the Cystic Fibrosis Foundation by participating in a Great Strides fundraising

walk. Find out when the next Great Strides fundraiser will be in Austin, Texas.


22/23

Document D: Exit Survey


23/23

CFF Final Report

Documents

Transcript of CFF Final Report