presentation-designing and user testing new FIT instructions

| © 2013 Regional Health Education, The Permanente Medical Group, Inc. December 30, 2016

Finding the Perfect FIT: Designing and User Testing New FIT Kit Instructions

Jessica Alper, Graduate Student Intern August 14, 2013

2 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only. December 30, 2016

Agenda

FIT Kit User Error

Cognitive Load Theory in Health Education

User Testing

Outcomes

Conclusion

2012 FIT Kit Return Rate N=467,003


32.1% 67.9%

Not submitted to KPSubmitted to KP

6.4%

93.6%

Unprocessed due to user errorProcessed

2012 FIT Kit User Error N=317,095

Presenter

Presentation Notes

This project was developed to respond to the problem of FIT kit user error. About 67.9% of the FIT kits that we sent to members in 2012 were completed and mailed back. Of that 67.9% , 6.4% could not be processed due to user error. That comes out to about 20,294 kits. Although this is a small number relative to our region-wide membership, it’s significant because tests unprocessed due to user error cannot count toward our HEDIS score for colorectal cancer screening. In 2012, our colorectal cancer screening compliance rate for the region was 81.5%; the HEDIS goal was 82%. With margins this close, every screening counts.


Most Common User Errors (N=20,294)

49.3%

23.5%

16.5%

7.0% 1.9%

1.1% 0.7%

No collection dateNot labeledToo old to processUnable to ID memberNo specimen includedSpecimen leakedExpired collection tube

Presenter

Presentation Notes

No collection date (49.3%) Not labeled (23.5%) Too old to process (16.5%) Unable to ID member (7.0%) Nearly all of these user errors are issues we hope to be able to correct through revising the instructions so that they are even easier to understand and follow correctly.

When To User Test?

Presenter

Presentation Notes

When scoping out this project, I knew that user testing would be in important phase. However, user testing isn’t the first phase—the value of user testing is the opportunity to learn what you can’t glean from RHE’s expertise in health education and health communication. Therefore, user testing should be conducted AFTER a rigorous analysis and design process.


Cognitive Load Theory in Health Education

Cognitive Load Theory (Sweller, 1988) The Magic Number Seven, Plus or Minus Two: Some Limits

on Our Capacity for Processing Information (Miller,1956) “Chunking” optimizes memory’s limited capacity

Designing Communications That Support Cognitive Learning Processes

Presenter

Presentation Notes

The analysis and design phase is informed by the idea that the way information is presented greatly influences how easy or difficult that information will be to process and comprehend. We can optimize the effectiveness of health education materials by applying design principles and strategies grounded in psychological and educational research. Cognitive load theory is central to this approach. Cognitive load theory posits that people learn best when information is presented in ways that minimize the amount of mental activity needed to process information. One theory I applied was psychologist George Miller’s Magic Number Seven concept. Miller found that people can only hold 7, plus or minus 2, pieces of information in their short term memory at a time. So at any given moment, people can only remember 5-9 pieces of information. To optimize the capacity of short term memory, we can “chunk” information so as not to exceed 7 items. We do this naturally without realizing that we’re acting on empirically supported psychological research. Think of the way you remember a phone number—not as 7 individual digits, but as one set of 3 and one set of 4 numbers. The analysis and design phase is informed by the idea that the way information is presented greatly influences how easy or difficult that information will be to process. We can optimize the effectiveness of health education materials by applying design principles and strategies grounded in educational psychology research. Cognitive Load Theory is central to this approach. Cognitive load refers to the total amount of mental activity imposed on working memory at an instance in time. Cognitive Load Theory posits that people learn best when information is presented in a way that maximizes human cognitive processes and minimizes the amount of mental activity needed to process information. In his most famous paper, published in the Psychological Review in 1956, psychologist George Miller documented his research indicating that people can only hold 7, plus or minus two, pieces of information in their short term memory at a time. To optimize the capacity of short-term memory, people can “chunk” information into pieces—much the way we memorize phone numbers not as seven individual numbers, but as one set of three and one set of four numbers. Given that most people find it challenging to hold more than 5-9 pieces of information in their mind at a time, instructions that exceed that number of steps can be overwhelming, intimidating, and difficult to process. The current version of FIT directions has 10 steps. So, we applied design principles to “chunk” the directions in order to make them easier to process.


Presenter

Presentation Notes

I’d like to illustrate the power of using design principles that support cognitive processes with an activity based on an experiment conducted by Tullis in 1997. Looking at this screen, I’d like you to find the price of a double room at the Quality Inn in Columbia. When you’ve finished, please raise your hand.


Presenter

Presentation Notes

Now I’d like you to find the price of a double room at the Holiday Inn in Bradley. Again, please raise your hand when you’ve finished. In an early study, Tullis found that although both screens have the same density of information, with 31% of the page taken up by text, users took an average of 3.2 seconds to search the first screen and an average of 5.5 seconds to search the second screen for the same kind of information.


Presenter

Presentation Notes

In 2004, Wells conducted a similar study in which participants took more time to find the word Italian on this page--


Presenter

Presentation Notes

--then on this page. Wells found that people processed information more quickly when information was presented with borders and white space, than when information was presented using only color contrast. This suggests that borders can be used as an effective grouping method to help readers navigate and process information on the page. We applied this principle by “chunking” the directions using borders and white space.


Presenter

Presentation Notes

We applied these principles by rewriting the instructions from 10 steps into 3 and “chunking” them using borders and white space.


Presenter

Presentation Notes

We applied these principles by “chunking” them using borders and white space.


Presenter

Presentation Notes

Looking at this version, you may also notice that it is very text heavy. It features 12 pictures, but with the text taking up more space and located at the center of the page, the primary focus is on the text.


The Power of Pictures

— The Joint Commission (2010). Advancing Effective Communication, Cultural Competence, and Patient-Centered Care

“Using pictures, with or without written explanations, can greatly increase patient understanding and retention of information, particularly among patients with low functional health literacy skills.”

Presenter

Presentation Notes

As the Joint Commission, the CDC, and several other reputable sources have found, using pictures is a powerful way to increase comprehension and understanding, especially among people with low functional health literacy skills.


Presenter

Presentation Notes

The revised version relies much less on text to instruct the reader on how to complete each step and move through the FIT kit process. While the first version includes only 12 pictures, the new version features 19. The pictures take up more space on the page than words and occupy the center of the pages, making the pictures the focal point so that the text supports the images, rather than images supporting the text.

Not All Pictures Are Worth a Thousand Words Original Version Revised Version


Presenter

Presentation Notes

Simply adding more pictures is not enough to reap the benefits associated with the implementation of visuals. To add value to an instructional document, the pictures must be meaningful—they must be easy to recognize and should effectively illustrate directives. Creating pictures to visualize an unpleasant task is challenging, but shying away from the ick factor to avoid offending readers may result in pictures that fail to clarify and that may even confuse our members.


Presenter

Presentation Notes

In addition to revising existing images and adding new ones, the layout of those images as well as text on the page was carefully considered to make the instructions simple and easy to follow. In the original version, the list of “what’s in this kit” was divided into two sections: “what you need to collect a sample” and “what you need to mail your sample back to us.” That’s confusing in terms of sequencing—it simultaneously invokes the beginning of the FIT kit process as well as the final step, mailing the kit back.


Presenter

Presentation Notes

In the first revision that I received from our Content Management Team, at the top, that section has been simplified to read only, “What’s in this Kit.” However, this layout which lists a column of words on the left and a cluster of pictures on the right prompts the reader to scan back and forth to identify each item. And while after spending hours with the FIT kit you become very familiar with each of these items, it won’t necessarily be easy for a member completing his first FIT kit to recognize which item is which. The revised version, pictured here at the bottom, reads more intuitively left to right, which requires less mental activity to process. It’s a subtle change, but these kinds of details add up to the creation of a document that’s easier to read and understand.


User Testing

User (usability) testing—evaluates a product by testing it with users

“Watching users try to complete tasks with your interface is the

most effective and productive way to uncover usability problems.”

–Jakob Nielsen, Nielsen Norman Group

Presenter

Presentation Notes

Once you’ve applied design principles to develop a strong prototype, you’re ready to user test. Essentially, user testing involves recruiting users and observing them as they interact with your product to evaluate how well the product does what you want it to do. This allows you to gauge where members will succeed, where they will have difficulty, and how quickly and easily they will be able to use your product correctly.


Why Conduct User Testing?

— Mark Twain

“We tend to project our own rationalizations and beliefs onto the actions and beliefs of others.”

-Don Norman, The Design of Everyday Things “Supposing is good. Knowing is better.”

-Mark Twain

Presenter

Presentation Notes

User testing is an incredibly valuable technique to evaluate all member facing products, whether those are print materials, mobile applications, or programs. Until we test our products with the people they’re intended to serve, we simply can’t know how effective they will be.


User Testing 101

Recruit representative participants Conduct user testing:

– Facilitate representative tasks “Think out loud”

– Observe Rainbow spreadsheet

Evaluate – Standardized usability survey

Presenter

Presentation Notes

If you’ve never conducted user testing, it may sound complex or challenging. However, user testing is really quite simple—the basic protocol is to recruit representative participants, facilitate representative tasks, observe while encouraging participants to think out loud, and evaluate the results. I found it helpful to couple user testing with a standardized usability survey to gain additional data, and that will be included in the appendix to this presentation. I also video taped my user testing sessions and streamed them to webinar, which is an observation technique that enables you to go back and look for findings you may have missed in the moment.

Why You Only Need to Test With Five Users

Presenter

Presentation Notes

The best return on investment comes from small tests with only 5 users. Testing costs increase with each additional study participant, yet the number of findings quickly reaches the point of diminishing returns. There's little additional benefit to running more than 5 people through the same study; ROI drops like a stone with a bigger N . With 5 users, you get very close to the maximum cost-benefit ratio. In most cases, you’ll yield larger benefits running additional studies than testing more subjects.


User Testing Techniques

Probe with open-ended questions User speculation unreliable: reinforce data, not opinion

– Video clip

Think Aloud Protocol – Concurrent – Retrospective

Presenter

Presentation Notes

The most valuable information you can glean from user tests comes from what participants do, not what they say they do. Participants’ thoughts about what they will do in the future, or about what others will do, is not reliable. Probe for data, not opinions—reinforce thinking out loud, not simply talking out loud. Use open ended questions such as, “What about that was easy or difficult?” “How well did you understand step two?” This can be done during the user test—as users are completing representative tasks—as well as after, when they’re reflecting on what they just did. In this video clip, I’ve watched the user go through the directions and incorrectly perform the step that says to fill out the removable label and stick it on the sample bottle. When I ask him about his error, he offers his opinions about how to make the bottle stand out more. I needed to probe to understand WHY he did not follow the step correctly, not his opinion about how to make the label easier to see.

User Test Demographics (N=7)

1

4

2

Age

42-54 yrs55-64 yrs65-74 yrs

2

3

1

1 Race

Hispanic/Latino

White

Asian

Black/AfricanAmerican

4

3

Sex

MaleFemale

Presenter

Presentation Notes

In total, I conducted user testing on 7 members, whom I recruited through Member Voice Panel. Since colorectal cancer screening is recommended for users between the ages of 50 and 74 years old, I recruited members from that age group. And since I was aiming to reduce user error and my literature review suggested that the highest amount of user error occurs among Hispanic/Latinos, I sought to recruit high representation from Latino members.

25 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

Of 5 users in test one: 5 rated instructions as “very easy to understand.” 5 began FIT by laying out all items and comparing each with its

accompanying picture, without any prompting. 4 hesitated or expressed confusion regarding which label they

should use. 3 expressed discomfort or reservation about writing their name in

the return address portion of the envelope. 2 wrote PHI directly onto bottle label instead of peel off label.

December 30, 2016

Results Summary User Testing, Round One (N=5)

Presenter

Presentation Notes

My first user test consisted of 5 members. Of those 5, all rated the instructions as very easy to understand in the usability survey and also commented on how easy the instructions were to follow. They also all began the test by laying out each item and comparing it with its accompanying picture, which validated the design choice to present that information before the first step and also validated the new pictures developed by our Content Management Team. However, 4 of the 5 users hesitated or expressed confusion about whether to use the label provided in the instructions or to write on the label that comes on the sample bottle. And 2 of the 5 users actually performed that step wrong, which indicated the need for more revision to avoid that error.


Presenter

Presentation Notes

Mariana Almeida, Ali Goldstein and I decided to make the label yellow so that we could refer to it as “the yellow label,” and we added instructions to “peel off this label” and “use the yellow label to cover the blank label on the sample bottle.”


Key Learnings

Apply design principles that improve comprehension and recall – Chunking: borders and white space – Pictures – Format and sequencing

User testing is simple, robust, and effective

Presenter

Presentation Notes

By applying design principles and strategies grounded in educational psychology, we can improve members’ comprehension and recall, especially among people with low health literacy skills. I think we use some of these techniques, such as chunking with borders and white space, because we intuitively think doing so makes our materials look better. By being more aware and intentional about presenting information so that it’s easy to navigate and to process, we can improve the quality of our health education materials. Additionally, as I noted earlier, user testing is an incredibly effective way to evaluate ourmember-facing products. It’s a simple process that yields valuable results, enabling us to enhance materials to better serve our members.

presentation-designing and user testing new FIT instructions

Documents

Transcript of presentation-designing and user testing new FIT instructions