Developing Web Design and Usability Guidelines:

An Evidence-Based Approach

November 4, 2002 Craig Lafond

Mary Frances Theofanos

Communication Technologies BranchOffice of Communications • National Cancer Institute

Presentation Objectives

• Discuss why evidence-based web design is important

• Describe the process used to create the guidelines

• Highlight the expert review process used to create stronger guidelines

• We should base website design decisions on facts

• Attempt to:

- Move away from ‘country doctor’ model

- Move toward

– University-based Physician or

– Teaching Hospital Model

Evidence-Based Solutions

• Make the best web design decisions by putting the latest research (evidence) into practice early

• This leads to ensuring that sites always start at the highest level

• This prevents

– Exposure of users to unusable sites

– Wasting of resources

– Application of ineffective design ideas or design processes

Evidence-Based Web Design

Steps in the Process

1. Identify and translate research into guidelines

2. Create ‘Relative Importance’ ratings

3. Generate ‘Strength of Evidence’ ratings

• Identify existing guidelines– Yale Web Style Manual

– Ameritech Web Page User Interface Standards and Design Guidelines

– Sun Microsystems: Guide to Web Style

– IBM’s Web Design Guidelines

– The Library of Congress World Wide Web Style Guide

• Problems with style guides– Contained conflicting guidance

• Yale: “Use longer pages to ease page maintenance”• Sun: “Use shorter pages to make site more maintainable”

– Guidance too general

– No references

Step 1: Identify and Translate

Research into Guidelines

• May not be specific

– Audiences

– Contexts/Situations

• May not be explicit

• May not consider usability/design experience

• Evidence from other disciplines may not be reflected

Guideline Limitations

• Purpose

– Determine which guidelines practitioners think are most important, Create ‘Relative Importance’ Ratings

– Reduce and strengthen the list

• 16 Reviewers

– 8 web designers

– 8 usability specialists

• December 11-31, 2001

Practitioner Review

• No reliable difference between usability specialists and web designers in how they rated the guidelines

• ‘Relative Importance’ and ‘Strength of Research’ ratings are correlated

Step 2 - Phase 1 Results

• Allowed reviewers to confirm or change their ratings with knowledge of

– Their previous ratings

– The group mean for ‘Relative Importance’

• Attempted to arrive at consensus

Step 2 - Phase 2

• 287 guidelines

• Same 16 reviewers

• Allowed reviewers to confirm or change their ratings with knowledge of

– Their previous ratings

– The group mean for ‘Relative Importance’

• Attempted to arrive at consensus

Step 2 - Phase 2

New for Phase 2

Reviewers ratings from

Phase I

• Again no reliable difference between ratings of web designers and usability specialists

• No reliable difference between ‘Relative Importance’ and ‘Strength of Research’ ratings from Phase 1 to Phase 2

• ‘Relative Importance’ and ‘Strength of Research’ are still correlated in Phase 2 (r=.81 in both phases)

Step 2 - Phase 2 Results

1. Exclusively on experience

2. Mostly on experience, though somewhat familiar with the research

3. Mostly on experience, though very familiar with the research

4. Half on experience, half on knowledge of the research

5. Mostly on knowledge of the research, with strong confirmation from experience

6. Mostly on knowledge of the research, with weak confirmation from experience

7. Exclusively on knowledge of the research

Basis for ‘Strength of Research’ Rating

Purposes:

• Learn whether researchers will rate the guidelines differently than practitioners

• Assess the level of agreement on an evidence-based rating

• Collect additional references

• Identify where there is no evidence to support common web practices

• Establish an ongoing process for future reviews

Researcher Review

• Nine usability experts• All with

– Strong educational background

– Expert understanding of the usability literature

– Good understanding of experimental design issues

Step 3 Participants

Step 1

• Each reviewer put each of the 225 guidelines into one of 3 categories

– strong research support

– weak research support

– no research support

• If any one of the reviewers indicated that the guideline had any research support, it was kept in the process

• Reviewers also added new references

Researcher Review

Step 2

• There was little agreement on guidelines with ‘no research support’

• Researchers met on August 8 to establish a common framework for rating the guidelines

• Decided to classify references into 9 types (hypothesis-based, expert opinion, survey data, literature review, etc.)

Researcher Review

Step 3

Researchers rated each of the guidelines on a common scale:

Strong Research Support • Cumulative and compelling, supporting research-based evidence• At least one formal, rigorous study with contextual validity• No known conflicting research-based findings• Expert opinion agrees with the research  Moderate Research Support• Cumulative research-based evidence• There may or may not be conflicting research-based findings• Expert opinion

– Tends to agree with the research, and– Consensus seems to be building

Researcher Review

Researcher Review

Weak Research Support • Limited research-based evidence• Conflicting research-based findings may exist

and/or • There is mixed agreement of expert opinions

Strong Expert Opinion Support• No research-based evidence• experts tend to agree, although there may not be a consensus• Multiple supporting expert opinions (presented in textbooks, style

guides, newsletters, etc.) • Generally accepted as a 'best practice' (or reflects current 'state of

practice')

Weak Expert Opinion Support• No research-based evidence • Limited or conflicting expert opinion

Results to Date

• References by type– Observational study – 17– Experiment – 190– Model-based – 15– Expert opinion – 57– Literature review – 78– Survey – 7– Textbook – 27– Usability test – 24– Exploratory study - 20

Results to Date

• Research helps us better understand what works

• We need

- More research

- Better research

- Greater use of research findings

• Final goal: To help practitioners make design decisions that achieve higher-quality, professional user interfaces!

Conclusions