Ch 13. Asking Users & Experts

Team 3: Jessica Herron

Lauren Sullivan

Chris Moore

Steven Pautz

Asking Users and Experts: Introduction

• We want to find out what users do, want to do, like, or don’t like

• Several tools & techniques are available• Interviews

• Questionnaires

• Heuristic Evaluation

• Cognitive Walkthroughs

• Strengths/Weaknesses of these tools

Interviews

Unstructured are not directed by a script. Rich but not replicable.

Structured are tightly scripted, often like a questionnaire. Replicable but may lack richness.

Semi-structured guided by a script but interesting issues can be explored in more depth. Can provide a good balance between richness and replicability.

Interview Basics• Remember the DECIDE framework

– Determine Goals, Explore Questions, Choose eval paradigm, Identify issues, Decide on ethical issues, and Evaluate data.

• Goals and questions guide all interviews• Two Types of Questions:

Closed Questions Closed questions have a predetermined answer format, e.g., ‘yes’ or ‘no’. Closed questions are quicker and easier to analyze.

Open Questions Open questions do not have a predetermined format, they are trying to determine the feeling of the interviewee.

Avoid Interview Pitfalls

Long questions Compound sentences - split into two Jargon & language that the interviewee may not

understand Leading questions that make assumptions e.g.,

why do you like …? They might not actually like the product.

Be aware of unconscious biases e.g., gender stereotypes

Interview ComponentsIntroduction introduce yourself, explain the goals of the

interview, reassure about the ethical issues, ask to record, present an informed consent form.

Warm-Up make first questions easy & non-threatening.

Main Body present questions in a logical order

Cool Off include a few easy questions to defuse tension at the end

Closure thank interviewee, signal the end, e.g, switch recorder off.

The Interview Process

• Dress in a similar way to participants• Check recording equipment in advance• Devise a system for coding names of participants

to preserve confidentiality.• Be pleasant• Ask participants to complete an informed consent

form

Group interviews

• Also known as focus groups• Typically 3-10 participants• Provide a diverse range of opinions• Need to be managed to:

- ensure everyone contributes- discussion isn’t dominated by one person- the agenda of topics is covered, eliminate digressions.

Other Sources

• Telephone Interview– Can’t see participant’s body language or nonverbal

clues.

• Online Interviews– Asynchronous: E-mail or Synchronous: Chat Room

• Retrospective Interview– Check with the participant to be sure interviewer

correctly understood their remarks.

Analyzing interview data

• Depends on the type of interview• Structured interviews can be analyzed like

questionnaires• Unstructured interviews generate data like that

from participant observation• It is best to analyze unstructured interviews as

soon as possible to identify topics and themes from the data

Links

• UPA – (Usability Professionals Association)– Discovering User Needs Discovering User Needs

• Sage Services - User Research & Usability Techniques Available Through Sage.

Questionnaires

• Questions can be closed or open• Closed questions are easiest to analyze, and may

be done by computer• Can be administered to large populations• Paper, email & the web used for dissemination• Advantage of electronic questionnaires is that data

goes into a data base & is easy to analyze• Sampling can be a problem when the size of a

population is unknown as is common online

Questionnaire Style

• Questionnaire format can include:- ‘yes’, ‘no’ checkboxes- checkboxes that offer many options- Likert rating scales- semantic scales- open-ended responses

Designing Questionnaires

• Make questions clear and specific.• When possible, ask closed questions and offer a

range of answers.• Consider including a ‘no-opinion’ option for

questions that seeks opinions.• Think about the ordering of questions.• Avoid complex multiple questions.

Designing Questionnaires

• When scales are used, make sure the range is appropriate and does not overlap.

• Make sure the ordering of scales is intuitive and consistent.

• Avoid jargon.• Provide clear instructions on how to complete the

questionnaire.• Make a balance between white space and the need

to keep the questionnaire compact.

Encouraging a good response

• Ensure the questionnaire is well designed so that participants do not get annoyed.

• Provide a short overview section.• Include a stamped, self-addressed envelope.• Contact respondents through a follow-up letter.• Offer incentives.• Explain why you need the questionnaire to be

completed.

Online Questionnaires

• Advantages:– Responses received quickly.

– Copying and postage costs lower.

– Data can be transferred immediately to database.

– Time required is reduced.

– Errors in questionnaire can be corrected easily.

Online Questionnaires

• Disadvantages:– Sampling is problematic if population size is unknown.

– Preventing individuals from responding more than once.

– Individuals have also been known to change questions in email questionnaires

Developing Web-based Questionnaire

• Steps:1. Produce an error-free interactive electronic version from the

original paper-based one.

2. Make the questionnaire accessible from all common browsers and readable for different sized monitors and network locations.

3. Make sure information identifying each respondent will be captured and stored confidentially.

4. User-test the survey with pilot studies before distributing.

Asking experts

• Experts use their knowledge of users & technology to review software usability

• Expert critiques (crits) can be formal or informal reports

• Heuristic evaluation is a review guided by a set of heuristics

• Walkthroughs involve stepping through a pre-planned scenario noting potential problems

Heuristic evaluation

• Developed by Jacob Nielsen in the early 1990s• Based on heuristics distilled from an empirical

analysis of 249 usability problems• These heuristics have been revised for current

technology, e.g., HOMERUN for web• Heuristics still needed for mobile devices,

wearables, virtual worlds, etc.• Design guidelines form a basis for developing

heuristics

Nielsen’s heuristics• Visibility of system status• Match between system and real world• User control and freedom• Consistency and standards• Help users recognize, diagnose, recover from errors• Error prevention • Recognition rather than recall• Flexibility and efficiency of use• Aesthetic and minimalist design• Help and documentation

HOMERUN

• High-quality content• Often updated• Minimal download time• Ease of use• Relevant to users’ needs• Unique to the online medium• Netcentric corporate culture

Discount evaluation• Heuristic evaluation is referred to as discount

evaluation when 5 evaluators are used.

• Empirical evidence suggests that on average 5 evaluators identify 75-80% of usability problems.

3 stages for doing heuristic evaluation

• Briefing session to tell experts what to do• Evaluation period of 1-2 hours in which:

- Each expert works separately- Take one pass to get a feel for the product- Take a second pass to focus on specific features

• Debriefing session in which experts work together to prioritize problems

Heuristic evaluation of websites

• MEDLINEplus – medical information website

• Keith Cogdill, commissioned by NLM, evaluated this website

• Based evaluation on the following:– Internal consistency

– Simple dialog

– Shorcuts

– Minimizing the user’s memory load

– Preventing errors

– Feedback

– Internal locus of control

MEDLINEplus suggestions for improvement

• Arrangement of health topics– Arrange topics alphabetically as well as in categories

• Depth of navigation menu– Having a higher “fan-out” in the navigation would enhance

usability…meaning many short menus rather than a few deep ones.

Turning guidelines into heuristics for the web

• Navigation, access and information design are the three categories for evaluation

• Navigation– Avoidance of orphan pages

– Avoidance long pages with excessive white space (scrolling problems)

– Provide navigation support

– Avoidance of narrow, deep, hierarchical meus

– Avoidance of non-standard link colors

– Provide a consistent look and feel

Turning guidelines into heuristics for the web (cont.)

• Access– Avoidance of complex URLs

– Avoid long download times that annoy users• Pictures

• .wav files

• Large software files

• Information Design– Have good graphical design

– No outdated or incomplete information

– Avoid excessive color usage

– Avoid gratuitous use of graphics and animation

– Be consistent

Advantages and problems

• Few ethical & practical issues to consider• Can be difficult & expensive to find experts• Best experts have knowledge of application

domain & users• Biggest problems

- important problems may get missed- many trivial problems are often identified

Walkthroughs

• Alternative approach to heuristic evaluation• Expert walks through task, noting problems in

usability• Users are usually not involved• Several forms:

• Cognitive Walkthrough

• Pluralistic Walkthrough

Cognitive Walkthroughs

• Involves simulating the user’s problem-solving process at each step in the task

• Focus on ease of learning• Examines user problems in detail, without needing

users or a working prototype• Can be very time-consuming and laborious• Somewhat narrow focus• Adaptations may offer improvements

Cognitive Walkthrough Steps

• One or more experts walk through the design description or prototype with a specific task or scenario

• For each task, they answer three questions:• Will users know what to do?

• Will users see how to do it?

• Will users understand from feedback whether the action was correct or not?

Cognitive Walkthrough Steps

• A record of critical information is compiled• Assumptions about what would cause problems and why,

including explanations for why users would face difficulties

• Notes about side issues and design changes

• Summary of results

• Design is revised to fix problems presented

Pluralistic Walkthroughs

• Variation of Cognitive Walkthrough• Users, developers, and experts work together to

step through a scenario of tasks• Strong focus on users’ tasks• Provides quantitative data• May be limited by scheduling difficulties and time

constraints

Pluralistic Walkthrough Steps

• Scenarios are developed in the form of a series of screens

• Panel members individually determine the sequence of actions they would use to move between screens

• Everyone discusses their sequence with the rest of the panel

• The panel moves to the next scenario

Asking Users and Experts: Summary

• Various techniques exist for getting user opinions• Structured, unstructured, and semi-structured

• Interviews, focus groups, questionnaires

• Expert evaluations: heuristic evaluation and walkthroughs

• Various techniques are often combined for added accuracy and usefulness

Interview: Jakob Nielsen

• “Father of Web Usability”• Pioneered heuristic evaluation• Demonstrated cost-effective methods for usability

testing & engineering• Author of several acclaimed design books• www.useit.com• www.nngroup.com