Usability Evaluation Issues in Commercial and Research Systems Laila Dybkjær, Niels Ole Bernsen...
-
Upload
felix-preston -
Category
Documents
-
view
214 -
download
0
description
Transcript of Usability Evaluation Issues in Commercial and Research Systems Laila Dybkjær, Niels Ole Bernsen...
Usability Evaluation Issues in Commercial and Research Systems
Laila Dybkjær, Niels Ole BernsenNISLab, University of Southern Denmark
Hans Dybkjær SpeechLogic™, Prolog Development Center A/S
ASIDE 2005-11-10COST Workshop, Ålborg University
Slides available at www spokendialogue.dk/publications/2005k/ASIDE-2005-11-10.ppt
Spee
chLo
gi c&
N
ISLa
b
ASID
E 20
0520
05-1
1-10
It’s all about design – usable design ...
Users, prompts, modalities, media, ... Usability!
Spee
chLo
gi c&
N
ISLa
b
ASID
E 20
0520
05-1
1-10
Usability in academia and industry
Academia: New and challenging? Focus on advanced systems and new knowledge
Industry: Cost? ROI? Market? Customers? Focus on state-of-the-art and functionality
But they have much to learn from each other Lots of research results to streamline for industry Lots of ”simple systems” questions open to research Gap: EU research visions vs. industrial reality
What can they learn from each other?
Spee
chLo
gi c&
N
ISLa
b
ASID
E 20
0520
05-1
1-10
Three examplesSystem Traffic FAQ NICE HCA
Task / domain
Road traffic information
Holiday allowance information
H. C. Andersen’s life and fairytales edutainment
Purpose Commercial CommercialGov. support
Research
I/O Speech Speech Speech, gesture, 3D graphics
Language Da 50 words Da 500 words En 2000 words
Target Car drivers All employees Children 10-18
Who built it
PDC PDC, NISLab NISLab and 4 other EU partners
Wide range in purpose and complexity
Spee
chLo
gi c&
N
ISLa
b
ASID
E 20
0520
05-1
1-10
Cost and complexity
Academic focus: Prototypes – Industry focus: Final systems
Months
Traffic400 hoursSimple
FAQ4000 hoursComplex
NICE HCA40000 hoursVery complex
0 2 9 2313 36
P1 P22005
F1 F2 2002
F 2005
Spee
chLo
gi c&
N
ISLa
b
ASID
E 20
0520
05-1
1-10
Usability evaluation criteria
Difficult to select right criteria for given system• Is purpose to compare, to investigate,
or to define contract?• To make proper selection, one must know the range
and properties of criteria available
Many usability criteria vaguely defined • E.g. “adequacy” or “sufficiency” of …
Quantifiability often missing• Subjective or qualitative evaluation
New system types require new criteria• Must be clearly defined and operationalised
Standards may emerge, but new needs keep coming
Spee
chLo
gi c&
N
ISLa
b
ASID
E 20
0520
05-1
1-10
Core usability evaluation criteriaSystem Criteria
Traffic Interaction problemsCorrectness
Task and domain completeness
FAQ Interaction problemsCorrectnessTransaction success
Task and domain completeness
NICE HCA Conversation success NaturalnessReasoning capabilities Ease of useError handling
Scope of user modellingEntertainment and education valueUser satisfaction
Clearly different focus in academia and industry
Spee
chLo
gi c&
N
ISLa
b
ASID
E 20
0520
05-1
1-10
Usability evaluation methods
Which one to choose depends e.g. on• Evaluation purpose• Resources (who, time, money)• Stage of development process
Examples of methods• Walkthrough (early)• Focus groups (early, but ok any time)• Wizard-of-Oz (early-middle)• Field test (late)• Heuristic evaluation (best early but also ok later)• User interviews and questionnaires (any time)
Many current practice methods!
Spee
chLo
gi c&
N
ISLa
b
ASID
E 20
0520
05-1
1-10
Usability evaluation methodsSystem Methods
Traffic Walkthrough (using DialogDesigner)Semi-formal WOZ (using DialogDesigner)In-house scenario-based testExpert evaluation of domain information
FAQ Walkthrough (manually) In-house and external scenario-based testQuestionnaire on webMonitored scenario-based lab testsField data analysisExpert evaluation of domain information
NICE HCA WOZ in schools and in museumLab-test of first and second prototypePost-lab-test interviews
Industrial systems need broad range of methods
Spee
chLo
gi c&
N
ISLa
b
ASID
E 20
0520
05-1
1-10
Data and analysisSystem Data Analysis
Traffic Logfiles Problem identification via observation and feedbackLog-based analysis of problems
FAQ Logfiles; trans-criptions; trans-action annotations; questionnaires
Problem identification via ob-servation, feedback from users and domain experts, and ana-lysis of transcribed dialogues
NICE HCA
Logfiles; trans-criptions; topic annotation; Eng-lish evaluation; interviews
Analysis of WOZ and lab test data for design input; analysis of lab tests and interviews to get users’ opinion and develop new criteria
Research more data and analysis needed
Spee
chLo
gi c&
N
ISLa
b
ASID
E 20
0520
05-1
1-10
Electronic model IT tools possible
A lot of knowledge and theory can be made operational
Sketch, prompt design and recording, walk-through, WOZ, document, test, formal properties (coherence, well-formedness, ...), ..., you name it
Spee
chLo
gi c&
N
ISLa
b
ASID
E 20
0520
05-1
1-10
Academia and industry do meet ...
Industrial actions and challenges• Optimise existing processes• Automation (transcription support, annotation, ...)• Use known results and theories• Unknown effects of new technology
Academic challenges• Highly sophisticated technology• New factors to analyse, define, and measure• On-line adaptivity to users’ skills, expertise, …• Investigate troubles with ”simple” systems
... even though they are also different beasts