MEASUREMENT CHARACTERISTICS Error & Confidence Reliability, Validity, & Usability.
ECOVAL: A Framework for Increasing the Ecological Validity in Usability Testing
46
Suzanne Kieffer ECOVAL: A Framework for Increasing the Ecological Validity in Usability Testing Jean Vanderdonckt Ugo Braga Sangiorgi Université catholique de Louvain Louvain School of Management Louvain-la-Neuve, Belgium
-
Upload
jean-vanderdonckt -
Category
Software
-
view
151 -
download
0
Transcript of ECOVAL: A Framework for Increasing the Ecological Validity in Usability Testing
Suzanne Kieffer
ECOVAL: A Framework for Increasing the Ecological Validity in Usability Testing
Jean VanderdoncktUgo Braga Sangiorgi
Université catholique de Louvain
Louvain School of Management
Louvain-la-Neuve, Belgium
Ecological validity
Experimental design
Natural setting
x x
xxx
xxx
xx
xx
xx
xx
Environment
Tasks
Stimuli
Ecological validity
High
Low
Experimental design
Natural setting
≈
Experimental design
Natural setting
≠
Internal validityWas the study well done?
External validityCan we generalize the findings to particular persons, settings and times?
Ecological validityTo which extent can we generalize the findings?
User interface design
Ecological validity
Overall validity
Conclusionsof usability evaluations
What’s the problem with ecological validity?
No formal definition that enables
To objectively quantify the ecological validity of experimental designs
To capture the critical attributes of the real-world environment
LOGICAL IDITYVALECO ECOVALA framework for increasing the ecological validity in usability testing
Outline
The ECOVAL framework ECOVAL Guidelines GAMBIT
Case study Operability and usefulness of ECOVAL ROI of increased ecological validity Recommendations
The ECOVAL framework
The guidelines
No objects
Mock objects
Real objects
Only verbalized
Mimicked/verbalized
For real
GAMBIT – the supporting tool
Production of prototypes Easy Quick Cost-effective
Real user interface performances Interaction Navigation Responsiveness
GAMBIT – how to
1
Produce screens
GAMBIT – how to
2
Create behaviour
GAMBIT – how to
3
Conduct experiment
GAMBIT – how to
4
Download log file
• Q1: To which extent is ECOVAL applicable for controlling the ecological validity?
• Q2: What are the possible outcomes/benefits of increased ecological validity ?
Case study
Hot-Dip Galvanizing on Continuous Lines
Monitoring report
Task Control Result
Task Control Result
Artefact:paper checklist
Section > Subsection
Function – Name - Date – Shift
Section > Subsection
300 items a day
Monitoring report
Task Control Result
Task Control Result
Artefact:paper checklist
Section > Subsection
Function – Name - Date – Shift
Section > Subsection
Timeline
Task analysis
User profile
Usability goal
setting
Screen design
Usability evaluation
Usability evaluation
Paper versus GAMBIT prototype
Ecological validity
Ratio Paper-to-Gambit for ecological validity of 1:1.17
Balsamiq mockups
Experimentations
Paper GAMBIT
5 key users x 2
10 participants (30’) 18 participants (1 hour)
5 key users x 3+ 3 user representatives
V16 participants
V212 participants
New version of the prototype
Apparatus
Results and discussion
Early detection of usability problems Increased user efficiencies Increased user satisfaction
Cost-benefit analysis
130 hours
18 hours
Cost-benefit analysis (continued)
$74,480
$194,480
Paper GAMBIT
Ecologicalvalidity
1 1.17
Datacollection
Manual & face-to-faceDifficult & tedious Unstructured processObtrusive
Automated & remoteComfortableStructured processUnobtrusive
OutcomesCompliant withthe literature
Early detection of usability problemsIncreased user productivityDecreased late design changesIncreased organizational efficiency
Recommendations
Take away
{ECOVAL, Guidelines, GAMBIT}
Case study Operability Usefulness ROI
Recommendations
![Interfaces Design and User Evaluation of …critical factor in terms of the validity of the usability test [1] Figure 2: A comparison of set-up of usability testing in real world and](https://static.fdocuments.in/doc/165x107/5e433b9bf837cf5dc5659a70/interfaces-design-and-user-evaluation-of-critical-factor-in-terms-of-the-validity.jpg)
