Seeing is believing:
Using systematic observations in user experience research.
March 11th, 2011
Bram van Mil [email protected] consultant
Noldus Information Technology
Systematic Observations
Measuring Behavior with The Observer XT
Observation labs and other research tools
Agenda
International developer of professional software for behavioral research
From a single software license up to a complete observation lab
Complete with installation, training and support
HQ located in Wageningen, The Netherlands
Founded in 1989, currently ≈
115 employees
Customers in 75 countries
Installed base Observer
software ≈ 4400 organizations
Who we areNoldus Information Technology, since 1989…
Why study behavior?
Develop new drugs to cure diseases
Support medical diagnosis and therapy
Support human resource management
Improve educational systems
Improve animal welfare
Control insect pests
Create more usable products
Create safer work environments
Optimize team performance
Etc.
Questionnaires
Observations
Ways to study behavior
• Problem of meaning (interpretation of a question)
• Problem of omission (inadvertently leaving out our relevant information)
• Problem of memory
• Social desirability effect
• Question threat (questions may appear threatening)
• Interviewer characteristics (influence on participant)
• Gap between stated and actual behavior
Disadvantages of questionnaires
Record who does what, where, when and with whom
The alternative: Systematic observations
Objective
Accurate
Standardized
Quantify qualitative data
Advantages of systematic observations
A human observer watches the behavior of thesubjects and writes down the observations on paper
and uses a clock for time-information.
Disadvantages: Subjective measurements
When writing, the observer cannot observe
Very labor intensive
No integration with other modalities
Measuring Behaviorthe pioneers way…
Create a coding scheme before or during the observations and annotate behavior by a simple key press.
This will result in an event-log which contains time-stamped events that can be synchronized with multimodal data:
Videos
Physiological data
Eye tracking data
Etc.
Solution: The Observer XTfor structured observations
How to execute systematic observations
Three steps
1. Setup
2. Observe
3. Analyze
How to execute systematic observations
Step 1 – SetupObservation sources
How to execute systematic observations
Step 1 – SetupCoding scheme
Subjects
Behaviors
- Point events
- State events
Modifiers
How to execute systematic observations
Step 1 – SetupCoding scheme : Example
Time Subject, behavior, modifiers
09:35:12 Joe, run, fast, North
09:40:23 Mary, talk, Rose
09:40:24 Rose, smile
09:40:24 Rose, affect, 5
09:40:31 Mary, smile
How to execute systematic observations
Step 1 – SetupCoding scheme : Mutually exclusive behaviors
Mutually exclusive: one behavior ends the previous
How to execute systematic observations
Step 2 – Observe
Live scoring:
With or without videos
Offline scoring:
From digital video files
How to execute systematic observations
Step 3 – Analyze Data selection
Choose the data you want to analyze by filtering and nesting your data
This is done by data profiles:
In order to save different criteria for you data selection and data analysis
How to execute systematic observations
Step 3 – Analyze Visualization of your data
How to execute systematic observations
Step 3 – Analyze Descriptive statistics
Advanced analysis - Lag-sequential analyses - Reliability analysis
Statistical analysis - Export to Excel, SPSS or as ASCI file
Export highlight videos
How to execute systematic observations
Step 3 – Analyze: Numerical analysis on external data
Other tools
That can be used in behavioral research
Neutral
Happy
Sad
Angry
Surprised
Scare
Disgusted
Also: gender, age, ethnicity, glasses and facial hair
FaceReaderDetect facial expressions
Eye Tracking SystemsSee what your participants are looking at
Computer monitor / Standalone / Head mounted
Gaze tracks, heat maps, fixations and areas of interest
Physiological dataDetect mental load, physical load and emotional states
ECG – Electro Cardio Gram
EMG – Electro Myo Gram
EEG – Electroencephalogram
GSR – Galvanic Skin Response
Labs
Complete setups for behavioral research
Portable Usability Lab
Portable Observation Lab
Custom Mobile Observation Labs
Stationary Observation Lab
Game Experience LabGhent University
Two room Usability Lab for User Experience research in gaming on consoles and pc’s
Game Experience LabGhent University
Game Experience LabGhent University
Different facilities under 1 roof
– Observational-, sensory- and mood- labs
Observe eating behavior
– Introduce better suited products
Restaurant of the FutureWageningen University
Restaurant of the FutureWageningen University
Service, training and supportWe are there when you need us
On-site installation and training Behavioral scientists
Hardware engineers
Excellent technical support Helpdesk
Online knowledge base
Other Service contracts
Measuring Behavior conference
Bram van Mil [email protected] consultant
Top Related