Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… ·...
Transcript of Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… ·...
![Page 1: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/1.jpg)
Usability Evaluation of context-aware mobile systems:
A reviewStavros Garzonis
VODAFONE GROUP R&Dhttp://www.vodafone-rnd.com/
![Page 2: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/2.jpg)
Mobile Service Discovery
Humans discovering mobile servicesHow can context awareness assist in that?Evaluating context-aware mobile services could helpBut how do I do that??
![Page 3: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/3.jpg)
Decisions, decisions…
Field or Lab?Experts or Users?Task-based or exploratory learning?Co-operative or ‘strict’ think-aloud?Direct/Indirect Observation, Field Questionnaires and/or Logging?Grounded or Discount Analysis?
![Page 4: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/4.jpg)
Taking into account…
Low HW capabilitiesLimited Input/Output (UI)Mobility
Limited Attention, unstable hands etc.
Dynamic ContextSensing Context & adapting
![Page 5: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/5.jpg)
What literature has to offerGuidelines for field evaluations
Rowley (1994)Borntrager & Cheverest (2003)
Evaluation Techniques specifically for mobile devicesBrewster 2002: EarconsPirhoen, Brewster, Holguin 2002: TouchPlayer
Evaluation Methods specifically for mobile servicesVetere et al. 2003: Heuristic Walkthrough
Field or Lab debateKjeldskov et al. : Comparing 4 evaluation studiesKjeldskov & Stage: Comparing 6 evaluation studiesKjeldskov et al. : Is it worth the hassle?
![Page 6: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/6.jpg)
Walking Through Mobile Use: Novel Heuristics and their Application
F. Vetere, S. Howard, S Pedell, S. Balbo The University of Melbourne
OzCHI 2003
![Page 7: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/7.jpg)
Traditional Usability Heuristics do not…
Capture problems associated with changing context of useExplicitly deal with dynamic context
thus missing out ‘social transitions’
![Page 8: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/8.jpg)
‘Mobile Heuristic Walkthrough’
A hybrid evaluation technique ofCognitive Walkthrough andHeuristic Evaluation
with added Heuristics to capture the social aspects of mobility
![Page 9: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/9.jpg)
Evaluating the methodEvaluators did not have a sense of real world contextCognitive workload of being expert anduser is “rather high”The concept of ‘locale’ will probably need instantiating for each applicationCurrent heuristics provide more concrete results and redesign implications
![Page 10: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/10.jpg)
To Field or not to Field?
41% of Mobile HCI involved evaluation71% was done in labs
Kjeldskov & Graham (2003)
Might involve travellingEasier to find participantsHarder to record or think-aloudEasier to collect dataUncontrolled EnvironmentExperimental ControlReal ContextUnnatural SettingTime Consuming & ExpensiveFaster & Cheaper
FIELDLAB
![Page 11: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/11.jpg)
Evaluating the usability of a mobile guide: the influence of
location, participants and resources
J. KJELDSKOV, C. GRAHAM, S. PEDELL, F. VETERE, S. HOWARD, S. BALBO, J. DAVIES
The University of Melbourne & Aalborg University
Behaviour & Information Technology 2005
![Page 12: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/12.jpg)
Comparing 4 approaches
Field EvaluationLaboratory EvaluationHeuristic WalkthroughRapid Reflection
![Page 13: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/13.jpg)
Results
• All methods found 59% of the all the problems
• All methods found 80% of the critical problems
• Field Evaluation was the most effective in serious problems
• Field Evaluation was the least effective in cosmetic problems
![Page 14: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/14.jpg)
• Only ONE Serious Problem was mutually identified by all four methods
• Field and lab had considerable overlapping None of the cosmetic problems was found by all techniques
• Lab and HE identified the same set of cosmetic problems.
![Page 15: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/15.jpg)
LESSONS LEARNEDCritical Problems are most probably detectable, regardless the technique.Lab evaluation tend to produce more device-centred findings (e.g. labelling)Field evaluation misses out most of the cosmetic but was most effective in finding the serious problemsRapid Reflection proved very cost-effective but exhaustive data collection and analysis is good for building theoretical foundations
![Page 16: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/16.jpg)
New techniques for Usability Evaluation of Mobile Systems
J. Kjeldskov & J. Stage
The Aalborg University
International Journal of Human-Computer Studies (IJHCS) 2004
![Page 17: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/17.jpg)
Evaluating against 6 conditions
1. Sitting on a chair at a table2. Walking on a treadmill at constant
speed3. Walking on a treadmill at varying speed4. Walking at constant speed on a course
that is constantly changing5. Walking at a varying speed on a course
that is constantly changing6. Walking in a pedestrian street
![Page 18: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/18.jpg)
Results
The sitting technique was the most effectiveBut mainly identifies more cosmetic problemsTechniques involving movement & navigation find UI layout problems
![Page 19: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/19.jpg)
Is it worth the hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field
J. Kjeldskov, M. B. Skov, B. S. Als and R.T. Hoegh
Aalborg University
MobileHCI 2004
![Page 20: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/20.jpg)
Results
60 %100 %Cosmetic
52.6 %94.7 %Serious
87.5 %100 %Critical
114Uniquely identified
62.2 %97.3 %Problems found
6534Person-Hours spent
FIELDLABORATORY
![Page 21: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/21.jpg)
Distribution of Identified Usability Problems
![Page 22: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/22.jpg)
Context-aware related problems
Both condition identified all 7 context-aware related problems.Pushing Information was found to be more annoying in the fieldLack of data entry validation was picked up only in the field
![Page 23: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/23.jpg)
BUT…
The evaluation time in the lab lasted twice as in the field per participant The nurses participating in the field were less experienced
![Page 24: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/24.jpg)
Summary
There are not any evaluation methods to effectively address the specific challenges of context-aware mobile devicesThe major difference between Field and Lab evaluation is in cosmetic problemsThe studies mentioned used 5 or 6 participants per condition.
![Page 25: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/25.jpg)
Suggestions
Field or Lab?Experts or Users?Task-based or exploratory learning?Co-operative or ‘strict’ think-aloud?Direct/Indirect Observation, Field Questionnaires and/or Logging?Grounded or Discount Analysis?
![Page 26: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/26.jpg)
References 1Brewster, S. 2002, Overcoming the lack of screen space on mobile computers. Personal and Ubiquitous Computing, 6, 188-205Borntrager, T., Jameson, A., Kruger, A. and Butz, A., 2002, Location-aware shopping assistance: evaluation of a decision-theoretic approach. Proceedings of Mobile HCI 2002, Pisa, Italy, http://www.mguides.infoKjeldskov, Jesper ; Skov, Mikael B. ; Als, Benedkite S. ; Høegh, Rune T. Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field. I: Proceedings of Mobile HCI 2004. Springer-Verlag, Berlin, 2004. s. 61-73.
![Page 27: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/27.jpg)
References 2Kjeldskov J. and Stage J. (2004) New Techniques for Usability Evaluation of Mobile Systems. International Journal of Human-Computer Studies (IJHCS), 60, 599-620, ElsevierKjeldskov, J., Graham, C., Pedell,S., Vetere,F., Howard,S., Balbo,S., & Davies,J. Evaluating the Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of Behaviour and Information Technology. Volume 24/1.Kjeldskov J. and Graham C. (2003) A Review of Mobile HCI Research Methods. In Proceedings of the 5th International Mobile HCI 2003 conference, Udine, Italy, Springer- Verlag, LNCS
![Page 28: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/28.jpg)
References 3Rowley D. E. 1994, Usability Testing in the Field: Bringing the Laboratory to the User, Proccedings of CHI 1994Pirhonen, A., Brewster, S.A. and Holguin, C. 2002, Gestural and audio metaphors as a means of control for mobile devices. Proccedings of CHI 2002 (New York: ACM), pp. 291-298Vetere, F., Howard, S., Pedell, S. and Balbo, S., 2003, Walking through mobile use: novel heuristics and their application. Proceedings of OzCHI 2003, Brisbane, Australia (Brisbane: CHISIG), pp. 24-32
![Page 29: Usability Evaluation of context- aware mobile systems: A ... Files/Garzonis/ukubinet05_sl… · Usability of a Mobile Guide: a Comparison of Four Different Approaches. Journal of](https://reader033.fdocuments.in/reader033/viewer/2022050517/5fa096e5216fe9440f2cc946/html5/thumbnails/29.jpg)
Usability Evaluation of context-aware mobile systems:
A reviewStavros Garzonis
VODAFONE GROUP R&Dhttp://www.vodafone-rnd.com/