Chapter 10: Introducing Chapter 10: Introducing EvaluationEvaluation

Group 4: Tony Masi, Sam Esswein, Brian Rood, Chris Troisi

Chapter GoalsChapter Goals

Explain the concepts and terms used to Explain the concepts and terms used to discuss evaluation.discuss evaluation.

HutchWorld case study.HutchWorld case study.

Examine how different techniques are Examine how different techniques are used at different stages of development.used at different stages of development.

Discuss how developers cope with real-Discuss how developers cope with real-world constraints.world constraints.

Definition of EvaluationDefinition of Evaluation

Dictionary Definition – “To ascertain or fix Dictionary Definition – “To ascertain or fix the value or worth of.”the value or worth of.”

Book’s Definition – “The process of Book’s Definition – “The process of systematically collecting data that informs systematically collecting data that informs us about what it is like for a particular user us about what it is like for a particular user or group of users to use a product for a or group of users to use a product for a particular task in a certain type of particular task in a certain type of environment.”environment.”

Some Evaluation Companies:Some Evaluation Companies:

META Group, Inc. – META Group, Inc. – www.metagroup.comwww.metagroup.com

Canadian Innovation Centere - Canadian Innovation Centere - http://www.innovationcentre.ca/http://www.innovationcentre.ca/

Hitachi Data Systems - Hitachi Data Systems - http://www.hds.com/http://www.hds.com/

How To Evaluate Your Software - How To Evaluate Your Software - http://www.ageesw.com/how.htmhttp://www.ageesw.com/how.htm

Iterative design & evaluation is a continuous process that Iterative design & evaluation is a continuous process that examines:examines:

Early ideas for conceptual model Early ideas for conceptual model

Early prototypes of the new systemEarly prototypes of the new system

Later, more complete prototypesLater, more complete prototypes

Designers need to check that they understand users’ Designers need to check that they understand users’ requirements.requirements.

What to evaluateWhat to evaluate

Why to evaluateWhy to evaluate

Designers should not presume everyone is Designers should not presume everyone is like them or that following set guidelines like them or that following set guidelines guarantees usability.guarantees usability.

Evaluation is needed to check that users can Evaluation is needed to check that users can use the product and like it.use the product and like it.

Why to evaluateWhy to evaluate

www.AskTog.comwww.AskTog.com– ““Iterative design, with its repeating cycle of Iterative design, with its repeating cycle of

design and testing, is the only validated design and testing, is the only validated methodology in existence that will methodology in existence that will consistently produce successful results.consistently produce successful results. If If you don’t have user-testing as an integral part you don’t have user-testing as an integral part of your design process you are going to throw of your design process you are going to throw buckets of money down the drain.”buckets of money down the drain.”– – Bruce TognazziniBruce Tognazzini

Why to evaluateWhy to evaluate

Tognazzini’s 5 reasons to evaluate:Tognazzini’s 5 reasons to evaluate:– Problems are fixed before the product is shipped, not Problems are fixed before the product is shipped, not

after.after.– The team can concentrate on real problems, not The team can concentrate on real problems, not

imaginary ones.imaginary ones.– Engineers code instead of debatingEngineers code instead of debating– Time to market is sharply reducedTime to market is sharply reduced– Upon first release, sales department has a rock-solid Upon first release, sales department has a rock-solid

design it can sell without having to pepper pitches with design it can sell without having to pepper pitches with how well the NEXT release will work.how well the NEXT release will work.

Why to evaluateWhy to evaluate

USABILITY TESTING involves measuring the USABILITY TESTING involves measuring the performance of typical users on typical tasks.performance of typical users on typical tasks.

SATISFACTION can be evaluated through SATISFACTION can be evaluated through questionnaires and interviews.questionnaires and interviews.

Trends are towards evaluating more Trends are towards evaluating more subjective user-experience goals, like subjective user-experience goals, like emotionally satisfying, motivating, fun, etc.emotionally satisfying, motivating, fun, etc.

When to evaluateWhen to evaluate

Throughout designThroughout design

New productNew product– Use mockups, sketches, and other low-fidelity prototyping Use mockups, sketches, and other low-fidelity prototyping

techniques to elicit users’ opinionstechniques to elicit users’ opinions

– Goal is to asses how well a design fulfills users’ needs and whether Goal is to asses how well a design fulfills users’ needs and whether the users like itthe users like it

Upgrade existing productUpgrade existing product– Evaluations to compare user performance and attitudes toward new Evaluations to compare user performance and attitudes toward new

design with those of the previous versionsdesign with those of the previous versions

Evaluation is a key ingredient for a successful design.Evaluation is a key ingredient for a successful design.

Two main types of evaluationTwo main types of evaluation

Formative evaluationFormative evaluation is done at different stages of is done at different stages of development to check that the product meets users’ needs.development to check that the product meets users’ needs.– Design proceeds through iterative cycles of ‘design-test-Design proceeds through iterative cycles of ‘design-test-

redesign’redesign’– Helps ensure products success upon first arrival in marketHelps ensure products success upon first arrival in market

Summative evaluationSummative evaluation assesses the quality of a finished assesses the quality of a finished product.product.– Satisfy sponsoring agencySatisfy sponsoring agency– Check that standard is being upheldCheck that standard is being upheld

Chapter 10 focuses on Formative Evaluation.Chapter 10 focuses on Formative Evaluation.

Story of the 1984 OMS Story of the 1984 OMS (Box 10.1)(Box 10.1)

Background –Background –– Voice mail system for Olympic Games contestants and Voice mail system for Olympic Games contestants and

their families could send and receive messagestheir families could send and receive messages– Developed by IBMDeveloped by IBM– Could be used from almost any push-button phone Could be used from almost any push-button phone

system around the worldsystem around the world

Reasons for intense evaluation -Reasons for intense evaluation -– IBM’s reputation at stakeIBM’s reputation at stake– Olympics a high-profile eventOlympics a high-profile event

Story of the 1984 OMS Story of the 1984 OMS (Box 10.1)(Box 10.1)

Evaluation activities –Evaluation activities –– Use of printed scenariosUse of printed scenarios

– Iterative testing of user guidesIterative testing of user guides

– Development of early simulationsDevelopment of early simulationsKeypadsKeypads

ReactionsReactions

– Olympian on design teamOlympian on design team

– Interviews with other OlympiansInterviews with other Olympians

– Overseas testsOverseas tests

– Free coffee and donut testsFree coffee and donut tests

– ‘‘Try-and-destroy-it’ with Computer Science studentsTry-and-destroy-it’ with Computer Science students

– Pre-Olympic field testsPre-Olympic field tests

– Heavy traffic testsHeavy traffic tests

HutchWorld Case StudyHutchWorld Case Study

A Virtual CommunityA Virtual CommunityCollaboration:Collaboration:– Microsoft’s Virtual Worlds Research GroupMicrosoft’s Virtual Worlds Research Group– Fred Hutchinson Cancer Research CenterFred Hutchinson Cancer Research Center

Uses:Uses:– chattingchatting– storytellingstorytelling– discussionsdiscussions– emotional supportemotional support

Why? Why? – cancer patient isolation issuescancer patient isolation issues

How to design HutchWorld?How to design HutchWorld?

Needs:Needs:– usefuluseful– engagingengaging– easy-to-useeasy-to-use– emotional satisfactionemotional satisfaction

Early ideas:Early ideas:– what is typical cancer treatment?what is typical cancer treatment?– what resources are available to patients?what resources are available to patients?– what are the specific needs of the users?what are the specific needs of the users?– what kind of “world” should be the model?what kind of “world” should be the model?– how will users interact within the virtual community?how will users interact within the virtual community?– what should it look like?what should it look like?

No stone left unturnedNo stone left unturned

Interviews with patients, caregivers, family, Interviews with patients, caregivers, family, friends, clinicians, social support groups, friends, clinicians, social support groups, former patients, and expertsformer patients, and experts

Reading of latest research literature, and Reading of latest research literature, and HutchWorld web pagesHutchWorld web pages

Visiting Fred Hutch research facilities, and Visiting Fred Hutch research facilities, and the Hutch school for pediatric patients and the Hutch school for pediatric patients and juvenile patient family membersjuvenile patient family members

ProblemProblem

Inadequate non-verbal feedbackInadequate non-verbal feedback– potential for misunderstandingpotential for misunderstanding– no:no:

facial expressionsfacial expressions

body languagebody language

tone of voicetone of voice

ResearchResearch

Studies indicates social support helps Studies indicates social support helps cancer patients cope psychologically with cancer patients cope psychologically with their disease.their disease.

Patients also benefit in their overall Patients also benefit in their overall physical wellbeing.physical wellbeing.

Example: women with breast cancer Example: women with breast cancer receiving therapy lived on average twice receiving therapy lived on average twice as long as those that did not.as long as those that did not.

Features of HutchWorldFeatures of HutchWorld

Availability:Availability:– anytime, day or nightanytime, day or night– regardless of geographic locationregardless of geographic location

Designed to resemble the outpatient facility Designed to resemble the outpatient facility – This real-world metaphor helped users infer the This real-world metaphor helped users infer the

functionality.functionality.

Synchronous chat environment was selected for Synchronous chat environment was selected for realism (vs. asynchronous)realism (vs. asynchronous)

3D photographic avatars (p. 326)3D photographic avatars (p. 326)

Before testingBefore testing

Logistical issues:Logistical issues:– Who would provide training for patients and Who would provide training for patients and

testers?testers?– How many systems were needed for testing?How many systems were needed for testing?– Where should these systems be placed?Where should these systems be placed?

Testing HutchWorldTesting HutchWorld

Test 1:Test 1:– six computers six computers – scaled-back prototypescaled-back prototype– Microsoft specialists trained Hutch volunteersMicrosoft specialists trained Hutch volunteers– events were hosted in the prototypeevents were hosted in the prototype

Test 1 observations:Test 1 observations:– general usage of the prototypegeneral usage of the prototype– usage of the space during unscheduled timesusage of the space during unscheduled times

Testing HutchWorldTesting HutchWorld

Test 1 results:Test 1 results:– small user communitysmall user community– critical mass concept – not enough participants to fill critical mass concept – not enough participants to fill

the chat room for successful conversationthe chat room for successful conversation– lack of interestlack of interest– patient availabilitypatient availability– patients preferred asynchronous communication (via patients preferred asynchronous communication (via

email, journals, etc.)email, journals, etc.)– prototype did not include original computer uses prototype did not include original computer uses

patients played games and searched the internetpatients played games and searched the internet

Redesigning HutchWorldRedesigning HutchWorld

a more “unified” product was desired that a more “unified” product was desired that included a variety of communication, included a variety of communication, information, and entertainment tasksinformation, and entertainment tasks

new support:new support:– more asynchronous communicationmore asynchronous communication– information-retrieval toolsinformation-retrieval tools– email, a bulletin board, text-chatemail, a bulletin board, text-chat– games and other entertainment tasksgames and other entertainment tasks– a web page creation toola web page creation tool– a way to check if anyone is around to chat witha way to check if anyone is around to chat with

Usability TestsUsability Tests

Seven participantsSeven participants– four had used chat roomsfour had used chat rooms– all had browsed the weball had browsed the web– given five minutes to get familiar with softwaregiven five minutes to get familiar with software

A running commentary was given by each during A running commentary was given by each during exploration (what each was looking at, thinking, exploration (what each was looking at, thinking, or confused by)or confused by)After five minutes, a series of structured tasks were After five minutes, a series of structured tasks were given focusing on how the participants (p. 329):given focusing on how the participants (p. 329):– dealt with their virtual identitydealt with their virtual identity– communicated with otherscommunicated with others– retrieved desired informationretrieved desired information– found entertainmentfound entertainment

QuestionnaireQuestionnaire

After the test participants were asked to fill After the test participants were asked to fill out a questionnaire about their experience out a questionnaire about their experience with HutchWorldwith HutchWorld– What did you like about HutchWorld?What did you like about HutchWorld?– What did you not like about HutchWorld?What did you not like about HutchWorld?– What did you find confusing or difficult to use What did you find confusing or difficult to use

in HutchWorld?in HutchWorld?– How would you suggest improving How would you suggest improving

HutchWorld?HutchWorld?

Usability FindingsUsability Findings

The back button did not always work.The back button did not always work.Users ignored navigation buttonsUsers ignored navigation buttons– more prominent buttons were neededmore prominent buttons were needed

Users expected that objects in 3D would do something Users expected that objects in 3D would do something when clicked onwhen clicked on– provide links to web pages when objects are clickedprovide links to web pages when objects are clicked

Users did not realize other real people were interacting Users did not realize other real people were interacting with them in the worldwith them in the world– wording was changed in the overview descriptionwording was changed in the overview description

Users did not notice the chat window and instead Users did not notice the chat window and instead chatted with people on the participation listchatted with people on the participation list– instructions on where to chat were clarifiedinstructions on where to chat were clarified

Follow-upFollow-up

more rounds of observation and testing more rounds of observation and testing were conducted with new subjectswere conducted with new subjects

HutchWorld was installed at the Fred HutchWorld was installed at the Fred Hutchinson CenterHutchinson Center

observations continued on the users observations continued on the users – which parts of the system are being used?which parts of the system are being used?– when are they being used?when are they being used?– why are they being used?why are they being used?

Future of HutchWorldFuture of HutchWorld

evaluation of the effects of the software at the evaluation of the effects of the software at the Fred Hutchinson CenterFred Hutchinson Centerinvestigation will include:investigation will include:– How the computers and software impact the social How the computers and software impact the social

wellbeing of the patients and their caregivers?wellbeing of the patients and their caregivers?– What type of computer-based communication best What type of computer-based communication best

supports this patient community?supports this patient community?– What are the general usage patterns of the system?What are the general usage patterns of the system?– How might any medical facility use computers and How might any medical facility use computers and

software like HutchWorld to provide social support for software like HutchWorld to provide social support for its patients and caregivers?its patients and caregivers?