'Living Lab' for HCI - presentation made at HCI International 2009
-
Upload
ed-chi -
Category
Technology
-
view
3.783 -
download
1
description
Transcript of 'Living Lab' for HCI - presentation made at HCI International 2009
EdH.ChiAreaManagerandSr.ResearchScientistPaloAltoResearchCenter
2009HCIInternationalConference,SanDiego,CA
Asafield,earlyfundamentalcontributionsfrom:– Computerscientistsinterestedinchangesinwayswe
interactwithinformationsystems– Psychologistsinterestedintheimplicationsofthese
changes
Combustible,because:– Computerscientistswanttocreategreattools,butdidn’t
knowhowtomeasureimpact– Psychologistswanttogobeyondclassicalresearchofthe
brainandhumancognition
TheneedtoestablishHCIasascience– Adoptmethodsfrompsychology– GoodExamples:Fitts’Law,ModelsofHumanMemory,
CognitiveandBehavioralModeling,InformationForaging– Dualpurpose:understandnatureofhumanbehaviorand
buildupascienceofHCItechniques.
7/24/09 HCIC "Living Lab" 2
Manyproblemsdon’tfitthelaboratoryexperimentalmethodsanymore– Beyondauserinfrontofcomputer;Yetevaluationmethodsmostly
stayedthesame– Controlledlabstudyasthegoldstandardforacceptance
ChangesandTrendsinSocialComputingandUbiComp
7/24/09 HCIC "Living Lab" 3
Old Assumptions New Considerations
Single display Multiple displays
Knowledge work Games, communication, social apps
Isolated worker Collaborative and social groups
Stationary location Mobile and stationary
Short task durations Short and long tasks, and tasks with no time boundries
Controllable experimental conditions Uncontrollable experimental conditions
Artificialexperimentalsetupsareonlycapableoftellingusbehaviorsinconstrainedsituations
Hardtogeneralizetonewtaskcontexts(withinterruptions,othertasks,othergoals,unfocusedattention,moredisplays)
Hardtogeneralizetoothertools,apps Ecologicalconsiderations
Adoptionofmobiletechnology iPhonesinJapan,single‐handedinput[PARC] BestsellingphonesinIndonesiacomeswithacompass[Bell]
Impossibletoanswerquestionsaboutaggregatebehaviorsofgroups
AggregatebehaviorofWikipediaorDelicioususers
7/24/09 4 HCIC "Living Lab"
Conductresearchonrealplatformsandservices– Nottoreplacecontrolledlabstudies– Expandourarsenaltocovernewsituations
Someprinciples:– Embeddedintherealworld– Ecologicallyvalidsituations– Embracethecomplexity– Relyonbig‐data‐sciencetoextractpatterns
Notfirsttosuggestthis:– S.Carter,J.Mankoff,S.KlemmerandT.Matthews.Exitingthecleanroom:On
ecologicalvalidityandubiquitouscomputing.HCIJournal,2008– EClass[Abowd],PlaceLab[Intille],PlasmaPoster[ChurchillandNelson],Digital
FamilyPortrait[Rowan,Mynatt]
7/24/09 5 HCIC "Living Lab"
GroupLens / MovieLens [Riedl, Konstan, Univ. Minnesota]
Games with a Purpose [von Ahn et al]
7/24/09 HCIC "Living Lab" 8
World of Warcraft [Yee, Ducheneaut et al]
Wikipedia History Flow [Viégas et al]
Bucket Testing or A/B Testing [Kohavi et al]
A B
UbiFit [Consolvo et al]
7/24/09 HCIC "Living Lab" 13
Masterdegreewasincomputationalmolecularbiology Analogy:Justasbiologistsworkonmodelplantsand
genomesinthelab,thistellsusjusthowitbehavesinanisolatedenvironmentundercontrolledconditions,butnothowtheplantwillbehaveintherealworld.
Biologistsdon’tjuststudymodelsinthelab,butinthewildalso.
7/24/09 HCIC "Living Lab" 14
Twodimensions– 1.Whetherthesystemisunderthecontroloftheresearcher– 2.Whetherthestudyisconductedinthelaborinthewild
System Control System Not in Control
Laboratory (1) Build a system, study in the Lab
(2) Adopt a system, study in the Lab
Wild (Real World)
(4) Build a system, release it, study in the Wild
(3) Adopt a system, study in the Wild
7/24/09 15 HCIC "Living Lab"
TraditionalApproach;Numerousexamples FavoredbyHCIfieldreviewers Typicalsituationisthestudyofsomeinteractiontechnique
– Peninput,gestures,perceptionofsomevisualizeddata,readingtasks,mobiletextinput
Typicalmeasuresarequantitativeinnature– performanceintime,performanceinaccuracy,eyetracking,learning
measures,userpreferences
Issues:– Notalwaysecologicallyvalid– Hardtotakeallinteractionsintoaccount– Oftentime‐consuming;eventhoughwethoughtwecoulddoitfast.
7/24/09 16 HCIC "Living Lab"
Hardertofindintheliterature Oftencomparingagainstanoldersystemasbaseline Typicalcaseiscomparisonoftwosystems
– (onewebsitewithanother,onewordprocessorvs.another)– Whichhighlightingfeatureworksbetter– Twotextinputtechniqueonacellphone
Typicalmeasuresaresimilarto(1) Issues:
– Somesimilarissuesto(1)becauseit’sinlab– Systemfeaturenotincontrol,sonotabletocomparefairly,or
isolatethefeature
7/24/09 HCIC "Living Lab" 17
Twodimensions– 1.Whetherthesystemisunderthecontroloftheresearcher– 2.Whetherthestudyisconductedinthelaborinthewild
System Control System Not in Control
Laboratory (1) Build a system, study in the Lab
(2) Adopt a system, study in the Lab
Wild (Real World)
(4) Build a system, release it, study in the Wild
(3) Adopt a system, study in the Wild
7/24/09 18 HCIC "Living Lab"
Realapplicationsinecologicalvalidsituations Realfindingscanbeappliedtoarunningsystem Impactofresearchismoreimmediate,sincesystemisalready
running Typicalcaseisloganalyticswithlargesubjectpools
– logstudiesofwebsites,realmobilecallingusages,websearchlogs,studiesofWikipediaedits.
Typicalmeasuresarestickiness,amountofactivity,clusteringanalysis,correlationalanalysis
Issues:– Factorsnotincontrol,findingsnotcomparable– Factorscannotbeisolated– Reasonsforfailureisoftenjustguesswork
7/24/09 HCIC "Living Lab" 19
Hypothesis:ConflictiswhatdrivesWikipediaforward. Howtostudythis?
– JohnTukeyparadigm– Getalargepaper,andplotallofthedata!
– DownloadedallofWikipediaandalloftherevisions– Hadoop/MapReduce,MySQL,etc.
7/24/09 HCIC "Living Lab" 20
7/24/09 21
60%
65%
70%
75%
80%
85%
90%
95%
100%
2001 2002 2003 2004 2005 2006
Perc
enta
ge o
f tot
al e
dits
Article
User
Article Talk
User Talk
Other
Maintenance
HCIC "Living Lab"
Group A
Group B Group C
Group D
Number of users in user group A B C Total
Users with Korean point of view 10 6 0 16
Users with Japanese point of view 1 8 7 16
Neutral or Unidentified 7 3 6 17 7/24/09 22 HCIC "Living Lab"
Mediators
Sympathetic to parents
Sympathetic to husband
Anonymous (vandals/spammers)
7/24/09 23 HCIC "Living Lab"
7/24/09 HCIC "Living Lab" 24
7/24/09 25 HCIC "Living Lab"
7/24/09 26 HCIC "Living Lab"
7/24/09 27 HCIC "Living Lab"
Hypothesis:SocialTaggingdoesn’tscaleovertime. Howtostudythis?
– Crawlasmuchtaggingdataaswecan.– Studythenoiseinthesystem.
– 40machinesfor3months
7/24/09 HCIC "Living Lab" 28
Topics
Users Documents
Tags
T1…TnEncodingDecoding
Noise
7/24/09 29 HCIC "Living Lab"
Concepts
7/24/09 HCIC "Living Lab" 30
Source: Hypertext 2008 study on del.icio.us (Chi & Mytkowicz)
31 7/24/09 HCIC "Living Lab"
7/24/09 32
Guide
Web
Howto
Tips Help
Tools
Tip
Tricks
Tutorial
Tutorials
Reference
Semantic Similarity Graph
HCIC "Living Lab"
7/24/09 HCIC "Living Lab" 33
Twodimensions– 1.Whetherthesystemisunderthecontroloftheresearcher– 2.Whetherthestudyisconductedinthelaborinthewild
System Control System Not in Control
Laboratory (1) Build a system, study in the Lab
(2) Adopt a system, study in the Lab
Wild (Real World)
(4) Build a system, release it, study in the Wild
(3) Adopt a system, study in the Wild
7/24/09 34 HCIC "Living Lab"
Similarto(3),practicalforrunningsystems;ecologicallyvalid,impactcanbeimmediate.– Goodforcasesinwhicheconomicsmakessense[Google]– Changestosystemispossible;Factorscanbecontrolled.
TypicalcasemightbeA/Btesting,largesubjectpools Typicalmeasuresarebeingdeveloped
– Impactmeasures.Largevisit#andinterest(measuredbyblogposts?)NewBusinessinquiries?
– Usabilitymeasuresvs.Usefulnessmeasures
Issues:– Effortandresourcerequirementisdroppingbutstillsignificant– Hardforaresearchlabtotakeon
7/24/09 HCIC "Living Lab" 35
7/24/09 36
HCIC "Living Lab"
7/24/09 37
HCIC "Living
Lab"
7/24/09 38
HCIC "Living
Lab"
Personal Computing [Xerox PARC]
Evaluationmethodsarein‐separablefromthekindsofscienceandmodelsthatcanbebuildinafield.
Platformadvancesenablerealtechnologyinsertionintorealworldsituationscheaperandmoremanageable.
7/24/09 43 HCIC "Living Lab"
Characteriza7on Models
PrototypesEvalua7ons
ResearchVision:Understandhowsocialcomputingsystemscanenhancetheabilityofagroupofpeopletoremember,think,andreason.
LivingLaboratory:Createapplicationsthatharnesscollectiveintelligencetoimproveknowledgecapture,transfer,anddiscovery.
http://asc‐parc.blogspot.comhttp://[email protected]
WikiDashboard MrTaggy SparTag.us