'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer...
-
Upload
ed-chi -
Category
Technology
-
view
1.130 -
download
3
description
Transcript of 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer...
EdH.ChiAreaManagerandSr.ResearchScientistPaloAltoResearchCenter
2009HCICWorkshopatYMCACamp,Colorado
Asafield,earlyfundamentalcontributionsfrom:– Computerscientistsinterestedinchangesinwaysweinteract
withinformationsystems– Psychologistsinterestedintheimplicationsofthesechanges
Combustible,because:– Computerscientistswanttocreategreattools,butdidn’tknow
howtomeasureimpact– Psychologistswanttogobeyondclassicalresearchofthebrain
andhumancognition
Example:– anenduringinterestin‘augmentinghumanintellect’:V.Bush,
Licklider,Engelbart,inturninspiringStuCard,AlanNewell,AlanKay,andmanyothers.
2/13/09 HCIC "Living Lab" 2
TheneedtoestablishHCIasascience– Adoptmethodsfrompsychology– Convenientandfitswellwithproblemsathand
Issuesaroundpersonalcomputing(WIMP)– Dualpurpose:understandnatureofhuman
behaviorandbuildupascienceofHCItechniques.
– GoodExamples:Fitts’Law,ModelsofHumanMemory,CognitiveandBehavioralModeling,InformationForaging
– StuartK.Card,WilliamK.English,andBettyJ.Burr(1978).Evaluationofmouse,rate‐controlledisometricjoystick,stepkeys,andtextkeysfortextselectiononaCRT.Ergonomics,21(8):601–613,1978.
2/13/09 3 HCIC "Living Lab"
Beyondauserinfrontofcomputer– Yetevaluationmethodsmostlystayedthesame– PerceivedCHIpapertemplateforacceptance
Manyproblemsdon’tfitthelaboratoryexperimentalmethodsanymore– Yesterday’sdiscussionaboutLargeDataandHCIwaslargelydrivenby
howHCIevaluationmethodsneedtochangetofitthewild
Thebestexamples:TrendsinSocialComputingandUbiCompforceustothinkaboutnewcontextofuse
2/13/09 HCIC "Living Lab" 4
Old Assumptions New Considerations Single display Multiple displays Knowledge work Games, communication, social apps Isolated worker Collaborative and social groups Stationary location Mobile and stationary Short task durations Short and long tasks, and tasks with
no time boundries Controllable experimental conditions Uncontrollable experimental conditions
2/13/09 5 HCIC "Living Lab"
ThinkaboutresearchthathasbeendoneonUIanimationsorflashingicons.
Fromvisualperception,weknowmotionintheperipheryismorenoticeablethaninthefovealregion[DaVinci].
2/13/09 HCIC "Living Lab" 6
EvaluationssurroundingmanyHCIsystemsforknowledgeworkfocusonproductivityincrease,butwhataboutfactorsforadoption?– Argument:ifnoproductivityincrease,thenadoptionis
irrelevant– Buttheoppositeargumentisjustasright:ifnoadoption,no
amountofproductivityincreaseshownisrelevant!
Academicresearchoftenfocusonproductivityimprovements,increasingtheperceivedgulfbetweentheivorytowerandthetrenches
AnExample:Colorcopierstudies
2/13/09 HCIC "Living Lab" 7
Artificialexperimentalsetupsareonlycapableoftellingusbehaviorsinconstrainedsituations– Ecologicalconsiderations– Hardtogeneralizetonewtaskcontexts(withinterruptions,other
tasks,othergoals,unfocusedattention,moredisplays)– Hardtogeneralizetoothertools,apps– Impossibletoanswerquestionsaboutaggregatebehaviorsofgroups
Exampleproblems:– Adoptionofmobiletechnology
iPhonesinJapan,single‐handedinput[PARC] BestsellingphonesinIndonesiacomeswithacompass[Bell]
– AggregatebehaviorofWikipediaorDelicioususers Bigdataanalysisofeditlogs
2/13/09 8 HCIC "Living Lab"
Wasacomputationalmolecularbiologist Analogy:Justasbiologistsworkonmodelplantsand
genomesinthelab,thistellsusjusthowitbehavesinanisolatedenvironmentundercontrolledconditions,butnothowtheplantwillbehaveintherealworld.
Biologistsdon’tjuststudymodelsinthelab,butinthewildalso.
2/13/09 HCIC "Living Lab" 9
ObservationalStudies– Ethnography– SocialTechnicalDesign– IterativeDesign– DiaryStudies– LongitudinalStudieswithsingleoutcome
Problems:– Samplingfromnon‐normaldistributions– Treatvariationsinsocialcontextsas‘noise’– Whataboutadoption?
MixedMethodsisnowpopulartopartiallyameliorate– Convergentmeasurestellsuswe’regettingclosertothetruth
2/13/09 10 HCIC "Living Lab"
– similartoVentureBusinessmentionedbyDavidMillen
Conductresearchonrealplatformsandservices– Nottoreplacecontrolledlabstudies– Expandourarsenaltocovernewsituations
Someprinciples:– Embeddedintherealworld– Ecologicallyvalidsituations– Embracethecomplexity– Relyonbig‐data‐sciencetoextractpatterns
Notfirsttosuggestthis:– S.Carter,J.Mankoff,S.KlemmerandT.Matthews.Exitingthecleanroom:On
ecologicalvalidityandubiquitouscomputing.HCIJournal,2008– EClass[Abowd],PlaceLab[Intille],PlasmaPoster[ChurchillandNelson],Digital
FamilyPortrait[Rowan,Mynatt]
2/13/09 11 HCIC "Living Lab"
Twodimensions– 1.Whetherthesystemisunderthecontroloftheresearcher– 2.Whetherthestudyisconductedinthelaborinthewild
System Control System Not in Control
Laboratory (1) Build a system, study in the Lab
(2) Adopt a system, study in the Lab
Wild (Real World)
(4) Build a system, release it, study in the Wild
(3) Adopt a system, study in the Wild
2/13/09 12 HCIC "Living Lab"
TraditionalApproach;Numerousexamples FavoredbyCHIreviewers Typicalsituationisthestudyofsomeinteractiontechnique
– Peninput,gestures,perceptionofsomevisualizeddata,readingtasks,mobiletextinput
Typicalmeasuresarequantitativeinnature– performanceintime,performanceinaccuracy,eyetracking,learning
measures,userpreferences
Issues:– Notalwaysecologicallyvalid– Hardtotakeallinteractionsintoaccount– Oftentime‐consuming;eventhoughwethoughtwecoulddoitfast.
2/13/09 13 HCIC "Living Lab"
Hardertofindintheliterature Oftencomparingagainstanoldersystemasbaseline Typicalcaseiscomparisonoftwosystems
– (onewebsitewithanother,onewordprocessorvs.another)– Whichhighlightingfeatureworksbetter– Twotextinputtechniqueonacellphone
Typicalmeasuresaresimilarto(1) Issues:
– Somesimilarissuesto(1)becauseit’sinlab– Systemfeaturenotincontrol,sonotabletocomparefairly,or
isolatethefeature
2/13/09 HCIC "Living Lab" 14
2/13/09 HCIC "Living Lab" 15
http://www.flickr.com/photos/crush_images/3245220458/
Realapplicationsinecologicalvalidsituations Realfindingscanbeappliedtoarunningsystem Impactofresearchismoreimmediate,sincesystemisalready
running Typicalcaseisloganalyticswithlargesubjectpools
– logstudiesofwebsites,realmobilecallingusages,websearchlogs,studiesofWikipediaedits.
Typicalmeasuresarestickiness,amountofactivity,clusteringanalysis,correlationalanalysis
Issues:– Factorsnotincontrol,findingsnotcomparable– Factorscannotbeisolated– Reasonsforfailureisoftenjustguesswork
2/13/09 HCIC "Living Lab" 16
Hypothesis:ConflictiswhatdrivesWikipediaforward. Howtostudythis?
– Tukeyparadigm– Getalargepaper,andplotthedamndata!
– DownloadedallofWikipediaandalloftherevisions– Hadoop/MapReduce,MySQL,etc.
2/13/09 HCIC "Living Lab" 17
2/13/09 18
60%
65%
70%
75%
80%
85%
90%
95%
100%
2001 2002 2003 2004 2005 2006
Perc
enta
ge o
f tot
al e
dits
Article
User
Article Talk
User Talk
Other
Maintenance
HCIC "Living Lab"
Group A
Group B Group C
Group D
Number of users in user group A B C Total
Users with Korean point of view 10 6 0 16
Users with Japanese point of view 1 8 7 16
Neutral or Unidentified 7 3 6 17 2/13/09 19 HCIC "Living Lab"
Mediators
Sympathetic to parents
Sympathetic to husband
Anonymous (vandals/spammers)
2/13/09 20 HCIC "Living Lab"
2/13/09 HCIC "Living Lab" 21
2/13/09 HCIC "Living Lab" 22
2/13/09 23 HCIC "Living Lab"
2/13/09 24 HCIC "Living Lab"
2/13/09 25 HCIC "Living Lab"
Hypothesis:SocialTaggingdoesn’tscaleovertime. Howtostudythis?
– Crawlasmuchtaggingdataaswecan.– Studythenoiseinthesystem.
– 40machinesfor3months
2/13/09 HCIC "Living Lab" 26
Topics
Users Documents
Tags
T1…TnEncodingDecoding
Noise
2/13/09 27 HCIC "Living Lab"
Concepts
2/13/09 HCIC "Living Lab" 28
Source: Hypertext 2008 study on del.icio.us (Chi & Mytkowicz)
29 2/13/09 HCIC "Living Lab"
Similarto(3),practicalforrunningsystems;ecologicallyvalid,impactcanbeimmediate.– Goodforcasesinwhicheconomicsmakessense[Google]– Changestosystemispossible;Factorscanbecontrolled.
TypicalcasemightbeA/Btesting,largesubjectpools Typicalmeasuresarebeingdeveloped
– Impactmeasures.Largevisit#andinterest(measuredbyblogposts?)NewBusinessinquiries?
– Usabilitymeasuresvs.Usefulnessmeasures
Issues:– Effortandresourcerequirementisdroppingbutstillsignificant– Hardforaresearchlabtotakeon
2/13/09 HCIC "Living Lab" 30
2/13/09 HCIC "Living Lab" 31
CollaborativeFiltering
TapestrysystematPARC
GroupLens 6kusers Ecosystemwith
passionateusers
2/13/09 HCIC "Living Lab" 32
2/13/09 HCIC "Living Lab" 33
2/13/09 34
HCIC "Living Lab"
2/13/09 35
HCIC "Living
Lab"
2/13/09 36
HCIC "Living
Lab"
2/13/09 37
Guide
Web
Howto
Tips Help
Tools
Tip
Tricks
Tutorial
Tutorials
Reference
Semantic Similarity Graph
HCIC "Living Lab"
2/13/09 HCIC "Living Lab" 38
2/13/09 HCIC "Living Lab" 39
Interactioncostsdeterminenumberofpeoplewhoparticipate
Surplusofattention&motivationatsmalltransactioncosts
Therefore… Importanttokeep
interactioncostslow
Cost of participation #
Peop
le w
illin
g to
pro
duce
for “
free
”
40 2/13/09 HCIC "Living Lab"
In situ tagging while reading – No new window – Clicking vs typing
Tagging + highlighting
41 2/13/09 HCIC "Living Lab"
Exitingthecleanroom[Carteretal.]– SparseData– CriticalMass– Dealingwithrealisticenvironments– Unobtrusivenessofmonitoring– Ambiguityanderrorofinferencing
2/13/09 HCIC "Living Lab" 43
Evaluationmethodsarein‐separablefromthekindsofscienceandmodelsthatcanbebuildinafield.
Platformadvancesenablerealtechnologyinsertionintorealworldsituationscheaperandmoremanageable.
2/13/09 44 HCIC "Living Lab"
Characteriza7on Models
PrototypesEvalua7ons
ResearchVision:Understandhowsocialcomputingsystemscanenhancetheabilityofagroupofpeopletoremember,think,andreason.
LivingLaboratory:Createapplicationsthatharnesscollectiveintelligencetoimproveknowledgecapture,transfer,anddiscovery.
http://asc‐parc.blogspot.comhttp://[email protected]
WikiDashboard MrTaggy SparTag.us