1 Visualization and Evaluation at Microsoft Research George Robertson, Mary Czerwinski and VIBE...

1

Visualization and Visualization and Evaluation at Microsoft Evaluation at Microsoft

ResearchResearch

George Robertson, Mary Czerwinski George Robertson, Mary Czerwinski and VIBE teamand VIBE team

22

Visualization BenefitsVisualization Benefits

33

Visualization in Microsoft ProductsVisualization in Microsoft Products

Data VisualizationData Visualization• Excel chartingExcel charting

Information VisualizationInformation Visualization• Basic hierarchy visualization – TreeViewBasic hierarchy visualization – TreeView• Microsoft Business SolutionsMicrosoft Business Solutions• BizTalk ServerBizTalk Server

44

Visualization Research Categories Visualization Research Categories Information ManagementInformation Management

• Data Mountain (UIST’98)Data Mountain (UIST’98)• Photo Mountain (WinHEC 2001)Photo Mountain (WinHEC 2001)• DateLens (CHI 2003)DateLens (CHI 2003)• FacetMap (AVI 2006)FacetMap (AVI 2006)• FaThumb (CHI 2006)FaThumb (CHI 2006)

Principles: leverage spatial memory, animation, space-filling for scaling, provide tools for personalization

55

Document Management:Document Management:Data Mountain (UIST’98)Data Mountain (UIST’98)

Subject Layout of 100 Pages

• Size is strongest cue

• 26% faster than IE4

• After 6 months, no performance change

• Images help, but not required

• Faster retrieval when similar pages are highlighted

• Size is strongest cue

• 26% faster than IE4

• After 6 months, no performance change

• Images help, but not required

• Faster retrieval when similar pages are highlighted

EvaluationEvaluation

66

Wanted to test the spatial memory hypothesisWanted to test the spatial memory hypothesis Also wanted to know what the influence of Also wanted to know what the influence of

other factors wereother factors were• Thumbnail imageThumbnail image• Audio cuesAudio cues• Text summariesText summaries

MethodMethod

Gave subjects a “cue” to look for after they Gave subjects a “cue” to look for after they arranged their Data Mountainarranged their Data Mountain

Cue either had text summary, a thumbnail, an Cue either had text summary, a thumbnail, an audio cue or all 3audio cue or all 3

Time to retrieve the right page, number of Time to retrieve the right page, number of “misses” were dependent measures“misses” were dependent measures

After 6 months, had them do it againAfter 6 months, had them do it again• This time, 50% of the trails had the thumbnail This time, 50% of the trails had the thumbnail

images turned off!images turned off!77

88

Calendar Visualization:Calendar Visualization:Datelens (CHI 2003)Datelens (CHI 2003)

With Ben Bederson @ U. With Ben Bederson @ U. MarylandMaryland• Fisheye representation of dates • Compact overviews • User control over the view• Integrated search (keyword) • Enables overviews, fluid

navigation to discover patterns and outliers

• Integrated with outlook


First, prototyped on desktop to perform First, prototyped on desktop to perform formative evaluation but also tested against formative evaluation but also tested against existing UIexisting UI

Next built onto Pocket PCNext built onto Pocket PC Gave to PPC owners for 3 days useGave to PPC owners for 3 days use Performed benchmark tasks with them on 4Performed benchmark tasks with them on 4thth

day, satisfaction ratings over all 4 daysday, satisfaction ratings over all 4 days

99

Benchmark StudyBenchmark Study DateLens v. Microsoft’s Pocket PC 2002™ DateLens v. Microsoft’s Pocket PC 2002™ GoalsGoals

• 11stst iteration of UI with potential users iteration of UI with potential users• to compare its overall usability against an existing to compare its overall usability against an existing

productproduct Mary’s calendar, seeded with artificial calendar Mary’s calendar, seeded with artificial calendar

events, utilized events, utilized

MethodsMethods 11 knowledge workers (5 F) 11 knowledge workers (5 F)

• All experienced PC, not PDA usersAll experienced PC, not PDA users 11 isomorphic browsing tasks on each calendar11 isomorphic browsing tasks on each calendar

• All conditions counterbalancedAll conditions counterbalanced• All tasks had deadline of 2 minutes All tasks had deadline of 2 minutes • Find the dates of specific calendar events (e.g., birthdays)Find the dates of specific calendar events (e.g., birthdays)• Determine how many Mondays a month containedDetermine how many Mondays a month contained• View all bdays for the next 3 monthsView all bdays for the next 3 months

Task times, success rate, verbal protocols, user Task times, success rate, verbal protocols, user satisfaction and preferencesatisfaction and preference

Results—Task TimesResults—Task Times Tasks were performed faster using DateLens, Tasks were performed faster using DateLens,

F(1,8)=3.5, p=.08F(1,8)=3.5, p=.08• Avg=49 v. 55.8 sec’s for the Pocket PCAvg=49 v. 55.8 sec’s for the Pocket PC• Complex tasks significantly harder, p<.01, but Complex tasks significantly harder, p<.01, but

handled reliably better by DateLens (task x calendar handled reliably better by DateLens (task x calendar interaction), p=.04 interaction), p=.04

Results—Task TimesResults—Task TimesCompletion Times across Tasks

0

20

40

60

80

100

120

140

1 2 3 4 5 6 7 8 9 10 11

Task

Ave

rag

e T

ime

(Sec

on

ds)

FishCal

Pocket PC

Task SuccessTask Success Tasks were completed successfully significantly Tasks were completed successfully significantly

more often using DateLens (on average, 88.2% v. more often using DateLens (on average, 88.2% v. 76.3% for the PPC, p<.001. 76.3% for the PPC, p<.001.

In addition, there was a significant main effect of In addition, there was a significant main effect of task, p<.001. task, p<.001.

For the most difficult task (#11), no participant For the most difficult task (#11), no participant using the Pocket PC completed the task successfully. using the Pocket PC completed the task successfully.

Task SuccessTask SuccessPercent Task Completion

010

20304050

607080

90100

1 2 3 4 5 6 7 8 9 10 11

Task

Avg

. P

erce

nt

Co

mp

lete

d

FishCal

Pocket PC

Usability IssuesUsability Issues Many users disliked the view when more than 6 Many users disliked the view when more than 6

months were shownmonths were shown Concerns about the readability of text, needs Concerns about the readability of text, needs

to be customizableto be customizable Wanted more control about how weeks were Wanted more control about how weeks were

viewed (e.g., start with Sunday or Monday?)viewed (e.g., start with Sunday or Monday?) Needed better visual indicators of conflicts for Needed better visual indicators of conflicts for

both calendars, e.g., red highlights and/or a both calendars, e.g., red highlights and/or a “conflicts” filter“conflicts” filter

1818

FacetMap—Faceted Search Results FacetMap—Faceted Search Results of Digital Bitsof Digital Bits

Meant to use metadata of your digital stuff to Meant to use metadata of your digital stuff to aid in browsingaid in browsing

Abstract, scalable, space-fillingAbstract, scalable, space-filling Visual more than textualVisual more than textual Study showed favored over existing Study showed favored over existing

techniques for browsing taskstechniques for browsing tasks

1919

Small SizeSmall Size

2020

Large Size (Wall Display)Large Size (Wall Display)


2121

Wanted to test against textual search UI (existing Wanted to test against textual search UI (existing system)system)

Needed to use both text search and browse at Needed to use both text search and browse at various levels of depthvarious levels of depth• TargetedTargeted: Find the earliest piece of email Gordon : Find the earliest piece of email Gordon

received from Jim Gemmell (text search for received from Jim Gemmell (text search for “Gemmell”). “Gemmell”).

• BrowseBrowse: Name a document that Gordon modified in : Name a document that Gordon modified in the 3the 3rdrd week of May, 2000. week of May, 2000.

Also, needed to test search for different kinds of Also, needed to test search for different kinds of dimension (file type, date, people, etc.)dimension (file type, date, people, etc.)

The Text BaselineThe Text Baseline

2222

ResultsResults

2323

Average Task Times

0

50

100

150

200

250

300

350

Targeted Browse

User Interface

Ave

rag

e T

ask

Tim

es (

Sec

on

ds)

FacetMap

Memex

Question FacetMap Memex

Mental Demand 4.0 (1.8) 4.3 (1.6)

Physical Demand 3.6 (2.1) 3.6 (1.6)

System Response Time 4.8 (1.4) 5.7 (1.1)

Satisfaction 5.6 (1.4) 5.4 (0.8)

Preference over Existing Techniques

4.9 (1.2) 5.2 (1.4)

Browsing Support 5.9 (0.9) 5.9 (0.9)

Text Search Support 5.9 (1.4) 5.3 (0.8)

Aesthetic Appeal 5.3 (1.3) 4.1 (1.5)

2424

Visualization Research Categories Visualization Research Categories Task ManagementTask Management

• Scalable Fabric (WinHEC 2003)Scalable Fabric (WinHEC 2003)• Clipping Lists (summer 2005)Clipping Lists (summer 2005)• Change Borders (summer 2005)Change Borders (summer 2005)

Principles: leverage spatial memory and periphery to reduce clutter and improve glancability• Users stay in the flow of their tasks longer, switch Users stay in the flow of their tasks longer, switch

more optimallymore optimally

2525

Task Management:Task Management:Scalable Fabric (WinHEC 2003)Scalable Fabric (WinHEC 2003)

Beyond MinimizationBeyond Minimization• Manage Windows tasks

using natural human skills• Central focus area• Periphery: windows scaled• Cluster of windows = task• Works on variety of displays• Download available Aug.

2005 – 5000 downloads in 1st 2 months


Similar to TG, users lay out tasksSimilar to TG, users lay out tasks Simulate task switchingSimulate task switching Compare to TaskBarCompare to TaskBar Also, 3 weeks real usage + satisfactionAlso, 3 weeks real usage + satisfaction

2626

2727

Visualization Research Categories Visualization Research Categories Improved Productivity & ReadabilityImproved Productivity & Readability• Clipping Lists and Change BordersClipping Lists and Change Borders

• Principles: remove content of less Principles: remove content of less importance to get more info on the importance to get more info on the screen, reduce occlusion for readabilityscreen, reduce occlusion for readability

Study: compare abstraction techniquesStudy: compare abstraction techniques

Change detectionChange detection• signals when a change has signals when a change has

occurredoccurred Semantic content extractionSemantic content extraction

• pulling out and showing the pulling out and showing the most relevant contentmost relevant content

ScalingScaling• shrunken version of all the shrunken version of all the

contentcontent

Which will most improve Which will most improve multitasking efficiency?multitasking efficiency?

Our DesignsOur Designs

Study DesignStudy Design

no semantic content extraction

semantic content extraction

no change detection

change detection

Comparing TradeoffsComparing Tradeoffs

no semantic content extraction

semantic content extraction

no change detection

+ spatial layout

– no legible content

+ most relevant task info

– detailed visuals / text

change detection

+ spatial layout

+ simple visual cue for change

– limited info

+ most relevant task info

+ simple visual cue for change

User Study: ParticipantsUser Study: Participants

26 users from the Seattle area (10 female)26 users from the Seattle area (10 female)• moderate to high experience using computers and moderate to high experience using computers and

Microsoft Office-style applicationsMicrosoft Office-style applications

User Study: TasksUser Study: Tasks

Four tasks designed to mimic real world tasksFour tasks designed to mimic real world tasks• QuizQuiz - wait for modules to load - wait for modules to load• Uploads - wait for documents to uploadUploads - wait for documents to upload• EmailEmail - wait for quiz answers and upload - wait for quiz answers and upload

task documents to arrive task documents to arrive• PuzzlePuzzle - high-attention task done while waiting - high-attention task done while waiting

QuizQuiz

UploadsUploads

PuzzlePuzzle

User Study SetupUser Study Setup

left monitor right monitor

MeasuresMeasures

Overall performanceOverall performance task durationtask duration

Accuracy of task resumption timingAccuracy of task resumption timing time to resume tasktime to resume task

((e.g., e.g., time between upload finishing & user clicking on upload tool)time between upload finishing & user clicking on upload tool)

Task flowTask flow number of task switchesnumber of task switches

Recognition of windows and reacquisition of taskRecognition of windows and reacquisition of task number of window switches within a tasknumber of window switches within a task

User satisfactionUser satisfaction survey after each trial & the lab sessionsurvey after each trial & the lab session

Results: overall performanceResults: overall performance

Clipping Lists Clipping Lists faster task times faster task timesChange Borders Change Borders no significant improvement no significant improvement

Average Task Times

540

560

580

600

620

640

660

680

700

Average T

im

e in S

econds

SF Clippings + Change

ClippingsSF + Change

Average Task Times

540

560

580

600

620

640

660

680

700

Average T

im

e in S

econds



Average Task Times

540

560

580

600

620

640

660

680

700

Avera

ge T

ime in S

econds



Average Task Times

540

560

580

600

620

640

660

680

700

Avera

ge T

ime in S

econds



Average Task Times

540

560

580

600

620

640

660

680

700

Average T

im

e in S

econds



Average Task Times

540

560

580

600

620

640

660

680

700

Average T

im

e in S

econds



Average Task Times

540

560

580

600

620

640

660

680

700

Avera

ge T

ime in S

econds



Average Task Times

540

560

580

600

620

640

660

680

700

Avera

ge T

ime in S

econds



Results: task resumption timingResults: task resumption timing

Clipping Lists Clipping Lists trend toward more accurate task trend toward more accurate task resumption timingresumption timing

Average Time to Resume Quiz

0

10

20

30

40

50

60

70

80

90

Ave

ra

ge

T

im

e in

S

eco

nd

s




0

10

20

30

40

50

60

70

80

90

Ave

ra

ge

T

im

e in

S

eco

nd

s




0

10

20

30

40

50

60

70

80

90

Ave

rag

e T

ime

in

Se

co

nd

s




0

10

20

30

40

50

60

70

80

90

Ave

rag

e T

ime

in

Se

co

nd

s




0

10

20

30

40

50

60

70

80

90

Ave

ra

ge

T

im

e in

S

eco

nd

s




0

10

20

30

40

50

60

70

80

90

Ave

ra

ge

T

im

e in

S

eco

nd

s




0

10

20

30

40

50

60

70

80

90

Ave

rag

e T

ime

in

Se

co

nd

s




0

10

20

30

40

50

60

70

80

90

Ave

rag

e T

ime

in

Se

co

nd

s



Results: task flowResults: task flow

Clipping Lists Clipping Lists reduced switches reduced switchesChange Borders Change Borders increased switches for SF increased switches for SF

Average Task Switches

40

42

44

46

48

50

52

54

56

Average # of S

witches




40

42

44

46

48

50

52

54

56

Average # of S

witches




40

42

44

46

48

50

52

54

56

Avera

ge #

of

Sw

itches




40

42

44

46

48

50

52

54

56

Avera

ge #

of

Sw

itches




40

42

44

46

48

50

52

54

56

Average # of S

witches




40

42

44

46

48

50

52

54

56

Average # of S

witches




40

42

44

46

48

50

52

54

56

Avera

ge #

of

Sw

itches




40

42

44

46

48

50

52

54

56

Avera

ge #

of

Sw

itches



Results: recognition & reacquisitionResults: recognition & reacquisition

Clipping Lists Clipping Lists reduced window switches reduced window switchesChange Borders Change Borders no significant improvement no significant improvement

Average Quiz Window Switches

16.5

17

17.5

18

18.5

19

19.5

20

20.5

21

Average # of S

witches




16.5

17

17.5

18

18.5

19

19.5

20

20.5

21

Average # of S

witches




16.5

17

17.5

18

18.5

19

19.5

20

20.5

21

Avera

ge #

of

Sw

itches




16.5

17

17.5

18

18.5

19

19.5

20

20.5

21

Avera

ge #

of

Sw

itches




16.5

17

17.5

18

18.5

19

19.5

20

20.5

21

Average # of S

witches




16.5

17

17.5

18

18.5

19

19.5

20

20.5

21

Average # of S

witches




16.5

17

17.5

18

18.5

19

19.5

20

20.5

21

Avera

ge #

of

Sw

itches




16.5

17

17.5

18

18.5

19

19.5

20

20.5

21

Avera

ge #

of

Sw

itches



Average Upload Window Switches

0

24

68

10

1214

1618

20

Average # of S

witches




0

24

68

10

1214

1618

20

Average # of S

witches




0

24

68

10

1214

1618

20

Avera

ge #

of

Sw

itches




0

24

68

10

1214

1618

20

Avera

ge #

of

Sw

itches




0

24

68

10

1214

1618

20

Average # of S

witches




0

24

68

10

1214

1618

20

Average # of S

witches




0

24

68

10

1214

1618

20

Avera

ge #

of

Sw

itches




0

24

68

10

1214

1618

20

Avera

ge #

of

Sw

itches



Results: user satisfactionResults: user satisfaction

Clipping List UIsClipping List UIs rated better than those withoutrated better than those without

Change Border UIsChange Border UIs rated better than those withoutrated better than those without

Preferred UIPreferred UI• 1717 – Clipping Lists + Change Borders– Clipping Lists + Change Borders• 44 – Scalable Fabric + Change Borders– Scalable Fabric + Change Borders• 22 – Clipping Lists– Clipping Lists• 22 – Scalable Fabric – Scalable Fabric

Results SummaryResults Summary

Clipping Lists were most effective for all metricsClipping Lists were most effective for all metrics• Overall performance speedOverall performance speed• Accuracy of task resumption timing Accuracy of task resumption timing (not significant)(not significant)• Task flowTask flow• Recognition of windows & reacquisition of taskRecognition of windows & reacquisition of task• User satisfactionUser satisfaction

Improvements are cumulative, adding up to a Improvements are cumulative, adding up to a sizeable impact on daily multitasking productivitysizeable impact on daily multitasking productivity• Clipping ListsClipping Lists

29 seconds faster on average29 seconds faster on average• Clipping Lists + Change BordersClipping Lists + Change Borders

44 seconds faster on average44 seconds faster on average

Results: semantic content extraction…Results: semantic content extraction…

+ + benefits task flow, resumption timing, and benefits task flow, resumption timing, and reacquisition reacquisition

+ + improves multitasking performance more than improves multitasking performance more than either change detection or scalingeither change detection or scaling

Implication for design of peripheral interfaces Implication for design of peripheral interfaces that support multitasking:that support multitasking:• providing providing enough relevant enough relevant task info is more task info is more

important than very simplistic designsimportant than very simplistic designs

5050

Visualization Research Categories Visualization Research Categories Software VisualizationSoftware Visualization

• FastDASHFastDASH Principles: leverage usage data to expose most Principles: leverage usage data to expose most

important, relevant content to improve important, relevant content to improve discoverabilitydiscoverability

FastDASHFastDASH

5151

•Peripheral display for showing a dev team who has what checked out of a code base•Shows individual team members, what they’ve checked out, what method they’re in, what they’ve changed, where they may be blocked and need help•Display devotes more screen real estate to bigger files in code base


Developed coding scheme to quickly Developed coding scheme to quickly document communication and display usage document communication and display usage behaviors of teambehaviors of team

Code 2 days w/o FastDASHCode 2 days w/o FastDASH Insert FastDASH display on 3Insert FastDASH display on 3rdrd day day Code 2 days w/FastDASH displayCode 2 days w/FastDASH display Pre- and post- satisfaction and situation Pre- and post- satisfaction and situation

awareness surveysawareness surveys

5252

5353

Reduction in Use of Shared Reduction in Use of Shared ArtifactsArtifacts

Increase in Certain Increase in Certain CommunicationsCommunications

5454

5555

Visualization Research Categories Visualization Research Categories Large Information SpacesLarge Information Spaces

• Polyarchy (CHI 2002)Polyarchy (CHI 2002)• PaperLens (InfoVis 2004, CHI 2005)PaperLens (InfoVis 2004, CHI 2005)• Schema Mapper (CHI 2005)Schema Mapper (CHI 2005)• Treemap Vis of Newsgroup CommunitiesTreemap Vis of Newsgroup Communities

Principles: support interactive data exploration Principles: support interactive data exploration through highlighting, transparency, animation through highlighting, transparency, animation and focus + context techniquesand focus + context techniques

5656

Polyarchy Visualization (CHI 2002)Polyarchy Visualization (CHI 2002)

Multiple Intersecting Multiple Intersecting HierarchiesHierarchies• Show multiple hierarchiesShow multiple hierarchies• Show other relationshipsShow other relationships• Search results in contextSearch results in context


Systematically Systematically explored each explored each potential animation potential animation speed and transition speed and transition stylestyle

Also, keystroke Also, keystroke evaluationevaluation

5757

5858

Topic Trends Visualization:Topic Trends Visualization:PaperLens (InfoVis 2004, CHI 2005)PaperLens (InfoVis 2004, CHI 2005)

Understanding a conference• InfoVis (8 years)• CHI (23 years)

Helps understand: • Topic evolution• Frequently published

authors• Frequently cited

papers/authors • Relationship between

authors

Understanding a conference• InfoVis (8 years)• CHI (23 years)

Helps understand: • Topic evolution• Frequently published

authors• Frequently cited

papers/authors • Relationship between

authors


Formative evaluation with target end usersFormative evaluation with target end users Used the information visualization contest Used the information visualization contest

questions to make sure the prototype satisfied questions to make sure the prototype satisfied the requirementsthe requirements

Noted usability issues and redesignedNoted usability issues and redesigned Scaled up for CHI, required massive changesScaled up for CHI, required massive changes

5959

6060

Schema Mapper (CHI 2005)Schema Mapper (CHI 2005) Current techniques fail for large Current techniques fail for large

schemas/mapsschemas/maps

6161

Schema MapperSchema Mapper Emphasize relevant relationshipsEmphasize relevant relationships

• De-emphasize other relationshipsDe-emphasize other relationships


Systematically explored each new feature Systematically explored each new feature addition against shipping product doing addition against shipping product doing mapping tasksmapping tasks

Used real schema map designersUsed real schema map designers

6262

6363

Goals for FutureGoals for Future

Visual representations that:Visual representations that:• Exploit human perception, pattern matching and spatial Exploit human perception, pattern matching and spatial

memorymemory• Summarize and scale to very large datasetsSummarize and scale to very large datasets• Use animated transitions to help retain contextUse animated transitions to help retain context• Scale to a variety of display form factorsScale to a variety of display form factors• Move into collaborative/sharing task domainsMove into collaborative/sharing task domains

Challenges: user-centered design, creative Challenges: user-centered design, creative breakthroughs, need machine learning expertisebreakthroughs, need machine learning expertise

6464

Thanks for your attention!Thanks for your attention!http://research.microsoft.com/research/vibehttp://research.microsoft.com/research/vibe

1 Visualization and Evaluation at Microsoft Research George Robertson, Mary Czerwinski and VIBE...

Documents

Transcript of 1 Visualization and Evaluation at Microsoft Research George Robertson, Mary Czerwinski and VIBE...