Interacting with Visualizations

Ware Chapter 10

University of Texas – Pan AmericanCSCI 6361, Spring 2014

Interacting with Visualizations - IntroductionThe very big picture

• Best visualizations support productive interaction

– Interactive visualizations– Not merely static representations of data

• Though certainly has its place– Allows, e.g.,

• Inspection of underlying data from the visualization

• Transformation of data• Filtering – removal of data by some

criteria– E.g., visual analytics systems we have

seen clearly demonstrate use of highly interactive systems, indeed, across visual mappings

• E.g., “Overview first, zoom and filter, then details on demand”

– Shneiderman, 1996 (at class site)– Though in fact may see interesting detail,

zoom out, find others, zoom in, …

VxInsight, Sandia Labs

Recall, Amplifying CognitionNorman, 1993

• Humans think by interleaving internal mental action with perceptual interaction with the world

• This interleaving is how human intelligence is expanded– Within a task (by external aids)– Across generations (by passing on techniques)

• External graphic (visual) representations are an important class of external aids

• Don Norman is an influential cognitive scientist– The power of the unaided mind is highly overrated. Without external aids, memory,

thought, and reasoning are all constrained. But human intelligence is highly flexible and adaptive, superb at inventing procedures and objects that overcome its own limits. The real powers come from devising external aids that enhance cognitive abilities. How have we increased memory, thought, and reasoning? By the invention of external aids:

– It is things that make us smart. (Norman, 1993, p. 43)

• External Cognition

Introduction and Overview

• Visualization as an “internal interface”– Interface between human and computer in a man-machine problem-solving system

• Computer-based information system supports data gathering, calculation, and analysis• Augments investigator’s working memory

– Provides visual markers for concepts– Reveals structural relationships between problem components

• Some models of visualization – different takes on the same thing!– Overview, zoom, filter, details (Shneiderman)– Visualization Pipeline North (from Card et al.)– Knowledge crystalization (Card et al.)– Ware– Model human processor (Card et al.)

• Motor processor – Ex: Fitts’ law

• Viewing information spaces– Distortion techniques, fisheye views

• Navigation and Exploration

Example: “Overview, zoom and filter, details on demand” - Shneiderman

• VxInsight demonstrates:– “Overview, zoom and filter, details on

demand”– Saw earlier when talking about text

representations (visual mappings)– Again, visual analytics systems provide

• Developed by Sandia Labs to visualize databases

• “Elements of database can be “anything”

– For IV “abstract”– e.g., document relations, company profiles

• Example screens show grant proposals

– Shows interactive capabilities

VxInsight: Overview

vvv

VxInsight

• Interaction paradigm (Shneiderman):– Overview– Zoom– Filter– Details on demand– Browse– Search query

• Or (Ware) …– Lowest level

• Data manipulation loop

– Intermediate • Exploration and navigation loop

– Highest• Problem-solving loop

VxInsight - Overview

• Interaction paradigm

– Overview– Zoom– Filter– Details on

demand– Browse– Search query

VxInsight - Zoom

• Interaction paradigm

– Overview– Zoom– Filter– Details on

demand– Browse– Search

query

VxInsightv - Details

• Interaction paradigm

– Overview– Zoom– Filter– Details on

demand– Browse– Search

query

VxInsightv - Query

• Interaction paradigm

– Overview– Zoom– Filter– Details on

demand– Browse– Search

query

Recall, Visualization Pipeline:Or, another take on interaction: Mapping Data to Visual Form

• Most fundamentally – Visualizations are: – “adjustable mappings from data to visual form to human perceiver”

• Series of data transformations ( )– Multiple chained transformations

– Human adjusts the transformations - interaction

• Entire pipeline comprises an information visualization

RawInformation

VisualFormDataset Views

User - Task

DataTransformations

VisualMappings

ViewTransformations

F F -1

Interaction

VisualPerception

Visualization Pipeline: Human might adjust any of the visualization Stages

• Data transformations (rarely):– Map raw data (idiosynchratic form) into data tables (relational descriptions

including metatags)

• Visual Mappings (sometimes):• E.g., table to graph– Transform data tables into visual structures that combine spatial substrates,

marks, and graphical properties

• View Transformations (very often):• E.g., zooming, …, changing viewpoint– Create views of Visual Structures by specifying graphical parameters such as

position, scaling, and clipping

RawInformation

VisualFormDataset Views

User - Task

DataTransformations

VisualMappings

ViewTransformations

F F -1

Interaction

VisualPerception

Ware: Interactive Visualization: Interlocking Feedback Loops – Quick Look

• Interactive visualization – Process made up of interlocking feedback loops

• Lowest level: Data manipulation loop– Objects selected and moved– Relies on eye-hand coordination– Requires delay-free interaction

• Intermediate: Exploration & navigation loop– User finds way in large visual space– Searching a large data space part by part– Building a cognitive map of the data/simulation

• Highest: Problem-solving loop– Forming and testing hypotheses about data– Refines hypotheses through augmented visualization– Repeat through cycles, revising or replacing visualization

• New data added, problem reformulated, possible solutions identified

– Visualization as external representation of problem• Extension of cognitive process

Exploration

and Navigation

Problem Solving

DataManipulation

Interactive VisualizationRecall, Problem Solving, Cognitive Amplification, Knowledge Crystallization, (Card et al.)

• Knowledge crystallization: Gather knowledge, make sense of it, use it in task

Task

OverviewZoomFilterDetails BrowseSearch query

ReorderClusterClassAveragePromoteDetect patternAbstract

ExtractCompose

Read factRead comparisonRead patterManipulateCreateDelete

Task operations

Instantiate

Search for schema

Forage for data

Instantiate schema

Problem-solve

Write, decide, or act

Again, Ware’s Interlocking Feedback Loops

• Interactive visualization – Process made up of interlocking feedback loops

• Lowest level: Data manipulation loop– Objects selected and moved– Relies on eye-hand coordination– Requires delay-free interaction

• Intermediate: Exploration & navigation loop– User finds way in large visual space– Searching a large data space part by part– Building a cognitive map of the data/simulation

• Highest: Problem-solving loop– Forming and testing hypotheses about data– Refines hypotheses through augmented visualization– Repeat through cycles, revising or replacing visualization

• New data added, problem reformulated, possible solutions identified

– Visualization as external representation of problem• Extension of cognitive process

Exploration

and Navigation

Problem Solving

DataManipulation

1) Interacting Feedback Loops, and 2) Knowledge Crystallization, …

• Different time spans

• Problem Solving - outer– Longest time

• Exploration and Navigation

– Primary use of data and information visualizations

– Occurs for all elements of problem solving, knowledge crystallization

• Data Manipulation– Motor, etc.– Again, for all element

of exploration and navigation

Forage for data

Write, decide, or act

Problem-solve

Instantiate schema

Search for schema

Task

Exploration and Navigation

Data Manipulation

• Knowledge crystallization: Gather knowledge, make sense of it, use it in task

Interacting Feedback Loops – Another WayWare’s account with “gear” metaphor

• As “gears” …

Exploration

and Navigation

Problem Solving

DataManipulation

Problem Solving(knowledge crystalization)

Exploration and Navigation

Data Manipulation

Lowest Level: Data Manipulation Loop

Exploration

and Navigation

Problem Solving

DataManipulation

Lowest Level: Data Manipulation Loop

• Visual-Manual Control Loop

• Very carefully studied, for example …

– Choice reaction time: Hick-Hyman law • Reaction time = a + b log2 (C)

– 2D positioning and selection: Fitts’ law – quickly, more later• Part of ISO standard 9214-9

– Protocols for evaluating user performance and comfort when using pointing devices with visual display terminals

• Selection time = a + b log2 (D/W + 1.0)• Hitting smaller targets further away is harder• Adding latency severely increases difficulty• Fitts’ law, including lag

– Mean time = a + b (Human Time + Machine lag) log2 (D/W + 1.0)

– Control compatibility is important• Offset and scale is easy to deal with; rotation is hard

– Reaction time in making choices• >= 160 msec per doubling of the numbers of choices• Faster if allowed to make mistakes

Exploration

and Navigation

Problem Solving

DataManipulation

Model Human Processor + AttentionRecall

• A “useful” big picture - Card et al. ’83 plus attention– Senses/input f(attention, processing) motor/output– Notion of “processors”

• Purely an engineering abstraction

• Detail next

Model Human Processor + Attention

• Sensory store– Rapid decay “buffer” to hold

sensory input for later processing

• Perceptual processor– Recognizes symbols, phonemes– Aided by LTM

• Cognitive processor– Uses recognized symbols– Makes comparisons and

decisions– Problem solving– Interacts with LTM and WM

• Motor processor– Input from cog. proc. for action– Instructs muscles– Feedback

• Results of muscles by senses

• Attention– Allocation of resources

Model Human ProcessorRecall

• Card et al. ’83

• An architecture with parameters for cognitive engineering …

– Will see visual image store, etc. tonight

• Memory properties– Decay time: how long memory lasts– Size: number of things stored– Encoding: type of things stored

Model Human ProcessorMotor Processor

• Motor processor– M = 70 (range 30-70)– For repetitive tasks without

feedback

• Tasks with feedback involve all:– Perceptual processor– Cognitive processor– Motor processor

Motor Processing

• Motor processor can operate in two ways:

• 1. Open-loop control

– Motor processor runs a program by itself – no feedback about correctness

– Maximum rate, cycle time is M = Tmotor ~ 70 ms

– Experiment: Scribble without looking and trying to stay in lines

• 2. Closed-loop control– Experiment: Looking at lines, draw within the lines

– Muscle movements (or their effect on the world) are perceived by cognitive system and compared with desired result

– Cycle time is Tprocess + Tcognitive + Tmotor ~ 240 ms

Fitts’s Law - demo

• Fitts’s Law– Fundamental law of human

sensory-motor system• Fitts, P. M. (1954). The information c

apacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 47, 381-391.

– E.g., for direct (reach) and mouse use

– Demo: • Best – won’t run on class box• http://www.tele-actor.net/fitts/index.html

– Demo:• OK – no line plotted• http://fww.few.vu.nl/hci/interactive/fitts/

Fitts’s Law - demo

• Fitts’s Law– Fundamental law of human sensory-motor system

• “tele-actor” results from demo:

Fitts’s Law

• Fitts’s Law– Fundamental law of human sensory-motor system– E.g., for direct (reach) and mouse use– The time to acquire a target is a function of distance to and width (size) of target

• T = f (D, S)

• Time T to move your hand to a target of size S at distance D away:– T = ReactionT + MotorT

= a + b * log2 (2 * D/S)

– Depends only on index of difficulty log(2D/S)

Explananation of Fitts’s Law

• Moving hand to a target is closed-loop control– Vs. open-loop control we saw for Card et al. model

• Each (correction) cycle covers remaining distance D, with error εD– Smaller correction in position as get closer

• (because there is less distance with which to correct)

– Slower velocity • (because don’t go so fast with shorter distance)

Implications of Fitts’s Law

• Buttons, etc. should be reasonable size; – hard to click small targets.

• Edges and corners of the computer display are easy to reach– Mac single menubar better than multiple Windows menubars– Also, pointer is "caught" at the edges

• Popup menus can usually be opened faster than pull-down menus– User avoids movement

• Pie menu items are typically selected faster than linear menu items– Small distance from the center of the menu – Wedge-shaped target areas are large

Power Law of Practice

• Time to do a task decreases with practice

– Obviously– Involves all of perceptual-cognitive-

motor system

• Time Tn to do a task the nth time:– Decaying exponential rate

– Tn = T1n α

– α is typically 0.2-0.6

• Example:– Novices get rapidly better at task with

practice, but performance “levels off”– Though still increasing performance

Intermediate Level: Exploration, View Refinement and Navigation

Exploration

and Navigation

Problem Solving

DataManipulation

Intermediate Level:

Exploration, View Refinement and Navigation

• View navigation important when data space is too large to fit on screen

– Complex problem – Considers theories of pathfinding and map use, cognitive spatial

metaphors, direct manipulation, visual feedback

• Basic navigation control loop (below)– Left is human – cognitive and spatial model with which user

understands data space and progress through it• Maintaining data space for some time may become encoded in long-

term memory

– Right is system – visualization may be updated and refined from data mapped into spatial model

• Includes:– 3D Locomotion and viewpoint control– Pathfinding– Focus + context

Exploration

and Navigation

Problem Solving

DataManipulation

3D Locomotion and Viewpoint ControlNavigation in 3D

• Displaying data elements so looks like 3D landscape, vs. flat map, often used

– Follows from Gibsonian orientation• Affordances• Properties of the world perceived in terms of potential for

action (physical model, direct perception)• Problem with generalization to user interfaces/interaction• Nevertheless, important and influential

– Have examined depth cues

• Embed objects in space, navigate space– Flying viewpoint through the data space– Constrain user to useful parts of the space to reduce

cognitive load of navigation• Surface of the ground• Walkways within power plant• Particular paths of interest

• Examples– Web browser: Harmony– Clustering of text, Wise et al.

3D Locomotion and Viewpoint Control: Spatial Metaphors

• Evaluation– Exploration and Explanation– Cognitive and Physical Affordance– Task 1: Find areas of detail in the scene– Task 2: Make the best movie– 3D environments: Hallway, extended terrain, closed object.

• World-in-hand– Good for discrete objects– Poor affordances for looking scale changes – detail– Problem with center of rotation when extended scenes

• Eye-in-hand– Easiest under some circumstances– Poor physical affordances for many views– Subjects sometimes acted as if model were actually present

• Walking

• Flying vehicle control– Hardest to learn but most flexible– Non-linear velocity control– Spontaneous switch in mental model– The predictor as solution

Virtual scene

TreadmillController

c

Virtual scene

6 df HandleController

a

Virtual scene

6 df HandleController

b

Virtual scene

JoystickController

d

3D Locomotion and Viewpoint Control: Wayfinding, Cognitive, and Real Maps

• Worldlets– Can be rotated to facilitate recognition

Frames of ReferenceEgocentric, Exocentric

• Use of maps implies ability to apply another perspective

– To physical, • e.g., road map (view from above),

– Or abstract– … another frame of reference

• Egocentric – view from user

• Exocentric– View from outside the user– Road map just one of many

exocentric view

• Movement of body (vs. eyes) affects orientation most

– Pan, tilt, …, but not rotation, so dof constrained in practice

Frames of ReferenceTethered view, world view

• Various views illustrated

Mutiple Simultanous Views

• Represent data space in different forms in different views

• E.g., “spiral calendar”

Focus, Context, and Scale

• Saw this earlier, here, in Ware

Focus, Context, and Scale

• Problem of finding detail in larger context– Again, spatial navigation– Wayfinding problem may be considered as

discovering specific objects in a larger context

• Addressed by multiple views at differing spatial scales

– Movement between views at different scales (and frames of reference)

– Changing spatial scale• E.g., overview + detail

• Addressed, also, by changing structural scale

– E.g., collapsing lines of code in display of software systems

Focus, Context, and Scale: Overview and Detail

• Fred Brooks’ GRIP project at UNC

– Molecular structure solution, docking

– Architectural walkthrough

• Users always going from detail to overview

– Then overview to detail…– Then detail to overview…

• Options– Provide display of both– Provide easy, non-jarring

switch between them

• Multiple-Window Zoom with Callouts …

Focus+Context: Fisheye Views, 1

• Detail + Overview – Keep focus, while remaining aware

of context

• Fisheye views– Physical, of course, also ..– A distance function. (based on

relevance)– Given a target item (focus)– Less relevant other items are

dropped from the display– Classic cover

• New Yorker’s idea of the world

Focus+Context: Fisheye Views, 2

• Detail + Overview – Keep focus while remaining aware of context

• Fisheye views– Physical, of course, also ..– A distance function. (based on relevance)– Given a target item (focus)– Less relevant other items are dropped from

the display – Or, are just physically smaller – distortion

Distortion Techniques, Generally

• Distort space = Transform space– By various transformations

• “Built-in” overview and detail, and landmarks– Dynamic zoom

• Provides focus + context– Several examples follow

• Spatial distortion enables smooth variation

Focus + Context, 1

• Fisheye Views• Keep focus while remaining aware of the context• Fisheye views:

– A distance function (based on relevance)– Given a target item (focus)– Less relevant other items are dropped from the display.

• Demo of Fisheye Menus:– http://www.cs.umd.edu/hcil/fisheyemenu/fisheyemenu-demo.shtml

Focus + Context, 2

• Bifocal Lens– Database navigation: An Office Environment for the Professional by R. Spence and M.

Apperley

Focus + Context, 3

• Distorted Views– The Table Lens: Merging Graphical and Symbolic Representations in an Interactive

Focus + Context Visualization for TabularInformation by R. Rao and S. K. Card– A Review and Taxonomy of Distortion Oriented Presentation Techniques by Y. K.

Leung and M. D. Apperley

Focus + Context, 4

• Distorted Views– Extending Distortion Viewing from 2D to 3D by M. Sheelagh, T. Carpendale, D. J.

Cowperthwaite, F. David Fracchia

Magnification and displacement:

Focus + Context, 5

• Alternate Geometry– The Hyperbolic Browser: A Focus + Context

Technique for Visualizing Large Hierarchies by J. Lamping and R. Rao

• Demo

Other Navigation Techniques: GeoZui3D, Zooming + 2 dof rotations

• Translate point on surface to center

• Then scale

• Or translate and scale

View Refinement and Navigation(optional, from 2nd ed.)

• Transparency: – When there is the perception of

direct contact with the data, the interface becomes transparent

– Big idea in interfaces– Temporal feedback rapid (< 1/10

second)– Response is compatible with

interaction method

• Interactive adjustment of ranges– Zoom in on data area of interest– Sometimes nonlinear mapping

brings area of interest into range where patterns are easy to see (logarithmic)

Interaction vs. Animation

• Ware comments:

• Exploration (interaction) vs. Presentation (animation)– Flexibility vs. Efficiency

• Active vs. Passive Participation– Immediacy of response and engagement– Control promotes understanding

• Person moving learns more than partner watching• Active control increases sense of presence

End

• .