Brian Fisher

Interactive Arts and Technology & Program in Cognitive Science

Simon Fraser University

Visual Analytics as a Cognitive Science

Cognitive &

Perceptual

Sciences

VisualInformation

Systems

Graphic & Interaction Design

Mathematical & Statistical

Methods

EffectiveSituated

R&D

•“This science must be built on integrated perceptual and cognitive theories that embrace the dynamic interaction between cognition, perception, and action. It must provide insight on fundamental cognitive concepts such as attention and memory. It must build basic knowledge about the psychological foundations of concepts such as ‘meaning,’ ‘flow,’ ‘confidence,’ and ‘abstraction.’ “

•“Illuminating the Path” (IEEE Press)

“The science of analytical reasoning facilitated by interactive

visual interfaces”

Visual Analytics

•Technical, functional & operational drivers

• Availability of massive data & processing

• Perceptually rich interactive displays

• Cognitive performance is key

• Diverse users: Novices, experts, multidiscipline, working in collaboration

•Complex design space with weak metrics

•Design stalls for lack of knowledge sufficient to steer through complex design space

Need for a Scientific Approach

≠ ≠

Diversity of users ranging from novice ⇒ expert collaborating on tasks that require analysis, judgment, and coordination

Bill Buxton/Dave Kasik

•Some cognitive processes are hard-wired-

• Sensory processes, e.g. 3 primary colours

• Performance bottlenecks e.g. STM of 7 +- 2

•Other aspects can change over time, based on experience and knowledge

•Intuitions about architectural processing are usually inaccurate = “metacognitive gap”

Implications of Cognitive

Architecture

•The study of thought, learning, and mental organization, which draws on aspects of psychology, linguistics, philosophy, and computer modelling. (Oxford)

•Cognitive Science Society founded in 1979

Cognitive Science

Cognitive Science Society

•Daniel Bobrow - AI

•Eugene Charniak - AI

•Allan Collins - Psychology

•Edward Feigenbaum - AI

•Charles Fillmore - Linguistics

•Jerry Fodor - Philosophy

•Walter Kintsch - Psychology

•Donald Norman - Psychology

•Zenon Pylyshyn - Psychology

•Raj Reddy - AI

•Eleanor Rosch - Psychology

•Roger Schank - AI

Society Founders

Key figures not at Dallas

AILinguisti

csNeurosci

Philosophy

Psychology

Chomsky ✓ ✓

Miller ✓ ✓ ✓ ✓

Minsky ✓ ✓ ✓

Newell ✓ ✓

Simon ✓ ✓ ✓

•A division of labour?

•Simon’s“nearly decomposable problems”

•Conceptual & methodological “trading zone”?

•Galison: Image and Logic: a Material Culture of Microphysics

•Thagard: Trading Zones in Cognitive Science

What Happens in the Maze?

•“If politics is the art of the possible, research is surely the art of the soluble. Both are immensely practical-minded affairs”.

Sir Peter Medawar

Visual Analytics Science & Technology

"Il n'existe pas une catégorie de sciences auxquelles on puisse donner le nom de sciences appliquées. Il y a la science et les applications de la science, liées entre elles comme le fruit à l'arbre qui l'a porté"

Louis Pasteur

Pure Basic Research(Bohr)

Use-inspired Basic

Research (Pasteur)

Sampling,Description,Taxonomy(Audubon)

Pure Applied Research (Edison)

Quest for Fundamental

Understanding?

No

Yes

Consideration of Use ?

No

Yes

(1822–95)

Pasteur’s Quadrant (Stokes)

Mangle of Practice (Pickering)

Mangle 2 |ˈmøŋgəl| |ˈmaŋg(ə)l| noun

A large machine for ironing sheets or other fabrics,

usually when they are damp, using heated rollers.

chiefly Brit. a machine having two or more cylinders turned by a handle, between which wet laundry is squeezed (to

remove excess moisture) and pressed

CognitionPerceptual

ScienceMethods

SocialScienceMethods

Computation and

Visualization Methods

Graphic & Interaction DesignMethods

Cognitionin the wild

Mangle of Visual Analytics?

•Focused on specific situations:

•Understand processes of cognition as supported by interactive visualization

•Understand communication and distribution of cognition via technical means in organizations and communities

•Incorporate these in design and evaluation of technologies for augmenting cognitive processes

Visual Analytics Mangle

•External representations can support

• Increasing the memory and processing resources available to users

•Reducing the search for information by using visual representations to enhance the detection of patterns

•Engaging perceptual inference operations

•Using perceptual attention mechanisms for monitoring

•Supporting manipulation of information

Cognitive Amplification (Card)

•Design guidelines & evaluation methods

•Perceptual cognition

•Reasoning & problem solving

•Social cognition

•Cognitive linguistics

•Cognitive neuroscience

Cognitive Psychology

•Visual expertise - how do we assess, model, teach, and build for it

• Individual differences, quantitative predictions

•Visuomotor expertise - how do we assess, model, teach, and build for it

•Multimodality

Understand cognition supported by interactive

visualization

Cognitive,PerceptualSciencesMethods

•Unified Theories of Cognition (Newell)

• Models = theories

• Models should explain a range of behaviours

• “You can’t play 20 questions with nature and expect to win”

•Fodor “Modularity of Mind”

•Egon Brunswick “Conceptual Framework of Psychology”

Cognitive Architecture

•Pull key unanswered research questions from field work and conduct lab research to answer them.

•The innovation here is in the integration of HCI variables with measures from cognitive/perceptual psychology, hence greater predictive validity for cognitive tasks.

New Methods

•Cognitive processes emerge as a result of a “close coupling” of mental representations & processes with external representations & processes

•All information is equal-- there is “no privileged access” for internal representation to cognitive processes (Gray)

Cognition & Interactivity

•Focus on human-information discourse• Perceptual & motor learning and

expertise• Tight-loop perception/action

coordination• Cognitive processes as supervisory

control• Temporal coordination “Rhythms” of

events & flow of cognitive processing (individual and group)

• Customization for individual’s “personal equation”

• Tracking & training development of interactive expertise

Cognitive augmentation Engelbart

•VR Design Environment (GMR)

•Fishtank VR air traffic control (Hughes/Raytheon)

•Car interfaces (Nissan)

•Aircraft safety (Boeing)

Perception of depth in VR displays (Fechner)Ability to track targets in moving spaces (Pylyshyn - FINST)Perception of touch on the outer legDiscourse in “pair analytics” (JAT)

Enactive Cog VA Projects

•Build on social science (GT, JAT) approaches to understand organizations and cognitive work practices.

•The innovation here is in the extension of social science to bridge to the perceptual and cognitive science theories that apply to the use of visualization in analytical tasks.

Understand technological distribution of cognition

SocialSciences

•Activity theory (Vygotsky via Engeström)

• Multiple human actors and technology

• The activity is the unit of analysis

• Technology is typically passive, e.g. diagrams

•Psycholinguistics Joint Activity theory (Clark)

• Unit of analysis is the structure of joint actions that make up a joint activity

• Avoids metarepresentational infinite recursion

Distributed Cognition

•“Pair analytics” sessions

• Student visual analyst & trained domain expert collaborate on analytic task

• Student “drives”, expert “navigates”

• Video session & capture screen

•Analysis of session based on knowledge of human cognition & communication

• Analysis bridges socsci & cognitive science (Joint Activity Theory & Grounded Theory)

Dr. Richard Arias-Hernández (Postdoc)Daniel Ha (Doctoral Student)Aaron Smith (Doctoral Student)Andrew, Wade (Masters Student)

Distributed Cognition

Engineering Humans & Organizations

•Focus on technologies, but true visual analytics should have impacts on:

• Selection and training of people for perceptual, cognitive, and motor abilities needed to support human-technological cognitive systems

• Structures of organizations & communities that support technological distributions of cognition

•We certainly do not advocate that business schools, in revising MBA curricula, abandon science. Rather, they should encourage and reward research that illuminates the mysteries and ambiguities of today's business practices. Oddly, despite B schools’ scientific emphasis, they do little in the areas of contemporary science that probably hold the greatest promise for business education: cognitive science and neuroscience.

•Bennis & O’Toole, HBR, May 2005

“How Business Schools Lost Their Way”

•Case studies of technologies designed for specific people, situations and cognitive tasks

•Selected theories and methods from cognitive and social sciences that have proven useful

•Aggregate case studies to infer methods that are successful over a class of applications

Cognitive/Social Oriented Informatics

Graphic & Interaction

Design

Computation & Visualization

•Radical DCog would argue equivalence of actors as well as information

•Distribution of agency across human and non-human actors.• Emergent cognition >> summed

cognition of actors• Clark’s Joint Activity theory may

provide a framework for extending DCog theory to complex heterogenous actor networks.

•Scientific approach might address nearly decomposable sub-systems, not individual actors

Cognitive Systems

•System blends human & computational cognitive processes

•Bound together through high-bandwidth interface of vision/visualization for human-information discourse

• Stream processor, many modular subprocessors working in parallel, a learning system with large variation among individuals in methods, capabilities, and time course of processing

• Scalable visual analysis systems, automatic data analysis and interactive visualization for custom-designed processes for the exploration and analysis of complex information spaces.

Cognitive Systems

•Interdisciplinary events

• VAST and EuroVA, workshops

• HICSS - Business/tech

• Cogsci - Psyc, Neuro, AI, Anthro, Ling, Ed...

• Smart Graphics -interdisc. science/art/tech

•Disciplinary

• VSS - Vision Science

• APS - Psychological Science

•Need for a society, journal?

Conferences & workshops

•Visual Analysis of Massive Data for Decision Support and Operational Management

•Submissions due June 15, 2011

•http://www.hicss.hawaii.edu/hicss_45/apahome45.htm

•Proceedings in IEEE digital library, top 2% of IEEE Conferences in IEEE access

•Participants usually from over 40 countries (approximately 50% non-US).

•2nd in citation ranking among 18 Information Systems conferences

•The Australian Government's Excellence in Research project (ERA) has given HICSS an "A" rating, one of 32 Information Systems conferences so honored out of 241