Some Reflections on Some Reflections on Augmented CognitionAugmented Cognition

Eric HorvitzEric Horvitz

ISAT & Microsoft ResearchISAT & Microsoft Research

November 2000November 2000

Toward Augmented CognitionToward Augmented CognitionChallengeChallenge: : Volume of information, overall complexity of Volume of information, overall complexity of defense tasks continue to grow at a rapid pace, in defense tasks continue to grow at a rapid pace, in stark stark contrastcontrast to human cognitive abilities, which remain static: to human cognitive abilities, which remain static:

AttentionAttention MemoryMemory LearningLearning ComprehensionComprehension Sensory bandwidthSensory bandwidth Visualization abilitiesVisualization abilities Judgment and decision makingJudgment and decision making

In contrast, computational capabilities have continued In contrast, computational capabilities have continued to grow rapidlyto grow rapidly

Apply computation to support / augment human cognition

Developing new powers of understanding, Developing new powers of understanding, remembering, and decision making via new remembering, and decision making via new technologies custom-tailored for human—technologies custom-tailored for human—computer collaboration, symbiosiscomputer collaboration, symbiosis

Augmented CognitionAugmented Cognition

Methods, designs that harness computation and Methods, designs that harness computation and explicit knowledge about human limitations to open explicit knowledge about human limitations to open bottlenecks, address biases, deficits in cognition.bottlenecks, address biases, deficits in cognition.

Techniques and interfaces that allow for fluid mixed-Techniques and interfaces that allow for fluid mixed-initiative interaction to solve problems.initiative interaction to solve problems.

Continual background sensing, learning, inference to Continual background sensing, learning, inference to understand trends, patterns, situations relevant to a understand trends, patterns, situations relevant to a user’s context and goals.user’s context and goals.

Modern Psychology: Modern Psychology: Cognition as information processingCognition as information processing

Human cognition: Vast abilities coupled with Human cognition: Vast abilities coupled with highly characterized / characterizable highly characterized / characterizable limitations & bottlenecks limitations & bottlenecks

Mind as an architecture composed of Mind as an architecture composed of different subsystems that form, remember, different subsystems that form, remember, transmit representations of the world.transmit representations of the world.

Work on distinct dimensions of cognition: Work on distinct dimensions of cognition: Attention, memory, learning, concept attainment, Attention, memory, learning, concept attainment, visualization, judgmentvisualization, judgment

Results in Cognitive PsychologyResults in Cognitive Psychology

• Limitations in serial processingLimitations in serial processing

• AttentionAttention

• Concept attainmentConcept attainment

• Learning and memoryLearning and memory

• VisualizationVisualization

• Judgment and decisionsJudgment and decisions

• Collaboration, group decisionsCollaboration, group decisions

• PsychophysicsPsychophysics

• HCI expertise, resultsHCI expertise, results

Understanding of human as resource-limited reasoner,Understanding of human as resource-limited reasoner,

well-characterized bottlenecks, capabilities, biases well-characterized bottlenecks, capabilities, biases

Timing: Is Now the Time to Timing: Is Now the Time to Undertake Such an Effort?Undertake Such an Effort?

Timing: Is Now the Time to Timing: Is Now the Time to Undertake Such an Effort?Undertake Such an Effort?

Computational powerComputational power

• Raw computational resources (CPU, memory)Raw computational resources (CPU, memory)

• ConnectivityConnectivity

• SensingSensing

• Sensor fusionSensor fusion

• Inference and classificationInference and classification

• LearningLearning

• User modelingUser modeling

• Speech recognitionSpeech recognition

• VisionVision

• Interaction, inputInteraction, input

• UI, display designUI, display design

Leaps in computation, memory, algorithmic prowessLeaps in computation, memory, algorithmic prowess

Augmented Cognition EffortsAugmented Cognition Efforts

ExistingExisting

Psychological Psychological

Results on Results on

Cognitive LimitationsCognitive Limitations

TargetTarget

Cognitive TasksCognitive Tasks

MemoryMemory ConceptConcept

AttainmentAttainmentDivided Divided

AttentionAttention

ComputationComputation

Ab

ilities & efficien

ciesA

bilities &

efficiencies

Characterizing inefficiencies, problems, opportunities re: Characterizing inefficiencies, problems, opportunities re: performing critical tasks based on fundamental cognitive performing critical tasks based on fundamental cognitive limitationslimitations

Identify, apply results from cognitive psychology to build Identify, apply results from cognitive psychology to build computation solutionscomputation solutions

Identify new cognitive psychology researchIdentify new cognitive psychology research Identify new joint CS—cognitive psychology effortsIdentify new joint CS—cognitive psychology efforts

ExistingExisting

Psychological Psychological

Results on Results on

Cognitive LimitationsCognitive Limitations

TargetTarget

Cognitive TasksCognitive Tasks

MemoryMemory ConceptConcept

AttainmentAttainmentDivided Divided

AttentionAttention

Ab

ilities & efficien

ciesA

bilities &

efficiencies

Characterizing inefficiencies, problems, opportunities re: Characterizing inefficiencies, problems, opportunities re: performing critical tasks based on fundamental cognitive performing critical tasks based on fundamental cognitive limitationslimitations

Identify, apply results from cognitive psychology to build Identify, apply results from cognitive psychology to build computation solutionscomputation solutions

Identify new cognitive psychology researchIdentify new cognitive psychology research Identify new joint CS—cognitive psychology effortsIdentify new joint CS—cognitive psychology efforts

* * New Cog. Psych. New Cog. Psych. Research Research

* HCI, Aug. Cognition * HCI, Aug. Cognition ResearchResearch

Augmented Cognition EffortsAugmented Cognition Efforts

??

Enhance learning and memory via reminder Enhance learning and memory via reminder systems & methodssystems & methods

Automate specific aspects of problem solving & Automate specific aspects of problem solving & filteringfiltering

Modulate / triage communications Modulate / triage communications

Develop new visualizations, other info. rendering Develop new visualizations, other info. rendering to increase rate of “concept attainment”---raising to increase rate of “concept attainment”---raising effective human-computer bandwidtheffective human-computer bandwidth

Augmented Cognition: OpportunitiesAugmented Cognition: Opportunities

Augmented Cognition: OpportunitiesAugmented Cognition: Opportunities

Support sensory fusion, judgment, action Support sensory fusion, judgment, action under uncertaintyunder uncertainty

Provide vigilant monitoring for situations Provide vigilant monitoring for situations requiring attentionrequiring attention

Monitoring, knowledge about human and Monitoring, knowledge about human and machine errorsmachine errors

Extend abilities to coordinate, collaborate Extend abilities to coordinate, collaborate with other people and systemswith other people and systems

Design multimodal, context-aware systems Design multimodal, context-aware systems for more efficient, natural human—computer for more efficient, natural human—computer collaborationcollaboration

Low-Hanging(?) FruitLow-Hanging(?) Fruit

Information filtering and triageInformation filtering and triage Mixed-initiative interactionMixed-initiative interaction Context-sensitive UI / computingContext-sensitive UI / computing Intelligent remindingIntelligent reminding Managing attention and disruptionManaging attention and disruption Visualization & comprehensionVisualization & comprehension Human-error--aware systemsHuman-error--aware systems Automated sensor fusionAutomated sensor fusion Judgment de-biasing systemsJudgment de-biasing systems

Risks in Building a Strong, Risks in Building a Strong, Valuable ProgramValuable Program

Timing: Too early…?Timing: Too early…? Need more complete understanding of human Need more complete understanding of human

bottlenecks in real world?bottlenecks in real world? Need more sophisticated computational Need more sophisticated computational

procedures?procedures?

Strength of researchStrength of research Rigorous, foundational work may be neededRigorous, foundational work may be needed Poor initial stabs may lead to premature Poor initial stabs may lead to premature

discontinuation, disappointmentdiscontinuation, disappointment

FocusFocus Choosing some foci for early tractionChoosing some foci for early traction Avoiding replication of work in commercial sectorAvoiding replication of work in commercial sector

Several Sample ProjectsSeveral Sample Projects

VistaVista: : Models for limiting, controlling Models for limiting, controlling information based on coarse value of revealed information based on coarse value of revealed information.information.

Handsfree decision support:Handsfree decision support: Fluid Fluid interaction for providing recommendation, interaction for providing recommendation, updates; context-sensitive speech.updates; context-sensitive speech.

LumiereLumiere: : Context- and competence-based Context- and competence-based assistanceassistance

LookoutLookout: : Mixed-initiative interactionMixed-initiative interaction. .

PrioritiesPriorities: : Automated assignment of urgency, Automated assignment of urgency, alerting, communication, caching by value. alerting, communication, caching by value.

Several Sample ProjectsSeveral Sample Projects

Notification Platform:Notification Platform: Attention in HCI, Attention in HCI, notification architectures, UI, psych studies of notification architectures, UI, psych studies of disruption, reminders.disruption, reminders.

Continual computation:Continual computation: Use background Use background computation to forecast goals, attention; guide computation to forecast goals, attention; guide precomputing, caching. Background query, precomputing, caching. Background query, document tracking.document tracking.

Learning information goals:Learning information goals: Learning about a Learning about a user’s goals from behavior, log files.user’s goals from behavior, log files.

Qualia:Qualia: Models of attention, perception in Models of attention, perception in multimedia rendering and communicationmultimedia rendering and communication

Context-aware devices:Context-aware devices: Enhance UI with Enhance UI with implicit gestures, context-sensing.implicit gestures, context-sensing.

Project Drill Down & DemosProject Drill Down & Demos