Extending Cognitive Architectures with Mental Imagery
AGI-09Scott LathropJohn Laird 1
2
WHY MENTAL IMAGERY?
Cognitive Architectures • Amodal, symbolic representations & computations• No general reasoning with perceptual-based
representations (Barsalou 1999, 2008)
Resulting in… • Ad-hoc reasoning in tasks rich with spatial and visual
properties• “Bolted-on” task-dependent components
What we desire…• “Best of both worlds” multi-representational, task-
independent approach• Functionality• Computational advantage
• Link perceptual-based thought and cognition
3
MENTAL IMAGERY REPRESENTATIONS
Representation Processing Uses ExampleSymbolic
Amodal
Symbolic manipulation
General Reasoning/Control
Qualitative Spatial & Visual Reasoning
feature (letter-B, line)color(letter-B, blue)
feature(box, corner)color(box, yellow)
on(letter-B, box)Quantitative spatial
Amodal/Perceptual-based
(Spatial Imagery)
Mathematical manipulation
Dynamical Systems (motion models)
Spatial Reasoning(Generic Shapes)
Facilitate building visual depictive representation
letter-B location <2,3,-2> orient 0 length 2 height 4 box location <0,0,0> orient 20 length 6 height 4 width 2
Visual depictive
Modality-specific/Perceptual-based
(Visual Imagery)
Mathematical manipulation
Depictive manipulation
Spatial Reasoning(Specific Shapes)
Visual Feature Recognition
4
SCOUT DOMAINEnemy Scout-
2
Enemy Scout-
1
Scout-1
(Agent)
Scout-2(Teammat
e)
Enemy Scout-
3
Actual Situation
Scout-2(Teammate)
Enemy Scout-1
Teammate’s ViewObserved
EnemyScout-2
Scout-1(Agent)
Agent’s View
ObservedEnemy Scout-2
Hypothesized
Enemy Scout
Hypothesized
Enemy Scout
“Key”Terrain
Imagined Situation
SCOUT DOMAIN
5
6
ARCHITECTURE
SVI
• Quantitative spatial • Scene-graph•Viewpoint
Object Map STM
Refresher
Stimulus-Based
Refresh
“Rendered”
Image
Visual-Spatial STMObjec
t SetObjA,ObjB,…
Feature
Set
Spatial
Set
Feature
Set
Object
SetObjA1,ObjA2,…
Spatial
Set
• Visual depictive• Set of 2D bitmaps
Visual Buffer STM
• Shape•Texture• Spatial configuration
Perceptual LTM
“What” Inspectors
“Where” Inspectors
Soar
Symbolic STM•Goals
•Current State
Process Productions
MemorySymbolsVisual Symbols
DataPath
Control Path
Symbolic (Procedural) LTM IMAGERY OPERATORSConstruct
GenerateTransformInspect
TASK OPERATORS
Constructor
Manipulator
8
Pixel-level rewrites (Furnas, 2000)• Production-like rules (LHS/RHS) • Shared image (i.e. working memory)• LHS and RHS are pixel patterns• Conflict resolution scheme (sequencing)• Processing may match other
orientations• Active manipulation of shapes
Contrast with other image processing• Computer vision
o Sentential manipulations o Filters (e.g. Gaussian) where each pixel
is rewritten as a specific function of its neighbors’ values
• Cellular Automatao Finite State Machineso Next state of a cell based on current
state and state of its neighborso Rather than rewriting many local
configurations at once
= “don’t care”
“2x2 reduce
”
“nibble up”
“nibble left”
DEPICTIVE MANIPULATIONS(1 of 2)
9
Takes specific shape of no-go terrain and obstacles (yellow) into account Attention window shift controlled by Soar Pixel rewrite rules sent from Soar to SVI Determine approximate path coverage by overlaying each scout’s view frustum Simulate alternate view orientations to determine “best” path coverage
*
Steep Terrain
All Other
Obstacles
Mark Obstacles
**
**
Layer 0-1
*
Distance Field Flood
Layer 0-1
Layer 1-2
Mark Path Start-
Layer 2
Layer 2-1
Layer 1-0
Layer 1-0
DEPICTIVE MANIPULATIONS(2 of 2)
10
0 50 100
150
200
250
300
350
-1-0.8-0.6-0.4-0.2
00.20.40.6
Soar+SVISoar-SVIObserverTracker
Simulation Time
Info
rmat
ion
RESULTSAMOUNT OF INFORMATION
SUMMARY General reasoning with symbolic and perceptual-
based representations where the reasoning emerges from the combination of the representations
Soar• Symbolic representations and computations• Goals and high-level knowledge• Controls imagery processing
Spatial and Visual Imagery (SVI)• Quantitative spatial and visual depictive representations and
computations• Leverages perceptual mechanisms
Future Work - Push architecture closer to sensory input (Robotics)
11
Top Related