1

Episodic Memory for Soar AgentsEpisodic Memory for Soar Agents

Andrew NuxollAndrew Nuxoll11 June 200411 June 2004

2

What is Episodic Memory?What is Episodic Memory?

Memories of specific events in our pastMemories of specific events in our past Example: Your last vacationExample: Your last vacation

3

WorkingMemory

Soar-EpMem in ActionSoar-EpMem in Action

Cue

Retrieved

Episodes

A simple example…A simple example…

4

WorkingMemory

Soar-EpMem in ActionSoar-EpMem in Action

Cue

Retrieved

Episodes

New memories are recorded periodicallyNew memories are recorded periodically

5

WorkingMemory

Soar-EpMem in ActionSoar-EpMem in Action

Cue

Retrieved

Episodes

The agent creates a cueThe agent creates a cue

6

WorkingMemory

Soar-EpMem in ActionSoar-EpMem in Action

The cue is matched to episodic memoryThe cue is matched to episodic memory

Cue

Retrieved

Episodes

7

WorkingMemory

Soar-EpMem in ActionSoar-EpMem in Action

Cue

Retrieved

Episodes

The best match is retrieved into WMThe best match is retrieved into WM

8

Benefits of Episodic MemoryBenefits of Episodic Memory

Aids in decision making through predicting the Aids in decision making through predicting the outcome of possible courses of actionoutcome of possible courses of action

A recorded history can be used to answer A recorded history can be used to answer questions about the pastquestions about the past

To help keep track of progress on long-term To help keep track of progress on long-term goalsgoals

Learn from past experience when new Learn from past experience when new time/resources become availabletime/resources become available

Generalize knowledge by comparing multiple Generalize knowledge by comparing multiple events simultaneouslyevents simultaneously

9

Previous WorkPrevious Work

Episodic MemoryEpisodic Memory Psychology – Endel TulvingPsychology – Endel Tulving Cognitive Modeling – Erik AltmannCognitive Modeling – Erik Altmann

Case-Based ReasoningCase-Based Reasoning Continuous CBR – Ram and SantamarContinuous CBR – Ram and Santamarííaa

10

Key Differences:Key Differences:

Episodic Memory ResearchEpisodic Memory Research Architectural implementationArchitectural implementation Domain independentDomain independent

Continuous Case-Based ReasoningContinuous Case-Based Reasoning Qualitative vs. quantitative episode Qualitative vs. quantitative episode

contentcontent Differing scope of match and retrievalDiffering scope of match and retrieval

11

Pilot Implementation: EncodingPilot Implementation: Encoding

Encoding initiation:Encoding initiation: upon significant upon significant change in activation levelschange in activation levels

Episode determination:Episode determination: hand selected and hand selected and domain specificdomain specific

Feature selection:Feature selection: hand selected and hand selected and domain specificdomain specific

12

Pilot Implementation: StoragePilot Implementation: Storage

Episode structure:Episode structure: episodes are stored as episodes are stored as Soar productionsSoar productions

Episode dynamics:Episode dynamics: none none

13

Pilot Implementation: RetrievalPilot Implementation: Retrieval

Retrieval initiation:Retrieval initiation: deliberate retrieval in an deliberate retrieval in an agent-selected substate agent-selected substate

Cue determination:Cue determination: agent-selected data agent-selected data Retrieval:Retrieval: exact match exact match Retrieved episode representation:Retrieved episode representation: direct direct

modification of the agent-selected substatemodification of the agent-selected substate Retrieval meta-data:Retrieval meta-data: unique sequential id (to unique sequential id (to

provide an idea of temporal order)provide an idea of temporal order)

14

Pilot Implementation: IssuesPilot Implementation: Issues

Exact match led to encoding specificity Exact match led to encoding specificity issuesissues

Problems from overwriting the sub-stateProblems from overwriting the sub-state RecursionRecursion Spurious operator proposalsSpurious operator proposals Requires that agent create a sub-state to Requires that agent create a sub-state to

do a retrievaldo a retrieval Domain dependentDomain dependent

15

Current Implementation: ChangesCurrent Implementation: Changes

Partial match over a separate episodic Partial match over a separate episodic memorymemory Memories are no longer stored as rulesMemories are no longer stored as rules

Use of an architecture-specified buffer for Use of an architecture-specified buffer for query and retrieval (analogous to the ^io query and retrieval (analogous to the ^io link)link)

16

Current Implementation: EncodingCurrent Implementation: Encoding

Encoding initiation:Encoding initiation: one episode per agent one episode per agent actionaction

Episode determination:Episode determination: all of working all of working memory(!)memory(!)

Feature selection:Feature selection: the entire episode can the entire episode can affect retrievalaffect retrieval

17

Current Implementation: StorageCurrent Implementation: Storage

Episode structure:Episode structure: episodes are stored in episodes are stored in an internal data structurean internal data structure

Episode dynamics:Episode dynamics: still none still none

18

Current Implementation: RetrievalCurrent Implementation: Retrieval

Retrieval initiation:Retrieval initiation: cue is constructed in an cue is constructed in an architecture-specified bufferarchitecture-specified buffer

Cue determination:Cue determination: agent selected data agent selected data Retrieval:Retrieval: exact match exact match Retrieved episode representation:Retrieved episode representation: the episode is the episode is

recreated in an architecture-specified bufferrecreated in an architecture-specified buffer Retrieval meta-data:Retrieval meta-data: agent can retrieve the next agent can retrieve the next

memory in temporal sequence memory in temporal sequence

19

Working Memory ActivationWorking Memory Activation

Extension of the memory decay work by Extension of the memory decay work by Ron ChongRon Chong

Reimplementation by Michael James:Reimplementation by Michael James: Includes all of working memory Includes all of working memory Improvements in efficiencyImprovements in efficiency

20

Activation & MatchingActivation & Matching

Problem:Problem: All WMEs in an episode are All WMEs in an episode are weighted equallyweighted equally

Core Idea:Core Idea: The activation level of WMEs The activation level of WMEs indicates their relevance to current taskindicates their relevance to current task

Implementation:Implementation: Use the activation levels Use the activation levels of the WMEs in the episode to bias the of the WMEs in the episode to bias the matchmatch

21

Evaluation using EatersEvaluation using Eaters

Pac-Man-likePac-Man-like Two types of foodTwo types of food

Bonus food (10 pts)Bonus food (10 pts) Normal food (5 pts)Normal food (5 pts)

22

Create a memory cue (input-link + proposed direction)Create a memory cue (input-link + proposed direction)

East

South

North

Evaluate moving in each available directionEvaluate moving in each available direction

An Episodic Memory EaterAn Episodic Memory Eater

EpisodicRetrieval

Retrieve the best matching memoryRetrieve the best matching memory

RetrieveNext Memory

Retrieve the Retrieve the nextnext memory (in temporal order) memory (in temporal order)Use the change in score to evaluate the proposed actionUse the change in score to evaluate the proposed action

Move North = 10 points

23

Initial ResultsInitial Results

24

Problem #1: I-Support MaskingProblem #1: I-Support Masking

Problem:Problem: Testing an Testing an i-supported WME i-supported WME provides no provides no activation boostactivation boost

Solution = Pay it Solution = Pay it Backward:Backward: Testing Testing an i-supported an i-supported WMEs boosts the WMEs boosts the activation level of its activation level of its “set of o-support”“set of o-support”

25

Problem #2: New WME MaskingProblem #2: New WME Masking

Problem:Problem: A new WME starts at a fixed A new WME starts at a fixed activation levelactivation level

Solution = Pay it Forward:Solution = Pay it Forward: Activation of Activation of newly created WMEs is based upon those newly created WMEs is based upon those WMEs which were tested to create itWMEs which were tested to create it

26

ResultsResults

27

Current Challenge: PerformanceCurrent Challenge: Performance

28

NuggetsNuggets CoalCoal

Domain independent, Domain independent, architectural architectural implementation implementation

Performance issuesPerformance issues Still only tested in a Still only tested in a

single domainsingle domain