Ontologies Reasoning Components Agents Simulations Architectural Patterns for Agents Jacques Robin.

OntologiesReasoningComponentsAgentsSimulations

Architectural Patterns for AgentsArchitectural Patterns for Agents

Jacques Robin

Agents’ Internal ArchitecturesAgents’ Internal Architectures

Reflex agent (purely reactive) Automata agent (reactive with state) Goal-based agent Planning agent Hybrid, reflex-planning agent Utility-based agent (decision-theoretic) Layered agent Adaptive agent (learning agent)

Cognitive agent Deliberative agent

Reflex AgentReflex Agent

En

viro

nm

en

t

Sensors

Effectors

RulesPercepts Action

A(t) = h(P(t))

Remember … Remember …

Reasoning

AgentAgent

En

viro

nm

en

t

Sensors

Effectors

Goals

PerceptInterpretation:

I = f(P)

Action Choice:A = g(I,O)

A

P

So?So?

Goals

Percept Interpretation: I = f(P)

Action Choice:A = g(I,O)

En

viro

nm

en

t

Sensors

Effectors

RulesPercepts Action

A(t) = h(P(t))

A

P

Reflex AgentReflex Agent

Principle: Use rules (or functions, procedures) that associate directly percepts

to actions ex. IF speed > 60 THEN fine ex. IF front car’s stop light switches on THEN brake

Execute first rule which left hand side matches the current percepts Wumpus World example

IF visualPerception = glitter THEN action = pick see(glitter) do(pick) (logical representation)

Pros: Condition-action rules is a clear, modular, efficient representation

Cons: Lack of memory prevents use in partially observable, sequential, or

non-episodic environments ex, in the Wumpus World a reflex agent can’t remember which path

it has followed, when to go out of the cavern, where exactly are located the dangerous caverns, etc.

Automata AgentAutomata Agent

Enviro

nm

ent

Sensors

Effectors

(Past and) CurrentEnviroment Model

Percept InterpretationRules:percepts(t) model(t) model’(t)

Action ChoiceRules:model’’(t) action(t), action(t) model’’(t) model(t+1)

Model UpdateRegras: model(t-1) model(t) model’(t) model’’(t)

Goals

Automata AgentAutomata Agent

Rules associate actions to percept indirectly through the incremental construction of an environment model (internal state of the agent)

Action choice based on: current percepts + previous percepts + previous actions +

encapsulated knowledge of initial environment state Overcome reflex agent limitations with partially observable,

sequential and non-episodic environments Can integrate past and present perception to build rich

representation from partial observations Can distinguish between distinct environment states that are

indistinguishable by instantaneous sensor signals Limitations:

No explicit representation of the agents’ preferred environment states

For agents that must change goals many times to perform well, automata architecture is not scalable (combinatorial explosion of rules)

Automata Agent Rule ExamplesAutomata Agent Rule Examples

Rules percept(t) model(t) model’(t) IF visualPercept at time T is glitter

AND location of agent at time T is (X,Y)THEN location of gold at time T is (X,Y)

X,Y,T see(glitter,T) loc(agent,X,Y,T) loc(gold,X,Y,T).

Rules model’(t) model’’(t) IF agent is with gold at time T

AND location of agent at time T is (X,Y)THEN location of gold at time T is (X,Y)

X,Y,T withGold(T) loc(agent,X,Y,T) loc(gold,X,Y,T).

Automata Agent Rule ExamplesAutomata Agent Rule Examples

Rules model(t) action(t) IF location of agent at time T = (X,Y)

AND location of gold at time T = (X,Y) THEN choose action pick at time T

X,Y,T loc(agent,X,Y,T) loc(gold,X,Y,T) do(pick,T)

Rules action(t) model(t) model(t+1) IF choosen action at time T was pick

THEN agent is with gold at time T+1 T done(pick,T) withGold(T+1).

(Explicit) Goal-Based Agent(Explicit) Goal-Based Agent

Enviro

nm

ent

Sensors

Effectors

(Past and) CurrentEnvironment Model

Percept InterpretationRules: percept(t) model(t) model’(t)

Action ChoiceRules: model’’(t) goals’(t) action(t) action(t) model’’(t) model(t+1)

Model UpdateRules: model(t-1) model(t) model’(t) model’’(t)

Goal UpdateRules: model’’(t) goals(t-1) goals’(t) Goals

(Explicit) Goal-Based Agent(Explicit) Goal-Based Agent

Principle: explicit and dynamically alterable goals Pros:

More flexible and autonomous than automata agent Adapt its strategy to situation patterns summarized in its goals

Limitations: When current goal unreachable as the effect of a single action,

unable to plan sequence of actions Does not make long term plans Does not handle multiple, potentially conflicting active goals

Goal-Based Agent Rule ExamplesGoal-Based Agent Rule Examples

Rule model(t) goal(t) action(t) IF goal of agent at time T is to return to (1,1) AND agent is in (X,Y) at time T AND orientation of agent is 90o at time T AND (X,Y+1) is safe at time T AND (X,Y+1) has not being visited until time T AND (X-1,Y) is safe at time T AND (X-1,Y) was visited before time T THEN choose action turn left at time T X,Y,T, (N,M,K goal(T,loc(agent,1,1,T+N)) loc(agent,X,Y,T) orientation(agent,90,T) safe(loc(X,Y+1),T) loc(agent,X,Y+1,T-M) safe(loc(X-1,Y),T) loc(agent,X,Y+1,T-K)) do(turn(left),T)

Y+1

ok

Yv ok

A

X-1 X


Rule model(t) goal(t) action(t) IF goal of agent at time T is to find gold AND agent is in (X,Y) at time T AND orientation of agent is 90o at time T AND (X,Y+1) is safe at time T AND (X,Y+1) has not being visited until time T AND (X-1,Y) is safe at time T AND (X-1,Y) was visited before time T THEN choose action forward at time T X,Y,T, (N,M,K goal(T,withGold(T+N)) loc(agent,X,Y,T) orientation(agent,90,T) safe(loc(X,Y+1),T) loc(agent,X,Y+1,T-M) safe(loc(X-1,Y),T) loc(agent,X,Y+1,T-K)) do(forward,T)

Y+1

ok

Yv ok

A

X-1 X


Rule model(t) Rule model(t) goal(t) goal(t) goal’(t) goal’(t)//If the agent reached it goal to hold the gold, //then its new goal shall be to go back to (1,1) IF goal of agent at time T-1 was to find gold AND agent is with gold at time T THEN goal of agent at time T+1 is to be in location (1,1) T, (N goal(agent,T-1,withGold(T+N)) withGold(T) M goal(agent,T,loc(agent,1,1,T+M))).

Planning AgentPlanning Agent

Enviro

nm

ent

Sensors

Effectors

(Past and)Current

EnvironmentModel


Action ChoiceRules: model(t+n) = result([action1(t),...,actionN(t+n)] model(t+n) goal(t) do(action1(t))


Goal UpdateRules: model’’(t) goals(t-1) goals’(t)

GoalsPrediction of Future EnvironmentsRules: model’’(t) model(t+n) model’’(t) action(t) model(t+1)

HypotheticalFuture

EnvironmentModels

Planning AgentPlanning Agent

Percept and actions associated very indirectly through: Past and current environment model Past and current explicit goals Prediction of future environments resulting from different possible

action sequences to execute Rule chaining needed to build action sequence from rules

capture immediate consequences of a single action Pros:

Foresight allows choosing more relevant and safer actions in sequential environments

Cons: little point in building elaborated long term plans in, Highly non-deterministic environment (too many possibilities to

consider) Largely non-observable environments (not enough knowledge

available before acting) Asynchronous concurrent environment (only cheap reasoning can

reach a conclusion under time pressure)

Planning ThreadPlanning Thread

Goals

Current,past and

futureenvironment

model

Current Model Update

Percept Interpretation

Goal Update

Future Environments Prediction

Action Choice

Hybrid Reflex-Planning AgentHybrid Reflex-Planning Agent

Enviro

nm

ent

Sensors

Effectors

Reflex ThreadReflex ThreadReflex RulesPercepts Actions

Synchronization

Hybrid Reflex-Planning AgentHybrid Reflex-Planning Agent

Pros: Take advantage of all the time and knowledge available to choose

best possible action (within the limits of its prior knowledge and percepts)

Sophisticated yet robust Cons:

Costly to develop Same knowledge encoded in different forms in each component Global behavior coherence harder to guarantee Analysis and debugging hard due to synchronization issues Not that many environments feature large variations in available

reasoning time in different perception-reasoning-action cycles

Layered AgentsLayered Agents

Many sensors/effectors are too fine-grained to reason about goals using directly the data/commands they provide

Such cases require a layered agent that decomposes its reasoning in multiple abstraction layers

Each layer represent the percepts, environment model, goals, and actions at a different level of details

Abstraction can consist in: Discretizing, approximating, clustering, classifying data from prior

layers along temporal, spatial, functional, social dimensions Detail can consist in:

Decomposing higher-level actions into lower-level ones along temporal, spatial, functional, social dimensions

Abstract

Decide Abstractly

Detail

Act in DetailPerceive in Detail

Percept InterpretationPercept Interpretation

Am

bie

nte

Sensors

Effectors

Environment ModelEnvironment ModelEnvironment Model UpdateEnvironment Model Update

Action Choice and Execution ControlAction Choice and Execution Control

Layered Automata Agent

Layer0: f(x).dxy

Layer1: y).P(y)|P(zP(s)

Layer0: f(x).dxy

Layer1: y).P(y)|P(zP(s)

Layer2: q(A)r(B)B)s(A,

Layer2: q(A)r(B)B)s(A,

q(A)r(B)B)s(A, q(A)r(B)B)s(A, Layer2: Layer2:

Exemplo de camadas de Exemplo de camadas de abstração: abstração:

XX

YY

Abstraction Layer ExamplesAbstraction Layer Examples

XX

YY

Utility-Based AgentUtility-Based Agent

Principle: Goals only express boolean agent preferences among environment states A utility function u allows expressing finer grained agent preferences

u can be defined on a variety of domains and ranges: actions, i.e., u: action R (or [0,1]), action sequences, i.e., u: [action1, ..., actionN] R (or [0,1]), environment states, i.e., u: environmentStateModel R (or [0,1]), environment state sequences, i.e., u: [state1, ..., stateN] R (or [0,1]), environment state, action pairs,

i.e., u: environmentStateModel x action R (or [0,1]), environment state, action pair sequences,

i.e., u: [(action1-state1), ..., (actionN-stateN)] R (or [0,1]), Pros:

Allows solving optimization problems aiming to find the best solution Allows trading-off among multiple conflicting goals with distinct

probabilities of being reached Cons:

Currently available methods to compute (even approximately) argmax(u) do not scale up to large or diverse environments

Utility-Based Reflex AgentUtility-Based Reflex Agent

Enviro

nm

ent

Sensors

Effectors

Percept Interpretation:Rules: percept actions

Goals

Action Choice:Utility Functionu:actions R U(a))argmaxdo(

actionsa

Utility-Based Planning Agent

Enviro

nm

ent

Sensors

Effectors

Past &Current

EnvironmentModel

Percept InterpretationRegras: percept(t) model(t) modelo’(t)

Model UpdateRegras: model’(t) model’’(t)

Future Environment PredictionRegras: model’’(t) ação(t) model(t+1) model’’(t) model(t+1)

HypothesizedFuture

EnvironmentsModel

Action Choice])))action actionU(result([argmaxdo( i

ni

i1

actioni1

...

Utility Function:u: model(t+n) R

Learning Component

Performance Analysis Component

Adaptive AgentAdaptive Agent

Enviro

nm

ent

Sensors

Effectors

ActingComponent

New Problem Generation Component

• Reflex• Automata• Goal-Based• Planning• Utility-Based• Hybrid

• Learn rules or functions: • percept(t) action(t)• percept(t) model(t) modelo’(t)• modelo(t) modelo’(t)• modelo(t-1) modelo(t)• modelo(t) action(t)• action(t) model(t+1)• model(t) goal(t) action(t)• goal(t) model(t) goal’(t)• utility(action) = value• utility(model) = value

Simulated EnvironmentsSimulated Environments

Environment simulator: Often themselves internally follow an agent architecture Should be able to simulate a large class of environments that can

be specialized by setting many configurable parameters either manually or randomly within a manually selected range ex, configure a generic Wumpus World simulator to generate world

instances with a square shaped cavern, a static wumpus and a single gold nugget where the cavern size, pit numbers and locations, wumpus and gold locations are randomly picked

Environment simulator processing cycle:1. Compute percept of each agent in current environment2. Send these percepts to the corresponding agents3. Receives the action chosen by each agent4. Update the environment to reflect the cumulative consequences

of all these actions

EnvironmentEnvironmentSimulationSimulation

ServerServer

RedeRede

Environment Simulator ArchitectureEnvironment Simulator Architecture

SimulatedEnvironmentModel

SimulationVisualization

GUIEnvironment UpdateRules: model(t-1) model(t)action(t) model(t-1) model(t)

Percept GenerationRules: model(t) percept(t)

percepts

actionsAgent

Client 1

AgentClient N

...

KBA ArchitecturesKBA Architectures

A KBA is a: Reflex Agent? Automata Agent? Goal-Based Agent? Planning Agent? Hybrid Agent? Utility-Based Agent? Adaptive Agent? Layered Agent?

Can be anyone !

Is there any constraint between the reasoning performed by the inference engine and the agent architecture?

Adaptive agent requires analogical or inductive inference engine

AskTell

Non-Adaptive KBANon-Adaptive KBA

Enviro

nm

ent

Sensors

Effectors

Persistent Knowledge Base (PKB):rules, classes, logical formulas or probabilities

representing generic laws about environment class

Inference Engine for Deduction, Abduction, Inheritance, Belief Revision, Belief Update,

Planning, Constraint Solving or Optimization

Volatile Knowledge Base (VKB):facts, objects, constraints, logical formulas or probabilities

representing environment instance in current agent execution

Retract

Ask

Non-Monotonic

Engine

AskTell

Analogical KBAAnalogical KBA

Enviro

nm

ent

Sensors

Effectors

Persistent Knowledge Base (PKB):facts, objects, constraints, logical formulas or probabilities

representing environment instances in past agent executionsstructured by similarity measure

Inference Engine for Analogy



Retract

Ask

Remember the Planning Agent?Remember the Planning Agent?

Enviro

nm

ent

Sensors

Effectors

(Past and)Current

EnvironmentModel


Action ChoiceRules: model(t+n) = result([action1(t),...,actionN(t+n)] model(t+n) goal(t) do(action(t))


Goal UpdateRules: model’’(t) goals(t-1) goals’(t)

GoalsPrediction of Future EnvironmentsRules: model’’(t) model(t+n) model’’(t) action(t) model(t+1)

HypotheticalFuture

EnvironmentModels

How would be then aHow would be then aknowledge-based planning agentknowledge-based planning agent??

En

viro

nm

en

t

Sensors

Effectors

VKB: Past and Current

Environment Models

VKB: Hypothetical FutureEnvironment Models

VKB: Goals

PKB: PerceptInterpretation

PKB: EnvironmentModel Update

PKB: GoalsUpdate

PKB: Prediction ofFuture Environments

PKB: ActingStrategy

InferenceEngine

Alternative Planning KBA Alternative Planning KBA ArchitectureArchitecture

En

viro

nm

en

t

Sensors

Effectors


Environment Models


VKB: Goals



PKB: GoalsUpdate


PKB: ActingStrategy

InferenceEngine 1

InferenceEngine 2

InferenceEngine 3

InferenceEngine 4

InferenceEngine 5

Why Using Multiple Inference Why Using Multiple Inference Engines?Engines?

En

viro

nm

en

t

Sensors

Effectors


Environment Models


VKB: Goals



PKB: GoalsUpdate


PKB: ActingStrategy

InferenceEngine 1

InferenceEngine 2

InferenceEngine 3

InferenceEngine 4

InferenceEngine 5

Abduction

BeliefUpdate

Deduction

ConstraintSolving

Optimization

Off-Line Inductive Agent:Off-Line Inductive Agent:Training PhaseTraining Phase

Data, Examples or Case Base facts, objects, constraints or

logical formulas codifyingrepresentative sample of

environment entities

PerformanceInference Engine:

Any Reasoning TaskExcept Analogyand Induction

Intentional Knowledge Base (IKB):rules, classes or logical formulas

representing generic lawsabout environment class

InductiveInference Engine

Ask

Tell

Retract

HypothesisFormation

HypothesisVerification

Ask

Tell

Retract

Ask

Ask

Off-Line Inductive Agent: Usage Off-Line Inductive Agent: Usage PhasePhase

AskTell

Enviro

nm

ent

Sensors

Effectors

Inductively LearnedPersistent Knowledge Base (PKB):

rules, classes, logical formulas or probabilitiesrepresenting generic laws about environment class

Inference Engine for Deduction, Abduction, Inheritance, Belief Revision, Belief Update,

Planning, Constraint Solving or Optimization



Retract

Ask

Ontologies Reasoning Components Agents Simulations Architectural Patterns for Agents Jacques Robin.

Documents

Transcript of Ontologies Reasoning Components Agents Simulations Architectural Patterns for Agents Jacques Robin.