Depei Bao

What is the problem?●A robot with intense interaction with environmental

resources, such as another object, substance, surface or medium.

●The goal of the robot is to make some change to other object and/or itself mediated by body parts and/or other entities, i.e., tools.

What is the problem? (cont.)1.A general knowledge representation and reasoning

mechanism for rational decision making in any situation (independent of the specific knowledge structure)

2.Planning and reasoning with the aid of object ontological structure

3.Cognitive process to mediate sensory input and motor control in real interaction

4.Sensor signal recognition and control signal generation

What is the problem(cont.)

●The upper level constrains the lower level

●The lower level realizes the upper level

●Focus on 2 and 3

Basic Ideas●General decision making: means-end reasoning,

utility theory, preference handling, etc.●Domain: physical object manipulation• affordance-

based knowledge representation and schema●Cognitive bridge: embodied verification and planning●Sensor to knowledge and knowledge to controller

Pieces of Philosophy, Ontology and Psychology1.Reasoning and Verification

2.Affordance and Interactive properties

3.Embodied Knowledge and Reasoning

What is Reasoning?●Ways to retrieve relevant information from a

knowledge base represented and organized in a certain structure

●Typical structure: proposition or predicate logic

Relevant=Rational●Rational criteria

1.Consistent logic inference

2.Conform to mental states: desire, belief and intention

3.With constraints of physical world and social codes (beliefs)

4.Context-sensitive

Symbol structure manipulation?●The curse of AI: common sense and context

reasoning, such as frame problem●Solution 1: Nonmonotonic logic for reasoning●Solution 2: Capture the semantics of symbols?

symbol grounding problem

Verification●Determine the truth value of a symbol●Only the system can understand the semantics of

the symbol, i.e., the meaning a symbol represents●How?

1.Symbol network (such as semantic networks)

2.Knowledge is embodied

3.Grounded on sensor data

4.Symbol to controller mapping

Affordance

●A concept coined by ecological psychologist James Gibson

●Denote a resource or support that the environment offers an animal for action.

●A goal-biased way to perceive an object identified by its function

Why affordance?●Intuitively natural approach of our perception

and cognition●As a manner of knowledge representation

suitable to context-based physical behavior●Context: the environment elements combined

with the agent’s ability provide the opportunity (as the precondition) for the proper action/effect

The Semantics of Affordance●Sth O can afford an action A or an effect E (by some

implicit action).●=some properties of O satisfy the requirement of A or E. ●Four types of affordances

1.First-order or binding affordance: affords an action, e.g., can climb, or affords a status of the agent or other objects, e.g., can support.

2.Second-order or output affordance: affords changes to itself, e.g., afford to be struck into the wall or affords changes to other objects, e.g., hammer.

Interactive properties●How to represent properties requirement?

1.Properties requirement is relative

2.The meanings of properties are grounded on object properties

3.The mapping (or meaning) is different for different actions even though they share the same name

•interactive properties

Interactive properties (Cont.)

●In essence, interactive properties are constraints on object properties, e.g., hardness constrains on size, shape and material of an object

●Constraints also among interactive properties, e.g., sharpness and hardness

●Constraints also among affordances for a task.●Three levels of constraints satisfaction problem

(CSP) on object properties (for verification)

Object properties●Properties of an object independent of specific task

and can directly recognized from sensor data.●What the purpose they serve?

1.Object recognition: necessary properties to identify an entity as an instance of an object class.

2.Parameters to distinguish among different object classes and among instances of the same object class to meet specific functional requirement.

Question: object classes are contingent?●Two sorts of object classes possess different

ontological natures1.An artifact with a designed purpose2.A natural occurrence object identified only by its

substance●Share similar representational structure3.Common primitive properties: shape, material,

color, etc. (attribute-value)4.Compositional parts5.Intrinsic and extrinsic states

Identity of an object●An object class constraints the properties of its

compositional parts and their compositional relationships.

●For each primitive part, the constraints are imposed on the values of relevant attributes.

●An object may exhibit different properties in different internal state.

●Is-a relation and part-of relation

Identity of an object (Cont.)●A natural occurrence object, constraints are

determined only by natural properties●An artifact is identified by function, not by natural

properties●Designed function for an artifact class=generalized

affordance properties, ●a prototype, a looser constraint on properties than

its instances●Introduce attribute not common to all objects

Perception●Recognition of object properties is a process to

match sensor pattern●Refer to the determinable/determinates, e.g., color

and genus/species, e.g., shape

Situation awareness and tracking●A tracking process for the environmental situation,

purpose-independent perception●Obtained information can be used for verification

without the need to search from the beginning●Object identification, position, velocities and

orientations of objects and agents, spatial relation of object, subject-object relations, events, etc.

●States are tracked at every perception loop.

What is the meaning of embodiment●Knowledge is obtained, represented and verified

in terms of subject experience.●Body-related reasoning and verification. ●Body information is implied in affordance

representation●Empathy

Knowledge level embodiment●Every interactive property of an affordance implies a

body configuration, body dynamics and body properties, but the properties of a body part is not explicitly considered and/or set in verification.

●Past experience gives metrics to every such mapping from interactive properties to properties of an object for an action.

●Similarity comparison between an instance and properties prototype by the metrics for the verification of an affordance.

Controller level embodiment●Only at the control level, unconsciously, we consider the

body configuration (after verification of a symbol, this is a mapping from the semantics of the affordance to a controller).

●Control level embodiment: during an actual operation, we recognize the specific sensory feedback pattern to verify the interaction and dynamically adapt body configuration, i.e., the record of the real experience.

Schema●“a mental codification of experience that includes a

particular organized way of perceiving cognitively and responding to a complex situation or set of stimuli”

--Merriam-Webster's Collegiate Dictionary●A knowledge organization and retrieval method●Grouping relevant knowledge (precondition and

solution) of a task together

Symbol to control signal●A controller?

1.A low level dynamic feedback system

2.Generating spatial and kinematic trajectories of the agent’s body and apply a certain style of force onto the target

Symbol to control signal (Cont.)●Order=Mapping from embodied symbol knowledge●Order from cognition or not●Mapping Levels?●Feedback through cognitive channel or not

Body image●Under the assumption of embodied knowledge

representation, centralized body knowledge is rarely used for reasoning at an interaction

●The high-level self image only exists as a social element.

●At the low level, however, there is a body model for controller derivation.

Body-tool extension●Two levels: the cognitive level and the controller

level●The cognitive level: changed semantics of

affordance symbol●The controller level: body model extension (different

mapping from cognition)●The ‘feeling’ is still recognized from sensory pattern●Triggering? Learned Pattern for a specific action?

The framework●From ideas to realization●Glue various components●Process for reasoning from the goal to the plan

beyond symbol manipulation

Three steps●A single cycle of means-end reasoning

1.Target effect analysis (end)

2.Effectual action search (means)

3.Plan construction (refine the means in terms of the end and situation)

●Example: tries to hammer a nail into wall

Target effect analysis●Effect?

1.In multi-attribute decision theory terminology, effect=outcome of an action=an attribute vector (propositional or numeric) of states

2.Thus, effect=a subject/object status description capturing the static/dynamic meaning (states or states change) of the status

●Description Extension●Verb-centered representation

Verb-based representation●A verb has two semantic dimensions: the effect/state

and the manner of the action●Direct grounding of these two meanings: ‘remove’ a

state and ‘strike’ on a controller pattern●Otherwise, specify more meanings by extending

along verb links (relations) with context check

●Effect: the index to effectual action and plan schema

Verb relations●WordNet

1.Troponymy: the super-sub (is-a) relation, e.g., scribble and write

2.Causation: e.g. , give/have, reach/hold

3.(Temporal) inclusion: the part-of relation

4.Backward presupposition: prerequisite (not the cause), e.g., win/play

Example●the target state is (the depth would be specified,

which is dependent on the previous end).●The action is a direct description of the final state.

Effectual action search●An effectual action can be retrieved from the target

state category by some analysis on the situation●Why?

1.follow human problem solving pattern: schema

2.Two second-order affordances (passive and positive) are complementary and mutual dependent, fix the target.

Example●In the embedding category, measure the properties

of x’s tips (i.e., the shape and material of a nail) and y’s surface (wall), and then derive striking.

Plan Construction●Decision= action sequence schema + verification

schema + adjustment●Action sequence schema: an effectual action has a

prototype of action sequence specification, e.g., facilitating actions

●Verification schema:

1.an effectual action and its facilitating actions has corresponding affordance schema (precondition)

2.These affordances have relations, e.g., causal

Plan Construction (Cont.)●If necessary, a more general action is decomposed

into further primitive actions along the hierarchy until an action can be directly mapped to a controller

●Finally, a plan is represented as a finite automaton: state: action primitives, transitions: expected observed outcome, which can be mapped to a lower level controller (state) and sensor data (transition)

Example●By measuring the target(the nail, wall and depth), striking

(the action mode is impulsive) requires an object able to provide enough impulse on each blow.

●Two functional parts and the functional relation

●Interactive properties also include spatial requirement such as interface matching

●Available space verification, also embodied●A facilitating action: the nail affords to be stabilized●in the preparation phase

Decision making●Goal≠Plan Performance

1.Uncertainty/reliability of the plan: calculated as the result of verification

2.The cost of plan

3.The cost of verification●More sophisticated agent modeling

Verb extension and application●‘is-a’ relation: the basic verb organization

1.A sub-verb is the sub-type of a super verb with more semantic constraints.

2.e.g., ‘take’ is a kind of ‘shift’ focusing on the source whereas ‘give’ focuses on the recipient.

3.The constraint information is useful for grounding and planning, e.g., from ‘move’ to ‘step on’ (step 2 and 3)

●Causation: effect extension (step 1) and effectual action selection (step 2)

●(Temporal) inclusion: the part-of relation1.In plan schema selection (step 3)2.From a general action category (e.g., an effectual

action) to primitive actions●Backward presupposition: prerequisite (not the

cause), e.g., win/play (step 2 and 3, different conditions)

Functional primitives●Properties requirement: Mapped to object properties ●Nonmonotonic logic

1.There is no hope to check all possible preconditions and side-effects for an action

2.Defeat the conclusion (an affordance) if we can get extra information from other sources, e.g., object properties

Functional relations●behavior relations (also a kind of affordance) of two

or more functional parts, e.g., ‘swing together’●mapped to spatial relations and physical properties

(for the relation) of functional parts, e.g., for ‘swing together’, the effecter and the handle should be rigid and be connected in a way that affords swinging in the same direction.

Verification procedures●May also involve search for an object●The search is finally guided by object class

knowledge, i.e., properties constraints●Uncertainty in affordance verification (later)●CSP, heuristics for the sequence of verification

Failure●The effect is not achieved as expected●Cues to indicate the failure: e.g., sensor feedback●Diagnosis: attribute reasons to the failure

1.Sensor inaccurate

2.Uncertain verification

3.Unconsidered environmental conditions

4.Situation change

5.Disregarded properties

6.Side-effects to the target

Feedback mechanism●Sensor: vision, tactile, proprioceptional, balance(?)●Proprioceptional:

1.the sense of the relative position of neighboring parts of the body

2.Feedback traced by controller, return (map) to symbols in the cognitive part if such information (such as the current body posture) is needed

Verification of affordances●Mapping knowledge from the measurement of the

target properties to properties of the affording object bridged by interactive properties.

●Only for the target as an object, a subject is implicit●For four kinds of affordances

1.Pure action: target: action itself

2.Static status: subject/object parameters

3.Passive affordance: self-mapping

Example1.Positive affordance: the required effect/change to

another object. The effectual action determines the target properties to measure, which constrains the object’s properties in terms of its interactive properties.

●Example, measure the properties of the nail, the wall and the embedded depth.

Uncertainty of verification●Constraints often specify two thresholds for a

parameter t1 (guaranteed) and t2 (excluded)●Effectual process●Out of ranges: reasons against the effect●Reasons are uncertain●A number: uncertain properties for the extent to

which (or the probability of) the reason contributes to the failure of the process.

●Piercing: sharpness: the angle of the tip, from contacting the wall to the specified depth

Conclusion●An implementation?●A knowledge intensive approach to the problem

because of its generality●Domain specific information is represented as

knowledge rather than embedded in structure.●Cognitive framework, leave sensor pattern

recognition and controller construction