Peter J. Braspenning [email protected] Local I-MASS group

Communications Communications Research & Research & SemioticsSemiotics

(CR&S)(CR&S)

Assoc. Prof. Peter J. Braspenning, CR&S, UM1

15-01-2002

The Knowledge Technology of I-MASS (EC IST Research)

Communication, Computing and Interactive Networks

Peter J. Braspenning

[email protected] I-MASS groupLocal I-MASS group

Peter-Paul Kruijsen, Gabriel Hopmans & Peter J. Braspenning

Communications Research & SemioticsUniversity of Maastricht


(CR&S)(CR&S)


(CR&S)(CR&S)


15-01-2002

Signs & Meanings


(CR&S)(CR&S)


15-01-2002

ContentIntroductionWhat is (Computational) Semiotics?

Semiotic FrameworkLogo of CR&SCommunication Process among PeoplePerceive-Act Pathways

Multi-Modal User Interface:Semiotic EngineeringCognitive Engineering

Agent-Oriented ModelingSystem’s Modeling via Multi-Agent SocietiesComputational Semiotics for Agent Technology

Knowledge Landscape & VRRConclusions


(CR&S)(CR&S)


(CR&S)(CR&S)


15-01-2002

What is Semiotics?Semiotics: discipline of combining the theory of signs (representa-tions),

symbols (categories), and meaning extraction. It is an in-clusive discipline which incorporates all aspects of dealing with symbols and symbolic systems, starting with encoding and ending with the extraction of meaning.

Mathematical tools of semiotics include those used in control scien-ces, pattern recognition, neural networks, artificial intelligence, and cybernetics. Unified use within a computational semiotic framework leads to better treatments of the complexities (com-munication and computation) inherent in advanced intelligent systems. Semiotics is a strongly emerging multi-disciplinary field of study around a new paradigm for surpassing the classical mind-body dichotomy by focussing on all processes in which the triad object-sign-interpreter plays an essential role.

The pervasive use of icons in the interaction with communicative virtual environments (CVEs), is also part of Semiotics. A lot of different kinds of signs are exchanged while communication takes place.


(CR&S)(CR&S)


(CR&S)(CR&S)


15-01-2002

What is Semiotics? (continued)

• Semiotics is devoted to studying COM-MUNICATION: representations, their interpretation and usage

• It investigates SIGNS and the processes by which we take them to mean something to us and expect them to mean something to others

• It investigates the resolution of meanings in conversations, collective discourse and culture in general

• Semiotics also covers non-human commu-nication processes such as that of animals and machines/artificial systems


(CR&S)(CR&S)


15-01-2002

Peirce: sign = “something standing for something else for somebody in one or more respects”

interpretant

sign object

Semiotic frameworkComponents of Symbol

System[Schuyt]

interpretation system

groups acts & events


(CR&S)(CR&S)


15-01-2002

Logo of CR&S


(CR&S)(CR&S)


15-01-2002

Unlimited semiosis

“I like guys with dark hair”

medium

coding decoding

interpretant

interpretant

interpretant

message (signs)

Communication Process among People


(CR&S)(CR&S)


15-01-2002

Perceive-Act Pathways


(CR&S)(CR&S)


15-01-2002

• design intention• interaction principles

message

user system designer

Semiotic Engineering perspective of HCI


(CR&S)(CR&S)


15-01-2002

usage model

interaction

system imagedesigner user

=task model

+user model

design model

Cognitive Engineering


(CR&S)(CR&S)


15-01-2002

context

userdesigner

context

medium

Cognitive Engineering

Semiotic Engineering

context

Cognitive Engineering xSemiotic Engineering


(CR&S)(CR&S)


15-01-2002

Semiotic Engineering and Interface Evaluation

Communicability Concept– Communicability is the property of software that

efficiently and effectively conveys to users its underlying design intent and interactive principles

– The communicability evaluation method allows designers to appreciate how well users are getting the intended messages across the interface and to identify communication breakdowns that may take place during interaction


(CR&S)(CR&S)


15-01-2002

Assessing Communicability Utterances Symptoms Classes of Interaction

Problems

Where is?User searches for a specificfunction but demonstratesdifficulty in locating it

Navigation

What’s this? User explores possibilitiesof interaction Meaning Assignment

Oops!

I can’t do it this way.

User performs an action,but the outcome is differentthan expected / Cancels thesequence of actions &chooses a different path

Navigation

Meaning Assignment

Why doesn’t it?User expects an outcomebut doesn’t achieve it.Insists on the same path

Task Accomplishment

I can do otherwise.User ignores preferentialintended affordance presentin the interface

Missing of affordance


(CR&S)(CR&S)


15-01-2002

Agent-Oriented Modeling (issues)

• Intelligent Agent: an assistant that takes care of many gory details of many mundane tasks

• Additional properties are autonomy, sociability, and a human-like communication

• Often able to adapt to user's interests, habits and preferences

• Enabled to communicate with other agents it is potentially entering role-taking behavior and social commitments with other agents that allow it to function in a society of agents

Multi-Agent System: bring such agents together in a kind of abstract society, wherein coordination, cooperation and/or collaboration are of paramount importance in order to solve problems that no single agent could handle on its own

FIPA specifications represent a collection of standards, which are intended to promote the interoperation of heterogeneous agents and the services that they can represent.


(CR&S)(CR&S)


15-01-2002

System’s Modelingvia Multi-Agent Societies

• One has to decide how to provide efficient inter-agent communication support, what language should the agents talk, should the agents be stationary or mobile, and what technology should be used to build the architecture

• At present, there are not much experience reports

• Architecture of a multi-agent system can naturally be viewed as a computational organization

• Additional organizational concepts– organizational rules,

– organizational structures, and

– organizational patterns


(CR&S)(CR&S)


15-01-2002

I-MASS uses MAS’s perspective not just as a framework for inter-action, but more as forming abstract societies consisting of agencies (comparable to societal institutions), complex agents (in the sense of consisting of simpler agents), and agents (roughly comparable to individuals in a societal context).

We try to deal with content inter-operability issues at different abstraction layers of syntactics, semantics, pragmatics and social world. These layers fit into a coherent semiotics framework.

System’s Modelingvia Multi-Agent Societies (continued)


(CR&S)(CR&S)


15-01-2002

Computational Semiotics for Agent Technology

Coordination is cen-tral to building MASs

Coordinating behav-iors in MASs are often realized by forming groups in which both control and data are distri-buted.

Therefore, agents have to have some auto-nomy in performing their actions.

However, this autono-my may lead to un-coordinated activities due to uncertainty about the actions of each of the agents.


(CR&S)(CR&S)


15-01-2002

The relationship between uncertainty and the situation that

the agents have to handleThe uncertainty lowers as

the familiarity of the situation that needs to be handled increases!

Therefore, it makes sense to develop a framework in which agents know how to handle routine, familiar, and unfamiliar situations


(CR&S)(CR&S)


15-01-2002

Co-ordination among agents: guiding principles

• Coordination among agents is easier to esta-blish in routine than in unfamiliar situations;

• In general, communi-cation between agents will be more needed in unfamiliar situations than in routine situations.


(CR&S)(CR&S)


15-01-2002

Needed: an agent-architecturein which three kinds of interaction are adressed

• Conceptual models [J. Rasmussen, Information Processing and Human-Machine Interaction: An Approach to Cognitive Engineering, 1986]

– skills– rules– knowledge

• The knowledge representation should be adapted to these kinds of interactions.


(CR&S)(CR&S)


15-01-2002

Computational scenario• First, perceived information from the environment leads

the agent to execute an action if the correspond-ing situation is perceived in terms of action.

• If this is not the case, the agent tries to recognize the situation. It can recognize the considered situation in terms of an action or in terms of a goal. In the first case, it tries to execute the corresponding action, and in the second case it invokes the planning module.

• Finally, if the agent faces an ambiguity and cannot come to a decision, or faces many alternatives, then it invokes the decision making module (based on a Cognitive Map) to make a decision in order to commit to achieve a goal or an action. A goal leads an agent to plan, that is to produce a sequence of actions that achieve the chosen goal.


(CR&S)(CR&S)


15-01-2002

Three levels of control of human behavior

Knowledgeperception recognition decision planning execution

perception recognition decision execution

perception recognition planning a execution

Rulesperception recognition planning b execution

perception recognition execution

Skills perception execution

a the planning process adapts old cases to the new situation, and the adaptation is significant b the planning process adapts old cases to the new situation, and the adaptation is generally minor

B. Chaib-draa & P. Levesque, Hierarchical Model and Communication by Signs, Signals and Symbols in Multiagent Environments, 1998


(CR&S)(CR&S)


15-01-2002

Signals, Signs, and SymbolsSignals can be viewed as data

representing time space variables from a dynamic, spatial configuration in the environment and they can be processed directly by the agents as continuous varia-bles. In communication by signals, the signal delivered by an agent i has the end of simply being a releaser for the receiving agent j -- of simply eliciting a reaction by j. That is, the signal generally invokes a stimulus or a reaction, without passing through the memory (a data base in this model).

Signs indicate a state in the environ-ment with reference to certain norms for acts. In the case of communication by signs, the sender makes a sign which refers to some state of environment and which has the end of signifying, of letting the receiver knows the same reference. Of course, the sender and the receiver should share a set of signs with their references in order to communi-cate efficiently. For instance in urban traffic, communication between a driver and a policeman at a crossroad is generally done by signs. The policeman makes a specific sign which refers to a certain action and which is addressed to certain driver(s). The addressee(s) recognize(s) the reference of this sign and activate(s) stored patterns of behaviors.


(CR&S)(CR&S)


15-01-2002

Finally, agents can also communi-cate by symbols. Symbols repre-sent variables, relations and properties and can be formally processed. They are abstract con-structs related to and defined by a formal structure of relations and processes, which according to convention can be related to features of the external world.

In urban traffic for instance, a dialogue between a policeman and a driver in natural language reflects a symbol based communication. Another example of symbolic communication

is ``honk the car horn'', etc.

Information at knowledge and rule levels can act as symbols depending on the situation and the language used for com-munication. In familiar situa-tions corresponding to the rule level, agents can use a specific language (derived or not from a natural language). This lang-uage is generally constructed from repeated activities. When unfamiliar situations occur, agents do not dispose of any operative knowledge nor of any specialized language. They must then make use of a non specialized language (for example natural language), which is less concise but more flexible than their oper-ative language used in familiar situation.

Signals, Signs, and Symbols


(CR&S)(CR&S)


15-01-2002

Coordination: in summaryWith signals and signs, agents

do not force their cognitive control to a higher level (i.e. the knowledge level) than the demands of the situation requires.

In contrast, agents have a propensity for behaviors based on skills and rules. These behaviors are gene-rally fast, effortless and propitious to a better co-ordination between agents.


(CR&S)(CR&S)


15-01-2002

Recap• Communication and Semiosis are two sides of

the same coin;• Knowledge Representation has tradition-ally

only be treated in the context of solip-sistic systems. However, communication is constitutive for Knowledge (Represent-ation) & Reasoning Reflection;

• Agent Technology as a modeling metho-dology allows us to treat rather complex systems. Moreover, the semiotics perspec-tive sheds new light on issues concerning User Interfaces, Exploration Narratives, and all emerging kinds of New Media


(CR&S)(CR&S)


15-01-2002

Knowledge Landscape & VRR• A systemic approach via

Agent-Oriented Modeling & the development of agent-based tools, and

• An operational elaboration of the new concept of the Virtual Reference Room, by means of which contextualized access to heterogeneous objects can be realized, and more knowledge-based navigation by means of these contexts becomes feasible.


(CR&S)(CR&S)


15-01-2002

Virtual Reference Room Explanations about how to

explore the collections, pro-ducts and services of the institution;

Delivery of information about actual services of the Reference Room that are in force;

Pointers to where bibliographical services may be found;

Pointers to relevant exhibitions and other relevant events in connection with their search questions and associated references;

Orientation maps to more autonomously explore the facilities of the Reference Room and around the particular institutional collections maintained;


(CR&S)(CR&S)


15-01-2002

Virtual Reference Room (cont.)


(CR&S)(CR&S)


15-01-2002

D ig ita lS ou rces

M A N IP U L A TIO N S

U s e rInte rfa c e

Q ue ry/R e s po ns e a ndK L -B uild ing

K n o w le d g eL a n d s c a p e

(K L )

6

5

3S e m a ntic & S ynta c tic

Inte r-o pe ra bility

4

2

High- level OVERVI EWI -MASS

1


(CR&S)(CR&S)


15-01-2002

In crea sin gS em a n tic

E x p ressiv en ess

A M L

W r i t t e n i n

C o n s t r a i n t s

S e m an t i ci n t e r -

o p e r ab i l i t y

S yn t ac t i ci n t e r -

o p e r ab i l i t y

C o n t e n t

Tr an s fo r m e r

Tr an s fo r m e r

Tr an s fo r m e r

Tr an s fo r m e r

Tr an s fo r m e r

J D B C

Q u e r yR e s u lt

C o nte nt pro v ide rs





AB S TRACTION LAYER

K R L

C o n t e n t

X M L

C o n t e n t

O - O

C o n t e n t

R D F

C o n t e n t

S Q L

M e ta -da ta

M e ta -da ta

M e ta -da ta

P ro cess M o d els


(CR&S)(CR&S)


15-01-2002

P e r s p e c t i ve s

T H E V I R T UAL R EFER EN CE R O O M

Knowledge Landscape Manipulation - T oo l 1 - T oo l 2 - T oo l 3

Knowledge Landscape

I S I S IS

KO1

KO3

KO2

KO cR

KOaR

KObR

R eferen c e' W o rk



R

R '

R ''

Projectionontouser

screen


(CR&S)(CR&S)


15-01-2002

The big pictureR e a l W o rld

V irtu a lW o rld V irtua l R e fe re nc e E nviro nm e nt

D i g i t i s e dA r t e fac t

D i g i t i s e dA r t e fac t

V irtua l R e fe re nc e R o o m

Q u e r y S p e c i fi cV i e w o f S t r u c t u r e d

R e fe r e n c e L an d s c ap e

Synt

acti

c &

Sem

anti

cIn

tero

pera

bili

ty

Uni

form

Str

uctu

red

Vie

w o

f Dig

itis

edC

ultu

ral

Dom

ain

Dig

itis

ed C

olle

ctio

nsC

atal

ogue

s / D

icti

onar

ies

/ Art

efac

tsE

ncyc

lopa

edia

s / T

hesa

uri

/ Arc

hive

s ...

Q ue ry

W e b S i t e sW e b P ag e s

U s e r

H e rita g eIns titu tio ns


(CR&S)(CR&S)


15-01-2002

Conclusions w.r.t. I-MASS• Short term research will address the precise kinds of

enabling technologies (e.g., from information retrieval, ontological engineering and knowledge engineering) that the I-MASS system should incorporate to synthesize (configuring and presenting) pieces of information that seem to fulfill an apparently existing need for information at the side of the users;

• Also research about semantical (and pragmatical) inter-operability will require much effort, especially as it contributes quite a lot to the system’s ability to provide good answers;

• Longer-term research must address how the cultural domain may be modeled by means of process models that capture the relevant insights of cultural processes (e.g., the rise and fall of the Roman Empire, or the Renais-sance).


(CR&S)(CR&S)


15-01-2002

Conclusions

Leibniz’ (1646-1716) ambition was “to awake the sleeping child in us all”

I-MASS forces us to use Knowledge Tech-nology to the utmost and to use societal metaphors as much as possible!


(CR&S)(CR&S)

Peter J. Braspenning [email protected] Local I-MASS group

Documents

Transcript of Peter J. Braspenning [email protected] Local I-MASS group