A breadboard architecture for pervasive context-aware services in smart spaces: middleware...
-
Upload
john-soldatos -
Category
Documents
-
view
217 -
download
0
Transcript of A breadboard architecture for pervasive context-aware services in smart spaces: middleware...
ORIGINAL ARTICLE
A breadboard architecture for pervasive context-aware servicesin smart spaces: middleware components and prototypeapplications
John Soldatos Æ Nikolaos Dimakis Æ Kostas Stamatis ÆLazaros Polymenakos
Received: 31 January 2006 / Accepted: 20 September 2006 / Published online: 21 November 2006� Springer-Verlag London Limited 2006
Abstract We present an architectural framework
along with a set of middleware elements, facilitating
the integration of perceptual components, sensors,
actuators, and context-modeling scripts, comprising
sophisticated ubiquitous computing applications in
smart spaces. The architecture puts special emphasis
on the integration of perceptual components contrib-
uted by a variety of technology providers, which has
not been adequately addressed in legacy architectures.
Moreover, the introduced architecture allows for
intelligent discovery and management of resources.
Along with the description of this breadboard archi-
tecture, we present its non-functional features and as-
sess its performance. We also outline a rich set of
practical prototype pervasive services that have been
built, based on this architecture. These services
emphasize on providing non-obtrusive human-centric
assistance (e.g., memory aids, meeting recordings,
pertinent information) in the scope of meetings, lec-
tures and presentation, Experiences from building
these services manifest the benefits of the introduced
architecture.
Keywords Smart spaces � Autonomic computing �Ubiquitous computing � Context-awareness �Architecture � Middleware
1 Introduction
Pervasive and ubiquitous computing promise to trans-
form physical spaces into computationally active and
intelligent environments and envision a world, where
computers provide services to humans regardless of
time and users’ location [1]. Ubiquitous and pervasive
services are essentially context-aware, since they
automatically acquire and process information about
their surrounding environment [2]. Thus, context-
aware applications execute service logic not only based
on input provided explicitly by end users, but also
based on implicitly derived information [3]. Context
acquisition can be based on several approaches,
including reading tags to track objects and inferring
context (e.g. [4]), wearable computing systems (e.g.
[5]), as well as using sensors and effectors for natural
interaction in smart spaces (e.g. [6]). Sophisticated
ubiquitous computing applications are likely to lever-
age more than one of these approaches, leading to
hybrid schemes. For instance, wearable computing
applications and tag reading may operate in the scope
of smart spaces. Note that all the above approaches
rely on networked sensing infrastructures, which have
to be as non-obtrusive as possible. Apart from
numerous sensors, smart spaces also include actuators
facilitating the interaction between end-users and the
pervasive computing environment.
No matter which context-acquisition approach is
used, context-awareness relies on hardware sensors
J. Soldatos (&) � N. Dimakis � K. Stamatis �L. PolymenakosAthens Information Technology, 19.5 km Markopoulo Ave.,P.O. Box 68, 19002 Peania, Greecee-mail: [email protected]
N. Dimakise-mail: [email protected]
K. Stamatise-mail: [email protected]
L. Polymenakose-mail: [email protected]
123
Pers Ubiquit Comput (2007) 11:193–212
DOI 10.1007/s00779-006-0102-7
and signal processing algorithms. The latter extract
elementary forms of context (e.g., relating to people
and objects state), which can be combined toward
recognizing more complex contextual states. This
combination relies on middleware components under-
taking the fusing of the output of multiple sensors, as
well as to coordinate the interactions and information
from multiple perceptual components and recognition
algorithms. Additional middleware components are
required to facilitate integration, development and
deployment of complete applications. Integration is a
primary concern, given that pervasive computing
infrastructures are highly distributed and heteroge-
neous. Thus, additional middleware components pro-
viding directory services (i.e., registration and
discovery of components), facilitating interfacing to
sensors, actuators and perceptual components, fusing
information, and overall easing service development
are required.
Middleware components within a pervasive com-
puting environment can be reused across different
applications. Reusable middleware can significantly
ease service implementation, through allowing the
application developer to emphasize on the service lo-
gic, rather than on the integration with the underlying
sensors and context-aware components. Such middle-
ware components are a key prerequisite for the wide
adoption of ubiquitous computing, since they can sig-
nificantly minimize the effort required to develop and
deploy applications [7].
Major pervasive and ubiquitous computing projects
have developed middleware infrastructures facilitating
component integration, as well as development and
deployment of services. As a prominent example, the
Interactive Workspaces project (at Stanford university)
has developed the Interactive Room Operating System
(iROS) [8], which provides a reusable, robust, and
extensible software infrastructure, enabling the
deployment of component-based ubiquitous comput-
ing environments. iROS supports various modalities
and human–computer interfaces, by tying together di-
verse devices, each one having its own operating sys-
tem. The Oxygen project at MIT has produced the
MetaGlue system [9], which constitutes a highly robust
software agent platform. Agents are used to represent
both local resources and interactions with those re-
sources. MetaGlue relies on a custom distributed
communications infrastructure, enabling agents to run
autonomously of individual applications so they are
always available to service multiple applications. Sup-
port for implementing context-awareness is provided
through the GOALS architecture [10], which is the
evolution of the MetaGlue system. The EasyLiving
system developed at Microsoft research [11, 12] has
also produced an architecture enabling coordination of
devices, as well as fusion of contextual information.
Specifically, this architecture employs computer vision
technologies for person-tracking and visual user inter-
action and supports context-awareness based on a
geometric model of the world. It uses device-inde-
pendent communication and data protocol and
accordingly adapts the user interface. Another archi-
tectural model for pervasive computing has been
developed by the Carnegie Mellon University (CMU)
Aura project [13], which targets environments involv-
ing wireless communication, wearable or handheld
computers, and smart spaces. Aura provides software
architectural models that monitor an application and
guide dynamic changes to it. Adaptation takes into
account varying resources, user mobility, changing user
needs and system faults.
All these efforts justify the importance of ubiquitous
computing architectures. At the same time, they
manifest that there is no global unified framework
addressing all needs. Rather, the majority of these
architectures concentrate on one or more application
specific goals. While these goals (e.g., context-compo-
sition, coordination of devices, orchestration of
modalities, and adaptation to context) are common to
the majority of the pervasive computing systems, the
respective solutions emphasize different objectives,
depending on the nature of the target pervasive envi-
ronment. A fundamental limitation of these architec-
tures is that they assume that all context-aware
components (e.g., perceptual components, situation
modeling middleware) are contributed by the same
technology provider, which is not always the case.
Large-scale collaborative projects built demonstrators
from a variety of components that are contributed by
different providers. This is also in-line with the envis-
aged nature of emerging pervasive computing envi-
ronments, which will be composed by a variety of
distributed heterogeneous components from multiple
vendors. Another drawback of these architectures is
that they were built upon legacy technologies and
therefore do not benefit from emerging technologies
(e.g., the semantic web).
In this paper, we introduce an architecture for
ubiquitous, context-aware computing, which can
greatly facilitate the assembly and deployment of
pervasive applications based on components contrib-
uted by different technology providers. We character-
ize this architecture as ‘breadboard’ since it allows
flexible addition and replacement of hardware (e.g.,
sensors, actuators) and software elements. This archi-
tecture comes with a framework implementation of
194 Pers Ubiquit Comput (2007) 11:193–212
123
middleware components enabling the integration of
sensors, devices, actuators, perceptual components,
context modeling elements, as well as other informa-
tion fusion components. Special emphasis is laid on
integrating perceptual components, multi-modal
interfaces and situation recognition middleware. The
vendor-independent integration of other elements
(e.g., sensors, actuators, and devices) leverages existing
standards (e.g., the IEEE 1394 for video sensors) and
research results.
The motivation for implementing systems based on
this architecture was our participation in the Comput-
ers in the Human Interaction Loop (CHIL) project
[14], which is a one of the most prominent European
research initiatives in the areas of pervasive computing
and multi-modal interfaces. CHIL brings together
several research labs and therefore builds services that
leverage numerous perceptual technology components.
CHIL perceptual technologies comprise a rich collec-
tion of 2D-visual components, 3D-visual perceptual
components, acoustic components, audio–visual (i.e.,
multi-modal) components, as well as output perceptual
components like multi-modal speech synthesis and
targeted audio. As a result, CHIL services provide
ground for demonstrating the benefits of the intro-
duced breadboard architecture. Note that CHIL ser-
vices are built within prototype smart spaces, which
serve as test-beds for perceptual, multi-modal, and
middleware development. The authors operate a pro-
totype smart space comprising several sensors and
actuating devices, where several pervasive applications
have been built. Our experiences from building pro-
totype pervasive services provide a first class manifes-
tation of the introduced middleware framework.
While the introduced architecture focuses on the
integration of diverse sensors and perceptual compo-
nents, it also provides most of the features of other
middleware infrastructures for pervasive computing.
These features include directory services, coordination
of devices, autonomic communication, as well as a
variety of non-functional features. In several areas
(e.g., in the area of effective resources discovery) the
introduced architecture achieves significant improve-
ments over conventional approaches. We present all
these features, along with a performance assessment of
the middleware components comprising the architec-
ture. Note that the overall architectural concept takes
into account the authors’ work on more specific aspects
of a pervasive computing architecture (e.g. [15–19]).
The rest of the paper has the following structure:
Overall architecture framework section presents the
overall architectural concept for integrating sensors,
perceptual components, actuating services, information
fusion elements. Sensors and perceptual components
section illustrates how the architecture can flexibly
incorporate sensors and perceptual components, out-
lining also the middleware required for this incorpo-
ration. Our context modeling approach is described in
situation modeling and context-awareness section,
along with the middleware components that implement
it. Intelligent actuating services section, describes the
interfacing to actuators and actuating services. It also
elaborates on our approach for intelligent coordination
and orchestration of devices. Service logic and user
front-end section, describes the implementation of the
service logic executed by a ubiquitous computing
application, while Fault-tolerance and system man-
agement section describes non-functional features (i.e.,
scalability, reliability, and management of the archi-
tecture). Prototype services and applications section
presents prototype applications that leverage the
architecture and illustrates its tangible advantages.
Moreover, it presents some evaluation results targeted
to the approval of the Memory Jog. Performance
measurements section discusses performance issues
based on measurements that have been derived in the
context of a realistic operation of the system. Finally,
conclusions section draws the basic conclusions from
this work.
2 Overall architecture framework
Figure 1 presents the overall architectural framework
for pervasive services in smart spaces along with its
core middleware components. This framework can be
viewed as a composition of three tiers that operate on a
layered fashion in the sense that each tier exploits in-
put from the adjacent ones. The three tiers are as fol-
lows:
• A sensors tier comprising the transparent sensing
infrastructure of the smart space. Sensors collect
signals from the environment. Signals can accord-
ingly be processed to extract context.
• A tier of perceptual components based on signal
processing algorithms. Perceptual components ex-
tract context cues, typically by processing audio and
video signals. Context derived from perceptual
components relates primarily to the identity and
location of people and objects.
• A tier of agents that model and track higher-level
contextual situations, while at the same time
incorporating the service logic of the pervasive
computing services. Context tracking allows for
implementation of non-obtrusive service logic.
Pers Ubiquit Comput (2007) 11:193–212 195
123
Furthermore, implementing the service logic re-
quires access to, and management of, actuating
services. The use of agents in the service layer is
merely an implementation choice that offers several
advantages; other sorts of distributed systems also
could have been used as well. Note that all agents
are implemented based on the JADE [21] platform
and communicate with each other using standard-
ized Foundation for Intelligent Physical Agents
(FIPA) [22] messages/primitives.
Communication and data exchange between these
layers is achieved based on specialized third-party
middleware components, enabling distributed high-
performance access to sensor input, as well as to per-
ceptual components outputs. Specifically, perceptual
components access sensor output through the NIST
Smart Flow middleware [23]. The latter undertakes the
distributed high-performance transfer of streams to
any computer within the smart space, which facilitates
decentralized processing of sensor streams by multiple
consumers. Distributed processing is essential for two
main reasons:
• To allow more than one algorithm to leverage a
particular sensor stream. For example, a video
sequence may be needed by a body tracking
algorithm and a visual personal identification
method.
• To distribute the processing load to more than one
computers. This is particularly important in the case
of computationally demanding real-time multime-
dia processing.
Information exchange between the perceptual
components layer and the agent-based services layer is
achieved based on IBM’s CHILIX library [14]. CHI-
LIX is a middleware component enabling access to
perceptual components outputs in a language and
platform agnostic manner, based on XML over TCP
transfer of information. The information to be trans-
ferred is not a multimedia stream per se, rather the
result of its processing from a perceptual component.
Pervasive services that are built based on this
architecture feature an information flow as follows:
• Visual and acoustic sensors capture audio–visual
streams, which are transferred to perceptual com-
ponents through the NIST Smart Flow System.
Alternatively, sensors that can directly provide
context (e.g., temperature, motion sensors) and
convey this information to the services tier.
• The perceptual components tier processes audio
and video streams, toward identifying and locating
people and objects within the smart space.
• The services layer maintains a high-level context
model, which is assembled based on elementary
contextual information received from either sensors
Fig. 1 High-levelarchitectural framework andmiddleware infrastructure forpervasive computing servicesin smart spaces
196 Pers Ubiquit Comput (2007) 11:193–212
123
or perceptual components through CHILIX. The
service logic is triggered upon identification of
particular contextual states, as well as based on user
requests. The service logic execution is likely to
engage actuating services of the smart space.
In such heterogeneous environments, directory ser-
vices are a major concern. Transparent distributed
communication between all these components, presup-
poses mechanisms for locating components and ser-
vices. While numerous directory mechanisms exist (e.g.
[24–28]), we argue that they are not appropriate to deal
with the diversity of a pervasive computing environ-
ment. Technologies such as Universal Plug n’ Play
(UPnP) [24], Service Location Protocol (SLP) [25], and
Universal Description, Discovery, and Integration
(UDDI) [26] provide mechanisms for registering and
discovering resources and services. However, these
mechanisms are not particularly tailored to the range of
information and components that are essential to per-
vasive services. For example, UDDI and SLP are merely
service oriented, while UPnP is device oriented. The
limitations of these technologies are largely due to a lack
of standardized descriptions for perceptual compo-
nents, context models, and actuating services. Moti-
vated by these limitations, we have developed a
directory mechanism leveraging a knowledge base,
which is based on web ontologies. In particular, the
ontologies include all the concepts associated with
hardware, middleware, software, as well as with the
physical objects (people, artifacts) of the smart space.
Our web ontologies are described based on the Web
ontology language (OWL) [29] and accessed through a
variety of distributed access techniques [17–19].
Using the ontology-based directory services mecha-
nisms we can more intelligently answer queries, as is
often required in the scope of context-aware, human-
centric, ubiquitous computing services. The intelligence
lies in the ability to infer information from existing sets
of meta-data according to current context and user
intention. As an example, given the number of different
sensors in a smart space, a service may need to acquire a
reference to the best-view camera for a given situation,
e.g., the camera facing the door for recognizing a person
entering a room. The ability to answering such queries
requires inferring information based on existing meta-
data, based on a reasoning procedure. External rule-
based reasoning systems can access the ontological data
of our OWL-based directory service (e.g. [30]), which
overall results in intelligent directory services. Follow-
ing paragraphs elaborate on the structure, functionality
and implementation details of the main components of
the architectural framework.
3 Sensors and perceptual components
3.1 Sensor virtualization and sensor proxies
Smart spaces applications need to interface with sen-
sors, to acquire and process sensor streams. This
interfacing is based on Application programming
interfaces (APIs) facilitating access to the low-level
capabilities of the sensors. Given the large number of
different sensors and vendors, developers of ubiquitous
computing applications have to deal with a rich set of
diverse APIs (Fig. 2).
To facilitate seamless addition, replacement and
integration of sensors, our architecture provides a set
of vendor-independent APIs that alleviate this diver-
sity. Specifically, an hierarchy of sensor-related classes
and their operations have been defined. This class
hierarchy includes vendor-independent classes for
acoustic and video sensors comprising the sensing
infrastructure of the CHIL smart rooms. Moreover,
classes for other types of sensors have been defined as
well. Vendor-independent operations offer virtualized
access to the capabilities of the underlying sensors.
Any component (e.g., perceptual algorithm, signal
processing algorithm, NIST Smart Flow client) inter-
facing with a certain sensor type exploits the same
single API, regardless of the sensor vendor. As a result,
developers need only to learn and leverage a simple
API to interface with each sensor type. Current APIs’
abstractions cover all the sensors available in the CHIL
smart rooms, which include: active cameras with pan,
tilt and zoom (PTZ cameras), digital IEEE1394
Fig. 2 Sensor virtualization
Pers Ubiquit Comput (2007) 11:193–212 197
123
(Firewire) cameras, microphones, microphone clusters,
and NIST Mark III 64 channel microphone arrays [31].
However, the higher-level abstractions are appropriate
for other sensors as well [e.g., Radio-Frequency Iden-
tification Systems (RFIDs), temperature sensors]
(Fig. 3).
Pervasive computing environments tend to be vol-
atile, since components, devices and services can
dynamically join or leave. This dynamism is particu-
larly evident in mobile and nomadic computing
environments, which involve roaming users (e.g. [32]).
In-door environments (such as smart spaces) tend to
be less dynamic, since sensors and devices join or
leave at higher time-scales. The architecture can deal
with this dynamism at the hardware level, through
dynamic discovery of sensors and devices. This is
achieved, thanks to a set of middleware components
(i.e., JADE agents) called sensor proxies. A different
proxy component has been developed for each dis-
tinct sensor class, with a view to managing the sensor.
Accordingly, each sensor instance is associated with
its proxy instance. Proxies register the sensors to the
knowledge base and can be used to convey manage-
ment commands to the sensors (e.g., changing the
frame rate of a camera or the sampling frequency of a
microphone). Proxies support the above-mentioned
virtualized API, providing access to the capabilities of
the sensors. Note that this virtualization requires that
each sensor is wrapped to comply with this API. Once
such a wrapper is implemented, a sensor can be
seamlessly integrated to the overall ubiquitous com-
puting infrastructure.
3.2 Perceptual components APIs
Perceptual components are the primary consumers of
sensor streams. They are implemented as NIST Smart
Flow clients to allow high-performing distributed ac-
cess to sensor output. Perceptual technologies apply
novel algorithms on audio and video streams to extract
several forms of context pertaining to people and ob-
jects within the smart room. CHIL technologies in-
clude person localization and tracking, body detection,
head orientation, face detection and recognition, ges-
ture/posture recognition, head and hand tracking,
pointing gesture recognition, speech recognition
(including far-field), source localization, speech detec-
tion, speaker identification, acoustic and visual emo-
tion recognition, acoustic event classification, activity
recognition, and lips observation. A presentation of
these components is out of the scope of this paper.
Interested readers may access [33–36] for a description
of perceptual components available within our smart
room.
Several of these components are provided by more
than one technology sites within the CHIL consortium,
which creates the need to deal with heterogeneity at
the perceptual components level. To alleviate this
heterogeneity, APIs pertaining to each one of the
component types are defined. Perceptual components
APIs are structured as XML schemas and define input
and output parameters for each component. As soon as
a technology provider exposes this API for its (body
tracker in this case) component, the latter can be
integrated to the overall ubiquitous environment
Fig. 3 Perceptual components virtualization and APIs
198 Pers Ubiquit Comput (2007) 11:193–212
123
within minimal customization effort. To the best of our
knowledge this is a first-of-the-kind effort, to stan-
dardize inputs and outputs of perceptive algorithms.
Based on perceptual component APIs, consumers of
the contextual information produced by any perceptual
component can tolerate dynamic switching of tech-
nology providers. Consumers will use the same API
over the CHILIX middleware. In this way, the archi-
tecture acts as a breadboard, since a variety of per-
ceptual components can be integrated in a plug ‘n play
fashion. In addition, perceptual components are
dynamically registered, discovered and managed,
based on a proxy mechanism. Similar to the sensors
case, implementing a proxy for each perceptual com-
ponent provides a uniform way for discovering,
accessing and managing the component.
The seamless integration and operation of percep-
tual components to the overall ubiquitous computing
environment depends not only on an abstract API, but
also on a wide range of environmental parameters and
settings. For example, the performance of image pro-
cessing algorithms (being the cornerstone of video-
based body tracking, face detection, and face identifi-
cation) depends highly on the lighting conditions of the
smart space. Nevertheless, developing components that
are autonomous and adaptive to environment (e.g.,
through adaptive foreground/background segmenta-
tion) falls in the technology provider’s jurisdiction and
is therefore, out of the scope of the architecture.
4 Situation modeling and context-awareness
4.1 Network of situations approach
Perceptual components provide elementary context
cues. These mainly relate to the status of people and
objects within the smart space. Sophisticated context-
aware applications are likely to rely on composite
contextual states, which ask for composing perceptual
components outputs. To this end, the architecture of-
fers a context modeling approach that relies on the
network of situations paradigm [15, 37]. According to
this paradigm, the contextual states of interest are
structured into a (state-diagram) graph (ST, ed), where
the nodes denote the target states and the arcs the
possible transitions between contextual states. Specifi-
cally, edge edij denotes that it is possible to reach state
Sj from state Si.
Contextual states may be arbitrarily complex in
terms of their defining cues. The situation model
is accompanied by a truth table, which depicts the
underlying combination of perceptual component
values (i.e., outputs) that triggers each one of the
composite contextual states. To formally specify how
the situation transitions occur, assume that a ubiqui-
tous computing environment is supported by m com-
ponents having k1, k2,..., km outputs respectively and let
k = max(k1, k2,..., km). Without loss of generality, we
can represent the outputs produced by all perceptual
components at a given time instant (t) using the matrix:
POUTðtÞ ¼ pijðtÞ� �
; ð1Þ
where 1 < i < m and 1 < j < k.
It is assumed that pij(t) = 0 for j > ki, since there are
perceptual components that possess less than k-out-
puts. As a result, Pout (t) contains the measurements
observed by the perceptual components outputs at
time instant t.
For each situation STl (1 < l < n) targeted by a sit-
uation model, we define a matrix Sl comprising the
values of the perceptual components that define Sl., as
follows:
Sl ¼ fsijg; 1\i\m; 1\j\k; ð2Þ
where sij „ 0 if the jth output of the ith perceptual
component contributes in the triggering the state Sl
and sij = 0 otherwise. Toward associating the non-zero
sij-values with the observed outputs pij(t), we perform
an element-wise multiplication of Pout (t), with the
following matrix:
Al ¼ faijg; 1\i\m; 1\j\k; ð3Þ
where aij = 1 if sij „ 0 and aij = 0 otherwise (i.e.,
sij = 0). The result of the element-wise multiplication
will produce a Pl matrix filtering the observed outputs
in a way that only values defining the state STl are
retained:
Pl ¼ aij � pijðtÞ� �
; 1\i\m and 1\j\k: ð4Þ
STl occurs when all the elements of the matrices Pl
and Sl coincide, i.e.,
Sl � Pl ¼ sij � aij � pij
� �¼ Omk; ð5Þ
where O corresponds to the matrix having all its
elements equal to zero. In practice, it is rare to achieve
a simultaneous total agreement between target values
and observed values. Therefore, the triggering of the
situation may be defined as the case when the elements
of the two matrices almost coincide, thus allowing the
perceptual outputs to slightly fluctuate over the target
values:
Pers Ubiquit Comput (2007) 11:193–212 199
123
Sl � Pl ¼ sij � aij � pij
� �¼ El ¼ eij
� �; ð6Þ
where either eij fi 0.
In order for situation (STl) to be triggered, condition
(6) has to be fulfilled (for some eij-values), while at the
same time the situation model has to be on a state
(STp) that allows transition to Sl. Therefore,
STp! STlfoccurs whenever Sl�Pl¼El and edpl¼ 1:
ð7Þ
Note that this network of situations approach is
general and applicable not only to perceptual compo-
nents outputs but also to the more general class of
observations, which may also include sensor signals
(e.g., such as a temperature indication or a motion
sensor’s output). In practice, the matrix Sl is likely to
be very sparse, which can greatly simplify equation (6).
It is also noteworthy that Eqs. 1–6 assume numeric
outputs for perceptual components. While this may
sound limiting, it is in general, possible to encode
several other value domains as numbers.
4.2 Sample situation models
Figure 4 depicts a sample situation model for tracking
activities within a meeting. The situation model con-
sists of five states corresponding to the commencement
of a meeting, the start of a presentation during the
meeting, a question on the presentation, the end of a
presentation and the end of a meeting. The arcs in
Fig. 4 denote the possible transitions. For example, a
question can only occur while a presentation is in
progress, since there is no means to reach state S3
unless the model is in situation S2. Table 1 shows how
particular situation states are triggered based on
underlying perceptual components. NIL denotes the
starting state. As an example, the start of the meeting
(i.e., the transition NIL–>S1) occurs when an expected
number of people are speaking very close to the table.
Given a number of perceptual components (i.e.,
TablePeopleCount, WhiteBoardPeopleCount, Speech
Activity Detection, and Acoustic Localization) and
their APIs, Table 1 can be mapped to Eq. 6 in a
straightforward way. Nevertheless, the mapping is in-
general, application specific since the elements of the
matrices E1, E2,...,E5 (see Eq. 6) are likely to be de-
fined based on the problem at hand.
Figure 5, depicts an augmented version of the
meeting tracking situation model. In particular, this
situation model is enhanced to track people entering
(i.e., person entered) and leaving the meeting (i.e.,
person left). Moreover, it tracks situations, where a
person returns to a meeting that is in progress (i.e.,
person returned). Accordingly, Table 2 associates the
extended model with underlying perceptual compo-
nents. Extending situation models to capture/track
additional states, is one direction toward the scalability
of the network of situations solution. The architecture
makes it possible to use more than one situation
models for context tracking.
4.3 Situation model implementation
From an implementation perspective, situation models
are described in XML format, and loaded by the Sit-
uation Modeling Agent (SMA) within the JADE agent
society during the startup sequence. SMA access the
underlying perceptual components through their APIs
(over CHILIX). Accordingly, they keep track of the
current situation state, as well as whether Eq. 7 is ful-
filled for some transition. To this end, they leverage
perceptual component observations along with XML-
based situation model definition. Whenever Eq. 7 is
met, the SMA advances the tracked situation state to
the next one. Note also that the SMA includes data
structures holding information about people, objects
and events that are tracked by the situation model. As
information from perceptual components is fused to-
ward the SMA, these data structures are filled with the
appropriate information.
5 Intelligent actuating services
Actuating services provide the means for the smart
space to communicate with end users, for example by
outputting a voice prompt or displaying a message. TheFig. 4 Situation model tracking activities within a meeting
200 Pers Ubiquit Comput (2007) 11:193–212
123
proposed architecture supports dynamic registration,
discovery and invocation of services, based on the al-
ready described proxy mechanism. Just like perceptual
components and sensors, actuating services come with
a proxy that allows them to be registered in the
Knowledge Base and accordingly discovered, invoked,
and managed. Furthermore, the architecture allows for
implementation of ambient services that intelligently
select the optimal actuating mechanism in a given
context. This implementation is performed at the agent
layer. Specifically, for any target intelligent service, a
special agent incorporating ambient service selection
logic needs to be implemented. This (JADE) agent has
the means to intercept incoming requests for actuating
services and to delegate them to the appropriate ser-
vice proxy among a set of actuating services that are
appropriate for the requested service. The service
developer has to extend this agent and provide the
service selection algorithm. The following paragraphs
describe two characteristic examples of intelligent
actuating services that are operational within our smart
room.
Table 1 Mapping perceptualcomponent outputs tosituation transitions
Situation transition Combinations of perceptual components outputs
NIL fi S1 TablePeopleCount = n (i.e., n people in table area)and Speech Activity Detection = 1
S1 fi S2 WhiteBoardPeopleCount = 1 (1 in speaker area),TablePeopleCount = n – 1 (n – 1 in table area)
Acoustic localization = within the whiteboard areaS2 fi S3 Acoustic localization = within the table areaS3 fi S2 Acoustic localization = within the speaker areaS2 fi S4 Face Detection (speaker area and table area)S4 fi S2 WhiteBoardPeopleCount = 1 (1 in speaker area),
TablePeopleCount = n – 1 (n – 1 in table area)Acoustic localization = within the whiteboard area
S4 fi S5 TablePeopleCount = 0 (i.e., 0 people in table area)
Fig. 5 Situation modeltracking activities within ameeting
Pers Ubiquit Comput (2007) 11:193–212 201
123
5.1 Intelligent display
The intelligent display service selects the optimal
display device according to the location of the target
person(s), within a smart room. A smart space may
have more than one means (e.g., devices, projectors,
and smartboards) to visually display messages as well
as presentations, posters, etc. The goal of the intel-
ligent display services is to select the device that is
more convenient for the room participants. This is
accomplished as follows: whenever a display service
is requested, a special display agent intercepts this
request and examines it in relation to the current
context. The context of interest includes the location
and orientation of participants, which are tracked by
appropriate perceptual components. Specifically, body
trackers provide information about people location,
while a special video processing component provides
a metric of ‘faceness’ for the target participants [38].
Based on this information, a display selection algo-
rithm has been implemented and incorporated within
an appropriate agent, aiming at choosing the display
that best suits the location and orientation of par-
ticipants within the smart room. The algorithm at-
tempts to provide a satisfactory view for as many
participants as possible.
In obtaining contextual information (number of
participants, locations of participants, orientation of
participants) the display agent accesses the situation-
modeling agent. The latter collects and maintains this
information through body trackers and ‘faceness’
modules, in accordance with the already presented
architectural concepts.
5.2 Intelligent meeting recording: best camera
selection
The architecture offers the capability of storing camera
streams. This is based on a recording service, which can
be instantiated for any of the cameras within the smart
space. Storing camera streams is a key prerequisite to
recording meetings, lectures and presentations within
the smart space. A realistic meeting recording service
is expected to operate like an automated intelligent
cameraman. In particular, an ambient recording should
select the optimal camera view based on the location
and orientation of the participants, as well as on their
activities and role within the group interaction (e.g.,
whether a person is a presenter or speaker) (Fig. 6).
We have implemented such an intelligent meeting
recording service based on a Meeting Recording Agent
that continually monitors the location and orientation
of speakers toward storing the optimal camera stream.
Thus, the meeting recording agent combines informa-
tion from the body tracking, acoustic source localiza-
tion and ‘faceness’ components. This combination is
performed at the SMA, which continually notifies the
Meeting Recording Agent on changes of the orienta-
tion and location of speakers. Accordingly, the
Recording Agent communicates with other JADE
agents controlling the storage services for the various
cameras. A more thorough description of this record-
ing service, as a stand-alone ambient service can be
found in [38].
Table 2 Mapping perceptualcomponent outputs tosituation transitions
Situation transition Combinations of perceptual components outputs
NIL fi S1 BodyTracker = n + 1 (where n is previous number of people in the room)S1 fi S2 FaceID = i (i does not exist as a current ID in the system)S2 fi S4 BodyTracker = n AND TablePeopleCount = NS4 fi S5, S6 fi S5 TablePeopleCount = n – 1 AND WhiteBoardPeopleCount = 1 AND
Acoustic Localization = within the whiteboard areaS5 fi S6 TablePeopleCount = n – 1 AND WhiteBoardPeopleCount = 1 AND
Acoustic Localization = within the table areaS8 fi S9 BodyTracker = 0S4, S5, S6, S7 fi S8 BodyTracker = n – 1 (n is the current number of people in the room)S5, S6, S7 fi S1 BodyTracker = n + 1 (n is the current number of people in the room)
Fig. 6 Elements of the meeting recording service
202 Pers Ubiquit Comput (2007) 11:193–212
123
6 Service logic and user front end
The service logic is implemented at the agent society
tier of the architecture. Typically, service logic is trig-
gered either on end users’ demand or wherever a
contextual change occurs. In the former case, end users
need to provide explicit input to the pervasive system,
which is in line with the vast majority of legacy com-
puting applications. In the later case, the context-aware
triggering of the service logic can be based on situation
state transitions, as the latter are modeled in situation
models. Thus, upon the identification of a transition,
some piece of service logic is executed. No matter the
triggering mechanism, the service logic will instigate
requests to available actuating services. These actuat-
ing services will ensure ambient natural context-aware
interaction with the end-users. In the scope of several
pervasive applications however, end-users are likely to
interact with conventional terminal devices (such as
Laptops, PDA, smart phones). Apart from modular-
izing the service logic, the introduced architecture al-
lows for implementation of service specific graphical
user interfaces (GUIs), which may run on different
platforms and terminal devices.
The agent tier of the architecture is a multi-agent
system (depicted in Fig. 7) consisting of the following
agent types:
• Device desktop agent, which implements the user
interface required for accessing the ubiquitous
services. A ‘pluggable’ mechanism (supported
based on the ‘pluggable’ behaviors feature of the
JADE platform) allows the user interface to be
customized to the particular ubiquitous computing
service.
• Device agent, which enables different devices to
communicate with the framework (Fig. 8).
• Personal agent, which constitutes the proxy of the
end user in the agent world. The personal agent
conveys user requests to the agent manager (AM),
which redirects them to other agents that can
handle them appropriately. It maintains the user’s
profile in order to personalize the services to the
end user.
• Basic services agents: these agents incorporate the
service logic of basic services, which are tightly
coupled with the infrastructure of the smart space.
Basic services include the ability to track composite
situations, as well as the control of sensors and
actuators. Tracking of composite situations is per-
formed through the SMA based on the network of
situations context modeling approach. Control of
sensors and actuators is performed through the
Smart Room Agent (SRA). Furthermore a Knowl-
edge Base Agent (KBA), allows the agents of the
framework to dynamically access information on
the state of the components of ubiquitous comput-
ing environment (e.g., sensors, actuators, percep-
tual components), through a Knowledge Base
Server that is supported by the ontology manage-
ment system.
• Agent manager, which allows the system to be
dynamically augmented with additional Service
Agents. Moreover, the AM resolves requests for
agent services, through re-directing them to appro-
priate agents that match the request.
Moreover, the multi-agent system comprises service
agents that implement the non-obtrusive service logic
of the various context-aware services. Each pervasive
computing service is therefore implemented as a ser-
vice agent and accordingly plugged into the framework
based on the JADE ‘pluggable’ behaviors mechanism.
In the scope of the CHIL project, several service
agents corresponding to various ubiquitous computing
services are implemented and integrated into this
framework.
The service logic for a given service is implemented
as a separate Service Agent, which is modular and
‘pluggable’ to the overall architectural framework.
From an implementation perspective, ‘pluggability’
exploits the pluggable behaviors feature of the JADE
environment. Note that ‘pluggability’ supports the
overall concept and goals of the breadboard architec-
ture at the service logic level. Service agents interact
with the SMA, to access contextual information. Con-
text triggering is achieved through conveying state
transitions from the SMA to the Service Agent.
Moreover, the service agent may interact with agents
implementing the user front end for different terminal
devices. To facilitate implementation of Service
Agents (i.e., of the service logic), the architecture
provides template agents that can be directly extended
to incorporate the target service logic. Templates
facilitate the process of access contextual information
and obtaining contextual notifications, allowing the
service developer to emphasize on the service logic
rather than the underlying middleware.
The GUI implementation is incorporated into spe-
cial agents called Device Agents. Different types of
device agents exist for the various terminals (e.g.,
PDADeviceAgent, NotebookDeviceAgent). Device
Agents receive user requests through GUI actions and
convey them to the service logic. Furthermore, they
receive requests from the service logic for GUI inter-
action with the end user (through the user front end).
Pers Ubiquit Comput (2007) 11:193–212 203
123
These requests are appropriately filtered from a Per-
sonal Agent (which maintains the user’s profile and
implements personalization). Similar to the Service
Agents’ case, the architecture provides templates for
the Personal Agent and the Device Agents, with a view
to minimizing the service implementation or custom-
ization effort. In this way, reusability at the agent layer
is pushed to the extreme. A more thorough description
of the agent society, including detailed agent descrip-
tions and interactions is contained in [15, 40].
7 Fault-tolerance and system management
The proposed architecture addresses non-functional
requirements relating to the reliability, availability and
ease of management. These sets of non-functional
requirements define softer goals of the target pervasive
services and are not defined as part of use cases and
user requirements. In the sequel, we describe support
of the proposed architecture with respect to fault-tol-
erance, reliability, and system management. The fol-
lowing paragraph presents performance measurements
based on real prototype services that adopt this archi-
tecture.
7.1 Reliability: fault-tolerance
The introduced architecture incorporates recovery
functionalities at the agent layer. This functionality is
applied whenever an agent dies [e.g., due to a hardware
or software (Java Virtual Machine) failure]. The
implementation is based on a controlling agent called
Autonomic Agent Manager (AAM), which has as main
functionality the monitoring of specific agents regard-
ing their status (alive or not) and to act accordingly.
This monitoring is done using a ping-like functionality
(implemented as a JADE behavior), which is initiated
by the AAM to all monitored Agents.
JADE provides functionalities regarding message
exchange among the members of the Agent Society it
supervises. The ping-like behavior initiates an the
transmission of an Agent Communication Language
message (ACLMessage) by the AAM to all monitored
Agents. Upon reception of such a message (sent by the
AAM), the agent replies back with its status. In case of
a significantly delayed response, the AAM considers
that the agent is dead. Upon realizing that the agent
has failed, the AAM attempts to restart the agent on a
given alternative host, which depends on configuration
options. The restarting process distinguishes between
two different methodologies namely stateful and state-
less recovery.
Stateless recovery is initiated to agents, which do not
maintain a state during their execution. Examples of
such agents are all those that convey requests to
actuating services. After the registration of each agent
with AAM, the AAM maintains all the information
needed for re-starting all the agents it supervises. This
information consists of the class file and the name of
agent. Following the restart process, the ping behavior
Fig. 7 Anatomy of the multi-agent system comprising the agenttier
Fig. 8 Delivery of information to multiple terminal devicesthrough the agent layer
204 Pers Ubiquit Comput (2007) 11:193–212
123
responsible for listening for ping messages and
answering back with an ‘alive’ response is also started.
Stateful recovery is required for agents that main-
tain state. Characteristic example of such agents is the
SMA, which keeps track of the situation models. In
recovering such agents, we monitor and store their
state to a Relational Database Management System
(RDBMS), thus allowing the state to be retrieved upon
completion of the regeneration sequence. Following
the completion of the restart process (which is similar
to the stateless case), the serialized state is retrieved by
the rejuvenated agent, and becomes the current state.
From an implementation perspective, stateful agents
are stored as serialized objects in the relational data-
base. To this end, we leverage Hibernate [41], which is
a powerful, ultra-high performance object/relational
persistence and query service for high-level systems.
The stateful recovery process is shown in Fig. 9.
7.2 System management
Systems adopting the proposed architecture are in
principle highly heterogeneous and distributed, with no
single centralized control point. Thus, it is fundamen-
tally difficult to build a unified management system for
the whole range of hardware, software, and middle-
ware components comprising the pervasive environ-
ment. To this end, the system management relies on
the disjoint management sub-systems that monitor and
manage individual tiers of the architecture. These sub-
systems are based on tools developed as supplements
to the architecture, as well as on third-party tools.
Third-party tools include the JADE Directory
Facilitator (DF), as well as the NIST Smart Flow
Control Center (SFCC). The DF manages each one of
the agent instance according to sets of unique identi-
fiers. Whenever a new agent is instantiated, it registers
itself to the DF so that all other agents can find it when
and if they need to. This is also shown in Fig. 8, which
depicts that every rejuvenated agent, registers with the
DF. The SFCC provides a management interface used
to instantiate the Smart Flow servers so that perceptual
components access streams in a distributed fashion.
Along with the architecture, we have also developed
the Smart Space Resource Manager (SSRM) [19],
which is a utility enabling monitoring and control of
perceptual components, sensors, and actuating services
available in a smart space (e.g., smart room). Admin-
istrators of the smart space may use the SSRM utility,
to manage a variety of heterogeneous distributed
entities from a single entry point. SSRM has been
implemented as a JADE-based mutli-agent system,
with a twofold objective: (a) to facilitate interopera-
bility between the diverse hardware (i.e., sensing and
actuating devices) and middleware (i.e., perceptual and
actuating components) and (b) to leverage the JADE
access interface to the Knowledge Base Server, along
with the corresponding KBA.
The JADE based proxies of sensors, perceptual
components, devices, and services allow these compo-
nents to be managed by the SSRM. Agent proxies act
as gateways allowing other agents within the architec-
ture to interact with the device. As already outlined,
proxy agents provide a level of abstraction: each proxy
exposes a universal virtualized interface to the agent
framework (i.e., a common interface for sensors,
actuators, perceptual components of the same type).
The building blocks of the SSRM are shown in Fig. 10.
The current implementation of the SSRM can
graphically depict the contents of ontological knowl-
Fig. 9 Stateful agentrecovery based on de-serialization of the agent’sstate
Pers Ubiquit Comput (2007) 11:193–212 205
123
edge base through an easy-to-use interface. In par-
ticular, the SSRM dynamically retrieves the regis-
tered elements and presents them in a GUI.
Moreover, it can also issue control commands to
proxy agents of the cameras. Control commands (e.g.,
changing capture frame-rates, changing zoom levels)
on components and devices are handy in cases where
there is a need to adapt the operation of the devices
(e.g., a camera frame rate), to either a particular
context or to the operational requirements of a per-
ceptual processing component (e.g., a face recogni-
tion or visual tracking algorithm). Thus, the SSRM is
a useful tool that can support several processes en-
tailed in the development and deployment cycle of
ubiquitous computing applications. Monitoring of
sensors, actuating devices and perceptual components
is important for developing and debugging ubiquitous
computing services, since it facilitates identification of
component failures and failing factors. Moreover, it
can facilitate deployment through providing an
overall view of the infrastructure and the perceptive
capabilities of a smart space.
8 Prototype services and applications
Based on the presented architectural framework and
its middleware components, we have developed non-
trivial ubiquitous context-aware services within our
prototype smart space. This prototype smart space
consists of a variety of sensors, devices and perceptual
components, and is therefore used as a testbed for
application development in the areas of pervasive
computing and multi-modal interfaces. The floorplan
of this smart space is depicted in Fig. 11.
The middleware described in the scope of the pre-
vious section has greatly facilitated the development of
sophisticated computing services. It is also noteworthy
that the majority of the middleware components have
been proven reusable across different services.
8.1 Memory jog
The goal of the Memory Jog service is to provide non-
obtrusive assistance to humans in scope of meetings,
lectures, presentations, and conferences occurring
within the smart space. It exploits the architecture
along with its associated middleware components to:
• Identify participants and track their locations with-
in the smart room. Identity and location of partic-
ipants can be displayed within different devices.
• Keeping track of meeting progress based on an
agenda, which is known before-hand.
• Recording the meeting, lecture or presentation
based on the best-camera selection mechanism.
The recording is tagged with meta-data from the
situation model of the service, to allow selective
retrieval and post-processing of the recording.
• Provide context-aware assistance through display-
ing relevant information from past meetings (e.g.,
presentations, URLs). Information from past meet-
ings includes video segments recording through the
recording service.
KnowledgeBase
Sensor,Actuator,
Perceptual Component
Low-level driver(e.g. IEEE1394)
CHILIX JNI
JADE ProxyAgent
(sensor,actuator,
perceptualcomponent)
Knowledge Base Agent'KBAgent'
Smart Space ResourceManager(SSRM)
Console & Control Agent
DiscoveryControl
Register/De-Register/Update
StatusControl
Fig. 10 Building blocks of thesmart space resource manager(SSRM)
206 Pers Ubiquit Comput (2007) 11:193–212
123
• Assist participants by automatically carrying out
casual meeting tasks, such as opening presentations,
e-mail material to the participants and displaying
notes/messages.
The Memory Jog relies on a variety of perceptual
components including face recognition (e.g., to identify
participants), acoustic localization, visual body track-
ing (e.g., to track participant locations), face detection,
and speech activity detection. It also exploits the situ-
ation model depicted in Fig. 4, to follow higher-level
situations such as agenda tracking, question tracking,
meeting commencement, and meeting finish. This ser-
vice exploits actuating services (targeted audio, text-to-
speech, display services) to output information to
meeting participants. Moreover, information is ren-
dered in GUIs, which run both on PCs and PDAs
(Fig. 12).
The architecture framework kept development ef-
fort for the memory jog service to a minimum, despite
its rich functionality. Specifically, implementing the
Memory Jog service was a matter of:
• Configuring the situation model within the SMA,
i.e., producing the XML description for the Situa-
tion Model.
• Defining the information exchange primitives for
the communication within the agents.
• Implementing a Memory Jog service agent that
incorporated the target service logic. This involved
implementing appropriate message handlers for
messages received from the Situation Model and
Perceptual Components, both through the SMAs.
• Implementing the target GUIs within the respective
device agents. This involved again the task of
implementing message-handling logic for informa-
tion primitives stemming from the Memory Jog
agent.
• Augmenting the recording service to store tagged
video, as well as the GUIs to retrieve and display
the tags.
Note that the following steps pertain to the service
logic and the functional features of the Memory Jog
pervasive service. Tedious tasks such as interfacing
with sensors, perceptual components and actuating
services were handled by the architecture’s middleware
components. Moreover, in implementing, testing,
debugging, and deploying the service, we took advan-
tage of Smart Room Resource Manager.
8.2 WHWIWA: what happened while I was away?
The what happened while I was away (WHWIWA)
service targets intelligent working and living environ-
ments, with the aim to provide people with summaries
of events that occurred during their absence from the
smart space. To this end, the service identifies people
and tracks the moments where they enter and leave the
Fig. 11 Floor plan of thesmart room of the Athensinformation technology
Pers Ubiquit Comput (2007) 11:193–212 207
123
smart space. Upon identification of the situation that a
person returns to the smart space, he/she is notified of a
series of contextual state that occurred during his/her
absence. Contextual states are tracked according to the
presented situation modeling approach. Information
can be conveyed to end user through auditory inter-
faces or even a conventional GUI. In its simplest form,
this service is less sophisticated than the Memory Jog,
since the sensors, perceptual components, and actuat-
ing services used for the Memory Jog are significantly
more than those used in the scope of the WHWIWA
service.
The two services can be combined to provide par-
ticipants in meetings or conferences with the option of
obtaining contextual information during short intervals
of absence. This can be supported by the situation
model depicted in Fig. 5. The overall implementation
required a series of implementation steps similar to the
Memory Jog case. Those steps concerned mostly
functional features and FIPA message processing to
realize them, rather than dealing with the low-level
interfacing with sensors, perceptual components, and
actuating services.
While these two service implementations demon-
strate that several components of the architecture can
be reused across services, it does not fully manifest the
importance of the perceptual component APIs. The
value of the APIs lies in that they allow exchange of
perceptual components from different technology
providers. Currently, we have internally tested this
potential for seamless exchange of perceptual compo-
nents through experimenting with different algorithmic
implementations for face identification, body tracking,
and speech activity detection components. Based on
the architecture, perceptual component developers can
work totally independently from service developers, as
long as they respect the APIs for these components.
Following this proof-of-concept validation, we are in
the process of exchanging components with different
technology providers within the CHIL project.
8.3 Evaluation
In evaluating these prototype services, we conducted a
series of six focus groups, involving 25 participants with
diverse backgrounds (students, faculty, technical peo-
ple, management staff). Some participants found indi-
vidual features (e.g., the meeting recording) more
appealing than the whole service. Many participants
expressed concerns about privacy issues, while others
thought that some features are not practical in small
smart rooms (e.g., people identity and location).
Reactions also varied for different groups. Neverthe-
less, all groups found several useful features in both the
Memory Jog and the WHWIWA services. The latest
remarks, along with the technological advances in
context-aware applications and smart spaces, lead us to
believe that smart space applications are likely to move
soon from the research labs to the market. In fact, we
believe that smart space applications will form the
second wave of enterprise-pervasive computing appli-
cations, following the widespread RFID applications
[42–43]. This is also evident from the numerous in-
stances of context-aware services for smart spaces,
which have been developed in the scope of deploy-
ments in smart homes, smart conference rooms and
systems for ambient assisted living (e.g. [43–44]). The
proliferation of such services will be greatly facilitated
by middleware components and libraries such as those
presented in this paper.
Fig. 12 Desktop and PDAversions of the memory jogservice GUI
208 Pers Ubiquit Comput (2007) 11:193–212
123
9 Performance measurements
While the presented middleware components facilitate
integration in a distributed multi-vendor environment,
they also incur a slight performance overhead. Wrap-
ping components through uniform interfaces implies
additional overhead in message passing and processing
across the sensing, perceptual processing, situation
identification, and service invocation chain. Given the
real-time nature of various pervasive systems, this
overhead may be essential. To assess this additional
overhead, we conducted a set of performance mea-
surements, while the Memory Jog service was in
operation.
Figure 14 depicts the evolution of the delay for
conveying contextual information from the perceptual
components tier to the agent tier, a sequence, which is
shown in more detail in Fig. 13. Specifically, the delay
is measured over body tracking messages sent from the
body tracker to the SMA. The delay figures/measure-
ments in Fig. 14 include the transfer network delay, the
processing/parsing delay at the CHILIX middleware
layer, as well as the processing delay at the situation-
modeling layer (i.e., the corresponding agent)1. This
aggregate delay has been seen in all cases below 1 s,
which is more than tolerable for the nature of the
applications at hand. Indeed, processing and conveying
body tracking messages to the service logic in less than
1 s, results in a real-time responsiveness in recognizing
situation states and triggering service actions.
The delay appears to be almost uniformly distrib-
uted among three different levels (i.e., 200, 400, and
600 ms), based on an incremental escalation of the
delay. The increments are due to the fact that the body
tracker messages at hand carried information about all
the participants of the memory-jog supported meeting.
Thus, as more and more participants join the meeting,
the body tracker produces and conveys a larger piece
of information. As a result, the delay is grows almost
analogous to the number of people participating in the
meeting. This is also shown in Fig. 15, which demon-
strates that the size of the messages exchanged be-
tween perceptual components and the agent tier
through CHILIX increases as the number of meeting
participants increases. Figure 15 shows also that the
message sizes pertaining to the FIPA communication
ontology increase also as a result of an increasing
number of meeting participants. Note that the
increasing number of participants increases the amount
of information that has to be processed and transmitted
through the overall distributed system. Thus, the
overall delay is inevitably increased.
Figure 16 depicts also the delay within the service
tier (i.e., the JADE layer) of the system. This delay is
again within acceptable levels (less than 0.4 s). In all
cases, absolute processing figures are only indicative,
given that processing time depends heavily on the
processing capacity of the hosting machine. Neverthe-
less, these indicative figures manifest that the archi-
tecture does not constitute a performance bottleneck
for the overall system.
10 Conclusions
Dealing with the heterogeneity and distribution of
software, middleware, and hardware components is a
key challenge in pervasive computing systems. In this
paper, we have presented an architectural framework
and a set of middleware components facilitating inte-
gration of sensors, actuators, perceptual components,
as well as sophisticated context acquisition and mod-
eling. The architecture provides flexibility in managing
and coordinating the various components, along with
their interactions. It operates like a breadboard, en-
abling seamless addition and removal of hardware,
software, and middleware components. To this end it
leverages a knowledge base registry, which is sup-
ported by semantic web technologies and ontological
management systems.
Fig. 13 The processingcomponents in the CHILsystem
1 Agent level processing was supported by a dual-processor(2.4 GHz) Zeon machine.
Pers Ubiquit Comput (2007) 11:193–212 209
123
One of the key features of this architecture is that it
incorporates schemes for modeling complex contextual
states. These schemes combine perceptual components
outputs toward recognizing situations, along with
transitions between contextual states. Apart from
sophisticated context modeling, the architecture en-
ables the implementation of intelligent actuating ser-
vices. Prominent examples of such services are an
intelligent display service and a meeting recording
service. The intelligent display service displays infor-
mation in devices based on the orientation of the
audience. Similarly, the meeting recording services
incorporates algorithms that select the best camera for
a given orientation of a speaker within the smart room.
Note that the architecture provides also a number of
reliability and system management features. The latter
facilitate development, debugging and deployment of
pervasive computing applications.
A number of real-life prototype applications have
been developed based on this architecture. The
development of these applications has been carried
out with minimal development effort. The major part
of this effort was allocated to producing the service
logic, since the architecture facilitated the integration.
Message size
020406080
100120140160180200
1 2 3 4
Number of people
Siz
e(b
ytes
)
CHiLiX
Ontology
Fig. 15 Size of messages exchanged in CHILiX and betweenagents (ontology) for varying numbers of meeting participants
Fig. 16 Delay contributed bythe agent layer
Fig. 14 Delay overheadevolution during a meeting
210 Pers Ubiquit Comput (2007) 11:193–212
123
Performance measurements demonstrated that the
integration and structural benefits of the architecture
come with a decent performance overhead. Overall, we
have introduced an architecture that offers several un-
ique features and benefits over existing architectures
developed in the scope of previous large scale ubiqui-
tous computing related projects. The main distinguish-
ing features of our architecture lie in its breadboard
nature toward multi-vendor components, as well as in
use of intelligent directory services.
Acknowledgments This work is part of the FP6 CHIL project(FP6-506909), partially funded by the European Commissionunder the Information Society Technology (IST) program. Theauthors acknowledge valuable help and contributions from allpartners of the project, especially of partners participating inWP2, which defines the software architecture of the project.
References
1. Weiser M (1991) The computer for the 21st century. Sci Am265(3):66–75
2. Dey AK (2001) Understanding and using context. PersUbiquitous Comput J 5(1):4–7
3. Dey AK, Salber D, Adowd GD (2001) A conceptualframework and a toolkit for supporting the rapid prototypingof context-aware applications. Hum Comput Interact J16:97–166
4. Want R, Hopper A, Falcao V, Gibbons J (1992) The activebadge location system. ACM Trans Inform Syst 10(1):91–102
5. Smailagic A, Siewiorek D (2002) Application design forwearable and context-aware computers. IEEE PervasiveComput 1(4):20–29
6. Johanson B, Fox A, Winograd T (2002) The interactiveworkspaces project: experiences with ubiquitous computingrooms. IEEE Pervasive Comput Mag 1(2):67–75
7. Yau SS, Karim F, Yu W, Bin W, Gupta SKS (2002) Re-configurable context-sensitive middleware for pervasivecomputing. IEEE Pervasive Comput (joint special issue withIEEE Pers Commun Context-Aware Pervasive Comput)1(3):33–40
8. Ponnekanti SR, Johanson B, Kiciman E, Fox A (2003)Portability, extensibility and robustness in iROS. In: Pro-ceedings of the 1st IEEE international conference on per-vasive computing and communications PERCOM. IEEEComputer Society, Washington, pp 11–19
9. Coen M, Phillips B, Warshawsky N, Weisman L, Peters S,Finin P (1999) Meeting the computational needs of intelli-gent environments: the Metaglue system. In: Proceedings ofthe 1st international workshop on managing interactions insmart environments. Dublin, Ireland, pp 201–212
10. Saif U, Pham H, Paluska JM, Waterman J, Terman C, WardS (2003) A case for goal-oriented programming semantics.In: Workshop on system support for ubiquitous computing(UbiSys’03), 5th international conference on ubiquitouscomputing (Ubicomp 2003). Seattle, 12 October 2003, pp 74–83
11. Brumitt B, Krumm J, Meyers B, Shafer S (2000) Ubiquitouscomputing and the role of geometry. IEEE Pers Commun7(5):41–43
12. Shafer S, Krumm J, Brumitt B, Meyers B, Czerwinski M,Robbins D (1998) The new EasyLiving project at microsoft
research joint DARPA/NIST smart spaces workshop. Gai-therburg, Maryland, 30–31 July 1998, pp 127–130
13. Garlan D, Siewiorek D, Smailagic A, Steenkiste P (2002)Project aura: towards distraction-free pervasive computing.IEEE Pervasive Comput 1(2):22–31
14. The CHIL project, http://www.chil.server.de15. Soldatos J, Pandis I, Stamatis K, Polymenakos L, Crowley JL
(2006) Agent based middleware infrastructure for autono-mous context-aware ubiquitous computing services, Com-puter Communications, Elsevier (in press) DOI 10.1016/j.comcom.2005.11.018
16. Soldatos J, Polymenakos L, Pnevmatikakis A, Talantzis F,Stamatis K, Carras M (2005) Perceptual interfaces and dis-tributed agents supporting ubiquitous computing services. In:Proceedings of the Eurescom Summit 2005. Heidelberg, pp43–50
17. Pandis I, Soldatos J, Paar A, Reuter J, Carras M, Polyme-nakos L (2005) An ontology-based framework for dynamicresource management in ubiquitous computing environ-ments. In: Proceedings of the 2nd international conferenceon embedded software and systems, Xi’an, 16–18 December2005, pp 195–203. ISBN 0-7695-2512-1
18 Pnevmatikakis A, Soldatos J, Talantzis F, Polymenkos L(2006) Robust multimodal audio-visual processing for ad-vanced context awareness in smart spaces. In: Magloniannis I,Karpouzis K, Bramer M (eds) 3rd IFIP conference in artifi-cial intelligence applications and innovations, AIAI 2006.Springer series: IFIP, vol 204. Springer, Berlin HeidelbergNew York
19. Soldatos J, Stamatis K, Azodolmolky S, Pandis I, Polyme-nakos L (2007) Semantic web technologies for ubiquitouscomputing resource management in smart spaces. Int J WebEng Technol (in press)
20. Tesauro G, Chess DM, Walsh WE, Das R, Segal A, WhalleyI, Kephart JO, White SR (2004) A multi-agent systems ap-proach to autonomic computing, AAMAS In: 3rd interna-tional joint conference on autonomous agents and multi-agent systems (AAMAS’04), vol 1. Columbia University,New York, pp 464–471
21. Java Agent Development Environment, http://jadetilabcom/22. FIPA—The Foundation for Intelligent Physical Agents,
http://wwwfipaorg23. Rosenthal L, Stanford VM (2000) NIST smart space: per-
vasive computing initiative. In: Proceedings of the 9th IEEEinternational workshops on enabling technologies: infra-structure for collaborative enterprises WETICE. IEEEComputer Society, Washington, 4–16 June 2000, pp 6–11
24. UPnP Forum (2003) UPnP device architecture 1.0. Availableat: http://www.upnp.org/
25. Guttman E, Perkins C, Veizades J, Day M (1999) RFC 2608:service location protocol, Version 2. Available at: http://www.ietf.org/rfc/
26. UDDI Consortium (2002) UDDI technical white paper.Available at: http://www.uddi.org/pubs/
27. Czerwinski SE, Zhao BY, Hodes TD, Joseph AD, Katz RH(1999) An architecture for a secure service discovery service.In: Proceedings of the 5th annual international conferenceon mobile computing and networking (Seattle, Washington,USA) Mobicom’99. ACM Press, New York, pp 24–35. DOIhttp://doi.acm.org/10.1145/313451.313462
28. Zhu F, Mutka M, Ni L (2003) Splendor: a secure, private,and location-aware service discovery protocol supportingmobile services. In: Proceedings of the 1st IEEE interna-tional conference on pervasive computing and communica-tions PERCOM. IEEE Computer Society, Washington, pp235–242
Pers Ubiquit Comput (2007) 11:193–212 211
123
29. W3C Recommendation (2004) OWL web ontology languageoverview. Available at: http://www.w3.org/TR/owl-features/
30. Moller R, Haarslev V (2004) RACER: renamed ABox andconcept expression reasoner, Hamburg. Available at: http://www.sts.tu-harburg.de/~r.f.moeller/racer/
31. Brandstein M, Ward D (2001) Microphone arrays: signalprocessing techniques and applications. Springer-Verlag,Berlin
32. Murphy A, Picco GP, Roman G-C (2001) LIME: a middle-ware for physical and logical mobility. In: Golshani F et al.(eds) Proceedings of the 21st international conference indistributed computing systems(ICDCS-21). Phoenix, AZ, pp524–533
33. Pnevmatikakis A, Polymenakos L (2005) An automatic facedetector and recognition system for video streams. In: Pro-ceedings of the 2nd joint workshop on multimodal interac-tion and related machine learning algorithms. Edinburg
34. Stergiou A, Pnevmatikakis A, Polymenakos L (2005) Audio/visual person identification. In: Proceedings of the 2nd jointworkshop on multimodal interaction and related machinelearning algorithms. Edinburg
35. Talantzis F, Constantinides AG, Polymenakos L (2005)Estimation of direction of arrival using information theory.IEEE Signal Process 12(8):561–564
36. Pnevmatikakis A, Polymenakos LC (2004) Comparison ofeigenface-based feature vectors under different impairments.In: Proceedings of the 17th international conference onpattern recognition (ICPR’04), vol 1. Cambridge, 23–26August 2004, pp 296–299
37. Crowley JL (2003) Context driven observation of humanactivity. Lecture notes in computer science (Ambient Intel-ligence), vol 2875, Springer, Berlin Heidelberg New York, pp101–118
38. Azodolmolky S, Dimakis N, Mylonakis V, Souretis G,Soldatos J, Pnevmatikakis A, Polymenakos L (2005) Mid-dleware for in-door ambient intelligence: the polyomatonsystem. In: Proceedings of the 2nd international conferenceon networking, next generation networking middleware(NGNM 2005). Waterloo, 2–6 May 2005
39. Minar N, Gray M, Roup O, Krikorian R, Maes P (200) Hive:distributed agents for networking things. IEEE Concurr8(2):24–33
40. Soldatos J (2006) Software agents in ubiquitous computing:benefits and the CHIL case study. In: Proceedings of thesoftware agents in information systems and industrial appli-cations (SAISIA 2006) workshop. Karlsruhe
41. Hibernate Relational Persistence for Java and .NET, http://www.hibernate.org
42. Stanford V (2003) Pervasive computing goes the last hun-dred feet with RFID systems. IEEE Pervasive Comput2(2):9–14
43. Stanford V (2002) Pervasive computing goes to work:interfacing to the enterprise. IEEE Pervasive Comput1(3):6–12
44. Stanford V (2002) Pervasive computing: applications—usingpervasive computing to deliver elder care. IEEE Distrib SystOnline 3(3):10–13
212 Pers Ubiquit Comput (2007) 11:193–212
123