L.C

. Pol

ymen

akos

Living in the WebMixed Reality and Intelligent Spaces

prof. Lazaros PolymenakosHead, Autonomic & Grid Computing

Athens Information Technology

L.C

. Pol

ymen

akos

Outline• Introduction• Evolving Context to Mixed Reality Spaces

– Intelligent Spaces– Connected Intelligent Spaces– Mixed Reality Spaces

• Applications– Memory Jog– Elderly Care: My-past– Aware Interactive Media– Conclusions

L.C

. Pol

ymen

akos

Web Evolution• The Web has transformed

Practically

every activity of our lives is mirrored into

the Web

The Web structure across all its levels

is resembling our social

organizational

structures.

From a connection oriented space (anytime, anywhere, any device, any document, any…thing)

To a social interaction oriented space (be close to the people that we work with, we are friends with, we are traveling with,

we are in general socializing and sharing the world with)

L.C

. Pol

ymen

akos

Context Evolution

Key component to intelligent systems is the ability to understand and act on the context, i.e., the who, where, what and, if possible, why

“Any information that can be used to characterize the situation of an entity, where an entity can be a person, place, or physical or computational object” (Dey & Abowd )

Representational or Interactional view of context

All observed static and dynamic information along with the internals of the observing computational system in order to infer the type and dynamics of a situation involving humans, objects, and entities (real or virtual).

L.C

. Pol

ymen

akos

Intelligent Spaces

L.C

. Pol

ymen

akos

Perceptual Component Technologies

• 3-D Video recognition and tracking of people and objects• 3-D Audio recognition and tracking• Fusion of audio and video information• Audio-visual Emotion Detection• Speech activity detection• Voice recognition

L.C

. Pol

ymen

akos

Abstract evolving 3-D description

MK3

C

A

D

B

Presentation Area

x

y

125cm3 cube spanning

3D space

(250,360,170) cm

(270,80,130) cm

(200,40,135) cm

(110,110,135 ) cm

L.C

. Pol

ymen

akos

Middleware Components

Context Middleware Interface

Context Middleware Interface

Service Simulators

1. Feature Extraction2. Service Simulator

3. Service and Logic Modules4. Visualization

L.C

. Pol

ymen

akos

Connected Intelligent Spaces

When connecting two or more intelligent spaces we have to deal with:• Fusing Information from different perceptual modalities and sensors.• Channeling of information to multiple output modalities and actuators

according to the user's situation and attention as well the physical room’s equipment.

• Rendering perceptual components robust and highly scalable, to provide stable reliable performance under a wide range of real world operating environments.

• Adapting services to accommodate a variety of sensor and actuator configurations for multimodal environments.

• Maintain a consistent model of the context and its evolution across time and geographical separated rooms.

The goal is to go beyond the simple and often limited avatar representations of self to interact with remote objects and people, but rather we focus on technology assisted presence and awareness across sites and services.

L.C

. Pol

ymen

akos

Mixed Reality Spaces: a) Intermeshing Real and Virtual Objects

Maintaining user seamless experience: • Virtual objects obey the laws of physics • They interact with real-world objects.Technologies used: • Physics/game engines• 3-D cameras, • 3-D tracking, • 3-D Gesture recognition• Actuators close the loop between virtual

objects and the changes they effect on real ones.

L.C

. Pol

ymen

akos

Mixed Reality Spaces: b) Intelligent Tabletops

Multi-touch surface is based on the technology of TFT monitors and Frustrated Total Internal Reflection (FTIR) technology that describes the internal reflection of light, inside a certain material.

Materials:The parts of a disassembled 19” TFT monitor,

84 IR LEDs, IR-pass filter, USB Camera, Crystal-clear Plexiglas

Frustrated Total Internal Reflection

IR LED arrays

Plexiglas

TFT panel

Light Diffusing paper

Diffusing Plexiglas

Backlight

FTIR panelApplication

projecting screen

Capturing hole

IR capturing camera

L.C

. Pol

ymen

akos

Applications: a) Memory Jog

L.C

. Pol

ymen

akos

Memory Jog: Context State Diagram

L.C

. Pol

ymen

akos

Applications: b) Elderly Care

13500mm.

1500

0mm

.

Couch

Camera for people coming in, also covering some part of the room

C

Camera facing sofa, wide-angle, covering

most of the room

Television

7-channel microphone arrayfacing the couch

2900

mm

.

Door

Close-Talking Microphone for

user

Coffee Table

L.C

. Pol

ymen

akos

Applications: c) Aware Interactive Media

L.C

. Pol

ymen

akos

Intelligent Entertainment

Myspace

Content Provider

Myspace

IP network

AN n

IP access network

Scene Analysis/ Person Tracking

Service PlatformAAA, DRM, Web Server

Users Profile DB

AN 1

Playlist Playlist

YouTube

Video MetadataDB

21 3a,b,c4a,b,c

5a,b,c

Signalling

Video

Best Videos Selection

1'2'

5a,b,c

L.C

. Pol

ymen

akos

Conclusions

• These new paradigms for services and infrastructure can change the way we live, work and entertain ourselves.

• Our lives will be immersed by these new technologies and we will practically fulfill our lives in these mixed reality spaces, experiencing both the social and the productive aspects of living.