Contextual multimodal interfaces and device-centric ...hci.stanford.edu/cs547/abstracts/05-06/Boda...

1 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB

Contextual multimodal interfaces and

device-centric sensory networks

BODA Péter Pál

Software & Applications Technologies Laboratory

Nokia Research Center

Helsinki, Finland

[email protected] seminar, Stanford

April 14, 2006


Outline

• Enabling technologies

• overview of past and current activities

• Context-aware & multimodal applications

• MobiLife EU project

• CoMMod & zappa

• Virtual Modality and Multimodal Integration

• Device-centric sensory networks

• Whither User Interface?


Motivation and Objectives

• Ultimate goal: provide enhanced user experience

• human-human communication relies on a full range of modalities

• seamless interaction, ease-of-use, naturalness, predictability

• Use of Advanced User Interface solutions for mobile applications and services

• speech & language technologies (recognition, synthesis, natural language understanding, Q/A:question-answering, etc.)

• multimodality (combining various intentional user inputs, e.g. speech & gesture)

• modularity, easy deployment, multiple devices

• Intelligent combination of additional information with current state of the system, i.e. howto affect the actual performance of the User Interface

• context-awareness: how the actions of the system can be enhanced by using parametersdescribing the existence of the user and the system in the physical (and virtual) world

• sensory inputs: both user-originated and environment related, ambient knowledge, etc.


Enabling technologies


Past and current activities on enabling technologies

• Speech recognition & text-to-speech synthesis• embedded name dialling, connected digit recognition

• embedded large vocabulary speech recognition

• TTS for device-based use

• Language Understanding• adapted speech recognition technology for semantic unit recognition

• discrete non-aligned HMM and genetic algorithm based approaches

• Multimodality• own implementations and development of integrated technology

• contribution to W3C standardisation

• Haptic/tactile• experimenting event-triggered haptic gesture based interaction

• exploring novel use of tactile channel

• User experience• user-centered design, usability studies, human factors


3. System

Mobile access to Internet - user experience building

blocks (Virpi Roto et al.)

1. User 2. Context Mobile device Application Connection Site

Need Physical context Appeal Usability Availability Value

Motivation Social context Connectivity Content support Speed Usability

Experiences Temporal context Usability Functionality Cost Appeal

Expectations Task context Memory space Trust Trust

Mood Battery lifetime Gateway

Resources Performance (Page optimization, Access restrictions)


Some of the related EU and national cooperations

• Finnish National project (PuttePuh): first Finnish continuous ASR

• COST 249, 278 – speech technology, dialogue systems

• LC/TC-STAR - database collection for speech technology development

• CATCH-2004• Developing conversational multilingual dialogue systems• Multimodal applications and architecture


NEW CONTEXT!

NEW CONTEXT!

NEW CONTEXT!

Movies

tonight?

…. starting time of the 1st?

…. duration? (in Finnish)

…. description?

NEW CONTEXT!

Movies withMichael Douglas?

Welcome to Program Guide!

How may I help you?

… it starts at 5.15pm

I found the following

programs …

… it is 1h 25min long(in Finnish)

… no text message,thanks. (in Finnish)

I can offer the

following

description ….

To text message?

What kind of info

I can offer next?(in Finnish)

Programs for youngsters?

(in Finnish)

… sorry, no programs

for youngsters(in Finnish)

Michael Douglas is

in Coma … Channels?

BBC World, CNN,

Eurosport, TCM

What’s on BBC Worldtonight at 10pm?

World News at 10pm

… it takes 5 minutes(in Finnish)

That’s all for now.

Good bye!

…. duration? (in Finnish)

Bi-lingual Program Guide Information System(one of the outcome of the CATCH-2004 EU project)


Combining context-awareness and

multimodality


Life Goes Mobile with MobiLife (EU project 2004-2006)Vision and Challenges

Vision: Mobile & wirelessapplications and servicesbecome everyday things

All

phot

os ©

Nok

ia

People shift between! different (blurring) roles in! participating different groups,! using multiple devices! with different modalities! over different networks

Challenge: How to helppeople, especiallyfamilies, in making theireveryday life easier

MobiLife Overview September 2004 Slide 11

IST-511607

Life Goes Mobile with MobiLifePartners

Suunto, ElisaUniversity of HelsinkiHelsinki U of TechnologyNokia, Ericsson

Ericsson

MotorolaUniversity of SurreyNEC Europe (site: Ger)

NEOSHewlett-PackardSiemensTelecom Italia

Alcatel, Motorola

Telematica

FOKUSUniversity of KasselDoCoMo Eurolabs

BellStream

UNIS

Finland

Germany

Poland

Czech Rep.

SwedenNetherlands

UK

France

Italy

Coordination:

App. own. & SMEs:

Operators:

Manufacturers:

Academia:

Countries:

Partners:

Nokia

5

3

8

6

9

22

MobiLife Overview September 2004 Slide 12

IST-511607

“Self-awareness”

• Local environment

• Automatic configuration arrangementof devices, services, and localconnectivity

• Multimodal interfaces, personalisation

“Group-awareness”

• Context and presence support

• Novel privacy and trust models

“World-awareness”

• Automatic support for seamlessaccess to and delivery of servicesacross different domains

Communication Spheres

BAN – Body Area Network• Bluetooth, RFID, sensors, …

PAN – Personal Area Network• Bluetooth, WLAN, …

WAN – Wide Area Network

• 3G, B3G, WLAN, *DSL, …

People shift between differentroles in participating differentgroups, using multiple deviceswith different modalities overdifferent networks

Life Goes Mobile with MobiLifeTechnical Viability and User Acceptance


The Scope

• Everyday things:• Things that are around us

• Things that are so obvious we do not even notice them

• Can we detect them?

• Contextual analysis and Sensory inputs

• Everyday life easier:• The user experience comes from the ease-of-use, seamlessness

• User Interface is the first touch to the application/system

• The simpler the better, the shorter interaction the more effective

• Imitation of human-human communication: multimodality

• Work Package 2:• Self-awareness, personalisation, profiling and its use

for modifying the actual UI

• Advanced user interface solutions

(multimodality, sensory inputs, tangible UIs)


The Approach

• User-Centered Design:

• Collect typical scenarios in the domain

CA

G

g1

g2

c1H a1

J B

• Analyse representative user behaviours

• Map them for technical requirements and functionalities

• Define functional and system architecture to implement or re-usefrom existing systems

MainWeather

info

Babyinfo

• Verify implementation via applications & intensive tests


Few examples of Multimodal Interaction

• Combining modalities naturally/seamlessly as the user prefers:• gestures: when spatial interaction is preferred (browsing, map navigation)

• speech: when quick selection, shortcut is useful

• gaze tracking: hands-busy situations (in noisy environment)

• tactile feedback: providing prompt alert or non-intrusive feedback

• etc.


Examples of contextuality & UI dynamics

• Using location information:• FamilyMap: “Is there any place to feed the baby nearby?”

• Deictic reference can be resolved by knowing current location

• Outcome: results in smoother and faster interaction, 1 turn saved

• Using personal settings, profile:• Infotainer: User jumps to the car " corporate news streamed automatically

• Knowing the date and time and user set preferences, the desired news are delivered on the instantlyavailable audio channel

• Using biomedical/sensory information:

• Wellness: detecting kids’ heart rate on a football training• Instant feedback to trainers and parents about the performance

• Data can be logged for later comparison, “competition against me”

• Detecting emotional state, stress level• Dialogue style, system guidance can be adapted accordingly


The implemented contextual UI architecture

DeaMon and DeGan:•Advanced modality and device controlmechanisms• Contextual inputs affect how modalitiesand devices used, as well as how content ispresented


Demonstration Applications

Content is

synchronised

between devices

User can switch

from one device

to another at

any moment

Multimedia Infotainer

FamilyMap

MainWeather

info

Babyinfo

“Logistic

navigation”,

contextual

proactiveness,

social networking

Being aware of

”wellness” condition

Use of sensors

as passive input

modalities

Wellness-Aware

GamingContext Watcher

Location clustering,

sport sensors,

finds POIs, buddy activity

show, blogs & pics


Multimedia Infotainer

Context-aware multimodality allows personalised access to content

(e.g. news) depending on current preferences, location and/or device(s)

Content can be showed in

different multimodal ways,

depending on available devices

(PC, mobile phone, TV)

Content is

synchronised

between devices

User can switch from one device

to another at any moment

• An advanced multi-functional application on a mobile device with abilities to:• Enable a user to communicate and receive personalised news via multiple channels;

• Retrieve content based on the user’s personal preferences;

• Support multiple multimedia output devices;

• Select best delivery method/media according to the user’s preferences and/or context reasoning engine


Wellness-Aware Multimodal Gaming System

• Wellness-Aware Multimodal Gaming System integrates fitness monitoring features and comparisoncapabilities:

• Allows family members to share wellness, environment and location information during/after training orgame;

• Family members and football coach are able to look at heart rate, speed and wellness condition of asingle user or a whole football team

Parents looking at the game together

Being aware of ”wellness” condition

Use of sensors as passive

input modalities

Using sensors and collecting sensory data allows users to share,

record and compare wellness information


• FamilyMap allows enriched contextual navigation:

• Provides baby-care related information for families with babies, especially on the move in unknowncities;

• Assists families to find “logistic” locations (where to feed a baby, change diaper, spend somerelaxed time with a baby, paths suitable for baby carriage);

• Social interaction: leave and receive virtual-post-its in hot-spots about preferred places dependingon context and preferences;

• Advanced multimodal interaction via multiple devices assembled to the baby carriage

Main

Navigate

Weather

info

Babyinfo

It will soon rain.

Find nearest

metro s tation "

FamilyMap

Weather change info

Navigation view

O K/YES

O ptions

More/Nav

NO

Weather

info

Babyinfo

1. There is playground in

200m named Sunnydale

playground

2. There is a baby clothes

shop after the next corner

O K/Main

Leave Msg

Weather

info

Babyinfo

Great place, my children

loved here. Ciao, Daniela

MainWeather

info

Babyinfo

FamilyMap provides navigation enriched with contextual information

(logistics, weather, point of interest, social interaction)

Social interaction,

virtual post-it

Point of interest

description


FamilyMap on the concept level

Main planned functionalities:

Provides baby-care related information

for families with babies, especially on the

move in unknown cities.

It assists families to solve practicalproblems, to find “logistic” locations,places where a parent can feed a baby (e.g.a microwave is available), change diaper,spend some relaxed time with a baby, canwalk easily with the baby carriage: i.e. pathhaving no stairs or construction going on.

Promoting social networking: parents canleave and receive virtual-post-its in hot-spots about preferred places, e.g.cafeterias, shops for babies, etc.,depending on context and preferences.

Navigation, tracking

(Nokia, Wayfinder)

GPS

System-User interaction

(e.g. iPaq, Nokia)

SUUNTO:

Sensory devices

(temp, speed, etc.)

Tactile

feedback


MainWeather

info

Baby

info

<map with POI>

Functionalities

More/Navigate

New

destination

Navigation view

Main

Navigate

Weather

info

Baby:Diaper

wet

Two hours on the road.

Find a place to change

the diaper "

Baby status info

OK/YES

Opinions

More/Navigate

NO

Weather

info

Baby

info

1. There is playground in 200m named

Sunnydale playground

2. There is a baby clothes shop

after the next corner.

Point-of-Interest

descriptions

OK/MainWeather

info

Baby

info

“Great place, my children loved here.

Ciao, Daniela”

Leave a

message

Social interaction,

virtual post-it

Main

Navigate

Weather

info

Baby

info

It will soon rain.

Find nearest metro station "

Weather change

info


FamilyMap system architecture

MS MapPoint

Server

FamilyMaps

ServerR

map

coordinate

edit PoI data

Nokia 9500

VibratorGPS

Nokia 9500

VibratorGPS

Nokia 9500

VibratorGPS


Links

• MobiLife: http://www.ist-mobilife.org/

• Context Watcher: http://www.lab.telin.nl/~koolwaaij/showcase/crf/cw.html

(links to Cell-ID repository, auto-blog, downloads, etc.)


Internal activities:

CoMMod, zappa


CoMMod: Context-based multimodal rule creation & visualisation

Problem

• Rule creation for context-aware applications isproblematic and might prevent wide-spreaduse: typically all personalization-basedapplications require substantial user interaction.

• Rules can be quite complex (understandingwhat conditions in what combinations shouldtrigger actions)

Solution

• Make rule management seamless, usemultimodal user interfaces

• By visualisation means make the user awareand able to understand why a certain actionhappened

Benefit

• User is in charge what rules are used

• Straightforward rule distribution

mng calls ring even in silent

get traffic info

Sue calls not urgent to voicemail

Most recent firingrule visible

Currently firing rule,conditions valid

Active rules,not firing

Inactive rules

on mtng

on mtng

5pm on weekday


Multimodal rule creation

• Process:

• Pre-defined building blocks for conditions and actions

• Drag & drop or speak out: “If Joe is calling then Send message .... to .... ”

If

or

Then

Conditions Actions

• Public &private rules

• Distributionover mForms

• Public rules’placeholders tobe filled up ondevice


Use case: Desired profile selection(contextual/sensory events trigger device-based core application)

Corporate meeting

if LOCATION= (workplace and meeting room)

and DATE= working day

and PROXIMITY= 2 or more colleagues

and PROXIMITY_MODE= (silent or meeting)

then PUT PHONE TO SILENT MODE


Use case: Manager calls

In a meeting….

if CALLER=own line manager

and LOCATION= (workplace and meeting room)

and DATE= working day

and MODE= (silent or meeting)

and ACTIVITY= not giving a lecture

then RING THE PHONE

NB.: it is fine to be ringed by the boss

during working hours even on a meeting,

but not when giving a lecture to a larger audience


Context-story simulator and distributor: zappa

• Problem:

• How to test (in laboratory environment) the operation of the developed technologies(e.g. visualisation of firing rules, rule execution)?

• How to detect users’ behaviour under controlled circumstances in a context-richenvironment ?

• Simulated contextual bits should be mixed with real-life events

• Approach:

• Generate stories that contain simulated contextual events (e.g. change of location,proximity of family members, colleagues, change in environmental conditions, etc.)

• Distribute stories either from a mobile device (can take real-life elements intoaccount, e.g. proximity, location) or from a server

• zappa!" zentralised autonomous probabilistic pragmatic appliance


t (sec)

0 5 10

ID: 1.1Story :Trav ellerSender: zappa_#1LOC*=“home”t=0sec

ID: 1.2Story :Trav ellerSender: zappa_#1LOC*=“plane”t=10sec

ID: 1.5Story :Trav ellerSender: zappa_#1LOC*=“work”t=20sec

ID: 1.9Story :Trav ellerSender: zappa_#1LOC*=“restaurant”t=35sec

ID: 1.3Story :Trav ellerSender: zappa_#1WEATHER*=“rain”t=10sec

ID: 1.4Story :Trav ellerSender: zappa_#1TRAFFIC=“heav y ;delay ”t=10sec

ID: 1.7Story :Trav ellerSender: zappa_#1IDENTITY=<colleague>t=20sec

ID: 1.8Story :Trav ellerSender: zappa_#1ACTIVITY*=“meeting”t=20sec

ID: 1.6Story :Trav ellerSender: zappa_#1ENVIRONMENT=proximity ;IDENTITYt=20sec

ID: 1.10Story :Trav ellerSender: zappa_#1ACTIVITY=“call;<spouse>”t=35sec

An example story


t (sec)

0 5 10

ID: 1.1Story:TravellerSender: zappa_#1LOC*=“home”t=0sec



Architectural setup

TopologyFile Edit LocateView Help

Mount

431 7437

1950 79%

/

NetworkTraffic Help

t (sec)

0 5 10




server

zappa

1. Create story on server, test, run locally then distribute to test device2. Test user start test by downloading the story and the corresponding playback parameters3. Story downloaded to test device. User starts test, zappa"is playing back according to

playback parameters4. Device acts as central story player, distributes story elements to further test devices, can

interleaves with real-life elements

1.

2.

t (sec)

0 5 10


ID: 1.2Story:TravellerSender: zappa_#1LOC*=“plane”t=10sec

ID: 1.5Story:TravellerSender: zappa_#1LOC*=“work”t=20sec

ID: 1.9Story:TravellerSender: zappa_#1LOC*=“restaurant”t=35sec

ID: 1.3Story:TravellerSender: zappa_#1WEATHER*=“rain”t=10sec

ID: 1.4Story:TravellerSender: zappa_#1TRAFFIC=“heavy;delay”t=10sec

ID: 1.7Story:TravellerSender: zappa_#1IDENTITY=<colleague>t=20sec

ID: 1.8Story:TravellerSender: zappa_#1ACTIVITY*=“meeting”t=20sec

ID: 1.6Story:TravellerSender: zappa_#1ENVIRONMENT=proximity;IDENTITYt=20sec

ID: 1.10Story:TravellerSender: zappa_#1ACTIVITY=“call;<spouse>”t=35sec

PLAYBACK PARAMETERS

TEST_ID=1234STORY1=trav el_car_home_workSTORY2=naPLAY=1DURATION=2MINRANDOM_STORY=naRANDOM_EVENTS=naINFINITE=na

3.

4.


Virtual Modality and

Multimodal Integration


…. when it comes to multimodality

• Satisfied users <== Full understanding of intention

• primary goal is to interpret the user’s intention accurately

• not only at the time of the interaction, but in respect to dialogue history, previous sessions, userpreferences, context (meant here in a wider sense), etc.

• Seamless interaction

• no explicit modality selection should be forced on the user – unless some extra circumstancedemands so (e.g. switch off ASR when too noisy)

• Consistency, equality

• all modalities should be handled in a similar fashion

• no reason to have a primary modality (weights can be allocated if usability study supports)

• our senses work continuously: equally, in a complementary and overlapping fashion

• Transparency

• Multimodal Integration (MMI) must support information flow across modalities

• neuropsychological evidence shows that cross-modal interaction occurs constantly, more stronglyin cases where one of the sensory inputs is degraded/missing


The model

symbolic output

sensory input

(e.g. speech)

Modality

pro

cessing p

ipelin

e

….….

symbolic output

sensory input

(e.g. gesture)

<map>

gramm

ar(s)

(x,y

)A semantic tier:

is the bridge across modalities


Notions and Implications

• Semantic tier:

• description of the domain, how the worldis perceived

• Dual view:

• speech should not know aboutcoordinates, gesture should not knowabout grammars

• Cross-modality:

• follows neuropsychological findings(McGurk effect, synaesthesia, etc.)

• see Coen: Perceptually motivated MMIand P. Haikonen about consciousmachines

• Eventually, this is ontology

• later might be Semantic Web

• Can be extended

• to any number of modalities

• sufficient: there is a quantitative descriptionin terms of resources for a modalityprocessor (e.g. grammar).

• Cross-modal triggering is:

• bi-directional

• can emerge at various points

• Why is it good for us:

• equal role in perception, no primarymodality

• triggering one modality’s perception in caseof early evidence(s) in another modality

• unimodal performance can be boosted (e.g.focusing to area, allowing only certainobjects, selecting grammar)

• mutual disambiguation


gramm

ar(s)

(x,y

)

<object>

smell ga

zed

gramm

ar(s)

(x,y

)

<object>

smell ga

zed

Generalised

gramm

ar(s)

(x,y

)

<object>

smell ga

ze


Further notices

• Eventually, this is a complex model

• Assumption:• Available multimodal data could be utilised in the development (testing & building) of

multimodal systems

• Learn/validate internal structure, train parameters, etc.

• Chicken-and-the-egg problem:• No real data can be collected w/o an operational system

• No data ! difficult to develop well-tuned complex systems


The tasks

• Acquire database:• purchase/collect/….

• ! transform/generate !

• Create a statistical (hybrid) integration framework:• which utilises the multimodal data in the training/testing phases

• statistical:• other modules are (speech reco-ASR, handwriting reco-HWR, natural language understanding-NLU, ….)

• huge amount of contextual data is/will be available

• MMI:• typically ad hoc methods available, except AT&T’s FST (finite-state transd.)

• challenging to create a unified framework


How to get multimodal data?

• There is a huge amount of data out there!

• Multimodal systems can also operate in speech-only mode

• Vast amount of speech data (also in textual form):

• Used for component training (speech recogniser, language modelling, …)

• Available as logs of operational spoken dialogue systems

• Idea:

• Take transcribed data and generate “multimodal” data

• Select pre-defined concepts (e.g. location references) and remove from the speech channel

• Replace them with deictic expressions

• Put the removed concepts to an abstract channel: Virtual Modality

• NB.: presence of deictic expressions is not necessary (cf. Oviatt et al. 1997)


Virtual Modality …

• … is a fictional modality channel, carrying semantically meaningful input elements

• … it can be virtually anything, we do not care how it evolved, sensed, etc.• Intentional input: any available input modality (speech, gesture, gaze, text, etc.)

• Passively available info: pragmatics, interaction histories, user preferences, “context”,sensory/affective data, location, etc.

• Utilisation:• Testing integration modules

• Study timing effect, mutual disambiguation

• Cross-modal interaction

• Building statistically motivated integration mechanism


Virtual Modality Implementation


Eventually ….

• Multimodal Integration:

Sp’MMI

• “Inverse” Multimodal Integration:

G

Sp’

VM

Sp

C


Parameters of the model

• Number of allowed deictic replacements per utterance

• Selection of deictic replacements:

• this, that, here, there, over here, over there

• this $, that $: e.g. this city, that university

• Inter-modal lags (deictic position vs. VM entry)

• Pre-defined user types (discrete lags: word position)

• Simulated user population:

• E.g. 80% quickie, 20% precise (cf. sequential & simultaneous)

Too-fast -2

Quickie -1

Precise 0

Slowhand +1

Everlate +2


Data generation

• Application domain:• Voyager (MIT/SLS) – traffic and city guide

• Conversational spoken language interface

• Original data:• Output of speech recogniser

(i.e. input sentences)

• Corresponding NLU parse frames

(used to identify location references)

Number of sessions 1099

Number of turns 6077

Average number of turns

per session 5.53

All location references 6982

Average number of

references per turn 1.14

Number of different

location reference patterns 203

The five most frequent

location reference patterns:

in <….>

of <….>

show_me <….>

on <….>

from <….> to <….>

9.95%

8.15%

6.39%

5.69%

3.95%


Two questions to be answered

• Is the generated data realistic?

• How well does the existing system work with the Virtual Modality data?


Data quality

• Real multimodal data from Voyager (MIT/SLS):

• 68% of turns have deictic expressions (5583)

• The first three most commonly used location reference patterns are the same as in thespeech-only case:

in …, of …, show_me …

• When two location references used in a sentence, the first one is with deictic:

how do i get there from kendall square

• The used deictic expressions: this, that, this one, here, there

• No occurrence of: over there, this university, that city, …

By courtesy of Jim Glass, MIT


Evaluation setup

Galaxy Communicator

Batchmodeserver

VM data MM eventdatabase

Original sentences

{c directions:subject 1:domain "Voyager":utterance_id 6:pred {p from

:topic {q university:name "Harvard University" } }

:pred {p to:topic {q university:name “MIT" } }

Original frame



:pred {p to:topic {q pronoun:name “here" } }

VM frame

?“From Harvard University to MIT”

“From Harvard University to here”

D1 = “MIT”

D1_begin = 4

D1_end = 5




After Context Resolution

Galaxy Communicator

Batchmodeserver

VM data MM eventdatabase

Original sentences




Original frame



:pred {p to:topic {q pronoun:name “here" } }

VM frame

?“From Harvard University to MIT”

“From Harvard University to here”

D1 = “MIT”

D1_begin = 4

D1_end = 5




After Context Resolution


Experiments with the VM data

• 436 non-identical sentences

• 1 deictic per sentence with zero lag:• yields 82.03% identical output

• Problem: preposition and randomly selected deictic sometimes results inungrammatical expressions

• e.g. “how about on over there?”

• Few cases were not covered by the parser

• 2 deictic per sentence with zero lag:• 78.26% accuracy

• Incorrectly handled highways and highway_numbers by the data generation process

• Temporal effects are not tested (server applies a fixed size window)


The integration framework

• How to integrate various inputs?• textual, symbolic, (maybe) physical measures

• If we think about the resolution of deictics• reminds Word Sense Disambiguation

• Can something be used here from NLP?

• Maximum Entropy framework• Elegant approach to combine various inputs via features

• See work of Adam Berger, Adwait Ratnaparkhi, ….


Maximum Entropy approach

• Use/say no more than known• no more assumptions than the data supports

• find smoothest solution

• Features:• logical connection between input and output domain

• binary or continuous values

• Exponential model:• basically a weighted distribution• (Engineers: like Fourier spectrum)

• weights are learned from data


Features

• bag-of-word type feature: the co-occurrence of a word w in the Sp channel or entrym in the VM channel and that of a concept C is checked

• contextual feature (NB. lingustic context): the co-existence of patterns in Sp anda given concept is measured. In the current set-up left context with 3 words relativeto the deictic expression is considered

• prepositional feature: the word before the deictic expression was checked whetherit is a preposition


Results

• Integration accuracy for the first 3-best:

• Problems:

–Preposition detection, handling no-preposition cases

–Too many labels (176) -> atomic labels?

–Amount of features -> data sparseness, smoothing parameters

86.772.833rd best

83.947.422nd best

76.521st best

Accuracy for 2 and 3 bestAccuracy %Rank

• Contextual info helps:

• utilising (virtual) gesture type info, can increase accuracy by 2-4% points


Sensory input on a large scale


SensorPlanet:

a global test platform for mobile-centric wireless sensor networkresearch

Owner: Henry Tirri

Contacts: Barbara Heikkinen, Boda Péter

Nokia Research Center, Helsinki


Why a phone should play any rolein Wireless Sensory Nets?

... because it is there ...


SensorPlanet

a Nokia-initiated global cooperation on large scale

wireless sensory networks

mobile phone oriented sensory input acquisition

a research platform that enables collection and analysis

of sensory data on a never seen quantity


Mobile Wireless Sensor Network Scenarios

Mobile phonesact as sensor

node and/or sinknode.

Mixed sensornetworks which

consists of ordinarysensors and mobile

sensors.

Sink node

Sink node

Central

Data

Repository

„Green phones“

prov iding

env ironmental data,

sharing

“Service manifolds”

Traffic, masses of people ...

the more data shared the more accurate

(© Copyright 2000 Martin Lamm)

(© Copyright 2006 Petri Koskelainen)

“Urban games”

Locations shared,

participation of the public ...


SensorPlanet University Cooperation

• Ultimate goal to have multiple SensorPlanet sites around the globe

• Data gathered on sites accumulated in central repository

• Common access to accumulated data for research purposes

• Single interface to sensory data from the world

• Universities:• MIT/CSAIL and Media Lab ! ongoing work on body sensor network and CARTEL

• UCLA

• NYU

• Stanford

• Berkley

• Copenhagen

• EPFL/Zurich

• tbd.


More on potential use cases


Mobile handset in Wellness

Glucose

Activity

Heart beat

Home systems

Services

Blood Pressure

Wrist

Applications


Recording everything (SharMe)

What I feel etc.

What I see

What I hear

What we say

What I write

Where I am


Tagging, Virtual Graffiti


„Dynamic highway pay”


Prototype Device Implementation – Wireless Motion

Band

• Wireless Motion Bands are designed forcomfortable wear in different body parts toenrich user experience in gaming.

• Wireless Motion Bands allow the use of

(full) body motion tracking in gaming:

• Direct 3D body motion control of a gamingcharacter

• Detection of body poses & gestures

• Local multi player gaming. Low-latency multidevice Bluetooth connectivity.

• Interoperability with Wireless Game Pads.

Design by Mikko Aarras, Nokia Design

By Courtesy of Jukka Salminen


User Interaction Examples – Virtual Snowboarding

• Direct bodymotion controlof a computercharacter

• Body poses

• Balance

• Direction

• Speed

• Jumps

• Landings


There is always a need of an ecosystem...

Mobile device asintelligent hub andportable applicationplatform

IR, Bluetooth,

RFID, USB

PC and server applications andhosted internet services

Server

PC

- Coach- Trainer- Doctor- Nurse- Caregiver- Operator DB

- Serviceproviders- Peercommunity

Internet services

Portfolio ofcommunication andmonitoring devices

Some sensors may beembedded in terminal

Ope

rato

r tr

affic


Some research challenges


Some key technological challenges

• ”Sensor data challenge”: how to make the high-volume sensor datauseful

• query/search problem

• data management problem

• ”Mobile adaptation challenge”: self-configuration and adaptation

• ”Integration challenge”: how to integrate (two-way) WSN to Internetbased services (create the “gateways”)

• “Power challenge”: even high-power mobile nodes are not enough

• “User interaction challenge”: how end-users will benefit, interact


What sort of raw (context) data management

problem we are facing?

• a multidimensional (2-30) vector of real values

• frequency 0.5s - 1 day

• typically “always-on”

• a 1-4M pixel image

• frequency 10 min - week(s)

• very irregular, high-intensity bursts (many images within minutes)

• a 100K-1M sound file

• frequency 1 min - days

• irregular; streaming

• Naturally many application domains require a mixture of these

10K phones - vector every 2 min results in 2.7 (US) billion vectors/year

200M phones - vector every 60 min results in about 1012 vectors/year


SensorPlanet Core handset(s)

• Nokia E70 Smartphone

• EGSM 900, GSM 1800, GSM 1900 and WCDMA (3GPP Release 4) networks

• Speech codecs: FR, EFR and AMR f or GSM and AMR f or WCDMA

• Weight: 127 g (with Nokia Battery BL-6C)

• Dimensions: 117 mm x 53 mm x 22 mm (closed), 102 cc

• Full messaging key board with f old open design

• Up to 75MB internal dy namic memory

• 2.0 Megapixel camera with resolution up to 1600x1200

• 8x digital zoom

• Recording f ormats (encoding): 3GPP, H.263 v ideo, and AMR radio

• Video resolution CIF (352 x 288), QCIF (176x144) or Sub QCIF (128 x 96 pixels)

• Pop-Port™ interf ace

• USB 2.0 f ull speed supported v ia Pop-Port™

• Connect to a compatible PC wirelessly or with Nokia Connectiv ity Cable CA-53

• Integrated inf rared module (IrCOMM, IrOBEX, IrTiny TP)

• Bluetooth wireless technology 1.2

• Install applications with Nokia PC Suite f or the Nokia E70 phone

• Video and audio streaming (3GPP and RealMedia)

• WCDMA (3GPP Release 4)

• WAP 2.0

• Multiconnectiv ity

• WLAN connection can be shared by multiple applications at the same time. For example: email, browserand sy nchronization

• Possible to use WLAN, Bluetooth technology , USB and inf rared at the same time

• Max 6 Bluetooth technology connections in use at the same time 4

• WCDMA (3GPP Release 4)

• EGPRS (Class B, MSC 10)

• GPRS (General Packet Radio Serv ice) Multislot Class 10

• Dual Transf er Mode (Multislot Class 9)

• Wireless LAN 802.11g, 802.11e and 802.11i


SensorPlanet Core handset(s)

• Nokia 770 Internet Tablet

• Weight: 230 g

• Dimensions: 141 x 79 x 19 mm

• Display

• High-resolution (800x480) touch screen with up to 65,536 colors

• Memory: Flash 128MB (>64MB for user)

• Memory card: 64MB RS-MMC (Reduced Size - MultiMediaCard)

• Operating system: Internet Tablet 2005 software edition

• Applications (in 2005 software edition) Web Browser (Opera 8), Flash Player version 6, EmailClient,Internet Radio, News Reader, Media players, Image viewer, PDF viewer, File Manager,Search...

• Audio: MP3, MPEG4-AAC, WAV, AMR, MP2

• Image: JPEG, GIF, BMP, TIFF, PNG, Animated GIF format,

• SVG-tiny, ICO

• Video: MPEG1, MPEG4, Real Video, H.263, AVI, 3GP

• WLAN: 802.11b/g

• Bluetooth specification: 1.2

• For Internet connection and file transfer via phone

• Profiles supported: Dial-up Networking, File Transfer, Generic Access, SIM Access and SerialPort profiles

• USB device mode for PC connectivity

• RS-MMC

• 3.5 mm stereo audio out


SensorPlanet demos

under development


SensorPlanet’s Horizontal & Vertical implementations

1) Magrathea 3)Virtual Graffiti

In-door attachment and

spotting of virtual objects

BT-based positioning

Server-based repository

Passing primitive & robust

codes from device to device

Report back mechanism

about spreading

Op

era

tion

Tim

es

Sq

ua

re

Dirk Trossen (NRC/Hki)

A to be Open-Source platform

4) Nokia Remote

Sensing Architecture

(N-RSA)

WL

AN

-ba

sed

Po

sitio

nin

g

Inst

an

t M

ee

ting

Ro

om

Acc

ess

Vir

al R

ese

arc

h T

oo

lkit

To

ijala

stu

de

nts

’ in

no

vatio

ns

5) NRC SiteImplementation

WLAN based indoor

positioning

Sensor platforms

2) Interactive

Services

Infrastructure for

building urban services


Virtual Graffiti Visualisation

• Virtual Graffiti develops a horizontal platformfor in-door tag placing

• Positions are determined by Bluetooth beacons

• User can leave/read out/modify/etc. virtualobjects left to physical places within theNRC/Toijala premises

• The current visualisation:

• Violet balls represent Bluetooth beacons andthe red ones at the centre of the picturerepresent clients. The red balls that constitutean oval represent other Bluetooth devices thatthe beacons have found.

• The thickness of the lines represents thefrequency of scans made between the clientand a Bluetooth device.

• On the top of the visualisation box the five lastevents are displayed.


Operation Times Square

• Real time reporton test users’location

• Game: find yourgroup, gettogether, send pic

• Location isidentified basedon cell-ID

(© Copy right 2005 FreeLargePhotos.com)

SensorPlanet.org


Play it again in Manhattan!

Get with your team to the Times Square,

with a little help ....

The Game (preliminary plan)

#1 Players are scattered around Manhattan

#2 Divide them into two teams on both side

of the Times Square

#3 Teams to get to Times Square

(© Copyright 2006 Petri Koskelainen)

(© Copyright 2000 Martin Lamm)


SensorPlanet.org …

soon to be launched


Links & contact

• Nokia Research Center external site: http://research.nokia.com

• More on SensorPlanet: http://sensorplanet.org (will be up soon)

• Contacts:

• Henry Tirri ([email protected])

• Boda Péter ([email protected])


Finally


Whither UI?

My personal view about the ultimate User Interface:

• no explicit input from the user, yet, accurate system action

• already now: an elevator can take you where you intend to go

• new palyers: context, sensory input, bio-sensory signals, ambient intelligence, etc.

• intentional user input only in “error correction” situation

• Will the actual user interaction be reduced to minimal?

• Will the traditional UI become useless? --> Useless Interface?

Gábor Dénes (1969):

"If enough people work hard enough on the problem of speech recognition, itwill be solved by mid next century.“

Surely he did not think of multimodality, context-awareness, etc.


Acknowledgements

• NRC/Helsinki:

• Agathe Battestini, Barbara Heikkinen, Mika Klemettinen, Jukka Linjama

• Jesper Olsen, Virpi Roto, Henry Tirri, Vétek Ákos

• NRC/Tampere:

• Kiss Imre, Pekka Ollikainen

• NRC/Budapest:

• Szomolányi Márton, Bisztray Dénes

• NRC/Toijala:

• Marko Mahkonen, Tuomo Heinonen

• MobiLife/EU partners


Thank you!

[email protected]


References

• Enabling Technologies:• Publication type: Conference paper to COLING-ACL 2006

• Author(s): Daniela Oria & Jesper Olsen

• Title: Statistical language modeling with semantic classes for large vocabulary speech recognition in embedded systems

• Status: submitted

• Publication type: conference paper (ICASSP 2006)

• Author(s): Jussi Leppänen, Imre Kiss.

• Title: Gaussian Selection with Non-Overlapping Clusters for ASR in Embedded Devices

• Status: accepted

• Publication type: conference paper (ICASSP 2006)

• Author(s): J. Olsen & D. Oria

• Title: PROFILE BASED COMPRESSION OF N-GRAM LANGUAGE MODELS

• Status: Submitted

• Publication type: workshop paper (W3C workshop on internationalizing the speech synthesis markup language, 2005)

• Author(s): Jilei Tian, Xia Wang and Jani Nurminen

• Title: SSML Extensions Aimed To Improve Asian Language TTS Rendering

• Status: Accepted

• Publication type: conference paper (Interspeech 2005)

• Author(s): Jussi Leppänen, Imre Kiss

• Title:Comparison of Low Footprint Acoustic Modeling Techniques for Embedded ASR Systems

• Publication type: conference paper (Interspeech/Eurospeech 2005)

• Author(s): Jilei Tian, Jani Nurminen, Imre Kiss

• Title: Duration Modeling and Memory Optimization in a Mandarin TTS System.

• Jukka Linjama & Topi Kaaresoja: Novel, minimalist haptic gesture interaction for mobile devices

• Demo paper -- NordiCHI, Tampere 23-27.10. 2004


References

• Enabling technologies:• Virpi Roto, Roland Geisler, Anne Kaikkonen, Andrei Popescu, Elina Vartiainen: Data Traffic Costs and Mobile Browsing User Experience, MobEA IV- Empowering the Mobile Web, 2006

• Matti Karjalainen, Péter Boda, Panu Somervuo, Toomas Altosaar (1997), “Applications for the hearing-impaired: evaluation of Finnish phoneme recognition methods”, Eurospeech'97, Rhodes, Greece, 22-25. September, 1997, pp. 1811-1814.

• P.P. Boda (1997), “From stochastic speech recognition to understanding: an HMM-based approach”, ASRU’97- IEEE Workshop on Automatic Speech Recognition & Understanding, Santa Barbara, CA,USA, 14-17. December, 1997

• Cristina Dobrin, Boda Péter Pál (1998), “Resolution of date and time expressions in a WWW-based dialogue system”, COST 249 (Continuous Speech Recognition over the Telephone) Workshop, Porto,Portugal, February, 1998

• P.P. Boda (1999), “Speech Understanding By Means Of Genetic Algorithms”, COST 249 (Continuous Speech Recognition over the Telephone) Workshop, Prague, Czech Republic, February, 1999

• Cristina Dobrin, Péter Boda & Kari Laurila (1999), “On Usability of Name Dialling”, ASRU'99 - IEEE Workshop on Automatic Speech Recognition & Understanding, Keystone, Colorado, USA, 14-16December, 1999

• Petri Haavisto & Boda Péter (2000), “On Voice Services and Data in Digital Mobile Systems” , VOTS-2000: Voice Operated Telecom Services: "Do they have a bright future?", Gent, Belgium, 11-12. May,2000, pp. 13-16.

• P.P. Boda (2000), “Visualisation of Spoken Dialogues”, ICSLP'2000 6th International Conference on Spoken Language Processing, Beijing, China, October, 2000, vol. I. pp. 146-149.

• P.P. Boda & Saija Lemmelä (2002), “Efficient Combination Of Type-In And Wizard-Of-Oz Tests In Speech Interface Development Process”, ICSLP'2002 7th International Conference on Spoken LanguageProcessing, Denver, USA, September 16-20, 2002.

• CATCH-2004:• P.P. Boda, Suresh Chande, Elviira Hartikainen, Nidhi Gupta (Nokia), Sirpa Autere (Elisa Communications), (2003), “Flexibility and efficiency through personalisation? Experiments with a conversational

Program Guide Information System”, EACL-03 Workshop on "Dialogue Systems: interaction, adaptation and styles of management", April 13-14, 2003, Budapest, Hungary

• Elviira Hartikainen & Boda Péter (2003), “Results and Observations from the Evaluation of a Telephony-based Conversational Information Retrieval System”,First ISCA Tutorial & Research Workshop on“Auditory Quality of Systems", April 23-25, 2003, Akademie Mont-Cenis, Germany

• Spoken Dialogue Systems:• Oria, Daniela, Vetek, Akos (2004) "Multilingual E-mail Text Processing for Speech Synthesis", in Proceedings of Interspeech 2004 - ICSLP, pp 841-844.

• Oria, Daniela, Vetek, Akos (2005) "Robust algorithms and interaction strategies for voice spelling", in Proceedings of Interspeech 2005 - Eurospeech, pp. 833-836.

• Oria, Daniela, Vetek, Akos (2005) "Automatic Generation of Speech Interfaces for Web-based Applications", in Proceedings of ASRU 2005 - IEEE Automatic Speech Recognition and UnderstandingWorkshop, pp. 397-402.


References

• MobiLife:• More publications on: http://www.ist-mobilif e.org/ --> Publications

• Context Watcher: http://www.lab.telin.nl/~koolwaaij/showcase/crf /cw.html (links to Cell-ID repository , auto-blog, downloads, etc.)

• A. Salden, R. Poortinga, M. Bouzid, J. Picault, O. Droegehorn, M. Sutterer, R. Kernchen, C. Räck, M. Radziszewski and P. Nurmi, “Contextual personalization of a mobile multimodal application”,International Conference on Internet Computing (ICOMP), 2005, Las Vegas, USA.

• P. P. Boda, R. Kernchen, K. Moessner, F. L. Dáv alos, J.Storgårds, M. Sutterer, O. Droegehorn, and G. Giuliani (2005). Dev eloping context-aware and personalisedmultimodal applications in the MobiLif e EU project. In International Conference on Multimodal Interaction (ICMI'2005), Trento, Italy , October 3-6 2005.

• Kernchen R, Boda P, Mößner K, Mrohs B, Boussard M, Giuliani G (2005). Multimodal user interf aces f or context-aware mobile applications, 16th Annual IEEEInternational Symposium on Personal Indoor and Mobile Radio Communications (PIMRC'2005), Berlin, Germany , September 12-14, 2005.

• P.P. Boda, D. Bonnef oy , E. Kurv inen, F. Lopez, M. Karlstedt & M. Salacinski (2006). Family Map: Dev elopment of a Multimodal Context-Aware Utility f or Families.Submitted to the MobileHCI 2006 Conference, Espoo, Finland, September 12-15, 2006

• Context-rule generation with multimodal interface:• D. Bisztray (2006). Peer-to-peer rule distribution in context-aware mobile applications. In microCAD International Scientific Conference (microCAD 2006), Miskolc,

Hungary , March 16-17 2006

• Multimodal Integration• P.P. Boda & E. Filisko (2004). Virtual Modality : a Framework f or Testing and Building Multimodal Applications. HLT/NAACL 2004 Workshop on Spoken Language

Understanding for Conversational Systems, Boston, Massachusetts, USA, May 7, 2004

• P.P. Boda (2004). Maximum Entropy Based Approach f or Multimodal Integration. Sixth International Conference on Multimodal Interfaces (ICMI'04), Pennsy lv aniaState Univ ersity , PA, USA, October 13th-15th, 2004

• P.P. Boda (2004). Modality Integration in a Wider Sense. Inv ited paper on the “Robust and Adaptive Information Processing for Mobile Speech Interfaces” DUMASFinal Workshop on COLING-2004, Genev a, Switzerland, August 28th-29th, 2004


References

• SensorPlanet:

• Nokia Research Center external site: http://research.nokia.com

• More on SensorPlanet: http://sensorplanet.org (will be up soon)

Contextual multimodal interfaces and device-centric ...hci.stanford.edu/cs547/abstracts/05-06/Boda...

Documents

Transcript of Contextual multimodal interfaces and device-centric ...hci.stanford.edu/cs547/abstracts/05-06/Boda...