1 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Contextual multimodal interfaces and
device-centric sensory networks
BODA Péter Pál
Software & Applications Technologies Laboratory
Nokia Research Center
Helsinki, Finland
[email protected] seminar, Stanford
April 14, 2006
2 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Outline
• Enabling technologies
• overview of past and current activities
• Context-aware & multimodal applications
• MobiLife EU project
• CoMMod & zappa
• Virtual Modality and Multimodal Integration
• Device-centric sensory networks
• Whither User Interface?
3 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Motivation and Objectives
• Ultimate goal: provide enhanced user experience
• human-human communication relies on a full range of modalities
• seamless interaction, ease-of-use, naturalness, predictability
• Use of Advanced User Interface solutions for mobile applications and services
• speech & language technologies (recognition, synthesis, natural language understanding, Q/A:question-answering, etc.)
• multimodality (combining various intentional user inputs, e.g. speech & gesture)
• modularity, easy deployment, multiple devices
• Intelligent combination of additional information with current state of the system, i.e. howto affect the actual performance of the User Interface
• context-awareness: how the actions of the system can be enhanced by using parametersdescribing the existence of the user and the system in the physical (and virtual) world
• sensory inputs: both user-originated and environment related, ambient knowledge, etc.
4 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Enabling technologies
5 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Past and current activities on enabling technologies
• Speech recognition & text-to-speech synthesis• embedded name dialling, connected digit recognition
• embedded large vocabulary speech recognition
• TTS for device-based use
• Language Understanding• adapted speech recognition technology for semantic unit recognition
• discrete non-aligned HMM and genetic algorithm based approaches
• Multimodality• own implementations and development of integrated technology
• contribution to W3C standardisation
• Haptic/tactile• experimenting event-triggered haptic gesture based interaction
• exploring novel use of tactile channel
• User experience• user-centered design, usability studies, human factors
6 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
3. System
Mobile access to Internet - user experience building
blocks (Virpi Roto et al.)
1. User 2. Context Mobile device Application Connection Site
Need Physical context Appeal Usability Availability Value
Motivation Social context Connectivity Content support Speed Usability
Experiences Temporal context Usability Functionality Cost Appeal
Expectations Task context Memory space Trust Trust
Mood Battery lifetime Gateway
Resources Performance (Page optimization, Access restrictions)
7 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Some of the related EU and national cooperations
• Finnish National project (PuttePuh): first Finnish continuous ASR
• COST 249, 278 – speech technology, dialogue systems
• LC/TC-STAR - database collection for speech technology development
• CATCH-2004• Developing conversational multilingual dialogue systems• Multimodal applications and architecture
8 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
NEW CONTEXT!
NEW CONTEXT!
NEW CONTEXT!
Movies
tonight?
…. starting time of the 1st?
…. duration? (in Finnish)
…. description?
NEW CONTEXT!
Movies withMichael Douglas?
Welcome to Program Guide!
How may I help you?
… it starts at 5.15pm
I found the following
programs …
… it is 1h 25min long(in Finnish)
… no text message,thanks. (in Finnish)
I can offer the
following
description ….
To text message?
What kind of info
I can offer next?(in Finnish)
Programs for youngsters?
(in Finnish)
… sorry, no programs
for youngsters(in Finnish)
Michael Douglas is
in Coma … Channels?
BBC World, CNN,
Eurosport, TCM
What’s on BBC Worldtonight at 10pm?
World News at 10pm
… it takes 5 minutes(in Finnish)
That’s all for now.
Good bye!
…. duration? (in Finnish)
Bi-lingual Program Guide Information System(one of the outcome of the CATCH-2004 EU project)
9 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Combining context-awareness and
multimodality
10 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Life Goes Mobile with MobiLife (EU project 2004-2006)Vision and Challenges
Vision: Mobile & wirelessapplications and servicesbecome everyday things
All
phot
os ©
Nok
ia
People shift between! different (blurring) roles in! participating different groups,! using multiple devices! with different modalities! over different networks
Challenge: How to helppeople, especiallyfamilies, in making theireveryday life easier
MobiLife Overview September 2004 Slide 11
IST-511607
Life Goes Mobile with MobiLifePartners
Suunto, ElisaUniversity of HelsinkiHelsinki U of TechnologyNokia, Ericsson
Ericsson
MotorolaUniversity of SurreyNEC Europe (site: Ger)
NEOSHewlett-PackardSiemensTelecom Italia
Alcatel, Motorola
Telematica
FOKUSUniversity of KasselDoCoMo Eurolabs
BellStream
UNIS
Finland
Germany
Poland
Czech Rep.
SwedenNetherlands
UK
France
Italy
Coordination:
App. own. & SMEs:
Operators:
Manufacturers:
Academia:
Countries:
Partners:
Nokia
5
3
8
6
9
22
MobiLife Overview September 2004 Slide 12
IST-511607
“Self-awareness”
• Local environment
• Automatic configuration arrangementof devices, services, and localconnectivity
• Multimodal interfaces, personalisation
“Group-awareness”
• Context and presence support
• Novel privacy and trust models
“World-awareness”
• Automatic support for seamlessaccess to and delivery of servicesacross different domains
Communication Spheres
BAN – Body Area Network• Bluetooth, RFID, sensors, …
PAN – Personal Area Network• Bluetooth, WLAN, …
WAN – Wide Area Network
• 3G, B3G, WLAN, *DSL, …
People shift between differentroles in participating differentgroups, using multiple deviceswith different modalities overdifferent networks
Life Goes Mobile with MobiLifeTechnical Viability and User Acceptance
13 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
The Scope
• Everyday things:• Things that are around us
• Things that are so obvious we do not even notice them
• Can we detect them?
• Contextual analysis and Sensory inputs
• Everyday life easier:• The user experience comes from the ease-of-use, seamlessness
• User Interface is the first touch to the application/system
• The simpler the better, the shorter interaction the more effective
• Imitation of human-human communication: multimodality
• Work Package 2:• Self-awareness, personalisation, profiling and its use
for modifying the actual UI
• Advanced user interface solutions
(multimodality, sensory inputs, tangible UIs)
14 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
The Approach
• User-Centered Design:
• Collect typical scenarios in the domain
CA
G
g1
g2
c1H a1
J B
• Analyse representative user behaviours
• Map them for technical requirements and functionalities
• Define functional and system architecture to implement or re-usefrom existing systems
MainWeather
info
Babyinfo
• Verify implementation via applications & intensive tests
15 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Few examples of Multimodal Interaction
• Combining modalities naturally/seamlessly as the user prefers:• gestures: when spatial interaction is preferred (browsing, map navigation)
• speech: when quick selection, shortcut is useful
• gaze tracking: hands-busy situations (in noisy environment)
• tactile feedback: providing prompt alert or non-intrusive feedback
• etc.
16 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Examples of contextuality & UI dynamics
• Using location information:• FamilyMap: “Is there any place to feed the baby nearby?”
• Deictic reference can be resolved by knowing current location
• Outcome: results in smoother and faster interaction, 1 turn saved
• Using personal settings, profile:• Infotainer: User jumps to the car " corporate news streamed automatically
• Knowing the date and time and user set preferences, the desired news are delivered on the instantlyavailable audio channel
• Using biomedical/sensory information:
• Wellness: detecting kids’ heart rate on a football training• Instant feedback to trainers and parents about the performance
• Data can be logged for later comparison, “competition against me”
• Detecting emotional state, stress level• Dialogue style, system guidance can be adapted accordingly
17 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
The implemented contextual UI architecture
DeaMon and DeGan:•Advanced modality and device controlmechanisms• Contextual inputs affect how modalitiesand devices used, as well as how content ispresented
18 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Demonstration Applications
Content is
synchronised
between devices
User can switch
from one device
to another at
any moment
Multimedia Infotainer
FamilyMap
MainWeather
info
Babyinfo
“Logistic
navigation”,
contextual
proactiveness,
social networking
Being aware of
”wellness” condition
Use of sensors
as passive input
modalities
Wellness-Aware
GamingContext Watcher
Location clustering,
sport sensors,
finds POIs, buddy activity
show, blogs & pics
19 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Multimedia Infotainer
Context-aware multimodality allows personalised access to content
(e.g. news) depending on current preferences, location and/or device(s)
Content can be showed in
different multimodal ways,
depending on available devices
(PC, mobile phone, TV)
Content is
synchronised
between devices
User can switch from one device
to another at any moment
• An advanced multi-functional application on a mobile device with abilities to:• Enable a user to communicate and receive personalised news via multiple channels;
• Retrieve content based on the user’s personal preferences;
• Support multiple multimedia output devices;
• Select best delivery method/media according to the user’s preferences and/or context reasoning engine
20 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Wellness-Aware Multimodal Gaming System
• Wellness-Aware Multimodal Gaming System integrates fitness monitoring features and comparisoncapabilities:
• Allows family members to share wellness, environment and location information during/after training orgame;
• Family members and football coach are able to look at heart rate, speed and wellness condition of asingle user or a whole football team
Parents looking at the game together
Being aware of ”wellness” condition
Use of sensors as passive
input modalities
Using sensors and collecting sensory data allows users to share,
record and compare wellness information
21 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
• FamilyMap allows enriched contextual navigation:
• Provides baby-care related information for families with babies, especially on the move in unknowncities;
• Assists families to find “logistic” locations (where to feed a baby, change diaper, spend somerelaxed time with a baby, paths suitable for baby carriage);
• Social interaction: leave and receive virtual-post-its in hot-spots about preferred places dependingon context and preferences;
• Advanced multimodal interaction via multiple devices assembled to the baby carriage
Main
Navigate
Weather
info
Babyinfo
It will soon rain.
Find nearest
metro s tation "
FamilyMap
Weather change info
Navigation view
O K/YES
O ptions
More/Nav
NO
Weather
info
Babyinfo
1. There is playground in
200m named Sunnydale
playground
2. There is a baby clothes
shop after the next corner
O K/Main
Leave Msg
Weather
info
Babyinfo
Great place, my children
loved here. Ciao, Daniela
MainWeather
info
Babyinfo
FamilyMap provides navigation enriched with contextual information
(logistics, weather, point of interest, social interaction)
Social interaction,
virtual post-it
Point of interest
description
22 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
FamilyMap on the concept level
Main planned functionalities:
Provides baby-care related information
for families with babies, especially on the
move in unknown cities.
It assists families to solve practicalproblems, to find “logistic” locations,places where a parent can feed a baby (e.g.a microwave is available), change diaper,spend some relaxed time with a baby, canwalk easily with the baby carriage: i.e. pathhaving no stairs or construction going on.
Promoting social networking: parents canleave and receive virtual-post-its in hot-spots about preferred places, e.g.cafeterias, shops for babies, etc.,depending on context and preferences.
Navigation, tracking
(Nokia, Wayfinder)
GPS
System-User interaction
(e.g. iPaq, Nokia)
SUUNTO:
Sensory devices
(temp, speed, etc.)
Tactile
feedback
23 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
MainWeather
info
Baby
info
<map with POI>
Functionalities
More/Navigate
New
destination
Navigation view
Main
Navigate
Weather
info
Baby:Diaper
wet
Two hours on the road.
Find a place to change
the diaper "
Baby status info
OK/YES
Opinions
More/Navigate
NO
Weather
info
Baby
info
1. There is playground in 200m named
Sunnydale playground
2. There is a baby clothes shop
after the next corner.
Point-of-Interest
descriptions
OK/MainWeather
info
Baby
info
“Great place, my children loved here.
Ciao, Daniela”
Leave a
message
Social interaction,
virtual post-it
Main
Navigate
Weather
info
Baby
info
It will soon rain.
Find nearest metro station "
Weather change
info
24 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
FamilyMap system architecture
MS MapPoint
Server
FamilyMaps
ServerR
map
coordinate
edit PoI data
Nokia 9500
VibratorGPS
Nokia 9500
VibratorGPS
Nokia 9500
VibratorGPS
25 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Links
• MobiLife: http://www.ist-mobilife.org/
• Context Watcher: http://www.lab.telin.nl/~koolwaaij/showcase/crf/cw.html
(links to Cell-ID repository, auto-blog, downloads, etc.)
26 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Internal activities:
CoMMod, zappa
27 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
CoMMod: Context-based multimodal rule creation & visualisation
Problem
• Rule creation for context-aware applications isproblematic and might prevent wide-spreaduse: typically all personalization-basedapplications require substantial user interaction.
• Rules can be quite complex (understandingwhat conditions in what combinations shouldtrigger actions)
Solution
• Make rule management seamless, usemultimodal user interfaces
• By visualisation means make the user awareand able to understand why a certain actionhappened
Benefit
• User is in charge what rules are used
• Straightforward rule distribution
mng calls ring even in silent
get traffic info
Sue calls not urgent to voicemail
Most recent firingrule visible
Currently firing rule,conditions valid
Active rules,not firing
Inactive rules
on mtng
on mtng
5pm on weekday
28 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Multimodal rule creation
• Process:
• Pre-defined building blocks for conditions and actions
• Drag & drop or speak out: “If Joe is calling then Send message .... to .... ”
If
or
Then
Conditions Actions
• Public &private rules
• Distributionover mForms
• Public rules’placeholders tobe filled up ondevice
29 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Use case: Desired profile selection(contextual/sensory events trigger device-based core application)
Corporate meeting
if LOCATION= (workplace and meeting room)
and DATE= working day
and PROXIMITY= 2 or more colleagues
and PROXIMITY_MODE= (silent or meeting)
then PUT PHONE TO SILENT MODE
30 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Use case: Manager calls
In a meeting….
if CALLER=own line manager
and LOCATION= (workplace and meeting room)
and DATE= working day
and MODE= (silent or meeting)
and ACTIVITY= not giving a lecture
then RING THE PHONE
NB.: it is fine to be ringed by the boss
during working hours even on a meeting,
but not when giving a lecture to a larger audience
31 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Context-story simulator and distributor: zappa
• Problem:
• How to test (in laboratory environment) the operation of the developed technologies(e.g. visualisation of firing rules, rule execution)?
• How to detect users’ behaviour under controlled circumstances in a context-richenvironment ?
• Simulated contextual bits should be mixed with real-life events
• Approach:
• Generate stories that contain simulated contextual events (e.g. change of location,proximity of family members, colleagues, change in environmental conditions, etc.)
• Distribute stories either from a mobile device (can take real-life elements intoaccount, e.g. proximity, location) or from a server
• zappa!" zentralised autonomous probabilistic pragmatic appliance
32 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
t (sec)
0 5 10
ID: 1.1Story :Trav ellerSender: zappa_#1LOC*=“home”t=0sec
ID: 1.2Story :Trav ellerSender: zappa_#1LOC*=“plane”t=10sec
ID: 1.5Story :Trav ellerSender: zappa_#1LOC*=“work”t=20sec
ID: 1.9Story :Trav ellerSender: zappa_#1LOC*=“restaurant”t=35sec
ID: 1.3Story :Trav ellerSender: zappa_#1WEATHER*=“rain”t=10sec
ID: 1.4Story :Trav ellerSender: zappa_#1TRAFFIC=“heav y ;delay ”t=10sec
ID: 1.7Story :Trav ellerSender: zappa_#1IDENTITY=<colleague>t=20sec
ID: 1.8Story :Trav ellerSender: zappa_#1ACTIVITY*=“meeting”t=20sec
ID: 1.6Story :Trav ellerSender: zappa_#1ENVIRONMENT=proximity ;IDENTITYt=20sec
ID: 1.10Story :Trav ellerSender: zappa_#1ACTIVITY=“call;<spouse>”t=35sec
An example story
33 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
t (sec)
0 5 10
ID: 1.1Story:TravellerSender: zappa_#1LOC*=“home”t=0sec
ID: 1.1Story:TravellerSender: zappa_#1LOC*=“home”t=0sec
ID: 1.1Story:TravellerSender: zappa_#1LOC*=“home”t=0sec
Architectural setup
TopologyFile Edit LocateView Help
Mount
431 7437
1950 79%
/
NetworkTraffic Help
t (sec)
0 5 10
ID: 1.1Story:TravellerSender: zappa_#1LOC*=“home”t=0sec
ID: 1.1Story:TravellerSender: zappa_#1LOC*=“home”t=0sec
ID: 1.1Story:TravellerSender: zappa_#1LOC*=“home”t=0sec
server
zappa
1. Create story on server, test, run locally then distribute to test device2. Test user start test by downloading the story and the corresponding playback parameters3. Story downloaded to test device. User starts test, zappa"is playing back according to
playback parameters4. Device acts as central story player, distributes story elements to further test devices, can
interleaves with real-life elements
1.
2.
t (sec)
0 5 10
ID: 1.1Story:TravellerSender: zappa_#1LOC*=“home”t=0sec
ID: 1.2Story:TravellerSender: zappa_#1LOC*=“plane”t=10sec
ID: 1.5Story:TravellerSender: zappa_#1LOC*=“work”t=20sec
ID: 1.9Story:TravellerSender: zappa_#1LOC*=“restaurant”t=35sec
ID: 1.3Story:TravellerSender: zappa_#1WEATHER*=“rain”t=10sec
ID: 1.4Story:TravellerSender: zappa_#1TRAFFIC=“heavy;delay”t=10sec
ID: 1.7Story:TravellerSender: zappa_#1IDENTITY=<colleague>t=20sec
ID: 1.8Story:TravellerSender: zappa_#1ACTIVITY*=“meeting”t=20sec
ID: 1.6Story:TravellerSender: zappa_#1ENVIRONMENT=proximity;IDENTITYt=20sec
ID: 1.10Story:TravellerSender: zappa_#1ACTIVITY=“call;<spouse>”t=35sec
PLAYBACK PARAMETERS
TEST_ID=1234STORY1=trav el_car_home_workSTORY2=naPLAY=1DURATION=2MINRANDOM_STORY=naRANDOM_EVENTS=naINFINITE=na
3.
4.
34 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Virtual Modality and
Multimodal Integration
35 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
…. when it comes to multimodality
• Satisfied users <== Full understanding of intention
• primary goal is to interpret the user’s intention accurately
• not only at the time of the interaction, but in respect to dialogue history, previous sessions, userpreferences, context (meant here in a wider sense), etc.
• Seamless interaction
• no explicit modality selection should be forced on the user – unless some extra circumstancedemands so (e.g. switch off ASR when too noisy)
• Consistency, equality
• all modalities should be handled in a similar fashion
• no reason to have a primary modality (weights can be allocated if usability study supports)
• our senses work continuously: equally, in a complementary and overlapping fashion
• Transparency
• Multimodal Integration (MMI) must support information flow across modalities
• neuropsychological evidence shows that cross-modal interaction occurs constantly, more stronglyin cases where one of the sensory inputs is degraded/missing
36 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
The model
symbolic output
sensory input
(e.g. speech)
Modality
pro
cessing p
ipelin
e
….….
symbolic output
sensory input
(e.g. gesture)
<map>
gramm
ar(s)
(x,y
)A semantic tier:
is the bridge across modalities
37 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Notions and Implications
• Semantic tier:
• description of the domain, how the worldis perceived
• Dual view:
• speech should not know aboutcoordinates, gesture should not knowabout grammars
• Cross-modality:
• follows neuropsychological findings(McGurk effect, synaesthesia, etc.)
• see Coen: Perceptually motivated MMIand P. Haikonen about consciousmachines
• Eventually, this is ontology
• later might be Semantic Web
• Can be extended
• to any number of modalities
• sufficient: there is a quantitative descriptionin terms of resources for a modalityprocessor (e.g. grammar).
• Cross-modal triggering is:
• bi-directional
• can emerge at various points
• Why is it good for us:
• equal role in perception, no primarymodality
• triggering one modality’s perception in caseof early evidence(s) in another modality
• unimodal performance can be boosted (e.g.focusing to area, allowing only certainobjects, selecting grammar)
• mutual disambiguation
38 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
gramm
ar(s)
(x,y
)
<object>
smell ga
zed
gramm
ar(s)
(x,y
)
<object>
smell ga
zed
Generalised
gramm
ar(s)
(x,y
)
<object>
smell ga
ze
39 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Further notices
• Eventually, this is a complex model
• Assumption:• Available multimodal data could be utilised in the development (testing & building) of
multimodal systems
• Learn/validate internal structure, train parameters, etc.
• Chicken-and-the-egg problem:• No real data can be collected w/o an operational system
• No data ! difficult to develop well-tuned complex systems
40 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
The tasks
• Acquire database:• purchase/collect/….
• ! transform/generate !
• Create a statistical (hybrid) integration framework:• which utilises the multimodal data in the training/testing phases
• statistical:• other modules are (speech reco-ASR, handwriting reco-HWR, natural language understanding-NLU, ….)
• huge amount of contextual data is/will be available
• MMI:• typically ad hoc methods available, except AT&T’s FST (finite-state transd.)
• challenging to create a unified framework
41 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
How to get multimodal data?
• There is a huge amount of data out there!
• Multimodal systems can also operate in speech-only mode
• Vast amount of speech data (also in textual form):
• Used for component training (speech recogniser, language modelling, …)
• Available as logs of operational spoken dialogue systems
• Idea:
• Take transcribed data and generate “multimodal” data
• Select pre-defined concepts (e.g. location references) and remove from the speech channel
• Replace them with deictic expressions
• Put the removed concepts to an abstract channel: Virtual Modality
• NB.: presence of deictic expressions is not necessary (cf. Oviatt et al. 1997)
42 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Virtual Modality …
• … is a fictional modality channel, carrying semantically meaningful input elements
• … it can be virtually anything, we do not care how it evolved, sensed, etc.• Intentional input: any available input modality (speech, gesture, gaze, text, etc.)
• Passively available info: pragmatics, interaction histories, user preferences, “context”,sensory/affective data, location, etc.
• Utilisation:• Testing integration modules
• Study timing effect, mutual disambiguation
• Cross-modal interaction
• Building statistically motivated integration mechanism
43 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Virtual Modality Implementation
44 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Virtual Modality Implementation
45 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Virtual Modality Implementation
46 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Eventually ….
• Multimodal Integration:
Sp’MMI
• “Inverse” Multimodal Integration:
G
Sp’
VM
Sp
C
47 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Parameters of the model
• Number of allowed deictic replacements per utterance
• Selection of deictic replacements:
• this, that, here, there, over here, over there
• this $, that $: e.g. this city, that university
• Inter-modal lags (deictic position vs. VM entry)
• Pre-defined user types (discrete lags: word position)
• Simulated user population:
• E.g. 80% quickie, 20% precise (cf. sequential & simultaneous)
Too-fast -2
Quickie -1
Precise 0
Slowhand +1
Everlate +2
48 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Data generation
• Application domain:• Voyager (MIT/SLS) – traffic and city guide
• Conversational spoken language interface
• Original data:• Output of speech recogniser
(i.e. input sentences)
• Corresponding NLU parse frames
(used to identify location references)
Number of sessions 1099
Number of turns 6077
Average number of turns
per session 5.53
All location references 6982
Average number of
references per turn 1.14
Number of different
location reference patterns 203
The five most frequent
location reference patterns:
in <….>
of <….>
show_me <….>
on <….>
from <….> to <….>
9.95%
8.15%
6.39%
5.69%
3.95%
49 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Two questions to be answered
• Is the generated data realistic?
• How well does the existing system work with the Virtual Modality data?
50 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Data quality
• Real multimodal data from Voyager (MIT/SLS):
• 68% of turns have deictic expressions (5583)
• The first three most commonly used location reference patterns are the same as in thespeech-only case:
in …, of …, show_me …
• When two location references used in a sentence, the first one is with deictic:
how do i get there from kendall square
• The used deictic expressions: this, that, this one, here, there
• No occurrence of: over there, this university, that city, …
By courtesy of Jim Glass, MIT
51 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Evaluation setup
Galaxy Communicator
Batchmodeserver
VM data MM eventdatabase
Original sentences
{c directions:subject 1:domain "Voyager":utterance_id 6:pred {p from
:topic {q university:name "Harvard University" } }
:pred {p to:topic {q university:name “MIT" } }
Original frame
{c directions:subject 1:domain "Voyager":utterance_id 6:pred {p from
:topic {q university:name "Harvard University" } }
:pred {p to:topic {q pronoun:name “here" } }
VM frame
?“From Harvard University to MIT”
“From Harvard University to here”
D1 = “MIT”
D1_begin = 4
D1_end = 5
{c directions:subject 1:domain "Voyager":utterance_id 6:pred {p from
:topic {q university:name "Harvard University" } }
:pred {p to:topic {q university:name “MIT" } }
After Context Resolution
Galaxy Communicator
Batchmodeserver
VM data MM eventdatabase
Original sentences
{c directions:subject 1:domain "Voyager":utterance_id 6:pred {p from
:topic {q university:name "Harvard University" } }
:pred {p to:topic {q university:name “MIT" } }
Original frame
{c directions:subject 1:domain "Voyager":utterance_id 6:pred {p from
:topic {q university:name "Harvard University" } }
:pred {p to:topic {q pronoun:name “here" } }
VM frame
?“From Harvard University to MIT”
“From Harvard University to here”
D1 = “MIT”
D1_begin = 4
D1_end = 5
{c directions:subject 1:domain "Voyager":utterance_id 6:pred {p from
:topic {q university:name "Harvard University" } }
:pred {p to:topic {q university:name “MIT" } }
After Context Resolution
52 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Experiments with the VM data
• 436 non-identical sentences
• 1 deictic per sentence with zero lag:• yields 82.03% identical output
• Problem: preposition and randomly selected deictic sometimes results inungrammatical expressions
• e.g. “how about on over there?”
• Few cases were not covered by the parser
• 2 deictic per sentence with zero lag:• 78.26% accuracy
• Incorrectly handled highways and highway_numbers by the data generation process
• Temporal effects are not tested (server applies a fixed size window)
53 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
The integration framework
• How to integrate various inputs?• textual, symbolic, (maybe) physical measures
• If we think about the resolution of deictics• reminds Word Sense Disambiguation
• Can something be used here from NLP?
• Maximum Entropy framework• Elegant approach to combine various inputs via features
• See work of Adam Berger, Adwait Ratnaparkhi, ….
54 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Maximum Entropy approach
• Use/say no more than known• no more assumptions than the data supports
• find smoothest solution
• Features:• logical connection between input and output domain
• binary or continuous values
• Exponential model:• basically a weighted distribution• (Engineers: like Fourier spectrum)
• weights are learned from data
55 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Features
• bag-of-word type feature: the co-occurrence of a word w in the Sp channel or entrym in the VM channel and that of a concept C is checked
• contextual feature (NB. lingustic context): the co-existence of patterns in Sp anda given concept is measured. In the current set-up left context with 3 words relativeto the deictic expression is considered
• prepositional feature: the word before the deictic expression was checked whetherit is a preposition
56 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Results
• Integration accuracy for the first 3-best:
• Problems:
–Preposition detection, handling no-preposition cases
–Too many labels (176) -> atomic labels?
–Amount of features -> data sparseness, smoothing parameters
86.772.833rd best
83.947.422nd best
76.521st best
Accuracy for 2 and 3 bestAccuracy %Rank
• Contextual info helps:
• utilising (virtual) gesture type info, can increase accuracy by 2-4% points
57 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Sensory input on a large scale
58 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
SensorPlanet:
a global test platform for mobile-centric wireless sensor networkresearch
Owner: Henry Tirri
Contacts: Barbara Heikkinen, Boda Péter
Nokia Research Center, Helsinki
59 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Why a phone should play any rolein Wireless Sensory Nets?
... because it is there ...
60 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
SensorPlanet
a Nokia-initiated global cooperation on large scale
wireless sensory networks
mobile phone oriented sensory input acquisition
a research platform that enables collection and analysis
of sensory data on a never seen quantity
61 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Mobile Wireless Sensor Network Scenarios
Mobile phonesact as sensor
node and/or sinknode.
Mixed sensornetworks which
consists of ordinarysensors and mobile
sensors.
Sink node
Sink node
Central
Data
Repository
„Green phones“
prov iding
env ironmental data,
sharing
“Service manifolds”
Traffic, masses of people ...
the more data shared the more accurate
(© Copyright 2000 Martin Lamm)
(© Copyright 2006 Petri Koskelainen)
“Urban games”
Locations shared,
participation of the public ...
62 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
SensorPlanet University Cooperation
• Ultimate goal to have multiple SensorPlanet sites around the globe
• Data gathered on sites accumulated in central repository
• Common access to accumulated data for research purposes
• Single interface to sensory data from the world
• Universities:• MIT/CSAIL and Media Lab ! ongoing work on body sensor network and CARTEL
• UCLA
• NYU
• Stanford
• Berkley
• Copenhagen
• EPFL/Zurich
• tbd.
63 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
More on potential use cases
64 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Mobile handset in Wellness
Glucose
Activity
Heart beat
Home systems
Services
Blood Pressure
Wrist
Applications
65 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Recording everything (SharMe)
What I feel etc.
What I see
What I hear
What we say
What I write
Where I am
66 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Tagging, Virtual Graffiti
67 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
„Dynamic highway pay”
68 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Prototype Device Implementation – Wireless Motion
Band
• Wireless Motion Bands are designed forcomfortable wear in different body parts toenrich user experience in gaming.
• Wireless Motion Bands allow the use of
(full) body motion tracking in gaming:
• Direct 3D body motion control of a gamingcharacter
• Detection of body poses & gestures
• Local multi player gaming. Low-latency multidevice Bluetooth connectivity.
• Interoperability with Wireless Game Pads.
Design by Mikko Aarras, Nokia Design
By Courtesy of Jukka Salminen
69 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
User Interaction Examples – Virtual Snowboarding
• Direct bodymotion controlof a computercharacter
• Body poses
• Balance
• Direction
• Speed
• Jumps
• Landings
70 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
There is always a need of an ecosystem...
Mobile device asintelligent hub andportable applicationplatform
IR, Bluetooth,
RFID, USB
PC and server applications andhosted internet services
Server
PC
- Coach- Trainer- Doctor- Nurse- Caregiver- Operator DB
- Serviceproviders- Peercommunity
Internet services
Portfolio ofcommunication andmonitoring devices
Some sensors may beembedded in terminal
Ope
rato
r tr
affic
71 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Some research challenges
72 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Some key technological challenges
• ”Sensor data challenge”: how to make the high-volume sensor datauseful
• query/search problem
• data management problem
• ”Mobile adaptation challenge”: self-configuration and adaptation
• ”Integration challenge”: how to integrate (two-way) WSN to Internetbased services (create the “gateways”)
• “Power challenge”: even high-power mobile nodes are not enough
• “User interaction challenge”: how end-users will benefit, interact
73 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
What sort of raw (context) data management
problem we are facing?
• a multidimensional (2-30) vector of real values
• frequency 0.5s - 1 day
• typically “always-on”
• a 1-4M pixel image
• frequency 10 min - week(s)
• very irregular, high-intensity bursts (many images within minutes)
• a 100K-1M sound file
• frequency 1 min - days
• irregular; streaming
• Naturally many application domains require a mixture of these
10K phones - vector every 2 min results in 2.7 (US) billion vectors/year
200M phones - vector every 60 min results in about 1012 vectors/year
74 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
SensorPlanet Core handset(s)
• Nokia E70 Smartphone
• EGSM 900, GSM 1800, GSM 1900 and WCDMA (3GPP Release 4) networks
• Speech codecs: FR, EFR and AMR f or GSM and AMR f or WCDMA
• Weight: 127 g (with Nokia Battery BL-6C)
• Dimensions: 117 mm x 53 mm x 22 mm (closed), 102 cc
• Full messaging key board with f old open design
• Up to 75MB internal dy namic memory
• 2.0 Megapixel camera with resolution up to 1600x1200
• 8x digital zoom
• Recording f ormats (encoding): 3GPP, H.263 v ideo, and AMR radio
• Video resolution CIF (352 x 288), QCIF (176x144) or Sub QCIF (128 x 96 pixels)
• Pop-Port™ interf ace
• USB 2.0 f ull speed supported v ia Pop-Port™
• Connect to a compatible PC wirelessly or with Nokia Connectiv ity Cable CA-53
• Integrated inf rared module (IrCOMM, IrOBEX, IrTiny TP)
• Bluetooth wireless technology 1.2
• Install applications with Nokia PC Suite f or the Nokia E70 phone
• Video and audio streaming (3GPP and RealMedia)
• WCDMA (3GPP Release 4)
• WAP 2.0
• Multiconnectiv ity
• WLAN connection can be shared by multiple applications at the same time. For example: email, browserand sy nchronization
• Possible to use WLAN, Bluetooth technology , USB and inf rared at the same time
• Max 6 Bluetooth technology connections in use at the same time 4
• WCDMA (3GPP Release 4)
• EGPRS (Class B, MSC 10)
• GPRS (General Packet Radio Serv ice) Multislot Class 10
• Dual Transf er Mode (Multislot Class 9)
• Wireless LAN 802.11g, 802.11e and 802.11i
75 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
SensorPlanet Core handset(s)
• Nokia 770 Internet Tablet
• Weight: 230 g
• Dimensions: 141 x 79 x 19 mm
• Display
• High-resolution (800x480) touch screen with up to 65,536 colors
• Memory: Flash 128MB (>64MB for user)
• Memory card: 64MB RS-MMC (Reduced Size - MultiMediaCard)
• Operating system: Internet Tablet 2005 software edition
• Applications (in 2005 software edition) Web Browser (Opera 8), Flash Player version 6, EmailClient,Internet Radio, News Reader, Media players, Image viewer, PDF viewer, File Manager,Search...
• Audio: MP3, MPEG4-AAC, WAV, AMR, MP2
• Image: JPEG, GIF, BMP, TIFF, PNG, Animated GIF format,
• SVG-tiny, ICO
• Video: MPEG1, MPEG4, Real Video, H.263, AVI, 3GP
• WLAN: 802.11b/g
• Bluetooth specification: 1.2
• For Internet connection and file transfer via phone
• Profiles supported: Dial-up Networking, File Transfer, Generic Access, SIM Access and SerialPort profiles
• USB device mode for PC connectivity
• RS-MMC
• 3.5 mm stereo audio out
76 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
SensorPlanet demos
under development
77 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
SensorPlanet’s Horizontal & Vertical implementations
1) Magrathea 3)Virtual Graffiti
In-door attachment and
spotting of virtual objects
BT-based positioning
Server-based repository
Passing primitive & robust
codes from device to device
Report back mechanism
about spreading
Op
era
tion
Tim
es
Sq
ua
re
Dirk Trossen (NRC/Hki)
A to be Open-Source platform
4) Nokia Remote
Sensing Architecture
(N-RSA)
WL
AN
-ba
sed
Po
sitio
nin
g
Inst
an
t M
ee
ting
Ro
om
Acc
ess
Vir
al R
ese
arc
h T
oo
lkit
To
ijala
stu
de
nts
’ in
no
vatio
ns
5) NRC SiteImplementation
WLAN based indoor
positioning
Sensor platforms
2) Interactive
Services
Infrastructure for
building urban services
78 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Virtual Graffiti Visualisation
• Virtual Graffiti develops a horizontal platformfor in-door tag placing
• Positions are determined by Bluetooth beacons
• User can leave/read out/modify/etc. virtualobjects left to physical places within theNRC/Toijala premises
• The current visualisation:
• Violet balls represent Bluetooth beacons andthe red ones at the centre of the picturerepresent clients. The red balls that constitutean oval represent other Bluetooth devices thatthe beacons have found.
• The thickness of the lines represents thefrequency of scans made between the clientand a Bluetooth device.
• On the top of the visualisation box the five lastevents are displayed.
79 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Operation Times Square
• Real time reporton test users’location
• Game: find yourgroup, gettogether, send pic
• Location isidentified basedon cell-ID
(© Copy right 2005 FreeLargePhotos.com)
SensorPlanet.org
80 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Play it again in Manhattan!
Get with your team to the Times Square,
with a little help ....
The Game (preliminary plan)
#1 Players are scattered around Manhattan
#2 Divide them into two teams on both side
of the Times Square
#3 Teams to get to Times Square
(© Copyright 2006 Petri Koskelainen)
(© Copyright 2000 Martin Lamm)
81 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
SensorPlanet.org …
soon to be launched
82 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Links & contact
• Nokia Research Center external site: http://research.nokia.com
• More on SensorPlanet: http://sensorplanet.org (will be up soon)
• Contacts:
• Henry Tirri ([email protected])
• Boda Péter ([email protected])
83 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Finally
84 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Whither UI?
My personal view about the ultimate User Interface:
• no explicit input from the user, yet, accurate system action
• already now: an elevator can take you where you intend to go
• new palyers: context, sensory input, bio-sensory signals, ambient intelligence, etc.
• intentional user input only in “error correction” situation
• Will the actual user interaction be reduced to minimal?
• Will the traditional UI become useless? --> Useless Interface?
Gábor Dénes (1969):
"If enough people work hard enough on the problem of speech recognition, itwill be solved by mid next century.“
Surely he did not think of multimodality, context-awareness, etc.
85 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Acknowledgements
• NRC/Helsinki:
• Agathe Battestini, Barbara Heikkinen, Mika Klemettinen, Jukka Linjama
• Jesper Olsen, Virpi Roto, Henry Tirri, Vétek Ákos
• NRC/Tampere:
• Kiss Imre, Pekka Ollikainen
• NRC/Budapest:
• Szomolányi Márton, Bisztray Dénes
• NRC/Toijala:
• Marko Mahkonen, Tuomo Heinonen
• MobiLife/EU partners
86 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
Thank you!
87 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
References
• Enabling Technologies:• Publication type: Conference paper to COLING-ACL 2006
• Author(s): Daniela Oria & Jesper Olsen
• Title: Statistical language modeling with semantic classes for large vocabulary speech recognition in embedded systems
• Status: submitted
• Publication type: conference paper (ICASSP 2006)
• Author(s): Jussi Leppänen, Imre Kiss.
• Title: Gaussian Selection with Non-Overlapping Clusters for ASR in Embedded Devices
• Status: accepted
• Publication type: conference paper (ICASSP 2006)
• Author(s): J. Olsen & D. Oria
• Title: PROFILE BASED COMPRESSION OF N-GRAM LANGUAGE MODELS
• Status: Submitted
• Publication type: workshop paper (W3C workshop on internationalizing the speech synthesis markup language, 2005)
• Author(s): Jilei Tian, Xia Wang and Jani Nurminen
• Title: SSML Extensions Aimed To Improve Asian Language TTS Rendering
• Status: Accepted
• Publication type: conference paper (Interspeech 2005)
• Author(s): Jussi Leppänen, Imre Kiss
• Title:Comparison of Low Footprint Acoustic Modeling Techniques for Embedded ASR Systems
• Publication type: conference paper (Interspeech/Eurospeech 2005)
• Author(s): Jilei Tian, Jani Nurminen, Imre Kiss
• Title: Duration Modeling and Memory Optimization in a Mandarin TTS System.
• Jukka Linjama & Topi Kaaresoja: Novel, minimalist haptic gesture interaction for mobile devices
• Demo paper -- NordiCHI, Tampere 23-27.10. 2004
88 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
References
• Enabling technologies:• Virpi Roto, Roland Geisler, Anne Kaikkonen, Andrei Popescu, Elina Vartiainen: Data Traffic Costs and Mobile Browsing User Experience, MobEA IV- Empowering the Mobile Web, 2006
• Matti Karjalainen, Péter Boda, Panu Somervuo, Toomas Altosaar (1997), “Applications for the hearing-impaired: evaluation of Finnish phoneme recognition methods”, Eurospeech'97, Rhodes, Greece, 22-25. September, 1997, pp. 1811-1814.
• P.P. Boda (1997), “From stochastic speech recognition to understanding: an HMM-based approach”, ASRU’97- IEEE Workshop on Automatic Speech Recognition & Understanding, Santa Barbara, CA,USA, 14-17. December, 1997
• Cristina Dobrin, Boda Péter Pál (1998), “Resolution of date and time expressions in a WWW-based dialogue system”, COST 249 (Continuous Speech Recognition over the Telephone) Workshop, Porto,Portugal, February, 1998
• P.P. Boda (1999), “Speech Understanding By Means Of Genetic Algorithms”, COST 249 (Continuous Speech Recognition over the Telephone) Workshop, Prague, Czech Republic, February, 1999
• Cristina Dobrin, Péter Boda & Kari Laurila (1999), “On Usability of Name Dialling”, ASRU'99 - IEEE Workshop on Automatic Speech Recognition & Understanding, Keystone, Colorado, USA, 14-16December, 1999
• Petri Haavisto & Boda Péter (2000), “On Voice Services and Data in Digital Mobile Systems” , VOTS-2000: Voice Operated Telecom Services: "Do they have a bright future?", Gent, Belgium, 11-12. May,2000, pp. 13-16.
• P.P. Boda (2000), “Visualisation of Spoken Dialogues”, ICSLP'2000 6th International Conference on Spoken Language Processing, Beijing, China, October, 2000, vol. I. pp. 146-149.
• P.P. Boda & Saija Lemmelä (2002), “Efficient Combination Of Type-In And Wizard-Of-Oz Tests In Speech Interface Development Process”, ICSLP'2002 7th International Conference on Spoken LanguageProcessing, Denver, USA, September 16-20, 2002.
• CATCH-2004:• P.P. Boda, Suresh Chande, Elviira Hartikainen, Nidhi Gupta (Nokia), Sirpa Autere (Elisa Communications), (2003), “Flexibility and efficiency through personalisation? Experiments with a conversational
Program Guide Information System”, EACL-03 Workshop on "Dialogue Systems: interaction, adaptation and styles of management", April 13-14, 2003, Budapest, Hungary
• Elviira Hartikainen & Boda Péter (2003), “Results and Observations from the Evaluation of a Telephony-based Conversational Information Retrieval System”,First ISCA Tutorial & Research Workshop on“Auditory Quality of Systems", April 23-25, 2003, Akademie Mont-Cenis, Germany
• Spoken Dialogue Systems:• Oria, Daniela, Vetek, Akos (2004) "Multilingual E-mail Text Processing for Speech Synthesis", in Proceedings of Interspeech 2004 - ICSLP, pp 841-844.
• Oria, Daniela, Vetek, Akos (2005) "Robust algorithms and interaction strategies for voice spelling", in Proceedings of Interspeech 2005 - Eurospeech, pp. 833-836.
• Oria, Daniela, Vetek, Akos (2005) "Automatic Generation of Speech Interfaces for Web-based Applications", in Proceedings of ASRU 2005 - IEEE Automatic Speech Recognition and UnderstandingWorkshop, pp. 397-402.
89 © 2006 Nokia Context-aware multimodality and device-centric sensory networks.ppt / 2006-04-14 / PB
References
• MobiLife:• More publications on: http://www.ist-mobilif e.org/ --> Publications
• Context Watcher: http://www.lab.telin.nl/~koolwaaij/showcase/crf /cw.html (links to Cell-ID repository , auto-blog, downloads, etc.)
• A. Salden, R. Poortinga, M. Bouzid, J. Picault, O. Droegehorn, M. Sutterer, R. Kernchen, C. Räck, M. Radziszewski and P. Nurmi, “Contextual personalization of a mobile multimodal application”,International Conference on Internet Computing (ICOMP), 2005, Las Vegas, USA.
• P. P. Boda, R. Kernchen, K. Moessner, F. L. Dáv alos, J.Storgårds, M. Sutterer, O. Droegehorn, and G. Giuliani (2005). Dev eloping context-aware and personalisedmultimodal applications in the MobiLif e EU project. In International Conference on Multimodal Interaction (ICMI'2005), Trento, Italy , October 3-6 2005.
• Kernchen R, Boda P, Mößner K, Mrohs B, Boussard M, Giuliani G (2005). Multimodal user interf aces f or context-aware mobile applications, 16th Annual IEEEInternational Symposium on Personal Indoor and Mobile Radio Communications (PIMRC'2005), Berlin, Germany , September 12-14, 2005.
• P.P. Boda, D. Bonnef oy , E. Kurv inen, F. Lopez, M. Karlstedt & M. Salacinski (2006). Family Map: Dev elopment of a Multimodal Context-Aware Utility f or Families.Submitted to the MobileHCI 2006 Conference, Espoo, Finland, September 12-15, 2006
• Context-rule generation with multimodal interface:• D. Bisztray (2006). Peer-to-peer rule distribution in context-aware mobile applications. In microCAD International Scientific Conference (microCAD 2006), Miskolc,
Hungary , March 16-17 2006
• Multimodal Integration• P.P. Boda & E. Filisko (2004). Virtual Modality : a Framework f or Testing and Building Multimodal Applications. HLT/NAACL 2004 Workshop on Spoken Language
Understanding for Conversational Systems, Boston, Massachusetts, USA, May 7, 2004
• P.P. Boda (2004). Maximum Entropy Based Approach f or Multimodal Integration. Sixth International Conference on Multimodal Interfaces (ICMI'04), Pennsy lv aniaState Univ ersity , PA, USA, October 13th-15th, 2004
• P.P. Boda (2004). Modality Integration in a Wider Sense. Inv ited paper on the “Robust and Adaptive Information Processing for Mobile Speech Interfaces” DUMASFinal Workshop on COLING-2004, Genev a, Switzerland, August 28th-29th, 2004
Top Related