ICAI'09 -The2009 International Conferenceon ArtificialIntelligencesonntag/icai09.pdf ·...

ICAI 2009 - General AI and Applications | 1

ICAI'09 - The 2009 International Conference on Artificial Intelligence

Design and Implementation of Combined Mobile and

Touchscreen-based Multimodal Web 3.0 Interfaces

Daniel Sonntag Matthieu DeruSimon Bergweiler

German Research Center for AI (DFKI), July 16, 2009, Las Vegas.


DFKI

Kaiserslautern Saarbrücken

Bremen

Berlin


Outline

» Introduction to Web 3.0 Interfaces

» Semantic Interface Elements (SIE) Architecture

» Combined Interaction Scenario

» Conclusions


Introduction to Web 3.0 Interfaces


Web 2.0 and 3.0

»Web 2.0: Folksonomies and Wikis»A wiki is a website that uses wiki software, allowing the easy creation and editing of any number of interlinked Web pages (Wikipedia).

» A folksonomy is a system of classification derived from the practice and method of collaboratively creating and managing tags to annotate and categorize content (Wikipedia).

»Web 3.0 = Semantic Web + Web 2.0»Provide machine understandable data for effective retrieval of commercial, scientific, and cultural data in a universal medium

»Provide new opportunities for semantically-enabled userinterfaces


No Presentation without Representation

»Anticipatory user interfaces should be human-centred and require human-like interactive functions.

»Rely on Semantic Web data structures.

»Transcend the traditional keyboard and mouse interaction metaphors.

»Provide the representation structures for more complex, collaborative interaction scenarios.


Common Data Model

Multimedia

Layer

GUI Layer

Data Content

Layer

Discourse

Layer

DialogueModel

Interaction

Model

Rendering

Videos

Pictures

Texts

Web 3.0 Access

Ontology Structures

Augmented Virtuality

Web 3.0 interfaces can either access Web 3.0 information on the Semantic Web and/or base the user interaction on Web 3.0 data


Discourse Layer

<http://www.semvox.de/ontology/odp#Hypotheses><http://www.semvox.de/ontology/odp#identifier><![CDATA[id_1]]>

<![CDATA[Search for images of Nelly Furtado]]></http://www.semvox.de/ontology/odp#completeUtterance>

</http://www.semvox.de/ontology/odp#WordLattice> </http://www.semvox.de/ontology/odp#hasContent> <http://www.semvox.de/ontology/odp#timeStamp><![CDATA[1247599485981]]>

</http://www.semvox.de/ontology/odp#timeStamp> <http://www.semvox.de/ontology/odp#confidence>

»<![CDATA[0.8]]><http://www.semvox.de/ontology/odp#attribute>

<![CDATA[utterance]]> </http://www.semvox.de/ontology/odp#attribute> <http://www.semvox.de/ontology/odp#value><![CDATA[S_SHOW_PICTURE_FROM_ARTIST_X]]>

</http://www.semvox.de/ontology/odp#value> …

<http://www.semvox.de/ontology/odp#attribute><![CDATA[artist_name]]>

</http://www.semvox.de/ontology/odp#attribute> <http://www.semvox.de/ontology/odp#value><![CDATA[Nelly Furtado]]>

</http://www.semvox.de/ontology/odp#value> </http://www.semvox.de/ontology/odp#NLUInfo>

</http://www.semvox.de/ontology/odp#hasNLUInfo> <http://www.semvox.de/ontology/odp#grammarId><![CDATA[comet_grammar.grxml]]>

</http://www.semvox.de/ontology/odp#grammarId> <http://www.semvox.de/ontology/odp#language><![CDATA[de]]>

…

</http://www.semvox.de/ontology/odp#Hypotheses>

Recognizer Interpretation

“Search for images of Nelly Furtado.”


Goals

»We discuss a combined mobile and touchscreen-based multimodal Web 3.0 interface which brings together »mobile scenarios,

»speech-based interaction, and

»touchscreen installations.

» The particular challenge we address is the implementation of a collaborative user scenario where the main application is installed on the mobile interface and a second one is installed on a touchscreen terminal.


Semantic Interface Elements (SIEs) Architecture


SIEs Technical Architecture

The most important part of the architecture is the Display Managerwhich observes the behaviour of the currently displayed SIEs(e.g., whether it was moved by the user or dropped somewhere).


SIEs YouTube Demo

Video is available at http://www.youtube.com/watch?v=hAAwKxeoCrk

SIEs

Spotlets forMedia content in the CoMET System

Implemented

by


SIEs Implementation

»Activate multimedia elements visible on the touchscreen

»Trigger requests to Web Services

»Semantic output representations» Tables with selectable cells

» Lists with selectable elements and Faceted Browsing

» Graphs with nodes, which can be selected

» Containers for elements of lists, tables, graphs, and several different media types

»Support “select”, “drag & drop”, or “drag-to-function” commands

»“Input” or semantic “drop-zone”


YouTube Spotlet

»A search can be made by dropping the object onto a spotlet in order to retrieve similar songs or artists.


Combined Interaction ScenarioAnd Speech-based Parallel Multimodality


Special Requirements

»This terminal is designed to be installed in a public place (or a café) and especially emphases the exchange of free music samples or video trailers which you can find on the Web. Drawbacks of mobile interaction:

» Often, direct software downloads are slow and cost-intensive.

» On a mobile devices, searching for information in large information spaces, such as online music and video repositories, is tedious.

» People cannot easily share and exchange their files.


Application to connect to the Touchscreen Terminal


Combined Scenario, Recognition of Devices and Media Exchange

Data Matrix Barcode software


Speech-based Input Integration Competence

Liquid List

“Search for more songs of this

artist.” (Nelly was put in focus, e.g.,

by using a pointing gesture).

Connect to YouTube, Seeqpod API[1], and the equivalent APIs for Lastfm[2]

and Flickr[3].

[1]http://www.seeqpod.com/api.php[2]http://www.lastfm.de/api/intro[3]http://www.flickr.com/services/api/


Speech-based Interaction Example

» U: Start client applications and register several iPods on the terminal. (WLAN IP connection)

» U: Place, e.g., 2 registered iPods on the touchscreen (ID3 tagged)

» S: Shows media circles for iPod contents

» U: Can select and play videos and search YouTube and Lastfm etc. by saying:

» “Search for more videos and images of this artist.”

» S: Delivers additional pictures, video clips and replies:

» “I found 40 videos and selected 5.

» Do you want me

» to play them?”

» U: “Yes, play the first two hits.”

» S: Initiates two video spotlets and plays the videos.

» U: Drag image of Nelly Furtado onto second iPod; Remove second iPod from the touchscreen.

» S: The media is transferred via IP connection.


Conclusions


Conclusions

»Multimodal speech-based interfaces offer great opportunities when designing mobile Human Computer Interfaces.

»We implemented a collaborative user scenario where one application is installed on the mobile interface and a second one is installed on a touchscreen exchange terminal.

» Following representation principle

»Refer to all presentation elements at input processing as a side-effect of profound data structure design

»Future Work: more complex dialogue scenarios, more spotlets(e.g., we are working with DBpedia contents), and the exploration of more fine-grained co-ordinated and synchronised multimodal presentations in mobile environments.


Thank you!

Questions?

CoMET System @ Cebit 2009

ICAI'09 -The2009 International Conferenceon ArtificialIntelligencesonntag/icai09.pdf ·...

Documents

Transcript of ICAI'09 -The2009 International Conferenceon ArtificialIntelligencesonntag/icai09.pdf ·...