Workshop on Multimodal Interaction on Mobile Devices

Multimodal and multichannel interactions for pervasive issues

José Rouillard – LIFL

José RouillardLIFL / USTL Lille

© ETSI 2007. All rights reserved

Workshop on Multimodal Interaction on Mobile Devices - 18th and 19th November 2008

Workshop on Multimodal Interaction on Mobile Devices

Overview� Introduction

� Multi-Device� Multi-Modal� Multi-Channel� Multi-DMC referential

� Context-awareness� Adaptation

José Rouillard 2

� Plasticity� Pervasive computing

� About QR Codes� Using QR Codes in context� PerZoovasive system

� Application logic� Using a workflow� Dynamic access to rules

� Conclusion

Workshop on Multimodal Interaction on Mobile Devices

Introduction� Multi-Device

A multi-Device system allows to use multiple devices (PC, PDA, Smartphone, etc.)in order to do the same task [Calvary & al. 2003].

Example : migration from smartphone to PC [Demumieux & al. 2008].

José Rouillard 3

Workshop on Multimodal Interaction on Mobile Devices

Introduction� Multi-Modal

A multi-Modal system allows to use multiple modalities (voice, gesture, brain, etc.).

José Rouillard 4

Workshop on Multimodal Interaction on Mobile Devices

Introduction� Multi-Channel

A multi-Channel system allows to use multiple way of communicate with anorganization (fax, web, SMS, etc.).

José Rouillard 5

Workshop on Multimodal Interaction on Mobile Devices

Introduction� Multi-DMC referential

José Rouillard 6

Workshop on Multimodal Interaction on Mobile Devices

Context-awareness

“A context-aware system is a computing system using context to provide relevant information and/or services to the user, where relevancy depends on the user’s tasks.” [Anind Dey, Ph.D thesis, 2000 ].

Indeed, one of the main goals of ubiquitous computing is to provide

José Rouillard 7

Indeed, one of the main goals of ubiquitous computing is to provide relevant information, at the right time and place and under the right form.

Context-aware systems should help filter information.

This allows users to concentrate on the task rather than the technology.

Workshop on Multimodal Interaction on Mobile Devices

Context-awareness� Adaptation

José Rouillard 8

Workshop on Multimodal Interaction on Mobile Devices

� Plasticity

Plasticity is the capacity of a user interface to withstand variations of both the system physical characteristics and the environment while preserving usability [Thevenin & Coutaz 1999]

Context-awareness

José Rouillard 9

Workshop on Multimodal Interaction on Mobile Devices

Pervasive ComputingMark Weiser’s vision of Pervasive and Ubiquitous co mputing (Weiser 1991) underlines the need of seamle ssly unifying computers and humans around the notion of rich environment.

“Everyware” (Greefield 2006): neologism to encompas s the terms of ubiquitous computing, pervasive comp uting, ambient computing and tangible media.

In the domain of object identification, different k inds of codes and technologies are used in order t o store, retrieve and manage information.

José Rouillard 10

RFID (Radio-Frequency IDentification)

NFC(Near Field Communication)

BARCODE (1D or 2D)

Barcodes are still interesting: basic technology and simplicity of the concept.

Workshop on Multimodal Interaction on Mobile Devices

� About QR Codes

Datamatrix (Semacode)

Rohs’ VisualCodes

Aztec Code

Codablock F

BeeTagg

No license fee to be paid to use neither DataMatrix nor QR Codes.

Pervasive Computing

José Rouillard 11

MaxiCode

High Capacity

Color Barcode

Paper Memory Code

Shot Code (SpotCode)

PDF417 (Portable Data File)

ColorCode

EZ CodeVericode

VS CodeHotScan

Codablock FQR Codes

Workshop on Multimodal Interaction on Mobile Devices

� About QR CodesQR Codes* (2D barcodes) introduced byDenso-Wave in 1994. Used to encode and decode data at a rapid rate.

It contains information in both the vertical and horizontal directions.Maximum QR Code Data capacity:

• Numeric only : 7,089 characters • Alphanumeric : 4,296 characters

Pervasive Computing

José Rouillard 12

• Alphanumeric : 4,296 characters • Binary (8 bits): 2,953 bytes • Kanji/Kana : 1,817 characters

Dirt and Damage Resistant. Error Correction capacity:• Level L: Approx. 7% of codewords can be restored. • Level M: 15% • Level Q: 25% • Level H: 30%

• Approved as AIM International (Automatic Identification Manufacturers International) standard [1997]• Approved as ISO international standard (ISO/IEC18004) [2000]

Using camera phones to read two dimensional barcodes for various purposes is currently a popular topic in both research and in practical applications.

Workshop on Multimodal Interaction on Mobile Devices

Impossible for human to manually decode QR Codes but they are easily processed by scanning equipment.

QR Codes are part of daily life in Asia: Japan, Korea, Taiwan, Hong Kong, China …and, many cellular phones are now natively equipped with a QR Code-reading software.

McDonalds uses codes to inform users about the nutritious value of its burgers.

Apple advertised the new i-Pod on billboards with QR codes.

QR Codes Usage

José Rouillard 13

Apple advertised the new i-Pod on billboards with QR codes.

QR Codes used in a Nike advertising campaign allows direct access to a dedicated mobile site.

In Japan, some teachers are using QR codes to distribute resources to learners.

QR codes now appear in magazines, advertisements, product wrappings, T-shirts, passports, business cards and on subway billboards in Japan.

Workshop on Multimodal Interaction on Mobile Devices

To speed up flight check in and boarding.

Bus information

Nutrition information

QR Codes Usage

José Rouillard 14

and boarding. Nutrition information

Cocacola.jp vending machineIn-store information

Workshop on Multimodal Interaction on Mobile Devices

Culture, entertainment, advertising …

QR Codes Usage

José Rouillard 15

Culture, entertainment, advertising …

Workshop on Multimodal Interaction on Mobile Devices

This act of linking from physical world objects is known as a hardlink or physical world hyperlinks.

Generating QR Codes: Users can scan existing QR Codes or they can generate and print their own. Creating QR Codes online is very easy: Kaywa, Snapmaze, Activeprint …

José Rouillard 16

Creating QR Codes online is very easy: Kaywa, Snapmaze, Activeprint …

Reading QR Codes:

QuickMark [http://www.quickmark.com.tw/] or I-nigma [http://www.i-nigma.com/] readers are good examples of free tools using this technique that are available for many manufactured models and devices.

Until now, information provided by QR Codes were static.

Workshop on Multimodal Interaction on Mobile Devices

Pervasive Computing� Using QR Codes in context

A contextual QR Code is the result of a fusion between a public part of information (QR Code) and a private part (the context) provided by the device that scanned the code [Rouillard 2008].

José Rouillard 17

Workshop on Multimodal Interaction on Mobile Devices

Pervasive Computing� PerZoovasive system

José Rouillard 18

Contextual usage of a QR Code at the Zoo (Turtle’s cage):The teacher could retrieve general information abou t this kind of animal (species, origin, speed, food…) but also personal data about t hat particular turtle (name, age, birthday, parents, etc.) [Rouillard & L aroussi 2008].

Workshop on Multimodal Interaction on Mobile Devices

Pervasive Computing� PerZoovasive system

Smartphone HTC Kaiser

3G connection

José Rouillard 19

C# .Net

Web services (SOAP)

Tasman barcode Library

See Video(Zoo of Lille)

Workshop on Multimodal Interaction on Mobile Devices

Pervasive Computing� PerZoovasive system

José Rouillard 20

Cage # 8245, Level: CP, Language: English, Task: Le sson

Workshop on Multimodal Interaction on Mobile Devices

Pervasive Computing� PerZoovasive system

José Rouillard 21

Cage # 123, Level: CM2, Language: French, Task: Les son

Workshop on Multimodal Interaction on Mobile Devices

� PerZoovasive system

Pervasive Computing

José Rouillard 22

Cage # 123, Level: CM1, Language: English, Task: Qu iz, TTS enabled

Workshop on Multimodal Interaction on Mobile Devices

� Using a workflow

Application Logic

José Rouillard 23

Workshop on Multimodal Interaction on Mobile Devices

� Dynamic access to rulesApplication Logic

José Rouillard 24

See Demo

Workshop on Multimodal Interaction on Mobile Devices

� Moving a remote webcam with a smartphoneApplication Logic

José Rouillard 25

See Video(Multimodal

server)

Workshop on Multimodal Interaction on Mobile Devices

Conclusion

� “Context is Key” [Coutaz & al. 2005]

� Semantic is key of “context”

� Designing Multimodal interfaces is not easy (see Op enInterface)

José Rouillard 26

� Designing Multi-Modal & Multi-Channel interfaces :

� Middleware� Mobile interfaces� Pervasive & Ubiquitous computing� Semantic (rules and logic application)

� We need to work together…