New Interface Design for Human Experience and

Expression

Sidney FelsHuman Communication Technologies Laboratory

Dept. of Electrical and Computer Engineering&

Media and Graphics Interdisciplinary Centre

University of British ColumbiaVancouver, BC, Canada

ssfels@ece.ubc.ca

Human Communication Technologies Lab

• HCT lab started in 1998– Focus on human experience

• Intimacy, Embodiment• Aesthetics of relationships• Communication

– Interdisciplinary by practice and training– 8000ft2 of facilities, housed within ICICS complex

• MAGIC started in 1991– Focus on Media technologies and graphics

• Research• Industrial• Cultural• Education

– 2000ft2 of facilities housed in the Forest Sciences building

hct.ece.ubc.ca

Research Directions

• Three main areas of research– New interface design– Art, Music and Expression– Modeling, Tools and Graphics

Example Projects at HCT• New Interfaces

– Interaction with Large Display Surfaces• T. Tang

– Cubee: a 3D display• Ian Stavness & Florian Vogt

– Affective communication interface• 2 Hearts musical system (G. McCaig)• Interactive Yoga System (xxxx)

Example Projects at HCT• Art, Music and Technology

– Sound• Tooka• Tongue’n Groove• D’Groove

– Interactive Art• Iamascope• Swimming Across the Pacific• Plesiophone• 1,001,001 Faces• Sound Room & Sound Weave• 1 Kingsway: Flow

– Performance• Waking Dream• Forklift Ballet• Laika Space Program

Example Projects at HCT

• Modeling, Tools, Graphics– Biomechanical Modeling for

Articulatory Speech Synthesis

• large team of people

– Parallel Distributed Camera Array & Local Positioning System

• large team of people

New Interfaces, Art & Technology

Iamascope• Interactive multimedia

artwork• Participants are put inside

a large kaleidoscope• Participants movements

also map to music

Iamascope System

KaleidoscopicImage

ImageProcessing

Vision-to-music subsytem

ActiveVideo

RegionVideo

Camera

speakers

VideoImage

Kaleidscope subsytem

TextureMapping

TextureMemory Music

Production

MusicSynthesizer

170” Video Projector

Iamascope Video

Overview

Swimming Across the Pacific

• Swimming Across the Atlantic– Queen ElizabethII– Misheff, 1982– Southampton to New York

• Contemporary vision– Swimming in an airplane– Los Angeles to Tokyo

• Created swimming apparatus– Exhibition– Audience participation– Team of 8 people

QuickTime™ and aTIFF (LZW) decompressor

are needed to see this picture.

QuickTime™ and aSorenson Video 3 decompressorare needed to see this picture.

Tooka: A Two Person Flute

• Create musical instrument that needs two people to play

• Explore human intimacy

How Tooka Works

VolumePitch

Capture/SustainOctave

Pitch Bend

Vibrato

Cubee: 3D display

Hardware

Cubee Scenes

• Static object glued to box– e.g. teapot display case

• Static scene fixed in world– e.g. volumetric cut-plane

• Dynamic objects inside box– e.g. ball rolling through 3D maze

Glove-TalkII• Translates hand gestures to speech

– like a musical instrument

• mapping partially learned• speech is

– intelligible– expressive– slower than normal

Spectrum of Gesture-to-Speech Mappings

ArtificialVocalTract

PhonemeGenerator

FingerSpelling

SyllableGenerator

WordGenerator

Von

Kem

pel

en (

1790

)

Bel

l & B

ell (

1880

)

Du

dle

y et

al.

(193

9)

Fel

s &

Hin

ton

(19

94)

Kra

mer

& L

eife

r (1

989)

Fel

s &

Hin

ton

(19

90)

10-30 100 130 200 500

approximate time/gesture for connected speech

(msec)

Glove-TalkII Vocabulary• Loosely based on articulatory model of speech• Vowels

– open configuration of hand represents open vocal tract– X,Y position determines vowel sounds (like tongue)

• Consonants– constrictions in hand represent constriction in vocal tract

• Stop consonants produced with ContactGlove• Volume controlled with foot pedal (air pressure)• Pitch controlled by Z position of hand (vocal cord

tension)

SpeechOutput

Glove-TalkII System

Foot Pedal

x,y,z, roll, pitch, yaw(60 Hz)

10 flex angles4 abduction angles

thumb and pinkie rotationwrist pitch and yaw

(100Hz)

Right Hand Data

ContactSwitches

Preprocessor

Fixed PitchMapping

V/C DecisionNetwork

VowelNetwork

ConsonantNetwork

Fixed StopMapping

Synthesizer

CombiningFunction X

• 3 neural nets• Output: Parallel Formant Speech Synthesizer

– ALF, F1, A1, F2, A2, F3, A3, AHF, V, F0– 100 Hz, 6 bit quantization [0, 63]

Glove-TalkII VIDEO

Alphabet

Numbers

Sam I Am

Vocal Sounds

Intimacy and Embodiment

• Want interfaces that feel “good” to use

• Humans and machines intimately linked– degree of intimacy supported may

determine success• Types of relationships:

– human to human– human to machine

Intimacy• Intimacy is a measure of match between the

behaviour of an object and the control of that object.– extension of “control intimacy” from electronic

musical instruments analysis (Moore, 1997)

• High intimacy implies:– object feels like an extension of self– satisfaction derives from interacting with object– emotional expression flows

• requires cognitive effort to prevent

• Relationship dynamics– Aesthetics of interaction

Intimacy: Embodiment vs. Disembodiment

Case 1: Object disembodied from Self

Case 2: Self embodies Object

Case 3: Self disembodied from Object

Case 4: Object embodies Self

self object

objectself

objectself

object self

response

control

reflection

belonging

Aestheticsof Interaction

Directions

• How to achieve intimate experiences?– Two user-centric techniques:

• mirrors– Eliza (Weizenbaum, 1966), Iamascope (Fels &

Mase, 1999), Wooden Mirror (Rozin, 1999), bots

• masks

– many other complex techniques

Summary: HCT Activities• Modeling, Graphics and Tools

– ArtiSynth for articulatory speech synthesis– OpenVl for providing image processing infrastructure

• New Interface Design– approach to design using intimacy

• Art, Music and Technology– explore expressive directions of technology

http://hct.ece.ubc.ca/

Sidney Fels

ssfels@ece.ubc.ca

Contact Information:

Questions: