So nification of the facial expressio ns New Interaction Techniques Department of Computer Science...
-
date post
21-Dec-2015 -
Category
Documents
-
view
216 -
download
0
Transcript of So nification of the facial expressio ns New Interaction Techniques Department of Computer Science...
So nification of the facial expressio ns
New Interaction Techniques
Department of Computer Science University of Tampere
27 February 2003
Guo Zhiguo Ioulia Guizatdinova [email protected] [email protected]
Contents
• Introduction to the problem
• Background & related research
• Objectives
• Methods
• Resources & experimental setup
• Preliminary results
• Prospectives
p 02_23 27.02.03 Sonification of the facial expressions
Sonification is the use of non-speech audio to convey information
more specifically sonification is the transformation of data
relations into perceived relations in an acoustic signal for the
purposes of facilitating communication or interpretation [1]
p 03_23 27.02.03 Sonification of the facial expressions
Introduction to the problem
The main question for sonification of the facial expression is:
How to convey the emotional expressions/facial features/ of some
person presented like any visual imaging into the sounds
(e.g. earcons, music chords) or whatever ?
Introduction to the problem
p 04_23 27.02.03 Sonification of the facial expressions
The sonification system for facial expressions should allow the
transformation of the visual information (attributes specifically
describing emotional expression/state) into sound patterns
This kind of sounds (“eARmoticons”) should evoke similar
emotional feelings of the listener
that is, to provide an emotional access through “audio snapshot”
when visual perception is blocked or imaging is hidden for
observer
p 05_23 27.02.03 Sonification of the facial expressions
Introduction to the problem
Some facial expressions can hardly be
transformed by sound (e.g., congenital or
non-classified features)
Blind people have a wide experience with
real environmental sounds, while the
congenital blinded can have not any
association to visual facial imaging; only
tactile memory of some facial features can
be insufficient or non-adequate
p 06_23 27.02.03 Sonification of the facial expressions
Background & Related Research
There is not signs’ system/auditory or tactile/specifically & briefly describing facial features/expressions. Only some general emotional states are presented like artificial set of static visual/tactile symbols [2, 11].
p 07_23 27.02.03 Sonification of the facial expressions
Background & Related Research
Happiness
: )
Sadness
: (
Frustration
: / >:-(
Madness
There have been a lot of approaches to data sonification of like math
functions, simple graphics, diagrams, …
p 08_23 27.02.03 Sonification of the facial expressions
http://www.visualprosthesis.com/javoice.htm
The vOICe Java Applet
Background & Related Research
p 09_23 27.02.03 Sonification of the facial expressions
Relevantly, there are a lot of earcons used in MS Windows
and other software, sound signaling for special equipment
and application…
sound soundsoundsound
Background & Related Research
Objectives
The goal of this project is
designing and usability investigation of the sonification
system for attributes specifically describing facial expression
to provide an emotional access through eARmoticons when
visual perception is blocked or imaging is hidden for
observer
p 10_23 27.02.03 Sonification of the facial expressions
Schedule
Project’s activity Weeks Expected results
Overview on the topicDefinition of the interaction scenarios & research methods
1Sonification & visual
attributesScenarios
Design of software prototypeDesign & collection of sound files
1Software prototype
Sound database
Classification of the facial expressions by soundRevision of software
1Sonification System for
facial attributesRevised software
Usability exploration of the new interaction techniques
2 Data
Data analysis Report writing
12
Project paper
p 11_23 27.02.03 Sonification of the facial expressions
Research Methods
• constructive research to design and simulate the new techniques, software tools and doing usability evaluation of novel sound objects – eARmoticons
• empirical research is based on crossmodal information transfer and perception of intermodal events
p 12_23 27.02.03 Sonification of the facial expressions
Preliminary Modelling
p 13_23 27.02.03 Sonification of the facial expressions
Facial expression
(image, picture, video, etc)
sounds
Video processing
Facial expression-to-sound mapping transformation system
Recognition of the sound patterns to facial expression
As a whole
Facial landmarks
Sound Database
Resources & Experimental Setup
p 14_23 27.02.03 Sonification of the facial expressions
• Subjects (people with normal vision)
• Stimuli – eARmoticons – earcons specifically & briefly describing
facial features/expressions
FACS – Facial Action Coding System [ 7, 8]
a prototype of the sonification system for facial expression transformation
Resources & Experimental Setup
p 15_23 27.02.03 Sonification of the facial expressions
Experimental procedure
Resources & Experimental Setup
p 16_23 27.02.03 Sonification of the facial expressions
Data analysis:reaction timeaccuracy = error
number
Preliminary results
p 17_23 27.02.03 Sonification of the facial expressions
Classification of the emotions by Ekman & Friesen (1975)
http://www.paulekman.com/
surprise happiness angerdisgustsadness fear
NEUTRALPOSITIVE NEGATIVE
p 18_23 27.02.03 Sonification of the facial expressions
Inner brow raiser
Outer brow raiser
Brow lowerer
Upper lid raiser
Cheek raiser
Lid tightener
Lid droop
Slit
Eyes closed
Squint
Facial landmarks.Eye & eyebrow
Preliminary results
p 19_23 27.02.03 Sonification of the facial expressions
Lip pressor
Lip corner puller
Lips part
Jaw drop
Mouth stretch
Lip funneler
Lip suck
Cheek puffer
Dimpler
Lip corner depressor
Lower lip depressor
Lip puckerer
Lip stretcher
Lip tightener
Chin raiser
Facial landmark. Mouth
Preliminary results
p 20_23 27.02.03 Sonification of the facial expressions
Nose wrinkler
Upperupper lip raiser
Nasolabial deepener
Facial landmark. Nose
Preliminary results
The sonification system of facial expressions which
includes new sound mapping techniques to display
facial features and expressions
Prospectives
p 21_23 27.02.03 Sonification of the facial expressions
1. http://www.icad.org/websiteV2.0/Resources/glossary.html#S2. http://www.tsbvi.edu/Education/vmi/tactile_symbols.htm#EMOTIONS3. http://www.mcsdtechcenter.org/multimedia/sounds-samp.htm4. http://www.matthawkins.co.uk/html/sound_fx_index.htm5. http://kin-dza-dza.etel.ru/ http://www.soundobject.org/6. http://viswiz.gmd.de/~eckel/publications/eckel98a/eckel98a.html7. http://www-2.cs.cmu.edu/afs/cs/project/face/www/facs.htm8. http://www.paulekman.com/9. http://www.ifeelpixel.com/download/whatsnew.htm 10. http://www.symbols.net/emoticons.htm11. http://www.fingertek.com/smilies.htm12. http://www-unix.mcs.anl.gov/~kaper/Sonification/13. http://www-pablo.cs.uiuc.edu/Project/Pablo/PabloSonification.htm
references
p 22_23 27.02.03 Sonification of the facial expressions