Bruxelles, October 3-4, 2000 10036 Interface Multimodal Analysis/Synthesis System for Human...
-
Upload
trent-perez -
Category
Documents
-
view
221 -
download
0
Transcript of Bruxelles, October 3-4, 2000 10036 Interface Multimodal Analysis/Synthesis System for Human...
Bruxelles, October 3-4, 2000
10036 Interface
Multimodal Analysis/Synthesis System for Human Interaction to Virtual and
Augmented Environments
IST Concertation Meeting
Bruxelles - October 3-4, 2000
Bruxelles, October 3-4, 2000
The Consortium
DIST – University of Genoa I C
Lernout & Hauspie B PLinköpings Universitet S PUniversitat Politecnica de Catalunya E PEcole Polytechnique Fédérale de Lausanne CH PUniversity of Geneva CH P Informatics and Telematics Institute EL PTecnologia Automazione Uomo scrl. I PELAN Informatique F PUniversity of Maribor SI PCurtin University of Technology AU PUmea University S PCentre National de la Recherche Scientifique F PW Interactive SA F P
Bruxelles, October 3-4, 2000
A “man in the computer”
Natural SpeechNatural Expressions
Emotional Speech Synthesis
Emotional Face&Body Animation
?
Bruxelles, October 3-4, 2000
Man-to-machine action
• coherent analysis of audio-video channels
• high level interpretation and data fusion
• speech emotion understanding
• facial expression classification
Bruxelles, October 3-4, 2000
Machine-to-man feedback
• human-like audio-video feedback simulating a "person in the machine”
• MPEG-4 face and body animation
• text-to-speech with lip synchronization
• speech and facial expressions emotional synthesis
Bruxelles, October 3-4, 2000
WORK DONE SO FAR
• Specifications of the common sw platform
• Bimodal multi-language, multi-speaker corpus recording
• Development of preliminary tools
• Implementation of the intermediate common sw platform
• Finalization of the CfP for ICAV3D
• Participation to IBC2000
• Preparation toward IST2000
• Preliminary market study
Bruxelles, October 3-4, 2000
The INTERFACE platform
• Network Platform (large set of distributed independent tools)
• Integrated Platform (a reduced set of strongly integrated tools)
• Demonstration Platform (personalization of the Integrated Platform on specific application dependent context)
Bruxelles, October 3-4, 2000
Tools under development
• Low/high level video analysis
• Low/high level speech analysis
• High level facial animation
• Natural dialog manager
• 3D human and cartoon characters
• Phoneme markers to FAP conversion
• Emotional speech synthesis
• High level body animation
Bruxelles, October 3-4, 2000
The Network CSP
http://www.ist-interface.org
C
IDL
C++
IDL
Java
IDL
Client
C
IDL
C++
IDL
Java
IDL
Server
ORB - Object Request Broker
Bruxelles, October 3-4, 2000
Mapping for the 14 MPEG-4 visemes
p, b, m f, v T, D t, d
k, g tS, dZ, S s, z n, l
r A e I
O U
Bruxelles, October 3-4, 2000
Facial emotion modeling and synthesis
Happiness Sadness Anger Fear Disgust Surprise
Bruxelles, October 3-4, 2000
Bimodal emotional analysis : training
Classifier (GMM)
Video
Features
Audio Analysis
Audio Frames
Video Analysis
Video Frames Features
Selection
Frames #
Bruxelles, October 3-4, 2000
Bimodal emotion synthesis: runtime
Audio Analysis
Classifier (GMM)
Audio
Features
Audio Frames
Predefined Movements or
Expressions
“Expression Blender”
Bruxelles, October 3-4, 2000
Feature tracking tool: Track2FAP
Bruxelles, October 3-4, 2000
Applications
• Virtual speaker for web news announcing
• Virtual operator for web call centers
• Virtual teacher for remote teaching
• Virtual salesman in e-commerce
• Virtual friend for web chatting
• Virtual guide for Internet navigation