IST-1999-20410 MEGA Multisensory Expressive Gesture Applications Concertation Meeting Brussels, 3-4...
-
Upload
garey-edgar-glenn -
Category
Documents
-
view
215 -
download
1
Transcript of IST-1999-20410 MEGA Multisensory Expressive Gesture Applications Concertation Meeting Brussels, 3-4...
IST-1999-20410
MEGA Multisensory Expressive Gesture Applications
Concertation MeetingBrussels, 3-4 October 2000
Antonio Camurri (DIST-University of Genoa)MEGA Project Coordinator
Summary
• MEGA: Concepts, Objectives and Mission
• The Consortium
• Project Overview: Workplan, WPs, and Deliverables
MEGA: Concepts, Objectives and Mission
• Concepts:
– Multisensory
– Expressive (emotion, kansei, affect)
– Gesture
– Applications
MEGA: Concepts, Objectives and Mission
• Objectives:– modelling and communication of expressive and
emotional content in non-verbal interaction by multi-sensory interfaces
– design of novel paradigms of interaction – analysis, synthesis and communication of
expressive content in networked environments
– implementation of the MEGA System Environment– development of artistic public events showing the
usefulness, flexibility and power of the MEGA System Environment
MEGA: Concepts, Objectives and Mission
• Mission:– Consolidating a third original European route in
affective computing (USA) and KANSEI (Japan).– Raise the competitiveness of European Industry on
• Digital Music Instruments • Entertainment and Edutainment• Multimedia: e.g. VEs, e-commerce, intelligent
interfaces, ...
by building the MEGA System Environment and showing its effectiveness in public events.
Participant no.
Participant name Participant short name
Country Status*
1 DIST - University of Genoa Lab. on Musical Informatics
DIST I C
2 DEI - University of Padova DEI I P
3 IPEM – University of Gent IPEM B P
4 KTH - Royal Institute of Technology, Stockholm
KTH S P
5 Uppsala University, Department of Psychology
UUP S P
6 Telenor TLN NO P
7 Generalmusic GEM I A
8 Consorzio Pisa Ricerche CPR I P
9 Eidomedia EM I A
Participants
Workplan: Workpackages
• WP1 Project management• WP2 Requirements • WP3 Analysis in human movement and gesture • WP4 Analysis in musical gesture performance • WP5 Synthesis in visual media and movement • WP6 Synthesis in music performance • WP7 MEGA System Environment • WP8 MEGA Multimedia Performance • WP9 Dissemination and implementation
Workplan: gantt
Year 1 Year 2 Year 3WP Title1 2 3 4 5 6 7 8 9 10 11 12 1314 15 1617 18 19 2021 21 2324 2526 27 2829 30 31 3233 34 3536
WP1 Project management * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *WP2 Requirements * * * * * * * * * *WP3 Analysis in movement * * * * * * * * * * * * * * * * * * * * *WP4 Analysis in music * * * * * * * * * * * * * * * * * * * * *WP5 Synth. in movem. and vis. * * * * * * * * * * * * * * * *WP6 Synth. in music * * * * * * * * * * * * * * * * * * *WP7 MEGA System Environ. * * * * * * * * * * * * * * * * * * * * * * * *WP8 MEGA Multim. Perform. * * * * * * * * * * * * * * * * *WP9 Dissemin. and Implement. * * * * * * * * * * * * * * *
MILESTONES * * *
- Spiral approach- Three phases, each concluded at a project milestone (13, 25, 36 months).
WP2 Requirements • WP Leader: IPEM
• State of the art, share of results among partners, Requirements and setup for the research, the MEGA System Environment and the MEGA Multimedia Performance products.
• Definition of a standard communication protocol for expressive data. The protocol is defined in this WP and is implemented in the WP7 (MEGA System Environment).
• Contributes from all the partners.
• D3 (month 6) Requirements
• D9 (month 14) First requirement revision
• D12 (month 26) Final requirements
WP3-6: Research on analysis and synthesis of expressivity
• WP3 Analysis in human movement and gesture (DIST) • WP4 Analysis in musical gesture performance (DEI)• WP5 Synthesis in visual media and movement (EM)• WP6 Synthesis in music performance (KTH)• Results: a selection of techniques and algorithms, integrated
as s/w modules in the MEGA System Environment (WP7). These developed tools and modules will be continuously integrated and updated in the EyesWeb architecture, i.e., WP3-6 will deliver tested library modules and associated manual pages using the standard adopted in the project
• Deliverables: Reports (D5, D13), S/w tools (D6, D14)
WP3 Analysis in human movement and gesture
• WP Leader: DIST
• To extract expressive content from movement patterns.
• To evaluate and compare the results and performance of the obtained methods and algorithms (DIST and KTH) with the results obtained by psychological analysis of movement (UUP) with selected groups of users, both non-expert and expert (dancers, choreographers).
• To develop a project reference archive of microdances and movement patterns (DIST, KTH, UUP), for both psychological analysis and automated analysis, using the standard adopted in the project (audiovisual, mocap data).
WP4 Analysis in musical gesture performance
• WP Leader: DEI Analysis of the different cues characterising an expressive
musical performance (UUP, DEI); Analysis of the correlation between cues and the instrument
played by the performer (DEI); Development of a model to extract automatically an
expressive content of a musical performance (IPEM); Development of tools implementing the model (DEI,
IPEM); Evaluation and test of the effectiveness of the model by
means of performances analysis with the implemented tools (KTH, UUP);
WP5 Synthesis in visual media and movement
• WP Leader: EM
• Techniques inspired to the audiovisual language in cinema (EM) and their utilisation in virtual environments (TNR) and in visual and robot-human interaction (DIST).
• Development of models and methods to synthesise expressive visual media outputs (EM).
• Expressive synthesis of movements of synthetic characters and expressive modulation of VEs and Mixed Reality envs (TNR), depending on the expressive communication goals and content.
• Expressive movement of small robots in concrete apps on robot-human interaction and mobile scenery (DIST).
• Integration with Dovre VE architecture (TNR).
WP6 Synthesis in music performance
• WP Leader: KTH
• Development of tools both in MIDI and in digital audio format (KTH, DEI), to be used for the project products (WP7-8).
• Synthesis on MIDI format based on Director Musices (KTH)
• Adapting and integrating audio electronics for real-time expressive processing (GEM)
WP7 MEGA System Environment
• WP Leader: DIST
• Deliverable D4: EyesWeb software release, including tutorial and reference manuals (month 6)
• Deliverable D7: 1st version of MEGA Sys Env. (12)
• Deliverable D10: 2nd version of the MEGA Sys Env. (24)
• Deliverable D15: Final product - MEGA Sys Env (35)
• DIST: EyesWeb and architecture CPR: Wireless Sensors
• TNR: Dovre IPEM: Mapping Strategies, Eval
• EM: Wireless Sensors
MEGA System Environment (WP7)• Main components:
- the EyesWeb software architecture (from DIST),- the (open) libraries of software modules concerning
analysis and synthesis of expressive gesture (WP3-6), - the implementation of the protocol for expressive
communication (WP2),- the h/w electronics and sensor systems for wireless non-
invasive real-time motion capture (adapted from existing systems by CPR, DIST, EM), integration of commercial sensor systems (e.g., radio baton) and audio electronics systems (by GEM).
WP8 MEGA Multimedia Performance
• WP Leader: IPEM• Main components:- the specification of the artistic use of the MEGA System
Environment (WP7) in terms of the expressive content analysis and synthesis (IPEM, DEI);
- the support to specific multimedia artistic events, based on the MEGA System Environment (DIST, KTH);
- the evaluation of the MEGA System Environment with respect to audience acceptance, effectiveness, usability, and flexibility (IPEM).
WP9 Dissemination and implementation• WP Leader: TNR• To set up the Liason Board• To disseminate products (WP7-8) and research results
(Workshops, book, Frankfurt and other exhibits, etc). • Internet-based dissemination and exploitation strategy by a
free distribution to users of the software (under certain constraints and on-line contract forms),
• To enable external users to develop new “plugins” of the MEGA System Environment.
• Evaluate (UUP) the results from the project from WP7-8 and the internet dissemination strategy, with respect to the planned outputs.
• D2: Dissemination and Use plan (6)
• D17: Technological Implementation Plan (36)
Dissemination (WP8 and WP9)• via public performances (WP8)
• An internet site providing the software and support to users (WP9):– s/w: the full MEGA System Environment, incl. a
Wizard enabling users to develop new system plugins– library of plugins deriving from WP3-6 (research)– full documentation in the standard adopted
• Workshops, special tracks and sessions at IEEE CS, ACM Conf, at CEC Workshops, at Frankfurt Messe 2001 and/or 2002 (WP9)
Deliverables list
Del.No.
Del. Name WP no. Leadparticipant
Estimateperson-months
Del. Type* Security** Delivery
(proj.Month)
D1 Project Presentation WP9 DEI 1 Report Pub. 6
D2 Dissemination anduse plan
WP9 IPEM 8 Report Int. 6
D3 Requirements WP2 KTH 7 Report Int. 6
D4 EyesWeb s/w releaseincl. tutorial andreference manuals
WP7 DIST 4 Prototype Int. 6
D5 Analysis andsynthesis ofexpressiveness:techniques andmapping strategies
WP3WP4WP5WP6
UUP 15 Report Int. 12
D6 Tools for automaticanalysis andsynthesis ofexpressive content inmusic and danceperformance
WP3WP4WP5WP6
KTH 25 Demonstration Rest.(***)
12
D7 First version of theMEGA Sys. Env.
WP7 DIST 10 Prototype Pub. 12
D8 Report and s/wpatches from the firstmultimedia event
WP8 UUP 9.5 Report Int. 12
D9 First revision ofrequirements
WP2 DIST 5 Report Int. 14
D10 2nd version of theMEGA Sys. Env.
WP7 DIST 11 Prototype Pub. 24
D11 Report and s/wpatches from thesecond multimediaevent
WP8 IPEM 12 Report Int. 25
D12 Final requirements WP2 IPEM 5 Report Int. 26
D13 Final report onanalysis andsynthesis ofexpressiveness:models, techniques,and integrationissues
WP3WP4WP5WP6
KTH 34 Report Int. 31
D14 Tools for automaticanalysis of theexpressive content inmusic and danceperformance
WP3WP4WP5WP6
DEI 45.5 Prototype Rest.(***)
31
D15 Final product of theMEGA Sys. Env.
WP7 DIST 9 Prototype Pub. 35
D16 Report and s/wpatches from thethird multimediaevent
WP8 IPEM 16 Report Int. 36
D17 TechnologicalImplementation plan
WP9 TNR 10 Report Int. 36