The EyesWeb Open Platform

The DIST – InfoMus Lab Staff

www.eyesweb.org

InfoMus Lab – DIST – University of Genovawww.infomus.org

The EyesWeb Projectwww.eyesweb.org

• Research:– Computational models of expressive content, emotional, and

KANSEI communication in movement, music, visual media.– Integration of music, visual, movement (incl. dance)

languages from a multimodal perspective.

• Open h/w and s/w Architecture: – modular h/w and s/w real-time platform to support research,

enable experiments, support artistic productions and in general multimodal applications

• Applications: – music, theatre, interactive art, multimodal interfaces,

museums, edutainment, multimedia ...

EyesWeb...• Fast development process of experimental

Expressive Media applications:• Fully open and easy to extend• SDK for development of user’s new modules

• Industry standards compliance: MS COM/DCOM, Steinberg VST and ASIO (support to audio multi-channel h/w boards), OSC, MIDI, tcp/ip, serial, firewire, USB webcam, wide range of sensors and frame grabbers, ...

...EyesWeb• A number of specific libraries and extensions

available:– Motion analysis lib for real-time extraction of cues

from video streams • Low-level: e.g. trackers (e.g. Multiple Coloured Blobs

Tracking; LK; SMI/MEI; ...)

– Expressive gesture processing library• High-Level: expressive cues, General Space, Space Maps

and Shapes, trajectories analysis.

– Pro features to create commercial apps– Specific extensions available on demand– Third-party libraries and apps available

EyesWeb Users Community

• Growing users community (>10000)

• Software platform freely available:– http://www.infomus.org– http://www.eyesweb.org– Public newsgroups– Constant s/w and material udates from

web sites and EyesWeb public users newsgroup

Requirements for EyesWeb XMI• Defined starting from

– Requests and suggestions by users in the EyesWeb newsgroups

– Requests and suggestions by partners in EU and national projects

– Our own experience in EU and national research projects and in collaboration with industries

– Concrete issues emerged while designing and developing interactive multimedia installations and artistic performances

• Leaded to a very deep revision of the system with redesign and re-implementation from scratch of the most critical modules

• Deal with usability, performance, robustness, interoperability, optimization.

Support to Cross-Modality and Multimodality• The same module can operate on different

datatypes: e.g., the same “segmentation” Eyw module can be used for different streams (audio, movement cues, sensor data ...)

• Support to resampling, interpolation, adaptation to simultaneous processing of different streams

• Data conversion, synchronization, etc. • Extension toward object-oriented programming • These features are not emerging to the

programmer and the user unless requested, but are provided (and extensible) in the EyesWeb visual language

Usability

• Integration in the EyesWeb language of aspects related to multimodal processing of different data stream. – E.g., user do not have to explicitly care about

synchronization of streams, the language automatically support this.

• Subpatching mechanism fully supported – First model: different subpatches of the same type are

disjoint; modifying one subpatch does not alter the other instances

– Second model: different instances of subpatches of the same type do share the subpatch class: modifying a subpatch acts on all instances of such subpatch.

Usability

• Hiding inner implementation details – E.g., users do not have to explicitly care about threads,

clocks, active and passive blocks: these are automatically handled by the system

• Improved management of available blocks– Blocks grouped in catalogs– Mechanisms for finding a block easily, for filtering out

blocks belonging to a given catalog etc.• Redesign of the Graphical User Interface

– Improved editing of patches (e.g., in terms of links among blocks, pins, organization of blocks in a patch)

– Improved access to parameters

Performance and optimization• Optimization of the kernel for management of

audio and video streams• Support for multiprocessor systems

– The new version of the EyesWeb kernel is built in such a way that dual (or multi) processors computers can be exploited at best.

• Kernel objects– Have a privileged access to the kernel– Kernel can optimize access and use of kernel objects– Basic datatypes (e.g., images, sound buffers) are built-

in in the kernel as kernel objects– Kernel objects also include control flow blocks (switch,

for or while loops, conditional operations, etc.), basic clocks (multimedia timer) or devices (e.g., keyboard, serial, or mouse)

Robustness

• Complete separation of the kernel from the user interface– The kernel does not suffer of possible bugs of the user

interface– Different user interfaces can be developed and

employed, depending on the specific application• Standalone applications which do not need user

interface and have to operate for days in unsupervised environments can run with a minimal (or without) GUI

• Interaction between the interface and the kernel has been greatly simplified

Interoperability• Simplified connection of the EyesWeb platform

with other software platforms• Simplified hosting of standard plugins

– EyesWeb already hosts standard plugins (e.g., Steinberg VST, FreeFrame)

– The simplification of the communication paradigm between the EyesWeb kernel and the EyesWeb interface allows fast development of hosts for further plugin systems.

• Interaction with hardware managed by a new level of abstraction: devices– Available physical hardware resources can be mapped

to virtual devices used by blocks. – Patches are more portable among different computers,

even if the hardware configurations are different

Other features• Multiple versions allowed on a single PC

– Avoid using system registry/avoid centralized configuration

– Use configuration files instead of registry – A configuration file only affects one folder. Different

configurations can coexist on the same computer

• Multiple instances allowed at the same time• Portable patches

– Patches are saved as xml files– Support for xml is embedded in the kernel– Patches can be edited with standard text/xml editors– Portable among versions and Oss– Can be managed via source control systems

Other features• Better audio support

– Clock may be provided by devices– Clock of renderer modules has priority on other clocks– Try using a single clock for the whole patch (avoid

clicks/jitter)– Warns if different clocks must be used

• Parameters no more constrained to a limited set of predefined types

• New SDK no more COM dependent– C++ pure virtual interfaces– Easier access to inputs/outputs (no need to lock/unlock)– No multithreading issues

Future enhancements

• Multiplatform (Linux)

• Embedded systems

• Distributed systems

Expressive Gesture in Human Movement

• Human full-body movement as a main non-verbal communication channel

• Dance, the artistic expression of movement, as a testbed for studying expressive gesture mechanisms

• Development of – a multimodal conceptual framework– a collection of algorithms for gesture processing– experiments for testing and validation

Motion detection and tracking

E.g., energy, contraction, stability, kinematics

Motion segmentation and representation

High-level expressive information

Physical Signals

Cues

Gestures

Concepts

Conceptual Framework

• The Lucas – Kanade (LK) algorithm is used to track features in the input image.

• Trajectories are then analyzed (e.g., to extract information about motion directness, kinematics).

Motion tracking: LK tracking

Motion tracking by colors

Tracking of subregions

Tracking of multiple dancers

E.g., analysis of peaks in different bands can help in distinguishing movements of different body parts

Frequency analysis

Contour and convex hull

Global measures depending on full body movement (e.g., body orientation, overall motion direction)

Measures from psychological research (e.g., Boone & Cunningham’s amount of upward movement)

Cues from Rudolf Laban’s Theory of Effort (e.g., directness, impulsiveness)

Cues derived from analogies with audio analysis (e.g., Inter Onset Intervals, frequency analysis)

Kinematical measures such as velocity, acceleration, average and peak velocity and acceleration.

Expressive Cues

• SMIs (Silhouette Motion Images) carry information on variations of the silhouette in the last few frames. • SMI’s area (Quantity of Motion) is a measure of the detected amount of motion.

SMI and Quantity of Motion

• Ellipse approximating the body silhouette (analogy between image and mechanical moments). • Eccentricity related to contraction/expansion; orientation of the axes related to body orientation.

Silhouette Shape

A measure of how the dancer’s body uses the space surrounding it.

Contraction Index

Identified by Boone and Cunningham (1998) as a primary indicator of positive expressive intentions (e.g., joy).

Upward Movement

Asymmetry of a body posture computed with respect to the axes of its bounding box.

Asymmetry Index

• Motion Phases: the dancer is moving • Pause Phases: the dancer does not appear to move • Values of cues can be associated to each phase.

Gesture Segmentation

Duration, peak velocity, pauses expansion( 26 , 8 , 0.0324653 , 0.0242855 , 2 , 0.0391145 , -0.407343 , 0.342657 ).

pause( 34 , 3 ).contraction( 37 , 8 , 0.0462466 , 0.0494994 , 6 , 0.111976 , 0.297261 , 0.490809 ).

RHYTHMMovements emphasized through pauses

Languages for Choreography

Gesture Representation

• Gestures as trajectories in semantic spaces

• Clustering of such trajectories for classification

Quantity of Motion

Fluency

Gesture Representation

• A measure of how a trajectory is direct or indirect. • The trajectory can be the output of a (physical) motion tracker or a trajectory in a semantic space.

Directness Index

MotionAnalysis: motion trackers (e.g., LK), movement expressive cues (QoM, CI, ...).

SpaceProcessing: processing of (physical or abstract) spaces. Use of metaphors from physics: hysteresis, abstract potentials, superposition principle ...

TrajectoryProcessing: processing of 2D (physical or abstract) trajectories (e.g., kinematics, directness, …).

Machine learning techniques (e.g., SVMs, clustering, neural networks, …)

EyesWeb Expressive Gesture Processing Library

Experiments

• For understanding and analyzing the expressive content expressive gesture convey

• For evaluating and validating the developed models and algorithms

• For identifying cross-modal interactions among expressive gestures (e.g., in movement and in music)

For more info…

• www.eyesweb.org• www.infomus.org• news://infomus.dist.unige.it• info@infomus.org