Advanced real-time sound and data processing with...

Advanced real-time

sound and data

processing with FTM&Co NOTAM Workshop 24.–28.1.2011

presented by Diemo Schwarz

IMTR Real-Time Music Interaction TeamIrcam–Centre Pompidou

http://ftm.ircam.frFTM 2.5.0 release beta 19 for Max 5 on Mac OS X

FTM & Co — IMTR Team — IRCAM - Centre Pompidou

PlanFTM

Introduction

Basics

Examples and exercises

Persistency: SDIF, SQLite

GaborPrinciple

Examples and exercises

MnMOverview, examples and exercises

Gesture Follower

CataRT


What is FTM&Co?a real time object system for Max/MSP

representation of musical structures

representation of sound data and structures

abstraction from the real-time system (Max, PureData, ...)

Timeline:

Reference article:Norbert Schnell, R. Borghesi, D. Schwarz, F. Bevilacqua, R. Müller « FTM — Complex data structures for Max », International Computer Music Conference (ICMC), Barcelona, 2005

— Max/ISPW — Max/FTS — jMax — FTM — t

Introduction

courtesy of Mikhail Malt


FTM MotivationsAudio analysis/synthesis (Gabor)

representation of grains, waves/periods and frames

representation and organisation of sound descriptors

Gesture analysis, classification and recognition (MnM)

(linear) algebra and matrix processing

representation and (statistical) modelling of gestures gestalts

Score representation and following (FTM, suivi)

representation of a score or time series of data

representation of objects in a score

! modularity & flexible structuring of memory


FTM ComponentsFTMlib shared library for Max

message system extension

introducing complex data types

+Max externals

+Editors and graphical externals


FTMlib FeaturesObjects of predefined classes

mat, dict, track, fmat, expr, bpf, ...

Expression evaluation

File import/export (soundfiles, MIDI, SDIF)

Object serialization and persistence


TerminologyReal World

Object-Oriented Programming

FTM Max

a thing with a name, state, and properties (an alert black-and-

white dog named Otto)

object (data) object(some specific Max objects = externals)

an action and reaction(“sit!”)

method and return value

message andreturn value

message(and output)

a process (the vet)

(function, process)module

(external)Max object (external)

a class of similar things (dogs)

class class –

a more general or more specific class (pug dogs, animals)

superclass, derived class

– (types, related classes)

–


FTM Conceptsseparation of data and operators

! data

objects of optimised classes with methods

file import & export (text, audio files, MIDI files, SDIF)

persistence (save within patcher and in separate files)

conversion to Max values and lists

sent within Max values and lists (as references)

simple garbage collector (by reference count)

! operatorsFTM extended message box

FTM externals


The FTM Package Familiy

Gesture FollowerGesture Follower CataRTCataRT

Gaboratomic sound

MnMmaths+stats

Suiviscore

following

FTM externals

FTMdata classes, visualisation, editing



SDIF, SQLite, libsndfile

application packages

basic library

http://ftm.ircam.fr

http://imtr.ircam.fr

applications


DistributionFTM 2.5 current stable versions for Max5

FTM 2.3 and 2.4 for Max 4.6

FTM is free (libre) open source software (LGPL)

Gabor&MnM are free (as in free beer)both are distributed together

Web site http://ftm.ircam.fr

User mailing list [email protected] on http://lists.ircam.fr

Developer list [email protected]


Credits and ReferencesFTM&Co is developed in the Real-Time Musical Interactions team at IRCAM by Norbert Schnell and Riccardo Borghesi, Diemo Schwarz, Frederic Bevilacqua, Remy Müller, Jean-Philippe Lambert

with contributions by Julien Bloit, Baptiste Caramiaux, Arshia Cont, Pierre Duquesne, Sebastien Gulluni, Baptiste de la Gorce, Bruno Zamborlin

Reference article:Norbert Schnell, R. Borghesi, D. Schwarz, F. Bevilacqua, R. Müller « FTM — Complex data structures for Max », International Computer Music Conference (ICMC), Barcelona, 2005


G a b o r :Dennis Gabor, Acoustical Quanta and the Theory of Hearing, Nature, 1947

Multi-Representation grains, waveforms, spectral frames, partial sets

Real-Time extension modules for Max/MSP, PD

Analysis–Synthesis...and transformation in a unified framework

Reference article:Norbert Schnell et al. « Gabor, Multi-Representation Real-Time Analysis/Synthesis », COST-G6 Conference on Digital Audio Effects (DAFx), Madrid, 2005


Two Fundamental Ideas...1. Atomic sound particles in various descriptions

time-domain: grains, wave periods

spectral domain: FFT frames, phase vocoder frames, sinusoidal partials

metadata: cepstrum spectral envelopes, filter coefficients, sound descriptors

2. Arbitrary rate processing

picking grains

pitch synchronous

multiple frame rates


Overview


...their Realisation...1. Unified framework

sound particles represented as generic data objects

matrix of floating point numbers

streams as time-sequence of objects

(others: hash-table or matrix of objects)

sent in between specific analysis or synthesis modules

2. Sound processing via message passing (control)

64 bit time tags for sub-sample precise reconstitution of the signal

input/output buffers with interpolation


...and their Advantages1. Unified framework allows generic

manipulation, storage, and display of atoms

higher modularity

can use math

no longer needing a monolithic

analysis+synthesis+manipulation module

2. Arbitrary control rate processing

is adaptable to frequency/texture/rhythm of sound or analysis algorithm

not limited to fixed-rate audio DSP chain


From Signal to AtomWays to cut the input buffer into vectors:

arbitrary grain extraction: gbr.grain~

pitch synchronous elementary waveforms: gbr.psy~

based on yin pitch estimation algorithm

(de Cheveigné and Kawahara, JASA 2002)

constant rate frames: gbr.slice~

FFT

phase vocoder analysis

additive analysis


From Atom to SignalWindowing

Inverse FFT

FFT-1 (additive synthesis)

Overlap add: gbr.ola~

shortcut for paste and drain (output delay line buffer)


...and in between...Manipulation

of the data

of the timing

Display

Storage and retrieval

files (audio, SDIF)

databases

Future: network or inter-application send/receive via OSC


Gabor Summary

Ideas Realisation

atomic sound particles in multiple representations

unified framework of specific analysis–

synthesis, generic data

arbitrary rate processingevent processing with time-tags + delay lines


MnM package

Classification and recognition

motion capture data, sound and music

Basic matrix calculations

Linear algebra

PCA, GMM, HMM, NMD, EM, etc.

Reference article:Frederic Bevilacqua, R. Muller, N. Schnell « MnM: a Max/MSP mapping toolbox », New Interfaces for Musical Expression, Vancouver, 2005

“Mapping is not Music”

Richard Siegal - If/Then Dance Performance


MubuPackaging of frequently used signal analysis, representation and synthesis algorithms with perfect synchronization as MSP externals

Mailing list [email protected]

Reference article:Norbert Schnell, et al. R. Borghesi, D. Schwarz, F. Bevilacqua, R. Müller « Mubu & Friends - Assembling Tools for Content Based Real-Time Interactive Audio Processing in Max/MSP », International Computer Music Conference (ICMC), Montreal, 2009

Real-Time

Corpus-Based

Concatenative Synthesis:

CataRT

http://imtr.ircam.fr

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Principle of CBCS

Corpus-basedsounds + segmentation + descriptors (timbre, metadata, classes)

concatenative grains + overlap + chaining

real-time synthesis interactive + explorative (navigation)

SMC Tutorial #2 — Diemo Schwarz and Norbert Schnell — IMTR Team — IRCAM - Centre Pompidou

Motivation and Approachwork with all the nuances of real sound

large sound databases exist, ready to use

new method ! new sound creation

data-driven vs. rule-based approach

general superiority of data-driven approaches in many other domains


Influences and LinksInspired by concatenative speech synthesis

CBCS in real-time (as implemented in CataRT) can be seen as a content-aware extension to granular synthesis:

providing direct access to specific sound characteristic

not restricted to selection by position in a sound


Applications of CBCS

Interactive exploration of sound databases (browsing)

Resynthesis of an audio target (mosaicing)

Texture synthesis (ambiences, soundtracks)

High-level instrument synthesis (example: Synful)

Expressive speech synthesis


A Short History of Data-Driven Synthesis

I.Prehistory (1948)

Concrete and early electronic music

II.The Classics — revisited (1980, 1990)

Sampling

Granular Synthesis

John Oswald

III.The Modern Times (2000)

many research projects, often in mutual ignorance


Tape splicing

soft attack or decay

combined attack and decay of two sounds

medium attack or decay

hard attack or abrupt finish

softer and less abrupt than D

[figures: Mikhail Malt]

Karlheinz Stockhausen: Étude des1000 collants (1952)


John Cage: Williams Mix (1953)

[http://www.medienkunstnetz.de/works/williams-mix]


Related Work up to 2005

1948

19801990

1993

20002001

2001

20012002

2003

2003

2003

2003

2003

20042004

2004

20042005

2005

2005

Selection

Let them sing it for you

Granular

Samplers

Synful

Musescape

CataRT

MoSievius

stochastic SoundscapesSoundclustering

La legende des sieclesN.A.G.

MATConcat

targeted Directed Soundtracks

Soundmosaicing

Input driven resynthesisMPEG!7 Audio Mosaics

Soundspotter

automatic

Ringomatic

Musical Mosaicing

Caterpillar

Audio Analogies

manual segmental high!level

Musique Concretemanual

Plunderphonics

similarity descriptorssimilarityframe spectrum

Analysis

fixed

2005

2003

[Schwarz, JNMR 2006]

Current Related WorkTimbreID (William Brent) is a collection of PD externals that allow descriptor analysis, corpus-based concatenative synthesis and CataRT-style 2D interaction

Extending voice-driven synthesis to audio mosaicing (Jordi Janer, Maarten de Boer): a system for controlling audio mosaicing with a voice signal

Mused: Navigating the Personal Sample Library (Graham Coleman): a system for sample-based music composition

ScrambledHacks (2006): performance system for spectral lookup alongside ganged music video concatenation


CataRT Introductionflexible testbed to experiment real-time corpus-based concatenative synthesis: CataRT

realisation of CBCS in real-time

patch for Max/MSP with FTM, Gabor, MnM by Norbert Schnell et al. from http://ftm.ircam.fr

released as Free Software under GPL at http://imtr.ircam.fr

mailing list concat on http://list.ircam.fr

modular MVC architecture:building blocks for you to use

CatArt (?)

CataRT


CataRT Overview

CorpusAnalysis

DB

Synthesis

Audio Input Control Input


Analysis

Corpus

Soundfile+markers

+descriptors

Persistent Sound and Unit

Database

descriptor analysis

Soundfile+grain sizeor markers

RT audio input

Audio Input

onset detection

Soundfile


Synthesis

Corpusnearest

neighbour selection

Target descriptor values Control

transformation + concatenation

descriptor analysis

RT audio input

onset detection

controller

Composers and Projects using CataRT

Sound design and compositionLuis Naõn, Hans Tutschku, Matthew Burtner, Sebastien Roux, Hector Parra, Roque Rivas, Ben Hackbarth

InstallationPlumage (project Enigmes about the navigable score, Roland Cahen), Pierre Jodlowski, Cécile Babiole, Franck Leibovici

Live electro-acoustic music! !Dai Fujikura, Stefano Gervasoni, Miguel Angel Ortiz-Pérez, Sam Britton, Aaron Einbond, Bruno Ruviaro, Christopher Trapani, Marc

Vitoria, Eric Maestri, theconcatenator

Examples: myspace.com/catartsoftware

!"

!"

18

""

""

q = 60

q = 60

!"

!"

q = 102

q = 102

!#

!#

""

""

!"

!"

A. Fl.(smallgroup)

B. Cl.(smallgroup)

Tpt.(smallgroup)

1 Vln.(smallgroup)

1 Vc.(smallgroup)

electronics

Fl.

Ob. I

Ob. II

Cl. I

Cl. II

Bsn. I

Bsn. II

Hn. I

Hn. II

Hn. III

Hn. IV

Tpt.

Tbn. I

Tbn. II

Timp.

B. D.

Vln. I

Vln. II

Vla.

Vc.

Db.

!

"

!

"

!

"

!

"

!

"

!

"

!"

$SMALL GROUP

%&'%( % % %) % # % #

p

* + + ,ca. 20 - 21 sec.

, ,- , , , ,

$ % %( %&'(% % % #

p

%) * , , , ,- , , , ,

$ % % #

p

% #) . + , , , ,- , , , ,

$ %&'( #% % #

% % %p

+ / , , ,- , , , ,

0 $% %1 % % %p

% #) . , , , ,- , , , ,

$

section 3

*

4

%%%%%%%%%(

211( %%%%%%%

((

more and more dense and violent(grain size gets smaller and smaller)Harmony 3Harmony 2

gliss.

gliss.

+3 semitones,

-4 semitones,

%%%%%%%%%1

2222 %%%%%%%22(1 ,

CUT OFF

5

34 5 . * + /

$ , , , , ,- +marcato

*ff

$%'6 $%

'6$%

'6%7( %7($%

'6

%7 . . %7 . $%'

( 6$%

'6%7( %7 .$%

'6 %78 . * $%'6%78 . * .

$%'

6$%

'( 6

%7 .%7

*sffffz

$%'6%7'( % * * espress.

fff

. $%'6 %8 %( %

$%'6

%% #

$ , , , , ,- +marcato

*ff

$%'( 6 $%'

( 6%7 %71 * $%

'6 %78 . + $%'( 6 $%'6

%7 .%7(

.$%

'6$%'6%7 %7 * %

%')$%

'( 6

$%'6

%7 %7

. %7( *sffffz

%7'(8 . *

espress.

*fff

%) $%'6% %( # % %

$ , , , , ,- + marcato*ff

. %%'()

$%'6

%74 $%'6%7(

. . %7 .

$%'

( 6 %78 *$%

'6 %7(8 . * * $%'6

$%'6

%7 %7 . $%

'( 6 %7(8 * .

$%'6

sffffz

%78fff

%7'(. . % %8 . * + *

espress.

%6

$ , , , , ,- +marcato

*ff

%%') %74 .

%%

'

2 )%%')

$%'6

%7%7 .

%7

(.

%%')

%%

'

)%%

'

)%7( %7 %7* $%'( 6 $%

'6

%%

'

)

%7 %7(%%')

%%

'

)%7. %7 %7

. %%')

%%

'

)$%

'6

%7

( .%7( %7 .

sffffz fff

$%

'

( 6

%7'

(

%(. + *

espress.

%%')

%%(

%%'

()%

%

$ORCHESTRA

, , , , ,- +marcato

* .ff

%%')

$%'( 6

%74%%

'

2 )%7 .

%7. .

%%

'

) %%')%7 %7 . %

%')%%

'

) %%')

$%'

( 6

%7 %72. %7

$%'

( 6

%%')

%7( . . %71%%

'

)%72. %7

.%%

'

)%%')

sffffz

%7 %7fff

espress.

%%'

()$

%

'

(6

%7'% %( %

% #%

% #%

%%

'

)%

% %

9 , , , , ,- +marcato

*ff

$%'2 6%78 .$%

'6

$%'6%7 . %7 .

$%'

6$%

'6$%'2 6 $%

'6%7 %72 . %7 $%'6 $%'6

%%')

%7 %7 . %7%%')

$%'6

%7 . . %7$%

'6%7 .%7

.$%

'6 %78 .

sffffz fff

$%'

( 6 espress.

%7'

2%

%( . $%'

( 6%( %( % %) *

$%'6 % # %

9 , , , , ,- +marcato

*ff

%%') %74 . .

$%'6

$%'6%7

%7 . %

%'2) %%') $%

'6%7 %7 . %7 * %

%') %%')

$%'6

%7 %7$%

'6%7 . %7 . . %%') %74 .

$%'

6 %78 *sffffz

espress.

fff

%7' % %( % %

) * $%'

( 6 %( # % %) *

$ , , , , ,- + + marcato

ff

. $%'6

%74 * 9 . $%'6

$%'

2 6%7. %7 . $%

'6 $%'6%7 . %7 * . $%'6%78 * $%

'6 %78 . *sffffz

ff

$%'

( 6%7' * %7( * $%'6 %7(8 . + .$%'6 %78 *

$ , , , , ,- + 9+ marcato*ff

$%'2 6 %78 . .$%

'6 %728 *$%

'6%78 . *$%

'6%78 . * $%'6 %78 . * $%

'6 %728 .sffffz ff

$%'6%7'

. . %( . $%'( 6 %78 * $%'6

$%'

( 6%7 . . %7 + $

$ , , , , ,- + marcato9 *ff

$%'6%78 . .

$%'6%78 * * . $%

'6$%

'6%728 $%'6%7 . . %7 + * $%'

( 6 %78 . * $sffffz

%7'8 . *ff

*$%

'6 %7( 4 . * *

$%'

6%74 . * 9

$ , , , , ,- + 9*ff

marcato$%'2 6%78 . * $%'6%78 . . $%

'6%74 * * . %

%') %74 * . $%'6

$%'6

%74 %728 . * .$%

'6

sffffz

%78ff

%%')%7'( %7( * * $%

'( 6 %78 . * $ . $%'6 %7( 4 *

%%'

() %7( 4 .

$ , , , , ,- + + marcato

ff

$%'( 6%78 . * *$%

'6 %78 . .

$%

'6 %%

')%72

. %7. $%'( 6%78 * . $%

'6 %7(8 * *sffffz ff

$%'6%7' %7 * * . $%'6 %78 + .$%'6 %78 *

9 , , , , ,- , , ,(senza sord.)

sffffz

%7'2 4 . * + /

9 , , , , ,- , , ,straight mute

sffffz

%7'(8 . *super short staccato

+p

%7'( 7' 7' 7' 7'

5

gliss.

9 , , , , ,- , , ,p

soft sticks

%'( ' ' ' ' ' ' ' ' ' ' '3 3 3 3

gliss.

: , , , , ,- , , ,sffffz

%7'4 . * + /

$ , , , , ,- marcato+ *ff

$%'6 %7(8 .

$%'6

$%'

6%7( . . %7 * $%

'6 %7(8 .$%

'6 $%'6%7(

. .%7

* $%'6

%74 . . $%'

( 6 %7(8 *$%

'6 %74 .

sffffz

unis.

fff

$%'6 espress.%7'( %

%($%'6

% %% %( % %4 . *

$ , , , , ,-div.

marcato+ * .ff

$%'6 %7(8 * $%

'6 %78 . . $%'( 6 %7(8 * *

$%'6 %78 .

$%'6 $%

'( 6%7 . . %71 . $%

'( 6 $%

'6%%')%7(

.%7

. %7

$ , , , , ,-

div.

marcato

+ *ff

$%'( 6%78 .$%

'6$%

'6%7(. .

%7(*

$%

'6 %74 . %

%'()%%')

%7

( . . %7* . %

%') %78 .$%

'( 6

$%'

6sffffz

unis.%7* .

%7

2fff

$%'6 espress.

$%'

6%7'( %7 . %(

%) * /

$ , , , , ,-marcato

+ff

* .$%

'( 6

%718 * . $%'6 %7(8 .

$%'

( 6 %78 *

$%'6

$%'6%7( . %7 .

$%'6

$%'

6%7

. %7 .

$%'

6 %%')%72

. . %7

( *

0 , , , , ,-div.

marcato+ *ff

$%6%74 . $%

'2 6 %

%')%7 . . %7 * $%

'6 %78 . . $%'6 %72 4 *

$%'

( 6$%

'6%7 . %7 . %

%') %78 . $%'

2 6%74 .

$%'

6 %74 .

sffffz

unis.

fff

espress.%7' %( %$%

'6%

%( % %% #

% % *

0 , , , , ,- marcato+ff

* .$%( 6

%78 * $%'

2 6%74 . . $%'6 %72 4 *

$%'

( 6 %78 . *$%

'6$%

'6%7 . %7 . . $%

'6%78 * . %%') %72 4

9 , , , , ,- +marcato

*ff

$%'6

%74 . .$%

'6 %78 * $%'2 6$%

'6%7 . . %7 * $%

'6%74 . $%'6

%%')%7 . . %7 * .

$%'

6 %78 *

sffffz

unis.

%%'()

espress.

%7'%( .

fff

. %%')

% %($%

'( 6

% % % +

9 , , , , ,-div.

+marcato

*ff

.$%

'6%78 * $%

'6%74 . . $%

'6 %78 *$%

'6

$%'6%7

. %7 . $%'2 6

%%')%7 . . %7 +

$%'

6 %78 .

9 , , , , ,- +marcato

*ff

$%'6

%78 . $%'6$%

'6%7. %7 . .

$%'6

%78 * .$%

'2 6 %78 *

$%'6

%78 . * .$%

'6%78 +

sffffz

%7'%7( * *

fff

$%'6 %(

) + * %)

2

Installation GrainStick“Rainstick” gestural control metaphor

Demo of SAME (Sound And Music for Everyone Everyday Everywhere Every way) EU project

Composition of sounds and ambiences by Pierre Jodlowski

WFS spatialisation and Motion Tracking by Grace Leslie and the Room Acoustics team

Gesture analysis by Bruno Zamborlin and Diemo Schwarz

Current research and applications at the Real-Time Music Interaction Team (IMTR) — http://imtr.ircam.fr — Diemo Schwarz

Grainstick Example 1


Grainstick Example 2


Grainstick InstallationsPrototype: Agora Festival 2009, Ircam, Paris

Public installation: 6. June 2010, Cité des Sciences, Paris

including video projection, more collaborative elements, several universes

Supports et concept de diffusion

Structure tubulaire démontable autonome de 10 mètres de diamètre. Surface de projection en

toile de cinéma circulaire et système de diffusions vidéo et audio intégrées. Ouverture 1/6e de

la circonférence pour l'entrée du public.

Installation Xe-RocksTurning photocopiers into musical instruments, by video and sound artist Cécile Babiole

On show at the ECM Gantner, Bourogne


Installation PlumageCollaboration with national design school ENSCI led by Roland Cahen, Yoan Ollivier, Benjamin Wulf (ENSCI), Christian Jacquemin, Rami Ajaj (LIMSI), Diemo Schwarz (Ircam)

within the project ENIGMES about the navigable score

3D interface using Virtual Choreographer http://virchor.sourceforge.net


Plumage


Live Recording the CorpusCorpus-based improvisation with live sound input

Diemo Schwarz with Etienne Brunet

Live Algorithms for Music (LAM) concerts:

Improvisation of George Lewis, Evan Parker, Sam Britton, Diemo Schwarz and others

Rien du tout by Sam Britton, Diemo Schwarz

CataRT — http://imtr.ircam.fr — Diemo Schwarz

Electro–Acoustic Music (1)

Whisper Not by Stefano Gervasoni for viola and electronics, computer music realization by Thomas Goepfer

“response” of CataRT to the instrument

morphing between corpora of pizzicato viola and water-drops

Interaction with pre-recorded sound,

interpolation, rearranging, navigation

!"

!"

18

""

""

q = 60

q = 60

!"

!"

q = 102

q = 102

!#

!#

""

""

!"

!"

A. Fl.(smallgroup)

B. Cl.(smallgroup)

Tpt.(smallgroup)

1 Vln.(smallgroup)

1 Vc.(smallgroup)

electronics

Fl.

Ob. I

Ob. II

Cl. I

Cl. II

Bsn. I

Bsn. II

Hn. I

Hn. II

Hn. III

Hn. IV

Tpt.

Tbn. I

Tbn. II

Timp.

B. D.

Vln. I

Vln. II

Vla.

Vc.

Db.

!

"

!

"

!

"

!

"

!

"

!

"

!"

$SMALL GROUP

%&'%( % % %) % # % #

p

* + + ,ca. 20 - 21 sec.

, ,- , , , ,

$ % %( %&'(% % % #

p

%) * , , , ,- , , , ,

$ % % #

p

% #) . + , , , ,- , , , ,

$ %&'( #% % #

% % %p

+ / , , ,- , , , ,

0 $% %1 % % %p

% #) . , , , ,- , , , ,

$

section 3

*

4

%%%%%%%%%(

211( %%%%%%%

((


gliss.

gliss.

+3 semitones,

-4 semitones,

%%%%%%%%%1

2222 %%%%%%%22(1 ,

CUT OFF

5

34 5 . * + /

$ , , , , ,- +marcato

*ff

$%'6 $%

'6$%

'6%7( %7($%

'6

%7 . . %7 . $%'

( 6$%

'6%7( %7 .$%

'6 %78 . * $%'6%78 . * .

$%'

6$%

'( 6

%7 .%7

*sffffz

$%'6%7'( % * * espress.

fff

. $%'6 %8 %( %

$%'6

%% #

$ , , , , ,- +marcato

*ff

$%'( 6 $%'

( 6%7 %71 * $%

'6 %78 . + $%'( 6 $%'6

%7 .%7(

.$%

'6$%'6%7 %7 * %

%')$%

'( 6

$%'6

%7 %7

. %7( *sffffz

%7'(8 . *

espress.

*fff

%) $%'6% %( # % %

$ , , , , ,- + marcato*ff

. %%'()

$%'6

%74 $%'6%7(

. . %7 .

$%'

( 6 %78 *$%

'6 %7(8 . * * $%'6

$%'6

%7 %7 . $%

'( 6 %7(8 * .

$%'6

sffffz

%78fff

%7'(. . % %8 . * + *

espress.

%6

$ , , , , ,- +marcato

*ff

%%') %74 .

%%

'

2 )%%')

$%'6

%7%7 .

%7

(.

%%')

%%

'

)%%

'

)%7( %7 %7* $%'( 6 $%

'6

%%

'

)

%7 %7(%%')

%%

'

)%7. %7 %7

. %%')

%%

'

)$%

'6

%7

( .%7( %7 .

sffffz fff

$%

'

( 6

%7'

(

%(. + *

espress.

%%')

%%(

%%'

()%

%

$ORCHESTRA

, , , , ,- +marcato

* .ff

%%')

$%'( 6

%74%%

'

2 )%7 .

%7. .

%%

'

) %%')%7 %7 . %

%')%%

'

) %%')

$%'

( 6

%7 %72. %7

$%'

( 6

%%')

%7( . . %71%%

'

)%72. %7

.%%

'

)%%')

sffffz

%7 %7fff

espress.

%%'

()$

%

'

(6

%7'% %( %

% #%

% #%

%%

'

)%

% %

9 , , , , ,- +marcato

*ff

$%'2 6%78 .$%

'6

$%'6%7 . %7 .

$%'

6$%

'6$%'2 6 $%

'6%7 %72 . %7 $%'6 $%'6

%%')

%7 %7 . %7%%')

$%'6

%7 . . %7$%

'6%7 .%7

.$%

'6 %78 .

sffffz fff

$%'

( 6 espress.

%7'

2%

%( . $%'

( 6%( %( % %) *

$%'6 % # %

9 , , , , ,- +marcato

*ff

%%') %74 . .

$%'6

$%'6%7

%7 . %

%'2) %%') $%

'6%7 %7 . %7 * %

%') %%')

$%'6

%7 %7$%

'6%7 . %7 . . %%') %74 .

$%'

6 %78 *sffffz

espress.

fff

%7' % %( % %

) * $%'

( 6 %( # % %) *

$ , , , , ,- + + marcato

ff

. $%'6

%74 * 9 . $%'6

$%'

2 6%7. %7 . $%

'6 $%'6%7 . %7 * . $%'6%78 * $%

'6 %78 . *sffffz

ff

$%'

( 6%7' * %7( * $%'6 %7(8 . + .$%'6 %78 *

$ , , , , ,- + 9+ marcato*ff

$%'2 6 %78 . .$%

'6 %728 *$%

'6%78 . *$%

'6%78 . * $%'6 %78 . * $%

'6 %728 .sffffz ff

$%'6%7'

. . %( . $%'( 6 %78 * $%'6

$%'

( 6%7 . . %7 + $

$ , , , , ,- + marcato9 *ff

$%'6%78 . .

$%'6%78 * * . $%

'6$%

'6%728 $%'6%7 . . %7 + * $%'

( 6 %78 . * $sffffz

%7'8 . *ff

*$%

'6 %7( 4 . * *

$%'

6%74 . * 9

$ , , , , ,- + 9*ff

marcato$%'2 6%78 . * $%'6%78 . . $%

'6%74 * * . %

%') %74 * . $%'6

$%'6

%74 %728 . * .$%

'6

sffffz

%78ff

%%')%7'( %7( * * $%

'( 6 %78 . * $ . $%'6 %7( 4 *

%%'

() %7( 4 .

$ , , , , ,- + + marcato

ff

$%'( 6%78 . * *$%

'6 %78 . .

$%

'6 %%

')%72

. %7. $%'( 6%78 * . $%


$%'6%7' %7 * * . $%'6 %78 + .$%'6 %78 *

9 , , , , ,- , , ,(senza sord.)

sffffz

%7'2 4 . * + /

9 , , , , ,- , , ,straight mute

sffffz


+p

%7'( 7' 7' 7' 7'

5

gliss.

9 , , , , ,- , , ,p

soft sticks

%'( ' ' ' ' ' ' ' ' ' ' '3 3 3 3

gliss.

: , , , , ,- , , ,sffffz

%7'4 . * + /

$ , , , , ,- marcato+ *ff

$%'6 %7(8 .

$%'6

$%'

6%7( . . %7 * $%

'6 %7(8 .$%

'6 $%'6%7(

. .%7

* $%'6

%74 . . $%'

( 6 %7(8 *$%

'6 %74 .

sffffz

unis.

fff

$%'6 espress.%7'( %

%($%'6

% %% %( % %4 . *

$ , , , , ,-div.

marcato+ * .ff

$%'6 %7(8 * $%

'6 %78 . . $%'( 6 %7(8 * *

$%'6 %78 .

$%'6 $%

'( 6%7 . . %71 . $%

'( 6 $%

'6%%')%7(

.%7

. %7

$ , , , , ,-

div.

marcato

+ *ff

$%'( 6%78 .$%

'6$%

'6%7(. .

%7(*

$%

'6 %74 . %

%'()%%')

%7

( . . %7* . %

%') %78 .$%

'( 6

$%'

6sffffz

unis.%7* .

%7

2fff

$%'6 espress.

$%'

6%7'( %7 . %(

%) * /

$ , , , , ,-marcato

+ff

* .$%

'( 6

%718 * . $%'6 %7(8 .

$%'

( 6 %78 *

$%'6

$%'6%7( . %7 .

$%'6

$%'

6%7

. %7 .

$%'

6 %%')%72

. . %7

( *

0 , , , , ,-div.

marcato+ *ff

$%6%74 . $%

'2 6 %

%')%7 . . %7 * $%

'6 %78 . . $%'6 %72 4 *

$%'

( 6$%

'6%7 . %7 . %

%') %78 . $%'

2 6%74 .

$%'

6 %74 .

sffffz

unis.

fff

espress.%7' %( %$%

'6%

%( % %% #

% % *

0 , , , , ,- marcato+ff

* .$%( 6

%78 * $%'

2 6%74 . . $%'6 %72 4 *

$%'

( 6 %78 . *$%

'6$%

'6%7 . %7 . . $%

'6%78 * . %%') %72 4

9 , , , , ,- +marcato

*ff

$%'6

%74 . .$%

'6 %78 * $%'2 6$%

'6%7 . . %7 * $%

'6%74 . $%'6

%%')%7 . . %7 * .

$%'

6 %78 *

sffffz

unis.

%%'()

espress.

%7'%( .

fff

. %%')

% %($%

'( 6

% % % +

9 , , , , ,-div.

+marcato

*ff

.$%

'6%78 * $%

'6%74 . . $%

'6 %78 *$%

'6

$%'6%7

. %7 . $%'2 6

%%')%7 . . %7 +

$%'

6 %78 .

9 , , , , ,- +marcato

*ff

$%'6

%78 . $%'6$%

'6%7. %7 . .

$%'6

%78 * .$%

'2 6 %78 *

$%'6

%78 . * .$%

'6%78 +

sffffz

%7'%7( * *

fff

$%'6 %(

) + * %)

2

Electro–Acoustic Music (2)

swarming essence by Dai Fujikura for orchestra and electronics, computer music realization by Manuel Poletti

harmonic selection of specially composed small instrument group and extended playing style

corpus-based orchestration, separate treatment

Re-orchestration, rearranging, navigation

!"

!"

18

""

""

q = 60

q = 60

!"

!"

q = 102

q = 102

!#

!#

""

""

!"

!"

A. Fl.(smallgroup)

B. Cl.(smallgroup)

Tpt.(smallgroup)

1 Vln.(smallgroup)

1 Vc.(smallgroup)

electronics

Fl.

Ob. I

Ob. II

Cl. I

Cl. II

Bsn. I

Bsn. II

Hn. I

Hn. II

Hn. III

Hn. IV

Tpt.

Tbn. I

Tbn. II

Timp.

B. D.

Vln. I

Vln. II

Vla.

Vc.

Db.

!

"

!

"

!

"

!

"

!

"

!

"

!"

$SMALL GROUP

%&'%( % % %) % # % #

p

* + + ,ca. 20 - 21 sec.

, ,- , , , ,

$ % %( %&'(% % % #

p

%) * , , , ,- , , , ,

$ % % #

p

% #) . + , , , ,- , , , ,

$ %&'( #% % #

% % %p

+ / , , ,- , , , ,

0 $% %1 % % %p

% #) . , , , ,- , , , ,

$

section 3

*

4

%%%%%%%%%(

211( %%%%%%%

((


gliss.

gliss.

+3 semitones,

-4 semitones,

%%%%%%%%%1

2222 %%%%%%%22(1 ,

CUT OFF

5

34 5 . * + /

$ , , , , ,- +marcato

*ff

$%'6 $%

'6$%

'6%7( %7($%

'6

%7 . . %7 . $%'

( 6$%

'6%7( %7 .$%

'6 %78 . * $%'6%78 . * .

$%'

6$%

'( 6

%7 .%7

*sffffz

$%'6%7'( % * * espress.

fff

. $%'6 %8 %( %

$%'6

%% #

$ , , , , ,- +marcato

*ff

$%'( 6 $%'

( 6%7 %71 * $%

'6 %78 . + $%'( 6 $%'6

%7 .%7(

.$%

'6$%'6%7 %7 * %

%')$%

'( 6

$%'6

%7 %7

. %7( *sffffz

%7'(8 . *

espress.

*fff

%) $%'6% %( # % %

$ , , , , ,- + marcato*ff

. %%'()

$%'6

%74 $%'6%7(

. . %7 .

$%'

( 6 %78 *$%

'6 %7(8 . * * $%'6

$%'6

%7 %7 . $%

'( 6 %7(8 * .

$%'6

sffffz

%78fff

%7'(. . % %8 . * + *

espress.

%6

$ , , , , ,- +marcato

*ff

%%') %74 .

%%

'

2 )%%')

$%'6

%7%7 .

%7

(.

%%')

%%

'

)%%

'

)%7( %7 %7* $%'( 6 $%

'6

%%

'

)

%7 %7(%%')

%%

'

)%7. %7 %7

. %%')

%%

'

)$%

'6

%7

( .%7( %7 .

sffffz fff

$%

'

( 6

%7'

(

%(. + *

espress.

%%')

%%(

%%'

()%

%

$ORCHESTRA

, , , , ,- +marcato

* .ff

%%')

$%'( 6

%74%%

'

2 )%7 .

%7. .

%%

'

) %%')%7 %7 . %

%')%%

'

) %%')

$%'

( 6

%7 %72. %7

$%'

( 6

%%')

%7( . . %71%%

'

)%72. %7

.%%

'

)%%')

sffffz

%7 %7fff

espress.

%%'

()$

%

'

(6

%7'% %( %

% #%

% #%

%%

'

)%

% %

9 , , , , ,- +marcato

*ff

$%'2 6%78 .$%

'6

$%'6%7 . %7 .

$%'

6$%

'6$%'2 6 $%

'6%7 %72 . %7 $%'6 $%'6

%%')

%7 %7 . %7%%')

$%'6

%7 . . %7$%

'6%7 .%7

.$%

'6 %78 .

sffffz fff

$%'

( 6 espress.

%7'

2%

%( . $%'

( 6%( %( % %) *

$%'6 % # %

9 , , , , ,- +marcato

*ff

%%') %74 . .

$%'6

$%'6%7

%7 . %

%'2) %%') $%

'6%7 %7 . %7 * %

%') %%')

$%'6

%7 %7$%

'6%7 . %7 . . %%') %74 .

$%'

6 %78 *sffffz

espress.

fff

%7' % %( % %

) * $%'

( 6 %( # % %) *

$ , , , , ,- + + marcato

ff

. $%'6

%74 * 9 . $%'6

$%'

2 6%7. %7 . $%

'6 $%'6%7 . %7 * . $%'6%78 * $%

'6 %78 . *sffffz

ff

$%'

( 6%7' * %7( * $%'6 %7(8 . + .$%'6 %78 *

$ , , , , ,- + 9+ marcato*ff

$%'2 6 %78 . .$%

'6 %728 *$%

'6%78 . *$%

'6%78 . * $%'6 %78 . * $%

'6 %728 .sffffz ff

$%'6%7'

. . %( . $%'( 6 %78 * $%'6

$%'

( 6%7 . . %7 + $

$ , , , , ,- + marcato9 *ff

$%'6%78 . .

$%'6%78 * * . $%

'6$%

'6%728 $%'6%7 . . %7 + * $%'

( 6 %78 . * $sffffz

%7'8 . *ff

*$%

'6 %7( 4 . * *

$%'

6%74 . * 9

$ , , , , ,- + 9*ff

marcato$%'2 6%78 . * $%'6%78 . . $%

'6%74 * * . %

%') %74 * . $%'6

$%'6

%74 %728 . * .$%

'6

sffffz

%78ff

%%')%7'( %7( * * $%

'( 6 %78 . * $ . $%'6 %7( 4 *

%%'

() %7( 4 .

$ , , , , ,- + + marcato

ff

$%'( 6%78 . * *$%

'6 %78 . .

$%

'6 %%

')%72

. %7. $%'( 6%78 * . $%


$%'6%7' %7 * * . $%'6 %78 + .$%'6 %78 *

9 , , , , ,- , , ,(senza sord.)

sffffz

%7'2 4 . * + /

9 , , , , ,- , , ,straight mute

sffffz


+p

%7'( 7' 7' 7' 7'

5

gliss.

9 , , , , ,- , , ,p

soft sticks

%'( ' ' ' ' ' ' ' ' ' ' '3 3 3 3

gliss.

: , , , , ,- , , ,sffffz

%7'4 . * + /

$ , , , , ,- marcato+ *ff

$%'6 %7(8 .

$%'6

$%'

6%7( . . %7 * $%

'6 %7(8 .$%

'6 $%'6%7(

. .%7

* $%'6

%74 . . $%'

( 6 %7(8 *$%

'6 %74 .

sffffz

unis.

fff

$%'6 espress.%7'( %

%($%'6

% %% %( % %4 . *

$ , , , , ,-div.

marcato+ * .ff

$%'6 %7(8 * $%

'6 %78 . . $%'( 6 %7(8 * *

$%'6 %78 .

$%'6 $%

'( 6%7 . . %71 . $%

'( 6 $%

'6%%')%7(

.%7

. %7

$ , , , , ,-

div.

marcato

+ *ff

$%'( 6%78 .$%

'6$%

'6%7(. .

%7(*

$%

'6 %74 . %

%'()%%')

%7

( . . %7* . %

%') %78 .$%

'( 6

$%'

6sffffz

unis.%7* .

%7

2fff

$%'6 espress.

$%'

6%7'( %7 . %(

%) * /

$ , , , , ,-marcato

+ff

* .$%

'( 6

%718 * . $%'6 %7(8 .

$%'

( 6 %78 *

$%'6

$%'6%7( . %7 .

$%'6

$%'

6%7

. %7 .

$%'

6 %%')%72

. . %7

( *

0 , , , , ,-div.

marcato+ *ff

$%6%74 . $%

'2 6 %

%')%7 . . %7 * $%

'6 %78 . . $%'6 %72 4 *

$%'

( 6$%

'6%7 . %7 . %

%') %78 . $%'

2 6%74 .

$%'

6 %74 .

sffffz

unis.

fff

espress.%7' %( %$%

'6%

%( % %% #

% % *

0 , , , , ,- marcato+ff

* .$%( 6

%78 * $%'

2 6%74 . . $%'6 %72 4 *

$%'

( 6 %78 . *$%

'6$%

'6%7 . %7 . . $%

'6%78 * . %%') %72 4

9 , , , , ,- +marcato

*ff

$%'6

%74 . .$%

'6 %78 * $%'2 6$%

'6%7 . . %7 * $%

'6%74 . $%'6

%%')%7 . . %7 * .

$%'

6 %78 *

sffffz

unis.

%%'()

espress.

%7'%( .

fff

. %%')

% %($%

'( 6

% % % +

9 , , , , ,-div.

+marcato

*ff

.$%

'6%78 * $%

'6%74 . . $%

'6 %78 *$%

'6

$%'6%7

. %7 . $%'2 6

%%')%7 . . %7 +

$%'

6 %78 .

9 , , , , ,- +marcato

*ff

$%'6

%78 . $%'6$%

'6%7. %7 . .

$%'6

%78 * .$%

'2 6 %78 *

$%'6

%78 . * .$%

'6%78 +

sffffz

%7'%7( * *

fff

$%'6 %(

) + * %)

2


Live Control and Composition

Beside Oneself, Aaron Einbond, 2008 for viola and electronics:

real time control of CataRT from audio analysis of the viola

What the blind see 2009 for small ensemble and electronics

real time control of CataRT from audio analysis of the viola

corpus-based “orchestration” or transcription

texture example:

1. Rain on leaves

2. CataRT resynthesis with extended technique instrument sounds

Aaron Einbond – What the blind see

B

&

&

ã

ã

t

&

ã

Alto

Cl.B.en Sib

Hp.

Pno.

Vib.

F

!

" " "

C10

C10

C10

a tempo

a tempo

a tempo

1936

T T .T#

Œ Ù .T

$T

%6

& . ym

bruit des clefs : doigtés ad lib.

"f "yn y

7

TmflM

ßym y ym ym y Ù TM

flßy ym y y yb

7

® x xx x xm

œSF neige to concatenation

Œ

7

‰ ® jœb œbp

œœ œ œm œ œ

7

‰ œ œ> œ œ .œ

#&

œétouffé avec papier sempre

p

œm

Ó

Ó ?

Œ ‰ jœb˝

‰ &6

˝superball

Vib.

p

3‰6˝ ‰

° sempre

ongles

%

%

6

® Tn -

pT JT Tb T

6

ym"f "

ym ym yy T- JT

%

Jœ

&

6 6 6 6 6>paume

ß

6

œ

pœ œ œm œb

œ œ

6æ j6æ

jyO>ß‰ 5& .

ym>

pplectre

y

˝

> . .y

yO

>

3

‰ œn ˝

® .

.œb˝

6&̋

6

œn ˝

& 6˝

.6˝

7

7

7‰ ® yæ gliss.(main)

arco

#IVyæ 3

yb y. y. y. y.%

6

Tym

ym y y y

6

yn ym ym yn ymT- JT Tb T .TbflM

ßTb Mflß

%

6

® œm œ

.œœ œb œ œ

œ .œ>7

œ .œ œ œ

7

‰ ® œ œb7

%

jyO

7

.yO

yb

#̋y

6

‰ ®6

6

6˝

œb ˝

6. ‰6

®œ˝

6>ß

28

1. snow melting on a metal roof 2. CataRT resynthesis with instrument sounds

3. manually edited transcription 4. live reading by ensemble

Bruno Ruviaro – Intellectual Improperty 0.6

Using Catork, a CataRT-based system for electronic music composition for laptop orchestrashttp://www.brunoruviaro.com/catork

Premiered by the Stanford Laptop Orchestra (SLOrk) April, 2010 (chamber version with six laptops), and June (full ensemble version)

New Concepts for Musical Applications

Re-arranging units by other rules than temporal order

Composition = Navigation through the sound space of heterogeneous

sound databases

exploit richness of detail of recorded sound

efficient control by perceptually and musically meaningful descriptors

Interaction with self-recorded sound: sound of a musician is available for interaction beyond simple repetition

Cross-selection and interpolation: apply sound characteristics from one corpus to another, morphing between corpora

Orchestration and re-orchestration: matching a mass of sounds to a harmonic or sonic target while retaining precise control over the result

Current and Future DirectionsInterface

multi-dimensional scaling by mass–spring–damper model

uniformisation of density

Sound texture synthesis

Modeling of transitions and articulations

40 60 80

0.05

0.1

0.15

0.2

0.25

light rain

40 60 80

0.05

0.1

0.15

0.2

0.25

medium rain

40 60 80

0.05

0.1

0.15

0.2

0.25

heavy rain


AcknowledgementsCataRT is based on FTM&Co by Norbert Schnell et al.

Thanks go to:

Alexis Baskind, Julien Bloit, Greg Beller, Miguel Angel Ortiz Pérez

Roland Cahen, Yoan Olliver, Benjamin Wulf (ENSCI),

Christian Jacquemin, Rami Ajaj (LIMSI)

Norbert Schnell, Riccardo Borghesi, Fréderic Bevilacqua, Rémy Muller, Jean-Philippe Lambert (IMTR Team)

the French National Agency of Research ANR and the projects Sample Orchestrator and Topophonie

Links:

CataRT wiki on imtr.ircam.fr,

mailinglist [email protected] and [email protected]

on http://lists.ircam.fr


Exercise SuggestionsAdapting CataRT

Integration in your concert patch

Adapting to your musical needs/ideas

Improving CataRT

Corpus Data

soundfile editor (list/delete sounds in corpus)

soundset editor (define sound sets, add/remove units)

descriptor range viewer/editor (e.g.: exclude units below threshold)

Analysis

segmentation editor

Selection

constraints: don’t repeat unit

Synthesis

forced pitch/loudness

multi-channel output

Control

by MIDI notes

by envelopes, sequencer

by live sound analysis

Diemo Schwarz – CataRT workshop

Architectureone catart.data per corpus

one catart.import for segmentation and analysis

any number of catart.data.proxy to access data

any number of catart.lcd for display and control

any number of catart.selection per corpus

any number of catart.synthesis per selection or per corpus


AcknowledgementsCataRT is based on FTM&Co by Norbert Schnell et al.

Thanks go to:

Alexis Baskind, Julien Bloit, Greg Beller, Miguel Angel Ortiz Pérez

Roland Cahen, Yoan Olliver, Benjamin Wulf (ENSCI),

Christian Jacquemin, Rami Ajaj (LIMSI)

Norbert Schnell, Riccardo Borghesi, Fréderic Bevilacqua, Rémy Muller, Jean-Philippe Lambert (IMTR Team)

the French National Agency of Research ANR and the RIAM project Sample Orchestrator

Links:

CataRT wiki on imtr.ircam.fr,

mailinglist [email protected] on http://lists.ircam.frand [email protected]

Advanced real-time sound and data processing with...

Documents

Transcript of Advanced real-time sound and data processing with...