Heterogeneous data orchestrationInteractive fantasy under SuperCollider

Sébastien ClaraCIEREC UJM – PhD student

35 rue du 11 novembreSaint-Étienne, France, 42000

sebastien.clara@univ-st-etienne.fr

Abstract

For L’Imaginaire music ensemble, I composed apiece of interactive music involving acousticinstruments and surround electronic music. Theinteraction between live musicians and electronicsis based on data collected in real time fromacoustic instruments. This data is further used toadjust timbre and synchronize electronics with therest of the music.

Keywords

SuperCollider, interactivity, music mixed

1 Introduction

For L'Imaginaire1 music ensemble, I composeda piece of interactive music mixed. Mixed music isa term used in musicological literature to refer to amusical genre. It is defined by the alloy ofinstrumental music and electronic music. For thispaper, I wish to focus on an interactive property ofmy piece.

Historically, the electronic part of mixed musicis composed in a studio and fixed on a magnetictape2. This technique makes impossible theinteraction between the interpreters and theelectronic sound. Feedback helps to improve thecompositional process of the tape. The othertechnique is to perform audio processing of theacoustic instruments in real time3. In this case, we

1Musical ensemble composed by Keiko Murakami(flutes), Philippe Koerper (saxophones) and MaximeSpringer (piano). http://www.limaginaire.org/.

2The first pieces of music using this technique:Orphée 51 and Orphée 53 by Pierre Schaeffer andPierre Henry (1951 and 1953) and Musica su duedimensioni by Bruno Maderna (1952 and 1958).

3The first pieces of music using this technique:Mixtur (1964) and Mikrophonie I (1964) by KarlheinzStockhausen.

are talking about a device that increases the sonicpossibilities of the acoustic instrument. Feedbackis used to regulate electronic sound by a human oran automaton.

I use these two techniques of electronic soundaccompaniment in my work, but the increase incomputing power and the versatility of the toolshave allowed a median way. In the next chapter, Ipresent my problematic. Thereafter, I will showmy issue with an example which could beextrapolated to other devices.

2 Problematic

According to Robert Rowe, "Interactivecomputer music systems are what are the changesin response to musical input. Such responsivenessallows these systems to participate in liveperformances, of both notated and improvisedmusic" [1]. How can a system listen to a musicianand make an appropriate decision to generate asound response? In his book, Rowe analyzessystems that use the MIDI standard tocommunicate between instruments and computers.But how can we use traditional instruments?

The audio descriptors4 correspond to parametersthat describe an analyzed sound. A set ofdescriptors is used to construct a data set andcreate spaces for representing the sound. Theparameters that are extracted can be described indifferent ways, depending on what is expectedfrom the information conveyed by the parameter.

The sound acquisition sensor for the processingunit is a microphone. Therefore, the use of audiodescriptors in interactive computer music systemsallows the use of standard acoustic instruments.

4With SuperCollider, it is necessary to install itsextension package to benefit from a wide choice ofdescriptors : https://github.com/supercollider/sc3-plugins.

To control electronic sound with the sound of anacoustic instrument, many descriptors can be used.To do this, it is necessary to match a soundcharacteristic with the values of the parametersthat describe it. The preparation of this report willallow us to establish a particular threshold andonce it is crossed, the system can trigger aresponse.

However, the interval between the extremevalues of a parameter depends on its nature and onthe analyzed sound source. To use this techniqueof interaction between an instrumentalist andelectronic sound, a first difficulty is to negotiatewith the heterogeneity of the data produced by thedifferent audio descriptors.

For example, I want to use the nuance and pitchof a sound to build a particular accompaniment.The amplitude descriptor returns a number that canrange from 0 to 1 and the pitch descriptor returns afrequency. The range of extreme values returnedby the pitch descriptor depends on the ambit of theanalyzed sound source. Determining a triggerthreshold to activate a response of a systeminvolves crossing values of different magnitudessometimes from sound sources of different natures.

Moreover, this technique uses sound capture andthis information is variable depending on itsenvironment. Consequently, elaborate settings in astudio with a particular electro-acoustic chainwould be less effective in another location with adifferent electro-acoustic chain. So, how can wetreat the heterogeneity of the data produced bydifferent audio descriptors, different soundsources, different electro-acoustic chains anddifferent concert locations, to achievehomogeneous electronic music accompanimentfor each interpretation of the same piece?

3 Creating a repository

The first step is to establish a reference. Thiswill serve as a standard for a single piece, aspecific electro-acoustic chain and a particularplace. We will be obliged to renew it each timeone of these three parameters changes.Furthermore, this repository will allow us tocharacterize our data and to determine thresholdsfor performing a particular electronic soundaccompaniment.

In the following example, we use the amplitudedescriptor (Amplitude.kr) and pitch (Pitch.kr). Thevalues are transmitted by the OSC protocol(SendReply.kr) to the SuperCollider clients at the

/dataTrigger address. The data is sent wheneveran onset is detected (Onsets.kr). Wheninstantiating the synthesizer, two arguments areavailable. The first determines the number ofinputs of the signal to be analyzed on our audiointerface (SoundIn.ar). The second determines theonset detection threshold.

Execution of figure 1 will only give thedefinition of your synthesizer to SuperCollideraudio server5. Synthesizer is not instantiated, so itdoes not work and it does not ask for resources toyour hardware. We will run6 it only when we wantto receive data (figure 2).

We now have to build a data collector toconstitute our repository. To do this, we use theOSCFunc object. It is fast responder for incomingOSC messages. We configure it with thepreviously defined OSC address. When a newmessage arrives from the analyzer, it executes afunction. In this case, this function saves theamplitude and pitch of the signal in array globalvariables.

We set to -9 dB the onsets detection threshold.We can change this parameter for change thedensity of the data reception. The analyzer listensto the first input of our audio interface (defaultsetting). In figure 2, running the first block startsthe acquisition of the data. Running the last linekills the synthesizer and responder instances, freesmemory and processor usage. The collected

5http://doc.sccode.org/Classes/SynthDef.html.6http://doc.sccode.org/Classes/Synth.html.

information is stored in variable arrays. We can

The graph shows the evolution of the parameterover time and allows us to make a correspondencebetween a sound characteristic and some values.The histogram provides a representation of thedistribution of the values of the parsed parameter.This observation allows us to characterize thedistribution produced by a descriptor.

4 Using the repository

In our system, we determine the value of athreshold to trigger a response to accomplishdynamic electronic music accompaniment. Thischoice may be arbitrary or be determined inresponse to a specificity of the analyzed soundsource. Above a certain value, our programtriggers a response for example. We can alsochoose this value according to its frequency ofappearance in the distribution and according to thesound result produced by this choice, we canincrease or decrease the density of our electronicaccompaniment by modifying the value of ourthreshold. However, how do we handle values ofdifferent magnitudes?

We manipulate repository by the requestedpercentile. To use this method, it is necessary toinstall an additional library. For that, you can usethe SuperCollider package manager7 to installMathLib. This one gives us access to additionalstatistics methods for arrays.

The percentile rank corresponds to theproportion of the values of a distribution less thanor equal to a determined value. We manipulate ourdata with float values from 0 to 1. For example, ifwe want to know the value equal to 90% of ourdata, we use 0.9.

During the adjustment phase of our system, wecan tune several parameters of differentmagnitudes transparently with a single scale.

7http://doc.sccode.org/Classes/Quarks.html.

5 System Response

In order for our system to respond to certainstimuli, we must attribute to it a means of soundproduction. To do this, we define an arbitrary

To implement a concrete example, we assumethat our device listens to two types of percussion.One of the percussions emits sounds high-pitchedthan the other. We decide that our system willrespond only to the percussion which emits themost high-pitched sounds and to the most loudsounds. With onsets detection threshold, ourcondition for triggering a response depends ontwo others parameters:

If the frequency of the answers does not suit us,we can return to the choice of the values of thesevariables to modify the sensitivity of our system.

The implementation of a response for oursystem follows the same structure as Figure 2. Aresponder wait for incoming OSC messages fromthe analyzer. When a new message arrives, if itmeets the previously formulated conditions (high-pitched and loud), a response is issued.

This response is customized according to theanalysis data. This connection is made by asonification process, "technique of renderingsound in response to data and interactions" [2]. Ido not deal with the mapping technique in thispaper, but its mastery is a source of variation andexpressiveness for electronic music. To make thisrelation, we map the synthesizer parameter froman input range to an output range. We can set theinput range with the repository information andadjust the output according to the desired soundquality. Figure 7 is the implementation of theresponse of our system.

At the end, we must free objects for freesmemory and processor usage.

6 For further

For this paper, we concentrated our system to itssimplest expression. In this chapter, we wish todevelop it design. Some of these ideas wereconceived during the development of our piece andothers afterwards.

Robert Rowe divides his interactive computermusic system (Cypher) into two sections [1]. Thelistener analyzes the data produced by a musicianand the player delivers a musical response. Thestructure of SuperCollider source code implies thisorganization. Our analysis synthesizer is thelistener and the function of the responder objectfor incoming OSC messages is the player. We keepthese terms to locate the following points.

Our listener can also have an implicit function oftime master. Indeed, we transmit the data of theanalysis every time an onset is detected. But wecan transmit them at a given frequency. Theresponses delivered by the system would then behave a beat.

We can use our listener to produce anautomation (with Env and EnvGen.kr objects). Anautomation allows to control and to automate thevariation of a parameter over a given time. In thisway, we determine the evolution of any thresholdor parameter.

We can parallelize other listeners who analyzeother sound qualities (brightness, noise,dissonance, etc.) to achieve other triggersthreshold and make complex electronic musicaccompaniment executed by our system.

For our player, we can define a maximumnumber of synthesizers executed in parallel inorder to preserve the resources of the system and /or to control the acoustic density so as not tosaturate our perception.

In addition, we can perform a certain musicalprocess8 or that feeds on our repository instead ofrunning a simple synthesizer. The interactivesystem developed by Jean-Claude Risset for hisduets for one pianist [3] is very interesting for thisway. He uses MIDI data (pitch, velocity andduration) which he transforms according totraditional compositional operations:transposition, reversal, canon, etc.

7 Implementation

I control my audio processor by a graphicalinterface (Figure 9) and MIDI controller. Thedifferent audio tracks allow me to adjust theintensity of the electronic sound layers. The flute,sax and piano tracks deal with interactiveelectronic sound accompaniment. The synth trackmanages my non-real time composite electronicmusic. Finally, the live track amplifies theacoustic instruments.

The creation of the repository is realizeddirectly in the interface and makes this actiontransparent. Graphical interface allows a soundengineer to play my work and this interface makesrehearsals and concerts easier.

A MIDI pedal assigned to a performer managesthe overall setting. Fourteen key moments

8http://doc.sccode.org/Tutorials/A-Practical-Guide/PG_01_Introduction.html.

articulate the electronics for a duration of fifteenminutes.

To create my electronic accompaniment, I usethe descriptors of amplitude, pitch, centroid andnoise. The operation of coupling between the dataof the analysis and the parameters of thesynthesizers [2] makes it possible to control theform of electronic music accompaniment byscaling, transposition or reversal. This relation canbe fixed or variable.

Generally, the sound source of my electronicmusic accompaniment does not come from soundsynthesis, but acoustic instruments. When thedensity of the responses of the system is notlimited, the responses are superimposed tocontinuously transform the timbre of the electronicsound. When the density of the responses of thesystem is limited, the responses can arrive insuccessive waves and produce a dynamicaccompaniment.

8 Epilogue

To design our system, our initial motivation wasto simplify the use of different descriptors, tosimplify our system settings, to customize theresponses of the system according to the sensitivityof the interpreter and to produce a homogeneouselectronic accompaniment to each interpretation ofthe same notated music under different conditions.

But in addition, we obtain an open interactivesystem that can be adapted from a specific modelto the intuition of a musician. We identified foursteps to explore in order to implement our practicalsolution and develop an interactive scenario [4].

The first step focuses on the sound of theinstrumentalist. What particularities of sound dowe want to relate to our system? What filter do wewant to use to trigger an answer? In the exampledeveloped for this paper, we make our filter withthe parameters of onset detection, pitch andamplitude. The thresholds established to constitutethis filter allow us to play on the sensitivity or theparticularity of the answers delivered by oursystem.

The second question to implement our solutionis to choose the type of response to trigger. Shouldthe answer be monophonic, polyphonic,contrapuntal, etc.? In other words, whatorganizational model should we use to develop ourresponse? In our example, we produce one itemper answer. This element is strongly correlatedwith the sound analyzed and the operations applied

to determine the sound characteristics of ourresponse are conceived during the last step of thispractical solution.

The third step in using our system is todetermine which synthesizer we want to assign toour system. Controls can be implemented insynthesizers. This possibility can give us solutionsto build a previously chosen model. For example,a synthesizer can perform a glissando. In theexample developed for this paper, we use a simplegranular FM synthesis.

The final step in implementing our solution is todetermine the type of relationship between theanalysis data and the parameters of oursynthesizer. How to get expressive sounds withsonification process [2] ? Should our relationshipbe static or dynamic? How should the plan ofconnections between these various elements beestablished? For the example developed in thispaper, we have established a one-to-many andmany-to-one static connection plan. Theamplitude determined by the analysis is correlatedwith the granular density of the synthesis, themodulation index and the duration of theresponse. The pitch determined by the analysis iscorrelated with the pitch of our response. Atransposition is performed by a scaling operation.Finally, the amplitude and the pitch analyzedserve to determine the modulation frequency ofthe FM synthesis of the response delivered by oursystem.

9 Conclusion

For this paper, we have implemented our openinteractive system under SuperCollider - platformfor audio synthesis and algorithmic composition.We could have implemented this system on othersoftware. Moreover, use free software increasesthe durability of our work. Laurent Pottier recallsthe history of the precariousness of technologiesin electronic music [5] and free software answersto this problem.

An example concerns the portage of Pluton byPhilippe Manoury from the 4X9 to Max [6]. Thepiece did not really sound exactly the same wayon both platforms. After a thorough study of the4X, the engineers discovered that a 4X hardwarelimitation influenced the sound result. Thislimitation was implemented in the Max patch tofind an electronic music equivalent [7].

94X is real time effect processors designed byGiuseppe di Guigno at IRCAM in the 1970s.

Free software is an important factor of durabilityand reproducibility in the digital art. The ubiquity[8] of free software allows more flexibility toimagine original devices [9]. In the end,researchers have no lock to study and increase thecommon good.

10 Acknowledgements

Thanks to Laurent Pottier for his encouragementand his pertinent advices.

References

[1] R. Rowe. 1993. Interactive Music Systems:Machine Listening And Composing. MIT Press,Cambridge.

[2] T. Hermann, A. Hunt, J. G. Neuhoff. 2011. TheSonification Handbook, Logos Verlag, Berlin.

[3] J.-C. Risset, S. Van Duyne. 1996. Real-TimePerformance Interaction with a Computer-Controlled Acoustic Piano. In Computer MusicJournal 20/1, pp. 62–75, MIT Press, Cambridge.

[4] B. Laurel. 2014, Second Edition. Computers asTheatre. Addison-Wesley, Crawfordsville.

[5] L. Pottier. 2015. L’évolution des outilstechnologiques pour la musique électronique, enrapport avec la pérennité des œuvres. Constat,propositions. A. Saemmer, editor, E-Formes 3,Les frontières de l’œuvre numérique, pp. 245-261, PUSE, Saint-Étienne.

[6] M. Puckette. 2002. Max at Seventeen. InComputer Music Journal 26/4, pp. 31-43, MITPress, Cambridge.

[7] J. Szpirglas. 2012. Composer, même avectrois bouts de ficelle… Entretien avec PhilippeManoury. http://etincelle.ircam.fr/1077.10.html.

[8] S. Letz, S. Denoux, Y. Orlarey. 2014. AudioRendering/Processing and Control Ubiquity? aSolution Built Using the Faust DynamicCompiler and JACK/NetJack. In ProceedingsICMC|SMC, Athens.

[9] M. Lallement. 2015. L'âge du faire.Hacking, travail, anarchie. Le Seuil, Paris.