1

TOWARD INTELLIGENT MUSICAL INSTRUMENTS: NEW WIRELESS MODULAR GESTURAL CONTROL INTERFACES

By

CHESTER JAMES UDELL III

A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT

OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY

UNIVERSITY OF FLORIDA

2012

2

© 2012 Chester James Udell III

3

To my wife

4

ACKNOWLEDGMENTS

I thank Dr. James Paul Sain for his continual guidance through this long journey. I

also thank Dr. Karl Gugel, who had the sense of adventure to entertain a crazy

composition student’s desire to pursue electrical engineering. Furthermore, I thank all of

the professors who invested considerable time and effort in me including Dr. Paul

Richards, Dr. Paul Koonce, Dr. Silvio Dos Santos, and Dr. Welson Tremura. You all

have made a profound impact on my career and on my outlook of music, technology,

and life. I am also obliged to Dr. A. Antonio Arroyo and Dr. Arthur Jennings for their

advice and input during my doctoral study and dissertation. And finally, I would not have

made it this far if not for the continual support and wisdom of my wife. I love you,

Monique.

5

TABLE OF CONTENTS page

ACKNOWLEDGMENTS .................................................................................................. 4

LIST OF FIGURES .......................................................................................................... 7

LIST OF OBJECTS ......................................................................................................... 8

LIST OF ABBREVIATIONS ............................................................................................. 9

ABSTRACT ................................................................................................................... 11

CHAPTER

1 INTRODUCTION .................................................................................................... 13

Current Issues for Augmented Music Instruments .................................................. 15

Proposed Solutions ................................................................................................. 17 Modular ............................................................................................................ 17 Reversible ........................................................................................................ 17

Non-invasive ..................................................................................................... 18 Reconfigurable ................................................................................................. 18

Limitations and Scope............................................................................................. 19

2 HISTORIC TRAJECTORY ...................................................................................... 22

Anatomy of a Musical Instrument ............................................................................ 22 Musical Instruments after Electricity........................................................................ 23

Early Electronic Instruments (The Primacy of the Keyboard) ........................... 24

Music for Loudspeakers ................................................................................... 28 Computing, From Laboratory to Stage ............................................................. 29

MIDI and Other Serial Communication Protocols ............................................. 32

3 ELECTRIC VERSUS ELECTRONIC MUSIC INSTRUMENTS TODAY .................. 34

Electric VS Electronic.............................................................................................. 34

Instrument Taxonomy ............................................................................................. 35

Alternative Controllers............................................................................................. 36 Augmented Instruments .......................................................................................... 39 Augmentations of the Trombone ............................................................................. 44

4 ON MUSICAL GESTURE ....................................................................................... 49

Sound, Motion, & Effort ........................................................................................... 50 Gesture............................................................................................................. 52 Musical Gesture ............................................................................................... 53

6

Instrumental Gesture ........................................................................................ 54

Spectro-morphology ............................................................................................... 55 Gesture & Paralanguage ........................................................................................ 57

5 INTERFACING WITH THE ACOUSMATIC. ........................................................... 58

Sensor Interfaces .................................................................................................... 58 Related Work: Wireless Sensor Interfaces ............................................................. 61 Current Trends ........................................................................................................ 62

Single-Serving Innovations ............................................................................... 63

Transparency ................................................................................................... 65 Accessibility ...................................................................................................... 65

The Dysfunctions of MIDI ........................................................................................ 66

6 EMOTION: RECONFIGURABLE MODULAR WIRELESS SENSOR INTERFACES FOR MUSICAL INSTRUMENTS ..................................................... 70

Overall Design Philosophy ...................................................................................... 70

Sensor Nodes ......................................................................................................... 72 Node Design Considerations ............................................................................ 74

Node Types ...................................................................................................... 75 The MCU ................................................................................................................ 76 Addressing Protocol ................................................................................................ 76

Radio Specifications ............................................................................................... 77 Receiver Hub .......................................................................................................... 79

Hub Design Considerations .................................................................................... 80 Software Client ....................................................................................................... 81

Data Input ......................................................................................................... 81 Data Processing ............................................................................................... 84 Data Mapping ................................................................................................... 85

Implementation: Augmented Trombone Using eMotion .......................................... 86

7 CONCLUSIONS AND FUTURE DIRECTIONS ...................................................... 89

Broader Impacts ..................................................................................................... 90 Future Directions .................................................................................................... 90 Conclusions ............................................................................................................ 94

APPENDIX: MUSICAL SCORE: CAPOEIRISTA FOR FLUTE, BERIMBAU, AND LIVE ELECTRONICS .................................................................................................... 95

LIST OF REFERENCES ............................................................................................. 107

BIOGRAPHICAL SKETCH .......................................................................................... 113

7

LIST OF FIGURES

Figure page 3-1 An illustration of where in the process electricity is introduced for electric

instruments. ........................................................................................................ 34

3-2 An illustration of where in the process electricity is introduced for electronic instruments ......................................................................................................... 35

3-3 Based on Miranda-Wanderley’s 2006 text on Taxonomy of DMI Types. ............ 36

5-1 Market-available wired sensor interfaces and their specifications as of 2011. ... 60

5-2 Market-available wireless sensor interfaces and specifications as of 2011. ....... 62

6-1 System Overview ................................................................................................ 71

6-2 Sensor Nodes ..................................................................................................... 72

6-3 A comparison of various market-available wireless data transceivers. ............... 79

6-4 Software client workflow. .................................................................................... 81

6-5 Software client screenshot of sensor modules. .................................................. 83

6-6 Screenshot of Data processing window. Processing flows from top-down. ........ 84

6-7 Screenshot: Patch Bay Window ......................................................................... 85

6-8 Alternative Mapping Screenshot ......................................................................... 86

6-9 Implementation of the eMotion System on the Bass Trombone. ........................ 88

7-1 Workflow of a Novel DDW Software Client. ........................................................ 93

8

LIST OF OBJECTS

Object page A-1 Sound file of “Capoeirista,” recorded November 30 2011 (.wav file 77MB) ........ 95

9

LIST OF ABBREVIATIONS

a Address Bit

A/D, ADC Analog to Digital Converter

AHRS Attitude Heading Reference System

CSIR Council for Scientific and Industrial Research

CSIRAC Council for Scientific and Industrial Research Automatic Computer

CTIA International Association for the Wireless Communications Industry

d Data Bit

DAW Digital Audio Workstation

DCM Direction Cosine Matrix

DDW Digital Data Workstation

DIY Do-it-yourself

DMI Digital Musical Instrument

DOF Degrees of Freedom

EEPROM Electrically Erasable Programmable Read-Only Memory

EVI Electronic Valve Instrument

EWI Electronic Wind Instrument

FSR Force Sensitive Resistor

GENA_1 General Analog Node, 1 Input

GENA_6 General Analog Node, 6 Inputs

GPIO General-purpose Input/Output

HMSL Hierarchical Music Specification Language

IMU Inertial Measurement Unit

IRCAM Institut de Recherche et Coordination Acoustique/Musique, Paris

IP Internet Protocol

10

LED Light-emitting Diode

MAC Media Access Control

MCU Microprocessor

MIDI Musical Instrument Digital Interface

MIT Massachusetts Institute of Technology

NIME New Interfaces for Musical Expression

OSC Open Sound Control

RF Radio Frequency

RX, PRX Receive, Primary Receiver

Sig-CHI A special Interest group for Computer-Human Interactivity in the Association for Computing Machinery

STEIM Studio for Electro-Instrumental Music, Amsterdam

TPE Trombone Propelled Electronics.

TX, PTX Transmit, Primary Transmitter

UDP User Datagram Protocol

UI User Interface

USB Universal Serial Bus

WSN Wireless Sensor Network

11

Abstract of Dissertation Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy

TOWARD INTELLIGENT MUSICAL INSTRUMENTS: NEW WIRELESS MODULAR

GESTURAL CONTROL INTERFACES

By

Chester James Udell III

May 2012

Chair: James Paul Sain Major: Music

This dissertation is dedicated to combining some of the latest ultra-low power

microprocessor and radio frequency (RF) wireless technology to build a new wireless

sensor network (WSN) for musical instruments called eMotion. The hardware is

designed to address three current issues in the field of augmented musical instruments:

single-serving innovation, accessibility, and transparency. This interface implements a

unique design approach when compared to other similar music interfaces currently

available. To this end, the eMotion hardware will be modular, reversible, non-invasive,

and reconfigurable – permitting a musician to interact with computers in live

performance using the physical gestures of their instrument.

Beginning with a general account of historic developments and technological

precedents – the aesthetics of augmented instrument design will be discussed in relief

to other design approaches. In response to the issues facing augmented instrument

development in the literature, a conceptual framework for the design of the eMotion

system will be constructed based on the notions of musical gesture. The second half of

the dissertation consists of technical documentation, construction, limitations, and

12

potential applications of this unique system. An account implementing this new wireless

gestural interface on the bass trombone will also be discussed.

Also included is a piece composed by the author for Flute, Berimbau, and Live

Electronics. The original version is a piece for live performers and electroacoustic

sound for four channels. It is presented here as a stereo reduction along with the

MaxMSP software performance program on a data CD.

13

CHAPTER 1 INTRODUCTION

Human interaction with sounding objects lies behind instrumental gesture. The passage from object experimentation to the creation of a musical instrument involves the increasing refinement of hitting, scraping or blowing, such that sophisticated and detailed techniques of control are consciously elaborated as a result of the performer-listener's conscious exploration of the interactive relationship. Gradually a performance practice evolves. However refined, specialized or rarefied the instrument, its morphologies and associated techniques may become, the primal hit, scrape or blow never fades from the listener's mind. It is a knowledge possessed by everyone, and its sophisticated instrumental guise is something all can appreciate.

– Denis Smalley. The Listening Imagination [1]

Throughout history, musical instruments have followed a process of co-evolution

with developing technology to become what we know them as today. From sticks and

bone, to cat-gut strings, to brass, to the well-tempered piano-forte, to electronic

synthesizers, music instruments reflect the major technological breakthroughs of an

epoch shaped by culture and the natural human predisposition to interact with

“sounding objects.” As music technology rapidly advances and computers become ever

more accessible, faster, and smaller, a question is raised: how can acoustic instruments

and traditional performance practice be interfaced to present-day technological

developments? Ten years ago, Kim Binsted, in the paper Sufficiently Advanced

Technology: Using Magic to Control the World, contended that computers were

ubiquitous (this is before the Apple® iPod and iTouch emerged on the market) [2]. In

this past decade, the saturation of new mobile and personal network technology has

significantly shaped the paradigm of what it means to interact with computing. Having

become invisibly interwoven into the fabric of our daily routine, these conceptual

14

developments have only begun to marginally effect conventional music instrument

design and performance practice.

This dissertation is dedicated to combining some of the latest ultra-low power

microprocessor and radio frequency (RF) wireless technology to build a new wireless

sensor network (WSN) for musical instruments called eMotion. A WSN is comprised of

several individual wireless sensors – called nodes – that measure environmental

conditions including temperature, pressure, ambient light, motion, and sound. This data

is wirelessly transmitted to a central hub for monitoring. A special focus is placed on the

trombone – the author’s instrument of formal training. A method of attaching and

removing individual sensor nodes will be devised as might be required for performance

with live interactive electronics. This interface implements a unique design approach

when compared to other similar music interfaces currently available. The eMotion

system will be modular, reversible, non-invasive, and reconfigurable – permitting a

musician to interact with computers in live performance using the physical gestures of

their instrument. With this technology the performer, improviser, and composer will

have the basic building blocks to easily combine or reconfigure unique sensor groupings

to control parameters such as: effects processing, algorithmic and generative computer

music, and virtual sonic environments, all the while requiring no knowledge of

microprocessors or programming. For instance, what if the clarinet (an omni-directional

instrument) could physically localize its sound to a specific area in a concert hall based

on the directional orientation of the instrument? Or if the amount of distortion effect on

an electric guitar could be controlled not with a foot pedal, but by simply tilting the

guitar?

15

Current Issues for Augmented Music Instruments

In the late 1970s and early 1980s, when the size and cost of microprocessors and

sensors significantly decreased, composers began to experiment with retrofitting

musical instruments with this new technology to communicate gestural data to

computers. The principle behind augmented instruments is that they maintain their

original acoustic properties and, to some degree, extend their traditional performance

practice. Historical perspectives, context, pros and cons, and aesthetic issues of

instrument augmentation presented in relief with other forms of digital music instrument

design are detailed in Chapter 3. The implementation of WSN technology is an attempt

to respond to three emergent issues that have been raised in the literature on

augmented instruments over these last four decades: single-serving innovations,

transparency, and accessibility.

Based on an observation of performances for specific augmented instruments –

Tod Machover’s Hyper Cello, Ben Neill’s Mutant Trumpet, and Matthew Burtner’s

Metasax to name a few. It seems they are not only technological extensions of the

instruments themselves, but also specific to the composer/performer for whom it is

designed. For example, Wanderley and Orio posit, “extended instruments have been

designed to fit the idiosyncratic needs of performers and composers, but as such they

have usually remained inextricably tied to their creators [3].” Furthermore, the particular

innovative developments these augmented instruments employ have commonly been

disseminated only in a limited number of instances, namely conference presentations.

This single-serving approach to instrument development is not necessarily counter-

productive. On the contrary, individual experimentation is a catalyst for a rich diversity

of solutions and methods. However, this trend may also be a contributing factor in the

16

general failure to disseminate such innovations and performance practice to the broader

community.

Another major issue facing the design of augmented instruments is the

invasiveness of the sensing technology. Instrument augmentation may require physical

alterations that might be difficult to undo or are completely irreversible. Additionally,

wires tethering to the computer and the size and weight of the hardware components

tend to throw off the balance of the instrument and often interfere with traditional

performance practice. Using smaller sensors and microprocessors along with new

wireless technology to mitigate the unwieldy nature of this process is a primary goal of

this study.

Finally, despite valuable advancements in wireless networking and

microprocessors, along with a growing community of developers for embedded devices

like ARDUINO™, the accessibility of this technology to the average classically trained

musician remains largely out of reach. Currently, the technology is at a point that still

requires at least a hobbyist’s knowledge of microprocessors, programming, and sensors

to even begin to experiment with novel interactive systems. This limits the use of these

systems almost exclusively to those who can acquire a certain level of technical

expertise beyond music making. In a time where an elementary school child can

effectively use a cell phone (a fantastically sophisticated piece of technology) without

any knowledge of how to fabricate or program one, trends in current conference

proceedings for both Computer Music (NIME) and Electrical Engineering (SIG-CHI for

example) point towards a growing need for composers to employ (and for classically

17

trained musicians to use) physical modes of interactivity between an instrument and

electroacoustic sound using computers without any expert knowledge [4–7].

Proposed Solutions

To this end, eMotion is step towards a general-purpose solution based on the

following design criterion: modular, reversible, non-invasive, and reconfigurable.

Modular

The eMotion system is comprised of individual sensor nodes (a transmitter/sensor

pair) and a single universal receiver that collects the sensor transmissions and sends

compiled packets to the computer via USB. Each node has its own unique ID (based

on its sensor type and instance) and dedicated to only transmitting the data of its

attached sensor. For instance, one node may be dedicated to transmitting sonar data

and another node may be dedicated to measuring ambient light. The user is then free

to utilize one, both, or neither of these sensor nodes by turning them on or off –

attaching or detaching. Multiple sensor nodes of the same sensor type may also be

used. In this manner, one can acquire a collection of general sensor nodes and use a

subset of them in any combination as needed for a particular application.

Reversible

The use of eMotion should not require any destructive alterations to the music

instrument itself so that the user can easily revert back to their original acoustic

instrument when desired. Thus, various methods including “hook-loop” fasteners, weak

removable adhesives, and bendable soft materials are being explored for attaching and

detaching these sensor nodes.

18

Non-invasive

One of the issues plaguing musical instrument augmentation is the network of wire

often required. Similar musical interfaces available on the market have only one

relatively sizable wireless transmitter to which all sensors must be connected by wires.

This issue can be mitigated when each sensor node wirelessly transmits its own data;

localizing only short wires to the specific position on the instrument. The sensor nodes

are designed to be exceptionally small (the largest prototype node is 1.26 inches in

diameter) to minimize weight and clutter on the instrument.

Reconfigurable

The nature of this interface design allows the user to easily reconfigure the

placement and combination of individual sensor nodes for any given project and

instrument. This enables the user to find the most optimized network combination,

sensor-instrument placement, and mapping to meet their unique aesthetic goals without

having to redesign the technology itself. The sensor nodes may even be distributed

amongst several instruments or non-instrumental performers, like dancers, to create

novel modes of musical interactivity. For instance, a dancer may control the spectral

filtering of a flute player through their body movements.

These eMotion sensor nodes can be viewed analogously to a piece of hardware

that musicians are generally familiar with: the mute. For brass and string instruments,

mutes are placed inside of the bell or on the bridge to alter the sound with interesting

effect. These are readily available for musicians to acquire. eMotion sensor nodes may

be regarded as an extension of this tradition, where a musician can easily acquire and

place these objects on an instrument to extend its sonic and expressive capacities with

the aid of the computer and microphone. With these nodes, the composers/performers

19

can intuitively construct and reconfigure their own customized wireless physical

interfaces between instruments and computers. The possibilities and interactive

paradigms may reach a new level of flexibility, expression, and accessibility.

Limitations and Scope

The scope of this document is limited to detailing the historic trajectory, conceptual

framework, technical development, and implementation of eMotion and a novel software

client designed by the author called a Data DAW. Additionally, this document specifies

the general aesthetic issues in the fields of augmented instruments and acousmatic

music to which eMotion is responding. Although this document focuses primarily on

developing a wireless sensor array for the trombone in particular, the broader impact of

this research is to create an open system where this technology will be easily

transferable to any musical instrument, dancer, or other object in a live performance

setting. This sensor data could also be mapped to control stage lighting and video

projections in addition to sound, but this is also beyond the scope of the dissertation.

Furthermore, there are classes of technology geared towards reducing the amount of

effort (which will be defined in Chapter 4) to perform certain tasks (which is a necessary

mantra for increasing quality of life and extending opportunities for the disabled).

However, it is not the aim of this research to make instrument performance easier per

se. Rather, the agenda is to augment the expressive capacities of the instrument and

take full advantage of effort that is naturally exerted by the performer. A musical

instrument may be viewed as an extension of the performer’s physiology (breath,

hands, arms, fingers, etc) to make sound. Sensing the physical gestures of the

performer and broadcasting this data to computers will enable new indicative

relationships between a performer, instrument, and acousmatic sound. To this end, it is

20

the natural, physical virtuosity of the performer that stems from countless hours spent in

a practice room that eMotion will take advantage of.

In Chapter 2, a general account of the historic trajectory and technological

precedents leading up to current Digital Musical Instrument (DMI) [8] design types will

be presented. Chapter 3 offers a look at DMIs today the concept of co-evolution

between music instruments and technology and the aesthetics of augmented instrument

design in contrast to other classes of digital instruments. The dichotomy between

augmented instruments and alternative controllers will briefly be addressed. However, it

is not the intention of the author to quantify augmented instruments as a “better”

approach than alternative controllers (which are musical controllers that do not emulate

traditional instruments for the sake of purposely avoiding traditional performance

paradigms). Alternative controller design is a valuable field of exploration, but is well

beyond the scope of this dissertation’s concerns.

Chapter 4 poses the question: what does one mean by the word gesture, and

more specifically, musical gesture? This chapter’s aim is not to propose a unified

general definition of the term, but rather to define it within the context of this document’s

focus. Contributing towards this framework will be the perception of cross-modal

dissonance for composing and experiencing music for physical musician with non-

physical acousmatic sound. The ideas of Trevor Wishart (gesture as musical

paralanguage) and Denis Smalley (spectro-morphology and surrogacy in acousmatic

sound) will be implemented to construct the point of convergence between physical and

non-physical acoustic phenomenon.

21

Sensor Interfaces and their technical limitations will be reviewed in Chapter 5

along with figures comparing several different market-available products. The

inadequacy of the MIDI protocol to encode and represent the intimate and necessary

physical gestures of the performer to interact with sound will also be discussed.

The technical development behind eMotion is detailed in Chapter 6 starting with

the overall design philosophy. The individual components such as microprocessor type,

sensors, and wireless technology will be presented along with the author’s experiential

reasons behind selecting them over other popular alternatives. Specification tables

including data rates, range, and current consumption illustrate the capability and

capacity of eMotion for reference. Finally, a client designed in MaxMSP to receive and

dynamically map this data in real-time to available musical parameters will be detailed.

The application of the eMotion system for trombone will also be illustrated.

In the final chapter, the author concludes the dissertation by summarizing how

eMotion attempts to address the issues discussed in previous chapters, the potential

ramifications for music composition and performance practice, current limitations of the

hardware, and proposing future directions for eMotion.

22

CHAPTER 2 HISTORIC TRAJECTORY

What we want is an instrument that will give us continuous sound at any pitch. The composer and electrician will perhaps have to labor together to get it. At any rate, we cannot keep working in the old-school colors. Speed and synthesis are characteristics of our epoch. We need twentieth-century instruments to help us realize them in music.

– Edgard Varèse. Making Music Modern [9]

Throughout the history of Western music, instruments have been continuously

evolving and every period has given rise to new or modified instruments and playing

techniques. This chapter details a brief history from the earliest electronic instruments

leading up to the development of microcomputers. From there, four taxonomic branches

of electronic instruments will be outlined as supported in the current literature, with a

particular focus on augmented instruments.

General historic accounts for electronic music instruments, especially for the years

1750 to 1960, are many [8], [10–12]. The objective for this chapter is not to thoroughly

reiterate the existing literature. The intent is to outline a path that may serve to establish

a contextual trajectory and to provide a highlight reel or montage focusing on electronic

music interfaces leading up to the development of new technology to be discussed in

Chapter 6.

Anatomy of a Musical Instrument

Musical instruments are a specialized form of technology. They extend the

expressive capacities of human movement to produce acoustic phenomena beyond

what our bodies can make alone. Musical instruments are comprised of two

fundamental components – a performance device and a sound generator. The

performance device is what the performer physically manipulates to control the sound

23

(e.g., the valves on a trumpet, bow of a violin, or keys on a clarinet) and will be referred

to within the scope of this dissertation as the gestural interface. The sound generator for

a given instrument propagates the sound (vibrating strings or air columns, body

resonance, shape, and material of the instrument) and it influences qualities like

resonance and timbre [10]. For conventional acoustic instruments, the physical interface

and sound generator are inextricably related, meaning that the device the performer

manipulates and the object that propagates sound are fundamentally one cohesive

system. In this manner, the physical gestures performed on an instrument (like buzzing

lips, bowing strings, and striking material) directly and visibly produce and shape sound;

sound and gesture are one. All musical instruments were bound to this causal gesture-

to-sound paradigm, until the first electronic instrument was developed in 1759 [11].

Musical Instruments after Electricity

The ability to harness the power of the electron in the mid 1700s set the stage for

a quantum leap in the evolution of musical instruments. The Clavecin Electrique

(electric harpsichord), invented by Jean Baptiste Delaborde, is one of the first electric

instruments to mimic the form and performance paradigm of traditional acoustic musical

instruments [11]. Unlike its acoustic counterpart, the electric harpsichord generated

acoustic tones mediated by electricity. The mechanical separation of performance

device (a harpsichord keyboard) from the sound generator (electromagnetically

vibrating tines) set the stage for subsequent developments in electronic instrument

design and is arguably the first musical instrument of record to have fractured the

conventional gesture-to-sound paradigm. For these instruments, physical gesture and

the resulting sound is mediated by electrical processes. This disconnect between

24

gesture and sound has served as the basis for both the prospects and problems in

moving forward with electronic instrument design.

Early Electronic Instruments (The Primacy of the Keyboard)

With all the creative potential that comes with the ability to electronically

synthesize any tone, timbre, and temperament, the extant literature suggests an

interesting limitation consistently imposed on a significant proportion of early electronic

instruments [13]. Conventional 12-tone chromatic keyboard interfaces served as the

primary control interface of choice. A factor contributing to this trend might be the

limitations of the technology itself. Electronic wind instruments didn’t emerge until the

1960s and 70s and electronic string controllers didn’t emerge until the 1980s. While

alternative means to control electronic instruments beyond the 12-tone keyboard were

well explored during this period, few of these electronic instruments met with any

significant level of success (defined by longevity and widespread use).

In 1906, Thaddeus Cahill’s Telharmonium (resembling a complex organ) piped the

first Muzak via telephone wire to nearby hotels, restaurants, museums, and resident

subscribers. Maurice Martenot premiered his Ondes Martenot at the Paris Opera in

1928. This instrument featured a unique electronic ribbon-controller (a looped string with

a ring attached to the middle). A performer controlled pitch by placing their finger in the

ring and sliding along the length of the ribbon. The left hand could manipulate a variety

of controls to vary loudness and timbre. A keyboard diagram was soon added to inform

the performer of pitch location and eventually a keyboard controller was fully integrated

into the design.

Laurens Hammond introduced the Hammond Organ in 1935, the first commercially

successful electric music instrument. Within the first 3 years of production, a Hammond

25

could be found in 1,750 churches and was manufacturing roughly 200 instruments a

month by the mid 1930s [14]. The original model-A generated sound using 91 rotating

tone wheels driven by a synchronous motor and had two 5-octave manuals and a 2-

octave pedal board. Electronic oscillators eventually replaced the tone wheels in the

1960s.

Hugh LeCaine, a Canadian scientist and electronic music pioneer invented the

Electronic Sackbut (a sackbut being the medieval ancestor of the trombone) in 1948.

Ironically, it did not resemble a brass instrument at all, but it vastly expanded the

expressive capacity of the diatonic keyboard controller. Each key was not only sensitive

to vertical pressure (to control articulation and amplitude) but also horizontal placement.

In this manner, a musician could achieve glissandi spanning an octave in either

direction by sliding their finger across the keys – resembling the sliding pitch of a

trombone. Its timbre was also changeable by mixing “noise” into the sterile purity of

electronic tones to emulate more natural acoustic phenomena (e.g. the airy tone of a

flute or the fuzzy articulations of an oboe).

Despite the prominence of keyboard-controlled electronic instruments in early

years, it is important to note that numerous non-keyboard controlled electronic

instruments were developed throughout this period and some achieved notable

widespread use. For example, a non-contact method to convert physical gesture into

electrical signals emerged in post-revolutionary Russia in 1920. Russian cellist and

electrical engineer Leon Theremin unveiled the aetherphone, also known as the

theremin, a television-size cabinet with two antennas sensitive to the proximity of

human hands (electrical capacitance). With this instrument a performer could control

26

pitch using the antenna sticking straight up with one hand and amplitude using the

“loop” antenna with the other hand. Theremin took his invention to New York where

many performances for this instrument were staged. It is interesting to note that the

theremin was often used to perform classical literature, including performances of

Rachmaninoff’s Vocalize and, in one case with the New York Philharmonic, Liszt’s

Hungarian Rhapsody no 1 for four theremins and orchestra.

Another example includes Friedrich Trautwein’s non-keyboard instrument invented

in 1928, inspiring Paul Hindemith to write a solo concerto for the instrument [10].

Discrete control over pitch and articulation was possible using the Trautonium. A

performer controlled pitch by moving their finger along a length of wire with the right

hand and articulated tones by pressing a bar with the left (resembling the right/left hand

performance practice of the theremin). The instrument was so successful that Oskar

Sala (Trautwein’s student) developed an improved model of the instrument called, the

Mixturtrautonium. It was used to produce sound effects for movies including Hitchock’s

the Birds and bell sounds for the 1950s Beyreuth production of Wagner’s Parsifal [12].

In the late 1950s and 60s, Raymond Scott (an accomplished bandleader, pianist,

composer, and engineer) designed a variety of electronic music instruments. Some

instruments, like the Clavivox and Videola were controlled primarily with a keyboard.

However, as Robert Moog pointed out, “Raymond quickly realized there were more

elegant ways of controlling an electronic circuit [15].” He implemented photoresistors in

his Circlemachine, an electronic sequencer comprised of lights mounted on spinning

wheels inside of wheels. It was autonomous, modulating speed and sound through its

own movements. The Electronium was the most successful of Scott’s instruments and

27

versions were sold to Motown Records. Referred to as the “instantaneous

composition/performance machine,” the Electronium algorithmically generated musical

material on its own. An operator could interact with the process using an array of knobs,

buttons, sliders, and switches. This led Motown executive Guy Costa to refer to the

instrument as the “idea machine [15].”

The invention of the transistor allowed for smaller and more cost effective design

methods, moving technology away from expensive, fragile and sizable vacuum tube

technology. This enabled Robert Moog to develop Modular Voltage-Controlled

Synthesizers based on his research presented at the Audio Engineering Society

Conference in 1964 [16] and his associations with pioneers like Raymond Scott and

Herbert Deutsch [15]. His modular methodology significantly changed the approach to

electronic instrument design and is a principle that informs the present document.

Studios, engineers, and composers often developed hardware to perform specific and

limited tasks, requiring the development of new hardware from scratch each time.

Moog’s solution was to break down the process of electronically generated sounds into

discrete “blocks” where each module performed a specific function (generally there

were three types: signal generating, signal modifying, and voltage control). The user

could then combine and configure these blocks to synthesize a variety of sounds

without having to redesign the hardware. Despite the fact that any interface could have

been used to control Moog’s synthesizers, most continued to utilize the diatonic

keyboard interface. Ribbon controllers, knobs, patch bays, and sliders were often

incorporated into Moog’s designs, but played secondary functions to the keyboard.

Therefore despite several early innovations and experiments, the literature suggests

28

that musicians interacted with early electronic instruments almost exclusively through

the conventional keyboard.

Music for Loudspeakers

Developed in parallel with early electronic instruments was the emergence of

commercially available sound recordings. While evidence of earlier prototypes exist

[17], Thomas Edison is credited with the development of technology capable of

recording sound around the year 1877. Recording allowed sound to be disembodied

from its original acoustic source and be reproduced via loudspeaker in a separate time

and space. As a result, loudspeakers essentially became a universal instrument,

capable of representing virtually any acoustic phenomenon. This notion sparked novel

methods for composing with and experiencing sound. Musique concrète started in Paris

in 1948 by Pierre Schaeffer. The term ‘concrete’ was used to distinguish a method for

working directly (concretely) with sound material versus that of working indirectly (or

abstractly) with sound using a system of notation, which had to be realized though

instrumental or vocal performance. Thus, material for musique concrète was derived

exclusively from pre-recorded sound. The medium is often associated with recordings

derived from “real-world” sounds including natural environments, instruments, and other

objects. Conducive to the aesthetic of collage, the sounds are to be appreciated beyond

their original source and context to include their abstract musical properties (i.e. timbre,

pitch, and rhythm) and relationships between one sound and another. By the 1950s,

‘electroacoustic’ music was promoted as a better term to describe a synthesis between

the concrete and electronic approaches of working with sound [18]. The relationship

between a sound’s attached associations and its abstract musical properties became

the basis for the acousmatic movement.

29

Meanwhile in the United States, Bebe and Louis Barron created one of the earliest

electroacoustic music studios in 1948 in New York. The Barrons often crafted novel

electronic circuits for projects to create the sounds they needed and eventually

amassed a collection of these devices to use for compositional purposes. Cage hired

the Barron’s studio to engineer his sound for Williams Mix 1952.

Also in 1952, Raymond Scott designed perhaps the first multi-track tape machine

in the world, capable of recording 7 to 14 parallel tracks on a single reel that resulted in

several patents for magnetic tape technology. Hugh LeCaine later devised a way to mix

down six separate tracks of tape in 1955.

Despite the limitations of early technology, composers worked to accommodate

live performance situations with electroacoustic music. Some of the earliest examples of

compositions for acoustic instruments and acousmatic sound include: Ottorino

Respighi’s use of a phonograph to capture and play back a nightingale song in his

orchestra work Pines of Rome (1924), John Cage’s Imaginary Landscape no. 1 (1939)

for phonographs and live instruments, and Milton Babbitt’s Vision and Prayer (1961) for

tape and soprano. This gave rise to a new compositional aesthetic: pieces for live

instruments and fixed media. At this point in time, the burden remained on the musician

to “chase” or synchronize with the tape.

Computing, From Laboratory to Stage

In the 1950s, computers became a significant compositional tool. The earliest

instance of a computer explicitly programmed to generate musical tones was the CSIR

Mk1, Australia’s first computer. Later called CSIRAC, Geoff Hill programmed it to play

popular melodies as early as 1950 [19]. An Australian composer and resident of the

United States, Percy Grainger collaborated with Burnett Cross to build his Free Music

30

Machine in the mid 1950s. This was not a computer per se, two large rollers fed four

sets of paper into a series of mechanical arms. The arms rolled over the contours cut

into the paper and controlled pitch, timbre, and amplitude of eight tone oscillators.

Voltage fluctuations corresponded to patterns on the paper, sonifying the graph

drawings [11].

Despite these early breakthroughs, the establishment of computer music is

credited to Max Matthews’s work at Bell Labs, which began in 1957. While developing

methods to convert sound into computer data for playback and for telephones,

Matthews became inspired by the musical implications. His program, Music I, resulted in

an IBM 704 playing 17 seconds of music. Alongside the efforts of many engineers and

composers including John Pierce, Milton Babbit, Jean-Claude Risset, James Tenney,

Pril Smiley, and Emmanuel Ghent, Matthews continued to develop his computer music

program. Subsequent versions, Music II through Music V, served as predecessors to

many computer music programs in use today including Csound. Later, the Groove

system was developed by Matthews in collaboration with F Richard Moore in 1968.

Equipped with potentiometers, joysitck, keyboard, and typewriter (for inputting

commands), the Groove system allowed a user to record and edit gestural input in real-

time, a process that Michael Roads called “edited improvisations [20].” It is important to

note that only the gestural data was recorded in real-time to later be used as control

input for voltage-controlled oscillators and amplifiers of an analog synthesizer.

The RCA Mark II Electronic Music Synthesizer was installed at the Columbia-

Princeton Electronic Music Center in 1959. The manner of programming this computer

was similar to that used with CSIRAC, punching holes in paper cards using binary

31

number sequences and then feeding it into the machine. The process was time-

consuming, tedious, and according to Milton Babbit, required the “patience of Job [10].”

Despite the limitations, the possibilities of this system provided unprecedented control

over musical elements including timbre, timing, and texture.

In the 1970s Charles Moore created an interesting hybrid compiled/interpreted

programming environment called Forth [21]. The appeal of this approach is the ability to

program software using an interactive shell environment. Programming could be done

virtually in real-time, executing subroutines, testing, and debugging without having to

recompile each time. This had an instant appeal to composers, as a moment of

inspiration could quickly be realized in sound. It also made algorithmic composition and

interactive performance more accessible and resulted in what George Lewis calls a

“dialogic creative process, emblematic of an improvisor’s way of thinking [22].”

Successors of Forth include Hierarchical Music Specification Language (HMSL) [23]

and Formula (FORth MUsic LAnguage) [24].

In 1976, Giuseppi di Giugno developed the first all-digital computer controlled

synthesizer with the encouragement of Luciano Berio [25]. The first model, the 4A,

allowed a user to program not only the individual control function devices, but also the

interconnections between the devices themselves; a kind of virtual patching [26]. The

next interactive computer music system developed at the Institut de Recherche et

Coordination Acoustique/Musique (IRCAM) was an outgrowth of Giugno’s system

(versions progressed from the 4A, to 4B, 4C, then 4X). IRCAM’s 4X system was

comprised of three processors running in parallel (to compute audio), an external

32

computer (to run programs), and a graphic display. For control input and feedback,

sliders, buttons, organ keyboard, joysticks, and a clarinet controller were provided.

Developing in tandem with this system at IRCAM in the 1980s was the work of

Miller Puckette, who named his programming environment in honor of Max Matthews

[26]. “Max” became the first program designed for live performance and to be widely

distributed to a significant user community. Initially used to program the 4X at IRCAM,

additional collaborations between Puckette and David Zicarelli allowed for valuable

refinements. This ultimately resulted in the program MaxMSP. Still in wide use today,

this graphical programming environment allows users to design their own unique music

programs by connecting basic blocks of code together. Even while the program is

running, the user can reconfigure and modify the program and immediately observe the

results in real-time. It has become a significant platform allowing musicians to program

and interact with computers on stage.

The capacity and power of computer processing has increased exponentially over

time while the size of the hardware has decreased [27]. Computational hardware that

once filled up rooms of space can fit on a microchip. This has alleviated many technical

performance barriers on stage. Computationally intensive processes like live pitch

tracking and gesture analysis can serve as points of interactivity and control in music.

Beyond MaxMSP, other significant music programs including Csound and SuperCollider

have expanded their function to include real-time applications and support new control

interfaces as well.

MIDI and Other Serial Communication Protocols

While physical interfaces could have been mapped to any sound beginning with

the first electronic instrument, practical applications were limited by the available

33

technology in a given era. Before MIDI and other serial communication protocols

developed, gestural controls were still mechanically bound to the physical synthesis

device (i.e. one could argue that the controller and the sound generator were still one

cohesive instrument by design). Disconnecting the keyboard controller from a

synthesizer to use somewhere else was not originally supported in early technology.

The inception of music hardware communication protocols dissolved this final tether –

modularizing the control mechanism and the sound generator as two separate entities.

This paradigm shift resulted in modular control devices for musical instruments. An

issue that emerged, however, was that each manufacturer used a proprietary serial

communication protocol between controllers and synthesizers. As a result, getting

musical controllers and hardware to universally communicate was difficult. The solution

was the development of the MIDI protocol, which established an industry standard for

musical hardware communication in 1983 [28]. Facilitated by the MIDI protocol, an

alternative controller could be readily detached from a synthesizer or computer and

reattached to another, or be used to interact with multiple sound generating devices

simultaneously.

Other music-based serial protocols continue to emerge. MLAN (developed by the

YAMAHA® corporation) was publically released in 2000 and allows users to

communicate not only audio, but also controller information on a single IEEE 1394

cable. Matthew Wright and Adrian Freed developed Open Sound Control (OSC) at

CNMAT at UC Berkley to address the shortcomings of MIDI in 1997. Despite MIDI’s

limitations, it has yet to be completely usurped by these newer, more powerful, and

flexible protocols. However, current trends are moving in that direction [29].

34

CHAPTER 3 ELECTRIC VERSUS ELECTRONIC MUSIC INSTRUMENTS TODAY

Broadly speaking, there are two major categories of instrument that employ

electricity. The distinction between the two is determined by where in the conversion

process electricity is employed [30].

Electric VS Electronic

Electric musical instruments retain a natural gesture-sound paradigm where the

physical gesture directly initiates an acoustic sound. For example, plucking a string on a

guitar or striking a string in a piano with a hammer. In the case of electric instruments,

the acoustic sound is converted into electrical signals via a transducer, or pickup, to

extend the sonic palate of the instrument through amplification, effects processing, etc.

The electric guitar and electric piano are common examples.

Figure 3-1. An illustration of where in the process electricity is introduced for electric

instruments.

Electronic musical instruments convert human gesture input into electrical energy

at the outset of the process and generate the sound entirely through electronic means.

For instance, pressing a key on a MIDI keyboard controller generates an electrical

impulse. The resulting electrical impulses may be mapped to control electronic

phenomenon to generate sound using a loudspeaker. Some examples of electronic

35

instruments include: the HAMMOND® Organ, MOOG Music® synthesizers, and the

YAMAHA® DX7. With notable exceptions like the EWI (electronic wind instrument), EVI

(electronic valve instrument), and YAMAHA® WX7, most commercially successful

electronic instruments today remain oriented towards the conventional keyboard

interface.

Figure 3-2. An illustration of where in the process electricity is introduced for electronic

instruments

Instrument Taxonomy

As new technologies and interactive paradigms emerge, composers and

musicians continue to experiment with the traditional relationships between gesture and

sound. A plethora of new musical controllers with a staggering variety of morphologies

and functions have evolved as a result of the natural human predisposition to

experiment and explore. Each device is truly unique, meeting the personal aesthetics

and technical requirements of each project. In Miranda and Wanderley’s Book “New

Digital Musical Instruments: Control and Interaction Beyond the Keyboard,” a

morphology comparing Digital Music Instruments based on their resemblance to

existing acoustic instruments is proposed (Figure 3-3) [8].”

36

Figure 3-3. Based on Miranda-Wanderley’s 2006 text on Taxonomy of DMI Types [8].

A comprehensive taxonomy of electric and electronic instruments is beyond the

scope of this historical review, but a few useful resources exist that categorize these

instruments into families based on varying criterion (including [8] and [31]). However,

identifying the fundamental differences between Alternative Controllers and Augmented

Instruments will be of significant value here. This distinction is governed principally by

two different aesthetics. Those who approach performance in electroacoustic music as

a continuation of previous traditions are apt to approach instrument design starting with

conventional instruments and performance practice (i.e. augmented instruments).

Those who wish to demonstrate that technology affords radical differences from the

past are likely to avoid emulating conventional instruments and paradigms (i.e.

alternative controllers).

Alternative Controllers

An alternative controller is defined within the scope of this document as:

“Alternative” – a physical interface that facilitates novel modes of interactivity between

performer and sound by avoiding conventional instrument models (like the diatonic

37

keyboard); and “controller” - the device is a control interface that is to be mapped onto a

sound-producing mechanism, and is not the sound-producing mechanism itself. It is

important to note that there were electronic instruments dating back to the original

theremin that utilized alternative methods to control sound. However, because the

control mechanism was inextricably part of the synthesis system by design, these early

instruments are not technically controllers. This document may be the first to make a

distinction between alternative controllers versus alternatively-controlled electronic

instruments. The MIDI theremin developed by Moog’s company, Big Briar, versus the

original analogue theremin is such an example. Controllers may either be built from

scratch using a collection of sensors or re-appropriated from other existing objects (like

game controllers and smart phones). Nicolas Collins in his article, A Soldier’s Tale,

recounts watching a performance of Vespers, a piece by Alvin Lucier where blindfolded

performers use sonar instruments to audibly navigate around a performance space.

This event retained core elements of live performance but bore little resemblance to a

conventional music recital. The point here is that Collins attributes the groundbreaking

nature of these kinds of works to the abandonment of traditional instruments, “with all

their ‘cultural baggage’ and [the] embrace of new electronic resources [32].” Collin’s

goal, along with others who engage in this approach, is to “disassociate music and

sound from the limited types of objects sold in music stores and, through this

disassociation, prompt new musical discoveries [32].”

Robert Boie developed an early alternative controller at Bell Labs under the

supervision of Max Matthews. The Radiobaton is a percussion-based instrument with

radio transmitters embedded in two mallets. This device broke the mold for alternative

38

music control in many respects. It was capable of detecting the three-dimensional

spatial position of each mallet over a sensor board and the pressure of a mallet’s impact

[26]. Several composers have written extensively for this controller including Max

Matthews and Richard Boulanger.

The prospect for sculpting sound with hands is intriguing, and a class of glove-

based alternative controllers soon emerged. At first experimental and designed only by

individuals for particular projects, the glove controller design eventually fund its way into

the commercial market through the Nintendo® Power-glove and the P5. It is interesting

to note that the success for these interfaces on the market was short-lived. One of the

first documented instances of glove controllers was built at STEIM for composer Michel

Waisvisz. The Hands controller looks nothing like gloves, but rather a set of controls

ergonomically mounted around his hands and wrists. Laetitia Sonami, after having

worked with a pair of dishwashing gloves with magnetic sensors decided to build one of

the first of what is now commonly known as the data glove. Her Lady’s Glove contained:

microswitches for the fingers and flex sensors for the fingers and wrist, a sonar

measuring hand distance from her waist, magnetic sensors and a pressure pad.

A current trend in the field is the use of game controllers as a means to

interactively interface with musical computing. This approach takes advantage of the

gestural skills people develop who regularly engage with video games. Controllers also

offer unique affordances for complex control of many simultaneous elements. There is

also an inherent accessibility (familiarity) for new users to easily engage with sound.

Little to no knowledge of hardware is needed to use these in a project: just plug-and-go.

Some of the more popular controllers used are Nintendo® Wii-motes, perhaps because

39

of the focus on visual gestural elements. Jeffery Stolet has composed several works for

the Wii-mote, treating it as an extended conductors wand. Flight controllers like the

SAITEK X45 also maintain a good compromise between interesting gesture types

visible to the audience and a variety of user control options.

According to CTIA, an international association for the wireless telecom industry,

96% of US citizens own a cell phone, growing from a modest 13% in 1995 [33].

Composers immediately began composing works for these devices, like Dialtones by

Levin and Gibbons where the entire concert is executed through audience participation

through use of their ring tones. Most recently, the family of Apple® iphone/touch/pad

devices along with DROID™ phones has made the use of personal wireless

communication devices a viable gestural controller in music performance. Apps like

TouchOSC by hexler.net facilitate communication of music and sensor data between a

wireless handheld device and computer. Displaying an array of knobs and sliders the

mobile device is transformed into a kind of control surface.

Augmented Instruments

A general reason for a growing desire to break clean from acoustic instruments

and cultural baggage is the existence of an ever-widening rift, with slow-developing

conventional acoustic instruments and performance practice on one side and rapidly

developing technologies with novel modes of interactivity, experimentation, and

expression on the other. Remarkably, the instruments of the symphonic orchestra have

maintained their fundamental features since the mid 1800s despite the accelerating rate

of technological development in other aspects of life [34]. Conventional instrument

fabrication and performance practice represent a thoroughly established tradition that

has naturally evolved and undergone refinements over the course of centuries. The

40

possibilities that emerging technology and electroacoustic music offer to extend the

expressive capacities of traditional acoustic instruments are vast. Proponents for

designing augmented instruments acknowledge and embrace the baggage of traditional

performance practice to extend these well-established traditions into the realm and

possibilities of computers and acousmatic music. There are several approaches to

instrument augmentation and each instance employs a unique combination of

methodologies. These include the use of sensors to collect performance gestures on

the instrument, amplification and realtime analysis of the instrument’s sound (like pitch

tracking), and converting the acoustic instrument into a resonant transducer for

electronic sounds (i.e. the instrument itself is used as a kind of loudspeaker). The

earliest attempts at augmenting conventional instruments sacrificed the original acoustic

sound and functionality (notable examples include Martin Hurni’s Synthophones [35]

and Nicolas Collins’s Trombone-Propelled Electronics [36]). A notable criterion that

distinguishes augmented instruments from instrument-like and instrument-inspired

controllers is that the original acoustic and mechanical function of the instrument is

maintained, preserving and extending traditional performance practice. The following

discussion will focus on augmented instruments within the context of extending

traditional instruments and performance practice.

Some of the earliest examples of instrument augmentation principally employed

amplification (e.g. Robert Ashley’s Wolfman (1964) and Stockhausen’s Stimmung

(1968)). Extending acoustic instruments beyond amplification became possible when

computing became available on-stage with the birth of integrated circuits. Experiments

with indeterminacy and improvisation using electronic circuits gave rise to early

41

interactive electronic music in the 1960s. A notable example involves John Cage, David

Tudor, Marcel Duchamp, David Behrman, Gordon Mumma, and Lowell Cross using a

photo-resistor chessboard on stage in 1968. Playing chess generated electronic sonic

and visual events [37]. The year before, Gordon Mumma applied these capacities to live

instrument performance. Hornpipe (1967), for horn and “cybersonic” interface produced

an interactive music performance between the horn’s sound, its resonance in the

performance space, and the electronic circuitry.

As affordable sensors and microcontrollers emerged on the market in the 1970s

and 1980s, retrofitting conventional acoustic instruments with sensor interfaces became

more accessible than ever. Computers can be programmed to use the sensor data to

influence musical algorithms, signal processing, or other non-musical elements such as

stage lighting.

MIT’s Tod Machover and his hyperinstruments is the first major project to interface

acoustic instruments with computers using microprocessors and sensors while

maintaining the acoustic properties of the original instrument. It began in 1986 with the

goal of designing expanded musical instruments, using technology to give extra power

and finesse to virtuosic performers. Hyperinstruments were designed to augment

guitars, keyboards, percussion, strings, and even conducting. A famous application is

the hypercello (an acoustic cello with sensors and computer-controlled synthesized

sounds) played by Yo-Yo Ma in the early 1990s. The hypercello allows the cellist to

control an extensive array of sounds through performance nuance. Wrist

measurements, bow pressure, position sensors, and left hand fingering position

indicators enable the computer to measure, evaluate, and respond to a variety of

42

aspects of the cellist’s performance. The computer then controlled aspects of the sound

of the cellist’s performance. The sound of the performance was affected by triggering

and modifying the synthesized sounds that accompanied the acoustic cello through the

performers own gestures on the instrument [38].

John Impett’s collaboration with Bert Bongers in 1994 resulted in the Metatrumpet.

The goal was to extend the performance practice inherent with playing the trumpet into

the realm of continuous control and interaction for computer music. His intent was to

build “presence into the structure of production,” in contrast to a performer simply

chasing the tape [39]. Instead of closely acting out a script, one could create musical

situations that allowed the performer to explore and improvise. The trumpet was

interfaced to a computer using a variety of sensors, a STEIM Sensorlab interface, and

pitch-to-midi conversion hardware. The 2-dimensional position of the trumpet was

calculated using sonar receivers attached on both sides and underneath the trumpet.

The player’s physical contact was measured with pressure sensors and mercury

switches. They discovered that breath pressure sensing could not be employed without

compromising the playing technique and the acoustic integrity of the instrument. This

constraint remains true for brass instruments even today. Instead, Impett and Bongers

relied on sound analysis from a microphone to get a ‘loudness’ parameter. All sensor

values were converted to MIDI, relayed to a computer, and then mapped to musical

control variables.

Matthew Burtner took a similar approach to Impett’s, extending the performance

practice of the saxophone using new technology to convert it into an electroacoustic

instrument: the Metasaxophone. His first attempt applied a variation of technology from

43

Gary Scavone at Stanford University and Parry Cook at Princeton to create the MIDI

Saxophone [40]. Using a Parallax Inc. Basic Stamp BIISX microprocessor fixed to the

bell to convert analog sensor data to MIDI information, Burtner attached force-sensitive

resistors (FSRs) to the keys and other areas of the saxophone, triggers, and a 2-d

accelerometer. He designed a software client in MaxMSP to read in the data and control

digital signal processing and synthesis algorithms. His next Metasax version explored

and augmented the natural acoustic characteristics of the instrument. Burtner designed

a 3-microphone amplification system that attaches to almost any location on the

saxophone using flexible tubing. Generally, one mic is designed for a location deep

inside the bell while the others are suspended outside of the horn to pick up the widest

range of frequencies. With MaxMSP, the audio signals can be mapped to modify the

functions of the MIDI data resulting in complex, multifunctional control over parameters.

Burtner mentions “the inseparability of the acoustic saxophone interface from either the

sound or the MIDI control changes… created a highly idiosyncratic but unified controller

[40].”

Curtis Bahn mentions in his paper, “Physicality and Feedback [41],”

Musical performance in a cultural context has always been inextricably linked to the human body, yet, the body has played only a minor role in the creation and performance of electronic music. …We find physicality, feedback, and gesture – the reintegration of the body in electronic music – are all key to maintaining and extending musical/social traditions within a technological context.

To this end, Bahn has worked to use sensing technology to interface traditional

instruments and dancers to computers. One of his well-known achievements is the

SBass (Sensor Bass interface), a 5-string electric bass built by luthier Bill Merchant.

Bahn equipped a small mouse touch-pad under the fingerboard, several slide sensors,

44

FSRs, potentiometers, a 2-d accelerometer, and several extra buttons. A microcontroller

attached to the side of the bass converts all of the sensor data into MIDI information,

which is sent to a computer running MaxMSP [42]. This system gives Bahn sensitive

gestural control of musical parameters while performing on stage with a computer and

other electronics.

Augmentations of the Trombone

Trombonists are a peculiar breed of person, and it is of no coincidence that the

instrument has been subjected to a plethora of experiments as new technologies

emerge. Because the author places particular focus on applying sensor interface

technology to the trombone, it is appropriate to cover a few notable examples.

Nicolas Collins’s Trombone Propelled Electronics (TPE) vastly expanded the

horizon for the technology that could be employed on an acoustic trombone. It is also

interesting to note he has never conventionally played the trombone [36]. He used the

trombone as a point of departure to create a novel controller to explore territory beyond

traditional performance. The first version of TPE began when he interfaced a

Commodore 64 personal computer, keyboard, and monitor with a Stargate digital reverb

unit. By programming the Commodore’s EEPROM, Collins could get the digital

hardware to simulate an array of effects (reverb, time stretch, sample/loop). At a time

when personal computing still took up significant space, George Lewis once mentioned

that Collins was the “first to take a computer on stage – it was smuggled there in the

guise of a trombone [36].” The next step was to incorporate a rotary potentiometer and

a dog leash to measure the length of the slide as it moved in and out. A small keypad

was also attached to the slide, where pressing keys could be mapped onto any musical

parameter. A small speaker driver was attached to the mouthpiece to send the sound of

45

the Stargate into the resonant body of the trombone, turning the instrument into an

acoustic transformer of the digitally processed sounds. Collins performed with this

version till 1994 when it was run over by a taxi. Future versions included a STEIM

Sensorlab, ultrasonic transducers replacing the dog leash, and an Apple® iMac

speaker/amplifier to replace the old driver.

Although mentioned above that Nicolas Collins’s Trombone Propelled Electronics

(TPE) does not necessarily fall into the scope of augmented instruments (having

sacrificed the acoustic integrity of the instrument), it isn’t entirely just a controller either.

There is still an integral acoustic function as the instrument serves as a resonant

transformer of sound.

George Lewis is a trombonist, improviser, and computer music pioneer. Lewis

created Voyager – a non-hierarchical interactive musical environment – at STEIM in

Amsterdam in 1986 [22]. The program interfaces with any instrument capable of

sending MIDI data and parses data for up to two performers (though Voyager

autonomously generates music without the need for any human performer interaction).

In the case of acoustic instruments like Lewis’s trombone, a pitch-to-midi converter is

employed. Up to 64 asynchronous layers, or MIDI instruments, generate some form of

musical behavior. There is an arbitration layer that decides which layers are present and

determine musical characteristics like tempo, pitch density, reverb, etc. The performer’s

MIDI data stream is analyzed and features like average pitch, speed, and tempo are

assigned to control parameters of the arbitrator. So while not interacting on a one-to-

one level, the musicians are affecting the musical behavior of the system on multiple,

complex levels. As opposed to the above examples where sensors are integral for

46

parametric control, this interaction takes place exclusively with sound – there are “no

veto buttons, foot pedals, or physical cues [22].” His approach stands out in that where

others focus on extension of an instrument into the realm of the computer, Lewis’s

approach is an extension of computer into the realm of the human – an autonomous

virtual improviser.

In 2003, Richard Karpen created a music composition for an amplified trombone

equipped with a Photoresistor Slide [43]. In collaboration with Chad Kirby, they

designed a thin, telescoping tube that could be easily attached to the slide that can

measure the length of extension. There is a photo resistor in one end of the enclosed

tube and a light source on the other. The light is angled in such a manner that the

photoresistor receives brighter or dimmer light corresponding to the length the tube is

extended by the slide. This data is converted and sent to a computer to interact with

digital signal processing. The composition was written for Stuart Dempster and called

“Anterior View of an Interior with Reclining Trombonist: the Conservation of Energy.”

The slide data included position, speed of change, and direction of movement and was

mapped to process the sound of the amplified trombone and play back (or “scrub”)

prerecorded speech and choral samples using just the slide.

Neil Farwell composed a trombone work called, “Rouse” in 2006. Commissioned

by trombonist and engineer Hilary Jeffery, Farwell was to expand on a virtual

performance environment controlled by a sensor-equipped trombone mute Jeffery had

already designed called the tromboscillator [44]. The outcome resulted in three

components: the uSlide, an ultrasonic distance finder sensor to measure slide position;

the eMouth, a small Apple® iPod tweeter driver that replaces the trombone’s

47

mouthpiece so that the trombone plays itself; and the eMute, a hand-held loudspeaker

that is used like a plunger mute that can function either as a pickup or can drive sound

into the bell of the instrument to change its acoustic properties [45]. With the uSlide, the

ultrasonic sensor measures time-of-flight, which can be used to accurately calculate

distance (with extremely slight variance based on the temperature of air). The slide data

is mapped to control parameters of the eMouth and eMute. The eMouth expanded

Collin’s TPE to drive the trombone with an actual physical model of buzzing lips. In this

manner, the virtual lips could actuate the sound of the trombone much like the lips of a

human performer. When propped up on stage, the trombone appears to be playing with

no aid of an actual performer. The eMute similarly drives sound into the bell of the

trombone using an acoustic trombone physical model. This changes the muting

characteristics of the horn while playing and also provides tactile feedback that one can

feel in the embouchure. The player can then use this haptic sensitivity to effectively

tease out different pulsations. A USB controller circuit was connected to the back of the

eMute so the performer could switch the function from driver (output) to pickup

(input/amplification), and also turn it on and off.

In the augmented instrument examples above, the underlying goal has been to

create a more causal link between the physical gestures performed on a traditional

instrument and electroacoustic music. As stated earlier, a paradigm shift occurred in the

relationship between sound and its originating source when electricity became a

mediating factor in sound production. The notion of gesture serves as an invaluable link

when reconciling the aesthetics of non-visual ‘acousmatic’ music with the aesthetics of

traditional performance practice. In the following chapter, the concept of musical

48

performance gesture and the notion of gesture in electroacoustic music will serve as a

kind of conceptual base for the significance of the technology presented in Chapter 6.

49

CHAPTER 4 ON MUSICAL GESTURE

Music controllers and the protocol that supports their communication with synthesis algorithms should be founded on a hierarchical structure with the performance gesture, not the score-based note list, as its unit of currency.

– Joseph Paradiso. Current Trends in Electronic Music Interfaces [7]

The literature on music and gesture is an ever-expanding field including the

ongoing work by Cadoz, Wanderley, Paradiso, Payne, and Godøy/Leman [13], [46–50].

Institutions like IRCAM, MIT, and STEIM are continually extending the horizons of

gesture capture technology applied to musical control. Although entire dissertations

could be written on this topic alone, the aim of this chapter is only to pull from the

gesture-sound literature to establish a framework linking non-visual (incorporeal)

acousmatic music with visual (physical) instrumental performance. Ever since sound

and gesture could be mediated by electricity (e.g. any arbitrary gesture may be mapped

to produce any sound), the translational relationships between gestures performed on

stage and the sound heard by an audience has remained a point of major concern in

the field of electroacoustic music. How does one convincingly reconcile acousmatic

music (sound for loudspeakers alone) with the traditions of live performance?

Musical gesture lies at the epicenter of these two domains. Even in the context of

acousmatic music, the listener seeks stable gesture-sound relationships to create

mental visual associations with sound [1]. Only recently have technologies emerged that

allow the empirical capture, analysis, and study of gesture in sufficient detail [50].

Emerging technology is spurring sudden growth in new sensor interfaces and fueling

heightened interest in gesture-sound relationships (such as the XBOX Kinect®

contactless motion sensor transforming the gaming industry). The following discussion

50

offers a conceptual framework behind the technology developed by the author, which is

a proposed step toward facilitating a more intimate relationship between physical music

performance and the non-visual morphological structures of sound.

Sound, Motion, & Effort

The traditional relationship between a performer and their musical instrument is

more than simply manual (i.e. physically holding it). The tactile interface of a musical

instrument is multi-faceted, including breath control, embouchure, key action, lungs,

diaphragm, throat, reverberations (of teeth, fingers, hands, and head), posture,

pressure, tension, a complex feedback loop between the sound an instrument makes

and the musician’s ear, etc [51]. A musician’s entire body seems to vibrate in sympathy

with the instrument to produce sound. This intimacy has long been sought after in

electronically mediated musical instruments.

When one is asked about what technology is, the responses often tend towards

identifying a tool that makes performing a certain task easier or more efficient [52]. Even

the computer is thought to be primarily an effort-saving device. The presence of too

much effort in a system may be perceived as an indicative sign of error or inefficiency.

Minimizing physical effort is a trend that can be observed in the production of electronic

musical instruments as well. Think about the physical conditioning it takes to effectively

play a soprano saxophone or clarinet versus the effort required to play a Yamaha®

WX7. This is not to say that unique flexibility and coordination isn’t developed when

engaging with electronic instruments. The comparison may be analogous to writing with

a pencil versus typing with a keyboard (i.e. what is possible to do with a pencil versus a

keyboard and how long it takes to gain proficiency at these skills). However, effort may

have an integral utility for music instrument design. Similar to Mark Applebaum’s

51

Mousetrap, it may be more interesting for an audience to observe someone contending

with an incredibly complex instrument (making control as difficult as possible) rather

than a rudimentary one. In fact, the notable characteristic of instrument performance (as

well as any spectator sport) may fundamentally be the witnessing of the display of

conditioned, physical effort.

Because of the separation of sound source from the physical interface, designing

systems to emulate intuitive gesture-sound relationships for electronic instruments

(instrument-like, instrument-inspired, and alternative controllers) is not without

complications. This is partially due to the disparity of haptic force-feedback in these

control surfaces. Haptics refers to the sense of touch (tactation) and the sense of

motion and body position (kinesthesia) [53]. While causal and complex haptic

relationships exist between a performer and a conventional musical instrument, touch

feedback is hardly used for electronic instruments. “The feel of a key that plays a

synthesized tone will always be the same irrespective of the properties of the sound

[54].” The sense engaged by electronic musical instruments is primarily auditory and

visual feedback is often displayed on LCD or computer screens. Though it is possible to

artificially produce reaction forces like vibrations within these instruments, doing so in a

convincing manner has yielded persisting difficulties [55]. This is not to say that

ergonomic and expressive alternative controllers have not emerged, but the range of

expressiveness and sensitivity in electronic instruments that mimic acoustic (and

electric) counterparts has not yet reached a sufficient level of sophistication [54].

On the other hand, conventional musical instruments have been greatly refined

over the centuries. The primary motivations have been to increase ranges, accuracy

52

and nuance of sound, not to minimize the physical requirement of effort [52]. Performing

a conventional musical instrument is the act of mapping complex gestural territories

onto a grid of musical parameters (pitch, harmony, articulation, etc). Thus, effort is

closely related to expression. Musical instruments serve as a kind of transducer,

translating every nuance of the performer’s body into a wide range of affective quality in

acoustic sound [56]. This attribute is also what makes musical instruments difficult to

play well. However, the range of expression makes overcoming this difficulty fulfilling for

both the performer and the audience. When compared to the physiological effort

required to play a note on the flute and the sensitivity it affords the performer, a simple

button press seems inadequate for musical control.

To gain proficiency at performing with an instrument (as with learning to write) the

difficulty of the task comes from being forced to use the generic capacities of the human

motor system in order to master an arbitrary gestural code (as with learning to write with

a pencil) [46]. Music performance is fundamentally a conditioned, specialized form of

movement. Musicians move to create sound and listeners often move intuitively in

response (sometimes in the form of mimicking instruments or dancing). This is the

gesture-sound paradigm; experiencing music is inextricably linked to experiencing

movement.

Gesture

There is no one common definition of the word gesture. The use of the word

covers a vast territory depending on its context in different fields including neuroscience,

physiology, computer science, engineering, linguistics, semiotics, etc [46]. The author’s

aim is to focus primarily on the word gesture and its use in music and not to propose an

all-inclusive definition of the word. Thus, it is important to define the scope of the word’s

53

meaning within the context of this document. Based on work from Wanderley and

Battier (2000), two premises should be defined to set the context for the rest of the

chapter [50]:

The word gesture necessarily makes reference to human corporeality and its physiological behavior – whether this behavior be useful or not, significant or meaningless, expressive or inexpressive, conscious or subconscious, intentional or reflexive, applied or not to a physical object, effective or ineffective, or suggested.

Phenomena that are not necessarily produced by human bodily behavior also commonly have the word gesture ascribed to them. Take, for instance, natural movements like flux (wind, waves, sand), falls, crumbling, etc. Gesture, associated to these phenomena carries an anthropomorphic intent. The perception of a gestural sound or gestural contour in sound (i.e. a Mannheim Rocket or Denis Smalley’s sonic object) is such an associative metaphor. Gestures identified in sound itself are human physiological associations the listener ascribes to the stimulus.

In other words, gesture is intimately tied to human physiological experience.

Musical Gesture

Jensenius et al. define “musical gesture” as body movement that is associated

with sounding music [50]. It should be pointed out that this definition might be too

exclusive. The presence of audible music may be sufficient, but not necessary. Sound

may evoke mental images of human gesticulation. However, visual stimuli can also

evoke thoughts of unheard sound. Take for instance Mark Applebaum’s piece Tlön for

three conductors and no musicians. Three people conducting an imaginary piece are

kept in time using headphones and a click track. The gestures are still surprisingly

musical in nature, evocative of an inaudible music, despite the lack of sound in the

room. Leman and Godøy describe musical gesture as being comprised of two levels.

The primary plane of gesture is an extension of the human body (physical movement

itself) while the secondary plane resides with intention (that which is imagined,

54

anticipated, or conveyed) [50]. Thus, “musical gesture” is a synergy of movement (the

displacement of an object in space) and meaning (a communicative message from

sender to recipient) relating to sound (audible or not).

From a listener’s perspective, sound itself may be gestural or have encoded within

it notions of physicality (e.g the galloping sensation of the William Tell Overture). A

study by Shove and Repp (1995) suggests that music is essentially an audible

embodiment of motion. “The listener does not merely hear the sound of a galloping

horse or bowing violinist; rather, the listener hears a horse galloping and a violinist

bowing [57].” Denis Smalley goes further to say that the action of instrumental and vocal

performance defines what music essentially is for most people [1]. From this

perspective, the perception of music with the impression of physical motion is

inextricably connected because of our cultural and social experiences, a kind of

generalized stimulus-response.

Instrumental Gesture

Instrumental gesture is musical gesture viewed specifically within the context of

performing with musical instruments. Wanderley (2000) and Cadoz (1994) describe

instrumental gesture as comprised of three interrelated components [46]:

ERGOTIC. Referring to material action, modification, and transformation. No communicative information is encoded.

EPISTEMIC. One’s capacity to touch, maintain contact with, perceive, and hone. Notions of virtuosity and effort reside within this plane.

SEMIOTIC. The communication of information, meaning, and intent.

For instrumental gesture, all three functions coexist interdependently. Exertion is

applied to a physical object, specific phenomena are produced whose forms and

55

dynamic evolution can be mastered by the subject, and these phenomena support a

communicative function.

Similarly, Delalande (1988) posits that when a musician performs, four

instrumental gesture types may be identified. However, it is important to note that any

instrumental gesture may fit equally well into several categories [50]:

SOUND-PRODUCING. Gestures that mechanically produce sound serve to either excite a new sound or to modify a sustained one.

COMMUNICATIVE. Also referred to as semiotic gestures, meaning is encoded and intended as indicative cues to another performer or audience member. This includes foot tapping or bobbing of the head, shoulders or instrument to indicate tempo, etc.

SOUND-FACILITATING OR ANCILLARY. Gestures not directly involved in sound production that still affect the sound to some degree. For the case of a trombone player, the visible inhalation, expansion of the abdomen, the posturing of the shoulders and puckering of the embouchure all influence the moment leading up to the buzzing of the lips to create sound. The phrasing-gesture is another type of sound-facilitating motor activity. Wanderley has shown that these movements are consistent and reproducible even over long periods of time [58]. Research conducted by Campbell et al. and Quek et al. studying ancillary gestures in clarinetists show these reproducible motor patterns are integrally connected to the performance of musical phrases and are often related to the movement of the clarinet bell [50]. These gestures may also function as communicative signals, enhancing a perceiver’s experience of the phrasing of sound and are especially displayed (and even exaggerated) amongst virtuoso performers. However, the gestures still persist (albeit less) when a performer is asked to play immobilized, suggesting these gestures are tied to shaping sound and are fundamentally separate from gestures intended purely for communicative purposes [58].

SOUND-ACCOMPANYING. Gestures that have no role in sound production or modification. They follow the contour of the music and cover the gamut from complex choreography to simple fist pumping or head bobbing.

Spectro-morphology

In 1986, Denis Smalley designated the term “spectro-morphology” to describe the

field of acousmatic composition. Comprised of four primary components – spectrum

typology, morphology, motion, and structuring process – this laid the theoretical

56

framework to describe morphological developments in sound spectra over time. His

writing on the gestural nature of sound is integral to understanding the complex

interrelationships between performance gesture and morphology of sound. The concept

of acousmatic music engages deeply with reinforcing or confounding visual associations

between sound and its source. In this sense, acousmatic music is a highly “visual”

music. Despite the lack of visual stimuli, the listener’s imagination proactively and

subjectively attempts to fill in the missing information based on gestural and

morphological cues heard in the sound.

With respect to instrumental performance, Trevor Wishart posits that the

morphology of human physiological gesture may be directly translated into the

morphology of sound through the use of the vocal apparatus or some instrumental

transducer [56]. It is the instrument itself that forms an interesting barrier in the

translation from human performance gesture into the morphology of a sound. In this

sense, Wishart posits that the vocal apparatus is the most sensitive gesture-to-sound

transducer, capable of complex timbre, amplitude, frequency modulation and

articulations. When human utterance is heard in the context of acousmatic music it is an

unmistakably human gesture. Similarly, all wind instruments (dependent on the

continuous breathing of the performer) and bowed instruments (where sound is

produced through continuous physiological action) are also gesturally sensitive

transducers. The ability to continuously modify sound over time is integral for Wishart’s

conception of a gesturally sensitive transducer. Percussive instruments like drums and

pianos, are viewed as less gesturally sensitive due to the fact that they lack the ability to

modify a sound after excitation.

57

Gesture & Paralanguage

Expressivo, Cantable, Molto Agitato. These are arbitrary symbols a composer

purposely marks on a score, placing the burden of interpretation on the performer. Most

audience members are aware of and can appreciate a performer who can make a

single note sound urgent, relaxed, happy, mournful, or regal. Music, like all arts, has a

semiotic goal: communication from artist to audience. As Ray Kurzweil mentions, “this

communication is not of unadorned data, but of the more important items in the

phenomenological garden: feelings, ideas, experiences, longings [27].”

In this sense, the communicative/semiotic plane of instrumental gesture and

morphology in spectro-morphology may be seen as serving an analogous purpose.

They both function as the contextual paralanguage that the listener uses to interpret an

experience when faced with sound. This idea is essential to the definition of gesture.

Even though the notions of non-visual acousmatic music versus the traditions of live

instrumental performance seem diametrically opposed, the paralanguage of musical

gesture can serve as a common territory that causally relates the two. What may have

been a translational disparity between acousmatic sound and physical performance

gesture may instead be an opportunity to explore the gestural territory between these

two domains. However, current musical protocols like MIDI encode almost exclusively

sound-producing gestures (pitch, onset, velocity, duration, etc), not communicative

gestures. The following chapter will detail the current hardware and issues central to

interfacing musical instruments with acousmatic sound.

58

CHAPTER 5 INTERFACING WITH THE ACOUSMATIC.

What was lost in [early] digital technology was the gestural, tactile immediacy of the analog world, so here you have this incredible computing power and just a mouse to control it with.

– Peter Otto. Electric Sound [10]

Sensor Interfaces

Traditional acoustic instruments are often employed in compositions for live

electroacoustic performance. Historically, sonic media (phonograph, magnetic tape)

was ‘fixed,’ with few exceptions, and the burden largely remained on the musician to

maintain synchronicity. In this manner, the musician could be influenced by the

electroacoustic sound, but not the other way around. Once the medium shifted to a

malleable digital data stream and computing became available on-stage, real-time

interactivity between musicians and computers for live performance became a

possibility. Hardware was soon developed that enabled musicians to influence

electroacoustic sound in a number of ways (e.g. sensing performance gestures or pitch

tracking) [59]. This chapter’s focus is on the use of sensor interfaces to capture and

map performance gesture onto interactive sound. However, this is far from the only

possible method to achieve interactivity with electroacoustic music.

Before market-accessible microprocessors, composers and instrument designers

often reappropriated computers and sensors from other hardware to attach to their

instruments. Some examples mentioned in previous chapters include Gordon Mumma’s

Cybersonics and Nicholas Collins’s Trombone Propelled Electronics. By the mid-1980s,

microcontrollers with sufficient peripherals became available (Motorola’s HC-11 for

example) allowing instrument designers to more readily design custom hardware. In

59

1983 the MIDI protocol was officially established as the industry standard for musical

data, and by 1989 STEIM had developed the Sensor-Lab (a portable microcomputer

that converts sensor voltages into MIDI data). These developments enabled musicians

to bypass a significant learning curve in hardware design and created an environment

ripe for new alternative musical controllers to emerge. For the first time, the interactive

landscape between music performance gesture and electroacoustic sound became

accessible to a wider community of musicians, including those with only moderate levels

of engineering experience, who could now purchase, program, solder, and construct a

unique interface to meet the aesthetic demands of such projects.

Performers today have more options to control sound than ever before and need

not limit themselves to the chromatic piano keyboard or the early digital computer-

keypad and mouse paradigm. As for approaches with sensor interface design,

individuals fall along a spectrum ranging from DIY (do-it-yourself) to out-of-the-box

solutions. In most cases, an approach restricted to either extreme is impractical for

broad application. The complex and economical nature of current technology leads

even the most advanced DIY hobbyist to start from off-the-shelf components at some

point (e.g. purchasing microchips, integrated circuits, and other common electrical

components). Conversely, many currently available out-of-the-box interfaces, like game

controllers, fail to meet the requirements of each unique project without modification.

The appeal for DIY is the seemingly endless options to design custom built

projects and the flexibility to alter designs as necessary. The hobbyist market is an ever-

expanding field with thorough documentation, tutorials, and sample code for almost any

design issue. The components often include microcontrollers (i.e. Arduino, Paralax,

60

Propeller, BasicStamp, Teensy, etc.), embedded programmers, sensors, serial

interfaces, and wireless transceivers (zigbee, xbee, IR, and RF). The downside (just like

crafting anything from scratch such as jewelry or clothes) is that it is often more

expensive to buy individual components and requires considerable more time (and risk)

to assemble the components by hand.

Alternatively, various out-of-the-box solutions have the advantage of mass

production (lowering cost) and can save substantial development time. These benefit

musicians and composers because the vast knowledge required for designing and

programming the hardware can be bypassed to a certain degree. However, one is still

often required to understand minimal circuit design and programming. This approach

also benefits those who have the requisite engineering background, allowing more time

for prototyping, testing, and assembly. Exhaustive lists of available sensor interfaces

with manufacturers, specifications, and prices already exist including [6], [60], but a few

notable systems worth mentioning here include: Teabox by Electrotap, MIDIsense

(LadyAda), MicroSystem (I-cubeX), Phidgets, Eobody (Eowave), Arduino, and Gluion.

Figure 5-1 details the technical specifications for the above-mentioned interfaces as of

2011.

Figure 5-1. Market-available wired sensor interfaces and their specifications as of 2011.

61

These systems generally consist of a single, pre-programmed microprocessor to

convert a given number of analog sensor inputs into digital data. The processor then

compiles the sensor data using any number of protocols (e.g. MIDI, OSC, etc) and

sends it to a computer via USB, MIDI, or other wired serial connection. Generally, these

devices offer quick and easy sensor interface solutions with minimal hardware

programming. Many versions also have General Purpose I/O pins and analog outputs to

control motors, lights, and other devices. However, applications involving these systems

require at least one wire tether from the interface to a computer. While this may not be

an issue for installation art, this is not ideal for a musician (or dancer) who desires to

move freely. Another challenge is the size of the hardware unit itself and the cabling

between it and the many sensors, making this setup obtrusive to the performer. In some

cases visible cables may be a desired visual component to a project’s aesthetic, but too

many wires run the risk of accidental disconnection of sensors, shifting the instrument’s

center of gravity, or hindering the performer’s full range of motion [8]. Having to alter

ones performance technique to contend with obtrusive technology has fueled the

development of wireless-based sensor interfaces. However, these devices do not

include the sensors themselves. This may require the user to purchase them

individually (significantly increasing cost) or design their own.

Related Work: Wireless Sensor Interfaces

Hardware for wireless communication and the protocols that facilitate it have

become increasingly assessable in recent years (e.g. Bluetooth, 802.11, UDP, etc).

Many companies that design “wired” sensor interfaces also provide “wireless” versions

of their systems at significantly greater cost. A few examples of these include: IRCAM’s

WiSe Box, Wi-microDig (I-cubeX), Sensemble (MIT), MIDITron-Wireless, EcoMotes,

62

and La Kitchen (IRCAM - discontinued). Specifications, as of 2011, are shown in Figure

5-2.

Figure 5-2. Market-available wireless sensor interfaces and specifications as of 2011.

The primary advantage of these units is the severing of the umbilical tether

between the sensor interface and the computer, allowing the musician or dancer to

move freely. However there are disadvantages associated with the currently available

wireless systems as well. For example, cable is still required to connect the sensor

circuits to the central processor, the hardware of many systems is too sizable to be

inconspicuous or light and while wireless units may be worn as a belt pack, they do not

easily attach to an instrument. Therefore, the risk of disconnected sensors and

restricted spatial freedom of the performer may still persist.

Current Trends

As mentioned in Chapter 1, three emergent issues endure across the full range of

market-available sensor interface solutions introduced over the last four decades:

single-serving innovations, lack of transparency, and limited accessibility. While a single

solution may not exist to address these complex matters, bringing the technology to a

usable and accessible state for classically trained musicians would be an important step

in the right direction.

63

Single-Serving Innovations

Charles Dodge mentions, “One of the impediments to writing enduring computer

music is the relatively short lifetime of most computer and electronic instruments [59].”

Consider the life cycle of instruments and musical repertoire before electronic music

(sometimes spanning centuries) compared to the life of the average computer operating

system. The lifetime of the average augmented instrument is shorter still, often

undergoing significant alterations from one project to the next. Furthermore, each

sensor interface developed using the hardware described above invariably becomes

one-of-a-kind and idiosyncratic to individual aesthetics and projects. Based on the

literature for specific augmented instruments – Tod Machover’s Hyper Cello, Ben Neill’s

Mutant Trumpet, and Matthew Burtner’s Metasax to name a few – it seems these

instruments are not only technological extensions of the originals, but are specific to the

composer/performer for whom it is intended. For example, Wanderley and Orio posit,

“extended instruments have been designed to fit the idiosyncratic needs of performers

and composers, but as such they have usually remained inextricably tied to their

creators [3].” This staggering variety has made it difficult for those in the field of HCI

(human-computer interactivity) to evaluate just how effective one method is compared

to another. The particular innovative developments that these augmented instruments

employ are commonly disseminated only in a limited number of instances, namely

conference presentations. This single-serving approach to instrument development is

not necessarily counter-productive. On the contrary, individual experimentation is a

catalyst for a diversity of solutions. This trend may even usher in a new era of

personally unique musical instruments designed to evolve and adapt to the individual.

However, the ad-hoc and volatile methodologies may also be a contributing factor in the

64

general failure to proliferate such innovations and performance practice to the broader

community.

One solution may be to modularize the intricacies of instrument augmentation into

more manageable ‘blocks.’ This often happens as technology, code, language, etc

develops in complexity and may stem from how the human mind handles information

and learning. The cognitive process known as “chunking” is a means of assembling

multi-step processes into a single routine (like waving hello or tying a shoe) [61]. One

can then assemble and combine these chunks to perform tasks of greater complexity

(which may subsequently be “chunked” and so on). Technology evolves in this manner

and can be observed throughout history. Take for example the automotive industry’s

move towards the assembly line, or the IKEA® approach to furniture design (quality

notwithstanding, it allows one to build a complex office desk in your living room without

a wood shop or experience). In electronic music history, sound engineers like Bebe

Barron and Raymond Scott had to design custom circuits to address project-specific

issues – a worthwhile yet inefficient mode of working that was latter solved when Robert

Moog proposed a modular approach to sound synthesis hardware. Miller Puckette’s

Max and the more recent Scratch from MIT are programming environments that take a

modular approach to code, where users can quickly program complex games and other

media by interconnecting blocks or objects. If applied to sensor interface design, this

may preserve the flexibility for individuals to assemble sensor blocks to meet unique

aesthetic demands, while reducing the complexity of lower-level design methods. This

approach could also aid the evaluation of these methods by observing how efficient

each sensor is at addressing specific tasks and noting common trends that emerge

65

amongst a community of users. For example, if one observes a significant demand for

placing accelerometer blocks on guitars, perhaps one day, guitar manufacturers may

integrate them into the standard design.

Transparency

As mentioned previously, the display of wires may well lend itself to a technically

appealing visual aesthetic. The tradeoff is the need for a performer to significantly alter

their playing style to contend with shifting weight, decreased flexibility, and the risk of

disconnecting sensors. Thus, wires are still meddlesome factors even for ‘wireless’

sensor interfaces. It would be of great value to eliminate the need for wires entirely.

Systems have been developed that utilize the architecture of wireless mesh

networking. A wireless transmitter is used for each individual sensor in the network

whereby all sensors can wirelessly communicate to a central receiver, or each other.

For systems like FireFly developed at Carnegie Mellon University, each node comes

preprogrammed to transmit light, audio, temperature, humidity, and acceleration [62].

Several of these can be deployed to monitor areas of an environment. The

unprecedented flexibility these systems offer is conducive for industrial, military, swarm

robotics, and environmental applications. However, a system has yet to be designed

specifically for music performance applications such as augmenting acoustic

instruments.

Accessibility

Despite (or perhaps as a result of) the phenomenal progress of technology, the

knowledge required to design musical sensor interfaces spans a territory well beyond

that of music making. New interfaces are designed almost exclusively by persons who

embody a unique combination of backgrounds including circuit design, digital systems,

66

wireless networks, data protocols, microcontroller and computer programming, as well

as having some degree of musicianship. A classically trained musician with the desire to

engage with these paradigms must overcome a significant technical learning curve.

Furthermore, the time spent at the workbench tediously developing hardware extends

the latency between the initial musical inspiration and the final product, greatly affecting

the creative process.

The following analogy may help illustrate the issue: CTIA wireless association

reports that 80% of U.S. teenagers own a cell phone [63]. Yet none of them have the

capacity to build and program one completely from scratch. This fact does not prevent

teens from effectively using the hardware, often out-performing adults (even those who

know how to fabricate or program phones) in common tasks like texting. This begs the

question – with the technology at our fingertips, why is it necessary for musicians to

design their own hardware to accomplish even rudimentary interaction with computers?

The problem extends beyond acquiring a level of technical facility. When one

purchases components one-by-one for specific projects, the cost quickly adds up.

Often, what is made with the components is particular to a specific project. When a new

project is begun, more of the same components are purchased again. A system

designed specifically to be modular and reconfigurable by the user would solve many of

these issues, but has yet to emerge.

The Dysfunctions of MIDI

The very popularity of MIDI based systems testifies to their utility and widespread use… MIDI is great. MIDI is good. Now let us examine what is wrong with it.

- F. Richard Moore. The Dysfunctions of MIDI [29]

67

Market-available sensor interfaces often communicate to a computer using a

musical data protocol like MIDI. The advantage is the hardware’s ability to connect

directly to any other MIDI device. However, nearly all of these interfaces are connected

directly to a computer. Thus, the user is also forced to purchase a MIDI interface,

increasing the complexity and total price of the system. It is also important to note that

for the tables listed in this chapter, the resolution column in each table refers to the bit-

depth of the A/D hardware only (the process where the analog sensor voltage is

converted into a digital representation). However, it is the data protocol that determines

the ultimate bit resolution when data reaches the computer. For instance, MIDI only

allows a 7-bit resolution for control data (a range of 0-127). Most A/D hardware offers at

least 10-bit resolution (a range from 0-1023). The MIDI protocol is a kind of bottleneck

at the end of the data pipeline. MIDI was established as a standard partially for its low

cost, but at the cost of speed and data transfer. While proven effective for studio

situations, its usefulness is limited for many types of music and performance methods

[59].

The pitch/velocity/duration-oriented nature of MIDI is almost exclusively a

byproduct of piano keyboard based control interfaces that have traditionally dominated

the industry [29]. Furthermore, each data parameter is independent from one another.

This may be appealing for the engineer or composer desiring unprecedented control.

However, the parameters of acoustic musical instruments are interdependent. For

example, a saxophone player can change pitch and timbre not only by depressing keys,

but also with air pressure and bending the embouchure (even environmental conditions,

such as the air temperature of the room, might be considered). Other recent protocols

68

including OSC and raw USB/serial are gaining momentum, addressing some of these

problems by offering higher resolution, lower latency, and better control.

Of the four types of musical gesture (sound-producing, semiotic, ancillary, and

accompanying) suggested by Delalande (discussed in Chapter 4) only sound-producing

gestures are reliably encoded with the MIDI protocol [50], limiting the gestural exchange

that takes place between the performer, musical instrument, and the audience to one-

fourth its potential. The context, nuance, and semiotics encoded in the other three

gesture types (communicative, ancillary, and sound-accompanying) are completely “lost

in translation.” What remains is a sterile stream known as the MIDI event list.

To be fair, a portion of the gesture-sound disconnect in musical data does not

originate with the limitations of MIDI itself, but from the context of the system that came

before: Western musical notation. As with MIDI, the three salient features of this

notational tradition (pitch, velocity/amplitude, and duration) do not inherently prescribe

gesture. Trevor Wishart notes that, “gestural structure is the most immediate and yet

notationally the most elusive aspect of musical notation [56].” While this rings especially

true for Western notation, it should be acknowledged that there are forms of

contemporary notation that do engage with effective use of gesture like that of Helmut

Lachenmann. Why then has Western notation ‘worked’ over the course of its history

while MIDI remains problematic for many live performance settings?

The critical difference between these two systems occurs during the phase when

information is sonified. It is expected (under conventional notions) that a human

performer will interpret a musical score. Here, the performer fills in a great deal of

missing information based on previous experience, training, performance practice, and

69

creativity. The continuation of this practice suggests at least some subjective degree of

success. Therefore a perfect analogy might imply that musical data (MIDI events) will be

interpreted by a computer or other electronic device in the same way. However, under

most circumstances, the device instead expects that the data provided is complete, and

that there is nothing more to fill in. The fundamental fault might not lie with the

cartographer (the one who creates the musical score in western notation or MIDI), but

rather with the interpretive abilities of the map-reader. Incredible work is being

accomplished in neural networks and machine learning, and perhaps this point will

someday become moot when computers, like people, can bring previous experiences,

training, and ‘creativity’ to score sonification. In the mean time, it would be useful for

musical data protocols to reach beyond the limitations of MIDI to also include the

communicative/semiotic, ancillary, and accompanying gesture information to create

more sensitive relationships between musicians and computers.

Moore states, “For subtle musical control to be possible, an instrument must

respond in consistent ways that are well matched to the psycho-physiological

capabilities of highly practiced performers [29].” The performer must receive consistent

aural and tactile feedback from a musical instrument – otherwise the instrumentalist has

no hope of learning how to perform on it expressively. One can turn this statement on its

head to say that computers must receive consistent aural and tactile (gestural) feedback

from the performer/instrument, or else there is no hope of translating critical gestural

nuance into sound.

70

CHAPTER 6 EMOTION: RECONFIGURABLE MODULAR WIRELESS SENSOR INTERFACES FOR

MUSICAL INSTRUMENTS

Any sufficiently advanced technology is indistinguishable from magic.

– Arthur C. Clarke. Profiles of the Future [64]

Overall Design Philosophy

The eMotion system is an all-purpose wireless sensor mesh network that a user

can implement to quickly and intuitively assemble wireless sensor nodes for

control/interaction of computer processes including sound and visual effects. The

novelty of this system with regards to all other human/computer interfaces on the

market resides in the network architecture. Each embodiment of this network is

distinctive from user-to-user depending on the unique aesthetic and technical demands

of the individual or project. Although sensor configurations and mapping may vary

widely from one use to the next, the core hardware/software itself need not be

redesigned. The purpose of this chapter is to provide a technical description of the

hardware and software developed for this dissertation.

The design philosophy adheres to four general principles. The system must be

transparent, reversible, reconfigurable, and modular.

Figure 6-1 illustrates the overall system flow. There are three major hardware

components to the system: a computer running a software client, a central hub receiver

connected to the computer, and a collection of wireless sensor nodes. The user

acquires a set of sensor nodes and places them on the desired sensing locations (an

The content of this chapter is protected by U.S. patent-pending.

71

instrument, dancer, environment, etc.) depending on their aesthetic and technical

needs.

Figure 6-1. System Overview. The system flow (from left to right): Wireless nodes, each with specialized sensors are placed by the user at the desired sensing locations (instrument, body, or environment). A central hub receives all node transmissions and echoes the raw packets to the attached computer. A software client allows the user to re-assign IDs, calibrate, and apply processing effects to the incoming streams of sensor data. The client also allows the user to graphically map desired data streams to control musical or visual parameters during performance and convert the data into musical protocols like MIDI and OSC.

Nodes independently transmit data wirelessly to the central hub. Next, the hub

parses the data streams based on an addressing protocol developed by the author and

sends the streams via USB to its attached computer. Then, a software client on the

computer enables the user to process, mix, and map the data streams to interact with

computer processes. The computer software client allows the user to convert sensor

data into industry-standard MIDI and OSC protocols, or web based UDP. This enables

the user to interact with virtually any hardware or software regardless of location. Each

component of the system will be broken down and explained below.

Node 1

Ex. Sonar

Node 2

Ex. FSR

Node 3

Ex. IMU

Computer

Mapping > Processing

Base Hub

72

Sensor Nodes

Figure 6-2. Sensor Nodes. Each sensor node consists of a single sensor type, an ADC,

a processor, flash memory, power source, and 2.4 GHz wireless RF transceiver. The data packet transmitted by each node consists of a list of sensor values and a unique node ID.

Each node transmits only the data of its attached sensor. For example, one node

may be dedicated to transmitting sonar data and another node may be dedicated to

measuring ambient light. The user is then free to utilize one, both, or neither of these

sensor nodes by turning them on or off – attaching or detaching. Multiple sensor nodes

of the same sensor type may also be used (for example, a sonar matrix array comprised

of a group of sonar nodes may be constructed). In this manner, one can acquire a

collection of general sensor nodes and use a subset of them in any combination as

needed for a particular application. When one sensor is turned off (in the event of

battery depletion or user preference) the other sensors on the network remain

unaffected. The nature of this modular interface design allows the user to easily

reconfigure the placement and combination of individual sensor nodes for any given

project and instrument. This enables the user to find the most optimized sensor

combination, sensing locations, and mapping scheme to meet their unique aesthetic

goals – without having to redesign or reprogram the hardware itself. The sensor nodes

73

may even be distributed amongst several instruments or non-instrumental performers,

such as dancers to create novel modes of interactivity. For instance, a dancer may

control the spectral filtering of a flute player through their body movements.

Each sensor node consists of a single sensor type, an ADC, a processor, flash

memory, power source, and a 2.4 GHz wireless RF transceiver. Each node also

includes a red and a green status LED to provide the user with visual feedback of the

transmission stream. For each data packet the node transmits, the red LED is

configured to blink if it is out of range of the receiver or does not get an indication that

the receiver is on. The green LED blinks when the sensor node is within range of a

receiver and working properly. The wireless data packet transmitted by each node

consists of data-byte pairs: the unique node ID and sensor value. The ID byte is hard-

coded into each sensor and will be presented in further detail below.

Fulfilling the other three design philosophies, the nodes themselves are:

TRANSPARENT. The physical size of each node is exceptionally small compared to similar devices on the market (the largest eMotion prototype node is 1.26 inches in diameter). Because each node is wireless, the usual tangle of wires on the instrument or body can be eliminated completely. This transparency is intended to mitigate the burden on the performer, allowing him to perform as conventionally or dynamically as he would have prior to the modification.

REVERSIBLE. In order to avoid mechanical and other non-reversible alterations to musical instruments or performers utilizing the wireless sensor network, the sensor nodes can be temporarily fixed to an instrument, body, or other surface in a noninvasive manner. “Hook-loop” fasteners and removable adhesives were used for particular implementations of this prototype system, which enables the user to easily revert back to their original acoustic instrument or physical self.

RECONFIGURABLE. The user may rearrange the sensor placement and combination at any time (even during live performance) and switch sensors on and off with no adverse effects to the overall system.

74

Node Design Considerations

The design considerations for the sensor nodes evolved over time as the author

experimented with various methods for programming the embedded devices. Originally,

the aim was to develop intelligent or “smart” sensor nodes, meaning that a number of

sophisticated processes would be embedded in each device (e.g. dynamic sensor

calibration, peer-to-peer networking, dynamic ID assignment, filtering/integration of

data, etc). After experimentation, the author opted to streamline the sensor nodes and

strip down the program logic to only the bare essentials, placing the major computations

(like auto-calibration and filtering) inside of the computer’s software client where power

consumption and processor speeds were not an issue.

The advantages of stripped-down sensor nodes are vast. Fewer computations per

cycle means less power consumption with faster data transmit cycles. The nodes

communicate only with the hub transceiver and do not communicate with each other

(although the modules may be configured for peer-to-peer communication in the future)

simplifying the protocol. The addressing scheme changed from dynamic to static,

requiring less initialization logic and ensuring that nodes appear to be the same device

from one use to the next. This is the equivalent of using a hard-coded MAC address

instead of a dynamic IP.

Another feature added was a “discover” button. When multiple data hubs are in

use, how does a node know which hub it should transmit to? When a hub is fabricated,

it gets a hard-coded random RF channel. The nodes also come programmed with a

factory default channel. The user can press a pin-hole button on the nodes, which

causes it to scan RF channels to synchronize with the closest hub. Once synchronized,

the correct channel is saved in memory for the next time the node is turned on.

75

Node Types

Because each node is dedicated to transmitting the data of its attached sensor,

each node type varies in the number of analog inputs, transmit rates, and number of

data streams, and it may incorporate an i2c interface (a common 2-wire multi-master

serial bus) in addition to analog inputs.

GENA_1: General Analog, one input/output. This node type has a sensor with one analog sensor channel. The node is configured to transmit the data of the embedded sensor at a rate of up to 1kHz (1,000 times per second). This is separate from the radio transmit rate which transmits data at a speed of 1 megabyte per second. To save battery power and bandwidth on the network, this node type transmits new data only when the sensor value has changed. This ensures that the network does not get bogged down and the nodes broadcast only when necessary.

GENA_6: General Analog, six inputs/outputs. This node type is similar to GENA_1, except that it handles up to six analog sensor inputs like 3-axis accelerometers and gyroscopes. In this case, if one value changes on any channel, all values are transmitted on the network. This maintains synchronicity for all axes of sensor measurement.

IMU: Inertial Measurement Unit, six inputs/outputs. This node type is equipped with a single chip that contains a 3-axis accelerometer and 3-axis gyro to measure inertial acceleration and tilt. A special kind of Direction Cosine Matrix (DCM) filtering [65] can be implemented in a computer’s software client to determine the 3-dimensional orientation of an object. Although the DCM algorithms may easily be implemented on the node’s microprocessor, it was determined that only raw values are transmitted by the device. Advantages include less calculations for the node, resulting in higher transmit rates and significantly lower power consumption. A secondary advantage is the ability to painlessly update filtering algorithms and firmware when needed on the software client without updating the hardware itself. However, the IMU works on a principle known as dead reckoning in that this unit only measures displacement from a specified origin. The minor errors in the filtering eventually compound, which results in wandering for the Yaw axis. Also known as “heading,” Yaw is the Euler angle that corresponds to compass degrees on a flat plane. The unit is still exceptionally useful for orientation sensing and cheaper to produce than the more elegant AHRS node described below. Transmissions are at 50Hz and run on an internal timer interrupt, which is the frequency of update the DCM filter expects on the software client.

AHRS: A full, low-power, attitude-heading reference system. This builds on the IMU structure to further include an i2c 3-axis magnetometer and outputs an additional 3 values: mag_x, mag_y, mag_z. This adds three-dimensional magnetic

76

orientation to the filter, which eliminates wandering inherent in the IMU. The wandering is eliminated due to the fact that magnetic orientation is a kind of discrete data (i.e. geographic magnetic variations and hard iron interference aside, magnetic coordinates are roughly dependable and repeatable). Adding discrete values to the relative accelerometer and gyroscope values eliminates the wandering inherent in a dead-reckoning system. The cost to produce this unit is slightly more, but a reliable Yaw axis is invaluable for many implementations. The transmit rate is 50Hz (the frequency of the filter update on the software client). The result is a wander-free three-dimensional representation of an objects orientation at the sensing point. At the time of the writing of this document, the AHRS described above is significantly more cost-effective (on the order of 60%) and user-friendly than other similar AHRS devices available on the market.

The MCU

The selection of a microcontroller was dictated by (in order of importance) power

consumption, required peripherals, cost, bit-depth, clock speed, and memory. After

working with Texas Instruments’ line of DSP processors, and investigating their other

line of ultra-low power devices, the author selected the MSP430 F2272 Ultra-low power

MCU as the central controller. The data sheet with specifications is located on the

Texas Instruments™ website [66].

Addressing Protocol

Due to the modular nature of the sensor network, where individual nodes may be

added, removed, or reorganized at the user’s discretion, the author had to devise

addressing protocol so the client could reliably handle and reconfigure the dynamic

node architecture. The original intent was to have the client dynamically assign an ID

number to each node based on the order in which they are turned on (this scheme is

used in many popular wireless controller systems for gaming). The problem with this

method is repeatability.

Many users will likely want to preserve a specific configuration and mapping of

sensor nodes after deriving the desired setup. Order-based ID assignment forces the

77

user to turn on the devices in a particular order each time. This may be acceptable

when turning on four game controllers, but can be aggravatingly tedious when dealing

with multiple sensor nodes. Therefore a more ideal structure involves devising a “hard-

code” or unique serial ID for each node (a kind of MAC address) where a software client

may recognize each node’s explicit identity. The six-bit-long Instance ID is generated at

compile time (the time the processor is programmed) during the fabrication stage of the

hardware. It is a permanent number that is transmitted each time the node sends a

packet of data and essentially tags each incoming data stream to its specific device

origin. The Nordic nRF24L01+ RF radios are capable of receiving six simultaneous data

streams called “pipes.” Each pipe is reserved for particular sensor types. Thus, the

receiver is capable of recognizing up to 64 possible instances for each of its six different

sensor types. The protocol can be easily expanded in firmware to extend the addressing

scheme to include more devices in the future. The hub and software client’s ability to

determine the sensor type and the unique instance of each node of the network allows

the user to preserve configurations and mapping for specific nodes.

In this manner, data is sent from each node to the receiver in address/data byte

pairs: aaaa aadd – dddd dddd, aaaa aadd – dddd dddd, … where “a” indicates the

Instance ID bits and “d” indicates data bits. The two most significant data bits are

integrated into the address byte space, allowing 10-bit resolution for sensor values (0-

1023). This is a vast improvement from MIDI’s 7-bit resolution (0-127).

Radio Specifications

Three kinds of market-available, wireless transceiver hardware protocols were

considered: RF, Bluetooth®, and XBee®/ZigBee®. An exhaustive description of each

protocol is beyond the scope of this document. However, there are vast resources

78

available that detail the history and specifications of each protocol [67–69]. The author

explicitly chose to use RF (radio frequency) transceivers over other available market

options for two primary reasons: power consumption and range. Although streaming

audio was not a primary goal, the author selected a radio that would still have the

capacity to stream PCM standard audio rates for future upgrades. A standard

requirement across all transceivers was a standard operating voltage of 3.3V (the same

levels as the node microprocessor). Additionally, all transceivers support point-to-point,

point-to-multipoint, and peer-to-peer network configurations and reside within the

2.4GHz ISM (Industrial, Scientific, Medical) short-range frequency band - a commonly

used frequency for wireless equipment including cordless phones, microwave ovens,

and wireless LAN. Although the 2.4 GHz band is utilized by a wide range of devices,

cross-talk and interference between these devices can be mitigated through Address

Masking, which creates unique addressing protocols that pick out only the intended

device. The transceivers shown in the figure below all have special protocols that

address the risk of interference, including channel hopping, auto-acknowledge, and

address masking. Note that the specifications in the list are taken from the device-

specific datasheets and often do not reflect actual values in testing situations [68], [70].

For budgetary reasons, the author tested the nRF24L01+ and TI CC2500 devices only.

The rest of the specifications should be used for reference purposes only.

It should be noted that some XBee® and Bluetooth® modules include ADC and

GPIO hardware, eliminating the need for a separate processor, making these devices

extremely appealing. The ultimate decision to select the Nordic transceiver came down

79

to power consumption and transmit range – even with the requirement for an external

processor compared to Bluetooth.

Figure 6-3. A comparison of various market-available wireless data transceivers.

The fully designed prototype eMotion nodes oscillate between 5 and 15mA current

consumption for standby and TX modes respectively. With a lithium-ion rechargeable

battery of 110 mA hours, each node can run about 10 hours between charges. Charge

time is roughly 2 minutes for a depleted battery. The line of sight range is roughly 100m,

suitable for any stage performance situation.

Receiver Hub

A single data hub is used to receive the data streams transmitted by the sensor

nodes and sends each data stream to the computer. If multiple computers are to have

access to the sensor data, a single computer with the hub may act as a networked host

and share the data to the other computers using the software client. Multiple data hubs

may also be used. The data hub includes an RF transceiver with antenna, an

asynchronous receiver/transmitter (UART), a processor, indicator LED, and a USB

interface for connecting to a computer.

80

Hub Design Considerations

The hub uses the same MCU (MSP430) and radio (Nordic RF Transceiver) as the

nodes to maximize compatibility. It is interfaced to the computer by a UART, which is

converted from RS232 to TTL serial levels using a chip by FTDI (TTL-232R-3v3-pcb)

[71]. This allows the computer to access the Hub as a serial device on its USB port. The

single indicator LED blinks when a transmission was successfully received,

acknowledged, and sent to the serial port. The hub hardware runs at 8MHz, eight times

faster than the transmitter nodes. This ensures that the hub can handle the incoming

data at a faster rate and alleviate potential network bottlenecking.

The hub waits in standby until a node transmits data. Once a transmission is

received, the transceivers’ Enhanced Shockburst TM protocol initiates in an auto-

acknowledge (ACK) routine. This is all handled by the radio hardware (all ACK

computations happen off of the hub processor requiring no extra computations). The

hub radio sends an ACK message to the specific node that transmitted the data, then

toggles an onboard LED to indicate a successful transaction with that node.

Because different nodes send different packet lengths (from one up to nine pairs

of data bytes) due to the varying kinds of sensors, the hub parses the received data

based on a node’s hard-coded sensor type. Once the data/address pairs are parsed,

the hub sends the bytes to the host computer via USB. The software client then has

accesses to the raw serial data streams.

81

Software Client

Data Input

Figure 6-4. Software client workflow. There are three major processes in the client

program-flow. A) Raw serial address/data-byte pairs are sent directly to the client and placed in a global buffer. B) The user processes the incoming data streams, which may include distortion, filtering, gesture or analysis. C) Raw and processed data streams are sent to a graphic UI for the user to assign to control/interact with musical or visual parameters; or to other computers or programs using the UDP, OSC, or MIDI output options.

Once the hub sends the address/data pairs to the computer, the software client

places all raw data into a global buffer that can be accessed by virtually any program

that receives data on a serial port (MaxMSP, Csound, supercollider, etc.). There are

three major processes in the client program-flow. A) Raw serial address/data-byte pairs

are sent directly to the client and placed in a global buffer. B) Each sensor type comes

with its own software module that “pulls” data from the global stream using the address

protocol. Here, the user processes each sensor’s data stream, which may include

floating-point calibration, filtering, gesture recognition or analysis. C) Raw and

processed data streams are sent to a graphic UI for the user to assign to control/interact

with musical or visual parameters – or to other computers, mobile devices, or programs

using the client’s UDP, OSC, or MIDI output modules.

The software client for the author’s prototype is designed in MaxMSP/Jitter. It was

chosen for its ease and speed for designing interactive user interfaces, access to

OpenGL and Javascript, and the author’s familiarity with the platform. The software

architecture is similar to the hardware in that all components are broken down into

82

modules that perform specific tasks. For each active sensor on the network, a

corresponding module is activated. Each module handles several important functions

including the following:

Data display: each module graphically displays the raw incoming data of its corresponding sensor. If the user creates multiple data buses during processing, each processed data bus is also graphically displayed.

Auto-calibrate: Sensors are going to output different ranges of data from one use to the next depending on a number of factors including environment, battery power, user, and varying physical placement. The software client, however, expects a total range of data between 0.0 and 1.0. A button on the UI allows the user to run an auto-calibrate routine on the incoming data. Calibration uses this linear mapping function: y=(x-xmin)/(xmax-xmin)*(ymax-ymin)+ymin; where ymin and ymax are the permanent lower and upper boundaries (0.0 and 1.0) and xmin and xmax are the movable boundaries of incoming data. Pressing the calibrate button erases the xmin and xmax values. When the user extends the sensor across the expected range on startup, the xmin/max values adjust to reflect the sensor’s actual range for that moment and continues to adjust the X boundary values during use. Whatever the actual sensor ranges, the linear mapping function ensures the software client receives the expected floating-point values between 0.0 and 1.0 regardless of extraneous and unpredictable factors. Originally, all sensor nodes were programmed with a physical button and an auto-calibrate routine in hardware. This resulted in the user having to physically press a small button on each sensor node. Depending on the node placement and performance situation, this method quickly became ungainly. Instead, placing the routine within the software modules allows the user, or a technician with the computer off-stage, to simply click all of the buttons on the screen to calibrate the whole network if desired – even in mid-performance.

The following is a breakdown of each software module for the prototype.

Hub Input: This module polls the serial port for all incoming sensor data at 50 Hz. The fastest motor reflex reaction latency for humans rests somewhere between 10 to 15 hz. The standard frame rate for cinema is 24 frames per second (fps). Thus, a 50 hz gesture capture resolution is more than sufficient for this system. The data stream’s address/data pairs are placed in a global buffer so other software modules can pull specific data from the stream. The Hub module detects new sensors on the network by combining the device ID and unique instance ID to derive each node’s serial number and saves the list of devices in a file. If the serial number is novel, the Hub module automatically generates a popup window visualizing the data with its corresponding device and instance information.

GENA_1: The user can open this module when sonar, force-sensitive resistor, or other GENA_1 device appears on the network. This module pulls the data only

83

with GENA_1-type addresses from the global data stream. An instance of this module is generated for each GENA_1 device on the network. The raw streams are displayed and can be accessed by the user for further processing and mapping.

GENA_6: This module pulls the data only with GENA_6-type addresses from the global data stream. Each raw stream is displayed and can be accessed by the user for further processing and mapping.

IMU Module: This module should be opened by the user to access IMU devices on the network. OpenGL is used to render a 3-demensional visual representation of the IMU device, providing orientation feedback to the user. The raw data of each axis from the accelerometer and gyroscope is also displayed and can be accessed by the user.

AHRS Module: This module is the same as the IMU module above except it also receives magnetometer data and uses a slightly different algorithm to visualize the orientation of the object.

A B Figure 6-5. Software client screenshot of sensor modules. The client detects new

devices as they appear on the network and automatically generates an instance of the corresponding module using the unique instance ID. The interface visualizes the data and give the user options to process and assign each data stream. A) represents a sonar module instance and B) illustrates an IMU module.

84

Data Processing

A user may modify the raw sensor data before using it at the output. For instance,

the user may want the data stream from the sonar to have inverted values before

mapping it to control a parameter. Each module allows the user to open up a processing

window. Various kinds of processes are available like filtering, gesture analysis, and

even physical interactive models. The user may access data along any point in the

process chain using data-bus outputs. These data-buses are stored in a list by the client

as potential data outputs, which can be mapped by the user to control musical or visual

parameters in the mapping window.

Figure 6-6. Screenshot of Data processing window. Processing flows from top-down.

85

Data Mapping

A visual matrix-mapping scheme resembling a patch bay provides a user

configurable method of connecting and disconnecting (i.e. patching) data streams to

parameter outputs, in a one-to-one, one-to-many, many-to-one, or many-to-many setup.

One data stream may control multiple parameters. In addition, multiple streams may be

used to affect a single parameter (multi-modal control).

Figure 6-7. Screenshot: Patch Bay Window. Displays data outputs (vertical axis) and the available list of output control parameters. Connections are made by clicking on the points where respective outputs and inputs intersect.

Additionally, a Digital Audio Workstation (DAW) method was explored for a user to

manage data streams and control assignments using data “tracks.” On a left-side

toolbar, a user selects from a list of available data streams. The data is visualized in the

center, which can be subsequently assigned to control a parameter by choosing from a

86

list in the right-side toolbar. Either of these methods allows the user to experiment with

the most optimal and intuitive mapping scheme for a project in real-time. The outputs

available for mapping can be virtually any software or devices. For example, musical

processes, FX processes, video projections, and lights can be controlled by the mapped

data.

Figure 6-8. Alternative Mapping Screenshot. Like a DAW, the user can create data

“tracks” and then assign input and outputs using the left and right dropdown menus.

Implementation: Augmented Trombone Using eMotion

Figure 6-9 illustrates the particular implementation of the eMotion prototype

retrofitted onto the author’s instrument of formal training: the bass trombone. Sensor

nodes are reversibly fixed to the trombone A at strategic sensing locations. The first

sensor node B, a force sensitive resistor (FSR), may be placed on the valve shifting

mechanism of the trombone, or at a position where the left hand would make contact

when a musician is playing the instrument. The function of the FSR in this location is

twofold: to detect when the trigger is depressed (on/off) and to indicate after-touch

pressure (squeezing the mechanism). A second FSR sensor node C is placed at a

position where the right hand would make contact when a musician is playing the

instrument (e.g., on or near the main slide hand grip). The objective is to make the

points of contact between the musician and the instrument sensitive to tactile pressure,

which can be mapped onto intuitive sonic phenomena. For example, the average

87

pressure detected could be mapped to the amplitude of effects processing for the

trombone’s sound. In this manner, the musician could control the amount of

distortion/overdrive applied to their signal by squeezing the instrument.

A third sensor node D, sonar, is placed on a fixed part of the main slide near the

mouthpiece to measure how far the slide is extended. A lightweight dowel E can be

attached to provide a detectable object for the sonar of the sonar node to continuously

detect. The dowel is attached onto the mobile part of the slide a particular distance from

the mounted position of the sonar sensor node D. The sonar device attached to the

node is an XL-MaxSonar-EZ3, operating at 3.3v levels. The particular distance spacing

between the sonar D and the dowel E depends on the minimum detectable distance the

sonar can detect (20 cm for this sensor). As the slide extends, the sensor values

increase with a 1cm (0.39 in) resolution out to a 765cm total length. The sonar software

module allows the user to auto-calibrate the sonar’s range to the particular user’s arm

length. Data is low-pass filtered and linearly mapped to a 0. – 1. range.

A fourth sensor node F, a six degree-of-freedom (6dof) IMU, is placed on or near

the bell. The IMU is the 6dof sparkfun razor board and includes a 3-axis accelerometer

and 3-axis gyroscope. When attached to the instrument, it measures overall instrument

orientation and tilt. An IMU operates on a level called “dead reckoning,” meaning that

the exact 3-demensional posture of the device cannot be exactly determined. An IMU is

calibrated at a known position to the user. The device proceeds to measure inertial

forces and calculates the position relative to the original known origin. Although the

inertial sensor values are useful in their own right, the exact posture of the device is

reliable for only a short time due to wandering and accumulative error. To maintain an

88

exact sense of 3-demensional orientation, discrete sensors must be used in conjunction

with the relative sensors. In this case, a 3-axis magnetometer is included in the IMU to

provide a full Attitude-Heading Reference System (AHRS).

Figure 6-9. Implementation of the eMotion System on the Bass Trombone.

A single data hub G is used to receive the data transmitted by the sensor nodes

and transfer data to an attached computer H via USB. The software client parses and

processes the incoming data, which is subsequently mapped by the user to control

various musical and visual parameters.

89

CHAPTER 7 CONCLUSIONS AND FUTURE DIRECTIONS

Could an instrument become intelligent, and adapt in an automated manner to a musician’s playing style? Could it learn the preferences of a particular musician, and modify itself in response to what it learns?

– Ken Jordan. Sound Unbound [15]

This dissertation presents a new technology developed by the author for

augmenting traditional musical instruments with the most up-to-date ultra-low power

microprocessor and RF wireless technology. More precisely, the system is a modular,

reversible, non-invasive, and reconfigurable wireless sensor mesh network combined

with a graphical user interface. A musician (or dancer, conductor, etc) is able to

wirelessly connect a variety of small individual wireless nodes where each performs a

particular sensing task (e.g. orientation, acceleration, distance, or pressure). This gives

the user the unprecedented ability to choose particular combinations of these nodes

and place them on desired sensing locations depending on the unique demands for a

given project. Nodes may also be recombined, rearranged, removed, remapped, and

turned on or off in real time with no adverse affects to the system itself. Though the

technology does not propose to completely solve the variety of aesthetic and technical

issues in the field of augmented musical instruments, this system embodies a significant

step in a positive direction.

Called eMotion, this system essentially allows someone with no knowledge in

microprocessors, analog circuit design, or programming (requisite knowledge to perform

similar tasks without this system) to intuitively control interactive sound, algorithmic

music, lights, or any number of effects using the gestures of their musical instrument.

The eMotion system is also useful for engineers and hobbyists with technical

90

backgrounds to quickly and easily deploy wireless sensing technology in projects,

saving considerable development time. However, the author acknowledges that there

are multitudes of methods to interact with digital media beyond instrument augmentation

and wireless sensing technology (including alternative controllers) and it is not intended

to suggest this system is the “best” overall approach for all projects.

Broader Impacts

Although developed to respond to current trends in musical instrument

augmentation, the capacity of the eMotion system has ramifications that span well

beyond making music or other forms of digital media. Some other potential areas

include home automation, environmental, urban, and industrial sensor monitoring,

providing alternative methods for access and interaction for the disabled, robotics, and

personal area networking technologies (e.g. health and sport monitoring).

Future Directions

The first prototype developed by the author took place over a number of years

(from 2008 to 2011). The intellectual property has been registered with the University of

Florida office of Technology Licensing and the Mk II version of the prototype is already

underway. The Mk II prototype will address a number of improvements including the

following:

Miniaturization: Surface-mount components and double-sided circuit boards will decrease the size minimally by 33%. Further miniaturization will result from employing the latest MEMS sensing technology. For example, the IMU described in the previous chapter had 3 ICs (integrated circuits) to measure 3 axes of acceleration and 3 axes of rotation. Currently, a single IC that is smaller than any of the previous ones is being used, which can manage all 6 axes of sensing.

Unique “stack” charging scheme: Sensor nodes will be designed to “stack” onto each other to interface with a charging station (built into the receiver hub) when not in use.

91

Battery indicator: A battery indicator IC will be integrated into the sensor node design to inform the user when the battery needs charging

Improved mounting hardware: A non-abrasive weak-adhesive strip with adaptor is being explored as a means to reversibly attach sensors to desired sensing locations. Each sensor node will have a special groove etched into the back of the housing that couples with the adhesive adapter. Hook-loop fasteners have been successfully deployed, but repeated use may wear the fasteners over time, decreasing their effectiveness. Bendable “wings” coated in non-slip rubber may be adapted to the housing of the sensor nodes that allow the user to bend the nodes to conform with the surface of their instrument.

Furthermore, a novel software client based on Digital Audio Workstation (DAW)

models will be developed for Processing & Mapping sensor data.

The user interface extends the existing software module system described earlier

by taking advantage of a generic layout most musicians are familiar with: the DAW.

However, instead of audio signals (or in addition to them), this processing system

utilizes/visualizes the incoming data from the sensor nodes of the wireless sensor

network. Examples of DAWs used in audio production include Apple Logic Pro, Avid Pro

Tools, Cakewalk Sonar, and Steinberg Cubase. Accordingly, a similar multi-track

interface can be used in order to further facilitate a musician’s interaction with the

wireless sensor network and processing system. Using this model, a user interface for a

computer-based Digital Data Workstation (DDW) is under development, having a layout

similar to a DAW’s transport controls (e.g. play, pause, and record), track controls, a

data mixer, and multi-track waveform display of data streams. In addition, the user can

save processing and mapping settings for a given session to a file for later access. The

settings can be saved as a session or as specified presets within a session. This allows

the user to not only explore the most optimal processing and mapping of their network,

but to save settings for later access as well. The user can change network configuration

presets (and thus the behavior of the sensor network) in mid-performance. By setting

92

presets, the user can change from one discrete state to another, or progressively tween

(a kind of cross-fade from one state to another) from one preset to the next.

The unique ID of incoming data streams are saved and compared to detect when

a new node is activated. If a novel address ID is detected, the software client prompts

the user that “a new node of sensor type x is on the network.” For instance, if the user

turns on a new sonar node, the client detects a novel address. Using the addressing

protocol, the software client prompts the user that a new sonar node is active and a new

track streaming live data appears in the DDW UI window. The sensor type, instance

number, and live data track stream are all displayed in the DDW front end.

Data Mixer: A data mixer resembling a DDW’s audio mixing console is in

development to handle the Processing and Mixing stages of the program flow. Each raw

data track corresponds to a corresponding track of the data mixer. Here, the user can

create additional data buses (similar to auxiliary sends on an mixer) of that track and

then assign a number of processes to each of those busses. The processed data

busses are visualized within the main UI console and appear, in order, below the raw

data inside of its original track. Each bus may be mapped to control user-defined

parameters. The slider for each mixer track differs from an audio slider in that there is a

min-slider and max-slider. Like adjusting the gain on an audio track, the min and max

sliders correspond to the ymin and ymax variables in the calibration linear function. This

gives the user unprecedented flexibility to ensure the sensor data rests within a desired

range of values for each track (default is floating point values between 0.0 and 1.0).

As an alternative to the patch bay matrix scheme, a more intuitive (or at least

engaging) method was demonstrated by Usman Haque for a project called

93

Reconfigurable House V.2 [72]. For this project, users could reconfigure sensor

mappings to control various actuation outputs by using an intuitive representation of

floating icons. Sensors were represented by one set of icons and outputs were

represented by another set. The user could drag these icons around using a touch

screen. If sensor and output icons are within proximity of each other, a connection is

formed. When dragged apart, the connection is broken. In this manner, the user is free

to arrange the sensors and control outputs in one-to-one, one-to-many, and many-to-

many groupings by arranging icons on a screen.

Figure 7-1. Workflow of a Novel DDW Software Client.

In the more distant future, the author hopes to develop the technology on a

number of levels. This includes (1) interactive instructional video games for traditional

musical instruments (e.g. a virtual lessons instructor), (2) exploring alternative methods

to power the sensor nodes, (3) creating a “smarter” software client using gesture

recognition algorithms, and (4) integrating this technology fully into traditional musical

94

instrument fabrication (e.g. musical instruments are purchased with this technology

already embedded).

Conclusions

History has shown that continued engagement with technological innovation is

integral for the evolution of musical instrument design and performance practice. As

mentioned previously, the instruments of the symphonic orchestra have remained

largely unchanged since the mid-1800s [34]. Conventional musical instruments must

continue to evolve to take advantage of current technological capabilities if they are to

remain in the contemporary dialog of stylistic progress. Trends in Imagine a time when

instruments purchased off-the-shelf still function as they always have, but also carry the

capabilities of the smart phones in our pocket. We have the capacity today.

95

APPENDIX MUSICAL SCORE: CAPOEIRISTA FOR FLUTE, BERIMBAU, AND LIVE ELECTRONICS

Object A-1. Sound file of “Capoeirista,” recorded November 30 2011 (.wav file 77MB)

96

97

98

99

100

101

102

103

104

105

106

107

LIST OF REFERENCES

[1] D. Smalley, “The listening imagination: Listening in the electroacoustic era,” Contemporary Music Review, vol. 13, no. 2, p. 77, 1996.

[2] K. Binsted, “Sufficiently advanced technology: using magic to control the world,” in CHI ’00 extended abstracts on Human factors in computing systems, New York, NY, USA, 2000, pp. 205–206.

[3] M. M. Wanderley and N. Orio, “Evaluation of Input Devices for Musical Expression: Borrowing Tools from HCI,” Computer Music Journal, vol. 26, no. 3, pp. 62-76, Nov. 2011.

[4] H.-C. Steiner, “Firmata: Towards making microcontrollers act like extensions of the computer,” in Proceedings of NIME, 2009.

[5] P. R. Cook, “Principles for designing computer music controllers,” in Proceedings of the 2001 conference on New interfaces for musical expression, 2001.

[6] A. R. Jensenius, R. Koehly, and M. M. Wanderley, “Building Low-Cost Music Controllers,” in Computer Music Modeling and Retrieval, vol. 3902, R. Kronland-Martinet, T. Voinier, and S. Ystad, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2006, pp. 123-129.

[7] J. A. Paradiso, “Current Trends in Electronic Music Interfaces,” Journal of New Music Research, vol. 32, p. 2003, 2003.

[8] E. R. Miranda and M. Wanderley, New Digital Musical Instruments: Control And Interaction Beyond the Keyboard, 1st ed. A-R Editions, Inc., 2006.

[9] C. J. Oja, Making Music Modern: New York in the 1920s. Oxford University Press, 2003.

[10] J. Chadabe, Electric Sound: The Past and Promise of Electronic Music. Prentice Hall, 1996.

[11] S. Crab, “120 Years of Electronic Music,” 2005. [Online]. Available: http://120years.net/. [Accessed: 13-Jul-2009].

[12] L. Cross, “Electronic Music, 1948-1953,” Perspectives of New Music, vol. 7, no. 1, pp. 32-65, Oct. 1968.

[13] J. A. Paradiso, “Electronic Music Interfaces,” 1998. [Online]. Available: http://web.media.mit.edu/~joep/SpectrumWeb/SpectrumX.html. [Accessed: 20-Jun-2011].

[14] H. Davies, “Hammond organ,” Grove Music Online, 2007.

108

[15] P. D. Miller, S. Reich, and J. Winner, “The World of Sound: A Division of Raymond Scott Enterprises,” in Sound Unbound: Sampling Digital Music and Culture, The MIT Press, 2008, pp. 181-201.

[16] R. A. Moog, “Voltage-Controlled Electronic Music Modules,” Audio Engineering Society Convention 16, vol. 13, no. 3, pp. 200-206, Jul. 1965.

[17] “Library of Congress National Recording Registry,” 2010. [Online]. Available: http://www.loc.gov/rr/record/nrpb/registry/nrpb-2010reg.html.

[18] S. Emmerson and D. Smalley, “Electro-acoustic music,” Grove Music Online.

[19] P. Doornbusch, The Music of CSIRAC: Australia’s First Computer Music. Common Ground, 2005.

[20] C. Nichols, “The VBOW An Expressive Musical Controller Haptic Human-Computer Interface (Dissertation).” Stanford University, Aug-2003.

[21] L. Brodie and Brodie, Starting Forth, 2nd ed. Prentice Hall, 1987.

[22] G. E. Lewis, “Too Many Notes: Computers, Complexity and Culture in Voyager,” Leonardo Music Journal, vol. 10, pp. 33-39, 2000.

[23] L. Polansky, P. Burk, and D. Rosenboom, “HMSL (Hierarchical Music Specification Language): A Theoretical Overview,” Perspectives of New Music, vol. 28, no. 2, pp. 136-178, Jul. 1990.

[24] D. P. Anderson and R. Kiuvila, “Formula: a programming language for expressive computer music,” Computer, vol. 24, no. 7, pp. 12-21, Jul. 1991.

[25] P. Griffiths, “Visions of Music Like Wheels In Space; Electronic Challenges Posed By Berio’s New Composition - New York Times,” 21-Oct-1997. [Online]. Available: http://www.nytimes.com/1997/10/21/arts/visions-music-like-wheels-space-electronic-challenges-posed-berio-s-new.html?pagewanted=all&src=pm. [Accessed: 02-Mar-2012].

[26] P. Manning, Electronic and computer music. Oxford University Press US, 2004.

[27] R. Kurzweil, The Age of Spiritual Machines: When Computers Exceed Human Intelligence. Penguin (Non-Classics), 2000.

[28] D. M. Huber and R. E. Runstein, Modern Recording Techniques, Seventh Edition, 7th ed. Focal Press, 2009.

[29] F. R. Moore, “The Dysfunctions of MIDI,” Computer Music Journal, vol. 12, no. 1, pp. 19-28, Apr. 1988.

109

[30] B. Schrader, Introduction to Electroacoustic Music. Longman Higher Education, 1982.

[31] J. Drummond and G. Paine, “TIEM: Taxonomy of Realtime Interfaces for Electronic Music Performance,” 07-Nov-2008. [Online]. Available: http://vipre.uws.edu.au/tiem/?page_id=676. [Accessed: 25-Aug-2010].

[32] N. Collins, “A Solder’s Tale: Putting the ‘Lead’ Back in ‘Lead Users’,” Pervasive Computing (IEEE), vol. 7, no. 3, pp. 32-38, Sep-2008.

[33] CTIA, “U.S. Wireless Quick Facts,” Jun-2010. [Online]. Available: http://www.ctia.org/advocacy/research/index.cfm/aid/10323. [Accessed: 08-Sep-2011].

[34] K. Bijsterveld and M. Schulp, “Breaking into a World of Perfection: Innovation in Today’s Classical Musical Instruments,” Social Studies of Science, vol. 34, no. 5, pp. 649-674, Oct. 2004.

[35] M. Hurni, “Softwind Instruments,” 1986. [Online]. Available: http://www.softwind.com/. [Accessed: 15-Aug-2010].

[36] N. Collins, “The Evolution of ‘Trombone-Propelled Electronics.’”Aug-2009.

[37] L. Cross, “‘Reunion’: John Cage, Marcel Duchamp, Electronic Music and Chess,” Leonardo Music Journal, vol. 9, pp. 35-42, Jan. 1999.

[38] T. Machover, “Hyperinstruments - A Progress Report 1987 - 1991.” MIT Media Laboratory, Jan-1992.

[39]“A Meta-Trumpet(er).,” in Proceedings of the International Computer Music Conference, San Francisco: International Computer Music Association, 1994, pp. 147 - 150.

[40] M. Burtner, “The metasaxphone: Concept, implementation, and mapping strategies for a new computer music instrument.,” Organised Sound, vol. 7, no. 2, pp. 201-213, 2002.

[41] C. Bahn, “Physicality and feedback: a focus on the body in the performance of electronic music,” Proceedings of the International Computer Music Conference, vol. 2001, p. 44--51, 2001.

[42] C. Bahn, “SBASS - Sensor Bass,” Cutris Bahn, “SBass” - sensor bass interface. [Online]. Available: http://www.arts.rpi.edu/crb/Activities/sbass.htm. [Accessed: 10-Sep-2011].

110

[43] R. Karpen, “Richard Karpen Selected Research Projects,” Anterior View of an Interior with Reclining Trombonist: The Conservation of Energy, 2003. [Online]. Available: http://dxarts.washington.edu/dxdev/profile_research.php?who=karpen&project=anterior. [Accessed: 09-Sep-2011].

[44] H. Jeffery, “tROMBOSCILLATOR,” 2006. [Online]. Available: http://www.mti.dmu.ac.uk/~jrich/kreepa/tromb.html. [Accessed: 20-Sep-2011].

[45] N. Farwell, “Adapting the trombone: a suite of electro-acoustic interventions for the piece Rouse,” in Proceedings of the 2006 conference on New interfaces for musical expression, 2006.

[46] C. Cadoz and M. M. Wanderley, “Gesture - Music,” in Reprint from : Trends in Gestural Control of Music, Ircam - Centre Pompidou, 2000.

[47] M. Wanderley, J. Malloch, and O. Lähdeoja, “Instrument Augmentation Using Ancillary Gestures For Subtle Sonic Effects,” in SMC Conference 2009, 2009.

[48] J. A. Paradiso, “Dual-Use Technologies for Electronic Music Controllers: A Personal Perspective,” NIME Proceedings, pp. 228-234, 2003.

[49] G. Paine, “Interactivity, Where to from Here?,” Organised Sound, vol. 7, no. 3, pp. 295-304, 2002.

[50] R. I. Godøy and M. Leman, Musical Gestures: Sound, Movement, and Meaning. 2009.

[51] Y. Harris, “Inside-out instrument,” Contemporary Music Review, vol. 25, no. 1-2, pp. 151-162, Feb. 2006.

[52] J. Ryan, “Some Remarks on Musical Instrument Design at STEIM,” STEIM [ texts ], 16-Feb-2010. [Online]. Available: http://www.steim.org/steim/texts.php?id=3. [Accessed: 23-Aug-2011].

[53] P. R. Cook, Music, Cognition, and Computerized Sound: An Introduction to Psychoacoustics. The MIT Press, 2001.

[54] B. Bongers, “Physical Interfaces in the Electronic Arts. Interaction Theory and Interfacing Techniques for Real-time Performance,” Trends In Gestural Control, p. 41--70, 2000.

[55] N. Orio, N. Schnell, M. M. Wanderley, and I. C. Pompidou, “Input Devices for Musical Expression: Borrowing Tools from HCI,” Computer Music Journal, vol. 26, no. 3, pp. 62-76, 2002.

[56] T. Wishart, On Sonic Art (Contemporary Music Studies). Routledge, 1996.

111

[57] P. Shove and B. Repp, “Musical motion and performance: theoretical and empirical perspectives.,” in The Practice of Performance: Studies in Musical Interpretation, Cambridge University Press, 2005, pp. 55-83.

[58] M. M. Wanderley, B. W. Vines, N. Middleton, C. McKay, and W. Hatch, “The Musical Significance of Clarinetists’ Ancillary Gestures: An Exploration of the Field,” Journal of New Music Research, vol. 34, no. 1, p. 97, 2005.

[59] C. Dodge and T. A. Jerse, Computer Music: Synthesis, Composition, and Performance, 2nd ed. Schirmer, 1997.

[60] TIK WSN Research Group, “The Sensor Network Museum,” 2011. [Online]. Available: http://www.snm.ethz.ch/Main/HomePage. [Accessed: 18-Oct-2011].

[61] D. T. Willingham, Cognition: The Thinking Animal- (Value Pack w/MySearchLab), 3rd ed. Prentice Hall, 2009.

[62] A. Rowe, R. Mangharam, and R. Rajkumar, “FireFly: A Time Synchronized Real-Time Sensor Networking Platform,” CRC Press Book Chapter, Nov. 2006.

[63] CTIA, “U.S. Teen Cell Phone Habits, Expectations, & Dream Phone Wishes,” Sep-2008. [Online]. Available: http://ctia.org/media/press/body.cfm/prid/1774. [Accessed: 08-Sep-2011].

[64] A. C. Clarke, Profiles of the Future: An Inquiry into the Limits of the Possible, Rev Sub. Henry Holt & Co, 1984.

[65] S. Madgwick, “Orientation estimation algorithms for IMUs and MARG sensor arrays.,” 30-Apr-2010. [Online]. Available: http://code.google.com/p/imumargalgorithm30042010sohm/.

[66] Texas Instruments, “MSP430x2xx datasheet.” SLAU 144e-2008.

[67] W. Hayward, Introduction to radio frequency design. American Radio Relay League, 1994.

[68] Digi International Inc., “XBee®/XBee‐PRO® RF Modules ‐ 802.15.4 ‐ v1.xEx [2009.09.23] (datasheet).”

2009.

[69] Bluetooth SIG, Inc., “Bluetooth Technology 101,” Bluetooth Technology 101, 2011. [Online]. Available: http://www.bluetooth.com/Pages/Fast-Facts.aspx. [Accessed: 23-Aug-2011].

[70] Roving Networks, “RN-1 Class 1 Bluetooth® Module (datasheet).” 04-Aug-2009.

[71] FTDI, “TTL-232R-PCB Serial Converter Datasheet.” 2008.

112

[72] U. Haque, “Reconfigurable House,” Reconfigurable House, 2006. [Online]. Available: http://www.haque.co.uk/reconfigurablehouse.php. [Accessed: 02-Aug-2009].

113

BIOGRAPHICAL SKETCH

From the ancient cypress swamps of Wewahitchka, FL, Chester Udell studied

Music Technology and Digital Art at Stetson University. He earned a Master of Music

degree in composition at the University of Florida in 2008 and a Ph.D. in music

composition with outside studies in electrical engineering at the University of Florida in

the spring of 2012.

Some of his honors include: SEAMUS/ASCAP Student Commission Competition

2010 1st prize, Prix Destellos 2011 Nominee, and Finalist for the Sound in Space 2011

International Composition Competition. His music can be heard on the SEAMUS and

Summit record labels and is also featured in Behavioural Processes, a peer-reviewed

scientific publication.