Thesis Final Draft
-
Upload
kidcranium -
Category
Documents
-
view
150 -
download
0
Transcript of Thesis Final Draft
1
Gesture Control
Table of Contents
Introduction LoudSpeaker Revolution
Gesture in the context of Musical Performance
Human Movement
Man-Machine Interaction
Paper Construction
Chapter I Background
Applications in the Past:
Applications in the Present:
Chapter II What Kind of Gesture
Spatial Relationships
Body Movement
Chapter III What Kind of Sensors
Different sensors for measuring body movement
Resistors
Gyroscopes
Location Tracking
Interconnections
System Latency
Chapter IV Design Concepts
Performance Environments
Instrument Design
2
Chapter V System Design
Construction
Analog-to-Digital conversion
802.11b
PC-104
Chapter VI Software
MAX/MSP
Chapter VII Mapping and the Aesthetic Consideration
My Considerations, How I chose to manipulate data.
Chapter VIII Dance and Music
Chapter IX The Performance
Lexicon, Lighting, Eventide, Spatialization
Engine27
Diagram of the Performance
Chapter X
Perception
Conclusion Future Applications
Wearable computing, augmented reality
Conclusion
Appendices
Resources
3
Gesture Control
Introduction
Why use gesture to control sound? It is necessary to answer this question by first
defining gesture. Webster’s definition of gesture:
2 : a movement usually of the body or limbs that expresses or emphasizes an idea,
sentiment, or attitude
3 : the use of motions of the limbs or body as a means of expression
This definition holds true in the context of musical performance. The gestures
associated with musical performance have the ability to heighten a musical experience.
However, the gestures associated with musical performance have been removed from the
listening experience. It is necessary to reintroduce the gestures associated with music
performance. It is possible to deconstruct the gestures of a musical performance and re-
associate the gesture with various sound control parameters. This can be accomplished
with the introduction of technology. Before we discuss how technology can be
implemented to map gesture to sound control parameters. It is best to first look at the
initial cause for the removal of gesture, from the musical experience.
LoudSpeaker Revolution
Elliot Schwartz states that the “Loudspeaker Revolution” transformed the nature of
the musical experience in profound ways. Today we may consider this revolution part of
a larger phenomenon – evidenced by the increased use of amplification and electric
instruments in live performance, the explosive growth of radio and television, and such
4
recent developments as the compact disc, videocassette recorder, and personal stereo.”
(Schwartz, pg. 26) The “Loudspeaker Revolution” has taken our music listening
experience into the home, the car, and has given us a means to introduce music into any
environment. This shift in environment has disassociated the performer from the
performance. In the past, performances given by musicians provided a visual stimulus to
the listener. The listener’s perspective has shifted in proportion to the environmental
shift. There is no longer a connection between the visual stimulus given by a performer in
a live performance and the listening experience given in our current environment. The
listener’s shift in perspective has led to a disassociation with the visual gesture of music
making.
Gesture in the context of Musical Performance
To understand the visual stimulus provided in a musical performance, it is necessary
to first focus on the interaction between the musical performer and the musical
instrument. Musical instruments transform the actions of one or more performers into
sound. An acoustic instrument consists of an excitation source under the control of the
performer(s) and a resonating system that couples the vibrations of the excitation to the
surrounding atmosphere. This affects, in turn, the precise patterns of vibration. In most
acoustic instruments (apart from organs and other keyboard instruments) the separation
between the input and the excitation subsystems is unclear. This separation has been
further increased by the introduction of synthesis and MIDI.
5
“The separation between gestural controllers and sound generators as standardized by
MIDI has led to the creation and development of many new alternative controllers with
which to explore new creative possibilities.” (e.g. Chadabe 1996; Wanderely and Battier
2000). MIDI was implemented as an interconnection between various digital
synthesizers. To further the connection between performer and instrument, Richard
Moore, a computer-music researcher at the University of California at San Diego, has
identified compatibility between performer and instruments as “Control Intimacy”:
“Control intimacy determines the match between the variety of musically desirable
sounds produced and the psycho-physiological capabilities of a practiced performer.” The
match between the variety of desirable sounds and the capabilities of the performer may
be determined differently depending on the terms used to represent the performance
process. “The fact that musical performance can be represented in different ways must be
taken into account when modeling musical performance. Reference to an auditory
process such as “hearing” for instance, implies a different representation action than does
reference to the same process as “listening.” Similarly the terms “moving” and
“gesturing” reflect different representations of the performance process.” (Mulder, LMJ,
1996). Let us broaden our focus and look at gesture in areas other than musical
performance. This will allow us to later investigate the gestures produced by a dancer.
Human Movement
Gesturing can take other forms such as gesticulation and signing, among others. All
forms of gesturing can involve tactile feedback, but very often do not. Similarly, they
may involve target-oriented movements, but very often do not. Adam Kendon, who
6
conducts research into nonverbal communication, defines gesticulations as “idiosyncratic
spontaneous movements of the hands and arms during speech” and signing “as the use of
a set of gestures and postures for a full-fledged linguistic communication system”
Kendon suggest that different forms of gesturing can be ordered according to their use of
linguistic forms. Gesticulation, language-like gestures, pantomime, emblems and signing
are different forms of gesturing that form a continuum, which proceeds from
gesticulation (least structured as a language) to signing (most structured as a language).
“The presence of such a continuum indicate that in human communications, gestures and
speech are part of a single system of channeling more or less linguistic meanings.”
(Mulder, 1996) Let back our focus out even further and look at all human movement in
general so that we can further our understanding of gesture as a control.
Some of the key concepts used in the exploration of human-motion intention are
taken from Rudolf Laban (1963). In his theory of effort, he notes the dynamic nature of
movement and the relationships between movement, space, and time. Laban’s approach
is an attempt to describe, in a formalized way, the characteristics of human movement
without focusing on a particular kind of movement or dance expression. Effort-theory
principles can be applied to dance and to everyday work practices. At the center of
Laban’s theory is the concept of effort, a property of movement. From an engineering
point of view, we can consider it with a vector of parameters that identifies the quality of
a movement performance.
7
The most important concept is a description of the quality of movement. Laban's
theory of effort is not concerned with degrees of joint rotation or movement directly but it
considers movement as a communication media and tries to extract parameters related to
its expressive power. During a movement performance, the vector describing the motion
quality varies in effort space. Mr. Laban studies the possible paths followed by this vector
and the intention they can express. Therefore, variations of effort during the movement
performance should be studied. This movement performance is what we will begin to
look at now.
“There is a fundamental connection between human movement and music.
Gesture interfaces reintroduce in music a concern for physicality. Composition and
execution, while remaining clearly, two different activities, share characteristics of
performance. Execution can influence structure and form because of the inherent
structural character of movement and space.” (Chabot, 1989) This reintroduction of
physicality can only be made with an introduction of technology.
Man-Machine Interaction
With the introduction of technology into a musical performance it behooves one
to consider the interaction between man and machine. The interaction between man and
machine is important to our discussion because the gestures of man will be controlling
machine. Johannes Goebel states the following about “man-machine interaction”: Quite
obviously, people who use the term ”man-machine interaction” do not use it in a way that
would imply that man reacts to a machine or that two machines react to each other. “Man
8
–machine interactions” wants to go beyond the known manipulations taking place
between man and machine, it aims for the realm of mutual, reciprocal action between two
partners who act on the ground of equal potential. It may not be a misinterpretation if I
assume that the implied aim is to engineer the quantitative states of a machine in such a
way that in the ultimate state men can act in their environment of natural languages, that
they can act out of their trains of thought which they will not have to formalize but which
a machine will comprehend by adapting itself to its own would of executable order.
“To react” has already undergone an extension from animated to in-animated: “1)
to act in turn or reciprocally 2) to act in opposition 3) to respond to a stimulus 4) to act
with another substance in producing a chemical change.” “Man-machine reaction”
instead of man-machine interaction? “Man-machine reaction” describes the current
situation more accurately, too accurately – and that is a problem. It could be understood
in pejorative ways: man only reacts to the stimuli produced by a machine and the
machine executes instructions the result of which will need man’s interpretation to be
another reason for his or her reaction which again will make the machine move or not
move. A machine will never know if it reacts or if it interacts, a program will never know
if tit is capable of learning. But human beings feel much better if they think of their
working-environment as being interactive instead of reactive. “Man-machine action”
implies the traditional hierarchy between man and tool. “Man-machine reaction:
describes a hierarchy where man reacts upon stimuli presented by a machine (and where
the production of such stimuli are a result of man’s labor). For obvious reasons we do not
like to put under such condition, even though and because a large part of western
9
civilization is based upon such working conditions, even though the production of
machines considered potentially interactive is based upon such conditions.(Goebel,
ICMC, 1998) Under the assumption that Goebel’s is correct about “man-machine”
interaction, it appear that the best interaction between man and machine lies in the
environment the two exist within. Interaction between gesture and control based on
environment-driven, rather than a performance-driven aspect will lead to a more visual
and aurally aesthetic end. Now that we have discussed the breakdown of gesture as
related to musical performance, and have started to investigate the nature of “man-
machine interaction”, I would like to discuss how the rest of this paper will be
constructed.
Paper Construction This paper will be constructed as follows:
I will briefly discuss in Chapter I, the history of gesture controllers, moving from
the Theremin, up to the present. In Chapter II, I will investigate various forms of
gestures. After discussing the various forms gesture can take, I will briefly describe, in
Chapter III, what kinds of sensors or systems have been used to capture these various
gestures. Chapter IV will be a discussion of various design concepts. My own designs
will be the topic of Chapter V, specifically what kind of gestures dancers can generate
and the challenges involved in capturing their gesture. Chapter VI will discuss different
types of software that can be used to manipulate the data coming in from these sensors.
Chapter VII will discuss the aesthetic decision that can be made in making the connection
between gesture and sound manipulation. Chapter VIII will discuss the connection
between dance and music. Then, in Chapter IX, I will talk about a performance that I
10
staged which, is a culmination of all the ideas that I will be discussing in this paper.
Following this discussion, Chapter X will briefly discuss the connection between visual
and aural stimulus and how this effects our perception. And finally, I will discuss
direction for the future applications of my gesture capturing system, and make my
conclusion.
Chapter I Background Applications in the Past:
Gesture control has been used for all types of applications since the beginning of
the 20th
Century. In 1920 a Russian physicist named Lev Termen (his name was later
changed to Leon Theremin) was engaged in designing an electronic burglar alarm, in an
early practical application of electrical technology. Theremin’s alarm was designed to
respond to changes in electrical capacitance caused by the approach of a foreign body
(Darreg, Experimental Instruments, 1991). The Theremin is unique in that it is played
without being touched. Two antennas protrude from the Theremin - one controlling pitch,
and the other controlling volume. As a hand approaches the vertical antenna, the pitch
gets higher. Approaching the horizontal antenna makes the volume softer. The historical
significance of a musical instrument that was played not by touching, but by gesture was
groundbreaking. This allowed for a myriad of other innovations, in experiments with
other types of musical instruments that could be played with gesture and movement.
In an experiment with one of these new gestural instruments, optical light sensors,
were used in John Cage’s Variations V. Variations V was first composed and performed
11
in 1965 and subsequently toured through Europe. A classic example of electronic
Gesamtkunstwerk, the work was a collaboration between Cage, Merce Cunningham,
musicians Gordon Mumma and David Tudor, Bell Labs engineers Billy Klüver and Max
Mathews (optical light triggers), visual artists Robert Rauschenberg, Nam June Paik and
Jasper Johns (projections) and the filmmaker Stan Vanderbeek. Derived from the
happening, in which the line between art and life is dissolved, the work is a surreal
juxtaposition of “real-life” movement, objects and images, in non-synchronous
combination using chance technique. Source material is taken from home movies and TV
sitcoms. Music is generated from taped sounds and live electronic synthesizers. The
movement of the dancers triggers sound and projections as a result of the optical light
sensors.
Another important innovation in the field of gesture control was produced by
Professor Max Mathews, a distinguished computer music pioneer and the inventor of
sound synthesis, is perhaps best-known for having co-authored "The Technology of
Computer Music" in 1969, which practically defined the entire field of computer music.
He is also known for his early work in gestural input for human-computer interaction,
which he did during his long career at Bell Telephone Laboratories. During the 1960s, he
and L. Rosler developed a light-pen interface, which allowed users to trace their musical
intentions on a display screen and see its graphical result before processing it.
Professor Mathews also developed the "GROOVE" system with F. R. Moore at
Bell Labs in 1970. GROOVE was an early hybrid configuration of a computer, organ
12
keyboard, and analog synthesizer, with a number of input devices including joysticks,
knobs, and toggle switches. The development of this system was important to the greater
field of computer music later on, because it was with this system that Professor Mathews
determined that human performance gestures could be roughly approximated as functions
of motion over time at a sampling rate of 200 hertz. This became the basis for the
adoption of the MIDI (Musical Instrument Digital Interface) standard for computer music
data transmission, by which eight 14-bit values can be transmitted every five
milliseconds. These various applications mentioned above have spawned a whole field of
research, which, brings us to the present to discuss what application are taking place in
the present.
Applications in the Present:
There are too many examples of these new gestural instruments to mention them
all. I will focus mainly on systems that produce music from controlled body movements,
and dance. These systems basically differ on the type of sensors used and on their
musical capabilities. For more examples IRCAM’s e-book “Trends in Gestural Control of
Music” presents an overview of gesture and motion sensing for music making. I will give
two examples of suit’s that can be worn by a dancer and used to control sound, as this
will be the focus of our discussion through the rest of this paper. I will also briefly
mention “The Hands” developed by Michael Waisvisz.
Mark Coniglio has built a costume (the "MidiDancer") that hides eight sensors
placed in crucial points, such as elbows and knees. Through the sensors, the movements
13
of the body are detected by a tiny computer that codifies the signal in such a way that it
can be sent to a receiver/decoder plugged to the computer, by means of a radio
transmitter.
Another such device is Digital Dance a research project conducted at DIEM
involving hardware and software development as well as the creation of a rule-based
compositional system. The project sprang from a vision that a dancer’s body movements
could control musical processes during a performance. The goal of the project was to
create a computer music composition that allows a dancer to directly influence musical
processes in a meaningful way. In November 1995 Jens Jacobsen, an electrical engineer,
began working at DIEM in collaboration with DIEM’s director, Wayne Siegel, to
research existing interactive dance interfaces and develop a new interface. Advantages
and disadvantages of several systems were considered and a new interface system was
designed in collaboration with the Aarhus School of Engineering. After years of
development and refinement the DIEM Digital Dance System version 2.0 is now being
made commercially available on a limited basis.
A different type of device uses the gesture of just the hands of a performer.
Michel Waisvisz has created a system called appropriately called “The Hands.” The
instrument consists of a number of sensors and keys mounted on two small keyboards
that are attached to the player’s hands. The combination of many different sensors to
capture the movements of the hands, the fingers and the arms is still unique and make
“The Hands” still one of the most refined and musical MIDI-controllers. Waisvisz states
that “a growing number of researchers/composers/performers work with gestural
14
controllers but to his astonishment he hardly sees a consistent development of systematic
thought on the interpretation of gesture into music, and the notion of musical feedback
into gesture.”
One can analyze and create distinct relationships between the character changes
of a gesture, and the change of musical content - and context - in a way that ones musical
intentions are clearly grasped by listeners. It is possible to have the freedom to create any
gestural relationship with a vast area of sound. Before we discuss these relationships, it is
necessary to look at the gestures to be captured and then look at how to capture these
gestures.
Chapter II What Kind of Gesture
Different gestures lend themselves to different applications. The movement of the
body can be measured in many ways. There are hundreds of sensors to capture the
movement of the body. Many researchers/composers/performers have built different
systems with various sensors and put a dancer in control of the music. It is necessary to
decide what type of gesture you want to capture so that you can determine what type of
sensor to use. “The category of gesture that has to be captured will determine the kind of
sensor to use.” (Depalle et al. 1997) “Creating a new instrument that responds to gesture
in a single fixed way might result in truly new music, but this originality could not be
sustained indefinitely. The novelty would fade as the instrumental technique became
standardized.” (Rubine, CMJ, 1990) So as Rubine states it is necessary to create a system
which, is re-configurable and modular. This discussion will focus on the interaction
15
between dancing and music, and will look specifically at the different types of gestures
that can be produced by a dancer.
Spatial Relationships
The spatial relationship between multiple dancers is one of the most important
gestures of dance. Spatial sensors detect location space by proximity to hardware sensors
or by location within a projected grid. There is a connection that can be made between
the spatial relationships of a dance performance and the spatial metaphors that play an
important role in characterizing material of music. The concept is that sounds that are
similar to each other are placed proximate to each other in the spatial representation. A
variety of spatial representations that go beyond the one-dimensional layout of a
traditional keyboard have been proposed for pitch. The spatial metaphor that maps
perceptual similarity to spatial proximity in the control interface can be generalized to
more complex musical processes and structures, such as those involved in the generation
of melody, rhythm and complex textures. One notable example is the musical texture
space provided by David Huron (Huron 1989, 2001). Gerog Hajdu(1994) has
demonstrated the use of Huron’s texture space as a control structure. It should be noted
that “meaningful spatial control configurations can be produced without the
multidimensional scaling of tediously collected similarity judgments for all pairs of
sounds in the set under consideration.” (Wessel, 2002) The discussion will return to this
topic of spatial relationships.
16
Body Movement
The dancer’s orientation to the audience, as well as the dancer’s orientation to the
floor can be characterized as another important gesture in dance. Many important
gestures also happen in the legs, arms, head, and torso. This is where most of the
attention is given by capturing system designers. On-body sensors measure angle, force,
or position of various parts of the body. For example, piezo resistive, and piezo-electric
technologies are very common for tracking small body parts such as fingers. We will now
discuss what types of sensors can be used to capture the gestures mentioned above.
Chapter III What Kind of Sensors Different sensors for measuring body movement
The discussion will now focus on what types of sensors to use for the different
gestures that can be created by a dancer. Human gesture has to be noise filtered,
calibrated, and quantized in order to have a normalized representation. Such data can then
be analyzed in order to extract pertinent information. Calibration process does not
necessarily lead to data reaction with an implicit impoverishment of the gestural
information, but helps to get homogenous information from heterogeneous input
channels. Studies on behavior from kinetic approach point out that human beings hardly
reproduce their gesture with precision. Furthermore, by dealing with gesture it is hard to
separate the sign of gesture from the signal it conveys and then to extract it significance.
There is no general formalization of movement that may help us to recognize its
meaning. As detailed by (Cadoz, 1994) gesture allows humans to interact with their
17
environment to modify it, to get information and to communicate, the ergotic, epistemic,
and semiotic functions associated with the gestural channel of communication. (Sapir)
Resistors
To capture these different gestures, it is necessary to use different types of
sensors. A vary common sensor used throughout almost every system, are flex sensors.
A flex sensor is a variable resistive strip that when bent will manipulate voltage. These
flex sensors are generally placed on the joints of the body to measure the bend angle of
the particular body part you want to capture.
Hardware Example 1.) is a flexible resistive sensor. When this sensor is bent it changes
the voltage that is output. This change in voltage increases the more the sensor is bent.
Hardware Example 1.
Gyroscopes
To measure orientation it is necessary to use some type of gyroscopic sensor, this
sensor will generally output its orientation in degrees or rotation. These sensors can be
place about the body to measure any rotational joint such as; the hands, head, legs, or
directly on the torso, to measure the orientation of the dance within the room.
18
Hardware Example 2.) is a ceramic angular velocity sensor.
Hardware Example 2. Murata GyroStar, ceramic angular velocity sensor. This particular
sensor, outputs a change in voltage that is equivalent to the speed of rotation.
Location Tracking
To measure spatial relationships or track location, it is necessary to have some
sort of fixed transmitter, and movable receiver. Many systems use ultrasound, in a sonar-
type fashion whereby, an output pulse is sent to the receiver and the time between the
pulses is used to measure the dancers location. In most cases it is necessary to have
multiple transmitters, which, would allow one to triangulate the position within a room.
Other systems use sensors external to the body such as cameras. These systems detect
changes from continuous frames (e.g. Rokeby’s Very Nervous System) or changes
between the current frame and a frame of reference (e.g., STEIM’s BigEye).
Hardware Examples 3. and 4.) show the cricket location tracking pair. Example 3. is the
Beacon/Transmitter of the pair. And Example 4. is the Listener/Receiver of the pair.
19
Hardware Example 3. Hardware Example 4.
“The Cricket” location tracking system is a very effective way to capture location
within a room. This device is defined as a location-support system for in-building,
mobile, location-dependent applications. It allows applications running on mobile and
static nodes to learn their physical location by using listeners that hear and analyze
information from beacons spread throughout the building. Cricket is the result of several
design goals, including user privacy, decentralized administration, network heterogeneity,
and low cost. Rather than explicitly tracking user location, Cricket helps devices learn
where they are and lets them decide whom to advertise this information to; it does not
rely on any centralized management or control and there is no explicit coordination
between beacons; it provides information to devices regardless of their type of network
connectivity; and each Cricket device is made from off-the-shelf components and costs
less than U.S. $10. “The Cricket” implements randomized algorithms, used by the
beacons to transmit information. The use of concurrent radio and ultrasonic signals are
20
used to infer distance. The listener uses inference algorithms to overcome multi-path and
interference. The system is very practical, the beacon configuration and positioning
techniques help improve accuracy. My experience with “The Cricket” has included
several location-dependent applications such as in-building active maps and device
control, these applications can be developed with little effort or manual configuration.
Interconnections
After having defined what type of sensors are placed where on the body of the
dancer, one can then translate these measurements into meaningful values. These values
can then be sent to sound control parameters. It is important for all the components to be
accurate and well scaled. Most of the sensors mentioned above are analog and output
changes in voltage. Therefore, it is necessary to have an Analog-to-Digital converter
(ADC) to use the data provided by the sensors with a computer. For use within a dance
performance it is necessary for the interconnection between the output of the ADC and
the computer to be wireless. This wireless connection has a few components. One
component is power, the other data transmission. To power the system it is necessary to
use batteries. Transmitting the data has historically been achieved with RF. Using an RF
transmitter- receiver pair. The receiver is then connected to a computer. Once in the
computer the various sensors can be used to various sound control parameters.
System Latency
One of the major roadblocks in developing a gesture capturing system is latency.
Latency is the elapsed time between a stimulus and the response. Some kinds of latency
21
are relatively easy to measure for example, the time between a computer’s receipt of a
control message and the beginning of the audio event generated in response.
At this point, I will diverge into the territory of the Linux operating system. I will
explain how Linux can be modified to achieve latency results of up to 10 times faster
than any other operating system. What is real-time? -- I think the IEEE definition
(supplied by Douglass Locke) is a good one, and a clear one. Namely: "a real-time
system is a system whose correctness includes its response time as well as its functional
correctness." The system doesn't just run software to plough through the process; it needs
to do it in a timely manner. So, real-time systems care about timeliness, but just how
timely isn't clear until you add a modifier to "real-time". In other words, real-time is very
much a gray region. "Response to what?" To "events", which are either external events,
communicated to the system via its input/output interfaces (which include things like
Ethernet as well as audio, keyboard, or mouse input), or to timer-triggered events within
the system itself.
I will discuss what modifications I made to the Linux kernel, to run in real-time
on the wearable computer that I developed for my performance. A Linux 2.4.17 kernel
patched with a combination of both preemption and low-latency patches yielded a
maximum scheduler latency of 1.2 milliseconds, a slight improvement over the low-
latency kernel. However, running the low-latency patched kernel for greater than twelve
hours showed that there are still problem cases lurking, with a maximum latency value of
215.2ms recorded. Running the combined patch kernel for more than twelve hours
22
showed a maximum latency value of 1.5ms. This data seems to indicate that the best
solution for reducing Linux scheduler latency is to combine both sets of patches. Latency
is really a shorthand term for the phrase latent period, which is defined by webster.com to
be "the interval between stimulus and response". In the context of the Linux kernel,
scheduler latency is the time between a wakeup (the stimulus) signaling that an event has
occurred and the kernel scheduler getting an opportunity to schedule the thread that is
waiting for the wakeup to occur (the response). Wakeups can be caused by hardware
interrupts, or by other threads. Large scheduler latencies have numerous causes.
One culprit are drivers that do lots of processing in their entry points or interrupt
service routine (ISR). Another is kernel code that stays in a block of code for a long
period of time, without explicitly introducing scheduling opportunities. Both of these
things cause one problem: the kernel does not have the opportunity to perform scheduling
calculations for a long period of time. Since the scheduler is the mechanism for
determining what process/thread should run, it needs to be run with a relatively high
frequency for the kernel to be responsive. The bottom line goal of someone trying to
reduce scheduler latency is to insure that opportunities to run the scheduler occur
regularly, so that tasks needed to service events get a chance to run as quickly as possible.
A different strategy for reducing scheduler latency called the low-latency patches was
introduced by Ingo Molnar and is now maintained by Andrew Morton. Rather than
attempting a brute-force approach (ala preemption) in a kernel that was not designed for
it, these patches focus on introducing explicit preemption points in blocks of code that
may execute for long stretches of time. The idea is to find places that iterate over large
23
data structures and figure out how to safely introduce a call to the scheduler if the loop
has gone over a certain threshold and a scheduling pass is needed (indicated by
need_resched being set). Sometimes this entails dropping a spinlock, scheduling and then
reacquiring the spinlock, which is also known as lock breaking. The ability to modify the
operating system and to have it “react” with low latency was a key process to beginning
the investigation into the development of a gesture capturing system. This low latency
removed many of the problems that many system designers have encountered.
Real-time computing technology and development of human gesture tracking
systems may enable gesture to be introduced again into the practice of computer music.
The study and the analysis of human gesture in the music area for the control of digital
instruments along with standardization requirements coming from the industry have led
to the birth of MIDI devices and their communication protocol. “This protocol has often
been judged insufficient, because of its narrow bandwidth and its limitations due to the
pianistic gesture production. This is even more true today as the relationship between
composer and technology is no more exclusive but is extended to together forms of
artistic projects on which multimedia interact with other forms of expression such as
dance, theatre, or video art.” (Sapir, JNMR, 2002) Now that we have looked at the
various forms that gesture can take via dance, and the what kind of sensors we can use to
capture these gestures, it is time to start discussing various concepts of design for an
interactive system.
24
Chapter IV Design Concepts Performance Environments
The relationship between interactive environments and compositional process is
too often glossed over, even though a truly interactive performance environment should,
almost by definition, challenge basic compositional thinking. That interactive
environment, whether it is driven by dancers moving within a defined space, or by a
single performer playing a more conventional device, such as a MIDI keyboard or wind
controller, becomes a central part of the composition itself.
Robert Rowe in his excellent “Interactive Music Systems” distinguishes between
“score-driven” and “performance-driven” interactive systems. Score-driven systems are
those in which predetermined, or “scored” materials are compared or matched with
incoming performance data, placing the composer in his or her traditional role as the final
arbiter of the piece. Performance-driven systems, on the other hand, rely on interpreting
typically improvisational performance data, attempting to recognize classes of data, such
as repetition, speed, pitch range, and so on. Richard Povall has advocated the addition of
a third classification, “environment-driven” which in some ways synthesizes Rowe’s
classifications, and in other ways lies entirely outside them. “In an environment-driven
interactive performance system the performer is working within an algorithmic
environment in which there are elements of both score-driven and performance-driven
systems. The environment is listening to the performance data, which in its turn can
trigger predetermined or algorithmic, or even aleatoric processes. By the same token, the
25
performer is also reacting to the environment, placing herself into a fully interactive
feedback situation.” (Povall, JNMR, 1995) Feedback can is an advantageous addition to
an environment-driven system. Feedback between the performer and the system provides
a much needed interconnection between the performer awareness of what is being
manipulated within the system. I tend to agree with Povall on his summation that it is
possible to have an interactive environment, which, is a combination of the two systems
that Rowe discusses.
Instrument Design
“The conception and design of new musical interfaces is a burgeoning
multidisciplinary area where technological knowledge (sensor technology, sound
syntheses and processing techniques, computer programming, etc.) artistic creation, and a
deep understating of musicians’ culture must converge to create new interactive music
making paradigms. If new musical interfaces can be partially responsible for shaping
future music, these new musical paradigms should not be left to improvisation.” (Jorda,
CMJ, 2002) I disagree with Jorda on his point that the paradigms he discusses should not
be left to improvisation. In a properly created environmentally-driven system,
improvisation with feedback is the key to an aesthetically heightened musical experience.
Often, more difficult-to-master instruments lead to richer and more sophisticated
music’s (e.g., the piano vs. the kazoo), but expressiveness does not necessarily imply
difficulty. In that sense, one of the obvious research trends in real-time musical
instrument design can be the creation of easy-to-use and, at the same time, sophisticated
26
and expressive systems (considering that the best way to understand and appreciated any
discipline, whether artistic or not and music in exception is by doing and being part of it).
More efficient instrument that can subvert the aforementioned effort-result quotient will
bring new-sophisticated possibilities and the joy of real-time active music creation to
non-trained musicians.
Whether following a parallel design or maintaining two independent approaches, the
physical and logical separation of the input device from the sound production necessitates
multiple way of processing and mapping the information coming from the input device.
Mapping becomes therefore an essential element in designing new instruments. We will
come back to this design process when we start the discussion of Aesthetic
Considerations.
Chapter V System Design Construction
The discussion will now focus on how I developed a gesture capturing system for
use in a performance with three dancers. The system that I developed has the following
components: ten flex sensors placed on one finger of each hand, the wrists, elbows, the
upper thighs, and the backs of the knees. The system also included two angular velocity
sensors mounted perpendicularly to each other allowing me to capture the dancers,
orientation within the room, as well as their orientation to the floor. The angular velocity
sensors output a voltage that was equal to the angle of rotation. The final capturing
component was an indoor location tracking system developed by MIT called “The
27
Cricket.” The Cricket allowed for the ability to track location in two dimensions with an
accuracy of two centimeters.
Analog-to-Digital conversion
To do Analog-to-Digital conversion I chose to use a BX-24 micro-controller. The
micro-controller was not only used to do A/D conversion but also allowed me to
multiplex the aforementioned sensors together before transmitting the data. The
multiplexing worked by giving each A/D input on the micro-controller an ID from 1-10.
The ID’s where then coupled with the value of the A/D input from each sensor. This data
was then sent in a stream out of the micro-controller’s serial interface.
Hardware Example 5.) is the BX-24 on the left mounted to a development board with a
power regulator on the right.
Hardware Example 5.
28
After doing A/D conversion, the micro-controller, was connected via serial into a
wearable CPU.
802.11b
To transmit the data from the sensors an 802.11b wireless card was used. This
method of transmission was not only more reliable, it also allowed for greater amounts of
bandwidth. The increase in data transmission greatly lowered latency. To add wireless
Ethernet (802.11b) to the system I chose to investigate wearable computing. It was
important to use off-the-shelf components to minimize development time.
PC-104
The decision was made to use a form factor called PC-104, these units are
approximately 3” square, and have a 104-pin expansion bus. The104-pin bus allows for
different modules to be plugged in for the expansion of the CPU’s functionality. The PC-
104 CPU module runs on an x86 compatible 66MHz processor, and was expanded with a
PCMCIA module that allowed for the addition of an 802.11 wireless Ethernet card.
Hardware Example 6.) is the TopView of the PC-104 with the PCMCIA module and
wireless Ethernet card.
29
Hardware Example 6.
The system boots from a compact flash card. The compact flash card was loaded
with a custom Linux distribution that allowed me to run various applications for
transmitting the sensor data. The system is powered by modified laptop battery. All of the
components are mounted inside of a plexi-glass housing, this housing has straps similar
to a backpack for the ease of motion and wearability. The above configuration allowed
me to have a compact, affordable, off-the-shelf, wireless system to capture all of the
dancers gestures.
Hardware Example 7.) show the BottomView of the PC-104 with a 64MB memory
module on the left and a 128MB CompactFlash card on the right.
30
Hardware Example 7.
Hardware Example 8.) shows the complete system including A/D converter, CPU and
PCMCIA module with Ethernet, and the location tracking system, all mounted inside of
the plexi-glass housing.
Hardware Example 8.
31
Hardware Example 9.) is a block diagram of the gesture capturing system worn by the
dancer.
Hardware Example 9.
Chapter VI Software
Once the system is operational, the next step is to start considering how the
gesture can be mapped to a sound control parameter. This mapping can be easily
achieved in software. “A software layer is always necessary between gesture acquisition
and sound production: gesture management software and performance software.” (Sapir,
JNMR, 2002) I will now discuss the various software systems that can be used for
mapping gesture to sound control. In a joint effort with Morton Subotnick, Mark Coniglio
has developed "Interactor" a Macintosh software that allows the computer to interpret
data coming from a set of sensors placed on the performer's body or somewhere on the
Flex Sensors
Gyroscopes
Cricket
Listener
Micro-
Controller
PC-104 with
802.11b
wireless
Ethernet
32
stage, so allowing the performer herself to directly control different devices such as
synthesizers, theatrical lighting, digital audio effects, video apparatus and even robots.
MAX/MSP
I chose to use the widely available and pseudo-modular MAX/MSP. After having
configured my gesture capturing system to be wireless, it was necessary to configure a
small wireless network between the dancers and a machine running Linux. The machine
running Linux ran a custom application called SerialOverIP, this software can run in a
client-server configuration, whereby the dancer’s wearable computer ran the server
portion and the machine running Linux ran the client portion, this software let me bind,
the serial interfaces on all my machine to an IP address. The machine running Linux had
four serial interfaces these serial interfaces were connected to a laptop running
MAX/MSP. Using the “serial” object to input the sensor data into MAX/MSP. Once the
sensor data was in MAX/MSP, the ID’s for each sensor were stripped and routed to the
appropriate objects.
The following is a run through of the MAX/MSP patch that I developed to input,
de-multiplex, and scale the sensor data coming from my gesture capturing system.
Software Example 1.) is the main patch where all the modules are connected via sub-
patches.
33
Software Example 1.
Software Example 2.) shows the serialin sub-patch which takes in the input from the
machine running Linux.
Software Example 2.
Software Example 3.) shows where the data is de-multiplexed with the MAX/MSP patch.
This sub-patch take the input from the serial object strips the ID’s from the micro-
controller via the sub-patch serial routing, this stripped data is then sent to the scaling sub
patch which, make the data suitable for MIDI control, and then outputs to the averaging
schema.
34
Software Example 3.
Software Example 4.) shows the serial_routing sub-patch, the ID’s from the micro-
controller’s A/D inputs were stripped from the data stream.
Software Example 4.
Software Examples 5. and 6.) show the scaling sub-patches. Here the data stream was
scaled to achieve MIDI’s minimum and maximum values of 0-255.
35
Software Example 5. Software Example 6.
Software Example 7.) is a block diagram of the software data flow.
Software Example 7.
The later portions of this diagram “Averaging of Sensor Values” and “Output to MIDI
controller” will be discussed in the following Chapter.
Chapter VII Mapping and the Aesthetic Consideration
This discussion will focus on the aesthetic consideration given to mapping a
gesture to a sound control parameter. I will discuss the techniques presently used to map
gesture to sound control. The mapping problem is at the center of the effectiveness of an
Serial Input from
Linux Machine
De-Multiplex
Scaling and
Routing
Averaging of
Sensor Values
Output to MIDI
controller
36
interactive environment. To a certain extent, as described by Winkler (1995), we can
consider human movement in space as a musical instrument if we allow the physicality of
movement to impact on musical material processes, and if we are able to map physical
parameters of movement on high level musical parameters down to low level sound
parameters.
One characteristic of the mapping from gestural parameters to sound control
parameters, in the case of most traditional instruments, is its simplicity and directness.
The position of a finger on the piano keyboard maps directly to pitch the downward
velocity of the finger maps directly to amplitude. In these and many other examples, a
single gestural parameter maps directly onto a single sound control parameter. (The
timbre of the piano also changes with velocity, but since the changes are linked
inseparably with the amplitude changes, we consider the pair to be a single sound control
parameter of the piano.) This directness in not lost on the listener, who can often infer the
form of the gesture from the sound generated. Cadoz (1988) states that “the sound
phenomenon produced by a natural object or instrument is an indelible trace of the
gesture” and that the perceptual system of the brain “infers possible causes,” i.e., possible
gestures that made the sound.
For the sake of simplicity many electronic instruments use basic one-to-one linear
mappings. More complex mappings such as one-to-many, many-to-one, and many-to-
many, with optional non-linear relations, memory, and feedback (so that the output is not
only a function of the last input value, but also of previous input and/or output values)
37
involve deeper design strategies. “Such strategies do not exist in a separated conception
approach. They also require more demanding programming techniques not easily
attainable in the environments commonly used for mapping design purposes, such as
Max and similar data-flow graphical programming languages.” (Puckette 1988; Puckette
and Zicarelli 1990; Winkler 1998)
“Unlike the “one-gesture-to-one-acoustic-event” paradigm, our framework allows
generative algorithms that can produce complex musical structures consisting of many
events. In this model, the performers gestures can guide and control high-level
parameters of the generative algorithms rather than directly triggering each event. One of
our central metaphors for music control is that of driving of flying about in a space of
musical processes. Gestures move through time, as do the musical processes.” (Wessel,
CMJ, 2002)
Many people have chosen to have one-to-one relationships between the gesture and
the type of data manipulation. For instance, a dancer, or performer will move their finger
in and out, and a light may turn on or off. This is not the direction that we should be
headed with gesture capturing technology. It is important to break down the gestures, and
then reconstruct a method for how the gesture should manipulate the data.
My Considerations, How I chose to manipulate data.
A performance was staged as a test bed for the gesture capturing technology that I
developed. This section is going to be a discussion of the mapping decisions used in the
38
performance. Each dancer had 16 different values associated with them. The values
included the flex of each of their ten joints, 4 different measurements of orientation and
X,Y coordinates for location. Using three dancers, nearly fifty different values were
being generated at any given time. It was necessary to start combining and averaging the
data, which would in turn generate different curves, and distributions. The different
curves and distributions could then effect the entire environment on a global level. These
curves were then mapped to different sound control parameters and effected very subtle
changes in the system. The averaging of the sensor data from the dancers was done as
follows: The total value of all the sensors on each arm was averaged, then, the values of
the legs; thighs and knees, for both legs was averaged. The orientation of the dancers
position to the floor, as well as their orientation within the room was combined with their
location in the room to effect the spatialization within the environment. Aesthetically this
type of mapping was much more pleasing conceptually as well as visually, and aurally.
The following is an example of what mapping took place within MAX/MSP
Software Example 7.) shows the averaging of the sensor data.
39
Software Example 7.
Software Example 8.) shows the connection of the output of the above scaled and
averaged sensor data to an Eventide Orville
Software Example 8.
40
Chapter VIII Dance and Music
In order to achieve a coherent integration of music and dance, we consider their
analogous principles. These include exposition of ideas, links, and transitions between
sections, variation strategies, embellishments methods, and structural issues. In dance
exposition of ideas is achieved through the repetition of movements in different space
regions, combined with variations, jumps twists and falls. Such variations incorporate
new movements to those already exposed. The general structure is normally divided into
sections, each with its own expressive criteria characterized normally by different
scenarios, light, and clothing. “These pose several problems, including what information
should be captured, how to characterizes this information and how to relate this
information in a way that is coherent with the music.” (Morales-Manzanares, 2001)
“Traditionally, music and dance have been complementary arts. However, their
integration has not always been entirely satisfactory. In general, a dancer must conform
movements to a predefined piece of music, leaving very little room for improvisational
creativity.” (Morales-Manzanares, 2001) I disagree with Morales-Manzanares on this
point, the room from improvisation can be immense when combining a dancer with an
environmentally-driven system that uses gesture as the control.
41
Human gesture has always been at the center of the musical creation process either
with the writing form of a score of for the instrumental and conducting performance. On
the contrary listening to music often leads to emotions, which reverberate at the body
level: a phenomenon, which is at the basis of dance and choreography.
It is very natural for a listener to put the gesture back into the listening experience,
many listeners play Air-guitar, or Air-drums to “get into” the music, and feel what it is
like to play the instrument, thus adding visual stimulus back into the music.
Chapter IX The Performance
The designer of the performance introduces sound and music knowledge into the
system, along with the compositional goals and the aspects of integration between music
and gesture (including a model of interpretation of gestures). “This may imply an
extension of the music language toward action, gesture languages, and visual languages.
This example raises important issues about new perspectives of the integration of music
and movement languages. For example, the performance designer in a live electronics
configuration now cooperates with dancers and performers.” (Vidolin 1997)
Lexicon, Lighting, Eventide, Spatialization
I wrote a vocal quintet in three movements, this particular piece was written to be
manipulated by reverb, and delay once it was recorded. I found a very interesting space
called Engine27, to do a performance of the vocal work with dance.. Engine27 has a 16-
42
speaker system with many high-end effects processing units, including a Lexicon 960L
with 16-ins and outs, as well as an Eventide Orville with 8-ins and outs. The 16-speaker
system allows for sound to be moved from any speaker to any speaker.
Engine27
I will now discuss my aesthetic decision on connecting three dancers into
Engine27’s facilities. First it was necessary to get a recording of my vocal piece, this was
recording was made in pro-tools. I took the pro-tools recording and ran it through the
Eventide, and the Lexicon, this effected mix then ran through MAX/MSP. MAX/MSP
was where the music was spatialized through the 16-speakers system. I ran a laptop that
could take the input from the dancers, transform the input, and then output midi to the
Eventide, the Lexicon, Engine27’s Horizon Lighting System, and the MAX/MSP
spacialization system. The arms and legs were averaged for each dancer and a peak and
trough were generated from their movements, these low and high values were used to
create a curve between the minimum and maximum values. The orientation was tracked
in a histogram, and was used in conjunction with the location data to generate different
random distributions for spacialization. The lighting system was also effected by time
and location. At different points in time a lighting cue was selected by the dancers
location. The general effect that I wanted to get was a global change with multiple
variables instead of specific change with one variable.
Performance Example 1.) is the Max patch used to control spatialization.
43
Performance Example 1.
Performance Example 2.) is the Eventide input output portion of the spatialization patch.
Performance Example 2.
44
Performance Example 3.) is the Lexicon and Eventide Spatialization output portion of the
patch.
Performance Example 3.
Performance Example 4.) Shows the setup for each movement of the vocal composition
Performance Example 4.
Performance Example 5.) Shows the sub-patch pan_sub, this sub-patch received location
and orientation tracking information off of the dancer to control what sound would be
placed in each speaker.
45
Performance Example 5.
Performance Example 6.) Shows the sub patch quikPan
Performance Example 6. This patch was developed by Dafna Naphtali. The input can be
sent to any of the 16 speakers. This is controlled by the coll object which, holds a matrix
for the different speaker combinations.
Diagram of the Performance
Performance Example 7.) is the block-diagram of the entire performance setup.
46
Performance Example 7.
A DVD is included with this paper. This DVD includes the performance titled
“Epistemology” which is mixed in 5.1, as well as the above MAX/MSP patches.
Wearable
Gesture Capture
System
Wearable
Gesture Capture
System
Wearable
Gesture Capture
System
Wireless
Hub Linux
Machine
Laptop
Pro-Tools
playback
and output
Orville
Lexicon
Max
Spatial
System
Sound Web Amplifiers
Horizon
Lighting
System
Audio
Output
47
Chapter X
Perception
The perceived control of the parameters affected by the dancers was not an
obvious one-to-one connection. The mapping and aesthetic decisions made in the above
mentioned performance helped to draw the audience into the piece, peaking the
audience’s awareness. The awareness was keyed into linking the dancers movement to
the music; to listen more intently to the changes in the music being effected by the
dancers, and watching the dancers movement to hear the change. This heightening of
awareness was a necessary component to the performance.
The technological description begs the compositional issues at the head of this
paper. Certainly the composer is forced to approach his/her composition in a different
way. There is no possibility here of building a score that can be stepwise triggered in
some way by the performers – Rowe’s (Rowe 1994) “score-driven” interactive system –
neither are the performers simply extending material provided for, or playing some kind
of extended instrument – Rowe’s “performance-driven” system. In the case of my
performance and others like this performance, the performer(s) are entering a virtual
environment that has precomposed, probabilistic, algorithmic and aleatoric elements. The
virtual environment is as much an instrument to be played as a score to be read.
Traditional notions of form; subject, development, repetition, become less important,
even somewhat impossible in a world of indeterminacy and eccentricity. Yet this in not a
truly indeterminate music neither is it purely improvisational. Central to the
48
compositional design of the piece is the algorithmic manipulation of data – manipulation
that may not necessarily mirror its input in any recognizable way.(Povall).
Michel Waisviz, designer of THE HANDS (Waisvisz 1985), has provided one of
the best counter-examples to the notion that electronic music is inherently lifeless.
Following Waisvisz’s example, designers of new controllers should attempt to maintain
or even expand the bandwidth and parallelism of expression beyond that of traditional
instruments. Furthermore, new controllers should allow the performer to flexibly redirect
expression at whatever functional level that is appropriate. For example one might want
to maintain careful control of the timer of a cell in a quested environment or during a
solo, but then redirect one’s expressive capabilities to the control of orchestral timber
during a finale. In the latter context, the orchestra can be thought of as a large, expressive
instrument.
Gestural expression and listening are constrained by a context based on stylistic,
syntactic, and semantic conventions, including those that involve qualities such as
emotional “depth”. A musical instrument designer may want to implement relation
between a sound parameter or musical quality described in terms of one representation
and a movement parameter or gestural quality described by means of another. To
implement such relations, the designer will first need to define a way to transform the
first representation in to the second. Failure to do so implies a return to a single
representation of musical performance.
49
How does the re-association of visual stimulus effect our perception of a
performance? This visual stimulus provided by a performance give the listener a more
heightened sense of awareness. This heightened awareness draws the audience into the
music that is being performed and is ultimately more satisfying.
Conclusion Future Applications Wearable computing, augmented reality
I would like to expand my knowledge of gesture capturing systems into many
different areas. Firstly, I plan on condensing many of the interconnections, associated
with this system. I am already developing a MAX/MSP object that will allow me to go
directly from the wearable computer into a laptop. This will allow me to remove all of the
serial connection from the system. Also, under consideration for the expansion of the
system, I would like to discontinue the usage of MAX/MSP, and move to a real-time
system such as Roger Dannenberg’s AuraRT. AuraRT will allow me to directly
synthesize sound off a laptop running Linux. A Revision of the hardware used for the
wearable computer is in progress as well. I want to further decrease the size of the
wearable computer. This will be accomplished by the use of Stanford University’s
Matchbox PC. The Matchbox PC, is 5 cubic inches and weighs 3.3ozs. This will be the
direction for further development with my device. I have considered different
applications for my device as well.
The gesture capturing system that I developed could be used as a video-game
controller. Allowing the player to map any body movement to a movement within the
50
game. I would like to expand on this idea by first. Getting the opportunity, to do another
performance of my vocal quintet where video could be manipulated in real-time, as well
as audio, and lighting. I would like to also explore how the gesture capturing system
could be used on the Internet utilizing the 802.11 capabilities. Possibly controlling
shockwave or flash movies. Once, connected to the Internet, the system could be used to
manipulate data in a database. I would like to hook the system into a database, and do
relational data manipulations, using Multi-Dimensional object (M-OLAP)
transformations. This transformed data could be used to manipulate video and audio data
streams coming off a streaming server running java servlets. Also, the system could be
used in motion capturing. This could be further explored with the possibility of
connecting the gesturing system, via the python scripting language, into the open-source
real-time 3D rendering application Blender. There are applications in augmented reality
that could be exploited as well, the CPU module has the capability for an LCD display,
thus allowing one to connect retinal scanning glasses. These glasses would allow a
performer to view the real-world, while mapping other data over the top of it. This would
give the performer more feedback of the environmental changes that they are effecting.
Conclusion
To conclude, gesture has to be used to control sound. The concept of gesture
control has already defined itself as a field within music composition and music
technology. Technology has progressed far enough for us to implement real-time
computing, with the use of highly accurate sensors, to develop affordable gesture
capturing systems. With low-latency and high accuracy, these gesture-capturing systems
51
have applications throughout various fields, outside of music composition and music
technology. The decision of mapping gesture to sound control parameters, is then the area
where further compositional control can be exercised. The idea of gesture mapping is a
burgeoning field, which with these advances in technology, will hopefully, now begin to
be further explored. Once the mapping of gesture to sound control has been clearly
defined, highly effective compositions can be created. These compositions can make
gesture, utilizing the ideas of an environmentally-driven performance, a new instrument.
These performance’s will allow composers to re-introduce the visual stimulus that has
been missing from musical performance for the past fifty years.
52
Appendices
Code for SerialOverIP, this application was written in C, and is being ported to
MAX/MSP.
/* * ---------------------------------------------------------------------------- * serialoverip * Utility for transport of serial interfaces over UDP/IP * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or * (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA * */ #include <stdio.h> #include <stdlib.h> #include <unistd.h> #include <errno.h> #include <sys/types.h> #include <sys/stat.h> #include <sys/socket.h> #include <netinet/in.h> #include <arpa/inet.h> #include <fcntl.h> #include <termios.h> #include <signal.h> #define MAXMESG 2048 char*pname; int s[2],st[2];
53
void help(){ fprintf(stderr,"\ SerialOverIP comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under GNU General Public License. Usage: %s <source1> <source2> where <source1> and <source2> are one of the folowing: -s <IP> <port> UDP server on IP:port -c <IP> <port> UDP client for server IP:port -d <device> sss-dps local serial device sss is speed (50,..,230400) d is data bits (5,6,7,8) p is parity type (N,E,O) s is stop bits (1,2) ",pname); return; } int setserial(int s,struct termios*cfg,int speed,int data,unsigned char parity,int stopb){ cfmakeraw(cfg); switch(speed){ case 50 : { cfsetispeed(cfg,B50) ; cfsetospeed(cfg,B50) ; break; } case 75 : { cfsetispeed(cfg,B75) ; cfsetospeed(cfg,B75) ; break; } case 110 : { cfsetispeed(cfg,B110) ; cfsetospeed(cfg,B110) ; break; } case 134 : { cfsetispeed(cfg,B134) ; cfsetospeed(cfg,B134) ; break; } case 150 : { cfsetispeed(cfg,B150) ; cfsetospeed(cfg,B150) ; break; } case 200 : { cfsetispeed(cfg,B200) ; cfsetospeed(cfg,B200) ; break; } case 300 : { cfsetispeed(cfg,B300) ; cfsetospeed(cfg,B300) ; break; } case 600 : { cfsetispeed(cfg,B600) ; cfsetospeed(cfg,B600) ; break; } case 1200 : { cfsetispeed(cfg,B1200) ; cfsetospeed(cfg,B1200) ; break; } case 1800 : { cfsetispeed(cfg,B1800) ; cfsetospeed(cfg,B1800) ; break; } case 2400 : { cfsetispeed(cfg,B2400) ; cfsetospeed(cfg,B2400) ; break; }
54
case 4800 : { cfsetispeed(cfg,B4800) ; cfsetospeed(cfg,B4800) ; break; } case 9600 : { cfsetispeed(cfg,B9600) ; cfsetospeed(cfg,B9600) ; break; } case 19200 : { cfsetispeed(cfg,B19200) ; cfsetospeed(cfg,B19200) ; break; } case 38400 : { cfsetispeed(cfg,B38400) ; cfsetospeed(cfg,B38400) ; break; } case 57600 : { cfsetispeed(cfg,B57600) ; cfsetospeed(cfg,B57600) ; break; } case 115200 : { cfsetispeed(cfg,B115200); cfsetospeed(cfg,B115200); break; } case 230400 : { cfsetispeed(cfg,B230400); cfsetospeed(cfg,B230400); break; } } switch(parity|32){ case 'n' : { cfg->c_cflag &= ~PARENB; break; } case 'e' : { cfg->c_cflag |= PARENB; cfg->c_cflag &= ~PARODD; break; } case 'o' : { cfg->c_cflag |= PARENB; cfg->c_cflag |= PARODD ; break; } } cfg->c_cflag &= ~CSIZE; switch(data){ case 5 : { cfg->c_cflag |= CS5; break; } case 6 : { cfg->c_cflag |= CS6; break; } case 7 : { cfg->c_cflag |= CS7; break; } case 8 : { cfg->c_cflag |= CS8; break; } } if(stopb==1)cfg->c_cflag&=~CSTOPB; else cfg->c_cflag|=CSTOPB; return tcsetattr(s,TCSANOW,cfg); } void gotint(int x){ if(st[0]&2){ tcflush(s[0],TCIOFLUSH); close(s[0]); } if(st[1]&2){ tcflush(s[1],TCIOFLUSH); close(s[1]); } printf("%s exiting.\n",pname); exit(1); }
55
int main(int argc,char**argv){ int i,n,w,clen[2],nonblock[2],speed,data,stopb; unsigned char c,buf[MAXMESG],*p,parity; struct termios cfg; struct sockaddr_in addr[4][4]; struct sigaction newact,oldact; pname=argv[0]; if(argc!=7){ help(); return 1; } for(i=0;i<2;i++){ st[i]=0; switch(argv[3*i+1][1]){ case 's': st[i]=1; case 'c': bzero((char *) &(addr[i][0]), sizeof(addr[i][0])); addr[i][0].sin_family = AF_INET; addr[i][0].sin_addr.s_addr = inet_addr(argv[3*i+2]); addr[i][0].sin_port = htons(atoi(argv[3*i+3])); bzero((char *) &(addr[i][1]), sizeof(addr[i][1])); addr[i][1].sin_family = AF_INET; addr[i][1].sin_addr.s_addr = 0; addr[i][1].sin_port = htons(0); if((s[i]=socket(AF_INET,SOCK_DGRAM,0))<0){ fprintf(stderr,"%s: can't open datagram socket",pname); return 3; } if(bind(s[i],(struct sockaddr*)&addr[i][!st[i]],sizeof(addr[i][!st[i]]))<0){ fprintf(stderr,"%s: can't bind local address",pname); return 4; } break; case 'd': st[i]=2; if((s[i]=open(argv[3*i+2],O_RDWR|O_NDELAY))<0){ fprintf(stderr,"%s: could not open device %s\n", pname,argv[3*i+2]); return -1; }
56
n=sscanf(argv[3*i+3],"%d-%d%c%d",&speed,&data,&parity,&stopb); if(n<4){ fprintf(stderr,"%s: invalid argument %1d from %s\n", pname,read+1,argv[3*i+3]); return 3; } if(setserial(s[i],&cfg,speed,data,parity,stopb)<0){ fprintf(stderr,"%s: could not initialize device %s\n", pname,argv[3*i+2]); return 7; } break; default:help();return 2; } clen[i]=sizeof(addr[i][1]); nonblock[i]=!(st[i]&1); } signal(SIGINT,gotint); i=0; while(1){ if(st[i]&2)n=read(s[i],buf,MAXMESG); else{ n=recvfrom(s[i],buf,MAXMESG,nonblock[i]*MSG_DONTWAIT, (struct sockaddr*)&addr[i][st[i]],&clen[i]); nonblock[i]=1; } p=buf; while(n>0){ if(st[!i]&2)w=write(s[!i],p,n); else w=sendto(s[!i],p,n,0, (struct sockaddr*)&addr[!i][st[!i]],clen[!i]); if(w>0){ n-=w; p+=w; }else{ fprintf(stderr,"%s: write error\n",pname); break; } } i=!i; } return 0; }
57
The following is code from the Linux kernel with kernel pre-emptive and low-latency
patches.
+A preemptible kernel creates new locking issues. The issues are the same as
+those under SMP: concurrency and reentrancy. Thankfully, the Linux preemptible
+kernel model leverages existing SMP locking mechanisms. Thus, the kernel
+requires explicit additional locking for very few additional situations.
+
+This document is for all kernel hackers. Developing code in the kernel
+requires protecting these situations.
+
+
+RULE #1: Per-CPU data structures need explicit protection
+
+
+Two similar problems arise. An example code snippet:
+
+ struct this_needs_locking tux[NR_CPUS];
+ tux[smp_processor_id()] = some_value;
+ /* task is preempted here... */
+ something = tux[smp_processor_id()];
+
+First, since the data is per-CPU, it may not have explicit SMP locking, but
+require it otherwise. Second, when a preempted task is finally rescheduled,
+the previous value of smp_processor_id may not equal the current. You must
+protect these situations by disabling preemption around them.
+
+
+RULE #2: CPU state must be protected.
+
+
+Under preemption, the state of the CPU must be protected. This is arch-
+dependent, but includes CPU structures and state not preserved over a context
+switch. For example, on x86, entering and exiting FPU mode is now a critical
+section that must occur while preemption is disabled. Think what would happen
+if the kernel is executing a floating-point instruction and is then preempted.
+Remember, the kernel does not save FPU state except for user tasks. Therefore,
+upon preemption, the FPU registers will be sold to the lowest bidder. Thus,
58
+preemption must be disabled around such regions.
+
+Note, some FPU functions are already explicitly preempt safe. For example,
+kernel_fpu_begin and kernel_fpu_end will disable and enable preemption.
+However, math_state_restore must be called with preemption disabled.
+
+
+RULE #3: Lock acquire and release must be performed by same task
+
+
+A lock acquired in one task must be released by the same task. This
+means you can't do oddball things like acquire a lock and go off to
+play while another task releases it. If you want to do something
+like this, acquire and release the task in the same code path and
+have the caller wait on an event by the other task.
+
+
+SOLUTION
+
+
+Data protection under preemption is achieved by disabling preemption for the
+duration of the critical region.
+
+preempt_enable() decrement the preempt counter
+preempt_disable() increment the preempt counter
+preempt_enable_no_resched() decrement, but do not immediately preempt
+preempt_get_count() return the preempt counter
+
+The functions are nestable. In other words, you can call preempt_disable
+n-times in a code path, and preemption will not be reenabled until the n-th
+call to preempt_enable. The preempt statements define to nothing if
+preemption is not enabled.
+
+Note that you do not need to explicitly prevent preemption if you are holding
+any locks or interrupts are disabled, since preemption is implicitly disabled
+in those cases.
+
+Example:
+
+ cpucache_t *cc; /* this is per-CPU */
+ preempt_disable();
+ cc = cc_data(searchp);
+ if (cc && cc->avail) {
+ __free_block(searchp, cc_entry(cc), cc->avail);
+ cc->avail = 0;
+ }
59
+ preempt_enable();
+ return 0;
+
+Notice how the preemption statements must encompass every reference of the
+critical variables. Another example:
+
+ int buf[NR_CPUS];
+ set_cpu_val(buf);
+ if (buf[smp_processor_id()] == -1) printf(KERN_INFO "wee!\n");
+ spin_lock(&buf_lock);
+ /* ... */
+
+This code is not preempt-safe, but see how easily we can fix it by simply
+moving the spin_lock up two lines.
60
Resources
Bauer, Will; Foss, Bruce GAMS: An integrated media controller
system. Computer Music Journal, Vol. 16;
Issue 1; spring 1992; pp. 19-24; Illus. ISSN: 0148-
9267.
Bongers, Bert An interview with Sensorband.
Computer Music Journal, Vol. 22; Issue 1; spring 1998;
pp. 13-24; Illus., port. ISSN: 0148-9267.
Camurri, Antonio; Coletta, Paolo; Ricchetti, Matteo; Volpe, Gualtiero
Expressiveness and physicality in interaction.
Journal of New Music Research, 2000; Vol. 29; Issue
3; Sept 2000; pp. 187-198; Illus., charts, diagr.
ISSN: 0929-8215.
Camurri, Antonio; Hashimoto, Shuji; Ricchetti, Andrea; Ricci, Andrea;
Trocca, Riccardo; Volpe, Gualtiero EyesWeb: Toward gesture and
affect recognition in interactive dance and music systems.
Computer Music Journal, Vol. 24; Issue 1; spring
2000; pp. 57-69; Bibliog., illus., charts, diagr.
ISSN: 0148-9267.
Causse, Rene Recherche et facture instrumentale: Logiques de
l'innovation. [Research into the making of music
instruments: The logic of innovation.] Les cahiers de
l'IRCAM: Recherche et musique, Issue 7; 3rd quarter 1995;
pp. 67-76 ISSN: 1242-8493.
Cerana, Carlos Gesture control for musical processes: A MAX
environment for Buchla's "Lightning". Organised Sound:
An International Journal of Music Technology, Vol. 5;
Issue 1; Apr 2000; pp. 3-7; Charts, diagr. ISSN:
1355-7718.
Cerana, Carlos Gesture control of musical processes: A MAX
environment for the lightning. Porto Alegre, Brazil: Sociedade
Brasileira de Computacao; pp. 169-178; 1999
61
Chafe, Chris Tactile audio feedback. San Francisco:
International Computer Music Association; pp. 76-79; 1993; Charts,
diagr.
Damiani, Furio; Manzolli, Jonatas; Mendes, Gilberto Controle
parametrico MIDI usando interface gestual ultrasonica. [
Parametric control in MIDI using an ultrasonic gestural interface.
] Porto Alegre, Brazil: Sociedade Brasileira de Computacao;
pp. 55-60; 1998
Darreg, Ivor; Hopkin, Bart Still nothing else like it: The
theremin. Experimental Musical Instruments
, Vol. 8; Issue 3; Mar 1993; pp. 22-26; Illus., bibliog.
ISSN: 0883-0754.
Downes, Pat Motion sensing in music and dance performance.
New York: Audio Engineering Society; pp. 165-172; 1987
Ghazala, Qubais Reed Circuit-bending and living instruments: The
odor box. Experimental Musical Instruments
, Vol. 8; Issue 2; Dec 1992; pp. 17-23; Illus.
ISSN: 0883-0754.
Harada, Tsutomu; Sato, Akio; Hashimoto, Shuji; Ohteru, Sadamu
Real time control of 3D sound space by gesture. San
Francisco: International Computer Music Association; pp. 85-88; 1992;
Illus.
Hobus, Steffi Wittgenstein uber Expressivitat: Der Ausdruck in
Korpersprache und Kunst. [Wittgenstein on expressivity:
Expressions in body language and art.] Hannover:
Internationalismus-Verlag; 211 p.; (PhD diss.: U. Bielefeld, 1994)
ISBN: 3-922218-67-9.
Keislar, Douglas [Editor] Computer music journal. XXVI/3 (fall
2002): New performance interfaces. Computer Music
Journal, Vol. 26; Issue 3; fall 2002; pp. 11-76; Illus.,
bibliog., charts, diagr. ISSN: 0148-9267.
Keislar, Douglas [Editor] Computer music journal. XXV/4 (winter
2001): Sound in space. Computer Music Journal
, Vol. 25; Issue 4; winter 2001; pp. 21-90; Illus.,
port., facs., music, bibliog., list of works, charts, diagr., sound
recording ISSN: 0148-9267.
62
Logemann, George W. Experiments with a gestural controller.
San Francisco, CA: Computer Music Association; pp. 184-185;
1989
Manzolli, Jonatas The development of a gesture interfaces
laboratory. Porto Alegre, Brazil: Sociedade Brasileira de
Computacao; pp. 81-84; 1995
Modler, Paul Interactive control of musical structures by hand
gestures. Porto Alegre, Brazil: Sociedade Brasileira de
Computacao; pp. 143-150; 1998
Moog, Robert A. Position and force sensors and their
applications to keyboards and related control devices. New
York: Audio Engineering Society; pp. 173-181; 1987
Mulder, Axel Getting a grip on alternate controllers: Addressing
the variability of gestural expression in musical instrument design.
Leonardo Music Journal, Vol. 6; 1996;
pp. 33-40; Illus. ISSN: 0961-1215.
Mulder, Axel Sound sculpting: Performing with virtual musical
instruments. Porto Alegre, Brazil: Sociedade Brasileira de
Computacao; pp. 151-164; 1998
Povall, Richard Compositional methods in interactive performance
environments. Journal of New Music Research
, Vol. 24; Issue 2; June 1995; pp. 109-120; Port.,
charts, diagr. ISSN: 0929-8215.
Rubine, Dean; McAvinney, Paul Programmable finger-tracking
instrument controllers. Computer Music Journal
, Vol. 14; Issue 1; spring 1990; pp. 26-41
ISSN: 0148-9267.
Rubine, Dean; McAvinney, Paul The VideoHarp. Koln,
Germany: Feedback Studio; pp. 49-55; 1988; Bibliog., tech. Drawings
Rubine, Dean; McAvinney, Paul Programmable finger-tracking
instrument controllers. Computer Music Journal
, Vol. 14; Issue 1; spring 1990; pp. 26-41
ISSN: 0148-9267.
Settel, Zack; Holton, Terry; Zicarelli, David Remote control
applications using "smart controllers" in versatile hardware
63
configurations. San Francisco: International Computer Music
Association; pp. 156-159; 1993
Tanaka, Atau Musical technical issues in using interactive
instrument technology with application to the BioMuse. San
Francisco: International Computer Music Association; pp. 124-126; 1993
Abstract: The BioMuse is a bioelectrical musical controller used in composition and
concert performance. (author)
Tobenfeld, Emile A system for computer assisted gestural
improvisation. San Francisco: International Computer Music
Association; pp. 93-96; 1992
Ungvary, Tamas; Vertegaal, Roel Designing musical
cyberinstruments with body and soul in mind. Journal of
New Music Research, 2000; Vol. 29; Issue 3; Sept 2000;
pp. 245-255; Illus. ISSN: 0929-8215.
Wanderley, Marcelo M.; Battier, Marc; Depalle, Philippe; Dubnov,
Shlomo; Hayward, Vincent Gestural research at IRCAM: A progress
report. Publications du Laboratoire de Mecanique et
d'Acoustique, 1998; Issue 148; May 1998; pp. D2.1-D2.8
ISSN: 1159-0947.
Wessel, David L. Instruments that learn, refined controllers,
and source model loudspeakers. Computer Music Journal
, Vol. 15; Issue 4; winter 1991; pp. 82-86; Port.,
bibliog. ISSN: 0148-9267.