POLYPHONIC EMBOUCHURE ON AN INTRICATELY EXPRESSIVEMUSICAL KEYBOARD FORMED BY AN ARRAY OF WATER JETS

Steve Mann, Ryan Janzen

University of TorontoDepartment of Electrical and Computer Engineering, Toronto, Canada

mann@eecg.toronto.edu, rejanzen@gmail.com

ABSTRACT

Touching, diverting, restricting, or obstructing water jets con-stitutes a new type of user-interface for immersive multime-dia environments such as totally acoustic, totally electronic,or hybrid musical instruments. The result is a richly expres-sive input device. In the context of a musical instrument,this device is called a hydraulophone.

Developments in the rich expressivity of the hydraulo-phone are presented, as a new type of embouchure control.This paper presents a new concept called “finger-jet em-bouchure” in which each “key” (water jet) on the instrumentis governed by fluid-dynamics, rather than by solid key mo-tion. The direct coupling between a musician’s finger andphysical sound production in the liquid (which can be de-tected by underwater microphones and fed into a computer,thus creating a hyperacoustic user-interface) leads to highlyexpressive performance styles. By designing each note to beassociated with one water-jet “mouth” on the instrument, wegive the performer an ability to fluidly interleave the dynam-ics of many notes simultaneously, producing a phenomenonnamed “polyphonic embouchure”. The result is the abilityto perform richly expressive music, whether on an acous-tic hydraulophone, or on an enhanced hydraulophone withsound pickups for hyperacoustic computer performances.

1. INTRODUCTION

The hydraulophone is a newly-invented1 musical instrumentwhich uses water as its sounding mechanism. Unlike otherinstruments that used water merely as a source of pressure todrive air through sounding pipes, the hydraulophone is theworld’s first instrument that has been able to produce soundpurely from vibrations in water itself, without requiring anyair to vibrate whatsoever [6].

The research required to accomplish this, based on thefluid dynamics of moving liquid, derives its inspiration fromthe screeching sounds made by defective faucets, and other

1The FUNtain (TM) Hydraulophones and water keyboards describedin this paper are covered by patents issued or filed in multiple countries,including CA2499784, CA2517501, and others pending. For license (non-profit, research, or commercial) contact author S. Mann.

valves with liquids passing through them, giving rise to thedevelopment of various water-based multimedia devices andmusical instruments [5]. In particular, various underwatermusical instruments were made from simple devices (somehand-cranked or pumped like an accordion, others motor-ized) that rapidly turned water jets on and off, or forced wa-ter through resonant orifices, and, additionally, other water-based musical instruments and interfaces such as organ pipeswith water actually flowing through the pipe and fipple mech-anism, were explored [5]. These new interfaces were a bighit with children at public pools. Additionally, we use someversions of the hydraulophone as an electronic input devicefor various multimedia applications [8].

By placing hydrophones (underwater microphones) inthe water flow, the acoustic sound can be picked up and usedas a broadband input signal source for controlling computermusic applications [8].

Our work on using underwater sound pickups with hy-draulophones [2] has led to the creation of hyperacousticinstruments [7] which utilise not only audible sound fromthe physical sound production, but also sound in the sub-sonic and ultrasonic ranges as well, and shift it into an audi-ble frequency spectrum. (This is in contrast to simply aug-menting a hydraulophone to become a hyperinstrument [4]).Since the turbulent water flow is highly controllable, a mu-sician can control subsonic and ultrasonic turbulent acousticsignals, and experience them transformed into the audiblerange, or transformed into controllable signals for computermusic devices [5].

Acoustic hydraulophones have been found to be highlyexpressive musical performance instruments [8]. Rich ex-pressivity is possible because of: (1) the close coupling be-tween the performer’s fingers and the turbulent flow associ-ated with sound production [7][2]; (2) acoustic/visual/tactilefeedback to the performer, from hearing the turbulent sound,and seeing and feeling the water flow [5][9]; (3) fluidly con-tinuous control over the sound by intricately touching thewater jets [9]. These elements exist regardless of whetherthe acoustic sound production is used as an input source forcomputer music, or whether the hydraulophone is left as apurely acoustic musical instrument.

Proceedings of the International Computer Music Conference (ICMC 2009), Montreal, Canada August 16-21, 2009

545

1.1. Examples of HydraulophoneWe have designed and built a number of hydraulophone pro-totypes, many of which are human-powered and entirelyacoustic, and others which have been electro-acoustic, opto-acoustic, as well as some that are completely electronic.

Examples of purely acoustic hydraulophones, which em-body the hydraulophone concept in its purest form, includea user-interface consisting of an array of water jets that eachfunction like a key on a keyboard instrument. Often the jetsare arranged in rows, like the keys on a piano, so that thehydraulophone is played by touching, pressing or blockingone or more of the water jets in succession.

Of our purely acoustic hydraulophones, some use un-derwater organ pipes as the sounding mechanism. (Fig. 1illustrates the H2Organ.) Others are built using spatial watermodulators that acoustically pulse the water flow at an audi-ble frequency [6]. The more the finger blocks a jet, the morefluid is diverted into the sounding mechanism, causing thenote, for example, to become louder, and brighter in timbre.(However, with the water modulator, unlike the H2Organ,the note stays in tune, regardless of the quantity, pressure,of other flow changes in the hydraulic fluid.)

Therefore, unlike even a true tracker organ, which givesthe musician some control over the attack envelope, hydraulo-phones give far greater control over both the attack envelopeand the steady state sound (volume, timbre, etc.).

2. EMBOUCHURE WITHOUT A HUMAN MOUTHEmbouchure is a concept ordinarily only associated withwind instruments. Wind performers refer to “embouchure”as the interaction between the mouth of a musician and themouthpiece of the instrument [1][10]. It is a highly control-lable coupling mechanism between mouth and mouthpiece,which greatly affects the sound produced.

The hydraulophone, as a water instrument rather than awind instrument, has a number of “mouths” through whichwater enters and escapes, allowing the performer to expres-sively play the instrument without having to continuously“blow” water into the instrument.

Water is required at the user-interface, so that the fullrange of musical expression can be achieved. This is doneby a musician’s interacting with the fluid turbulence, whichleads to acoustic oscillations in the water itself, inside theinstrument, through a mechanism of finger-jet embouchure.

Expression is achieved by rocking the finger over thewater flow, via roll, pitch and yaw, and by approaching thecentre of a jet along different pathways and speeds to soundvarious chiffs at the onset of a note. With practice, a per-former can play a note on hydraulophone while varying itsintensity, timbre and pitch. 2

2Two other consequences are as follows: (1) As with many wind instru-ments, the performer can subconsciously tune a note dynamically when theinstrument’s pitch strays due to varying temperature; (2) By using a fingerto expressively play the instrument’s “mouth”, the performer’s own mouthis freed to sing or speak.

a

CBA

...

...

31F

FLUID CHEST

USER

FINGER HOLE

STEE FITTING

TEE FITTING

HYDRAULOPHONEPIPE

2

STEE DISCHARGE

TMSINGLE−JET H Organ

TM

2H Organ

Figure 1: Acoustic hydraulophone example: The H2Organ consists ofa fluid user-interface [5] which is played by placing one’s fingers on oneor more water jets, and blocking or restricting the water flow. Blocking ajet causes water flow to be diverted into an underwater organ pipe. Thediagram has the first jet shown in a blocked state, where hydraulic fluid isforced through the side discharge to the sound-producing mechanism. Theother jets are shown in an unblocked state where, by way of the Bernoulliprinciple, a small vacuum is drawn on the side-discharge pipe, to silence thecorresponding sound. (Diagram adapted from Canadian Patent 2517501).Lower, a demonstration prototype is being played underwater. Audiencemembers immerse one or both ears in the water in order to hear the instru-ment. For larger audiences that include some non-bathers too far from thepool to hear the sound directly, underwater microphones or hydrophonesmay be used to broadcast the sound to a larger audience.

3. POLYPHONIC EMBOUCHURE

Moving the mouth from the human (musician) to the instru-ment, has already been described in terms of one mouth.The real advantage of the hydraulophone, however, is itsmany “mouths,” each with independently controllable intri-cate embouchure expression.

Thus the hydraulophone combines the intri-cate embouchure control of woodwind instruments with thepolyphony of keyboard instruments, resulting in somethingwe call polyphonic embouchure.

Many woodwind instruments have a row of holes whichare played with a particular somewhat complicated fingeringpattern that must be remembered, for each tone. Since theholes are used in various combinations of this sort, theseinstruments are purely monophonic.

The hydraulophone, however, associates one hole witheach separate tone. (As an educational tool alone, the hy-draulophone thus becomes easier for children to learn toplay simple monophonic melodies.) Moreover, concert hy-draulophones are often built with a chromatic scale, with theholes arranged in a piano-like pattern (Fig. 2).

Even though the hydraulophone resonates like a flute, itcan play chords. What’s more, each tone in a chord can bedynamically and continuously varied, even so much so that

Proceedings of the International Computer Music Conference (ICMC 2009), Montreal, Canada August 16-21, 2009

546

Figure 2: Example of piano-style layout of hydraulophone jets.Concert hydraulophones, and other performance-ready hydraulophones,can be seen at: http://en.wikipedia.org/wiki/Hydraulophone

one chord can be slowly blended into becoming another, soit is impossible to strictly define which chord is being playedat any one time.

3.1. Theoretical Origins of Polyphonic Jet-EmbouchureAn intuitive understanding of polyphonic embouchure per-formance styles comes from examining the mechanism ofwater flow in a single jet outlet channel in a hydraulophone.

The velocity flow profile is distributed unevenly acrossthe jet, and in fact approaches a quadratic (parabolic) dis-tribution, in the limit of laminar, steady-state, incompress-ible, fully-developed, viscous flow. This effect is known asPoiseuille flow, and is illustrated in Fig. 3(a).

This quadratic profile applies to cylindrical, axisymmet-ric fluid channels of constant cross-section, driven by a pres-sure gradient (i.e. a pressurized fluid source), which is thecase inside the hydraulophone’s jet outlet channels. Thesteady-state laminar velocity profile may be expressed as

uz(r) =1

4µ(−d p

dz)(r2

o− r2), (1)

with r as radial distance from centre-axis, ro channel radius,µ fluid viscosity, p pressure, and uz(r) streamwise velocity.

4. PERFORMING MUSIC WITHPOLYPHONIC EMBOUCHURE EXPRESSION

Important musical consequences come from the Poiseuilleflow in hydraulophone jet outlet channels. When a musi-cian touches his or her finger to one of the hydraulophone’sjet “mouths”, there is a non-uniform effect on the intensityof the note (amount of flow diverted to a sounding mech-anism). To a first-order approximation, the note becomeslouder in proportion to the surface area on the jet mouthwhich is restricted. To second-order, we consider the non-uniform Poiseuille outlet flow.

Blocking the innermost area of a jet (where momen-tum would have been highest otherwise) strongly affects thenote amplitude, while blocking the outer areas (radius r ap-proaching ro) has a softer, less marked effect.

A musician gains a distinctly gentle embouchure controlof the note because of this non-uniform response. When oneintrudes his/her finger radially towards the jet opening, then

FluidJet

Uz(r)

Uz(r)

r

z

Fluid source

Jet Outlet

CylindricalFluid channel

ro

(a)

21

Am

Am

#

Dm Am

BAA baa SHEEople, HAVE you any CASH

Am

AmAm

YES sir, YES sir, a THREE bag STASH

EDm

Am AmDm

Am Am EDm

Dm E

#ONE for my COUNTry, and ONE for my STATE;... and

ONE for the LITTle boy with a HIGH tax RATE.

E

#

Am

Legend for playing harmonywith the left hand.(same inversion of chordsas in the melody)

Nessie clef

Angrilyor sadly

Big holes indicate complete blockage of the water jet;Smaller holes indicate less blockage of the water jet. (blockage indicated by hole size)

Music based on "Baa Baa Black Sheep"

"Baa Baa Sheople", −− Steve and Christina Mann, 2004, for hydraulophone, otherwise affectionately known as Nessie the sea monster...

(b)Figure 3: (a) Poiseuille flow profile in a cylindrical water jet outlet. Theparabolic flowfield velocity profile lends itself well to selection of variouspoints along the profile, using finger-jet embouchure.(b) Polyphonic embouchure notation: “Hole-note” notation example.

the note intensity increases gently at first, and, as the flowrestriction increases (until the flow is completely blocked)the note intensity increases more quickly.

Finally, when the finger lies centred over the “mouth”, itis well positioned to control the note quickly and with a largeresponse on even small finger motions (useful if a section ofthe music calls for fast staccato notes) As the finger exits thejet’s cross section, the note dies down gently.

Significantly, the fact that a hydraulophone has many“mouths” means that a hydraulist can play with polyphonicembouchure, with a completely independent rich expressiv-ity for many notes played at once. A hydraulist eventuallydevelops the capability of gently and intricately interleavingthe expression of many notes in and out amongst each otheramong several notes within a chord. One gets the feeling ofhaving an entire ensemble of flutes and flute players, playingat once, all at the control of one’s fingertips.

Pieces of music that require polyphonic embouchure in-clude “Suite for Hydraulophone” (R. Janzen, 2007), and thesame composer’s single-instrument arrangements of multi-instrument pieces such as Faure’s Sicilienne, and the folksong “She’s Like the Swallow”3.

Another new technique, harmelody, evolved through us-ing polyphonic embouchure on compact portable hydraulo-phones with a very limited compass (number of notes).Harmelody is the combination of overlapping harmony andmelody, into “melodically weighted chords”. The melody isembedded within the accompaniment by way of continuousvolume variations in the individual notes of the accompani-ment, independently adjusting each note’s volume, as shownby example in Fig. 4.

3Samples of compositions for hydraulophone can be heard at:http://individual.utoronto.ca/janzen/composing

Proceedings of the International Computer Music Conference (ICMC 2009), Montreal, Canada August 16-21, 2009

547

Figure 4: A unique aspect of the hydraulophone is that the embouchure(finger-jet embouchure) is visible, unlike instruments that use air. The re-sult is a useful visual feedback mechanism to the performer, and a usefulteaching tool for music education. This figure illustrates harmelodic per-formance technique, using two phrases of a simple childrens’ song: (top)“Twinkle twinkle little star” and (bottom) “How I wonder what you are”.In the first figure we see a C-major chord being played with emphasis onthe first note of the melody, C (the flow of the C-jet is fully restricted),while the E and G jets are being partially restricted. In the next figure, thesame three jets are being blocked, but the emphasis is shifted fluidly andcontinuously toward the G jet, without any disruption in the harmony.

4.1. Notation for Polyphonic EmbouchureThe unique ability of the hydraulophone to produce notesthat continuously vary in amplitude, pitch, timbre, etc., re-quires a new form of musical notation.

Figs. 3(b) and 5 show two new musical notation forms,featuring variations on common children’s songs played ina harmelody style. We have devised these two differentwritten forms, as two different ways to communicate mu-sic which uses polyphonic embouchure.

In “hole-note” notation, each note represents how mucha finger should restrict the jet-flow out of a hole. The verticalplacement of notes corresponds to the holes in a left-to-rightseries on the hydraulophone, just as with a piano keyboard.

On a piano, though, all notes decay after being struck,and therefore must be re-struck every time the pianist wantsto update the amplitude. Unlike a piano, the hydraulophonecan sustain its notes for as long the jets are pressed, and con-tinuously responds to the manner in which they are pressed.

These capabilities of the hydraulophone are only partlyexpressed by hole-note notation. Hole-note notation is dis-cretized in time, whereas the actual capability of the hy-draulophone goes beyond this discretization because ampli-tudes can be varied continuously.

“Fluid music” notation is a further improvement overhole-note notation, where continuously changing notes areinstead drawn as a continuous function of time on the staff.The change is drawn as a varying line thickness. The linealways lies with its centre over the correct pitch on the staff.As a departure from existing modern music notations [3],this polyphonic notation is a type of time-frequency (e.g.spectrogram) density plot, with intensity mapped to line thick-ness, in which the frequency axis is logarithmic.

HOW I

Twinkle Twinkle Little Star, traditional, arranged by S. Mann 2004,for hydraulophone (written in fluid music notation).

TWIN− KLE TWIN− KLE LIT− TLE WON− DER WHAT YOU ARE −−−−−−STAR −−−−−

2TT

Figure 5: Example of harmelody represented using “Fluid music” nota-tion. Notes change fluidly and continuously, in order to combine harmonyand melody within an overlapping compass. The fluidly flowing changesin polyphonic intensity lends to the music a uniquely expressive quality.Moreover, components of the harmelody throb in volume, to convey a flu-idly sculpted sense of rhythm. Note the time signature, where the fractionaldenominator, T , is an arbitrary (“undigital”) analog quantity of time, as setforth by the continuous (fluid) nature of the musical process. In particular,notes need not necessarily be discretized in duration.

5. CONCLUSIONFluid user-interfaces were explored. In particular, it wasfound that an array of water jets formed a new and usefulinput device that functioned like a “soft” and expressive key-board, by way of polyphonic finger-jet embouchure.

By having a hydraulophonic sound production process(sound generated purely from vibrations in water) directlycoupled to a user-interface consisting of the same liquidmedium, we created an expressive user-interface that hadauditory, visual and tactile feedback to the musician.

Hydraulophones which were fitted with underwater soundpickups (functioning as input devices for computer musicapplications), and also purely acoustic hydraulophones, weremade capable of highly expressive performance techniques—now used by new compositions which specifically require ahydraulophone for the full expression of the composition.

6. REFERENCES[1] A. Baines, “Embouchure,” The Oxford Companion to

Music, ed. A. Latham, (accessed 2009).[2] R. Janzen, S. Mann, “Arrays of water jets as user inter-

faces,” Proc. ACM MM, Sept. 24-29 2007, pp. 505–8.[3] E. Karkoschka, Notation in new music. NY: Praeger;

Translated by R. Koenig, 1972.[4] T. Machover, “Hyperinstruments,” Cyberarts, Ed.

W. Freeman, San Francisco: Spartan, 1991.[5] S. Mann, “flUId streams,” Proc. ACM MM, 2005, Sin-

gapore, pp. 181 – 190.[6] S. Mann, R. Janzen, “Fluid samplers,” Proc. ACM

MM, Sept. 24-29 2007, Augsburg, pp. 912–921.[7] S. Mann, R. Janzen, R. Lo, “Hyperacoustic instru-

ments,” Proc. IEEE ICME 2008, pp. 89–92.[8] S. Mann, R. Janzen, R. Lo, C. Aimone, “Inventing

new instruments...,” Proc. ICMC, August 27-31 2007,Copenhagen, Denmark, vol. 1, pp. 105–112.

[9] S. Mann, R. Janzen, and M. Post, “Hydraulophonedesign considerations,” Proc. ACM MM, Oct. 23-272006, Santa Barbara, USA., pp. 519–528.

[10] G. B. Webster, “Embouchure,” Grove Music Online,ed. L. Macy, (accessed 2009).

Proceedings of the International Computer Music Conference (ICMC 2009), Montreal, Canada August 16-21, 2009

548