Literature ReviewLewis Sykes, The Augmented Tonoscope

1. A Personalised Introduction

1.1 The purpose of this literature review

Through this literature review I’m attempting to develop a clear line of argument that links with the evidence I’ve uncovered through my reading and online research. I’m trying to develop a critical discussion, showing insight into differing arguments, theories and approaches and a synthesis and analysis of relevant published work - linked to my own purpose and rationale. I’m trying to show I have read and understood relevant published work in my field and am using it to avoid duplication of research, develop a framework for my enquiry, raise questions for further work and suggest further research.

This literature review attempts to define and limit my research, place it in a historical perspective and artistic lineage and relate my own findings to previous knowledge. It serves to contextualise my research and provide the frame of reference from which I operate. Through it I’m also attempting to evaluate promising research methods. Most importantly I’m endeavouring to highlight the need for my own research through a description of existing artwork and literature in the field, indicating how my work builds on past practice and theory and fills an ‘absence’.

1.2 About The Augmented Tonoscope

The Augmented Tonoscope is an artistic study into the aesthetics of sound and vibration through its analog in visual form - the modal wave patterns of Cymatics. Dr. Hans Jenny coined the term Cymatics (Greek: κῦμα “wave”) when he studied this subset of modal wave phenomena using a device of his own design - the ʻtonoscopeʼ. I’m designing, fabricating and crafting a contemporary version of Jenny’s sound visualisation tool - a sonically and visually responsive hybrid analogue/digital instrument that produces dynamic Visual Music - The Augmented Tonoscope. I then plan to play, record and interact with it to produce a series of artistic works for live performance, screening and installation.

I’m asking the overarching research question:How far can artistic investigation into Cymatics - the study of wave phenomenon and vibration - contribute towards a deeper understanding of the interplay between sound and image in Visual Music?

1.3 A Hermeneutical Approach

Despite it’s relatively populist appeal, I’ve found limited academic research into Cymatics and its application in a Visual Music context. So I’ve not taken the traditional approach of studying research in the field in order to reveal an absence and so define a research question. Rather I’ve trusted my implicit practitioner knowledge, gained through my years of practice and curation, to identify the focus of my investigation and then steer my research through the surrounding terrain. Considering my project as a trans-disciplinary topic, I’ve looked to alternative fileds and perspectives to help inform my understanding and provide new insights. As such I’ve drawn on ideas and theories ranging from acoustics, engineering, philosophy and art theory to musicology, cognitive psychology and computer science.

I’ve not followed a traditional model of progressing through the discreet stages of analysis, synthesis, conceptualisation and epistemology applying deductive reasoning to form my argument. Instead, I’ve taken an alternative approach in which I break my overarching research question down in to a series of component parts and then use inductive and abductive reasoning to search for hermeneutic connections between them. In doing so, I’m attempting to describe and define my research as something more akin to an emergent system.

- What insights have I gleaned about Cymatics - the study of wave phenomenon and vibration? - What constitutes artistic investigation into Cymatics? - How have I developed a deeper understanding into the interplay between sound and image? - How can I contextualise this work within Visual Music and wider audiovisual culture?

Like Robert Fludd, the prominent C17th English astrologer, mathematician, cosmologist and Qabalist, I’m interested in an almost alchemic approach to research. Not through the specialism of a single discipline but through broad, diverse and by contemporary standards at least, potentially conflicting lines of enquiry. While I do aspire to audiovisual innovation, I’m pretty certain it won’t be through the (misguided) pursuit of the novel, but more likely by discovering ‘unfound’ linkages between dissociated strands of research. I intend to show suitable evidence that this strategy has helped me to both identify, explore and hook into underlying paradigms as well as uncover markers and hints towards creative possibilities that can inform and guide my ongoing research and practice.

Sources are distributed throughout with key ones marked in bold. The use of the term ‘this research’ refers exclusively to The Augmented Tonoscope.

2. Insights into Cymatics - the study of wave phenomenon and vibration

The study of modal wave phenomena has a long and rich history and a significant body of empirical evidence, theory and practical application most notably: - Robert Hookeʼs observation in 1860 of nodal patterns associated with the modes of vibration of glass

plates:• Daintith, J. and Gjertsen, D. (eds.) (1999) Oxford Dictionary of Scientists. Oxford University Press,

p. 101; - Ernst Chladni’s development in the 1780s of a technique of drawing a bow over a thin metal plate

sprinkled with sand and his documentation of the patterns that emerged, from which he developed Chladni’s Law, a simple algebraic relation for approximating the modal frequencies of the free oscillations of plates and other bodies, as detailed in:• Chladni, E.F.F. (1787) Entdeckungen über die Theorie des Klanges (“Discoveries in the Theory of

Sound”). Leipzig: Weidmanns Erben und Reich. Chladni’s Law may well provide the essential mathematics to be modelled in the virtual physics of the

emulated tonoscope, realised in code with the help of examples from: • Soedel, W. (2005) Vibrations of Shells and Plates. New York: Marcel Dekker Inc., c. 11.4 • Raichel, D.R. (2006) The Science And Applications Of Acoustics. New York: Springer Science+Business

Media Inc., c. 6; - John William Strutt aka Baron Rayleigh’s two-volume, major treatise on sound and vibration in the late

1870s, still considered a classic and including a chapter on the vibrations of plates:• Strutt, J.W. (1945) The Theory Of Sound, 2 Vols. New York: Dover Publications;

- and Margaret Watts-Hughes, inventor of the Eidophone in 1885, who writes in a 1891 magazine article about her work “...the extreme sensitiveness of the eidophone as a test for musical sounds detecting and revealing to the eye... what the ear fails to perceive” (The Century, 1891: 39) suggests the potential, enhanced through the capabilities of modern design, for an instrument of particular sensitivity and subtlety:• Hughes, M.W. (1891) “Visible Sound. I. Voice Figures”. The Century 42(1), 37-40. [Online]. http://

digital.library.cornell.edu/cgi/t/text/pageviewer-idx?c=cent;cc=cent;rgn=full%20text;idno=cent0042-1;didno=cent0042-1;view=image;seq=00047;node=cent0042-1%3A1 [Accessed 7 April 2011]

Absolutely central to any discussion on Cymatics is Hans Jenny - the Swiss medical doctor turned Cymatics researcher who’s empirical studies into the effects and manifestations of sound and vibration on matter still define the field.

• Jenny, H. (2001) Cymatics: A Study of Wave Phenomena and Vibration. New Hampshire: Volk, J. (firstprintedasVol.11967,Vol.21972)

Despite a recognition of the effects of sound on materials dating back to Leonardo Da Vinci and including observations, mathematical models, treatise and practical experiments by many others, including the aforementioned pioneers, Dr. Hans Jenny’s two volumes on Cymatics - a term coined by Jenny himself to describe the study of visible modal wave phenomena (to kyma, the wave; ta kymatika, matters pertaining to waves, wave matters), are commonly accepted as the defining works in the field. His work has no prior or subsequent match in terms of a detailed empirical investigation into the effect of sound and vibration on physical matter and the phenomena he investigated, documented and contemplated therein has been a source of inspiration for many, including artists, ever since.

Key to Jenny’s analysis of Cymatic effects are notions of the underlying periodic phenomena in nature expressed through common rhythmicity, oscillation and seriality. Jenny himself hoped that his research into Cymatics would open the eyes of others to these underlying periodic phenomena. He emphasises the ‘triadic nature’ of Cymatics - hear the sound, see the pattern, feel the vibration - highlighting three essential aspects and ways of viewing a unitary phenomenon and suggesting the potential for a multi-modal sensory instrument. Jenny also considered the whole system in his analysis, taking account of the complex structures within the vibrational field and the shift from form to form, not just the discrete geometric patterns that emerged at certain frequencies, “Hence it is possible not only to produce vibration patterns and investigate the laws to which they continuously conform, but also, and more especially, to make a close study of the transitions as one figure gives way to another” (Jenny, 2001: 22).

Particularly relevant for a Practice as Research project he favoured a practical experimentation over a mathematical model, “What we are concerned to do then, is not to formulate hypotheses about backgrounds and final causes, but rather to press on step-by-step with our exploration into this field and to find methods of giving tangible expression to this phenomenology” (Jenny, 2001: 20).

This combined body of research has inspired contemporary scientific research into Cymatics as a means to reveal a deeper understanding into areas such as acoustics:

• Reed, J.S. (2008) CymaScope. [Online]. http://www.cymascope.com/ [Accessed on 28 October, 2010]

John Stuart Reed is coinventor of the patented Cymascope (an instrument that makes sound visible) and arguably the current world leader in Cymatic research focused on scientific exploration such as the interpretation of dolphin communication. However, Reed’s approach is in stark contrast to the experimental artistic approach and ‘open source’ development that this research proposes.

3. Artistic investigation into Cymatics

Cymatics has inspired and informed artistic investigation and research across a range of associated disciplines - most notably architecture and music - in the exploration of areas such as spatial form and harmonicism:

• Goldstein, B. (2009) Cymatica. [Online]. http://cymatica.net/ [Accessed on 1 March, 2011]

Benlloyd Goldstein cites Goethe - “Architecture is the frozen music; music is the flowing architecture…” as rationale for his architectural thesis investigation exploring the synthesis of spatial proportion and form generated from sound. By declaring a set of definitions for terms such as ‘Wave Lattice’ and ‘Cymatics’ he formalises his approach “In Search of Cymatic Architecture” (Goldstein, 2009:online) and succinctly describes his evolving practice. Goldstein declares “The goal of unlocking the cymatic language is not to display truth or beauty - but to be able to translate and re-interpret the communication of sound through form; to better understand how to create atmosphere and effect, to compose architecture more rapidly; to play by ear” (Goldstein, 2009:online).

Not only are Goldstein’s sentiments appealing but the ‘harmonic proportion’, ‘waves’ and ‘cymatic language’ terminology of his thesis has plenty of resonance with this research. Particularly interesting is his use of

code in the “attempt to interpret the intelligent proportions of waves we commonly experience as sound into expressions of formal proportion in architecture” (Goldstein, 2009:online) realised as images, animations and 3D printed cymatic forms. An especially useful and interesting crossover lies in his explorations of Cymatic motion - a point of commonality with the musical rather than architectural focus of this research.

• Telfer, J. (2010) Cymatic Music. [Online]. http://www.cymaticmusic.co.uk/ [Accessed on 1 March, 2011]

John Telfer’s research - “an audiovisual science and music project that investigates the possibilities of creating a system of visual, or rather visible music” (Telfer, 2010:online) - is of specific interest and provides much food for thought. Telfer also asks if cymatic patterns can be interpreted musically but argues that the Equal Temperament, the twelve equal steps of a keyboard octave embedded in the piano and fretted string instruments which have dominated Western Music since the C18th, obscures the fact that this musical structure has harmonic inconsistencies. Telfer asserts “it is worthwhile developing a music system with firm acoustic foundations, particularly if this system is to be used as a tool for cymatic enquiry” (Telfer, 2010:online). He responds with his theory of harmonicism, a mathematically and musically consonant system of proportional Just Intonation, and his work in developing the two distinct harmonic (overtone) and lesser known arithmetic (undertone) progressions of the Pythagorean Lambdoid, which re-emerged in the C19th in its completed form as the Lambdoma matrix, as a practical creative resource in music.

While this research is unlikely to involve an investigation of micro-tonality, Telfer’s research is compelling, particularly if it does indeed “take advantage of the creative cross-fertilisation between musical harmony and physical patterning” (Telfer, 2010:online), though it does suggest that a correspondence between tone and cymatic pattern and form will most likely not be found within the Even Temperament of Western Music but within other musical traditions.

Furthermore, Telfer favours manufacturing acoustic musical instruments to interpret the harmonicism within the Lambdoma matrix, but recognises and points to, the possibility of an electronic (and digital) approach which this research plans to adopt.

There’s also inspiration and insights to be gleaned from the realisation, process and methodology of wide range of contemporary artistic output exploring Cymatics: • Goto, S. (2011) Cymatics. [Sound installation]. http://www.toshare.it/cymatics/index.html [Accessed on

13 April, 2012]• Joynes, G.J. (2011) Frequency Painting Series. [Sound installations]. http://clinkersound.com/ [Accessed

on 2 May, 2011]• Blore, D. and Meinema, J. (2009) Cymatics.org. [Photography]. http://cymatics.org/ [Accessed on 28

October, 2010]• Marko, A. (2008) Kymatika. [Live performance and video]. http://vimeo.com/2919023 [Accessed on 28

October, 2010]• Meehan, J. (2008) Journal of Cymatics. [Online]. http://cymatica.com/ [Accessed on 28 October, 2010]• Richards, J. and Wright, T. (2008) Cymatic Controller. [Instrument] http://www.mti.dmu.ac.uk/~jrich/

instru.html [Accessed on 7 February, 2011]• Richards, J. and Wright, T. (2008) Auto, Crack, Slimework and the Seven Leg Spider. In the 14th

International Symposium on Electronic Art. Proceedings of the ISEA2008 Conference, pp. 388-9. [Online] [Accessed on 7 February, 2011] Available from: http://www.isea-webarchive.org/mmbase/attachments/115304/Auto,_Crack,_Slimework_and_the_Seven_Leg_Spider_-_John_Richards_Tim_Wright.pdf

• Hall, A. (2003) Coffee Cup Oscillator. [Sound installation]. http://www.antonyhall.net/coffee-oscillator.html [Accessed on 7 February, 2011]

• McIntosh, T. with Hynninnen, M. and Madan, E. (2002) Ondulation. [Sound installation]. http://www.ondulation.net/index.html [Accessed on 7 February, 2011]

• Nicolai, C. (2000) Milch. [Lambda print on aluminium]. http://www.carstennicolai.de/?c=works&w=milch [Accessed on 7 February, 2011]

As well as software and ‘open source’ creative coding modelling its effects and manifestations:• Falstad, P. (n.d) Math & Physics Applets. [Online]. http://www.falstad.com/mathphysics.html [Accessed

on 28 October, 2010]• Wakefield G. (2009) Chladni 2D/3D. [Online]. http://www.mat.ucsb.edu/~wakefield/investigations.html

[Accessed on 28 October, 2010] Graham Wakefield’s Max/MSP patches which simulate a Chladni plate pattern in 2D and speculate

what the equivalent might be for 3D have a particularly appealing aesthetic.• Dill, K. and Mikhailov, S. (2008) Software Tonoscope. [Online]. http://softwaretonoscope.com/ [Accessed

on 28 October, 2010]• Hodgin, R. (2005-2011) Flight404 blog. [Online]. http://www.flight404.com/blog/ [Accessed on 28

October, 2010]• Schmidt, K. (2000-2005) Toxi archive. [Online]. http://toxi.co.uk/ [Accessed on 28 October, 2010]• Schmidt, K. (2008-2011).PostSpectacular. [Online]. http://postspectacular.com/ [Accessed on 28

October, 2010]

3.1AresponsetoSuguruGoto’sCymatics

Goto’s current interactive sound installation is a contemporary artistic investigation into Cymatics that is most akin to The Augmented Tonoscope. Since it potentially replicates this research it requires a specific response. While not seen in person, online research via its website, press materials and associated video documentation has enabled an understanding of its conceptual frameworks, technological underpinnings and production values.

Cymatics (Goto, 2011) is realised as two discrete but allied works, each housed within the gallery space in its own approximately 3m3 box - one black the other white - each with a single door. Within the black box is a ~1m2 metal tray mounted into the top of a waist high plinth. Water in the tray is vibrated by a series of computer controlled transducers underneath which playback a pre-arranged musical composition, resulting in a shifting vibrational pattern across the surface of the liquid. This is simultaneously projected in close up on one wall of the room. Within the white box is a waist high ~1m2 plinth with four ~10” speakers mounted in the top. Non-Newtonian fluid is vibrated within each of the vinyl covered cones producing dynamic, globular forms that rise and fall out of the material. Custom applications on an iPad in both of the rooms allows interaction with the works and so the resulting cymatic forms through control of the audio.

Cymatics (Goto, 2011) is a commissioned art work - supported and produced by Action Research, an initiative sponsored by the Turin Chamber of Commerce. “Production of the work was developed over four stages: design; structural testing at the Interdisciplinary Mechatronics Lab (LIM) of the Polytechnic of Turin (Verres campus); assembly and finishing at the Silvano Bauducco company; and finally, musical composition and rehearsals at Hiroshima Mon Amour” (Cymatics, 2011:online). As such there’s been a reasonably substantial investment of time and money spent in developing, fabricating and producing the work as well as a P.R. and Marketing campaign to promote it.

It’s clear that the work employs a relatively sophisticated combination of off-the-shelf, commercial wave drivers, transducers, speakers and amplifiers tested and developed in a well equipped lab with professional tone generation and testing equipment. A production video shows Goto overviewing the testing alongside a team of technicians responsible for the actual construction of the devices. This documentation is open enough to show a series of unsuccessful attempts, particularly with the non-Newtonian fluid, before the final configuration and technical set up was settled upon. Goto created music compositions to drive both devices (and though the sound quality of the documentation isn’t that clear) with a heavily sequenced score using mechanical and industrial sounds, perhaps influenced by his recent “RoboticMusic” works. The materials chosen to demonstrate the cymatics effects - water and a non-Newtonian fluid (most likely corn starch in water) - are very conventional in Cymatics terms and the outputs of the works are analogue only. While input via an iPad control interface is certainly contemporary, they appear to be virtual control interfaces emulating traditional physical hardware of buttons, knobs and sliders.

The language employed in describing the works is effusive and almost poetic “The creative process involved building a synergy between music to be seen and images to be heard, so as to create a sensorial performance for the complexity of human perception.” (Cymatics, 2011:online). It also makes some rather ambitious claims for the work in creating a connection between the material, technological, artistic and spiritual “Cymatics creates real spaces that are metaphysical and spiritual at the same time. A place where art is a bridge between the material and the spiritual, between technology and nature, and between the humanities and science.” (Cymatics, 2011:online) though this may be the P.R. agency’s spin on Goto’s sentiments.

Finally, Cymatics (Goto, 2011) seems to be quite a divergence from Goto’s more contemporary practice of robotic musicians, technologically augmented performance and telematic presences in augmented realities, though it may mark a return to his earlier instrument based work such as The Superpolm, a virtual violin built in 1996.

• Piemonte Share (2011) Cymatics - Production. [Online]. http://vimeo.com/32333689 [Accessed on 12 April, 2012]

• This Fluid World, Imperica, 2011. [Online] http://www.imperica.com/features/this-fluid-world [Accessed on 12 April, 2012]

• Goto, S. (2005) Suguru Goto website. [Online]. http://suguru.goto.free.fr/Contents/SuguruGoto-e.html [Accessed on 12 April, 2012]

In some ways this all sounds remarkably familiar. However there are marked differences between Cymatics (Goto, 2011) and The Augmented Tonoscope, as outlined in the RD1 and arguably still intact, which reaffirm this research’s authenticity and uniqueness in the field.

The Augmented Tonoscope is primarily intended as an audiovisual instrument - a means to perform live and produce refined audiovisual works in the form of short audiovisual films. While it’s likely to be exhibited as an interactive sound installation, it’s only one iteration of several possible outputs for the research. Moreover, the main focus of this research is the integration of a virtual tonoscope into the physical one - the analogue cymatic patterns of the latter captured and analysed and then projected and superimposed with digital cymatic forms derived from the former secondary but integrated emulated tonoscope. This virtual system, modelled according to real world physics and mathematical laws, will behave like the analogue device, yet have the ability to be extended, twisted and abstracted in ways the analogue never could.

So despite the digital control aspects of Goto’s Cymatics, it is the close integration of analogue and digital in The Augmented Tonoscope that is an essential difference. Accordingly, this research argues that it is this real-time, dynamic and aesthetic interplay between audio and augmented cymatic visual outputs that has the potential to provide new insights and understanding into the relationship between sound and image within Visual Music - which can only be addressed by combining the analogue and digital domains through the design, fabrication and crafting of a new hybrid device.

Furthermore, this research involves building an instrument from scratch, with minimal help and at own expense and with a distinct bias towards technology misuse - “the use of technological systems and products in ways that were never intended by their manufacturers” - by repurposing tools, modifying commercial products and using ‘found’ items for purposes other than that for which they were designed. In a search for subtle differences that may result in new behaviours, this research avoids using conventional mediums to visualise physical cymatic effects, (a prerequisite for most studies to date - including Goto’s), preferring to experiment with unconventional materials such as the glass beads most frequently used in highly reflective motorway signs. Also early prototypes indicate a complex union of modular components that will collectively produce the functionality of The Augmented Tonoscope - more akin to the haptic, tactile and almost organic form of the modular synthesiser than to a reductionist, minimal, integrated audiovisual system such as Goto’s Cymatics.

Finally, there’s no spiritual dimension and only a passing nod to metaphysical contemplation in this research, the critical focus being the aesthetic, artistic and scientific aspects. Cymatic effects may well induce awe in the viewer by revealing a previously hidden aspect of nature, but The Augmented Tonoscope isn’t intended

as a modern counterpart, as Goto’s Cymatics arguably is, to those Devices of Wonder - “the surprising and seductive ancestors of modern cinema, cyborgs, computers and other optical devices” (J. Paul Getty Trust, 2001:online) explored through this online exhibition. The focus is on an artistic exploitation of an infamous natural physical phenomenon for creative purposes - to generate audiovisualisations in which there is a real-time, direct and elemental link between what you see and what you hear.

• Devices of Wonder exhibition, Getty Research Institute, November 13, 2001, through February 3, 2002 [Online] http://www.getty.edu/art/exhibitions/devices/html/index.html [Accessed on 12 April, 2012]

3.2 In search of a direct visual equivalence

In exploring what constitutes an artistic investigation into Cymatics and how it might be used to create a correspondence between sound and image this research asks the question: - Is it possible to develop a cymatic visual equivalence to the auditory intricacies of melody, harmony and

rhythm?

Searching for prior attempts to find visual equivalence to auditory intricacies has led to an exploration of the artistic lineage of this research by looking back to the work of seminal experimental filmmaker such as Rudolph Pfenninger and Oskar Fischinger and particularly their synthetic sound production experiments. Using a technique of ‘direct sound’, of printing regularly repeating geometric patterns directly into the optical soundtrack of the film, Pfenninger and Fischinger used the photoreceptor of the projector to turn visual forms directly into music and so explored a direct visual correspondence to sound and music.

Fischinger generally held his abstract imagery to contain qualities parallel to the ones found in music and the actual soundtrack used as means to attract the attention of the audience to recognise this, but he especially believed his synthetic sound production experiments held extraordinary potential for the future of musical composition and sound analysis: “Now control of every fine gradation and nuance is granted to the music-painting artist, who bases everything exclusively on the primary fundamental of music, namely the wave - vibration or oscillation in and of itself.” (Fischinger, 1932:online).

While The Augmented Tonoscope aspires to approach something of the artful mastery of Oscar Fischinger’s films - as Moritz (1976) observes: “Ah, but those visuals contain formulas and gestures that communicate with us subconsciously, directly, without being appreciated or evaluated.” (Moritz, 1976: online) - Fischinger and many of his contemporaries were guarded and secretive about their methodologies and production processes. They were averse to reveal the intricacies of their techniques and details of their machinery and so accordingly there is little surviving documentation of how they actually made their films.

• Fischinger, O. (1932) “Klingende Ornamente”, Deutsche Allgemeine Zeitung, Kraft Und Stoff. 30, [Online]. http://www.centerforvisualmusic.org/Fischinger/SoundOrnaments.htm [Accessed on 7 Apr 2011]

• Moritz, W. (1976) “The Importance of Being Fischinger”, Ottawa International Animated Film Festival Program, 2-6. [Online]. http://www.centerforvisualmusic.org/library/ImportBF.htm [Accessed on 10 March 2011]

3.3 On the complementarity of music and visual art

So despite a strong corollary and artistic lineage for this research in the work of Fischinger and Pfenninger using the technology and mediums of their time, a deeper pedagogic connection has been sought in the more recent work and writing of John Whitney Sr. - the seminal experimental computer-aided filmmaker. His work holds particular resonance for this research because it exists at the interface between analog and digital technologies - albeit at the infancy of this endeavour.

• Whitney,J.(1980)Digital Harmony: On the Complementarity of Music and Visual Art. Peterborough, N.H.: McGraw-Hill

Whitney generated his motion graphic patterns through programmes of simple algorithms run on early computers output to a CRT monitor which he captured frame-by-frame via a film camera. He then applied his mastery of experimental filmmaking to select, optically print, colour, overlay and edit these sequences into his completed works. Whitney describes the process, “a library of… many trial/action sequences was accumulated on film over a period of months… each action was selected by that slow process of judging one at a time… of selecting only those which seem to fit my growing discernment of the broad design for a composition.” (Whitney, 1980: 8).

He created a series of remarkable 16mm films of abstract animation that used early computers to create a harmony - not of colour, space, or musical intervals - but of motion. He championed an approach in which animation wasn’t a direct representation of music, but instead expressed a “complementarity” - a visual equivalence to the attractive and repulsive forces of consonant/dissonant patterns found within music.

In Digital Harmony, John Whitney Sr. argues that harmony’s charge and discharge of tension in the tonal and chordal sequences of music could be matched in visual art and particularly within computer generated animation. He proposed the idea of harmonic correspondence: “This hypothesis... assumes the existence of a new foundation for a new art… a broader context in which Pythagorean Laws of Harmony operate… that the attractive and repulsive forces of harmony’s consonant/dissonant patterns [also] function outside the dominion of music”. (Whitney, 1980: 5).

Though Whitney’s films brilliantly demonstrate how computer animation can effectively create a temporal sensation of attraction and repulsion, of tension and resolution, within his lifetime he never managed to take the next step of creating music to correspond directly to those visual images. John Whitney recognised that his work represented the infancy of an art form, but suggested that,“Composers will discover a congruence of aural-visual partnership... grounded on valid harmonic interrelationships equally applicable to sound and image.” (Whitney, 1980: 18).

As an artist, John Whitney Sr. is particularly open about his methodology and approach - as he declares in Digital Harmony, ”the purpose is to document my own approach and propose the seminal idea of making an approach.” (Whitney, 1980: 5). So Whitney is an inspiration in that he not only shared his attempts “to define, as much as I understand them, the principles of harmony as they apply to graphic manipulation of dynamic motion pattern by computer.” (Whitney, 1980: 6) but that despite embracing new technology he applied his experience and expertise in the traditions of filmmaking to maintain the integrity of his creative outputs without succumbing too far to the lure of novel technology. This is both a legacy and challenge.

What is particularly intriguing about Whitney’s ‘complementarity’ is the notion of a visual equivalence to the consonant and dissonant patterns found within music. Is there a parallel between Whitney’s dynamic motion algorithms and the static and dynamic cymatic figures and forms produced by sound? Is it possible to match both the charge and the discharge of tensions within tonal and chordal sequences cymatically?

3.4 The Pythagorean Law of Harmony

N.B.: An analysis of the Pythagorean Law of Harmony and its impact on 2,500 years of Western Music Theory is not really appropriate in this Literature Review, though it will most likely constitute part of separate chapter in the final thesis. However, its centrality to this research justifies an attempt to overview the essentials through a reduction of select sources and useful, albeit nonacademic Wikipedia references. Without this section this literature review is ~10,000 words.

Pythagoras is commonly held to be the first individual to study the relationship between music and mathematics, though the principles were previously utilised by the Mesopotamians and Indians centuries earlier. He established that the perfect consonances of music were simple, whole-number ratios - octave (1:2), perfect fifth (2:3) and perfect fourth (3:4) - and that significant musical relationships, such as the tonal pitches of a given scale and the string lengths used to produce them, could be described mathematically. Maor (2007) concurs “Pythagoras concluded that numerical ratios rule the laws of musical harmony - and by extension the entire universe. It was to become an idée fixe with the Pythagoreans and the cornerstone

of their world picture.” (Maor, 2007: 17). Aristotle claimed that Pythagorean ideas exercised a marked influence on Plato - and so through him as a consequence, all of Western philosophy.

Harmony - derived from the Greek ἁρμονία (harmonía), meaning “joint, agreement, concord”, from the verb ἁρμόζω (harmozo) “to fit together, to join” - is a central tenant in the Pythagorean view of the universe. The term was often used for the whole field of music, while “music” referred to the arts in general. Musical harmony is the use of simultaneous pitches (tones, notes) or chords, which permit harmoniousness (sounds that ‘please’), by conforming to certain preestablished compositional principles such as consonance. Harmony is often said to refer to the ‘vertical’ aspect of music, as distinguished from melodic line, or the ‘horizontal’ aspect.

Musical tunings based on Pythagorean perfect consonances, in which the frequencies of notes are related by ratios of small whole numbers, is known as pure intonation or Just Intonation (sometimes abbreviated as JI). Any interval tuned in this way is called a pure or just interval and the two notes in any just interval are members of the same harmonic series. Harmonic intervals come naturally to horns and vibrating strings. As such there are several conventionally used instruments which, while not associated specifically with Just Intonation, can handle it quite well, including the trombone and the violin family. A cappella groups that depend on close harmonies, such as barbershop quartets, usually use Just Intonation by design.

A basic system of Just Intonation is Pythagorean tuning, perhaps the first tuning system to be theorised in the West, in which all the notes within its 12-tone scale can be found using the powers of the ratio 3:2, the interval of the perfect fifth (the next simplest after the ratios 2:1 which produces ascending octaves and 1:2 which produces descending octaves). Pythagorean tuning may be regarded as a ‘3-limit’ tuning system, because the ratios are obtained by using only powers of ‘n’, where ‘n’ is at most 3.

So starting from D (an example of D-based tuning), six other notes are produced by moving six times a ratio 3:2 up (an ascending fifth - i.e. an increase in frequency by a perfect fifth) and the remaining notes by moving six times a ratio 2:3 down (a descending fifth - i.e. a decrease in frequency by a perfect fifth), giving the notes sequence:

Ab- Eb—Bb—F—C—G—D—A—E—B—FB—CB—GB

This succession of twelve 2:3 and 3:2 intervals - each seven chromatic notes apart - spans a wide range of frequencies and so these notes are then adjusted back into a single or ‘base’ octave range by applying multiples of the ascending or descending octave ratios 1:2 or 2:1. To produce a twelve tone scale, one of the enharmonic (equivalent but differently named) notes at each end of this sequence is arbitrarily discarded - usually the Ab. But because a system of perfect fifths doesn’t quite fit into an octave range there is a small difference in pitch between the Ab and GB, known as the Pythagorean comma - about a quarter of a modern semitone of 100 cents. This means that one of the fifths, the wolf fifth (in D-based tuning specifically from GB to Eb) is badly out-of-tune.

So a drawback of Pythagorean tuning is that the wolf fifth in this scale is unusable (though overall the tuning system is manageable when the music isn’t particularly harmonically challenging) which in turn leads to the more significant limitation that it’s not possible to change key without retuning the instrument. Attempts to overcome these inconsistencies and limitations led to a series of alternative tuning systems and ultimately to the adoption by Western Music in the C18th of the twelve-tone Equal Temperament - a system of tuning, in which the octave is divided into a series of 12 equal steps with identical frequency ratios between successive notes.

However, there have also been efforts to extend the basic system of Just Intonation in more sophisticated ways to create more flexible 12-tone and multi-tonal scales - including 5-limit, 7-limit, 17-limit - as well as specific JI tuning systems employed by the likes of Harry Partch, an American twentieth-century composer and instrument creator. Partch was one of the first composers to work extensively and systematically with micro-tonal scales, writing much of his music for custom-made instruments that he built himself, tuned in 11-limit (43-tone) Just Intonation.

This musical evolution results in a rich Western Music history and tradition centred on Just Intonation - from the Pythagorean tuned music of ancient Greece, through plainsong - the body of chants used in the liturgies of the early Catholic Church, the subtle harmony through inference rather than full chordal structures conveyed through the counterpoint of the Baroque, to contemporary composers such as Harry Partch exploring micro-tonal composition developed from sophisticated Just Intonation scales. There’s also numerous other non-Western musical traditions, such as South Asian art music and the Arabic Maqam scales, which while not wholly based on Just Intonation do rely heavily on the just interval within their organisation.

A key issue then, is how might the principles on Just Intonation impact on The Augmented Tonoscope? How will this shape its development? What existing instruments will it be most akin to? Which musical traditions might best support the search for an equivalence between musical tone, analogue cymatic pattern and digital motion graphics?

• Maor, E. (2007) The Pythagorean Theorem: A 4,000-Year History. New Jersey: Princeton University Press• Mathieu, W.A. (1997) Harmonic Experience: Tonal Harmony from Its Natural Origins to Its Modern

Expression. Inner Traditions International• Achilles, C. (n.d.) harmonictheory.com. [Online] http://www.harmonictheory.com/introduction/ [accessed 08-04-12]• Pythagoras, Wikipedia. [Online]. http://en.wikipedia.org/wiki/Pythagoras [accessed 08-04-12]• Harmony, Wikipedia. [Online]. http://en.wikipedia.org/wiki/Harmony [accessed 08-04-12]• Harmonic series (music), Wikipedia. [Online]. http://en.wikipedia.org/wiki/Harmonic_series_(music)

[accessed 08-04-12]• Musical acoustics, Wikipedia. [Online]. http://en.wikipedia.org/wiki/Physics_of_music [accessed 08-04-12]• Music and mathematics, Wikipedia. [Online]. http://en.wikipedia.org/wiki/Mathematics_of_musical_

scales [accessed 08-04-12]• Just intonation, Wikipedia. [Online]. http://en.wikipedia.org/wiki/Just_intonation [accessed 08-04-12]• Pythagorean tuning, Wikipedia. [Online]. http://en.wikipedia.org/wiki/Pythagorean_tuning [accessed

08-04-12]• Five-limit tuning, Wikipedia. [Online]. http://en.wikipedia.org/wiki/5-limit_tuning [accessed 08-04-12]• Equal temperament, Wikipedia. [Online]. http://en.wikipedia.org/wiki/Equal_temperament [accessed 08-04-12]• Enharmonic, Wikipedia. [Online]. http://en.wikipedia.org/wiki/Enharmonic [accessed 08-04-12]• Music of ancient Greece, Wikipedia. [Online]. http://en.wikipedia.org/wiki/Music_of_ancient_Greece

[accessed 08-04-12]• Harry Partch, Wikipedia. [Online]. http://en.wikipedia.org/wiki/Harry_Partch [accessed 08-04-12]• Hindustani classical music, Wikipedia. [Online]. http://en.wikipedia.org/wiki/Hindustani_classical_music

[accessed 08-04-12]

3.5 Beyond Whitney

Also of interest is the work of composer, writer and video artist Bill Alves - one of Whitney’s last collaborators before his death in 1995. It’s been particularly useful to re-read Whitney’s ideas and approaches summarised so succinctly by Bill Alves and recognise his influence and legacy through Alves video work. Alves built upon the principles outlined by John Whitney Sr., extending his differential dynamics into three-dimensions and exploring new interpretations of dynamic Just Intonation.

“The fact that whole number proportions create these arresting patterns of visual resonance suggest a correspondence, or complementarity, to consonant musical sonorities created by whole number frequency ratios, that is, Just Intonation…. In my first video based on these principles, Hiway 70 (1997), I extended the polar coordinate curves of Whitney’s Permutations to three-dimensional graphics. But the most important way in which my work was distinguished from his is that, approaching this work as a composer, I created a soundtrack in tandem with the visual composition, carefully synchronising movement between points of tension and dissonance and points of stability and tonal consonance. I created the music entirely in Just intonation, using harmonies which were often direct analogues of the patterns of visual symmetry”

• Alves, B. (2004) Digital Harmony of Sound and Light. Computer Music Journal, Volume 29, Number 4, Winter 2005

4. A deeper understanding into the interplay between sound and image

By linking sound and image in such a direct and elementary way this research hopes to find an audiovisual amalgam that engages the viewer in a subtlety shifted way - a synchronisation between the senses of sight and hearing that results in a ‘co-sensing’ of a ‘co-expressiveness’ - where the mind is not doing two separate things, it’s doing the same thing in two ways. While this argument is proposed by David McNeill (1992) in his study into the relationship between speech and gesture it seems to particulalry resonate with this research.

• McNeill, D. (1992) Hand and Mind: What Gestures Reveal about Thought. Chicago, The University of Chicago Press

Yet this idea also seems to be supported, somewhat inadvertently, by contemporary cognitive psychology research such as key papers by MacDonald and McGurk (1978) and Shams et al. (2000) and their seminal studies into sensory illusions - now commonly termed the ‘McGurk Effect’ and ‘Double Flash Effect’ respectively. This research investigates sensory modality - how we attain an awareness of the world around us by interpreting sensory information.

4.1 Sensory integration

A reading of these papers and other recent studies listed below indicates that our senses aren’t distinct - they interact with and influence one another all the time. When we perceive images and sounds occurring coincidentally, either in space or in time, they are often bound together to form a multi-modal percept - a multi-sensory object of perception. Evolution has designed this innate multi-sensory integration to take advantage of situations where different senses can provide information on the same object.

It’s relatively easy to recreate these multi-sensory illusions - the ‘McGurk Effect’ and ‘Double Flash Effect’ - that demonstrate multi-sensory integration in action, although it requires laboratory conditions to also provide an opportunity to test the four hypotheses that each advocate a condition for why one sense is dominating over another:1. the discontinuity hypothesis - advocates that the sense in which stimulation is discontinuous i.e. the

most broken up dominates;2. the modality appropriateness hypothesis - states that the sense more appropriate for the task at hand

dominates;3. the information reliability hypothesis - claims that the sense providing the more reliable information dominates;4. the directed attention hypothesis - states that the sense in which there is most competition between

multiple stimuli and so which requires the most attention is dominant.

Andersen et al. (2004) suggest that all of these multi-sensory integration hypotheses should be considered as factors which contribute to the relative dominance of each sense and shouldn’t be considered all-or-nothing conditions.

So are there useful principles for artists in the audiovisual neuroscience and psychology literature? While artists may seek much more explicit large scale behavioural rules than scientists can with confidence supply there are certainly markers and hints towards creative possibilities.

This raises intriguing questions about whether it is possible to link the senses together more closely by exploiting the way in which sensory integration happens innately. Can a knowledge of multi-sensory integration be applied in such a way as to produce creative outputs that exploit this phenomenon for artistic purposes? Is it possible, with artistic intent, to manipulate a ‘multi-sensory object of perception’? If so, what will it be and how will it behave?

• Andersen, T.S. et al. (2004) Factorsinfluencingaudiovisualfissionandfusionillusions. Cognitive Brain Research, Vol. 21, Issue 3, pp. 301–308

• Collins, N. and d’Escriván, J. (eds.) (2007) The Cambridge Companion to Electronic Music. Cambridge University Press

• Hankins, T.L. and Silverman, R.J. (1995) Instruments And The Imagination. New Jersey: Princeton University Press

• Kastner, S. et al. (1998) Mechanisms of Directed Attention in the Human Extrastriate Cortex as Revealed by Functional MRI. Science, Vol. 282 No. 5386 pp. 108-111, 2 October - http://www.sciencemag.org/content/282/5386/108.abstract [accessed 10 November 2011]

• MacDonald, J. and McGurk, H. (1978) Visualinfluencesonspeechperceptionprocesses. Perception & Psychophysics, Vol. 24 (3), pp. 253-257

• Shams, L., Kamitani, Y. and Shimojo, S. (2000) Illusions: What you see is what you hear. Nature, Vol. 408, Dec. 14, p 788

• Shams, L., Kamitani, Y. and Shimojo, S. (2002) Visual illusion induced by sound. Cognitive Brain Research, Vol. 14, Issue 1, pp. 147–152

and useful, albeit nonacademic references:• Is Seeing Believing?, BBC, Horizon (2010-11). [Online]. http://www.bbc.co.uk/programmes/b00vhw1d

Available from: http://www.youtube.com/watch?v=G-lN8vWm3m0&feature=related (Accessed on 10 November, 2011]

• The Eye Hears, the Ear Sees (1970) IMDb. [Online]. http://www.imdb.com/title/tt0200625/ [accessed 10 November, 2011]

• McGurk effect, Wikipedia. [Online]. http://en.wikipedia.org/wiki/McGurk_effect [accessed 10 November, 2011]

• Perception, Wikipedia. [Online]. http://en.wikipedia.org/wiki/Perception [accessed 10 November, 2011]• Categorical perception, Wikipedia. [Online]. http://en.wikipedia.org/wiki/Categorical_perception

[accessed 10 November, 2011]• Neuroplasticity, Wikipedia. [Online]. http://en.wikipedia.org/wiki/Neuroplasticity [accessed 10 November,

2011]• Stimulus onset asynchrony, Wikipedia. [Online]. http://en.wikipedia.org/wiki/Stimulus_onset_asynchrony

[accessed 10 November, 2011]• Robert Fludd, Wikipedia. [Online]. http://en.wikipedia.org/wiki/Robert_Fludd [accessed 10 November,

2011]• Age of Enlightenment, Wikipedia. [Online]. http://en.wikipedia.org/wiki/Age_of_Enlightenment [accessed

10 November, 2011]

4.2 Devices that alter perception

• Reynolds, C. (Ed.) (2010) Devices That Alter Perception 2010, Lexington, Devices That Alter Perception Workshop

With the manipulation of perception on the agenda, a collection of abstracts, overviews, technological realisations and references for the diverse projects selected for the Devices That Alter Perception 2010 Workshop should have been welcome. The abstract of their ‘Call for Papers’ which outlines their interests seemed to be keenly relevant to this research and to considerations of the design of The Augmented Tonoscope.

“This workshop will promote the design and critique of systems whose explicit purpose is altered human percepts. Participants will be asked to present abstracts, images, videos and demonstrations that focus on devices that shape perceptual phenomena. The goals of the workshop are to: - document an emerging field of device design; - facilitate the development of these devices by sharing designs; - better understand the process of perception and how it informs the design of devices; and - debate the aesthetics, perceptual change, social and ethical issues as well as functional transformation

the presented works envision for the future.” (Reynolds, 2010: viii)”

Several of the topics suggested as examples of themes for paper submissions were of particular interest: - Phenomena such as perceptual illusions which can be exploited by systems and devices; - Media art that makes unconventional use of the viewer’s percepts; - Psychological and physiological studies that relate to the process of perception; - Displays that allow atypical perceptual experiences (such as temporal distortions, out of body experiences etc.); - Devices that simulate synaesthesia combining haptic visual and optical sensations; - Devices that map imperceptible phenomena onto the percepts.

The workshop organisers argue that systems such as those currently being developed to perform sensory substitution for sensory impairment, such as Low-fi Skin Vision for the blind and sight impaired, provide evidence of a growing genre of on-human perceptual devices. They talk of being keenly interested in philosophical accounts of perception and its relationship to tools and devices and of being deeply influenced by Alva Noë’s (2004) ideas as detailed in Action in Perception. They are inspired by Stelarc’s performances “illustrating the relentless hybridisation of human and technology” (Reynolds, 2010: ix). They also advocate use of perceptual biases and illusions - such as those of which the augmented reality research community is keenly aware as discussed by Drascic and Milgram (1996) - to create new devices that are more tightly integrated with our perceptual processes.

Unfortunately, with the notable exception of one project - Outre-Ronde, Counter-intuitive Sight Dilation by Anne-Sarah Le Meur - the described works were, for the most part, disappointing. As a collection of ideas representing an emerging ‘design concept’ they seemed to lack any engagement with subtlety, an accent on nuance or an interest in the fleeting which underpin [this researcher’s] thinking to date.

Perhaps the reference to Stelarc and his provocative and extreme body manipulation/modification experiments should have rung bells. The general emphasis on the sensationalist, alienating, fear inducing, intrusive, invasive and sensory depriving which seems to permeate much of this work simply failed to engage. A series of comical head mounted displays, artificial recreations of body parts, simplistic mappings of one sense to another, mechanical recreations of natural physiological responses, distressing emotion inducing experiences and rudimentary implementations of pop psychology seemed to suggest a dystopian technological future in which human perception is somehow bent, twisted and brow beaten to reveal that of which we are currently unaware.

Perhaps ironically, an extract from James Auger’s Keynote, Designing Devices that Alter Perception, seems to warn of these dangers, “A common approach in techno-centric domains is the focus predominantly on application and function for enhancement purposes, ignoring the contextual factors and implications. If we remove the connection to everyday life when researching a technology, the experiences or the effects facilitated by the device risk being purely a showcase; a kind of fairground ride where the aesthetic experience, however striking, is simply a temporary alteration of reality. This can create intrigue, thrill and fascination but the effect is rarely enduring, existing like a one-line gag.” (Reynolds, 2010: 1)

Danielle Wilde’s paper Devices That Alter… A Potted Enquiry, does hint at a more thoughtful and reflective consideration of the design principles that might underpin these devices when she asks, “What drives the artist and designers, engineers and tinkerers? What use can the devices possibly be?” (Reynolds, 2010: 40) But she falls short in only asking what criteria would help push this design concept forward “beyond an ever-growing, rich, yet straggled collection of ideas to a coherent forward thinking, and evolving body of knowledge” (Reynolds, 2010: 41) rather than suggesting a set herself.

These criticisms aside, the cover notes of Alva Noë (2004) Action in Perception does suggest it as a relevant source for future research: “‘Perception is not something that happens to us, or in us,’ writes Alva Noë. ‘It is something we do.’ In Action in Perception, Noë argues that perception and perceptual consciousness depend on capacities for action and thought—that perception is a kind of thoughtful activity. Touch, not vision, should be our model for perception. Perception is not a process in the brain, but a kind of skilful activity of the body as a whole. We enact our perceptual experience.”

• Noë A. (2004) Action in Perception. Cambridge, MA and London: The MIT Press

As does the abstract of Drascic and Milgram (1996) Perceptual Issues in Augmented Reality : “Between the extremes of real life and Virtual Reality lies the spectrum of Mixed Reality, in which views of the real world are combined in some proportion with views of a virtual environment… All Mixed Reality systems are limited in their capability of accurately displaying and controlled all relevant depth cues, and as a result, perceptual biases can interfere with task performance…”

• Drascic D. and Milgram P. (1996) Perceptual Issues in Augmented Reality. Proc. SPIE Vol. 2653: Stereoscopic Displays and Virtual Reality Systems III, San Jose, California, Feb. 1996. 123-134.

4.3 Seeing is spatial, hearing is temporal?

It’s generally held that we live in an ocular world - the visual sense is dominant and sound frequently plays a supporting role, albeit a significant one, in audiovisual experiences such as the film score. The predominant view is that vision is spatial and distal whereas audition is temporal and proximal. So when we see an object is it a distinctly different experience than hearing a sound? If so, is there an innate disparity between the senses of hearing and of sight and the acts of listening and seeing? Can a deeper insight into how we engage with the ocular and how we engage with the auditory help create audiovisual work that creates a deeper correlation between them? Is it possible to create a true and unique audiovisual harmony?

In Perceptual Coherence: Hearing And Seeing, Stephen Handel (2006) attempts to uncover the similarities and interactions between the senses of vision and audition and reframes object and event perception based on the dynamic perceiving of “things” in space and time. He compares hearing and seeing, or “listening to” and “looking at” by isolating what is common to perception and what is specific to each sensory system. With a particular perspective on what incoming sensory information is about - a single world containing objects, events, and various sources of light and of acoustic energy - he explains the heuristic methods used by these systems to build a coherent perceived world. In doing so he argues for a more sophisticated perspective on the senses “… I believe that the usual distinction that vision gives us objects and audition gives us events is a trap. It misleads us into thinking about vision as a spatial sense and about audition as a temporal sense.” (Handel, 2006: 5).

• Handel S. (2006) Perceptual Coherence: Hearing And Seeing. Oxford University Press.

In Audio-Vision: Sound of Screen, Michel Chion (1990) provides a rare theoretical framework for studying the mutual relationship between sound and image in audiovisual perception, albeit that he has a specific focus on the process of adding sound to image and a particular perspective on the ‘reassociation’ of image and sound as a fundamental upon which film sound is built. Chion argues that sound film qualitatively produces a new form of perception: we don’t see images and hear sounds as separate channels, we audio-viewa trans-sensory whole. Still, Chion contends there is no natural and preexisting harmony between image and sound and argues “Visual and auditory perception are of much more disparate natures than one might think. The reason we are only dimly aware of this is that these two perceptions mutually influence each other in the audiovisual contract, lending each other their respective properties by contamination and projection.” (Chion, 1994: 9). In proposing Synchresis - the spontaneous and irresistible mental fusion, completely free of any logic, that happens between a sound and a visual when these occur at exactly the same time - he highlights the perceptual possibility of audiovisual construction while intimating the infeasibility of a true and unique audiovisual harmony.

• Chion M. (1990) Audio-Vision: Sound of Screen. New York, Columbia University Press.

While Handel’s (2006) reframing of object and event perception based on dynamic perceiving of “things” in space and time seems to concur with Chion’s (1990) notion of the audio-viewa trans-sensory whole their otherwise divergent perspectives seem to fall either side of the fence - Handel calling for a more sophisticated perspective on the senses which he contends are actually more alike than commonly held while Chion argues that the senses are distinct but influence one another and effectively fuse when synchronised.

While this research errs towards Handel’s position, the growing realisation that there are deeply subtle and conflicting interactions between the senses, means that far more investigation is required, probably outside the scope of this Ph.D., before there’s any possibility of drawing conclusions on the perceptual and cognitive equivalence between vision and audition and how it might impact on audiovisual work.

4.4 Musical gestures

So an alternative route into perception has been found through research into the way the body affects perception and thinking - specifically through music and gesture. “Under the label of ‘embodied cognition’, we can now better understand the integration of gesture with perception and with thinking in general, including insights on how body movement is both a response to whatever we perceive and an active contribution to our perception of the world.” (Godøy and Leman, 2010:4).

• Godøy R.I. and Leman M. (eds.) (2010) Musical Gestures: Sound, Movement, and Meaning. New York & London, Routledge

Musical Gestures: Sound, Movement, and Meaning is an edited collection of papers addressing the fundamental issues of gesture in relation to music - spanning more than a decade of research sponsored by the European Science Foundation. It’s editors argue that experiences of music are intimately linked with experiences of movements, “We conceive of musical gestures as an expression of a profound engagement with music, and as an expression of a fundamental connection that exists between music and movement.” (Godøy and Leman, 2010: X). They go so far as to claim music is an art of sound and movement and that music means something to us because of this combination.

They also contend that musical gestures is an eminently interdisciplinary topic, drawing on ideas, theories and methods from disciplines such as musicology, music perception, human movement science, cognitive psychology and computer science.

While not all the perspectives outlined in Musical Gestures are relevant to this research, there are significant areas of interest. For example: - the development of new digital instruments and multi-modal interfaces with computers that try to make

meaningful combinations of sound and movement; - the application of technologies (such as motion capture) to open up new views on the fleeting; and - a growing interest in human perception and cognition as a multi-modal and embodied phenomena.

Notions such as ‘sound related gestures’, understood as movements in sound that are sonically rather than physically gestural (such as pitches rising, rhythms that have a galloping character); and ‘sound as metaphor’, movement implied through music - as described by Middleton, “How we feel and how we understand musical sounds is organised through processual shapes which seem to be analogous to physical gestures.” (1993, cited in Godøy and Leman, 2010:17) - may well feature in and shape works made with The Augmented Tonoscope.

Thinking about musical gestures may not only provide a deeper insight into the way movement is embedded within music, but in turn, could also provide insight into how the actual physical movement of glass beads on a vibrating diaphragm or particle systems in a virtual model within The Augmented Tonoscope could serve to communicate additional layers of meaning encoded within the harmonic progressions of the music itself.

But more than this, Godøy and Leman’s (2010) central argument - that in both the physical mechanics of our sense of hearing and in the cognitive act of listening, musical communication is fundamentally driven by movement - could have a profound impact on this research. On reflection, it not only validates the research method of crafting an instrument - justified in the innate connection between sound and movement - but also suggests movement as the mechanism by which to explore the perceptual connection between sound and image.

4.5 Movement as intermediary?

In Musical Gestures Godøy R.I. and Leman M. (2010) present compelling evidence that our experience of music is intimately linked to movement. So inductive reasoning suggests this should also be true of our ocular experience of visual representations of music? So is the most effective strategy for creating a perceptual connection between music and visuals to identify and respond to those aspects of vision which are particularly engaged when viewing movement?

In exploring the aesthetic and theoretical relationship between sound and image, what seems to be emerging as fundamental to this research, is the intermediacy of movement as the mechanism for an intimate perceptual connection between what you hear and what you see. This position puts it at odds with that body of practice that attempts to map a correspondence between the elements of colour (hue, saturation and value) and those of sound (pitch, amplitude, and tone) - hue and pitch being the most common mappings - as realised through the tradition of colour organs and Lumia. As well as that painterly tradition of translating sounds or music into a related visual presentation - of creating dynamic movement in the fixed medium of painting where “the time element becomes a spatial element” (Dickinson, 2011: online) - such as the art of Kadinsky or Klee.

• Dickinson, J. (2011)”Abstract”, Infinite Process from Finite Form. Seeing Sound 2. [Online] http://www.seeingsound.co.uk/programme-2/papers/ [Accessed on 22 December, 2011]

Though it will take more than Chion’s “Synchresis” - of synchronising images to sounds to create a spontaneous and irresistible mental fusion - to find an amalgam of sound and image that genuinely engages the viewer in a subtlety shifted way. This is a perspective shared by Moritz (1986) when he comments that “Too much of what passes [for Visual Music]… relies on just such false synchronization.” (Moritz, 1986: online) and argues that a major stumbling block toward the creation of audiovisuals of a rare quality is “the delusion of rhythm” - the belief by many practitioners that a relatively simplistic mapping of musical beats to visual rhythms and a matching of musical and visual tone and mood is what constitutes good Visual Music.

• Moritz, W. (1986) “Towards an Aesthetics of Visual Music”, ASIFA Canada Bulletin, 14(3), 1-3. [Online]. http://www.iotacenter.org/visualmusic/articles/moritz/moritz_aesthetics, [Accessed on10 March 2011]

What is required is a more sophisticated understanding of and approach to movement in both sound and image.

4.5 Movement in motion graphics

• Penner, R. (2002) “Motion, Tweening, and Easing“, Programming Macromedia Flash MX. Berkley: McGraw-Hill/Osborne, Pages 192-217. Available from: http://www.robertpenner.com/easing/penner_chapter7_tweening.pdf

There are well established techniques that describe the precise shift from point to point within motion graphics, in particular Robert Penner’s ‘tweening functions’, an essential component in every computer animator’s toolbox. These mathematical algorithms define a scalar interpolation from one position to another, a numerical change in position over time. In fact they determine position as a function of time. However, in the natural world objects accelerate and decelerate as they move away from and towards their resting positions. When we see the velocity of an object gradually changing, we infer that some force is steadily pushing or pulling it, we intuitively understand this to be the case and as a result the motion becomes more natural looking. So in order to make motion graphics look more natural Robert Penner’s tweening functions also include an element of ‘easing’ at the start and/or end of the motion - to simulate an acceleration, a change in speed or velocity.

Music, despite its complex structures of musical scale, time signatures, multi-timbral arrangement etc., can also be reduced to an essential change in pitch over time - and since pitch is scalar too, it is, like position, also a function of time.

So music/movement is interpolation from one pitch/position to another - a pitch/position for a specific time.

Considering both the dynamics of motion graphics and the transition between musical notes in terms of movement raises a series of intriguing questions. Is it possible to apply the tweening functions of motion graphics to sound? Can our innate understanding of movement in the real-world be mirrored in our understanding of movement within music? Do we intuitively comprehend and appreciate the notions of scale, cadence and rhythm as the structural framework of the scalar property of pitch and how it changes over time? Do we infer that some musical force is steadily pushing or pulling it?

4.6Movementinmusic

Surprisingly, there are a limited number of established terms and techniques within Western Music that describe the shift from pitch to pitch - and these seem to lack nuance and detail in the specific implementation of the effect: - “Portamento [It.]. A continuous movement from one pitch to another through all of the intervening

pitches, without, however, sounding these directly. It is principally an effect in singing and string playing, though for the latter and for other instruments capable of such an effect, the term glissando is often used.” (Randel, 1998: 673)

- “Glissando [It., abbr. bliss.; fr. Fr. glosser, to slide]. A continuous or sliding movement from one pitch to another. On the piano, the nail of thumb or of the 3rd finger or the side of the index finger is drawn, usually rapidly, over the white keys or the black keys, thus producing a rapid scale… On stringed instruments such as the violin, wind instruments (particularly, though not exclusively, the slide trombone), and on the pedal kettle drum, the sliding movement may produce a continuous variation in pitch rather than a rapid succession of discrete pitches… Some writers have preferred to restrict the meaning of glissando to the motion in which discrete pictures are heard, reserving portamento for continuous variation in pitch, but musical practice is not consistent in this respect…” (Randel, 1998: 353)

• Randel, D. M. (2003) The Harvard Dictionary of Music. Harvard University Press

Bret Battey (2004) concurs with these limitations, arguing “in comparison to Western classical music, numerous other musical traditions place far greater emphasis on the expressive shaping of the continuum between scale steps.” (Battey, 2004: 25). However, he also points out that while “The scalar framework on which such music is suspended may be highly codified theoretically… the detailed and highly subtle pitch curves, ornaments, and inflections of the tradition usually remain at most vaguely described… absorbed through oral transmission, and musical notation proves unequal in the task of representing them.” (Battey, 2004: 25).

• Battey, B. (2004) Bézier Spline Modelling of Pitch-Continuous Melodic Expression and Ornamentation. Computer Music Journal, Winter 2004, Vol. 28, No. 4, pp.25-39

It turns out that Robert Penner’s tweening functions, despite his efforts to “provide meaningful and predictable relationships between position and time” (Penner, 2002: 194) are essentially presets. More promising, interesting and relevant is Bret Battey’s research into how computers can effectively and convincingly render expressive melodic forms inspired by pitch continuum traditions and his technique of Bézier curve modelling - combining an identification of critical tonal points in the performance with a simple and intuitive two-dimensional designation of the curve between those points. While Battey’s software is designed to analyse music recordings it has inspired development of a performative, real-time, touchscreen driven Bézier curve controller to realise more controllable transitions between pitch.

5. Contextualising this work within Visual Music and wider audiovisual culture

From Isaac Newton’s colour circle correlating hues with musical notes which inspired a succession of colour organs and Lumia, to the synthetic sound production film experiments of Rudolph Pfenninger and Oskar Fischinger, there is a long and rich history of scientific and artistic study into the aesthetic and theoretical relationships between the aural and visual.

This evolving practice, broadly determined by the term Visual Music, has been defined by Keefer and Ox (2008) of the Center for Visual Music as: - “A visualization of music which is the translation of a specific musical composition (or sound) into a

visual language, with the original syntax being emulated in the new visual rendition. This can be done with or without a computer. This can also be defined as intermedia.

- A time based narrative visual structure that is similar to the structure of a kind or style of music. It is a new composition created visually but as if it were an aural piece. This can have sound, or exist silent. Theorist/inventor Adrian Klein wrote in 1930: “…somehow or other, we have got to treat light, form and movement, as sound has already been treated. A satisfactory unity will never be found between these expressive media until they are reduced to the same terms.” [Klein, A. B. (1930) Colour-Music: The Art of Light. London: The Technical Press Ltd.,. Second edition, p. 37.]

- A direct translation of image to sound or music, as images photographed, drawn or scratched onto a film’s soundtrack are directly converted to sound when the film is projected. Often these images are simultaneously shown visually. Literally, what you see is also what you hear. (An early example is filmmaker Oskar Fischinger’s Ornament Sound experiments c. 1932). There are many examples in Visual Music film of this process, e.g. McLaren, Spinello, Damonte and other contemporary filmmakers, including sections of Pengilly’s work in this show. This method has been called a “pure” type of Visual Music.

- A visual composition that is not done in a linear, time-based manner, but rather something more static like a 7’ x 8’ canvas. However, as in Klee, the movement of the painted elements can and have achieved a kind of Visual Music, serving as an artist’s visual interpretation of specific music.” (Keefer and Ox, 2008: online)

• Keefer, C. and Ox, J. (2008) On Curating Recent Digital Abstract Visual Music. [Online]. http://www.centerforvisualmusic.org/Ox_Keefer_VM.htm [Accessed 12 April 2012]

An alternative, more concise and production focussed albeit less academic definition, can be found in a summation of the ‘Visual music’ entry on Wikipedia: - methods or devices which translate sounds or music into a related visual presentation - possibly

including the translation of music to painting; - the use of musical structures in visual imagery; - systems which convert music or sound directly into visual forms (and vice versa) by means of a

mechanical instrument, an artist’s interpretation, or a computer.

• Visual music, Wikipedia. [Online]. http://en.wikipedia.org/wiki/Visual_music [Accessed 10 November, 2011]

While this research is located within this general context it sits especially well within the last of these definitions, since The Augmented Tonoscope will utilise all three of the suggested means - a mechanical instrument, artistic interpretation and a computer.

There is an active UK and international community of researchers and practitioners exploring a broad range of multimedia work highlighting the relationship between sound and image – including Cymatics – as illustrated through several conferences specifically on the theme of Visual Music: - Seeing Sound ’11 and Seeing Sound ’09, Centre for Musical Research, Bath Spa University, UK - Understanding VISUAL Music 2011, Hexagram-Concordia Centre for Research Creation in Media

Arts and Technologies in collaboration with the Department of Music Concordia University, Montreal, Canada

As well as numerous international conferences on audiovisual culture which feature Visual Music as a strand of their overall programme.

An exemplar in locating Visual Music in general - and my research in particular - within a contemporary but wider view of audiovisual culture is the See This Sound exhibition at the Lentos Kunstmuseum, Linz (August 28, 2009 to January 10, 2010) and its associated catalogue.

• RainerC.,Rollig,S.,Daniels,D.andAmmerM.(eds.)(2009)See This Sound. Promises in Sound and Vision, Köln: König W.

While there is a long and rich history of sound used as an artistic medium, Sonic (and by extension audiovisual) Art - with notable exceptions such as Turner Prize 2010 winner Susan Philipsz for her work LOWLANDZ 2008 / 2010 - has been essentially ‘tuned out’ by art institutions and operates at the margins of art practice.

What’s refreshing about the See This Sound exhibition and it’s accompanying catalogue is that it carefully traces the tradition of artists’ fascination with the relationship between sound and image but since the beginning of the C20th and with a modern and quite particular perspective. It accepts that there is a broader context of art that explores “the aesthetic and theoretical relationships between music and painting, between colours and sounds from the 19th to the mid-20th century” but argues that there is an alternative and distinct artistic tradition of “acoustic and visual interplay as it has developed and been artistically analyzed and treated since the early 20th century.” (Rollig and Daniels, 2009: 12).

This subtle but significant shift in perspective, from a painterly tradition that attempts to represent and interpret the dynamic, temporal, emotional and communicative aspects of music though graphical structures, shapes, colours and textures, towards an artistic investigation, usually through technology, of the interplay between sound and image, is particularly relevant and reflects the nature of this research. Furthermore, Rollig and Daniels (2009) argue that “the development of electronic and digital media has enabled a previously unimagined complexity in the coupling of images and sound” (Rollig and Daniels, 2009: 13) suggesting an unprecedented potential to develop new work in this oeuvre.

So while The Augmented Tonoscope is located within the broader context of Visual Music it has a clear focus in the direct and elemental relationship between visual and acoustic phenomena, best contextualised within the contemporary art theory, research and curatorial agendas advocated and disseminated through See This Sound: Promises in Sound and Vision (2009).