A Sense of Connection: Synesthetic Metaphor, Intermedia, and the Possibility of

Acquired Synesthesia

Dave Wall

1

The remarkable thing is not that "synesthetic art" turns out to be a chimera, but that it is such a compelling and useful chimera for the production of art. Paul Hertz

Indeed, there is good evidence to suggest that our senses aren’t as separate as you might think…Jamie Ward

Introduction

The world of the senses is not what many of us imagine it to be. Research has uncovered an array of senses that we use when we encounter the world; most of us think of five, but researchers have discovered anywhere between twenty-one and thirty-three. There are two views for the way in which we experience physical reality: the modularity thesis which most people hold to that assumes five independently functioning senses; and the unitary thesis which supports the idea of one integrated sensory organ with five suborgans (van Campen 1999, 9). The concept of five senses comes from Aristotle, who believed that each sense was linked to a sensory organ. In fact, there are a number of specialized cell receptors for each sense that respond to specific signals. The retina, for example, contains a cell that responds to light and another that responds to color, and the color receptors can be subdivided into three types. Depending on how we count these, the number of senses we come up with could be anywhere between twenty-one and thirty-three (Ward 2008, 33).

If we now add synesthesia – the fusion of one or more senses – to this line of thought, the possibilities for artistic work expands greatly. We may not experience the work of these specialized cells, but we can consider their existence metaphorically when we produce art. It makes sense that the more cell receptors a sense contains, the more range a metaphor based on synesthesia has, and the deeper it can potentially be. Language, and by extension metaphor, conditions how we ascribe meaning to experiences (Paul Hertz 1999, 9) and to works of art. The deeper the meaning, the more profound the artwork.

For artists, the fact of an existing neurological condition – synesthesia - is nowhere near as interesting in terms of artistic possibilities as using synesthesia as a metaphor to create new art. Chromesthesia – the fusing of color and sound – is the most common type of synesthesia, and dropping that possibility alone into the idea stream has led artists to metaphorically mix various sense combinations in the production of art.

It has been suggested that the synesthetic state can be produced by consciously creating associations between different senses. Some artists have made a practice of creating these associations so that, rather than working within the contrived situation of “making synesthetic art”, the artist can use an actual, personal synesthetic reaction as a tool. The fusion of the different senses becomes a natural, effortless act rather than a deliberate contrivance (Ox 1999a, 7- 8).

2

At this point, however, synesthetic art produced by non-synesthetes using synesthetic metaphor is for the most part a contrivance, something that needs to be worked out in the mind of the artist, not something that occurs naturally. Naturally occurring synesthesia means that connections are experienced spontaneously as personal reactions to the world.

Paul Hertz suggests that persons, not artworks, are synesthetic and offers the term “cross-modal art” as a more precise term than “synesthetic art” when describing non-synesthetic artists, while recognizing that the fusion of the senses represented by synesthesia is what fuels the imagination of these artists (Hertz 1999, 400). In the case of the art producer who is a synesthete, synesthetic art is an appropriate term for what they produce. While keeping Hertz’s distinction in mind I will, for the purposes of this paper, be using “synesthetic art” to describe the work of both synesthete and non-synesthete artists. Both artists are capable of creating synesthetic artistic experiences, which I think of as the simultaneous perception of multiple stimuli in a single experience. This type of experience in art goes by the name of intermedia, which I will look at later in the paper.

Both synesthete and non-synesthete artists use synesthetic imagery in order to access multiple, simultaneous meanings that can be freely combined, re-combined and un-combined. As Hertz points out, “we are tied neither to deep structures nor to arbitrarily determined symbolic codes, but free to play with the whole range between them” (Hertz 1999, 404). Synesthetic art gives us the ability “to use aleatoric juxtaposition, partial mappings and associations, parallel coupling and complementarity freely as a metalanguage of cross-modal composition, a counterpoint of rupture and cohesion” (Hertz 1999, 404). This freedom has led to difficulties in establishing universal correspondences between senses (i.e. this image equals that sound) and this is seen clearly in the production of graphic music scores and the use of color organs. The difficulty in finding a common visual language for extended instrumental technique in conventional music notation is also related to this lack of correspondence between the senses. While this is often seen as negative, it creates the sense of freedom outlined by Hertz above and can lead to the creation of personal, idiosyncratic systems.

I have touched on the main issues that I will look at in this paper – synesthesia as metaphor, intermedia, and the development of synesthesia in non-synesthetes. The latter is somewhat controversial and has been likened to a musician developing good relative pitch. It may well become that banal someday, but for now it exists more as an idea than as reality, and points toward an artist who is naturally adept at inhabiting whichever artworld he/she finds necessary for the effective communication of ideas.

A brief survey of synesthesia research and experimentation will serve to contextualize synesthesia and introduce some of the machines that were produced through the synesthetic metaphor, machines that helped keep the idea alive in the minds of artists and the public in general. I will also consider two painters, Paul Klee and Wassily Kandinsky in some depth, as contrasting examples of approach in using the synesthetic metaphor. I will also look at abstract cinema pioneer John Whitney as an example of someone who used the synesthetic metaphor to create a color music system of great depth and complexity.

3

Research and Experimentation: Synesthetic Metaphor in History

Pitch and color correspondences have been a source of fascination for philosophers and artists for centuries. Aristotle concentrated on lightness, ordering colors on a scale from

black to white and linking pitches to corresponding gray values. Pythagoras demonstrated that pitch could be measured by the length of strings, inspiring other philosophers to consider the possibility of a similar system of measurement for color.

John Locke, in his Essay Concerning Human Understanding of 1694, raised a hypothetical example of a blind man who saw scarlet when a trumpet was played. While he clearly believed that vision was necessary in order to experience color and was not using this as an example of synesthesia, the example nevertheless prompted heated international discussions during the eighteenth and nineteenth centuries concerning the possibility of correspondence between color and sound (Peacock 1988, 398).

In his Optiks of 1704, Isaac Newton, in an effort to create a quantifiable way to connect mathematical systems of optics with musical harmony, linked sound oscillation to the wavelength of light. He posited that the distribution of white light in the color spectrum is analogous to how tones are distributed in the octave, identifying seven discrete “light entities” which corresponded to the seven notes of the octave, an somewhat arbitrary correspondence since there is no natural relationship between the two systems.

Not long after, the synesthetic metaphor becomes actualized in the work of French mathematician Louis-Bertrand Castel who started to develop the first color harpsichord (clavecin oculaire) around 1720, inspired by Newton’s work. Though impractical in terms of the concert hall – strips of paper lit by candles did not have the power to create a convincing visual experience in such a large space – it was the first in a line of color organs that would stretch into the twentieth century, culminating in the work of Thomas Wilfred in the 1920s and 30s.

These instruments did not play sound (Peacock 1988, 402). They employed synesthetic metaphor by simply assigning color to each key on the keyboard, using music as a way of structuring a temporal visual art. Temporality is perhaps the most obvious musical attribute not present in painting, and many artists in the first decades of the twentieth century - futurists, members of De Stijl and Der BlaueReiter – were looking for ways to add it to their art through visual suggestions of movement (van Campen 1999, 11).

Thomas Wilfred was perhaps most influential figure in developing a color - music correspondence, and had the most sustained popularity of any previous color organist, developing what he called the art of Lumia. The quality of his work and his association with the Museum of Modern Art in New York contributed to his notoriety and brought his work to the attention of a great many other artists, and helping to keep the synesthetic metaphor alive.

4

After initially using the musical analogy to explain his work, Wilfred rejected the idea of a correspondence between light and sound, yet his art resembled music by including factors of time and rhythm in live performance. Music existed in his work as a structural idea and strongly influenced the production of his art even though he felt that “attempts

to design Lumia instruments in imitation of musical ones will prove as futile as attempts to write Lumia compositions by following the conventional rules laid down for music” (Eskilon 203, 67).

………………

Paul Klee and Wassily Kandinsky are important examples of the synesthetic metaphor influencing painting at the highest level. Both of these artists experimented with music as a way of influencing and structuring their work, and both developed a critical language based on music to describe their experimentation and to communicate a new approach to visual art.

Klee focused on developing an abstract visual language based on historical musical models, but his approach did not include overt references to synesthesia as did Kandinsky’s. He was reluctant to fully embrace the musical metaphor in his painting, but he nevertheless moved towards it. He concentrated mainly on rhythm, which he considered to not only mark the movement of time in music, but also in painting. His early works demonstrate his approach with “the translation of motifs taken from nature into free, rhythmical linear structures and tonal values…” (Duchting 2004, 16). Rejecting the idea that color could be mapped onto the musical scale, he struggled to find a technique that would communicate his sense of color.

Influenced by Robert Delauney, he began using contrasting colors to create structural features and continued to explore the relationship between music and painting. He created ‘fugue’paintings as a way of solving compositional problems involving temporality. Fugue in Red (fig. 1) demonstrates the way in which temporality is achieved as the various forms seem to emerge from dark to light. This piece is related to abstract cinema artists such as Hans Ruttman, who knew Klee, and Ludwig Hirschfeld-Mack who visited the Bauhuas where Klee was teaching and displayed a succession of colored forms expanding and contracting through the use of lamps of varying strength.

5

fig. 1

With the piece In Bach’s Style (fig. 2), he creates a linear framework reminiscent of a musical score, the fermata in the left-hand corner becoming more a physiognomic hieroglyphic (in this case an eye) than a musical symbol.

fig. 2

Rose Garden (fig. 3) is another example of what could easily be interpreted as a musical score, the roses becoming notes and the use of line and color creating rhythmic values and/or formal attributes.

6

fig. 3

The use of color as a metaphor for music continued to elude him. While he was able to quantify line and light/dark (note duration and pitch), color, for him, corresponded to the expressive, unexplainable, and irrational content of musical tones, not the tones themselves. As such, he felt they could not be quantified. As he said: “Color has always had something secretive about it that is difficult to grasp…The colors are the most irrational elements in painting” (Duchtung 2004, 45).

Our knowledge of Klee’s history as a musician along with the names he gave to some of his paintings encourage us to find musical meaning in them. In paintings like Ancient Sounds (fig. 4), shifting shades of darker color seem to imply harmonic structure while the lighter colors emerge as single lines of melody. His “polyphonic” paintings, “in which as many as eight colored planes are laid on top of one another” (Duchtung 2004, 67), create depth and the implication of musical polyphony. Dots, lines, and color combine in Rising Star (fig. 5) to imply a composition of many voices. Klee was after an “ecstatic truth” beyond the simple assignation of color to pitch, and used music as a “mediator between the world of appearances and artists’ efforts to fathom the laws of the universe” (Duchtung 2004, 88).

7

fig. 4

fig. 5

8

Wassily Kandinsky, by contrast, was seeking this ecstatic truth by blending the arts into a spiritually felt, synesthetic meshing of sensations. He considered colors to be living energies with their own intrinsic ‘sounds’. In his production of Mussorgsky’s Pictures at an Exhibition, he included abstract forms and colors, as well as lighting effects and movement in an effort to create a synesthetic art. In discussing the way in which different art forms affect different people, Kandinsky says, in Concerning the Spiritual in Art, “There reside...in arts which are outwardly different, hidden forces equally different, so that they may all work in one man towards a single result, even though each art maybe working in isolation” (Kandinsky 1977, 43). Kandinsky argues for what is in essence a synesthetic art, a fusion that results in “a single result,” pointing toward the development of intermedia.

Deeply affected by Arnold Schoenberg’s music, he conceived of an independence of forms related to the perceived independence of the instruments in Shoenberg’s music, freed from traditional harmony. Inspired by Shoenberg, he claimed that objects in a painting could be defined exclusively in terms of “tensions between forces” (Leggio 2002, 109). Kandinsky understood from music theory that the idea of art is the exteriorization of these tensions, which he thought of as the artist’s inner necessity leading, in Schoenberg’s music, to an experience that is purely spiritual.

Kandinsky’s love of the circle owes much to Schoenberg’s replacing of linear harmony with a circular conception in which the tensions created are not resolved in terms of harmonic closure. This is also evident in Wagner’s “endless melody” in Tristan und Isolde. Schoenberg saw this lack of closure as an “analogy with infinity” which “could hardly be made more vivid than through a fluctuating…unending harmony” (Leggio 2002, 115). Here we see the importance of the spiritual aspect so important to both these men, a goal Kandinsky saw being attained through synesthetic art or, as mentioned earlier, “a single result,” implying a “oneness.” His painting Several Circles demonstrates a sense of this spiritual aspect, evoking a sense of the cosmos and infinity (fig. 6).

9

fig. 6

In his book Point and Line to Plane, Kandinsky theorized on the individual graphic mark – the dot or point - which he feels is directly equivalent to the fundamental musical tone. This led to a graphic translation of passages from Beethoven’s Fifth Symphony (fig. 7).

fig. 7

10

The point’s “inner qualities…take on an increasingly powerful sound. These inner qualities…emerge one after another from the depth of its being...the dead point becomes a living being” (Leggio 2002, 105). This is, of course, a personal interpretation of the relationship between image and sound and as such is outside of a common, agreed upon language, much like Newton’s correspondence of the color spectrum to musical pitches, or the numerous color-tone analogies that have been created since Castel’s first color organ (Daniels 2010, 345).

Kandinsky’s painting Composition VIII (fig. 8) is a good example of his concept of the living point with different energy levels radiating from each point/circle in the painting. Kandinsky’s work is seen as an influence on composers producing graphic scores; this particular piece was sonified by Todd Barton, composer and composition teacher at Southern Oregon University, and posted to the Vimeo video site (http://vimeo.com/6766060).

fig. 8

John Whitney’s obsession with translating the “liquid architecture” of music into film led him to question the nature of music and to search for a visual code that would match that of music’s, and allow him to structure visual material in time. As implied earlier, however, the physical parameters of visual and auditory media provide us with no clear system of correspondences. We can invent things, but the fact remains that correspondences among media are arbitrary and are conditioned by “fuzzy cultural practices and psychological preferences” (Hertz 1999, 403). Nevertheless, Whitney was determined to overcome these obstacles and establish a system by which music corresponded logically to image. When attempts to map Shoenberg’s concept of the tone row onto filmstrip failed, he abandoned the movie camera and began an effort to invent a machine that he could use to create images that would accurately reflect music’s

11

abstraction. This was an effort to create a non-arbitrary language/system for producing synesthetic art.

Identifying certain musical truisms, such as the fact that music has motion and that the relationships between notes are what create music, provided insight into how to approach visual motion. He found that musical resolution of the tension created by musical motion and note relationships is directly related to a resolution to a “visual harmonic resonance,” which he described as follows: “First, motion becomes pattern if objects move differentially. Second, a resolution to order in patterns of motion occurs at points of resonance [or resolution]. And third, this resolution at resonant events…characterize differential resonant phenomena of visual harmony” (Whitney 1980, 40). “Differential resonant phenomena” refers to resonance (resolution) achieved through differential, or independent, ordered movement. This led to his detection of charge and discharge of tension in graphic pattern arrays and his connection of harmonic forces/tensions in music being related to the same in graphic display (Whitney 1980, 43). This recalls Kandinsky’s tensions between forces.

Rather than an up-down, left-right orientation that color organists had previously used to evoke the musical scale, Whitney proposed his concept of differential motion pattern, mentioned above. This involved defining sixty points along a line, each point moving a different amount. For instance, point #1 moves one step up, point #2 moves 2 steps up, point #3 moves 3 steps up until point #60 is reached. If point #1 moves 2 steps, then #2 moves 4 steps, #3 moves 6 steps. This could also be applied to a set of elements, such as an array of triangles. Whitney sums it up:

This is the differential principle, which is applied in various ways to all motion of typical scales of a hypothetical instrument. Any scale ascends a ladder of tensional structure in the time dimension. The progressive rise and fall of tension…is characterized by recurrent nodes, where tension is resolved to equilibrium (resonance) (Italics mine) (Whitney 1980, 49).

While this is idiosyncratic to Whitney, it represents a system less arbitrary than any presented thus far if only because of its detail, and its inclusion of musical tension in relation to visual tension. The depth and authority of the work would make it easy to accept as a standard operating procedure for marrying image with music.

This system is very personal to him, and it is difficult to see the relationship between sound and image in his films. Being able to determining his success in fusing the two is dependant on being able to watch his work with new eyes. Tensional structure is the clue. Watching for – and being able to see - changes in the images that are related to the rising and falling of musical tension/energy is key. When encountering his work we have to discard conventional notions of pitch/color relationships, or the conventional expectation

12

of music corresponding to visual movement in obvious ways.

This scratches the surface of Whitney’s thought and work in this domain, but demonstrates a practical, systematic approach to the synesthetic metaphor that goes deeper than any we have yet encountered. Two of his films – Matrix III and Arabesque – are included in the filmography.

Becoming Synesthetic

It is known that we are born as auditory-visual synesthetes. Until 6 months of age, responses to spoken language are recorded in the areas of the brain that process both hearing and sight. This ability gradually disappears until it is gone completely at 36 months of age (Afra, P. 2009, 35).

In a limited sense, however, we are all still synesthetes. Jamie Ward describes multi-sensory neurons in the brain that fuse sound and vision. These neurons become amplified in certain situations, and where we look becomes where we hear. This explains why, in a movie theatre, we experience the sound coming from the screen when the speakers are placed on the sides of the room. Similarly, a voice that goes unheard in a noisy room can be heard if we watch the speaker’s lips. These examples of sound-sight fusion represent an indirect, yet physical link to synesthesia. In these cases, entire areas of the brain that privilege sight or sound aren’t fused; individual neurons are. Our brains contain a type of sense fusion, indicating a possible path we can take to acquire synesthesia. Is the acquisition of synesthesia possible?

There are indications that it is. The new neuroscience of optogenetics, using, as the name implies, a combination of genetics and optics, is providing the tools for brain mapping with great precision. Essentially, neuroscientists can control brain circuits in laboratory animals, turning cells on or off with laser light, and replacing damaged circuitry with new wiring from a different location. As Dr. Tim Murphy of the Brain Research Centre in British Columbia says, “There is an actual sprouting of connections and structural rearrangements.” (Anne McIlroy, “Shining a light inside the brain,” The Globe and Mail, April 2, 2011). This research is very new and is being used to investigate brain function in stroke patients and the neural circuitry involved in mental illness. Given the ability to create connections and rearrange structure, it does not seem to me to be a stretch to say that this technology could be used to connect different areas of the brain that involve the senses.

As it stands now, synesthesia can be naturally occurring or, more interestingly for the purposes of this paper, acquired. It can be acquired through sensory loss or temporarily through the use of certain drugs. Ward states that, “…synesthesia is effectively an adaptation of the normal processes of multisensory perception that is universal among humans…” (Ward, p. 62) indicating the possibility of switching on pathways in the brain that permit synesthesia.

13

LSD and mescaline are known to activate serotonin 2A receptors. Auditory visual synesthesia has been reported with the activation of serotonin 2A receptors in the brain, also known as the “synesthesia receptor”. (Afra, P. 2009, 35) These drugs can produce profound results within an hour. Since it is inconceivable that the brain could rewire itself so quickly, the suggestion is that our brains are already intertwined and connected synesthetically (Ward 2008, 21). The drug effectively unblocks the connection. The research of Tor Wager, psychology professor at Columbia University indicates that, “all the drugs in the world are already in our brains; that’s why we have receptors that are able to make sense of synthetic or artificially introduced versions [of sensory events]. The real trick is figuring out how to prompt our brains to release the right stuff at the right time” (Hanus 2008, 1). Wager’s research indicates the possibility of voluntarily releasing the drugs necessary for unblocking connections in the brain that produce synesthesia.

Attempts to enable blind people to “see” by using software to convert visual images to sound using a hidden camera mounted on the wearer’s head have had success. Higher sounds denote objects higher up in the image and loudness is associated with brightness. The visual image is heard as a soundscape scanned from left to right. Pat Fletcher, blinded in an industrial accident and one of the first users, insists that it is like seeing, not hearing. She sees a visual representation of the soundscape, thus representing a person-made synesthesia (http://www.artificialvision.com/).

Many artists have stated it is possible to develop and/or increase synesthetic awareness simply by making it part of their practice while creating art. For example, Jack Ox has been experimenting for over twenty years with how to combine different media into one, and claims that it is now natural for her to see sonic forms (Ox, 1999, 7). This supports Ward’s conclusion that a slow reorganization of multisensory pathways is possible. (Ward 2008, 62) This reorganization often occurs after sensory loss of some kind, but the experiences of Ox and other artists indicate that sustained effort may also produce synesthesia. Kandinsky is perhaps the most famous example of this, having, as is commonly believed, “developed his abilities through associative techniques aimed at enhancing sensory exchange, much like one might develop relative pitch” (Ione 2004, 154). My own work as a musician has enabled me to develop a specialized type of sound – symbol synesthesia that every trained musician develops. If I see a melodic passage in musical notation, I can hear it. If I hear a melodic passage, I can see it in musical notation.

But why would we want to develop synesthesia? Would it create better artists? Ramachandran and Hubbard have this to say:

Synesthesia causes excess communication amongst brain maps…Depending on where and how widely in the brain the trait was expressed, it could lead to both synesthesia and to a propensity toward linking

14

seemingly unrelated concepts and ideas – in short, creativity (Ward 2008, 127).

Developing synesthesia would mean, essentially, breaking down or “eliding the division and hierarchies we apply to the senses” (Barker 2008, 2), creating a larger ground for the production of ideas. To recycle a statement from earlier in this paper, it would produce artists “who are naturally adept at inhabiting whichever artworld” (or simultaneous artworlds) “they find necessary for the effective communication of ideas.” This leads nicely to a discussion of intermedia.

Intermedia

Richard Wagner’s aesthetic approach of Gesamtkunstwerk, the uniting of the visual, the aural, and narrative text into a single artistic unity, is directly related to a synthesis of the senses and is an early example of multimedia. However, his idea of a single artistic unity – influencing Kandinsky and his idea of a “single result” - points towards an intermedial art with its ideas of “commingled, shared structural patterns coming from completely different worlds” (Ox 2001, 47). In other words, once someone thought of combining art forms as autonomous constructs in one space, it was only a matter of time before artists started combining different artistic syntaxes into a single thing.

Dick Higgins named and described intermedia in the 1960s (Higgins 1984, 2), and provided examples; Gene Youngblood defined a necessary component when he spoke of synergy in which the sum of the whole is greater than its parts. When these “parts behave with integrity and without self-consciousness” (Youngblood 1970, 110), intermedia is possible. The parts must fuse to create a stronger whole; independent parts existing in one space without fusing – as in Gesamtkunstwerk - is multimedia. Youngblood, writing about cinema in 1970, was convinced that the current environment involving a multi-dimensional, simulsensory network of information sources could most effectively function using a synesthetic aesthetic language (Youngblood, p. 77), resulting in intermedia. Refining the definition further, Yvonne Spielman believes that an obligatory part of intermedia is a “combinatory structure of syntactical elements that come from more than one medium but are combined into one and are thereby transformed into a new entity” (Ox 2001, 48).

The possibilities for the combining of different media into a new form have increased exponentially with the computer, making it much easier to perform intermedia. The popularity of chaos and complexity theories in the latter part of the twentieth century has contributed to this process of fusion. Systems are more easily connected and we also have a very precise collage tool. We can cut and paste multiple layers; our artistic procedures can achieve greater and greater complexity, creating new possibilities (Ox 1999b, 391). As Paul Hertz points out "computers make it possible to use these deep structures to build cross-modal works of art from the smallest perceptible element up” (Hertz 1999,

15

403).

In 1996, intermedia/synesthetic artist Jack Ox began work on the visual translation of Quanta and Hymn to Matter, a large-scale symphonic and choral work by the American composer, Dary John Mizelle. The visual representation was produced by what she named “The 21st Century Virtual Reality Color Organ.” I present this work as an example of contemporary intermedia being dependent on the use of technology for its realization.

The 21st Century Virtual Reality Color Organ is a computational system for translating musical compositions into visual performance. It uses supercomputing power to produce 3D visual images and sound from Musical Instrument Digital Interface (MIDI) files and can play a variety of compositions. Performances take place in interactive, immersive, virtual reality environments such as the Cave Automatic Virtual Environment (CAVE), VisionDome, or Immersadesk. Because it’s a 3D immersive world, the Color Organ is also a place—that is, a performance space (Ox, http://www.jackox.net/IEEE.pdf).

Ox mixed over 130 RGB colors to express the timbres of the orchestra, which included strings (fig. 9), woodwinds (fig. 10), brass (fig. 11), percussion, and vocals (fig. 12). Each section is represented by sets of colors in a graduated series of hues, with additions of other colors expressing the sound resulting from the application of different instrumental technique. Below are her charts for strings, woodwinds, brass, and vocals. She does not provide specific explanations of which colors in each chart express which parameters. She does, however, provide a page of the score over which she has laid color corresponding to the appropriate parameters. Timbre is represented by hue and dynamics are represented by saturation (fig. 13). Assuming a standard score with woodwinds at the top, followed by brass, percussion and strings gives us a little more insight into how the system works, but in the end we would need three things to truly understand her system: the score with overlaid color and instrument sections clearly marked; the original score without color so that specific musical indications are not obscured by overlaid colors; and a performance of the score. She has not provided a color chart for percussion.

16

fig. 9 strings fig. 10 woodwinds

17

fig. 11 brass

fig. 12 vocals

18

fig. 13

The poem on which the piece is based describes a very harsh landscape. In order give this landscape expression in the work, Ox travelled to deserts in California and Arizona and photographed landscapes that she felt expressed in their sculptural qualities sound

making concepts of different instrument families. She then made drawings which were transformed into 3D through modeling and placement in the VR world of the CAVE, a virtual reality theater originally developed by the Electronic Visualization Lab at the University of Illinois, Chicago. This hand drawn and modeled world is the canvas and the image source for surface textures on which the timbral colors lay.

This is a sketch that Ox made in order to show how the individual landscapes would be put together (fig. 14). Closest to the front is brass, followed by strings, with the ledge on which are placed the smaller units of percussion. Behind this ledge are woodwinds with vocals in the very back. Individual landscape images for each instrumental group can be found here: http://home.bway.net/jackox/colororgansite/.

19

fig. 14

In the virtual reality environment it looks like what we see in fig. 15. Referring back to the color and landscape charts for each instrumental section allows for a kind of synesthetic experience as the colors translate to sound. This takes a certain amount of study and interpretation. For instance, the green could either be strings, woodwinds or vocals, or a meshing of the three. The final product can be seen as a transcription translated into 3D in a virtual environment, but it is more than that. It is a work designed to evoke strong emotional responses in the viewer.

20

fig. 15

The music is represented by MIDI. The following is Ox’s description of how the music is realized in virtual reality:

Each note as it is played creates a rectangular strip located in a specific place over the landscape, which is determined by time and pitch. Pitch is the y axis and determines how high it is placed in space. Time is the x axis, going from left to right, so time says where the strip begins and ends in space. The initial attack of the note, which is how loud it begins, determines the width, or z axis of the rectangular strip, which remains constant until the note ends. Further changes in loudness or softness are reflected in the saturation of the timbre color which is part of the surface of the strip (Ox, http://home.bway.net/jackox/colororgansite/).

This piece falls under Higgins definition of intermedia as a work that exists between mediums (Higgins 1984, 49), combining the syntaxes of music, color, movement, and sculpture in a 3D virtual environment.

This quick time movie demonstrates the approach, using a chamber piece by Clarence Barlow. It is important to note that the viewer experiences the piece from within a world

21

of hand-drawn, black and white landscapes modeled in 3D. Imagine, while viewing the movie, that you are on the ground looking up at the colored strips, instead of floating above them (see fig. 15). As I understand it, this is one of the possible ways of experiencing pieces conceived in this environment. The video gives an idea of the environment, and demonstrates some of the parameters mentioned above: pitch, timbre, and loudness (http://www.jackox.net/media/Barlow_apple.mov).Time, rather than being expressed on a left to right x axis as described above, moves in a parabolic spiral path in order to visualize the structure Barlow used in his compositional process (http://www.jackox.net/pages/introImJan.html). Imposing the structure of the music on the visual medium combines the two syntaxes and provides another example of intermedia.

Conclusion

I have provided a limited exploration of the history of synesthesia research and experimentation along with a consideration of the synesthetic metaphor in order to demonstrate its influence on art, and to determine a way forward in terms of using it as a viable tool in the production of art. It has clearly had an influence, but research shows us that it is possible to take what it offers to a higher level, to use it as a naturally occurring tool in our neurobiological makeup rather than as a contrived strategy for creating art. This is particularly relevant to intermedia since its aim is to create any number of single, multifaceted entities out of a variety of the seemingly discrete sources that we presently think of as information received from individual sense organs.

With acquired synesthesia, these discrete sources of information become a single multifaceted entity if we voluntarily decide that it be so. With acquired synesthesia, we would not be forced to construct communication systems before creating art. We would simply be communicating felt experience. I do not mean to imply that felt experience is necessarily better than systems creation since engaging the intellect in the creation of systems is part of effective art. Since acquired synesthesia would allow us to use a synesthetic response voluntarily, we would continue to be able to exercise choice in this domain. We can make use of the intellect, immediate felt experience, or both. I believe that more options, wisely used, are beneficial in general term as well a in the creation of art.

22

BIBLIOGRAPHY

1. Afra, P., Funke, M. and Matsuo, F., ``Acquired auditory-visual synesthesia: A window to early cross-modal sensory interactions,'' Psychology Research and Behavior Management, Vol. 2, January 2009.

2. Barker, J. M. “Out of Sync, Out of Sight: Synaesthesia and Film Spectacle.” Paragraph, Volume: 31, Issue: 2 (July 1, 2008).

3. Baron-Cohen, Simon. “Is There a Normal Phase of Synaesthesia in Development?," Psyche,Vol. 2 (psyche.cs.monash.edu.auf/i,l ename:p syche-96-2-27-syn_development-baron_cohen). Quoted in Hertz, Paul. “Synesthetic Art: An Imaginary Number?” Leonardo, Vol. 32, No. 5, Seventh New York Digital Salon (1999), pp. 399-404.

4. Berman, Greta. "Synesthesia and the Arts," Leonardo 32, No. 1 (1999), pp. 15-22.

5. Brougher, Kerry, Jeremy Strick, Ari Wiseman, Judith Zilcher and Olivia Mattis. Visual Music: Synesthesia in Art and Music Since 1900. Los Angeles: Thames and Hudson, 2005.

6. Carpenter, S. “Everyday fantasia. The world of synesthesia.” APA Monitor on Psychology, vol. 32, No. 3 (2001).

7. Carrà, Carlo. "The painting of sounds, noises and smell."http://www.unknown.nu/futurism/paintsound.html

8. Conrad, Daniel. "The dichromaccord: Reinventing the elusive color organ." Leonardo, 32, no. 5 (1999), 393-398.

9. Cytowic, Richard. Synesthesia: A Union of the Senses. Cambridge, Mass: MIT Press,

2002.

10. Dailey, A., Martindale, C., Borkum, J. "Creativity, synesthesia and physiognomic

perception," Creativity Research Journal 10, 1-8 (1997).

11. Daniels, Dieter and Sarah Neuman. Audiovisuology: A Compendium. Koln: Walther Konig, 2010.

12. Duchtung, Hajo. Paul Klee: Painting Music. Munich: Prestel Verlag, 2004.

23

13. Eskilon, Stephen. “Thomas Wilfred and Intermedia: Seeking a Framework for Lumia.” Leonardo, Vol. 36, No. 1 (2003), pp. 65-68.

14. Galeyev, Bulat. ”What is synaesthesia: myth and reality” "Leonardo Electronic Almanac, vol.7 nr. 6 (1999). http://prometheus.kai.ru/mif_e.htm.

15. Galeyev, Bulat. “Synaesthesia and musical space. On Yavorsky's forgotten hypothesis "Leonardo, vol. 26, nr. 1 (1993), pp. 76 - 78.

16. Hertz, Paul. “Synesthetic Art: An Imaginary Number?” Leonardo, Vol. 32, No. 5, Seventh New York Digital Salon (1999), pp. 399-404.

17. Higgins, Dick. Horizons: The Poetics and Theory of the Intermedia. U.S.: Southern

Illinois University Press, 1984.

18. Howes, David. Scent, Sound and Synaesthesia: Intersensoriality and Material Culture

Theory. In: Tilley, Christopher et al. (eds.);. Handbook of Material Culture (pp. 161-72).

London, England: Sage, 2006.

19. Ione, A. "Kandinsky and Klee: Chromatic Chords, Polyphonic Painting and

Synesthesia." Journal of Consciousness Studies 11, no. 3-4 (2004), pp. 148-58.

20. James, David E. editor. Stan Brakhage: Filmmaker. Philadelphia: Temple University

Press, 2005.

21. Kandinsky, Wassily. Concerning the Spiritual in Art. New York: Dover, 1977.

22. Kandinsky, Wassily. Point and Line to Plane. New York: Dover, 1979. Quoted in Leggio, James, editor. Music and Modern Art. New York: Routledge, 2002.

23. Leggio, James, editor. Music and Modern Art. New York: Routledge, 2002.

24. Li, X., Tao, D., Maybank, S.J. and Yuan, Y. “Visual music and musical vision.”

Neurocomputing, Volume: 71, Issue: 10-12 (June 2008), pp: 2023-2028.

24

25. Mallary, Robert. "Spatial-synesthetic art through 3-D projection: The requirements of

a computer-based supermedium," Leonardo 23, no.1 (1990), pp. 3-16.

26. Miller, Dorothy, curator. 15 Americans, exh. cat., (New York: Museum of Modern Art, 1952). Quoted in Eskilon, Stephen. “Thomas Wilfred and Intermedia: Seeking a Framework for Lumia.” Leonardo, Vol. 36, No. 1 (2003), pp. 65-68.

27. Moritz, William. “The dream of color music, and machines that make it possible.”

http://www.awn.com/mag/issue2.1/articles/moritz2.1.html

28. Mueller, Robert Emmet. "Visic: A proposal for true color music." Leonardo, 32, no. 3

(1999) 177-179.

29. Ox, Jack and Peter Frank. “The Systematic Translation of Musical Compositions into Paintings.” Leonardo, Vol. 17, No. 3 (1984), pp. 152-158.

30. Ox, Jack "Color me synesthesia " Leonardo 32, no. 1 (1999a), 7-8.

31. Ox, Jack. "Synesthetic fusion in the digital age." Leonardo, 32, no. 5 (1999b), 391-392

32. Ox, Jack. "Creating a visual translation of Kurt Schwitters's 'Ursonate'," Leonardo Musical Journal 3 (1993), 59-61.

33. Poast, Michael. "Color music. Visual color notation for musical expression". Leonardo, 33, 3 (2000), 215-221.

34. Peacock, Kenneth. “Instruments to Perform Color-Music: Two Centuries of Technological Experimentation.” Leonardo, Vol. 21, No. 4 (1988), pp. 397-406

35. Ramachandran, V.S. and Hubbard, E.M. “Synaethesia – a window into perception, thought and language.” Journal of Consciousness Studies, 8, 3 – 34, (2001). Quoted in Jamie Ward. The Frog Who Croaked Blue. UK: Routledge, 2008.

36. van Campen, Crétien . The Hidden Sense: Synesthesia in Art and Science.

Cambridge: MIT Press, 2007.

37. van Campen, Crétien .“Artistic and Psychological Experiments with Synesthesia.” Leonardo, Vol. 32, No. 1 (1999), pp. 9-14.

25

38. Ward, Jamie. The Frog Who Croaked Blue. UK: Routledge, 2008.

39. Whitney, John. Digital Harmony: On the Complementarity of Music and Visual Art. Tennessee: Kinsport Press, 1980.

40. Wilfred. Thomas. “Composing in the Art of Lumia” Author(s): The Journal of Aesthetics and Art Criticism, Vol. 7, No. 2 (Dec., 1948).

41. Youngblood, Gene. Expanded Cinema. New York: E.P Dutton and Co., 1970.

FILMOGRAPHY

1. Arabesque by John Whitney and Larry Cuba. http://www.youtube.com/watch?v=HnlQa74H7is&feature=player_embedded#at=45 (accessed April 1, 2011).

2. John Whitney – Matrix III (1972) http://www.youtube.com/watch?v=ZrKgyY5aDvA&feature=player_embedded#at=134 (accessed April 1, 2011).

WEBSITES

1. Artificial Vision for the Totally Blind. http://www.artificialvision.com/(accessed March 16, 2011).

2. Center for Visual Music. http://www.centerforvisualmusic.org/(accessed March 17, 2011).

3. Colour Music. http://home.vicnet.net.au/~colmusic/ (accessed March 18, 2011).

4. Carpenter, Siri. “Everyday fantasia: The world of synesthesia.” American Pschological Association Monitor. Vol. 32, no. 3 (March 2001) http://www.apa.org/monitor/mar01/synesthesia.aspx (accessed March 24, 2011).

5. Galeyev, Bulat M. “Synesthesia: Myths and Reality.” http://prometheus.kai.ru/mif_e.htm (accessed March 25, 2011).

6. Hanus, Julie. Your Brain is a Pharmacy. The Utne Reader, September – October, 2008. http://www.utne.com/2008-09-01/Science-Technology/Your-Brain-Is-a-Pharmacy.aspx?page=2#ixzz1I7ZMgRfU (accessed April 2, 2011).

26

7. Heyrman, Hugo. “Dr. Hugo Heyrman (Motions of the Mind).” http://www.doctor. hugo.org/synaesthesia/index.htm

(accessed March 20, 2011).

8. McDonnell, Laura. “Colour and Sound.” .http://homepage.eircom.net/~musima/visualmusic/visualmusic.htm#top (accessed March 18, 2011).

9. Moritz, William. “The Dream of Color Music and the Machines that Made it Possible.” http://www.awn.com/mag/issue2.1/articles/moritz2.1.html (accessed March 18, 2011).

10. MULTI: Multi-user Laser Texture Interface. http://www.vacuumwoman.com/multi/index.html (accessed March 18, 2011).

11. Natalia Sidler. http://www.nataliasidler.ch/index.php (accessed March 30, 2011).

12. Ox, Jack. “The 21st Century Virtual Reality Color Organ.”http://home.bway.net/jackox/colororgansite/ and http://www.jackox.net/ and http://www.jackox.net/IEEE.pdf (accessed March 16, 2011).

13. Ox, Jack. “Im January am Nil.” Music by Clarence Barlow. http://www.jackox.net/pages/introImJan.html (accessed April 7, 2011)

14. RhythmicLight.com. http://rhythmiclight.com/index.html (accessed March 20, 2011).

15. Stadnick, George O. “The Art of Lumia.” http://www.photonlightguitars.com/ (accessed March 17, 2011).

16. Synesthesia Artforum. http://synesthesia-artforum.blogspot.com/ (accessed March 22, 2011).

17. The Online Visual Music Forum: Natural and Invented Visual Embodiments of Music. http://www.ronpellegrinoselectronicartsproductions.org/Pages/VisualMusicForum.html (accessed March 20, 2011).

18. Thorisson, K. R. and K. Donoghue. “Synthetic Synesthesia.” http://xenia.media.mit.edu/~kris/synthsyn.html (accessed March 22, 2011).

27

19. Wikipedia. “Color Organ.” http://en.wikipedia.org/wiki/Color_organ (accessed March 17, 2011).

20. Wikipedia. “Gesamtkunstwerk” http://en.wikipedia.org/wiki/Gesamtkunstwerk (accessed March 26, 2011).

21. Wikipedia. “History of Synesthesia Research.” http://en.wikipedia.org/wiki/History_of_synesthesia_research (accessed March 26, 2011).

22. Vimeo. “Kandinsky Sonified by Todd Barton.” (http://vimeo.com/6766060). (Accessed March 3, 2011).