Post on 08-Apr-2015
A Proposal for a Research Project Investigating a Vibrotactile Musical Instrument
IntroductionTechnology and art seem to continuously influence and challenge each other in interesting ways. For
example, the development of new technologies for composing, performing, transcribing and recording
music has been substantial. As new music art forms have been derived from these technologies (e.g.,
Hip-Hop using turn table “scratching”), new technologies have been developed to fully support the new
art form (e.g., software scratching tools are now available). Digital technologies have caused an
explosion of devices, technologies and techniques that have revolutionized many art forms. For
example, devices such as electronic synthesizers allow artists to compose, record and perform music in
ways that were not possible or even conceived of prior to the invention of computers. Digital processing,
sensing and storage technologies have driven new paradigms of music such as music controlled through
gestures or dance (Camurri et al., 2000; Ng, 2002; Winkler, 1995), or music without an audio component
or that is only seen (Chew & Francois, 2003; Smith & Williams, 1997), and now tactile music (Gunther &
O'Modhrain, 2003; Karam, Nespoli, Russo, & Fels, 2009). The focus of this research proposal will be the
exploration, development and evaluation of artistic and technological tools for tactile music.
New technology has enabled the possibility of a vibrotactile music system, in which musical patterns,
delivered entirely through vibration presented to the skin can be developed. This system would be
capable of supporting the creation and delivery of music, which can contain many, if not all, of the
elements of traditional audio based music, such as intensity, tempo, rhythm, even pitch and timbre, but
is completely devoid of an audio component. Instead, this form of music would be comprised entirely of
vibrotactile stimulation which is delivered to the skin and perceived through the tactile sensory channel.
A complete vibrotactile music system or instrument will have an input controller, which I will design and
build, and a vibrotactile display. This combination of an input control and a vibrotactile display will result
in the first vibrotactile instrument.
Research Objective The purpose of this research is to begin to scientifically explore a new art form called vibrotactile music
and some of the issues related to it. The purpose of this exploration is to begin to develop a model of
vibrotactile composition that will provide a foundation for this new art form. Using a pre-existing high
definition vibrotactile display called The Emoti-Chair and several versions of “yet-to-be-built” vibrotactile
music interfaces, some of the many technological, psychological, physiological, neurological and human
factor issues that vibrotactile music raises will be investigated. Research methods traditionally employed
in the field of Human Factors will be used to determine the affect of the vibrotactile music interface on
the creation of vibrotactile music, and on further iterations of the music interface design. Using methods
from the fields of psychology, psychophysics and neurology, the abilities of the human tactile system to
perceive the resulting vibrotactile music and the affect such stimulation has on people’s attitudes,
enjoyment and experiences of music will be explored.
BackgroundThis research will draw on several areas of scholarship and research including music theory and
composition, psychology, psychophysics, neuroscience, human factors, human computer interaction and
haptics. It will consist of a technological extension of the work carried on The Emoti-Chair (C. Branje,
Karam, Russo, & Fels, 2009; Karam, Branje, Price, Russo, & Fels, 2007; Karam & Fels, 2008; Karam,
Nespoli, Russo, & Fels, 2009) as well as an evaluation of artistic processes and audience reactions
resulting from the introduction of this extension. The Emoti-chair was developed to explore the
translation of audio-based music into vibrotactile-based music in order to provide greater access to film
sound and music for the deaf and hard-of-hearing. Initial studies have shown that conveying emotional
properties of music through the Emoti-Chair is possible (Branje, Fels, Russo, & Nespoli, 2010; Karam,
Branje, Price, Russo, & Fels, 2007).
The Emoti-ChairThe Emoti-Chair (C. Branje, Karam, Russo, & Fels, 2009) is a vibrotactile display capable of delivering
complex vibrotactile information to a seated user. Embedded in the Emoti-Chair are sixteen voice coils
that serve as vibrotactile stimulators. Various models of this vibrotactile display exist, however most
commonly the Emoti-Chair has eight independent information channels, meaning eight separate
vibrotactile signals can be delivered to the display simultaneously. Each channel is independent and
capable of vibrating at all frequencies within the tactile range (20 Hz - 1000 Hz). The Emoti-Chair uses
the conceptual framework of the Model Human Cochlea (MHC) developed by Karam et. al (2008). The
primary principle of the MHC is to model the distribution of frequencies on the skin according to how it
is accomplished by the human cochlea. The frequency components of an audio signal are divided and
spread across the human back as though the cochlea were uncoiled and stretched out forming a line
along the back.
Although many vibrotactile interfaces have been developed and examined (Gemperle, Ota, & Siewiorek,
2001; Hogema, De Vries, Van Erp, & Kiefer, 2009; Nanayakkara, 2009), the Emoti-Chair is one of the few
that use voice coils as the vibrotactile stimulator. Most often other devices use motors, such as those
found in massage chairs or cell phones. Using voice coils offers several advantages over using motors to
produce the vibrotactle stimuli. First, there is independent control of the frequency and amplitude
variables. This means that many traditional aspects of audio music such as melody and intensity will
translate easily to tactile music. Second, voice coils can be driven using widely available commercial
amplification equipment, making prototyping and eventual manufacture of a vibrotactile display using
voice coils economically feasible.
Audio Psychophysics The human sense of hearing facilitates speech, the enjoyment of music, navigation of the world and
many other human endeavours. There is a great deal of literature concerning the psychophysics of
human hearing going back almost a century. Since the early 20 th century, it has been accepted that the
human frequency range of hearing is approximately 20Hz to 20000 Hz, with a significant amount of
individual variability (Pumphrey, 1951). The commonly accepted threshold for the lowest energy sound
perceptible by humans is Root Mean Square (RMS) sound pressure of 20 µPa (Gelfand, 1998) but as with
the frequency range, this threshold is subject to individual differences. It will be important to consider
any knowledge about hearing when developing a tactile instrument because although the ear will not be
studied explicitly in this thesis, information known about the human ear and the sense of hearing will
influence the study of the tactile system. This is an approach similar to the one used to study the Emoti-
Chair in that human cochlea influenced the model human cochlea (MHC) used in the Emoti-Chair
(Karam, et al. 2007). Since the two perceptual systems are so similar it is likely that the vase amount of
knowledge knowing about the sense of hearing will be useful in the study of the tactile sense.
Tactile PsychophysicsTo a much lesser degree than for human hearing, there has been some work exploring the vibrotactile
perceptual ability of humans. It has been found that the ability of humans to detect vibration through
the tactile channel is analogous to the ability to perceive vibrations through the audio channel (Von
Bekesy, 1959)(reference), although the perceptual fidelity of the skin tactile system is much lower than
the auditory system (Von Bekesy, 1959). For example, the human frequency range of hearing is 20Hz to
20000 Hz but the range of vibrotactile frequency perception in the skin is approximately 20Hz to
1000Hz, with maximum sensitivity to amplitude at approximately 250Hz. In addition, the vibrotactile
receptors in the skin detect differences in frequencies with a much lower resolution than the auditory
system (Boothroyd & Cawkwell, 1970; Mahns, Perkins, Sahai, Robinson, & Rowe, 2006; Pertovaara &
Hamalainen, 1981; Pongrac, 2008; Verrillo, 1963). These important distinctions between audio and
tactile stimuli and the human ability to perceive them, must be taken into account when a vibrotactile
instrument is developed. For example, it would make little sense to present vibrotactile stimulation with
frequencies above 1000 Hz which the skin is extremely poor at perceiving. Similarly it would be ill-
advised to create vibrotactile “notes” with frequencies that are too close together since the skin would
not be able to differentiate them. This would be analogous to creating a piano (admittedly a very large
one) with its highest note at 30 kHz and intervals between notes of only 1 Hz.
Tactile NeurologyThere has been some neurological evidence collected through fMRI studies suggesting that the
processing of vibration stimulation applied to the skin as vibrotactation and to the ear as sound, may
occur in some overlapping areas of the brain (Foxe, 2009; Schürmann, Caetano, Hlushchuk, Jousmäki, &
Hari, 2006; Yau, Olenczak, Dammann, & Bensmaia, 2009). This may have important consequences
regarding the perception of vibrotactile music and may offer some explanation as to why participants
react similarly to music played auditorily and vibrotactally (C. Branje, Fels, Russo, & Nespoli, 2010;
Karam et al., 2007; Karam & Fels, 2008; Karam, Nespoli, Russo, & Fels, 2009). Although no explicit brain
function models will be developed as a result of this work, data collected through EEG may provide
additional evidence that corroborate the data collected through physiological measures or self reports
and may help to clarify what is occurring with the human audience members.
Tactile DisplaysTactile displays have been used by the deaf and hard-of-hearing for numerous applications and over
many years. The earliest known formal use of the tactile system for perceiving sound information is the
Tadoma method (Chomsky, 1986) in which a deaf person will place his hands on a speaker’s throat and
lips in order to feel the vibrations and perceive the speech produced. As electronic technology
improved, many devices were constructed to improve on the Tadoma method. Another simple and
common tactile display for used by the deaf and hard-of-hearing were simple audio speakers used
during what is called ``speaker listening``, where during which an individuals will place her his hands on
a single speaker in order to feel the vibrations produced. These early vibrotactile tactile devices often
used motors or voice coils arranged in an array to produce vibrations that were passively detected by
the skin. Other more recent developments, such as refreshable Braille which used piston-like actuators
(American Foundation for The Blind, 2010), depend on active perception. In most of these applications,
tactile displays were used to transmit speech, written text, directional information (Hogema et al., 2009)
or even visual information (Sampaio, Maris, & Bach-y-Rita, 2001). However, there has been little formal
research on the transmission of musical information through the tactile channel.
Tactile MusicGunther (Gunther & OʼModhrain, 2003) was one of the first researchers to investigate electronically
facilitated tactile music. He used a vibrotactile suit with 13 embedded transducers to explore the
production of tactile music. He suggested that vibrotactile music should contain elements of frequency,
intensity, duration, waveform and space similar to auditory-based music, but did not examine the
impact of tactile music on users/audiences, or the design and development of interfaces that were
explicitly intended for tactile music production. He did, however, remark that composing for the tactile
channel with tools designed for the audio domain was “painstaking” suggesting there is a need for a
tool.
The Emoti-Chair has been used to translate sound from the audio domain to the vibrotactile domain.
Although the skin’s ability to perceive vibration is analogous to the ear in many ways, the vibrotactile
perception of vibration produced from sound is still limited compared to that of the ear. This means that
it is likely not possible to perceive a large portion of the audio signal presented through a vibrotactile
display like The Emoti-Chair. For example, detecting different notes which differ in frequency by small
mounts is likely not possible with a vibrotactile display because of the reduced frequency discrimination
capability of the skin. These limitations of the tactile system may mean that a direct translation of music
from the audio domain into the tactile domain is not ideal or even possible. This suggests that perhaps
more focus should be paid to pure tactile music instead of trying to translate audio music into tactile
music.
One important advantage of the vibrotactile displays is that it is possible to spatially encode the tactile
stimuli. Tactile displays such as the Emoti-Chair can present vibrotactile signals to different locations on
the body which offers another dimension of encoding in addition to frequency and amplitude.
Vibrotactile stimuli of equal intensity and frequency can be presented at different spatial locations,
effectively distinguishing them. Although extremely complex information streams can be delivered
through the auditory channel it is nonetheless still only a two channel system as much of the
multiplexing abilities of the auditory system is due to signal processing functions of the brain. This fact
will have a large influence on how a vibrotactile instrument should be physically constructed and how
the vibrotactile information system will function.
Research MethodsA system that is capable of creating and delivering vibrotactile music will contain three main elements:
1) an input interface that allows users to interact with the system to express their musical intentions; 2)
display sub-system that provides vibrotactile music to audiences; and 3) a processing sub-system that
translates user interactions into the appropriate vibrotactile signals. Each of these sub-systems contains
hardware and software elements and functionality that must be created, integrated together and then
evaluated. I intend to use iterative create and evaluate cycles in order to construct the system. To begin,
I will use the processing and display sub-systems of the Emoti-chair and make modifications as
necessary. Current audio music interfaces including traditional software application such as Garage
Band, Adobe Audition and Pro Tools will be examined for applicability to the interface/interaction
development for my system. My main contributions will be the construction of the input interface or
“instrument”, the examination of the tactile music creation process by users, and the use and evaluation
of the entire system in actual performance situations.
For the iterative design, create and evaluate cycles, traditional usability research methods such as
usability studies with representative users and tasks, and focus groups will be employed to evaluate and
study the input interface (formative evaluation). The first objective in this phase of the process is to
examine ease of use and learning, match with the user’s content creation model and acceptance of the
control interface. In addition, the interface will be revised depending on issues and/or difficulties the
users encounter. Of particular interest in this aspect of the development process is the form factor for a
vibrotactile instrument interface should have in order to best facilitate the performance of vibrotactile
music.
Determining what is required to support the tactile musical creation process based on artist’s needs,
desires and abilities is the second objective in this first phase. Questionnaires, video analysis and
ethnographic techniques such as first hand observation and interviews? will be used to elicit and
understand the creative process. Artists will be commissioned to create vibrotactile compositions which
will be used to evaluate audience reactions in the next phase of the project.
In the next phase of the project, the entire system will be integrated together and then the impact of it
on audiences with be explored. Psychological and psychophysical methods will be employed to
investigate the effect it has on audiences. Psychophysical investigations such as just noticeable
differences will be used to determine optimal placement and distances for the placement of the tactile
feedback on an individual’s skin. User studies will be used to examine the impact of the system on user’s
understanding of the content, and experience of higher level concepts such as mood, rhythm and
contour. Biometric measures such as skin conductance level, heart rate variability, respiration, EMG
(measuring zygomaticus major and corrugator supercilii) will be used to probe the emotional status of
participants exposed to vibrotactile compositions. EEG measures may be considered to further
understand what is happening between the auditory and tactile centers in the brain as a result of
exposure to tactile music. In addition to the biometric measures, self report methods such as
questionnaires, interviews and continuous self report (Lottridge, 2008) will also be used. The goal of the
summative evaluation is to examine the impact of exposure to tactile music on an audience’s
enjoyment, appreciation and acceptance of tactile music. Statistical analyses such as means and
correlation testing will be used to analyze quantitative data collected through psychophysical
experiments, EEG, continuous or discrete self reporting. Thematic analysis will be used for qualitative
data such as that obtained through interview, open ended questionnaires or video recordings.
Proposed TimetableYear Title Milestones Months Description1 Initial Research Phase On Going
Literature Review
1-12
Emoti-Chair Workshops and Concerts with Musicians
8-12 Workshops with composers, singers, instrumentalists with current prototype of the high density vibrotactile display to get high level input on what vibrotactile music should or could be will be conducted. The information gained here will influence the first version of the input prototype
Public events where deaf and hearing Emoti-chair users can offer input into the design of the interface
Use video recording, questionnaires and interviewing techniques to collect data
Thematic analysis will be conducted on the video recordings and interview transcript will statistical analysis will be performed on any questionnaire data
2 Interface Construction / Display Evaluation
Continued Literature Review
25-36
Initial Design and Build Phase
13-19 Begin initial design of Vibrotactile Instrument Interface
Begin construction of Vibrotactile Instrument Interface
Collaborate with musicians and non-musicians in an interactive design process, incorporating feedback into design and construction of the improved prototype vibrotactile interfaces.
Publish findings on musician consultations and initial design of the vibrotactile instrument
Psychophysics 19-25 Complete an experiment that examines the
Experiments aesthetic quality of various types of vibration.
Characteristics such as frequency, amplitude, wave type in isolating and in combination with each other will be tested
Additional psychophysics experiment yet to be determined
Publish psychophysical findingsVibrotactile Music Created by Several Artists (Musicians, dancers, DJ’s)
25-29 Have musicians, composers and other artists create vibrotactile music to be later evaluated.
Traditional human factors methods such as interviews, video recording and questionnaires will be used to collect data during this phase
Give these musicians an emotional goal to strive for such as sad, happy, aggressive, frightening
3 Composition Evaluation
Continued Literature Review
25-36
Conduct emotional self report experiments of vibrotactile compositions
25-31 Compare the reaction to vibrotactile composition to the reaction to audio compositions using self report as the measure of affect
Publish emotional self report findingsConduct psychophysical experiments
25-31 Compare the reaction to vibrotactile composition to the reaction to audio compositions using self report as the measure of affect
Publish emotional self report findingsConduct EEG experiments
25-31 Compare the reaction to vibrotactile composition to the reaction to audio compositions using EEG as a measure of affect
Publish EEG findings4 Write-up / Follow-up Follow up
studies37-42 Additional studies to follow up on previous
studies as to be determinedComplete 37-48
Thesis
References
American Foundation for the Blind. (2010). Refreshable braille display. Retrieved 07/28, 2010, from http://www.afb.org/Section.asp?DocumentID=3652&SectionID=7&SubTopicID=97&TopicID=330
Boothroyd, A., & Cawkwell, S. (1970). Vibrotactile thresholds in pure tone audiometry. Acta Oto-Laryngologica, 69(1), 381-387.
Branje, C., Fels, D. I., Russo, F., & Nespoli, G. (2010). The effect of vibrotactile stimulation on the emotional response to horror films. Unpublished manuscript.
Branje, C., Karam, M., Russo, F., & Fels, D. I. (2009). Enhancing entertainment through a multimodal chair interface. Toronto International Conference – Science and Technology for Humanity, Toronto.
Camurri, A., Hashimoto, S., Ricchetti, M., Ricci, A., Suzuki, K., Trocca, R., et al. (2000). Eyesweb: Toward gesture and affect recognition in interactive dance and music systems. Computer Music Journal, 24(1), 57-69.
Chew, E., & Francois, A. R. J. (2003). MuSA.RT: Music on the spiral array. real-time. MULTIMEDIA '03: Proceedings of the Eleventh ACM International Conference on Multimedia, Berkeley, CA, USA. 448-449.
Chomsky, C. (1986). Analytic study of the tadoma method: Language abilities of three deaf-blind subjects. Journal of Speech and Hearing Research, 29(3), 332.
Cohen, B., & Kirman, J. (1986). Vibrotactile frequency discrimination at short durations. The Journal of General Psychology, 113(2), 179-186.
Foxe, J. J. (2009). Multisensory integration: Frequency tuning of audio-tactile integration. Current Biology, 19(9), R373-R375.
Gelfand, S. A. (1998). Hearing: An introduction to psychological and physiological acoustics (3rd ed., rev. and expanded. ed.). New York: Marcel Dekker.
Gemperle, F., Ota, N., & Siewiorek, D. (2001). Design of a wearable tactile display. Proceedings of the Fifth International Symposium on Wearable Computers, 5-12.
Goff, G. D. (1967). Differential discrimination of frequency of cutaneous mechanical vibration. Journal of Experimental Psychology, 74(2), 294-299.
Gunther, E., & OʼModhrain, S. (2003). Cutaneous grooves: Composing for the sense of touch. Journal of New Music Research, 32(4), 369-381.
Hogema, J. H., De Vries, S. C., Van Erp, J. B. F., & Kiefer, R. J. (2009) A tactile seat for direction coding in car driving: Field evaluation.
Karam, M., Branje, C., Price, E., Russo, F., & Fels, D. I. (2007). Towards a model human cochlea: Exploring vibetracks for crosssmodal audio-tactile displays. Submitted: GI'08: 34th Graphics Interface Conference,
Karam, M., & Fels, D. I. (2008). Designing a model human cochlea: Issues and challenges in crossmodal audio-haptic displays. Proceedings of the 2008 Ambi-Sys Workshop on Haptic User Interfaces in Ambient Media Systems Table of Contents,
Karam, M., Nespoli, G., Russo, F., & Fels, D. I. (2009). Modelling perceptual elements of music in a vibrotactile display for deaf users: A field study. Advances in Computer-Human Interactions, 2009.ACHI'09.Second International Conferences on, , 249-254.
Lottridge, D. (2008). Emotional response as a measure of human performance. CHI '08: CHI '08 Extended Abstracts on Human Factors in Computing Systems, Florence, Italy. 2617-2620.
Mahns, D., Perkins, N., Sahai, V., Robinson, L., & Rowe, M. (2006). Vibrotactile frequency discrimination in human hairy skin. Journal of Neurophysiology, 95(3), 1442-1450.
Nanayakkara, S., Taylor, E., Wyse, L., & Ong, S. H. (2009). An enhanced musical experience for the deaf: Design and evaluation of a music display and a haptic chair. Proceedings of the 27th International Conference on Human Factors in Computing Systems, 337-346.
Ng, K. (2002). Interactive gesture music performance interface. Proceedings of the 2002 Conference on New Interfaces for Musical Expression, 2.
Perovaara, A., & Hamalainen, H. (1981). Vibrotactile thresholds during background vibration of long duration. Scandinavian Journal of Psychology, 22(1), 41-45.
Pongrac, H. (2008). Vibrotactile perception: Examining the coding of vibrations and the just noticeable difference under various conditions. Multimedia Systems, 13(4), 297-307.
Pumphrey, R. (1950). Upper limit of frequency for human hearing. Nature, 166, 571.
Rothenberg, M., & Verrillo, R. (1976). Vibrotactile frequency for encoding speech parameters. The Journal of the Acoustical Society of America, 59, S69.
Sampaio, E., Maris, S., & Bach-y-Rita, P. (2001). Brain plasticity: ‘visual’ acuity of blind persons via the tongue. Brain Research, 908(2), 204-207.
Schürmann, M., Caetano, G., Hlushchuk, Y., Jousmäki, V., & Hari, R. (2006). Touch activates human auditory cortex. NeuroImage, 30(4), 1325-1331.
Smith, S. M., & Williams, G. N. (1997). A visualization of music. Visualization'97., Proceedings, 499-503.
Tommerdahl, M., Hester, K., Felix, E., Hollins, M., Favorov, O., Quibrera, P., et al. (2005). Human vibrotactile frequency discriminative capacity after adaptation to 25 hz or 200 hz stimulation. Brain Research, 1057(1-2), 1-9.
Verrillo, R. T. (1963). Effect of contactor area on the vibrotactile threshold. The Journal of the Acoustical Society of America, 35, 1962.
Von Bekesy, G. (1959). Similarities between hearing and skin sensations. Psychol Rev, 66(1), 1-22.
Winkler, T. (1995). Making motion musical: Gesture mapping strategies for interactive computer music. Proceedings of the 1995 International Computer Music Conference, 261–264.
Yau, J. M., Olenczak, J. B., Dammann, J. F., & Bensmaia, S. J. (2009). Temporal frequency channels are linked across audition and touch. Current Biology, 19(7), 561-566.