Mixing Interaction, Sonification, Rendering and Design - The art of creating sonic interactions

Mixing Interaction, Sonification, Rendering and Design - The art of creating sonic interactions

Eoin Brazil

EG_IT 22 & 23 October, Verona. Italy

Sonification

Using Sound to Effectively Convey Information

Ben Cohen, co-founder of Ben & Jerry’s Ice-cream Company

Talking against the size and cost of the nuclear stockpile in the United States.

Cartoonification

Maisy drinking with a straw

(From “Happy Birthday, Maisy” by Lucy Cousins, Candlewick Press, 1998.)

Chapter 9. High-level models: bouncing, breaking, rolling, crumpling, pouring 203

Figure 9.17: Maisy drinking with a straw (From “Happy Birthday, Maisy” byLucy Cousins, Candlewick Press, 1998.)

audio-cartoon of a filling process.As an example for a more abstract use of our sound objects, serves a son-

ification of a scene from children’s book. The combination of the streaming-object with a comb-filter could be interactively controlled simultaneously witha simple adjustable cardboard figure in the book: an “aristocratic mouse”,drinking lemonade with a straw! In the book, the reader can regulate the fluidlevel of the mouse’s drinking glass (see Figure 9.17).

That control might be coupled with the delay time (i.e. base frequency)of the comb filter (see Figure 9.18). Together with typical envelopes for astarting/finishing streaming-action an additional inspiring acoustic feedback isgiven. Of course, drinking with a straw generally does not generate a soundclose to this one, if any remarkable noise at all. Nevertheless, the free associ-ation of this moving liquid-like sound, with that cartoon scene, seems natural.Asked to spontaneously sonify this scene with their voice, many people wouldprobably produce similar noises, although it does not really sound like having-a-cocktail (hopefully).

Area-1 Area-2 Area-3 Area-4 Area-5 Area-6 Area-7 Area-8

0.333 1 0.333 0.692 0.176 0.138 0.354 0.058

0.554 1 0.554 1.232 0.074 0.198 0.358 0.065

1.028 1 1.028 0.576 0.070 0.077 2.401 0.162

0.217 1 0.217 0.885 0.259 1.087 0.739 1.005

0.507 1 0.507 0.862 0.198 0.309 0.578 1.107

0.570 1 0.570 0.866 0.301 0.434 0.526 0.674

0.554 1 0.554 1.232 0.074 0.198 0.358 0.065

0.530 1 0.530 0.042 0.852 0.250 0.053 0.185

0.497 1 0.497 1.590 1.365 1.175 1.278 0.101

0.167 1 0.167 1.261 0.031 0.658 0.306 1.289

0

0.4

0.8

1.3

1.7

2.1

2.5

Area-1 Area-2 Area-3 Area-4Area-5 Area-6 Area-7 Area-8

An Example of Auditory Graphing

Complex datasets - Adding sound can elaborate streams and temporality of the data

Sonic Interaction Design

• Sound and interaction are both process oriented

• Interactive systems are complex - sound is able to convey complex information very quickly

• Interactive systems are “disappearing”, sound can tell us about their presence and their status

• We have learned to understand complex soundscapes -> ubiquitous, visualisation, and many areas of computing may profit from the use of sound

SHAPE ExhibitionRe-Tracing the Past

Interactive Sound Object Models for Exploration & Education

SolidsFractureModel

Braking

Breaking Sliding

Footsteps on gravel

Can crushing

Falling coin

Rolling / breaking

wheel

Squeaking door

Paper crumpling

Struck string Struck bar Glass

cleaningBowed string

Rubbed glass

Fracturing

Walking

Crumpling

Crushing Bouncing

Hitting Rolling Rubbing

SqueakingDropping

ImpactModel

FrictionModel

Fluid flowingModel

Flowing Burbing

Splashing

BubbleModel

Dripping

FillingPouring

TurbulenceModel

Whoosing

Sucking

Burning Exploding Popping

Vacuum cleaner

Explosion Model

Liquids GasesLow LevelModels

Basic Eventsand Textures

DerivedProcesses

SimulationExamples

SOb-ifying the Exhibition

Impact Models for Hitting, Rolling, or Crumbling Textures / Basic Events

Virtual Touch

MachineInteracting with one of the four `mysterious’ objects

Using a multimodal (Audio-Haptic-Visual) interface to explore its materiality and size

Real-Time Interaction

SOb-ifying the Exhibition

http://www.soundobject.org/software/impact_modal.tgz















SOb - Sounding Object Project

Interactive Sound Object Models for Exploration & Education

Museum Installations - Lessons & Considerations

• Enhance interpretation

• Fun yet interactive, spark

• Active not passive environment / interfaces

• Design activities / consultations

✦ observational studies

✦ consult docents /experts

✦ understand physical/spatial constraints

Soundscapes & Train Stations

J. Tardieu, P. Susini, F. Poisson, P. Lazareff, and S. McAdamsFrance, 2008

BouncingBalls

Frequency / Amplitude

Airplane Synth

Mirror Triggers

Connecting Strangers at a

Train Station

L. Pellarin, N. Bottcher, J. M. Olsen, O. Gregersen, S. Serafin, and M. GuglielmiDenmark, 2005

Shannon Portal - Auditory Display for

Public Spaces

Interactive Sounds for Communication & Entertainment

Ping Photon Drive

Wind Shepard Tones

Analysis of the Sound & Space

Proceedings of the 15th International Conference on Auditory Display, Copenhagen, Denmark, May 18-22, 2009

ICAD09-3

3.1. First design: Ping

Our first display used an 80 ms decaying pulse from two

sine wave oscillators 4 Hz apart, pitched from 200 Hz to 800

Hz. The rate of pulses ranged from 0.3 to 5 Hz. The output was

amplitude panned left-right. In essence, one could describe this

simple sound design as “a machine that goes ping”. We tried the

following mapping:

• Amount of movement – Pulse rate

• Left-Right – Panning

• Front-Back - Pitch

Participants found this display to be clear but too intrusive

(“sounds like a truck reversing”). Part of the intrusiveness in the

sound design is due to the sharp onset, which is strongly

attracting attention. The pitch used was in the middle of the

background soundscape spectrum, hence would require to be

loud to avoid masking. All participants found the mapping easy

to understand.

3.2. Second design: PhotonDrive

Our second display used a pulse with approximately 0.3 s

attack and 0.3 s release from two sawtooth oscillators tuned 2%

apart, pitched from 50 Hz to 650 Hz. The output was amplitude

panned. Our nickname for this sound design was PhotonDrive.

The mapping used in our tests was the same as in the previous

trial.

The softer onset and richer spectrum resulted in participants

feeling that the sound design was less intrusive. Due to the

richer spectrum, the sound could be played at lower loudness

level, but was still found to be masked at times.

3.3. Third design: Wind

In our third display we tried a less intrusive sound, a wind-

like sound that was based on subtractive synthesis using white

noise sources that were bandpass filtered with a certain amount

of jitter in the center-frequencies to make it sound more

realistic. The average center frequency was mapped to the

amount of movement.

This sound was noted as not being intrusive as there was no

distinct onset, but due to its naturalness, basically being a

parametrically controlled auditory icon, and only two mappings

(activity – frequency, left-right panning), also this display was

discarded.

3.4. Fourth design: Shepard tones

In our fourth and final display we used a slightly more

radical approach. This sound design was based on the Shepard

tone illusion (Shepard, 1964). This illusion is perceived as

eternally rising or falling pitch and is due to the continuous

frequency change and mix of ten or more partials. We carefully

adjusted the partials to a quite unnatural timbre (bell-like) to try

to avoid masking effects in relation to the surrounding

soundscape.

• Amount of movement – loudness

• Left-Right – Panning

• Front-Back – Pitch direction

Participants found this display to be positively intriguing, as

it did not sound like anything they had heard before. Due a soft

onset, as we mapped amount of movement to loudness, it was

noted as both being pleasing as well as attention catching. A

participant or group of participants walking past and

accidentally being tracked and sonified immediately understood

that it was their movement that caused the sound. This finding

is similar to Todd Winkler’s installation Light around the edges

[10].

We finally moved the installation to Shannon airport where

it was installed for 3 weeks. We carried out both observations

and informal interviews during this period. Both passengers and

staff at the airport found the sound design contributing to their

experience and being non-intrusive, and our previous findings

confirmed. Our video analysis of people using the system on

site also confirmed that the auditory display served to catch

people’s attention when walking past the system.

In Figure 4, below, you can see an example sonogram of the

soundscape of Shannon Airport with the Shannon Portal

installed. Within the white square, you see the partials of the

auditory display increasing in frequency. The rest of the

sonogram is the background noise. Even in this visual

representation, the sonification stands out against the

background. The psychoacoustic features we are exploiting are

‘common fate’ and ‘good continuation’ [11].

Figure 4: Shannon Portal soundscape sonogram

4. DISCUSSION

There are a number of issues involved in this design. First

of all, it was important to create an auditory display that would

be somewhat subtle but always audible, but not annoying for the

staff working in the area. It also had to be immediately

informative, i.e. affordances [12-14] for immediate discovery

by casual users. The auditory display also had to be useful, i.e.

to actively provide users with information that increased their

feeling of engagement with the system.

Our overall design ambition was to develop a ubiquitous

computer system for a public setting. Instead of using

traditional metaphors, we aimed for a high degree if virtuality

[15], i.e. that the conceptual structure of the system would be

immediately obvious to the users instead of being weighed

down by traditional user interface clichés or forcing users to

read a manual before trying to use the system. This is always a

problem in computing for public environments, as most users

would never have encountered a system like this before (at

least, not the first time) and within a few seconds the system has

to communicate to potential users that it can interacted with and

how to proceed.

Throughout the design iterations, the first three designs had

some problems to fulfill the requirements, but the fourth and

final iteration was found to be a satisfactory solution to the

problem. We believe that the important finding here is to

facilitate multiple design iterations with evaluation, to help

Linking ubiquitious auditory displays design to evaluation in

public environments

Evaluate

Brainstorms, Concept Generation

Sonic Mood Boards

Sketching sound using clips (mood boards or

onomatopoeia)

Bodystorming / Roleplay the

Inteaction

Video Prototyping

Evaluate

Prototype using PD / Max/MSP or similar

Evaluate

Code interaction and sound object model

Evaluate

Auditory Public SpacesAuditory

Display Design Process for

Public Spaces

Proceedings of the 15th International Conference on Auditory Display, Copenhagen, Denmark, May 18-22, 2009

ICAD09-1

The Shannon Portal: Designing an Auditory Display for Casual Users in a

Public Environment

Author 1

Address

Address email@host

Author 2

Address

Address email@host

ABSTRACT

We developed an installation for a public environment and

casual users where auditory display was a significant element to

facilitate user interaction. We used an iterative design process,

starting from simple onomatopoeic representations, to complex

sound object models in Pure Data. The system was evaluated at

each stage, from the lab to the final public setting. The problems

addressed covered the representations of left-right, up-down,

and the amount of movement by the user or groups of users. In

addition to this, it was important that the auditory display would

attract attention when users were within control range of the

system, i.e. an affordance that invited and allowed users to

discover functionality.

1. INTRODUCTION

In our Shared Worlds project we explored designing

ubiquitous computing for public environments. Part of this

exploration was to design, build and evaluate installations in

public environments. The work reported in this paper is about

the Shannon Portal, a complex interactive installation in

Shannon Airport in the West of Ireland. The overall design is

described in [1], while in this paper we focus on the design of

the auditory display element of the installation. One of the

issues that we wanted to explore in the project was the use of

large ambient displays and public interaction with such

displays. As the installation at Shannon required an interactive

image gallery to be displayed, we designed a purpose-built

back-projection system. An overhead camera tracked users’

movements in front of the display, allowing the users to move a

virtual magnifying glass across the image gallery on screen, by

moving their body in front of the display. See Figure 1 and 2.

Over several iterations we developed hardware and software

for the display and explored different mappings of users’

actions to movement of the virtual magnifying glass on the

projection screen. For reasons outside the scope of this paper, it

was decided that it was not the users’ location that would

control the movement, but the users’ movement. The rationale

for this was that as there was only one virtual magnifying glass

but there could be multiple simultaneous users, this mapping

would encourage collaboration and engagement.

While testing and evaluating the system in the lobby outside

our lab in the Interaction Design Centre at the University of

Limerick, we noted that users who occasionally passed by the

display were not aware of that the display was interactive.

Furthermore, due to the nature of the mapping, some users

found it difficult to understand that it was their movement, not

their location, that controlled the virtual magnifying glass. To

remedy this, we decided to design an auditory display element

for the system.

Figure 1: Shannon Portal installation

Figure 2: Back-projection, loudspeakers and video

camera

1.1. The challenge of auditory display in public

environments

Is has been noted in numerous papers (e.g. [2-4]) that

auditory display can be annoying. A location such as the transit

hall of an airport has an inherently high background noise level,

at peak-time often over 85 dB. In the transit hall in Shannon

airport, the main contributors to the soundscape are people

walking, talking, handling luggage, eating and drinking, mobile

phone ring tones, tills in the Duty Free shop, all the activity in

the Sheridan Bar and bursts of announcements about flights

Left-Right Panning = Amplitude Panning

Front-B

ack =

Pitch

Direction

Movement within space = Loudness

1 2

3

Mappings used for fourth prototype (Shepard Tones)

Public Spaces - Lessons & Considerations

• Engaging (virtuality)

• Visitors / Working within the space

• Privacy / Intrusion

• Masking and soundscape issues (peak vs non-peak times)


✦ observational studies

✦ interviews

✦ understanding the physical /cultural / spatial constraints

Hanging Gardens - Auditory Installation

Collaboration with Jurgen Simpson

Linked two gallery spaces in Limerick and Belfast

Explore circadian rhythms of the spaces

Motion sensors and time of day, site-specific

Artistic Installations - Lessons & Considerations

• Interpretation vs artistic

• Engaging

• Group / Individual

• Consider temporal constraints


✦ aesthetic issues✦ distance /

repairability✦ understand

physical/spatial constraints

HCI Design and Interactive Sonification for Fingers and Ears

Action Sound Function

No touch N/A

Touch area outside button N/A

Enter button area Tick

Move finger on button Friction sound

Exit button area Tack

Lift finger off button Tock Select / Activate function

Where and How to start designing with Sound

• How to select and classify suitable sounds for a particular interaction design ?

• TaDa - select sounds to communicate the dimensions and directions with relation to their actions and to that of the system e.g. ears-lead-eyes design pattern

• Sounds linked to physical objects as dynamic entities with manipulation / interaction

• Multidimensional information carrier - size, shape, material, distance, speed, and emotional expression

Selection of Approaches Depending on Question

• Evaluating Perceptual Qualities of Single Auditory Stimuli

• Identification Tasks, Context-Based Ratings, Attribute Ratings

• Evaluating Relations Among Auditory Stimuli

• Discrimination Trials, Similarity Ratings, Sorting Tasks, MDS, CA, PCA, Pathfinder Analysis

• Evaluation of an application or an interface

• Surveys, Verbal Protocols, Heuristics, Task lists, etc.

Sound Creation

Sound Analysis

Context and Auditory Display Definition

ConsiderationsActivity, Sound Like,

Vocalisations / Onomatopoeia

Find Closest Match in F/X Library / Use

Foley

Analyse Sound andConsider Action/s and

Objects making it

Create the Sound and Test - Does it sound

right?

Musical Sounds

Everyday Sounds

ConsiderationsActivity, Mapping,

Instruments

Explore Existing Earcon Hierarchies /

Create New Hierarchy

Create Earcon Family and Test - Does it

sound right?

Evaluate Scalings / Mappings

Elicit Descriptors & Constructs

Rating of Constructs & Descriptor Categorisation

Personal ConstructsAssociated RatingsDescriptor Categories

Causal Uncertainty Measures Structuring Of Constructs

Definition Of AttributesConstruction Of Scales

Validation Of Scales

Category Refinement

Evaluation

Results

Attribute Scales

Groups of Constructs

Preliminary Results

Confused Sounds

Inconsistent Attributes or MappingsConfused Sounds

Inconsistent Attributes or MappingsConfused Sounds

Auditory Characterisation of Story / Scene / Account

TaDa Analysis Sonic Map Creation for Account / Scene / Story

Structuring Of ConceptsHearsay Analysis

Groups of Concepts

Evaluation

ResultsInconsistent Attributes or MappingsConfused Sounds

Mismatched Sounds

Narrative Sound Artefact Creation

1

2 2

22

2

3&4

3&4

5

7

12 13

14

15

16

17

6

8 8

11

17

9

10

Sound Creation

Sound Analysis

Review of existing approaches,

methods, and techniques.

Select Listening Panel, Sounds

& Attributes to explore

Play audio stimuli (~100-200)

Sta

ge

1 -

Dev

elo

p c

ons

ensu

s

des

crip

tive

lang

uag

e

Determine attribute scales,

descriptors, key audio groups

Sta

ge

2 -

Par

ticip

ant

per

form

ance

dat

a

Gather end

user

preference

Gather global

scores from

naive

participants

Sta

ge

3 -

Map

pin

g

dat

a to

pre

fere

nces

Map preference / performance

data using statistical methods

Select Listeners, Sounds &

Attributes to explore

Play audio stimuli (~100-200)

Stag

e 1 - Develo

p ind

ividual

descrip

tive languag

e Determine attribute scales,

descriptors, key audio groups

Stag

e 2 - Particip

ant

ranking / rating

of d

ata

Participant ranks, rates, or

groups data

Stag

e 3 - Map

ping

of d

ata

Mapping of data using

statistical methods

Group or Individual based approaches

Functional ArtefactsLinking design to evaluation

including basic design methods

Sonic Interaction Design Process for

Functional Artefacts

Interaction MaterialsAnalyse,Describe

Interaction Gestalt

Experience in Context

Interaction Artefact

Shape, Combine

Evaluate

Functional Artefacts CLOSED project, 2008, Ircam, Verona, Berlin

Rolling

Clapping 1

Clapping 2Examples:

Explorations and Methods

• Listening test approaches

• Textual descriptor analysis

• Sonic Mapping

• Causal uncertainty

• 2-D perceptual scaling

• Repertory grid technique

Background Sounds Foreground Sounds

Contextual

Sounds

Sig

nals

Sig

nals

Sig

nals

Emotions

Actions

Actions

Actions

Emotions

Emotions

Visible HiddenWritten descriptions of sounds and their type

Music

Speech

Everyday

Abstract /Unknown

*+0

?

*

*+ ?

00

Sonic MappingGraham Coleman, PhD ThesisScotland, 2008

Listening

Sorting

Classifying

1 2 3 4 5 6 7 8 9

1 2 3 4 5 6 7 8 91

2

3

4

5

6

7

8

9

1

2

3

4

5

6

7

8

9alarmed excitedastonished

afraid delighted

angry happy

annoyed

frustrated

pleased

content

serenemiserable

depressed

bored relaxed

tired sleepy

High Arousal

Low Arousal

Dis

ple

asu

re Pleasu

re

calm

none

close/sitstroke

petting

Zizi - The Affectionate

Couch

Stephen Barrass, Linda Davy & Kerry RichensAustralia, 2003

Repertory Grid George Kelly,USA, 1955

Listening

Sorting

Classifying

Source / Object / Type of

Interaction

Dendogram from Repertory Grid

George Kelly,USA, 1955

Clusters of `related’ or `similar’ sounds

Advice and tips for achieving better sound designs

Selecting and classify suitable sounds

Verify your mappings

Rapid prototyping with tools such as PD, SuperCollider, Chuck, etc.

Contexts / Users / Environment

Mixing Interaction, Sonification, Rendering and Design - The art of creating sonic interactions

Education

Transcript of Mixing Interaction, Sonification, Rendering and Design - The art of creating sonic interactions