−150 −100 −50 0 50 100 150

Texture Timbre Associations: Not Angry−Angry

-150 1500

−150 −100 −50 0 50 100 150

Texture Timbre Associations: Sharp−Smooth

-150 1500

−150 −100 −50 0 50 100 150

Texture Interval Associations: Happy−Sad

-150 1500

−150 −100 −50 0 50 100 150

Texture Interval Associations: Orderly or Simple−Chaotic or Complex

-150 1500

Simple - Complex

Happy - SadAngry - Not Angry

r = .69* MUSIC RATING

TSA r = .83** MUSIC RATING

TSA

r = .83** MUSIC RATING

TSA

Sharp - Smooth

r = .86**** MUSIC RATING

TSA

TIMBRES INTERVALS

The texture of musical sounds:Cross-modal associations from musical timbres and intervals to visual texturesJoshua Peterson, Thomas Langlois, & Stephen E. Palmer, Department of Psychology, University of California, Berkeley

Sound-Texture Choices

1.) Pick 3 textures that are most consistent with the sound.

Sound-Feature Ratings

HappySad

HappySad

Texture-Feature Ratings

METHODS

28 different textures synthesized using b-splines

1 2 3 4 5 6 7 8 9 10 11 12# Semitones:

Flute

RhodesPiano

MarimbaCeleste

Violin (Legato)Tuba

TrumpetTrombone

OboeFrench Horn

English HornClarinet

Cello Bassoon

Violin (Staccato) Harpsichord

TEXTURES

Minor SeventhMinor Third

Major Second

Major SeventhMinor Second

Perfect OctavePerfect Fifth

Perfect FourthMajor Third

Major SixthMinor Sixth

Diminished Fifth

TEXTURES

12 Intervals: Piano tone at middle C paired with all other possible notes in the chromatic scale

16 timbres generated using naturalistic recordings

STIMULI

BACKGROUND RESULTS

EXPERIMENT 1EXPERIMENT 1

Timbres and intervals clustered by similar texture profiles*

METHODS & RESULTS

Evaluate emotional content of sound

Do these findings generalize to another salient visual domain: line-based geometrical textures?

INTERVALS

2.) Pick 3 textures that are least consistent with the sound.

1 2

3

Rate features of each stimulus:

Perceptual Dimensions

Simple-Complex

Granular-Fibrous

Slanted-Not Slanted

Curved-Straight

Separate-Connected

Sharp-Smooth

Natural-Artificial

Emotional Dimensions

Happy-Sad

Calm-Agitated

Weak-Strong

Active-Passive

Not Angry-Angry

Harmonious-Disharmonious

TIMBRES INTERVALS

2AFC Task

Right BetterLeft Better

Associations among sound, texture, emotion, and non-emotional audio-visual features:Texture-Sound Associations (TSAD): The weighted average of the three textures picked as being most consistent with music (Cd,m) minus the weighted average of the three textures picked as being most inconsistent with the music (Id,m) along a given dimension D:

EXPERIMENT 2EXPERIMENT 2

Happy−Sad

Not Angry−Angry

Calm−Agitated

Weak−Strong

Active−Passive

Harmonious−Disharmonious

Simple−Complex

Granular−Fibrous

Slanted−Not−Slanted

Curved−Straight

Separate−Connected

Sharp−Smooth

Natural−Artificial

-1 -0.5 0 0.5 1

Cd,m = (3c1,d,m+ 2c2,d,m+ c3,d,m)/6 Id,m = (3i1,d,m+ 2i2,d,m+ 13,d,m)/6 TSAd,m = Cd,m- Id,m

1st 3rd Choices2nd

Happy−Sad

Not Angry−Angry

Calm−Agitated

Weak−Strong

Active−Passive

Harmonious−Disharmonious

Simple−Complex

Granular−Fibrous

Slanted−Not−Slanted

Curved−Straight

Separate−Connected

Sharp−Smooth

Natural−Artificial

-1 -0.5 0 0.5 1

Acknowledgements: We thank Will Griscom, Jan Flatley-Feldman, Vivian Wung, Candita Wager, Melissa Gertler, Sheila Raja-gopalan for their help with this project. This material is based on work supported by the National Science Foundation under Grants Nos. 1059088 and 0745820, and by a Google Gift to the senior author, Stephen E. Palmer.

REFERENCES

SEE THIS POSTER ONLINE

CONCLUSION

ThickThin

Dark Background

Medium

Bright Background

Brightest

Darkest Medium

Happy-Sad

Angry-Calm

Active-Passive

Strong-Weak

0 0.5 1

TIMBRES

INTERVALS

Previous research has provided evidence that cross-modal music-to-color associations are mediated by emotion, both for classical orchestral music and for a wide range of genres.

Similar results suggesting emotional mediation have been found using lower level musical stimuli, including musical melodies, two-note intervals, and instrumental timbres.

(Palmer et al., 2013, Whiteford et al., VSS-2013)

(Palmer et al., VSS-2011, Griscom & Palmer, VSS-2012)

Correlations between the emotional associations of the sounds, and the emotional associations of the colors chosen to go with the sounds.

People choose colors with matching

emotional content

Stimuli: 40 textures (10 patterns x 4 variations)

Vari

ance

Exp

lain

ed

Min

or S

econ

dM

ajor

Sec

ond

Perf

ect F

ourt

hPe

rfec

t Fift

hM

inor

Six

thPe

rfec

t Oct

ave

0

20

40

60

80

100

Bass

oon

Cele

ste

Cello

Flut

eHa

rpsi

chor

dM

arim

baPi

ano

Rhod

esTr

umpe

t

Viol

in (L

egat

o)

0

20

40

60

80

100

Vari

ance

Exp

lain

ed

TIMBRES INTERVALS

Lightness

Contrast

Thickness

PROCEDURESubjects compared every pair of the 4 variations of the same texture for fit with each sound.

FOLLOW-UP QUESTION

Do perceptual features of the textures such as lightness, contrast, and line thickness influence the fit of the texture with the sound?

Texture-Sound Association Task:People chose textures with matching emotional content, and matching non-emotional audio-visual content

Color-Music Association Task:People chose colors with matching emotional content

Regression model predicting texture choice from thickness, contrast, and lightness

The majority of variance is accounted for by lightness

The results suggest that associations from low-level musical stimuli to visual textures may be mediated by corresponding dimensions in particular perceptual and emotional qualities.

Listen to sounds

Evaluate emotional content and other

features of the sound

Griscom, W., Palmer, S., (VSS-2013). “Cross-modal Sound-to-Sight Associations with Musical Timbre in Non-Synesthetes” Presented at the 14th Annual meeting of the Vision Sciences Society, Naples, FL, May 2013.

Whiteford, K., Schloss, K., Palmer, S., (VSS-2013). “Bach to the Blues” Presented at the 14th Annual meeting of the Vision Sciences Society, Naples, FL, May 2013.

Griscom, W., Palmer, S., (VSS-2013). “The Color of Musical Sounds in Non-Synesthetes” Presented at the 13th Annual meeting of the Vision Sciences Society, Naples, FL, May 2012.

Palmer, S. E., Schloss, K., B., Xu, Z. & Prado-Leon, L., (2013) Music-color associations are mediated by emotion, Proceedings of the National Academy of Sciences. 110 (22), 8836-8841.

Correlation coefficient (r)

*Lightness of texture corresponds to frequency of choice across all subjects

Correlation coefficient (r) Correlation coefficient (r)

DIM

ENSIO

NS

PERCEPTU

AL

EMO

TION

AL

DIM

ENSIO

NS

PERCEPTU

AL

EMO

TION

AL

BassoonCeleste

CelloClarinet

English HornFlute

French HornHarpsichord

MarimbaOboePiano

RhodesTrombone

TrumpetTuba

Violin (Legato)Violin (Staccato)

BassoonCeleste

CelloClarinet

English HornFlute

French HornHarpsichord

MarimbaOboePiano

RhodesTrombone

TrumpetTuba

Violin (Legato)Violin (Staccato)

Minor Second

Major Seventh

Minor Seventh

Major Second

Diminished Fifth

Major Sixth

Minor Third

Minor Sixth

Perfect Fifth

Perfect Fourth

Perfect Octave

Major Third

Minor Second

Major Seventh

Minor Seventh

Major Second

Diminished Fifth

Major Sixth

Minor Third

Minor Sixth

Perfect Fifth

Perfect Fourth

Perfect Octave

Major Third

Bassoon

Celeste

Cello

Clarinet

English Horn

Flute

French Horn

Harpsichord

Marimba

Oboe

Piano

Electric Piano

Trombone

Trumpet

Tuba

Violin