The texture of musical sounds · Cross-modal associations from musical timbres and intervals to...
1
-150 150 0 -150 150 0 -150 150 0 -150 150 0 Simple - Complex Happy - Sad Angry - Not Angry r = .69* MUSIC RATING TSA r = .83** MUSIC RATING TSA r = .83** MUSIC RATING TSA Sharp - Smooth r = .86**** MUSIC RATING TSA TIMBRES INTERVALS The texture of musical sounds: Cross-modal associations from musical timbres and intervals to visual textures Joshua Peterson, Thomas Langlois, & Stephen E. Palmer, Department of Psychology, University of California, Berkeley Sound-Texture Choices 1.) Pick 3 textures that are most consistent with the sound. Sound-Feature Ratings Happy Sad Happy Sad Texture-Feature Ratings METHODS 28 different textures synthesized using b-splines 1 2 3 4 5 6 7 8 9 10 11 12 # Semitones: Flute Rhodes Piano Marimba Celeste Violin (Legato) Tuba Trumpet Trombone Oboe French Horn English Horn Clarinet Cello Bassoon Violin (Staccato) Harpsichord TEXTURES Minor Seventh Minor Third Major Second Major Seventh Minor Second Perfect Octave Perfect Fifth Perfect Fourth Major Third Major Sixth Minor Sixth Diminished Fifth TEXTURES 12 Intervals: Piano tone at middle C paired with all other possible notes in the chromatic scale 16 timbres generated using naturalistic recordings STIMULI BACKGROUND RESULTS EXPERIMENT 1 EXPERIMENT 1 Timbres and intervals clustered by similar texture profiles* METHODS & RESULTS Evaluate emotional content of sound Do these findings generalize to another salient visual domain: line-based geometrical textures? INTERVALS 2.) Pick 3 textures that are least consistent with the sound. 1 2 3 Rate features of each stimulus: Perceptual Dimensions Simple-Complex Granular-Fibrous Slanted-Not Slanted Curved-Straight Separate-Connected Sharp-Smooth Natural-Artificial Emotional Dimensions Happy-Sad Calm-Agitated Weak-Strong Active-Passive Not Angry-Angry Harmonious-Disharmonious TIMBRES INTERVALS 2AFC Task Right Better Left Better Associations among sound, texture, emotion, and non-emotional audio-visual features: Texture-Sound Associations (TSA D ): The weighted average of the three textures picked as being most consistent with music (C d,m ) minus the weighted average of the three textures picked as being most inconsistent with the music (I d,m ) along a given dimension D: EXPERIMENT 2 EXPERIMENT 2 Happy−Sad Not Angry−Angry Calm−Agitated Weak−Strong Active−Passive Harmonious−Disharmonious Simple−Complex Granular−Fibrous Slanted−Not−Slanted Curved−Straight Separate−Connected Sharp−Smooth Natural−Artificial -1 -0.5 0 0.5 1 C d,m = (3c 1,d,m + 2c 2,d,m + c 3,d,m )/6 I d,m = (3i 1,d,m + 2i 2,d,m + 1 3,d,m )/6 TSA d,m = C d,m - I d,m 1 st 3 rd Choices 2 nd Happy−Sad Not Angry−Angry Calm−Agitated Weak−Strong Active−Passive Harmonious−Disharmonious Simple−Complex Granular−Fibrous Slanted−Not−Slanted Curved−Straight Separate−Connected Sharp−Smooth Natural−Artificial -1 -0.5 0 0.5 1 Acknowledgements: We thank Will Griscom, Jan Flatley-Feldman, Vivian Wung, Candita Wager, Melissa Gertler, Sheila Raja- gopalan for their help with this project. This material is based on work supported by the National Science Foundation under Grants Nos. 1059088 and 0745820, and by a Google Gift to the senior author, Stephen E. Palmer. REFERENCES SEE THIS POSTER ONLINE CONCLUSION Thick Thin Dark Backgr ound Medium Bright Background Brightest Darkest Medium Happy-Sad Angry-Calm Active-Passive Strong-Weak 0 0.5 1 TIMBRES INTERVALS Previous research has provided evidence that cross-modal music-to-color associations are mediated by emotion, both for classical orchestral music and for a wide range of genres. Similar results suggesting emotional mediation have been found using lower level musical stimuli, including musical melodies, two-note intervals, and instrumental timbres. (Palmer et al., 2013, Whiteford et al., VSS-2013) (Palmer et al., VSS-2011, Griscom & Palmer, VSS-2012) Correlations between the emotional associations of the sounds, and the emotional associations of the colors chosen to go with the sounds. People choose colors with matching emotional content Stimuli: 40 textures (10 patterns x 4 variations) Variance Explained Minor Second Major Second Perfect Fourth Perfect Fifth Minor Sixth Perfect Octave 0 20 40 60 80 100 Bassoon Celeste Cello Flute Harpsichord Marimba Piano Rhodes Trumpet Violin (Legato) 0 20 40 60 80 100 Variance Explained TIMBRES INTERVALS Lightness Contrast Thickness PROCEDURE Subjects compared every pair of the 4 variations of the same texture for fit with each sound. FOLLOW-UP QUESTION Do perceptual features of the textures such as lightness, contrast, and line thickness influence the fit of the texture with the sound? Texture-Sound Association Task: People chose textures with matching emotional content, and matching non-emotional audio- visual content Color-Music Association Task: People chose colors with matching emotional content Regression model predicting texture choice from thickness, contrast, and lightness The majority of variance is accounted for by lightness The results suggest that associations from low-level musical stimuli to visual textures may be mediated by corresponding dimensions in particular perceptual and emotional qualities. Listen to sounds Evaluate emotional content and other features of the sound Griscom, W., Palmer, S., (VSS-2013). “Cross-modal Sound-to-Sight Associations with Musical Timbre in Non-Synesthetes” Presented at the 14th Annual meeting of the Vision Sciences Society, Naples, FL, May 2013. Whiteford, K., Schloss, K., Palmer, S., (VSS-2013). “Bach to the Blues” Presented at the 14th Annual meeting of the Vision Sciences Society, Naples, FL, May 2013. Griscom, W., Palmer, S., (VSS-2013). “The Color of Musical Sounds in Non-Synesthetes” Presented at the 13th Annual meeting of the Vision Sciences Society, Naples, FL, May 2012. Palmer, S. E., Schloss, K., B., Xu, Z. & Prado-Leon, L., (2013) Music-color associations are mediated by emotion, Proceedings of the National Academy of Sciences. 110 (22), 8836-8841. Correlation coefficient (r) *Lightness of texture corresponds to frequency of choice across all subjects Correlation coefficient (r) Correlation coefficient (r) DIMENSIONS PERCEPTUAL EMOTIONAL DIMENSIONS PERCEPTUAL EMOTIONAL Bassoon Celeste Cello Clarinet English Horn Flute French Horn Harpsichord Marimba Oboe Piano Rhodes Trombone Trumpet Tuba Violin (Legato) Violin (Staccato) Bassoon Celeste Cello Clarinet English Horn Flute French Horn Harpsichord Marimba Oboe Piano Rhodes Trombone Trumpet Tuba Violin (Legato) Violin (Staccato) Minor Second Major Seventh Minor Seventh Major Second Diminished Fifth Major Sixth Minor Third Minor Sixth Perfect Fifth Perfect Fourth Perfect Octave Major Third Minor Second Major Seventh Minor Seventh Major Second Diminished Fifth Major Sixth Minor Third Minor Sixth Perfect Fifth Perfect Fourth Perfect Octave Major Third Bassoon Celeste Cello Clarinet English Horn Flute French Horn Harpsichord Marimba Oboe Piano Electric Piano Trombone Trumpet Tuba Violin
Transcript of The texture of musical sounds · Cross-modal associations from musical timbres and intervals to...
−150 −100 −50 0 50 100 150
Texture Timbre Associations: Not Angry−Angry
-150 1500
−150 −100 −50 0 50 100 150
Texture Timbre Associations: Sharp−Smooth
-150 1500
−150 −100 −50 0 50 100 150
Texture Interval Associations: Happy−Sad
-150 1500
−150 −100 −50 0 50 100 150
Texture Interval Associations: Orderly or Simple−Chaotic or Complex
-150 1500
Simple - Complex
Happy - SadAngry - Not Angry
r = .69* MUSIC RATING
TSA r = .83** MUSIC RATING
TSA
r = .83** MUSIC RATING
TSA
Sharp - Smooth
r = .86**** MUSIC RATING
TSA
TIMBRES INTERVALS
The texture of musical sounds:Cross-modal associations from musical timbres and intervals to visual texturesJoshua Peterson, Thomas Langlois, & Stephen E. Palmer, Department of Psychology, University of California, Berkeley
Sound-Texture Choices
1.) Pick 3 textures that are most consistent with the sound.
Sound-Feature Ratings
HappySad
HappySad
Texture-Feature Ratings
METHODS
28 different textures synthesized using b-splines
1 2 3 4 5 6 7 8 9 10 11 12# Semitones:
Flute
RhodesPiano
MarimbaCeleste
Violin (Legato)Tuba
TrumpetTrombone
OboeFrench Horn
English HornClarinet
Cello Bassoon
Violin (Staccato) Harpsichord
TEXTURES
Minor SeventhMinor Third
Major Second
Major SeventhMinor Second
Perfect OctavePerfect Fifth
Perfect FourthMajor Third
Major SixthMinor Sixth
Diminished Fifth
TEXTURES
12 Intervals: Piano tone at middle C paired with all other possible notes in the chromatic scale
16 timbres generated using naturalistic recordings
STIMULI
BACKGROUND RESULTS
EXPERIMENT 1EXPERIMENT 1
Timbres and intervals clustered by similar texture profiles*
METHODS & RESULTS
Evaluate emotional content of sound
Do these findings generalize to another salient visual domain: line-based geometrical textures?
INTERVALS
2.) Pick 3 textures that are least consistent with the sound.
1 2
3
Rate features of each stimulus:
Perceptual Dimensions
Simple-Complex
Granular-Fibrous
Slanted-Not Slanted
Curved-Straight
Separate-Connected
Sharp-Smooth
Natural-Artificial
Emotional Dimensions
Happy-Sad
Calm-Agitated
Weak-Strong
Active-Passive
Not Angry-Angry
Harmonious-Disharmonious
TIMBRES INTERVALS
2AFC Task
Right BetterLeft Better
Associations among sound, texture, emotion, and non-emotional audio-visual features:Texture-Sound Associations (TSAD): The weighted average of the three textures picked as being most consistent with music (Cd,m) minus the weighted average of the three textures picked as being most inconsistent with the music (Id,m) along a given dimension D:
EXPERIMENT 2EXPERIMENT 2
Happy−Sad
Not Angry−Angry
Calm−Agitated
Weak−Strong
Active−Passive
Harmonious−Disharmonious
Simple−Complex
Granular−Fibrous
Slanted−Not−Slanted
Curved−Straight
Separate−Connected
Sharp−Smooth
Natural−Artificial
-1 -0.5 0 0.5 1
Cd,m = (3c1,d,m+ 2c2,d,m+ c3,d,m)/6 Id,m = (3i1,d,m+ 2i2,d,m+ 13,d,m)/6 TSAd,m = Cd,m- Id,m
1st 3rd Choices2nd
Happy−Sad
Not Angry−Angry
Calm−Agitated
Weak−Strong
Active−Passive
Harmonious−Disharmonious
Simple−Complex
Granular−Fibrous
Slanted−Not−Slanted
Curved−Straight
Separate−Connected
Sharp−Smooth
Natural−Artificial
-1 -0.5 0 0.5 1
Acknowledgements: We thank Will Griscom, Jan Flatley-Feldman, Vivian Wung, Candita Wager, Melissa Gertler, Sheila Raja-gopalan for their help with this project. This material is based on work supported by the National Science Foundation under Grants Nos. 1059088 and 0745820, and by a Google Gift to the senior author, Stephen E. Palmer.
REFERENCES
SEE THIS POSTER ONLINE
CONCLUSION
ThickThin
Dark Background
Medium
Bright Background
Brightest
Darkest Medium
Happy-Sad
Angry-Calm
Active-Passive
Strong-Weak
0 0.5 1
TIMBRES
INTERVALS
Previous research has provided evidence that cross-modal music-to-color associations are mediated by emotion, both for classical orchestral music and for a wide range of genres.
Similar results suggesting emotional mediation have been found using lower level musical stimuli, including musical melodies, two-note intervals, and instrumental timbres.
(Palmer et al., 2013, Whiteford et al., VSS-2013)
(Palmer et al., VSS-2011, Griscom & Palmer, VSS-2012)
Correlations between the emotional associations of the sounds, and the emotional associations of the colors chosen to go with the sounds.
People choose colors with matching
emotional content
Stimuli: 40 textures (10 patterns x 4 variations)
Vari
ance
Exp
lain
ed
Min
or S
econ
dM
ajor
Sec
ond
Perf
ect F
ourt
hPe
rfec
t Fift
hM
inor
Six
thPe
rfec
t Oct
ave
0
20
40
60
80
100
Bass
oon
Cele
ste
Cello
Flut
eHa
rpsi
chor
dM
arim
baPi
ano
Rhod
esTr
umpe
t
Viol
in (L
egat
o)
0
20
40
60
80
100
Vari
ance
Exp
lain
ed
TIMBRES INTERVALS
Lightness
Contrast
Thickness
PROCEDURESubjects compared every pair of the 4 variations of the same texture for fit with each sound.
FOLLOW-UP QUESTION
Do perceptual features of the textures such as lightness, contrast, and line thickness influence the fit of the texture with the sound?
Texture-Sound Association Task:People chose textures with matching emotional content, and matching non-emotional audio-visual content
Color-Music Association Task:People chose colors with matching emotional content
Regression model predicting texture choice from thickness, contrast, and lightness
The majority of variance is accounted for by lightness
The results suggest that associations from low-level musical stimuli to visual textures may be mediated by corresponding dimensions in particular perceptual and emotional qualities.
Listen to sounds
Evaluate emotional content and other
features of the sound
Griscom, W., Palmer, S., (VSS-2013). “Cross-modal Sound-to-Sight Associations with Musical Timbre in Non-Synesthetes” Presented at the 14th Annual meeting of the Vision Sciences Society, Naples, FL, May 2013.
Whiteford, K., Schloss, K., Palmer, S., (VSS-2013). “Bach to the Blues” Presented at the 14th Annual meeting of the Vision Sciences Society, Naples, FL, May 2013.
Griscom, W., Palmer, S., (VSS-2013). “The Color of Musical Sounds in Non-Synesthetes” Presented at the 13th Annual meeting of the Vision Sciences Society, Naples, FL, May 2012.
Palmer, S. E., Schloss, K., B., Xu, Z. & Prado-Leon, L., (2013) Music-color associations are mediated by emotion, Proceedings of the National Academy of Sciences. 110 (22), 8836-8841.
Correlation coefficient (r)
*Lightness of texture corresponds to frequency of choice across all subjects
Correlation coefficient (r) Correlation coefficient (r)
DIM
ENSIO
NS
PERCEPTU
AL
EMO
TION
AL
DIM
ENSIO
NS
PERCEPTU
AL
EMO
TION
AL
BassoonCeleste
CelloClarinet
English HornFlute
French HornHarpsichord
MarimbaOboePiano
RhodesTrombone
TrumpetTuba
Violin (Legato)Violin (Staccato)
BassoonCeleste
CelloClarinet
English HornFlute
French HornHarpsichord
MarimbaOboePiano
RhodesTrombone
TrumpetTuba
Violin (Legato)Violin (Staccato)
Minor Second
Major Seventh
Minor Seventh
Major Second
Diminished Fifth
Major Sixth
Minor Third
Minor Sixth
Perfect Fifth
Perfect Fourth
Perfect Octave
Major Third
Minor Second
Major Seventh
Minor Seventh
Major Second
Diminished Fifth
Major Sixth
Minor Third
Minor Sixth
Perfect Fifth
Perfect Fourth
Perfect Octave
Major Third
Bassoon
Celeste
Cello
Clarinet
English Horn
Flute
French Horn
Harpsichord
Marimba
Oboe
Piano
Electric Piano
Trombone
Trumpet
Tuba
Violin
