Visual Texture, Music, and Emotion:Cross-modal associations between visual textures and musicThomas Langlois, Joshua Peterson & Stephen E. Palmer, Department of Psychology, University of California, Berkeley
METHODS
STIMULI
BACKGROUND RESULTS
EXPERIMENT 1EXPERIMENT 1 CONCLUSION
RESULTSEXPERIMENT 2EXPERIMENT 2
REFERENCES Emotion Dimensions
Not Angry−Angry
Active−Passive
Smooth JazzPiano
FolkMozart
CountryBach
Big BandIndie
SoundtrackBalkin Folk
ReggaeHip Hop
Progressive Rock80s Pop
ElectronicArabic
Hindustani SitarBlues
DixielandJazz
AlternativeFunk
GamelanSalsa
BluegrassIrish
TranceClassic Rock
Progressive HousePsychobillyStravinsky
SkaDubstep
Heavy Metal
PianoBach
CountrySmooth Jazz
MozartBalkin Folk
FolkIndie
ReggaeBig Band
Hip HopSoundtrack
80s Pop Alternative
BluesGamelan
Progressive RockArabic
IrishJazz
DixielandHindustani Sitar
FunkSalsa
BluegrassElectronic
TranceProgressive House
PsychobillyStravinsky
Classic RockSka
DubstepHeavy Metal
Harmonious−Disharmonious Orderly or Simple−Chaotic or Complex
Sharp−Smooth
Happy−Sad
Texture-Music Association (TMAD ): The weighted average of the three textures picked as being most consistent with the music (Cd,m)
minus the weighted average of the three textures picked as being most inconsistent with the music ( Id,m) along a given dimension D:
Cd,m = (3c1,d,m+ 2c2,d,m+ c3,d,m)/6, Id,m = (3i1,d,m+ 2i2,d,m+ 13,d,m)/6, TMAd,m = Cd,m- Id,m
1st 3rd Choices2nd
2AFC Experiment
CountryBalkin FolkBachGamelan
FolkBig BandSmooth JazzReggae
DubstepMozartPsychobilly80s Pop
IrishPianoElectronicSka
Progressive RockSalsaHeavy MetalSoundtrack
Calm−Agitated
PianoBachFolk
MozartCountry
Smooth JazzBalkin Folk
IndieBig Band
SoundtrackReggaeHip Hop
BluesHindustani Sitar
JazzDixielandGamelan
Progressive Rock Alternative
80s PopArabic
IrishFunk
ElectronicBluegrass
SalsaPsychobilly
Classic RockProgressive House
StravinskyTrance
SkaDubstep
Heavy Metal
Heavy MetalDubstep
SkaStravinsky
Classic RockTrance
PsychobillyProgressive House
IrishFunk
BluegrassSalsa
GamelanElectronicDixieland
ArabicJazz
Hindustani SitarBlues
Progressive Rock80s Pop
AlternativesHip HopReggae
SoundtrackIndie
Big BandBalkin Folk
FolkCountry
Smooth JazzMozart
BachPiano
Ska Heavy Metal Classic Rock
Trance Progressive
House Psychobilly
Stravinsky Gamelan
Soundtrack Funk
Hip Hop 80s Pop Reggae
Big Band Alternative
Balkin Folk Country
Piano Indie
Mozart Hindustani Sitar
Smooth Jazz
Electronic Jazz
Blues Irish
Bach
Folk Salsa
Progressive Rock
Arabic Bluegrass
Direct Music Ratings
Texture Music Associations (TMA)
MozartPiano
FolkBach
CountrySmooth Jazz
Balkin FolkIndie
Big BandReggaeHip Hop
Soundtrack Alternative
80s PopGamelan
BluesIrish
Progressive RockJazz
DixielandBluegrass
ArabicHindustani Sitar
ElectronicFunk
SalsaClassic Rock
PsychobillyProgressive House
StravinskyTrance
SkaDubstep
Heavy Metal
Heavy MetalDubstep
SkaProgressive House
StravinskyTrance
PsychobillyClassic Rock Alternative
IrishBluegrassGamelan
BluesJazz
SalsaFunk
Hindustani SitarBig Band
80s PopProgressive Rock
DixielandHip HopReggae
ElectronicBachIndie
MozartArabic
FolkSoundtrack
CountryPiano
Balkin FolkSmooth Jazz
80s PopReggae
FolkSoundtrack
DixielandArabic
Smooth JazzCountry
MozartProgressive Rock
ElectronicFunk
Balkin FolkPianoBlues
Hip HopBig Band
Bluegrass Alternative
Progressive HouseTrance
GamelanHindustani Sitar
BachIrish
Classic RockIndie
SalsaPsychobilly
JazzHeavy Metal
StravinskySka
Dubstep
High correlations indicate support for the emotion mediation hypothesis (Palmer, S. E., Schloss, K., et al (2013)
MDS of Emotional Space for thirty-four genres of music based on pairwise correlations of average emotional ratings
Dimension 1
Dim
ensi
on 2
Happy
Sad
Agitated
Calm
-1.5 -1 -0.5 0 0.5 1 1.5-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
Salsa
Progressive Rock
ArabicBluegrass
Alternative
Blues
Gamelan
Bach
Balkan Folk
Country Classic Rock
Dixieland
TranceProgressive House
Ska
Electronic
Piano
IrishSoundtrack
Indie
Funk
Hip Hop
80s Pop
Psychobilly
Mozart
DubstepHeavy Metal
Hindustani Sitar
Stravinsky
Jazz
Reggae
Smooth Jazz
Big Band
Folk
Correlations between the emotional content of the musicand the corresponding emotional content of the colors chosen to go with it
p < .05
0.2
0.4
0.6
0.8
1.0
Corr
elat
ion
(r)
Calm/Agitated
Happy/Sad
Whim./Serious
Appeal./Disgust.
Pref.
Thirty-four musical genres clustered by similar texture pro�les*:
Associations among music, texture, emotion, and non-emotional audio-visual features:
SkaHeavy Metal
GamelanPsychobilly
Progressive HouseDixieland
SalsaDubstep
TranceBig Band
FunkReggae80s Pop
SoundtrackClassic Rock
BluegrassStravinskyElectronic
Smooth JazzHip Hop
ArabicJazz
Progressive RockIrish
Balkin FolkBlues
CountryHindustani Sitar
PianoFolk
BachIndie
AlternativeMozart
Granular−Fibrous
ArabicSalsa
DixielandElectronic
FunkProgressive Rock
Hindustani SitarSmooth Jazz
Soundtrack80s PopHip Hop
PianoReggae
BluegrassBalkin Folk
Big BandTrance
IndieJazzSka
BluesGamelan
StravinskyCountry
Classic RockMozart
FolkPsychobilly
Heavy MetalBach
DubstepIrish
Progressive House Alternative
Curved−Straight
Non-synesthetes make systematic associations between 34 musical genres and color.The associations are mediated by emotion (Whiteford, Schloss, Palmer, VSS-13).
Twenty-eight di�erent visual textures synthesized from spline curves:
Do cross-modal associations exist between music and visual texture?
Hindustani SitarIndieDixielandBluegrass
Thirty-four excerpts of songs, each representing a unique musical genre: StravinskyFunkTranceAlternative
1
Sound-Texture Choices
1) Pick 3 textures that are most consistent with the music.
Music-Feature Ratings
HappySad
HappySad
Texture-Feature Ratings
2) Pick 3 textures that are least consistent with the music.
1 2
3
Rate features of each stimulus:
Simple-Complex
Granular-Fibrous
Slanted-Not Slanted
Curved-Straight
Separate-Connected
Sharp-Smooth
Natural-Arti�cial
Happy-Sad
Calm-Agitated
Weak-Strong
Active-Passive
Not Angry-Angry
Harmonious-Disharmonious
r = .92**** MUSIC RATING
TMA r = .95**** MUSIC RATING
TMA
r = .89**** MUSIC RATING
TMA
r = .89**** MUSIC RATING
TMA r = .91**** MUSIC RATING
TMAr = .84**** MUSIC RATING
TMA
r = .55***MUSIC RATING
TMA r = .44**MUSIC RATING
TMA
JazzHip HopHouseBlues
Classic RockArabic
−150 −100 −50 0 50 100 150Dixieland
CountrySmooth Jazz
Hip HopPiano
Balkin FolkBach
SoundtrackFolk
Hindustani SitarGamelan
FunkMozart
JazzBig Band
IndieReggae80s Pop
Progressive RockSalsa
ElectronicBlues
SkaIrish
ArabicStravinsky
PsychobillyBluegrass
AlternativeHeavy Metal
Classic RockProgressive House
TranceDubstep
Strong Weak
r = .75**** MUSIC RATING
TMA
00.
51
00.
51
Visual Dimensions An
gry-
Not A
ngry
Calm
-Agi
tate
d
Wea
k-St
rong
Natu
ral-A
rti�c
ial
Natu
ral-A
rti�c
ial
Shar
p-Sm
ooth
Shar
p-Sm
ooth
Curv
ed-S
traig
htCu
rved
-Stra
ight
Gran
ular
-Fib
rous
Gran
ular
-Fib
rous
Sim
ple-
Com
plex
Sim
ple-
Com
plex
Harm
onio
us-D
ishar
mon
ious
Harm
onio
us-D
ishar
mon
ious
Activ
e-Pa
ssiv
eAc
tive-
Pass
ive
Right BetterLeft Better
ThickThin
Dark Background
Medium Brightest
Darkest Medium
BluegrassBlues
GamelanBach
Dixieland
HouseFunk
Piano
Heavy Metal
Folk
Varia
nce
Expl
aine
d
20
40
60
80
100
0
Thickness
Contrast
Lightness
As with music-color cross-modal associations, non-synesthetes appear tomake systematic cross-modal associations from music to texture.
Music-texture cross-modal associations were mediated by emotion, but unlike music-color associations, happy-sad was not relevant to texture. Music-texture associations were also mediated by other visual features.
Variance explained by perceived lightness,contrast, and thickness of visual textures
Shared features:Angry-Not Angry
Harmonious-DisharmoniousActive-PassiveWeak-Strong
Texture-Music features:Sharp-Smooth
Curved-StraightGranular-Fibrous
Color-Music features:Happy-Sad
Listen to Music
Evaluate Emotional content and other
features of the music Texture-Music Association Task:People chose textures with match-ing emotional content, and match-ing non-emotional audio-visual content
Color-Music Association Task:People chose colors with matching emotional content
Happy-Sad does not mediate texture-music associations
r = -.03MUSIC RATING
TMA
Visual Dimensions Emotional Dimensions
*Contrast of texture corresponds to frequency of choice across all subjects
Direct Music Ratings
Texture Music Associations (TMA)
Direct Music Ratings
Texture Music Associations (TMA)
Direct Music Ratings
Texture Music Associations (TMA)Direct Music Ratings
Texture Music Associations (TMA)
Direct Music Ratings
Texture Music Associations (TMA)
Direct Music Ratings
Texture Music Associations (TMA)Direct Music Ratings
Texture Music Associations (TMA)
Direct Music Ratings
Texture Music Associations (TMA)
Direct Music Ratings
(TMA)
First color picks for �rst twenty participants
Bright Background
Acknowledgements: We thank Will Griscom, Jan Flatley-Feldman, Vivian Wung, Candita Wager, Melissa Gertler, Sheila Rajagopalan for their help with this project. This material is based on work supported by the National Science Foundation under Grants Nos. 1059088 and 0745820, and by a Google Gift to the senior author, Stephen E. Palmer.
-Whiteford, K., Schloss, K., Palmer, S., (VSS-2013). “Bach to the Blues” Presented at the 14th Annual meeting of the Vision Sciences Society, Naples, FL, May 2013.
(r)
−150 −100 −50 0 50 100 150−150 −100 −50 0 50 100 150
−150 −100 −50 0 50 100 150 −150 −100 −50 0 50 100 150 −150 −100 −50 0 50 100 150
−150 −100 −50 0 50 100 150−150 −100 −50 0 50 100 150−150 −100 −50 0 50 100 150
−150 −100 −50 0 50 100 150
**** p < 0.00001 *** p < 0.0001 ** p < 0.001 * p < 0.01
-Palmer, S. E., Schloss, K., B., Xu, Z. & Prado-Leon, L., (2013) Music-color associations are mediated by emotion, Proceedings of the National Academy of Sciences. 110 (22), 8836-8841.
Top Related