McGurk Doesn’t Work: Evidence Against the McGurk Effect...
Transcript of McGurk Doesn’t Work: Evidence Against the McGurk Effect...
Synthetic-Lab Natural-Lab Natural-MTurk
ForcedChoice
OpenEnded
Auditory Fusion Visual Other
McGurk Doesn’t Work: Evidence Against the McGurk Effect as a Perceptual Illusion
RESULTS
Laura M. Getz & Joseph C. Toscano[laura.getz, joseph.toscano]@villanova.edu
DISCUSSION
Visualspeechcuesplayanimportantroleinspeechrecognition,andtheMcGurkeffectisaclassicdemonstrationofthis.
REFERENCES
CURRENT EXPERIMENTS
INTRODUCTION
MacDonald,J.,&McGurk,H.(1978).Visualinfluencesonspeechperceptionprocesses. Perception&Psychophysics.
Mallick,D.B.,Magnotti,J.F.,&Beauchamp,M.S.(2015).VariabilityandstabilityintheMcGurkeffect:Contributionsofparticipants,stimuli,time,andresponsetype. Psychonomic Bulletin&Review.
Massaro,D.W.(1998). Perceivingtalkingfaces:Fromspeechperceptiontoabehavioralprinciple.MITPress.
McGurk,H.,&MacDonald,J.(1976).Hearinglipsandseeingvoices. Nature.
Toscano,J.C.,&Lansing,C.R.(2017).Age-relatedchangesintemporalandspectralcueweightsinspeech. LanguageandSpeech.
Expt. Subjects Report“Ba”
Report“Ga”
Report“Da”
McGurk&MacDonald(1976)
3-5yr (n=21) 19% 0% 81%
7-8 yr (n=28) 36% 0% 64%
18-40 yr (n=54) 2% 0% 98%
MacDonald&McGurk (1978) 18-24 yr (n=44) 9% 27% 64%
Wesetouttosystematicallylookattheseindividualdifferences,investigatinganumberoffactorsthatcouldinfluencefusionrates.
Ø Participantdifferences:labvs.online• Lab:VillanovaUniversityIntroPsychologystudents
Agerange:18-21years• Online:Amazon’sMechanicalTurk(MTurk)
Agerange:21-72years
Ø Stimulusdifferences:syntheticvs.natural• Synthetic:Klatt-synthesizedaudio;/ɑ/vs./æ/vowelcontexts
Baldi visuallipmovements;/ba/vs./da/CombinedaudioandvideousingiMovie
• Natural:2maleand2femaletalkers(Mallick etal.,2015)CongruentAVstimuliseparatedandrecombinedinaudB-visG andvisG-audB combinationsusingiMovie
Ø Designdifferences:open-endedvs.3-alternativeforcedchoiceAskedtoreport:Whatdidthespeakersay?
HEAR SEE REPORT
“ba” “da”“ga”
StimulusParticipant Design
McGurk&MacDonad’s explanationfortheillusory“fusion”effectdealswiththewaythesoundsarearticulated.
Bilabial Alveolar Velar
Voiced /b/ /d/ /g/
Voiceless /p/ /t/ /k/
Morerecentworkshowsthattheeffectmaynotbeasrobustaspreviouslybelieved,astheproportionoffusionresponsesdependsonindividualandtaskdifferences(Mallick etal.,2015).
Ø Lower proportionoffusionresponsesoverallthaninoriginalexperiments• Open-endedMTurk fusionresponserate(0.38)similartoMallick etal.
(2015)with“tha”includedasafusionresponse
• Participantdifferences:more fusionresponsesonMTurk thaninlab• Onerelevantindividualdifferencemaybeage,witholderparticipants
more likelytoshowfusioneffect• Thissuggeststhatphoneticcueweightscontinuetochangeacrossthe
lifespan,inlinewithpreviouswork(Toscano &Lansing,2017)
• Stimulusdifferences:syntheticstimuliresultedinmore “other”responses,suggestingthatdespitehighcontrol,wemayneedtousenaturalstimulitoseefusioneffect
• Designdifferences:ineachexperiment,more fusionresponsesto3-alternativeforced-choicethanopen-endeddesign(cf.Mallick etal.,2015)• Similarproportionoffusionresponseswithsinglemodalitytrials
integrated withAVtrialsandblocked design
Ø Ratherthanarobustperceptualillusion,wearguethattheMcGurkeffectisaproductofindividualdifferencesandtaskdemands
• Maybeit’stimetofindamorereliableclassroomdemonstrationofvisualinfluenceonspokenwordrecognition?
SyntheticStimuli:LabParticipantsForcedChoice(N=24) Open-Ended(N=11)
BA DA GA BA DA GA combo otheraudioB 0.93 0.02 0.05 0.73 0.05 0.02 0.00 0.20audioD 0.01 0.95 0.04 0.01 0.72 0.02 0.00 0.25audioG 0.04 0.08 0.88 0.03 0.14 0.75 0.00 0.09visualB 0.89 0.05 0.06 0.52 0.04 0.04 0.00 0.40visualD/G 0.02 0.51 0.46 0.01 0.32 0.23 0.00 0.44AV-congruentB 0.89 0.05 0.06 0.41 0.00 0.02 0.00 0.55AV-congruentD 0.01 0.93 0.06 0.00 0.64 0.02 0.00 0.34AV-congruentG 0.00 0.02 0.98 0.00 0.01 0.94 0.00 0.03AV-audioB-visD/G 0.39 0.34 0.26 0.05 0.01 0.08 0.00 0.86AV-audioG-visB 0.10 0.02 0.89 0.04 0.02 0.88 0.00 0.04
NaturalStimuli:LabParticipantsForcedChoice(N=46) Open-Ended(N=46)
BA DA GA BA DA GA combo otheraudioB 0.98 0.02 0.00 0.98 0.00 0.00 0.00 0.01audioD 0.00 0.99 0.01 0.00 0.99 0.00 0.00 0.00audioG 0.00 0.01 0.99 0.00 0.00 1.00 0.00 0.00visualB 0.99 0.00 0.01 0.99 0.00 0.00 0.00 0.00visualD 0.00 0.91 0.09 0.01 0.85 0.11 0.00 0.03visualG 0.01 0.39 0.60 0.01 0.38 0.57 0.00 0.03AV-congruentB 0.99 0.01 0.00 0.99 0.00 0.01 0.00 0.00AV-congruentD 0.01 0.95 0.04 0.00 0.97 0.02 0.00 0.01AV-congruentG 0.00 0.06 0.94 0.00 0.05 0.94 0.00 0.00AV-audioB-visG 0.75 0.14 0.11 0.74 0.10 0.11 0.00 0.04AV-audioG-visB 0.21 0.01 0.79 0.23 0.00 0.74 0.01 0.00
NaturalStimuli:OnlineMTurk ParticipantsForcedChoice(N=37) Open-Ended(N=39)
BA DA GA BA DA GA combo otheraudioB 0.93 0.04 0.02 0.80 0.02 0.00 0.00 0.18audioD 0.02 0.91 0.07 0.00 0.94 0.03 0.00 0.03audioG 0.02 0.02 0.96 0.00 0.02 0.95 0.00 0.03visualB 0.90 0.08 0.02 0.89 0.02 0.02 0.00 0.06visualD 0.03 0.82 0.15 0.02 0.55 0.14 0.01 0.28visualG 0.04 0.50 0.46 0.01 0.35 0.43 0.01 0.20AV-congruentB 0.94 0.05 0.01 0.92 0.01 0.00 0.00 0.07AV-congruentD 0.01 0.93 0.06 0.00 0.95 0.03 0.00 0.02AV-congruentG 0.03 0.03 0.94 0.01 0.02 0.95 0.00 0.02AV-audioB-visG 0.49 0.41 0.09 0.37 0.17 0.06 0.00 0.40AV-audioG-visB 0.03 0.02 0.95 0.09 0.01 0.77 0.11 0.02
M=45years
M=35years
M=36years
M=39years