Costs and Cues in Code-switched Lexical Access

Alice Shen

ISB11

University of Limerick

Code-switching

• Is a common practice among bilingual speakers

Code-switching

• Is a common practice among bilingual speakers

• But incurs processing costs in:

o Spoken word recognition (Soares & Grosjean, 1984)

o Visual word recognition (Thomas & Allport, 2000)

o Language production (Meuter & Allport, 1999)

Code-switching

• Is a common practice among bilingual speakers

• But incurs processing costs in:

o Spoken word recognition (Soares & Grosjean, 1984)

o Visual word recognition (Thomas & Allport, 2000)

o Language production (Meuter & Allport, 1999)

Spoken word recognition: lexical access

Activation

Competition

Selection

Lexical Candidates

Bilingual spoken word recognition

• When a bilingual listener recognizes a spoken word, lexical candidates from both languages are activated

• Even if one language is not being used (Spivey &

Marian 1999, Schulpen et al 2003, Weber & Cutler 2004, Lagrou et al 2011…)

e.g. “pliers” English “playa” Spanish

Spoken word recognition: lexical access

Activation

Competition

Selection

Lexical Candidates

Code-switching

• Is a common practice among bilingual speakers

• But incurs processing costs in:

o Spoken word recognition (Soares & Grosjean, 1984)

• Spanish-English bilinguals use cues to anticipate code-switches

o Prosody (Piccinini & Garellek, 2014)

o Voice onset time (Balukas & Koops, 2014; Fricke et al, 2016)

Code-switching

• Is a common practice among bilingual speakers

• But seems to incur processing costs in:

o Spoken word recognition (Soares & Grosjean, 1984)

• Spanish-English bilinguals use cues to anticipate code-switches

o Prosody (Piccinini & Garellek, 2014)

o Voice onset time (Balukas & Koops, 2014; Fricke et al, 2016)

Mandarin-English prosodic cues

o Mandarin Chinese: lexical tone

o English: intonation

o Would expect prosodic changes when code-switching from English to Mandarin o E.g. Speaker increases F0 in order to prepare to code-

switch from English to a Mandarin word with a high level tone

Question

Do listeners use language-specific prosodic cues to anticipate a code-switch?

Study

Concept monitoring experiment

Concept monitoring

“She took the pencil from the case.”

Visual stimulus:

Audio stimulus:

Reaction time = target onset --- key press

English trial

Concept monitoring

“She bought the 杯子 for her sister.”

Visual stimulus:

Audio stimulus:

Reaction time = target onset --- key press

Code-switched trial

Participants

• 42 Mandarin-English bilinguals (35 F, 7 M)

– 18-31 years old (μ: 20)

English Mean (SD)

Mandarin Mean (SD)

Age of acquisition (yrs) 5.4 (2.7) 1.2 (0.6)

Self-rated reading (0-4) 3.8 (0.4) 3.8 (0.5)

Self-rated writing 3.7 (0.5) 3.5 (0.7)

Self-rated speaking 3.6 (0.5) 3.9 (0.4)

Self-rated understanding 3.8 (0.4) 3.9 (0.3)

Method

• Procedure

– Familiarization

– Randomized trial order

• Speaker

– Mandarin-English bilingual (F, 21)

• L1 Mandarin

• L2 English (acquired age 4)

• High self-rated proficiency for speaking + understanding both languages

Audio Stimuli

• 48 English sentences – 24 monolingual: English target nouns

• 12 sentence-medial • 12 sentence-final

– 24 code-switched: Mandarin target nouns • 12 sentence-medial • 12 sentence-final

• 2 versions of each sentence – Original – Target noun spliced in

• Each participant heard both spliced and unspliced stimuli, but never the same sentence content twice

Audio Stimuli

Condition Stimuli

English Original Recordings “She took the pencil from the case.” “She took the pencil from the case.”

Splicing “She took the pencil from the case.”

Code-switched Original Recordings “She bought the 杯子 for her sister.” “She bought the cup for her sister.”

Splicing “She bought the 杯子 for her sister.”

Splicing: if speakers produce prosodic cues and listeners use them, then manipulating the signal to remove those cues should affect the recognition process

Expected results

Condition

Expected results (Reaction time, ms)

English unspliced

English spliced

Code-switched unspliced

Code-switched spliced

Observed results

Condition

Observed results (Log-reaction time, RT in ms)

English unspliced

English spliced

Code-switched unspliced

Code-switched spliced 900

841

793

801

600 800 1000

Observed results

Condition

Observed results (Log-reaction time, RT in ms)

Final English unspliced

Final English spliced

Final Code-switched unspliced

Final Code-switched spliced

Medial English unspliced

Medial English spliced

Medial Code-switched unspliced

Medial Code-switched spliced 994

960

880

874

804

733

718

735

600 800 1000

Analysis

Log(reaction time)~Language*Splice*Position+(1+Language|Subject)+(1+Splice|Item)

• Linear mixed effects regression for log-transformed reaction time:

• Three-way interaction: Language*Splice*Position • Random slopes for Subject and Item

• Results • Longer reaction times for

• code-switches (p = 0.0595) = processing cost

• medial targets (p < 0.001)

• No significant interactions

Concept Monitoring Discussion

• Processing cost for code-switches

• Listeners don’t seem to use cues to anticipate code-switches

Concept Monitoring Discussion

• Processing cost for code-switch

• Listeners don’t seem to use cues to anticipate code-switches

Behavioral results show outcome, not process.

Question

Do listeners use language-specific prosodic cues to anticipate a code-switch?

Question

Do listeners use language-specific prosodic cues to anticipate a code-switch?

Is prosodic cue information incorporated in the lexical access of code-switches?

Study

Concept monitoring experiment

Eye tracking experiment

Method

• Same speaker as Exp 1

• Same participants as Exp 1 (excl. one)

• Different stimuli

• Procedure

– Familiarization

– Randomized trial order

Audio Stimuli

• 36 English sentences – 18 monolingual: English target nouns

• 9 sentence-medial • 9 sentence-final

– 18 code-switched: Mandarin target nouns • 9 sentence-medial • 9 sentence-final

• 2 versions of each sentence – Original – Target noun spliced in

• Each participant heard both spliced and unspliced stimuli, but never the same sentence

Visual World

Target

Filler Mandarin

phonological competitor

English phonological competitor

+

Visual stimulus:

Audio stimulus:

[Sentence containing target word]

Visual World

“We watched the goat from a distance.”

English Target

“goat”

Filler “kangaroo”

Mandarin phonological competitor “gŏu” (dog)

English phonological competitor

“gorilla”

+

English trial

Visual World

“We saw the mào zi in a tree.”

Mandarin target

“mào zi” (hat)

Filler “huā” (flower)

Mandarin phonological competitor

“mào jin” (towel)

English phonological competitor

“mouse”

+

Code-switched trial

Analyzing eye tracking data

• Growth curve analysis (Mirman, 2014)

– Advantages:

• Treats time as continuous variable

• Quantify group-level and individual-level patterns

– Orthogonal polynomials

• Capture shape of time course of fixations

Looks to Target

• Expect fewer looks to target in

– code-switched trials, compared to English trials = processing cost

– spliced, compared to unspliced, code-switched trials = absence of cues hinders listener

Looks to Target

Fewer looks to the target in: • code-switched trials (p < 0.001) • spliced code-switched medial trials (p = 0.002) •spliced trials (p < 0.001) • medial trials (p < 0.001)

~ (ot1+ot2)*Language*Splice*Position +(ot1+ot2+ot3|Subject)

Looks to Target

• Model does predict fewer looks to target in

– code-switched trials, compared to English trials = processing cost

– spliced, compared to unspliced, code-switched medial trials = absence of cues hinders listener

Looks to Mandarin Competitor

• Expect more looks to Mandarin competitor in

– code-switched trials, particularly when the target is not spliced, compared to English trials = presence of cues activates congruent language competitors

Looks to Mandarin Competitor

~ (ot1+ot2+ot3) * Language*Splice + (ot1+ot2+ot3 | Subject) + (ot1+ot2+ot3 | Subject:Language) + (ot1+ot2+ot3| Subject:Splice)

More looks to Mandarin competitor in: • code-switched trials (p < 0.001) • code-switched trials when target is unspliced, i.e. cues present (p = 0.0478)

Looks to Mandarin Competitor

• Model does predict more looks to Mandarin competitor in

– code-switched trials, particularly when the target is not spliced, compared to English trials = presence of cues activates congruent language competitors

Looks to English Competitor

• Expect more looks to English competitor in

– spliced code-switched trials = absence of cues means Mandarin words are less activated and listener is biased toward English

Looks to English Competitor

~ (ot1+ot2+ot3) * Language*Splice + (ot1+ot2+ot3 | Subject) + (ot1+ot2+ot3 | Subject:Language) + (ot1+ot2+ot3| Subject:Splice)

More looks to English competitor in: • spliced trials (p = 0.0512)

Looks to English Competitor

• Model does predict more looks to English competitor in

– Spliced, code-switched trials = absence of cues means Mandarin words are less activated and listener is biased toward English

Eye tracking discussion

• Confirms concept monitoring results: • Processing cost:

•CM: processing cost for code-switch •ET: target is less activated when code-switch

• Adds to concept monitoring results: • Using prosodic cues:

•CM: not reflected in behavioral results •ET: Prosodic cue information is incorporated into lexical access

•Mandarin competitors are more activated in code-switched trials when cues present •English competitors are more activated when cues absent

Conclusion

Theoretical

• This study provides support for bilingual listeners using cues in lexical access of code-switches.

– Prosodic cues affect lexical activation, but may not be enough to overcome processing cost, and/or

– may not be integrated quickly enough to be obvious in a reaction time task.

Conclusion

Theoretical • This study provides support for bilingual listeners using

cues in lexical access of code-switches. – Prosodic cues affect lexical activation, but may not be

enough to overcome processing cost, and/or – may not be integrated quickly enough to be obvious in a

reaction time task.

Methodological • Reinforces advantages of eye tracking studies • Position of the target word in the sentence matters • Stimuli more representative of natural speech:

sentences, not carrier phrases

Future Directions

• Relate results to participants’ language dominance and linguistic experience

– Dominance and activation?

– Frequent code-switcher better at perceiving code-switches?

• Explore more acoustic features of code-switched speech

Selected references

Balukas, C. and Koops, C., 2015. Spanish-English bilingual voice onset time in spontaneous code-switching. International Journal of Bilingualism, 19(4), pp.423-443.

Fricke, Melinda, Judith F. Kroll, and Paola E. Dussias. 2016. Phonetic variation in bilingual speech: A lens for studying the production-comprehension link. Journal of Memory and Language 89:110-137.

Lagrou, Evelyne, Robert J. Hartsuiker, andWouter Duyck. 2011. Knowledge of a second language influences auditory word

recognition in the native language. Journal of Experimental Psychology: Learning, Memory, and Cognition 37:952.

Meuter, Renata F., and Alan Allport. 1999. Bilingual language switching in naming: Asymmetrical costs of language selection. Journal of Memory and Language 40:25-40.

Mirman, Daniel. 2014. Growth curve analysis and visualization using r. CRC Press.

Piccinini, Page E., and Marc Garellek. 2014. Cross language speech-in-noise perception by early Spanish-English bilinguals and English monolinguals. The Journal of the Acoustical Society of America 135:2352.

Schulpen, Beryl, Ton Dijkstra, Herbert J. Schriefers, and Mark Hasper. 2003. Recognition of interlingual homophones in bilingual auditory word recognition. Journal of Experimental Psychology: Human Perception and Performance 29:1155.

Soares, Carlos, and Francois Grosjean. 1984. Bilinguals in a monolingual and a bilingual speech mode: The effect on lexical access. Memory and Cognition 12:380-386.

Spivey, Michael J., and Viorica Marian. 1999. Cross talk between native and second languages: Partial activation of an irrelevant lexicon. Psychological Science 10:281-284.

Tamargo, Rosa E. G., Jorge R. V. Kro, and Paola E. Dussias. 2016. Examining the relationship between comprehension and production processes in code-switched language. Journal of Memory and Language 89:138-161.

Thomas, Michael S.C., and Alan Allport. 2000. Language switching costs in bilingual visual word recognition. Journal of Memory and Language 43:44-66.

Weber, Andrea, and Anne Cutler. 2004. Lexical competition in non-native spoken-word recognition. Journal of Memory 50:1-25.

Thank you!

Advisors

Keith Johnson

Susanne Gahl

UC Berkeley Phon Lab

Ronald Sprouse

Susan Lin

LRAP

Jacqueline Ueng

Concept Monitoring Model

Log(Reaction time) ~ Language * Splice * Position + (1 + Language | Subject) + (1 + Splice | Item)

Estimate Std. Error df t value Pr(>|t|)

(Intercept) 6.56940 0.07348 59.09000 89.400 <2e-16 ***

Codeswitched-yes 0.09113 0.04766 79.03000 1.912 0.0595 .

Spliced-yes 0.04653 0.03367 58.68000 1.382 0.1722

Position-medial 0.21737 0.04620 63.27000 4.705 1.42e-05 ***

Codeswitched-yes:Spliced-yes -0.05988 0.05089 61.79000 -1.177 0.2438

Codeswitched-yes:Position-medial 0.01130 0.06449 66.3000 0.175 0.8614

Spliced-yes:Position-medial -0.01835 0.04742 58.17000 -0.387 0.7002

Codeswitched-yes:Spliced-yes:Position-medial

0.04015 0.07000 59.63000 0.574 0.5684

Target looks: model ~ (ot1+ot2)*Language*Position +(ot1+ot2+ot3|Subject)

+(ot1+ot2+ot3|Subject:Language) +(ot1+ot2+ot3|Subject:Position)

Mandarin competitor looks: model ~ (ot1+ot2+ot3)*Language*Splice +(ot1+ot2+ot3|Subject)

+(ot1+ot2+ot3|Subject:Language) +(ot1+ot2+ot3|Subject:Splice)

English competitor looks: model ~ (ot1+ot2+ot3)*Language*Splice +(ot1+ot2+ot3|Subject)

+(ot1+ot2+ot3|Subject:Language) +(ot1+ot2+ot3|Subject:Splice)

Target looks: empirical data + model fit

Target looks: empirical data + model fit

Mandarin competitor looks: empirical data + model fit

English competitor looks: empirical data + model fit