Bilateral semantic processing: Inferences in language, insight in problem solving Mark Jung-Beeman...
-
Upload
angelique-orvis -
Category
Documents
-
view
218 -
download
0
Transcript of Bilateral semantic processing: Inferences in language, insight in problem solving Mark Jung-Beeman...
Bilateral semantic processing:Inferences in language, insight in problem solving
Mark Jung-Beeman
Northwestern UniversityDepartment of Psychology
Neuroscience InstituteCognitive Brain Mapping Group
\
Bilateral semantic processing:Inferences in language, insight in problem solving
Northwestern University Drexel University Zoe Clancy John KouniosJason Haberman (UCDavis) Debbie GreenSandra Virtue (Depaul U) Jennifer Frymiare (U Wisc)Stella Arambel (deceased) Jessica Fleck Dianne Patterson Richard Greenblatt Todd Parrish
Paul Reber Bar-Ilan UniversityTerri Swan Miriam FaustKaruna Subramaniam Nira MashalEd Bowden
Research sponsored by NIDCD/NIH
OUTLINE:
• Drawing inferences from stories -- bilateral comprehension
• Three bilateral component semantic processes (to start)
• Insight -- bilateral, parallel processing during problem solving
Bilateral semantic processing:Inferences in language, insight in problem solving
Bilateral Activation, Integration, and Selection model of semantic processing
• Semantic activation - “Wernicke’s area”
– Bottom-up lexical-semantic activation: index of semantic representations (pMTG)
• Semantic integration - anterior Sup. Temp. Gyrus
– Compute semantic overlap - detect or generate (aSTG)
• Semantic selection - Inf. Frontal Gyrus
– Select among competing activated concepts (IFG)
OUTLINE:
• Drawing inferences from stories -- bilateral comprehension
• Three bilateral component semantic processes (to start)
• Insight -- bilateral, parallel processing during problem solving
Bilateral semantic processing:Inferences in language, insight in problem solving
Problems with view that language is purely a LH function
• General anatomical symmetry
• RH damaged patients - some language problems
• Recovery from aphasia, hemispherectomy, callosotomy
• Neuroimaging - always some RH signal, some tasks RH>LH
• Some tasks lvf-RH better than rvf-LH
Natural language, stories, discourse
• Higher level semantic processing (plus all lower levels)
As language input more complex (and natural):
• More anterior temporal lobes • More bilateral processing
Brain bases of comprehension of natural language
Causal bridging (coherence) inferences
“Before going to the wedding, John was sitting around in his jeans, so he went to his bedroom to find some clothes.”
Brain bases of cognitive processes whenpeople draw inferences from stories
Causal bridging (coherence) inferences
“Before going to the wedding, John was sitting around in his jeans, so he went to his bedroom to find some clothes.
He came out wearing his tuxedo, which had belonged to John's father, but looked like new.”
Brain bases of cognitive processes whenpeople draw inferences from stories
Causal bridging (coherence) inferences
“Before going to the wedding, John was sitting around in his jeans, so he went to his bedroom to find some clothes.
He came out wearing his tuxedo, which had belonged to John's father, but looked like new.”
CHANGECHANGE
Brain bases of cognitive processes whenpeople draw inferences from stories
We know people make such causal inferences
We know a lot about other types of inferences that people make - types of text, motivation, knowledge, capacity
We still don’t know much about component processes that support this seemingly complex behavior
Brain bases of cognitive processes whenpeople draw inferences from stories
RHD patients have difficulty drawing inferences
• Answer questions about inferable events less accurately than control subjects; intact on explicitly stated facts
(Brownell et al., 1986; Beeman, 1993)
• Do not show inference-related priming; control subjects do
(Beeman, 1993)
RH semantic processing and inferences
Proposed component processes of
inference generation• 1) Activation / integration (detect overlap)• 2) Selection • 3) Incorporation / integration (map overlap)
• Hemispheric cooperation
• RH activates information that may support inferences. Weak activation not reach consciousness.
Time course of inference related semantic activation
in both hemispheres during story comprehension.
“Before going to the wedding,, John was sitting around in his jeans,1 so he went to his bedroom to find some clothes.22 After a few minutes,tes,3 3 he came out wearing his tuxedo,44 which had belonged to John's father55, but was still fashionable and looked like new.”
- CHANGE- CHANGE
• (1) and (2): Predictive inference.
• (3): Transition.
• (4): Coherence or bridging inference.
• (5): Resolved and incorporated.
Right visual field
Left Hemisphere Right Hemisphere
Left visual field
“Before going to the wedding, John was sitting around in his jeans,1 so he went to his bedroom to find some clothes.2 After a few minutes,3 he came out wearing his tuxedo,4 which had belonged to John's father5, but was still fashionable and looked like new.” Brain and Language, 2000
Priming:
Inferencefaster than
Unrel
Asymmetric dynamic semantic fields: relatively coarser coding in RH; better selection in LH
footfoot
CUT
TOES
RULER
Right HemisphereLeft Hemisphere
Small but strongly activated;Focused on dominant or contextually relevant concepts- easy to select, interpret, output
Large but weakly activated;Diffuse, including secondaryand less relevant concepts- hard to select, output
foot
painglassglass pain
foot
RH coarse semantic coding: Increased likelihood of semantic overlap
for distant semantic relations
Bilateral Activation, Integration, and Selection model of semantic processing
• Semantic activation - “Wernicke’s area”
– Bottom-up lexical-semantic activation: index of semantic representations (pMTG)
• Semantic integration - anterior Sup. Temp. Gyrus
– Compute semantic overlap - detect or generate (aSTG)
• Semantic selection - Inf. Frontal Gyrus
– Select among competing activated concepts (IFG)
RH Middle & superior temporal gyrus involved in computing semantic integration
• Deriving theme from paragraphs (St. George et al.)
• Generating best ending (Kirchner et al.)
• Generating inferences? - moderately related sentence pairs (Mason & Just)
• Metaphoric over literal sentences (Bottini et al.)
• Detecting temporal/emotional inconsistency (Ferstl)
• Generating insight solutions (Jung-Beeman et al; Kounios et al)
Brain activity when people draw inferences on-line, as indexed by fMRI
Three ways to contrast inference versus no-inference conditions:
- Text: infernce versus no-inference; strong vs. weak constraint
- Individual differences: high versus low Working Memory
- Behavioral measures: recall of inferences
General Results:
Bilateral activity in pMTG; aSTG; IFG- modulated by constraint, WM, time
Brain activity when people draw inferences on-line, as indexed by fMRI
Inference:
… John was going to a wedding, but he had been sitting around the house in his jeans, so he went to his bedroom to find some clothes. Soon he came out wearing his tuxedo, * …
Explicit: …went to his bedroom to change his clothes. Soon he came out wearing his tuxedo ,* …
- High baseline, ongoing stories; small input difference
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Semantic integration at moment of implied events:Predominantly RH aSTG
L R Post Ant L R
Semantic integration at event point:Bilateral anterior Superior Temporal Gyrus
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
L RLower (ns) threshold, selected for LH STG
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Semantic activation and integration at coherence break (“tuxedo”):Predominantly LH STG
Semantic selection: High versus low working memory
High WM (reading span) subs show stronger, earlier evidence of semantic selection of inferences
(St. George et al; many behavioral)
• Completion requires selection, incorporation
Semantic selection: Inferior frontal gyrus
Selecting some concepts over competitors • Usually IFG in LH (Thompson-Schill et al; Barch;
Friston)
Some instances, RH IFG
• (Seger 2000; Friederici et al., 2000; Jung-Beeman et al.)
Semantic selection: Inferior frontal gyrus
Selecting some concepts over competitors • Usually IFG in LH (Thompson-Schill et al; Barch;
Friston)
Some instances, RH IFG• Unusual verb generation (cake -> “decorate”) (Seger 2000)
• Repair grammatical errors (Friederici et al., 2000)
• Utilize unintended meaning of ambiguous words in sentence– (Jung-Beeman et al.)
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Semantic selection: fMRI signal in IFG (LH > RH) at coherence break in High WM subs only (Fig: High WM > Low WM)
Replication and extension: Working memory and predictability
Unpredictable inferences: LH activation, IFG, pSTG• searching for connections
Predictable inferences: Bilateral activation, IFG, pSTG• building on connections
Higher WM (n=13) > lower WM (n=13): • building on connections• facile comprehension
RHpSTG
Successful integration versus continued activation: STG in High vs. Low WM subs at coherence break, Predictable inferences
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
RH IFG
High WM subsshow bilateral(stronger in RH)
Low WM show LH only
p<.001
Replication and extension: Working memory and predictability
Unpredictable inferences: LH activation, IFG, pSTG• searching for connections
Predictable inferences: Bilateral activation, IFG, pSTG• building on connections
Higher WM (n=13) > lower WM (n=13): RH activation, pSTG, IFG, and a little aSTG• building on connections• facile comprehension
Successful integration versus continued activation: STG in High vs. Low WM subs at coherence break, Predictable inferences
RH aSTG
High WM subsshow bilateral(stronger in RH)
Low WM show LH only, no aSTG
p<.005
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Replication and extension: Working memory and predictability
Unpredictable inferences: LH activation, IFG, pSTG• searching for connections
Predictable inferences: Bilateral activation, IFG, pSTG• building on connections
Higher WM (n=13) > lower WM (n=13): RH activation, pSTG, IFG, and a little aSTG
• building on connections• facile comprehension
Conclusions about inferences
• Semantic integration builds up as story hints that some event might occur: anterior STG; RH (?)
• At coherence break: integration and activation (STG), especially in LH
• completing the inference requires selection (IFG)
• RH contributes to facile inferencing/comprehension, not just kick in when demands are high
Current projects, Future directions
• Shift semantic distance for integration --> shift hemi asymmetry
• Closely tie to behavioral markers of inference activation, selection, incorporation
– Recall of inferences √
– Priming of inferences
• Successful integration versus effort of difficult integration
– Incorporation (recall study)
Recalled inferences
• If inferences recalled, must have been incorporated
• Working Memory correlates with
– total recall
– Recall of inferences
– NOT with recall of episodes w/o inferences
• Contrast fMRI signal of recalled infs versus recall episode, no infs
L R Post Ant L R
Inferences recalled versus Episode recalled, inf not recalled
L R R R
p<.005 , positive only
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Bilateral pMTG, stronger in RH RH aSTS, bilat IFG
So what?
Knowing where processing occurs informs and constrains what and how it occurs
OUTLINE:
• Drawing inferences from stories -- bilateral comprehension
• Three bilateral component semantic processes (to start)
• Insight -- bilateral, parallel processing during problem solving
Bilateral semantic processing:Inferences in language, insight in problem solving
Bilateral Activation, Integration, and Selection model of semantic processing
• Semantic activation - “Wernicke’s area”
– Bottom-up lexical-semantic activation: index of semantic representations (pMTG)
• Semantic integration - anterior Sup. Temp. Gyrus
– Compute semantic overlap - detect or generate (aSTG)
• Semantic selection - Inf. Frontal Gyrus
– Select among competing activated concepts (IFG)
Why does the RH code more coarsely?
Asymmetries in neural microcircuitry
Given topographic mapping of brain,broader input/output fields => coarser semantic coding
footfoot
CUT
TOES
RULER
Right HemisphereLeft Hemisphere
Small but strongly activated;Focused on dominant or contextually relevant concepts
Large but weakly activated;Diffuse, including secondaryand less relevant concepts
foot
painglassglass pain
foot
RH coarse semantic coding: Increased likelihood of semantic overlap
for distant semantic relations
Why a separate area for semantic integration?
• Could form associations in “activation” area
BUT
• Higher level relations, correlated co-occurrence, indirect
• Ability to extract, attend to, & manipulate relations
– Analogous to individual areas within vision (e.g., motion)
Why anterior STS/STG for semantic integration?
• Again, neural architecture
L R Post Ant L R
Patchy organization and multisensory integration(Beauchamp 2004)
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Why anterior STS/STG for semantic integration?
• Again, neural architecture
• More anterior = longer intrinsic conxns, better to integrate across patches
• RH = longer than LH
Important clarifications
• Not an “inference area”
– Semantic integration - participates in many functions
– Not specific to categories of inferences - varies with demand
• Tight comparison not reveal whole network
– Just areas that differ when storied imply versus explicitly state events
• RH and LH cooperate
OUTLINE:
• Drawing inferences from stories -- bilateral comprehension
• Three bilateral component semantic processes (to start)
• Insight -- bilateral, parallel processing in problem solving
Bilateral semantic processing:Inferences in language, insight in problem solving
Most problems solved with mix of analytic and insight processing
• Distinct computations, distributed across hemispheres, allows two approaches to proceed simultaneously (partially interactive)
• Hemispheric components, task shielding/switching
Brain bases of insight during problem solving: Aha! and antecedents
Archimedes and the crown
King’s crown - gold, or silver
Archimedes knew gold and silver differed in density
Archimedes knew weight, but couldn’t geometrically measure to
obtain volume (and compute density)
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Archimedes and the crown
Why has story persisted so long?
Archimedes and the crown
Why has story persisted so long?
• Resonates with our own experiences of
solving insight problems solving problems with
insight
Archimedes and the crown
• Solvers reach impasse (dead-end) - couldn’t measure
• Must reinterpret some aspect of problem
– Volume by water displacement
• Unconscious processing important
– If not thinking of crown, how recognize importance of water?
• Solution accompanied by “Eureka!”
Insight component processes?
Insight solutions associated with
• Switching to new strategy or associations (“restructuring”)
• Semantic integration -- solvers see connections that previously eluded them
– Right hemisphere?
Solving problems with insight
Characteristics of both “insight problems” and solving
processes similar to characteristics of discourse and
comprehension processes for which the Right Hemisphere
(RH) seems to make contributions
•Drawing inferences, understanding the gist
•Getting jokes, metaphors, connotations
•2ndary word meanings
Solving problems with insight
• Solvers reach impasse (dead-end)
• Must reinterpret some aspect of problem
• Unconscious processing important
• Solution accompanied by “Aha!”
Short insight problems:
RAT Compound Remote Associate Problems Bowden & Jung Beeman, 1998
Remote Associates Test: The RAT (Mednick, 1962)
child
scanlamesame strike
tennis
RAT Compound Remote Associate Problems Bowden & Jung Beeman, 1998
child
scanlamesame strike
tennis
Aha! experience
• Solution appears sudden and obvious
• As soon as you think of solution, you “just know” it works for all three words
– Comes as a whole, not part by part
• (vs strategic, step-by-step testing, etc)
Event-related fMRI design
• Insight solutions versus noninsight solutions
• Very “tight” comparison
– Not reveal whole network of problem solving
– Highlights just components that are uniquely engaged (or at least emphasized) for insight solutions
L R Post Ant L R
Insight effect in RH anterior Superior Temporal Gyrus: FMRI signal for insight > noninsight solutions.
L coronal R axial sagittal
p < .005, cluster > 500 mm3
-0.10
0.00
0.10
0.20
0.30
0.40
-2 2 4 6 8 10
-0.10
0.00
0.10
0.20
0.30
0.40
-2 2 4 6 8 10
Per
cent
sig
nal c
hang
e
Percent Signal change
Time (sec)
RH aSTG: Singal change across the active region
Signal change for insight Insight effectand noninsight solutions (Ins - non)
Signal change for insight and noninsight solutions, in aSTG
across hemispheres
-0.05
0.00
0.05
0.10
0.15
0.20
0.25
LH RHHemisphere
Percent signal
change
InsightNon
“Best” cluster within eachhemisphere!!
Parallel study with 128 channel EEG
•Temporal specificity
•Processing specificity - frequencies
Gamma band insight effects
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
Insight solving conclusions
Insight solutions associated with increased activity in RH aSTG
• Binding and conscious accessibility (gamma) over RH aSTG
• Preceded by visual gating (alpha) - RH temp/ occipital areas
Insight solving conclusions
Insight solutions associated with increased activity in RH aSTG
• Binding and conscious accessibility (gamma) over RH aSTG
– - Lexical or semantic integration
• Preceded by visual gating (alpha) - RH temp/ occipital areas
– - Sensory gating indicates cognitive control?
Replication plus… more areasNew data set: improved N, scanner, protocol
RH aSTG (distant semantic integration)
• Anterior Cingulate (monitoring response competition, switching)
• Posterior Cingulate - same?
• Hippocampus/parahippocampal gyri - memory, reorgnzn?
L R Post Ant L R
Insight effect in RH Superior Temporal Gyrus: FMRI signal for insight > noninsight solutions.
L coronal R axial sagittal
p < .001, cluster > 1000 mm3 ant and post STG
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
NONinsight effect in LH Inf. Frontal Gyrus: FMRI signal for NONinsight > insight solutions.
sagittal
p < .005, cluster > 1000 mm3
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
LH IFG - dominant semantic retrieval or selection
- turns on at problem onset- off at solution, esp’y Insight
RH IFG - unusual retrieval / selection
- off at problem onset- on at solution (I>NI, ns)
General vs specific mechanisms - Visual Aha!
•
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
L R Post Ant L R
Visual Aha! effect in RH anterior Mid Temporal Gyrus: FMRI signal for insight > noninsight recognition
L coronal R axial sagittal
p < .01, cluster > 500 mm3
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
L R Post Ant L R
Visual Aha! effect in RH anterior Mid Temporal Gyrus: FMRI signal for insight > noninsight recognition
L coronal R axial sagittal
p < .01, cluster > 500 mm3
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
L R Post Ant L R
Visual Aha! effect in RH Angular Gyrus: FMRI signal for insight > noninsight recognition
L coronal R axial sagittal
p < .01, cluster > 500 mm3 Also: RH Sup Frontal Gyrus
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
L R Post Ant L R
Visual Aha! effect in Bilateral M. Occipital Gyri: FMRI signal for NONinsight > insight recognition
L coronal R axial sagittal
p < .005, cluster > 500 mm3
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Visual Aha! conclusions
• NOT just for verbal problems
• Similarities - shared mechanisms (not “insight”, but…)
– Insight: top-down, cognitive control, integration
– RH -- unconscious, weak but mutually constraining, integration
– Recognition comes as a whole, not part by part
– Noninsight: bottom-up
• Some differences - Angular Gyrus somewhat surprising
General vs specific mechanisms - Visual Aha!
•
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Insight solving conclusions
Insight solutions associated with increased activity in RH aSTG
• Binding and conscious accessibility (gamma) over RH aSTG
• Preceded by visual gating (alpha) - RH temp/ occipital areas
Insight solving conclusions
Insight solutions associated with increased activity in RH aSTG
• Binding and conscious accessibility (gamma) over RH aSTG
– - Lexical or semantic integration
• Preceded by visual gating (alpha) - RH temp/ occipital areas
– - Sensory gating indicates cognitive control?
Insight solving conclusions
Insight solutions associated with
• Semantic integration -- solvers see connections that previously eluded them
• When “the light goes on…”
Bilateral Activation, Integration, and Selection model of semantic processing
• Semantic activation - “Wernicke’s area”
– Bottom-up lexical-semantic activation: index of semantic representations (pMTG)
• Semantic integration - anterior Sup. Temp. Gyrus
– Compute semantic overlap - detect or generate (aSTG)
• Semantic selection - Inf. Frontal Gyrus
– Select among competing activated concepts (IFG)
Insight preparation
Do different mental states influence how you solve problems?
• Brain activity during a “rest period” (fMRI) or at a “Ready?” prompt (EEG), prior to getting a problem
• Problems solved with insight versus without insight
Preparation for Insight
• Is there a general form of preparation for insight that begins before a problem is presented?
• We examined neural activity during the 2 sec immediately before each problem was presented.
• Compared neural activity preceding problems solved with insight to activity preceding problems solved without insight.
Conclusions
• Two forms of preparation.– Noninsight: Increased visual attention to displayed
problem.– Insight: Mobilization and control of cognitive resources;
activation of temporal lobe semantic regions; suppression of irrelevant thoughts.
Is insight really sudden? Part II: Antecedents of insight
Positive mood facilitates insight and creative problem solving (Isen et al.)
Insight and mood
Positive mood associated with increased creativity
– Better access to more distant associations
– Increased cognitive flexibility
• Anxiety associated with decreased creativity
– narrower focus of attention
Positive mood and insight
• Mood
– Positive mood enhances (anxiety impedes):
» Total solution rate
» % solved with insight
» Insight-like preparatory activity in ACC
Positive mood modulates prep activity in ACC
Insight >NonPrep activity
Pos Aff>Neg in prep activ
Convergence
General vs specific mechanisms - Visual Aha!
•
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Thank you!