Theory of language and semiotics Faculty of language and literature

Cognitive semiotics and neurodynamics.

RT: Cognitive semiotics, Cognitive science15th of September 2006

Early Fall School in Semiotics 11th to 18th of September 2006

Sozopol, Bulgaria

Wolfgang Wildgen

Theory of language and semiotics Faculty of Language and Literature

Wolfgang Wildgen2

Dynamic vs. cognitive semiotics • Dynamic semiotics starts from biological morphogenesis and expands

this thought to linguistics. In his holistic strategy René Thom (1923-2002) preferred a Gestaltist view on biology and not so much a mechanistic one, which takes the brain as the central (and finally) only “organ” responsible for thought, language and culture. This view is further developed in Wildgen (1994) introducing modes of cognitve self-organization in style of synergetics (cf. Haken and Stadler, 1991).

• The research line of cognitive semiotics was first developed in the philosophical atmosphere of logical empiricism (Quine) and formal syntax (Carnap). Whereas Chomsky created a compact mentalistic theory, Lakoff (since 1975) and with him Langacker and Talmy combined insights of gestalt-psychology with ideas stemming from issues of generative semantics. Cognitive semiotics have extended this methodology (cf. Brandt, 2002).

Theory of language and semiotics Faculty of Language and Literature

Wolfgang Wildgen3

• The current advances in neurodynamics due to the techniques of neuro-imaging and research in the neurodynamics of animals (mice, rats etc.) during perceptual and motor tasks give us a new understanding of what the brain does, when a subject (animal or human) has to solve specific problems.

• Dynamic semiotics (in Thom‘s tradition) must consider the neural “machinery” and specify the dynamics in the context of the brain.

• Cognitive semiotics must leave the theoretical isolation of structuralism, its primary concern with grammar and go beyond psychological analogies towards a foundation in the dynamics of the brain.

Ways of a neurodynamic expansion of theses models

Theory of language and semiotics Faculty of Language and Literature

Wolfgang Wildgen4

The representation of dynamics in cognitive grammar (Langacker)

Langacker's analysis of the verb ENTER.

Theory of language and semiotics Faculty of Language and Literature

Wolfgang Wildgen5

The ball kept rolling because of the wind blowing on it.

• intrinsic force tendency of the Agonist (right): towards rest (●),

• the Antagonist (left) is stronger (), • intrinsic force tendency of the Antagonist:

action (), • result of the force interaction: action ( ).

Schematisation of force-dynamics by Talmy (1988).

Description of a process in cognitive semantics (Talmy)

Theory of language and semiotics Faculty of Language and Literature

Wolfgang Wildgen6

The representation of dynamics in dynamic semantics

Derivation of a process-schema (right) from a path p in the vector-field of the cusp (A3): capture; A catches B.

Theory of language and semiotics Faculty of Language and Literature

Wolfgang Wildgen7

Are neural representations a map of external dynamics?

• Both the quasi-imaginistic and the dynamic model of meaning assume that a kind of abstract “pictures” of external events exist in the mind. Although neural representations must have an enactive capacity and be able to govern action in the external word (and social coordination in such actions), they have a very specific format and rather independent dynamics.

• At the local level, where specific observation and experimentation in a neuro-cognitive experiment can be achieved, one can not assume the appearance of such global mappings. This ma only appear at a higher level, where learned schemata are activated top-dawn. They must , however, be able to use the more basic binding dynamics explained in the forgoing sections.

Theory of language and semiotics Faculty of Language and Literature

Wolfgang Wildgen8

Shortcomings of both cognitive and dynamic semiotics

• If images filling Langacker’s articles represent only vague and fluctuating after-effects observed in the analysis of linguistic structures, on may just dispense of them and return to classical models.

• All concepts taken from gestalt psychology have since undergone a redefinition in terms of neurodynamics. Why stick to the state of the art fifty years ago?

• The concepts taken from morphogenesis by Thom (cf. his discussion with Waddingston) have also been redefined using more specific higher order knowledge on neurochemical processes.

• The neurodynamics of perception, motor programs and memory are nearer to language than basic morphogenetic processes.

Theory of language and semiotics Faculty of Language and Literature

Wolfgang Wildgen9

The dynamics of composition (Frege’s principle)

• In a simple case which avoids the complexity of verbal valence, and rather takes nominal syntax as a basic example, one may consider a noun related to the form of an object, say a “square”, an adjective of colour, say “red”, and a present participle of motion, say “moving”:

• red moving square• How does the brain compose a head-noun referring to form

with two satellites referring to colour and motion?

Theory of language and semiotics Faculty of Language and Literature

Wolfgang Wildgen10

• For Andreas Engel (2004) the binding process is one of temporal synchronization of neural assemblies, which form wholes (gestalts) from parts and one of desynchronization which distinguishes figure and ground.

• This type of analysis concerns only the composition in perception, attentiveness and memory, but one may conjecture a parallel process for words (related to perceptual information) and their composition in syntactic constructions involving nouns and adjectives.

Some elements of a model of neurolinguistic binding

Theory of language and semiotics Faculty of Language and Literature

Wolfgang Wildgen11

From: Engel et alii, 1997: 572

Parts or features of a visual whole are linked by the synchronic firing of a set of neurons (an assembly) during a short time interval.

In the example the parts and features of the cat and those of the woman are bound together by the internal synchrony of the assemblies 1 and 2 and they are distinguished by the asynchrony of these assemblies.

The basic idea of temporal binding

Theory of language and semiotics Faculty of Language and Literature

Wolfgang Wildgen12

Ambiguity and Binding

Picture a is ambiguous. If it is seen as one face (and a candle in front of it) the zones (1,2) and (3,4) (see series d) are bound; in the case two faces looking at each other are seen, the zones (1,3) und (2,4) are bound. The binding may be recognized by the synchronic firing rates in the series d versus e. From: Engel, Fries und Singer, 2001: 707

Theory of language and semiotics Faculty of Language and Literature

Wolfgang Wildgen13

Top down effects due to expectation and memory

At the left a Kanitza-triangle

At the right a non-Kanitza-triangle

If the tested person is instructed to recognize the non-Kanitza-triangle , the synchronization is higher for this configuration, although basic gestalt laws would predict the contrary.

From: Hermann, Munk und Engel, 2004:349

The remembered object produces higher synchronization at the γ-level (30 to 60 Hz)

Theory of language and semiotics Faculty of Language and Literature

Wolfgang Wildgen14

“ ‘What’ and ‘Where’ are the basic questions to be answered in visual perception. Not only must we recognize what we are looking at, but also we need to know where it is in order to respond appropriately.” Gazzaniga u.a. (1998: 165)

Two different pathways starting from the visual center V1 spezialize on the analysis of:

Where was it seen?

What was seen?

Theory of language and semiotics Faculty of Language and Literature

Wolfgang Wildgen15

Teisman (1999: 108) writes:• „It [the binding-by-synchrony hypothesis] also provides a plausible

reason for the attentional limit of around four objects that is widely observed in the perception of brief displays and in studies of visual working memory. The different firing rates that can be easily discriminated on a background of inherent noise and accidental synchronies may set a low limit to the number of objects that can be simultaneously bound.”

• The restrictions of valence patterns and of embeddings (recursive operations) have been discredited by Chomsky as performance effects. To the contrary, they are hints to the nature of the compositional process in language and thus more interesting than the algebraic notion of recursive operations.

Restrictions on compositionality due to temporal binding

Theory of language and semiotics Faculty of Language and Literature

Wolfgang Wildgen16

Some conclusions • Cognitive semiotics must try to have a neurodynamic

foundation of such basic concepts as the composition of meanings to a whole (Frege’s problem)

• Such a model should consider the topology of the brain, synchronization and desynchronization, and the coupling of subnetworks with self-organization (filtering, choice of dominant modes), self-reference and monitoring in consciousness, etc.

• As cognitive semiotics goes beyond cognitive semantics and grammar, it must also consider the composition of meaningfull entities in in the visual (art, media, architecture) and in the domain of audition (music) and motion (dance)

Theory of language and semiotics Faculty of Language and Literature

Wolfgang Wildgen17

Some bibliographical indications• Engel AK, Fries P, Singer W (2001) Dynamic predictions: oscillations and synchrony in top-

down processing. Nature Reviews Neuroscience 2: 704-716 • Engel AK, Roelfsema PR, Fries P, Brecht M, Singer W (1997) Role of the temporal domain for

response selection and perceptual binding. Cerebral Cortex 7: 571-582• Herrmann CS, Munk MHJ, Engel AK (2004) Cognitive functions of gamma-band activity:

memory match and utilization. Trends in Cognitive Sciences 8: 347-355• Gazzaniga, Michael S., Richard B. Ivry und George R. Mangun, 1998. Cognitive

Neuroscience. The Biology of the Mind, Norton, New York. • Machery, Edouard, Markus Werning, Gerhard Schurz (Hg.), 2005. The Compositionality of

Meaning, Bd. II: Applications to Linguistics, Psychology and Neuroscience, Ontos, Frankfurt/Main.

• Riesenhuber, Maximilian und Tonaso Poggio, 1999. Are Cortical Models Really Bound by the “Binding Problem”?, in: Neuron, 24: 87-93.

• Singer, Wolf, 1999. Neuronal Synchrony: A Versatile Code for the Definition of Relations?, in: Neuron 24: 49-65.

• Teisman, Anne, 1999. Solutions to the Binding Problem. Progress Through Controversy and Convergence, in: Neuron, 1999: 105-110.

• Werning, Markus, Eduard Machery, Gerhard Schurz (Hg.), 2005. The Compositionality of Meaning and Content, Bd. I: Foundational Issues, Ontos, Frankfurt/Main.

• Wildgen, Wolfgang, 1994a. Process, Image, and Meaning. A Realistic Model of the Meanings of Sentences and Narrative Texts, Benjamins, Amsterdam.

Theory of language and semiotics Faculty of language and literature

My home page is:http://www.fb10.uni-bremen.de/home

pages/wildgen.htm

You may find my presentation in Sozopol (when I have returned) together with

other conference papers of the last two years in a list at the end of my

homepage.